Re: [HACKERS] Buildfarm "master-next" branch?
On Tue, Apr 29, 2014 at 9:11 PM, Jim Nasby wrote: > On 4/17/14, 9:38 AM, Tom Lane wrote: > >> But the ability to easily spin up temporary branches for testing would >>also be great. Unfortunately, I suspect that only a minority of the >>buildfarm owners would choose to participate, which would make it less >>useful, but if we could solve that problem I'd be all in favor of it. >>> >... Of course, all this would be done in my copious spare time*cough*. >>> I'm >>> >>> >not sure this would be the best use of it. >>> >> I agree that this would not be worth the effort needed to make it happen. >> > > There's also a sizeable security risk there, of someone putting something > malicious in a branch and then triggering a run from that branch. I suppose > that could be overcome if this was purposefully limited to the main git > repo that only our core committers had access to, but we'd need to be > careful. I would suggest a separate repo to keep the main one "clean", but other than that, yes, it would have to be limited to the same committers as the rest I think. It's reasonably easy to set up build environments in containers/jais on many Unix boxes where that would actually not be a problem (just blow the whole jail away once the build is complete), but one of the main platforms that people would want to use this on I bet is Windows, which has no such facilities AFAIK. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/
Re: [HACKERS] Buildfarm "master-next" branch?
On 4/17/14, 9:38 AM, Tom Lane wrote: But the ability to easily spin up temporary branches for testing would >>also be great. Unfortunately, I suspect that only a minority of the >>buildfarm owners would choose to participate, which would make it less >>useful, but if we could solve that problem I'd be all in favor of it. >... Of course, all this would be done in my copious spare time*cough*. I'm >not sure this would be the best use of it. I agree that this would not be worth the effort needed to make it happen. There's also a sizeable security risk there, of someone putting something malicious in a branch and then triggering a run from that branch. I suppose that could be overcome if this was purposefully limited to the main git repo that only our core committers had access to, but we'd need to be careful. -- Jim C. Nasby, Data Architect j...@nasby.net 512.569.9461 (cell) http://jim.nasby.net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Buildfarm "master-next" branch?
On 04/17/2014 10:38 PM, Tom Lane wrote: > IMO the best single thing that could happen for the buildfarm is if > we had more critters (at least twice as many) running a wider variety of > platforms, compilers, and configuration options than there are today. > More frequent runs would come out of that automatically. I'll be bringing up a new Windows buildfarm member once I've got a current project knocked off. It's a pretty fast dedicated Windows Server 2012 box with a wide range of SDKs on it that can do 32-bit and 64-bit builds. Should help a little. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Buildfarm "master-next" branch?
Andrew Dunstan writes: > On 04/17/2014 09:17 AM, Robert Haas wrote: >> In terms of improving the buildfarm infrastructure, the thing I would >> most like to have is more frequent runs. IMO the best single thing that could happen for the buildfarm is if we had more critters (at least twice as many) running a wider variety of platforms, compilers, and configuration options than there are today. More frequent runs would come out of that automatically. >> ... But that would require more resources for the >> buildfarm machines, which are provided on a strictly volunteer basis, >> so it's hard to see how to arrange that. I don't think we've tried hard lately to get people to sign up. Maybe we should ask the -advocacy crew to do something. >> But the ability to easily spin up temporary branches for testing would >> also be great. Unfortunately, I suspect that only a minority of the >> buildfarm owners would choose to participate, which would make it less >> useful, but if we could solve that problem I'd be all in favor of it. > ... Of course, all this would be done in my copious spare time *cough*. I'm > not sure this would be the best use of it. I agree that this would not be worth the effort needed to make it happen. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Buildfarm "master-next" branch?
On 04/17/2014 09:17 AM, Robert Haas wrote: In terms of improving the buildfarm infrastructure, the thing I would most like to have is more frequent runs. It would be great if pushing a commit to the master repository triggered an immediate build on every buildfarm animal so that you could see all of the failures in a short period of time. But that would require more resources for the buildfarm machines, which are provided on a strictly volunteer basis, so it's hard to see how to arrange that. Some buildfarm owners run at pretty high frequency - I know there are cron jobs on some running every 15 minutes. My own Linux and FBSD machines run every hour. Windows builds take longer - depending on other use of resources they can run a couple of hours per branch. Also my two Windows machines doing buildfarm work are running a total of 5 animals, so the runs are staggered - on Windows 8 the two animals each run every 3 hours. Note that each run potentially builds all the branches, if there has been some backported change, and the windows animals are set up so that if animal A on the same machine is running when animal B's run time comes around animal B skips it scheduled run. So sometimes you do have to wait a bit. If someone were to providfe me with a bunch of nice fast Windows VMs I would set them up with one animal a piece with frequent runs and we might get a lot better coverage. But I am tapped out as far as the resources I can provide go. But the ability to easily spin up temporary branches for testing would also be great. Unfortunately, I suspect that only a minority of the buildfarm owners would choose to participate, which would make it less useful, but if we could solve that problem I'd be all in favor of it. I'm not volunteering to do the work, though. The buildfarm's original purpose was to give early warning of platform-specific problems of code we had *already* decided on. Now projects morph, so we might decide to do something like this. But we'd need to think long and hard about it. Postgres has not historically used short-lived branches. I don't much like Craig's idea of a long-lived testing branch that we're going to do commits and reverts on. If we're going to do something like this it would be much better to make some provision for short-lived topic branches. e.g. say we allowed branches with names like testme-platformname-featurename, (platformname here could be a magic "all", or a comma-separated list of names such as linux, freebsd, windows). Wnen testing is done, we could merge the branch if the testing worked out OK, or drop it if the testing proved to be a failure. There would be some work to make the buildfarm client suitable for this. And we'd probably need a "testing dashboard" so as to keep the main dashboard page free of test branch results. Of course, all this would be done in my copious spare time *cough*. I'm not sure this would be the best use of it. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Buildfarm "master-next" branch? (was: Dynamic Shared Memory stuff)
On Wed, Apr 16, 2014 at 8:24 PM, Craig Ringer wrote: > On 04/17/2014 12:08 AM, Robert Haas wrote: >> On Tue, Apr 15, 2014 at 10:46 PM, Amit Kapila >> wrote: >>> On Wed, Apr 16, 2014 at 3:01 AM, Robert Haas wrote: On Tue, Apr 15, 2014 at 12:33 AM, Amit Kapila wrote: > On Mon, Apr 14, 2014 at 10:03 PM, Robert Haas > wrote: >> For the create case, I'm wondering if we should put the block that >> tests for !hmap *before* the _dosmaperr() and check for EEXIST. What >> is your opinion? > > Either way is okay, but I think the way you are suggesting is better as it > will make code consistent with other place (PGSharedMemoryCreate()). OK, can you prepare a patch? >>> >>> Please find attached patch to address this issue. >>> One minor point to note is that now we have to call GetLastError() twice, >>> once inside error path and once to check EEXIST, but I think that is okay >>> as existing code in PGSharedMemoryCreate() does it that way. >> >> OK. I committed this blindly, but I don't have a Windows dev >> environment, so please keep an eye on the Windows buildfarm members >> and provide follow-on patches if any of them get unhappy about this. > > Given that we're doing this a fair bit, is it reasonable to define a > "master-next" branch in git and have the buildfarm (or at least the > Windows members) build that? > > Permit master-next to be rebased and reset. > > That way it's possible to fire stuff off and see what happens on the > buildfarm without introducing broken commits unnecessarily. > > Thoughts? In this particular case, I have a lot of confidence that Amit tested this on his own machine before sending in the patch; and moreover, he wrote that code in the first place. So it's no worse than it would have been if that change had been in the originally committed version, which I didn't personally test on Windows, either, but which has nevertheless mostly passed buildfarm testing. Arguably, if I'm going to be hacking on platform-dependent things very often, I should get my own Windows build environment set up so that I can test it myself, but it hasn't quite been worth it to me thus far, and Amit has proven to be pretty reliable in terms of getting things right. In terms of improving the buildfarm infrastructure, the thing I would most like to have is more frequent runs. It would be great if pushing a commit to the master repository triggered an immediate build on every buildfarm animal so that you could see all of the failures in a short period of time. But that would require more resources for the buildfarm machines, which are provided on a strictly volunteer basis, so it's hard to see how to arrange that. But the ability to easily spin up temporary branches for testing would also be great. Unfortunately, I suspect that only a minority of the buildfarm owners would choose to participate, which would make it less useful, but if we could solve that problem I'd be all in favor of it. I'm not volunteering to do the work, though. Honestly, I don't think we have a huge problem here today. Yeah, the buildfarm turns pretty colors on a fairly regular basis, but those issues are also generally fixed very quickly. With the unfortunate exception of the seemingly never-ending stream multixact-related bugs, a user who took a snapshot of our master branch at a randomly selected point during the 9.4 development cycle would likely have gotten code reliable enough to be run in production. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Buildfarm "master-next" branch? (was: Dynamic Shared Memory stuff)
On 04/17/2014 12:08 AM, Robert Haas wrote: > On Tue, Apr 15, 2014 at 10:46 PM, Amit Kapila wrote: >> On Wed, Apr 16, 2014 at 3:01 AM, Robert Haas wrote: >>> On Tue, Apr 15, 2014 at 12:33 AM, Amit Kapila >>> wrote: On Mon, Apr 14, 2014 at 10:03 PM, Robert Haas wrote: > For the create case, I'm wondering if we should put the block that > tests for !hmap *before* the _dosmaperr() and check for EEXIST. What > is your opinion? Either way is okay, but I think the way you are suggesting is better as it will make code consistent with other place (PGSharedMemoryCreate()). >>> >>> OK, can you prepare a patch? >> >> Please find attached patch to address this issue. >> One minor point to note is that now we have to call GetLastError() twice, >> once inside error path and once to check EEXIST, but I think that is okay >> as existing code in PGSharedMemoryCreate() does it that way. > > OK. I committed this blindly, but I don't have a Windows dev > environment, so please keep an eye on the Windows buildfarm members > and provide follow-on patches if any of them get unhappy about this. Given that we're doing this a fair bit, is it reasonable to define a "master-next" branch in git and have the buildfarm (or at least the Windows members) build that? Permit master-next to be rebased and reset. That way it's possible to fire stuff off and see what happens on the buildfarm without introducing broken commits unnecessarily. Thoughts? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
