Re: [HACKERS] removing old ports and architectures
On 10/17/2013 09:45 AM, Robert Haas wrote: >>> According to http://en.wikipedia.org/wiki/UnixWare, UnixWare is not >>> dead, although there have been no new releases in 5 years. Gee, I wonder why? I'll point out that SCO laid off all of its packagers three or four years ago. So nobody is packaging PostgreSQL for Unixware anymore. In general, I think we can reasonably expect that anyone still using Unixware isn't upgrading PostgreSQL, so if there's any effort at all in maintaining the port, we should dump it. The only reason we *have* the port in the first place is that SCO created it and used to maintain it. Oh, and +1 to dumping all of those other ports (IRIX, Tru64, Alpha), especially Alpha which I've been told is a maintenance issue. If we're concerned that there are users out there, the answer is to do some blogging and see if anyone speaks up. I'll post something. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Fri, Oct 18, 2013 at 8:04 PM, Peter Geoghegan wrote: > On Fri, Oct 18, 2013 at 9:55 AM, Ants Aasma wrote: >> FWIW, I think that if we approach coding lock free algorithms >> correctly - i.e. "which memory barriers can we avoid while being >> safe", instead of "which memory barriers we need to add to become >> safe" - then supporting Alpha isn't a huge amount of extra work. > > Alpha is completely irrelevant, so I would not like to expend the > tiniest effort on supporting it. If there is someone using a very much > legacy architecture like this, I doubt that even they will appreciate > the ability to upgrade to the latest major version. It's mostly irrelevant and I wouldn't shed a tear for Alpha support, but I'd like to point out that it's a whole lot less irrelevant than some of the architectures being discussed here. The latest Alpha machines were sold only 6 years ago and supported up to 512GB of memory with 64 1.3 GHz cores, something that can run a very reasonable database load even today. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-18 18:36:03 +0200, Stefan Kaltenbrunner wrote: > On 10/18/2013 06:29 PM, Andres Freund wrote: > > On 2013-10-18 18:24:58 +0200, Stefan Kaltenbrunner wrote: > >> hmm there are still some operating systems that "officially" support the > >> alpha architecture which will likely result in problems for their ports. > >> One example is OpenBSD both the current version (5.3) as well as the > >> upcoming release do fully support alpha and have binary packages and > >> source "ports" for postgresql and afaik they have no intention to stop > >> supporting that plattform. > > > > Hm. If you read their status page (which I think you linked to before): > > http://openbsd.org/alpha.html you can find stuff like X11 not > > working. So I don't see that forcing us to much. > > not sure that page is acurate and not sure how relevant X11 support is > for postgresql :) Only in as much as it's a major piece of software missing. So removing pg isn't exactly a shocking thing. > Anyway they do currently have packages (9.2 in -stable and 9.3 in > -current) available and I think we should consider packagers here as > well - I personally don't have any particular need for alpha but it is > clearly not as dead as some of the others we are discussing. > > > > Note also that we already don't support all openbsd platforms. > > sure - but does that also mean we should desupport without at least > considering it? We should consider it, yes. But I don't see it counting much in this case. Multiplatform OSs like (free|open|net)bsd and linux will mostly be the last to drop support for a platform. It will very rarely be the applications first. And openbsd still supporting it doesn't change the fact that there hasn't been new hardware for 10 years by the time 9.4 is going to be released. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Fri, Oct 18, 2013 at 9:55 AM, Ants Aasma wrote: > FWIW, I think that if we approach coding lock free algorithms > correctly - i.e. "which memory barriers can we avoid while being > safe", instead of "which memory barriers we need to add to become > safe" - then supporting Alpha isn't a huge amount of extra work. Alpha is completely irrelevant, so I would not like to expend the tiniest effort on supporting it. If there is someone using a very much legacy architecture like this, I doubt that even they will appreciate the ability to upgrade to the latest major version. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Thu, Oct 17, 2013 at 3:10 PM, Robert Haas wrote: > On Thu, Oct 17, 2013 at 12:22 AM, Noah Misch wrote: >> Removing support for alpha is a different animal compared to removing support >> for non-gcc MIPS and most of the others in your list. A hacker wishing to >> restore support for another MIPS compiler would fill in the assembly code >> blanks, probably using code right out of an architecture manual. A hacker >> wishing to restore support for alpha would find himself auditing every >> lock-impoverished algorithm in the backend. > > I had much the same thought last night. So I reverse my vote on > Alpha: let's drop it. I had thought that perhaps there'd be some > value in keeping it to force ourselves to consider what will happen > under the weakest generally-understood memory model, but in fact > that's probably a doomed effort without having the hardware available > to test the code. As you say, any future atomics support for such a > platform will be a major undertaking. FWIW, I think that if we approach coding lock free algorithms correctly - i.e. "which memory barriers can we avoid while being safe", instead of "which memory barriers we need to add to become safe" - then supporting Alpha isn't a huge amount of extra work. We only need a couple of extra barriers after atomic reads where I think we should have a comment anyway explaining that we don't need a read barrier because a data dependency provides ordering. In general I agree that we are unlikely to provide a bug free result without a build farm animal, so I'm ±0 on removing support. We can try to support, but we are unlikely to succeed. I also find it unlikely that anyone will create a new architecture with a similarly loose memory model. The experience with Alpha and other microprocessors shows that the extra hardware needed for fast and strong memory ordering guarantees more than pays for itself in performance. Regards, Ants Aasma -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 10/18/2013 06:29 PM, Andres Freund wrote: > On 2013-10-18 18:24:58 +0200, Stefan Kaltenbrunner wrote: >> On 10/18/2013 02:41 PM, Robert Haas wrote: >>> On Thu, Oct 17, 2013 at 5:41 PM, Peter Eisentraut wrote: On 10/17/13 12:45 PM, Robert Haas wrote: > The attached patch, which I propose to apply relatively soon if nobody > objects, removes the IRIX port. +1 >>> >>> Done. And here's a patch for removing the alpha architecture and >>> Tru64 UNIX (aka OSF/1) which runs on that architecture, per discussion >>> upthread. Barring objections, I'll apply this next week. >> >> hmm there are still some operating systems that "officially" support the >> alpha architecture which will likely result in problems for their ports. >> One example is OpenBSD both the current version (5.3) as well as the >> upcoming release do fully support alpha and have binary packages and >> source "ports" for postgresql and afaik they have no intention to stop >> supporting that plattform. > > Hm. If you read their status page (which I think you linked to before): > http://openbsd.org/alpha.html you can find stuff like X11 not > working. So I don't see that forcing us to much. not sure that page is acurate and not sure how relevant X11 support is for postgresql :) Anyway they do currently have packages (9.2 in -stable and 9.3 in -current) available and I think we should consider packagers here as well - I personally don't have any particular need for alpha but it is clearly not as dead as some of the others we are discussing. > Note also that we already don't support all openbsd platforms. sure - but does that also mean we should desupport without at least considering it? Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-18 18:24:58 +0200, Stefan Kaltenbrunner wrote: > On 10/18/2013 02:41 PM, Robert Haas wrote: > > On Thu, Oct 17, 2013 at 5:41 PM, Peter Eisentraut wrote: > >> On 10/17/13 12:45 PM, Robert Haas wrote: > >>> The attached patch, which I propose to apply relatively soon if nobody > >>> objects, removes the IRIX port. > >> > >> +1 > > > > Done. And here's a patch for removing the alpha architecture and > > Tru64 UNIX (aka OSF/1) which runs on that architecture, per discussion > > upthread. Barring objections, I'll apply this next week. > > hmm there are still some operating systems that "officially" support the > alpha architecture which will likely result in problems for their ports. > One example is OpenBSD both the current version (5.3) as well as the > upcoming release do fully support alpha and have binary packages and > source "ports" for postgresql and afaik they have no intention to stop > supporting that plattform. Hm. If you read their status page (which I think you linked to before): http://openbsd.org/alpha.html you can find stuff like X11 not working. So I don't see that forcing us to much. Note also that we already don't support all openbsd platforms. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 10/18/2013 02:41 PM, Robert Haas wrote: > On Thu, Oct 17, 2013 at 5:41 PM, Peter Eisentraut wrote: >> On 10/17/13 12:45 PM, Robert Haas wrote: >>> The attached patch, which I propose to apply relatively soon if nobody >>> objects, removes the IRIX port. >> >> +1 > > Done. And here's a patch for removing the alpha architecture and > Tru64 UNIX (aka OSF/1) which runs on that architecture, per discussion > upthread. Barring objections, I'll apply this next week. hmm there are still some operating systems that "officially" support the alpha architecture which will likely result in problems for their ports. One example is OpenBSD both the current version (5.3) as well as the upcoming release do fully support alpha and have binary packages and source "ports" for postgresql and afaik they have no intention to stop supporting that plattform. > > On a related note, I think we should update the paragaraph in > installation.sgml that begins "In general, PostgreSQL can be expected > to work on these CPU architectures". Any architecture that doesn't > have a buildfarm animal should be relegated to the second sentence, > which reads "Code support exists for ... but these architectures are > not known to have been tested recently." Similarly, I think the > following paragraph should be revised so that only operating systems > for which we have current buildfarm support are considered fully > supported. Others should be relegated to a sentence later in the > paragraph that says something like "code support exists but not tested > recently" or "expected to work but not tested regularly". seems like an improvement to me. Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Fri, Oct 18, 2013 at 9:39 AM, Tim Kane wrote: > Just to be pedantic, commit message shows > "support for Tru64 ended in 201." > > I think you mean 2012. Duh, I'm a dork. Thanks. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
Just to be pedantic, commit message shows "support for Tru64 ended in 201." I think you mean 2012. On 18/10/2013 13:41, "Robert Haas" wrote: >On Thu, Oct 17, 2013 at 5:41 PM, Peter Eisentraut wrote: >> On 10/17/13 12:45 PM, Robert Haas wrote: >>> The attached patch, which I propose to apply relatively soon if nobody >>> objects, removes the IRIX port. >> >> +1 > >Done. And here's a patch for removing the alpha architecture and >Tru64 UNIX (aka OSF/1) which runs on that architecture, per discussion >upthread. Barring objections, I'll apply this next week. > >On a related note, I think we should update the paragaraph in >installation.sgml that begins "In general, PostgreSQL can be expected >to work on these CPU architectures". Any architecture that doesn't >have a buildfarm animal should be relegated to the second sentence, >which reads "Code support exists for ... but these architectures are >not known to have been tested recently." Similarly, I think the >following paragraph should be revised so that only operating systems >for which we have current buildfarm support are considered fully >supported. Others should be relegated to a sentence later in the >paragraph that says something like "code support exists but not tested >recently" or "expected to work but not tested regularly". > >-- >Robert Haas >EnterpriseDB: http://www.enterprisedb.com >The Enterprise PostgreSQL Company > >-- >Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) >To make changes to your subscription: >http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Thu, Oct 17, 2013 at 5:41 PM, Peter Eisentraut wrote: > On 10/17/13 12:45 PM, Robert Haas wrote: >> The attached patch, which I propose to apply relatively soon if nobody >> objects, removes the IRIX port. > > +1 Done. And here's a patch for removing the alpha architecture and Tru64 UNIX (aka OSF/1) which runs on that architecture, per discussion upthread. Barring objections, I'll apply this next week. On a related note, I think we should update the paragaraph in installation.sgml that begins "In general, PostgreSQL can be expected to work on these CPU architectures". Any architecture that doesn't have a buildfarm animal should be relegated to the second sentence, which reads "Code support exists for ... but these architectures are not known to have been tested recently." Similarly, I think the following paragraph should be revised so that only operating systems for which we have current buildfarm support are considered fully supported. Others should be relegated to a sentence later in the paragraph that says something like "code support exists but not tested recently" or "expected to work but not tested regularly". -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company remove-alpha-arch.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 10/17/13 12:45 PM, Robert Haas wrote: > The attached patch, which I propose to apply relatively soon if nobody > objects, removes the IRIX port. +1 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Wed, Oct 16, 2013 at 1:25 PM, Stefan Kaltenbrunner wrote: > On 10/16/2013 07:04 PM, Robert Haas wrote: >> On Sat, Oct 12, 2013 at 8:46 PM, Andres Freund >> wrote: >>> I think we should remove support the following ports: >>> - IRIX >>> - UnixWare >>> - Tru64 >> >> According to http://en.wikipedia.org/wiki/IRIX, IRIX has been >> officially retired. The last release of IRIX was in 2006 and support >> will end in December of 2013. Therefore, it will be unsupported by >> the time PostgreSQL 9.4 is released. >> >> According to http://en.wikipedia.org/wiki/UnixWare, UnixWare is not >> dead, although there have been no new releases in 5 years. >> >> According to http://en.wikipedia.org/wiki/Tru64_UNIX, Tru64 has been >> officially retired. Support ended in December, 2012. This seems safe >> to remove. >> >> So I vote for removing IRIX and Tru64 immediately, but I'm a little >> more hesitant about shooting UnixWare, since it's technically still >> supported. >> >>> Neither of those are relevant. > > agreed The attached patch, which I propose to apply relatively soon if nobody objects, removes the IRIX port. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company remove-irix-port.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Thu, Oct 17, 2013 at 12:22 AM, Noah Misch wrote: > Removing support for alpha is a different animal compared to removing support > for non-gcc MIPS and most of the others in your list. A hacker wishing to > restore support for another MIPS compiler would fill in the assembly code > blanks, probably using code right out of an architecture manual. A hacker > wishing to restore support for alpha would find himself auditing every > lock-impoverished algorithm in the backend. I had much the same thought last night. So I reverse my vote on Alpha: let's drop it. I had thought that perhaps there'd be some value in keeping it to force ourselves to consider what will happen under the weakest generally-understood memory model, but in fact that's probably a doomed effort without having the hardware available to test the code. As you say, any future atomics support for such a platform will be a major undertaking. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Wed, Oct 16, 2013 at 10:04:29PM +0200, Andres Freund wrote: > On 2013-10-16 15:49:54 -0400, Bruce Momjian wrote: > > On Sat, Oct 12, 2013 at 06:35:00PM -0700, Peter Geoghegan wrote: > > > > - ALPHA (big pain in the ass to get right, nobody uses it anymore) > > > > > > Yes, for many years now ALPHA has only been useful as a way of > > > illustrating how bad it's possible for CPU memory operation reordering > > > considerations to get. So I quite agree. > > > > Are there any optimizations we have avoided, or 'volatile' designations > > added, only for Alpha? > > I am somewhat sure that some of the code we added in the last years > isn't actually correct for alpha (and others actually). It's just that > nobody actually runs on alpha anymore, so nobody notices. > > > Could we improve other things if Alpha support was dropped? > > I think the major thing is that if we're going to add more algorithms > that use less locks - which we'll have to, otherwise our scalability > will get more and more problematic - we'll have to adhere to the > weakest cache coherency model we support. And at least I am not > intelligent/experienced enough to blindly write correct code for Alpha. Removing support for alpha is a different animal compared to removing support for non-gcc MIPS and most of the others in your list. A hacker wishing to restore support for another MIPS compiler would fill in the assembly code blanks, probably using code right out of an architecture manual. A hacker wishing to restore support for alpha would find himself auditing every lock-impoverished algorithm in the backend. At the same time, as you suggest, the benefit to general PostgreSQL development from dropping alpha is greater in the same way. Dropping non-gcc MIPS reduces effort to complete the initial patch that adds the atomic primitives; dropping alpha reduces effort to implement every future algorithm that uses barriers. I do think dropping support for alpha is the right decision. That's a firm and likely one-way downgrade of support, much like we've done for compilers with no 64-bit type. Concerning cases like non-gcc MIPS, I'll mostly echo Tom's comments[1]. I'm comfortable with the project saying "We've added atomics for the architectures we have. Help us with the rest!" It's reasonable to introduce an improvement that entails architecture-dependent code without requiring that the initial patch and initial author address every interesting target. The bar to restore "support", even years later, will be pretty low. In that light, I don't favor ripping out longstanding architecture-specific code for borderline platforms. Doing so raises the bar for restoring support, without proximate benefit. [1] http://www.postgresql.org/message-id/4694.1381676...@sss.pgh.pa.us -- Noah Misch EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-16 16:10:18 -0400, Robert Haas wrote: > (though I don't see the code you're talking about wrt/32bit sparc) < v9 sparc doesn't support compare-and-swap like operations, that's the background. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-16 16:10:18 -0400, Robert Haas wrote: > On the other hand, I'm not convinced that we don't need to give any > thought to UNIX vendors that are still pushing their proprietary > compilers. Many of the old players are dead, but IBM's ICC and HP's > aCC definitely aren't, and I wouldn't be surprised to find one or two > other big ones as well, plus maybe half-a-dozen others that are > clinging to life. acc provides intrinsics for IA64, so that's easily supported. For IBM, do you mean XLC? If so, that provides intrinsics as well. So does sun studio. ICC as in Intel's compiler provides intrinsics as well. In fact, that's where gcc cribbed most of it's API from. > > Which? We only seem to disagree about M32R and m68k, right? I've > > recanted mips and superh days ago ;). > > If you find alpha important, that's fine, that's supported by gcc. I > > just doubt we'll get it even remotely right without tests... > > As far as spinlock support goes, you proposed removing VAX, univel, > sinix, sun3, natsemi 32k, superH, ALPHA, m86k, M32R, mips non-GCC, > s390 non-GCC, and 32bit/ those (though I don't see the code you're talking about wrt/32bit sparc), and you withdrew 2 of those suggestions, so I think there are > 6 that are still in doubt: vax, univel, ALPHA, m32r, m68k, and s390 > non-GCC. Well, you didn't sound like you deemed vax and alpha to be that important and I was only talking about architectures, not OSs... But anyway hardware architecture wise, all but m32r, m68k and pa-risc have the required hardware support for atomic add and cmpxchg. univel/unixware: Supports only x86 (these days at least), so writing the required assembly is trivial if it comes to that. Testing on the other hand... s390, s390x: We only support linux anyway, so I don't see the restriction to gcc being problematic. alpha: We use gcc inline assembly currently, so it's only gcc again. It is supported by gcc's intrinsics. We can easily support it if we trust ourselves to understand the cache coherency. mips: Besides gcc we only support IRIX, since you voted to remove that, the restriction to gcc doesn't cost anything. m68k: Only coldfire chips don't have working CAS. All support atomic add. m32r: Doesn't have CAS, doesn't support atomic add. pa-risc: Doesn't have CAS, doesn't support atomic add. > I do understand that it's going to be painful to carry multiple > implementations of some of these facilities. But at least from where > I sit I'm not sure we really have a choice. If we have a > --disable-atomics option, there's no reason we can't have regular > buildfarm coverage of that code; broken lwlocks tend to make things > die pretty horribly, so I'm relatively confident bugs will show up. I am pretty sure lots of that code will only be noticeably under noticeable concurrency. And we don't exercise that all that much in the regression tets. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Wed, Oct 16, 2013 at 3:26 PM, Andres Freund wrote: >> I don't agree with that policy. Sure, 97% of our users are probably >> running Linux, Windows, MacOS X, or one of the fairly-popular BSD >> variants. But I think a part of the appeal of PostgreSQL is that it is >> cross-platform, and doesn't require a lot of special hardware support, >> and I'm loathe to give that up. It's one thing to say, gee, we don't >> know whether this is actually going to compile and work on your >> platform, because it's not tested. It's quite something else to say, >> anything that's not on this short list of supported platforms has zero >> chance of working without jumping through major hoops. > > Well, I am not advocating to break platforms just because they are not > on the buildfarm, but I don't see how we can introduce new facilities > that require platform support if we don't have any way to test them on > some platforms. Well, on that we agree. But my answer to that is - any facilities that require additional platform support had better be optional. If we can't do it without raising the bar for platform support, then we don't do it. > I think by using architecture independent, compiler provided intrinsics > we take care of that for most non-mainstream platforms. Just about > everything even remotely new that is/will be out there is/will be using gcc > or something compatible to it (e.g. llvm). Those provide the __sync_* > intrinsics we'd wrap. > That's incidentally why I am advocating including pg_atomic_clear and > pg_atomic_test_set in the set of supported atomics. With those porting > to a new platform will often require exactly zero changes since > spinlocks will just use them. I grant that helps a great deal. The fact that gcc has added those atomics makes this a lot more feasible to contemplate that it would be otherwise. Now, given some of our other experiences with gcc, I am slightly worried that there may be bugs. But if that turns out to be the case we can figure out what to do about it. It's very nice to at least have a place to start. On the other hand, I'm not convinced that we don't need to give any thought to UNIX vendors that are still pushing their proprietary compilers. Many of the old players are dead, but IBM's ICC and HP's aCC definitely aren't, and I wouldn't be surprised to find one or two other big ones as well, plus maybe half-a-dozen others that are clinging to life. > Which? We only seem to disagree about M32R and m68k, right? I've > recanted mips and superh days ago ;). > If you find alpha important, that's fine, that's supported by gcc. I > just doubt we'll get it even remotely right without tests... As far as spinlock support goes, you proposed removing VAX, univel, sinix, sun3, natsemi 32k, superH, ALPHA, m86k, M32R, mips non-GCC, s390 non-GCC, and 32bit/> It's hard to say where to draw the line here. I don't want the >> illusion of support for platforms that don't in fact have a prayer of >> working to prevent us from making needed improvements. On the other >> hand, I also don't want to blithely rip out support for architectures >> that people may well still be using and where, in some cases, people >> have gone out of their way to add that support. If we get to the >> point where some relatively-obscure architecture is the only thing >> standing between us and improvement X, then we can weigh those things >> against each other and decide. But I don't really want to go rip out >> support for half-a-dozen semi-supported architectures without some >> clear goal in mind. That just doesn't seem friendly. > > Well, I only started to look at this somewhat seriously because more and > more people in the last year or so, both onlist and towards 2ndq were > complaining about massive spinlock contention (up to 97% spent in > s_lock) on somewhat bigger machines. The primary offender in many > workloads is the lwlock internal spinlock, quite often for LW_SHARED > acquisition where we shouldn't need to block. > > That triggered developing the wait-free LW_SHARED patch > http://www.postgresql.org/message-id/20130926225545.gb26...@awork2.anarazel.de > which indeed shows quite some promise. Unfortunately it pretty > fundamentally requires compare_swap and fetch_add. Now we could just > implement lwlocks in two pretty much independent ways, but that seems to > be pretty bad from a maintainability POV. > > The next big thing after that is getting rid of spinlocks around buffer > pinning (and by that in buffer hdr locking). That would end up in quite > some #ifdef's sprinkled around. I do understand that it's going to be painful to carry multiple implementations of some of these facilities. But at least from where I sit I'm not sure we really have a choice. If we have a --disable-atomics option, there's no reason we can't have regular buildfarm coverage of that code; broken lwlocks tend to make things die pretty horribly, so I'm relatively confident bugs will show up. Obviously,
Re: [HACKERS] removing old ports and architectures
On 2013-10-16 15:49:54 -0400, Bruce Momjian wrote: > On Sat, Oct 12, 2013 at 06:35:00PM -0700, Peter Geoghegan wrote: > > > - ALPHA (big pain in the ass to get right, nobody uses it anymore) > > > > Yes, for many years now ALPHA has only been useful as a way of > > illustrating how bad it's possible for CPU memory operation reordering > > considerations to get. So I quite agree. > > Are there any optimizations we have avoided, or 'volatile' designations > added, only for Alpha? I am somewhat sure that some of the code we added in the last years isn't actually correct for alpha (and others actually). It's just that nobody actually runs on alpha anymore, so nobody notices. > Could we improve other things if Alpha support was dropped? I think the major thing is that if we're going to add more algorithms that use less locks - which we'll have to, otherwise our scalability will get more and more problematic - we'll have to adhere to the weakest cache coherency model we support. And at least I am not intelligent/experienced enough to blindly write correct code for Alpha. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Sat, Oct 12, 2013 at 06:35:00PM -0700, Peter Geoghegan wrote: > > - ALPHA (big pain in the ass to get right, nobody uses it anymore) > > Yes, for many years now ALPHA has only been useful as a way of > illustrating how bad it's possible for CPU memory operation reordering > considerations to get. So I quite agree. Are there any optimizations we have avoided, or 'volatile' designations added, only for Alpha? Could we improve other things if Alpha support was dropped? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + Everyone has their own god. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Wed, Oct 16, 2013 at 1:52 PM, Andres Freund wrote: > On 2013-10-16 13:04:23 -0400, Robert Haas wrote: >> So I vote for removing IRIX and Tru64 immediately, but I'm a little >> more hesitant about shooting UnixWare, since it's technically still >> supported. > > I think if somebody wants to have it supported they need to provide a > buildfarm member and probably a bit of help. Doing any change in this > area for a platform that nobody has access to in any way seems > pointless because it will be broken anyway. I don't agree with that policy. Sure, 97% of our users are probably running Linux, Windows, MacOS X, or one of the fairly-popular BSD variants. But I think a part of the appeal of PostgreSQL is that it is cross-platform, and doesn't require a lot of special hardware support, and I'm loathe to give that up. It's one thing to say, gee, we don't know whether this is actually going to compile and work on your platform, because it's not tested. It's quite something else to say, anything that's not on this short list of supported platforms has zero chance of working without jumping through major hoops. Commit b8ed4cc9627de437e5eafdb81631a0d0f063abb3, from April of this year, updated our support for --disable-spinlocks. Spinlocks are a far more basic facility than the sorts of advanced primitives we're talking about requiring here. If we're going to support compiling under --disable-spinlocks, then we certainly can't rely on this more advanced stuff always being present. Even if we get rid of that, it's throwing up one more barrier in front of people who want to get PostgreSQL running on a non-mainstream architecture. I've spent enough time over the years working on odd hardware to have sympathy for people who want to compile stuff there and have it work, or at least work after some non-huge amount of hacking. >> I don't think we can desupport it just because it doesn't have CAS. >> CAS is very useful and I think we should start using it, but I think >> we should anticipate a --disable-cas or --disable-atomics option and >> regularly test that our code works without it. IOW, we can rely on it >> as an optimization, but not for correctness. Eventually, we can >> probably desupport all platforms that don't have CAS, but I see that >> as something that's probably 5 or 10 years away, not something we can >> do tomorrow. > > I think that will result in loads of barely tested duplicative code. If > there were any even remotely popular platforms requiring this, ok. But > unless I miss something there really isn't. > > We're talking about CPUs with mostly less than 100MHZ here, mostly with > directly soldered RAM in the one digit MB range. I really don't think > there's a usecase for running PG on them. And I doubt it still works on > many of the architectures we advocate. If stuff is completely ancient and obsolete, I think it's fine to kill it; it probably doesn't work anyway, and nobody's likely to try. But I think a lot of the stuff you're talking about is not in that category. >> I'm also not sure that this is dead enough to kill. >> >> > - M32R (no userspace CAS afaics) >> >> Support was added for this in 8.3 and it doesn't seem to be particularly >> dead. > > It's not? All I've read seems to point into a different direction. > > The newest supported CPU seems to be > http://www.renesas.com/products/mpumcu/m32r/m32r_ecu/32196/index.jsp > sporting a whopping 1024Kb of programmable RAM and only single precision > FPU. Well, so, they're an embedded chip. Yes, it's low spec. But then why'd somebody bother adding spinlock support for it in 2007? It's not as if that was a high-end chip *then* either. It's hard to say where to draw the line here. I don't want the illusion of support for platforms that don't in fact have a prayer of working to prevent us from making needed improvements. On the other hand, I also don't want to blithely rip out support for architectures that people may well still be using and where, in some cases, people have gone out of their way to add that support. If we get to the point where some relatively-obscure architecture is the only thing standing between us and improvement X, then we can weigh those things against each other and decide. But I don't really want to go rip out support for half-a-dozen semi-supported architectures without some clear goal in mind. That just doesn't seem friendly. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-16 13:04:23 -0400, Robert Haas wrote: > > - m86k (doesn't have a useable CAS on later iterations like coldfire) > > I don't think we can desupport it just because it doesn't have CAS. Btw, if necessary we could easily support the pre coldfire variants. Note that e.g. debian doesn't support coldfire either. Well, the unofficial m68k port after it has been dropped from core debian in *2007*. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-16 13:55:20 -0400, Robert Haas wrote: > >> - sinix (s_lock support remaining) > >> - sun3 (I think it's just s_lock support remaining) > >> - natsemi 32k > > Patch removing spinlock support for these three ports is attached. > This is not to say we couldn't remove more later, but these seem to be > the three spinlock implementations that are most sincerely dead. > Absent objections, I'll apply this tomorrow. Looks good to me. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
>> - sinix (s_lock support remaining) >> - sun3 (I think it's just s_lock support remaining) >> - natsemi 32k Patch removing spinlock support for these three ports is attached. This is not to say we couldn't remove more later, but these seem to be the three spinlock implementations that are most sincerely dead. Absent objections, I'll apply this tomorrow. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company desupport-ancient-spinlocks.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-16 13:04:23 -0400, Robert Haas wrote: > So I vote for removing IRIX and Tru64 immediately, but I'm a little > more hesitant about shooting UnixWare, since it's technically still > supported. I think if somebody wants to have it supported they need to provide a buildfarm member and probably a bit of help. Doing any change in this area for a platform that nobody has access to in any way seems pointless because it will be broken anyway. > > I think we should remove support for the following architectures: > > - VAX > > According to http://en.wikipedia.org/wiki/VAX#History, all > manufacturing of VAX computers ceased in 2005, but according to > http://en.wikipedia.org/wiki/OpenVMS#Major_release_timeline, OpenVMS > is still releasing new versions. I'm not sure what to make of that. And the last revisions are from 2000 even. I don't think anybody will actually run a new postgres installation on it. I also doubt our current version actually compiles there. > > - superH > > Support for spinlocks on SuperH was only recently added, in 9.0. I > don't think we can assume that no one cares any more. Yes, I since revised my opinion somewhere downthread. It's pretty much linux and gcc only, so it's really not that problematic. Note that our current implementation is broken on many older SuperH (< SH4) CPUs since their tas isn't safe... > > - m86k (doesn't have a useable CAS on later iterations like coldfire) > I don't think we can desupport it just because it doesn't have CAS. > CAS is very useful and I think we should start using it, but I think > we should anticipate a --disable-cas or --disable-atomics option and > regularly test that our code works without it. IOW, we can rely on it > as an optimization, but not for correctness. Eventually, we can > probably desupport all platforms that don't have CAS, but I see that > as something that's probably 5 or 10 years away, not something we can > do tomorrow. I think that will result in loads of barely tested duplicative code. If there were any even remotely popular platforms requiring this, ok. But unless I miss something there really isn't. We're talking about CPUs with mostly less than 100MHZ here, mostly with directly soldered RAM in the one digit MB range. I really don't think there's a usecase for running PG on them. And I doubt it still works on many of the architectures we advocate. > I'm also not sure that this is dead enough to kill. > > > - M32R (no userspace CAS afaics) > > Support was added for this in 8.3 and it doesn't seem to be particularly dead. It's not? All I've read seems to point into a different direction. The newest supported CPU seems to be http://www.renesas.com/products/mpumcu/m32r/m32r_ecu/32196/index.jsp sporting a whopping 1024Kb of programmable RAM and only single precision FPU. > > - mips for anything but gcc > 4.4, using gcc's atomics support > > - s390 for anything but gcc > 4.4, using gcc's atomics support > > I'm not clearly how broadly this would sweep, but MIPS doesn't appear dead. Downthread I noticed it's gcc 4.2 not 4.4. There's some API change in gcc's atomics support in 4.4 which is why I thought of 4.4 but it shouldn't affect us after looking in more detail. Mips seems to only be used with gcc these days, so I think that's ok. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 10/16/2013 07:04 PM, Robert Haas wrote: > On Sat, Oct 12, 2013 at 8:46 PM, Andres Freund wrote: >> I think we should remove support the following ports: >> - IRIX >> - UnixWare >> - Tru64 > > According to http://en.wikipedia.org/wiki/IRIX, IRIX has been > officially retired. The last release of IRIX was in 2006 and support > will end in December of 2013. Therefore, it will be unsupported by > the time PostgreSQL 9.4 is released. > > According to http://en.wikipedia.org/wiki/UnixWare, UnixWare is not > dead, although there have been no new releases in 5 years. > > According to http://en.wikipedia.org/wiki/Tru64_UNIX, Tru64 has been > officially retired. Support ended in December, 2012. This seems safe > to remove. > > So I vote for removing IRIX and Tru64 immediately, but I'm a little > more hesitant about shooting UnixWare, since it's technically still > supported. > >> Neither of those are relevant. agreed >> >> I think we should remove support for the following architectures: >> - VAX > > According to http://en.wikipedia.org/wiki/VAX#History, all > manufacturing of VAX computers ceased in 2005, but according to > http://en.wikipedia.org/wiki/OpenVMS#Major_release_timeline, OpenVMS > is still releasing new versions. I'm not sure what to make of that. VAX is also an officially supported OpenBSD port (see http://www.openbsd.org/vax.html) Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Sat, Oct 12, 2013 at 8:46 PM, Andres Freund wrote: > I think we should remove support the following ports: > - IRIX > - UnixWare > - Tru64 According to http://en.wikipedia.org/wiki/IRIX, IRIX has been officially retired. The last release of IRIX was in 2006 and support will end in December of 2013. Therefore, it will be unsupported by the time PostgreSQL 9.4 is released. According to http://en.wikipedia.org/wiki/UnixWare, UnixWare is not dead, although there have been no new releases in 5 years. According to http://en.wikipedia.org/wiki/Tru64_UNIX, Tru64 has been officially retired. Support ended in December, 2012. This seems safe to remove. So I vote for removing IRIX and Tru64 immediately, but I'm a little more hesitant about shooting UnixWare, since it's technically still supported. > Neither of those are relevant. > > I think we should remove support for the following architectures: > - VAX According to http://en.wikipedia.org/wiki/VAX#History, all manufacturing of VAX computers ceased in 2005, but according to http://en.wikipedia.org/wiki/OpenVMS#Major_release_timeline, OpenVMS is still releasing new versions. I'm not sure what to make of that. > - univel (s_lock support remaining) We removed the univel port in 9.2 and didn't get any complaints, but the s_lock support is still used by the SCO and UnixWare ports, except under GCC. > - sinix (s_lock support remaining) This seems to be quite thoroughly dead. The best information I can find indicates that development ended in 2002 and support in 2008. I think we can remove this. > - sun3 (I think it's just s_lock support remaining) > - natsemi 32k Both of these are so old I can hardly find any information on them. Seems clear to nuke these. > - superH Support for spinlocks on SuperH was only recently added, in 9.0. I don't think we can assume that no one cares any more. > - ALPHA (big pain in the ass to get right, nobody uses it anymore) It seems somehow a shame to let this one go, but I agree it's a big pain in the ass to get it right. > - m86k (doesn't have a useable CAS on later iterations like coldfire) I don't think we can desupport it just because it doesn't have CAS. CAS is very useful and I think we should start using it, but I think we should anticipate a --disable-cas or --disable-atomics option and regularly test that our code works without it. IOW, we can rely on it as an optimization, but not for correctness. Eventually, we can probably desupport all platforms that don't have CAS, but I see that as something that's probably 5 or 10 years away, not something we can do tomorrow. I'm also not sure that this is dead enough to kill. > - M32R (no userspace CAS afaics) Support was added for this in 8.3 and it doesn't seem to be particularly dead. > - mips for anything but gcc > 4.4, using gcc's atomics support > - s390 for anything but gcc > 4.4, using gcc's atomics support I'm not clearly how broadly this would sweep, but MIPS doesn't appear dead. > - 32bit/http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-16 12:26:28 -0400, Robert Haas wrote: > On Tue, Oct 15, 2013 at 8:33 AM, Andres Freund wrote: > > On 2013-10-13 16:56:12 +0200, Tom Lane wrote: > >> More to the point for this specific case, it seems like our process > >> ought to be > >> (1) select a preferably-small set of gcc atomic intrinsics that we > >> want to use. > > > > I suggest: > > * pg_atomic_load_u32(uint32 *) > > * uint32 pg_atomic_store_u32(uint32 *) > > We currently assume simple assignment suffices for this. Partially that only works because we sprinkle 'volatile's on lots of places. That can actually hurt performance... it'd usually be something like: #define pg_atomic_load(uint32) (*(volatile uint32 *)(atomic)) Even if not needed in some places because a variable is already volatile, it's very helpful in pointing out what happens and where you need to be careful. > > * uint32 pg_atomic_exchange_u32(uint32 *ptr, uint32 val) > > * bool pg_atomic_compare_exchange_u32(uint32 *ptr, uint32 *expected, uint32 > > newval) > > * uint32 pg_atomic_fetch_add_u32(uint32 *ptr, uint32 add) > > * uint32 pg_atomic_fetch_sub_u32(uint32 *ptr, uint32 add) > > * uint32 pg_atomic_fetch_and_u32(uint32 *ptr, uint32 add) > > * uint32 pg_atomic_fetch_or_u32(uint32 *ptr, uint32 add) > > Do we really need all of those? Compare-and-swap is clearly needed, > and fetch-and-add is also very useful. I'm not sure about the rest. > Not that I necessarily object to having them, but it will be a lot > more work. Well, _sub can clearly be implemented with _add generically. I find code using _sub much easier to read than _add(-whatever), but that's maybe just me. But _and, _or are really useful because they can be used to atomically clear and set flag bits. > > > * u64 variants of the above > > * bool pg_atomic_test_set(void *ptr) > > * void pg_atomic_clear(void *ptr) > > What are the intended semantics here? It's basically TAS() without defining whether setting a value that needs to be set is a 1 or a 0. Gcc provides this and I think we should make our spinlock implementation use it if there is no special cased implementation available. > > Ontop of that we can generically implement: > > * pg_atomic_add_until_u32(uint32 *ptr, uint32 val, uint32 limit) > > * pg_atomic_(add|sub|and|or)_fetch_u32() > > * u64 variants of the above > > Are these really generally needed? _add_until() is very useful for implementing thinks like usagecount where you don't want to increase values too high. The lwlock scaling thing needs the add_fetch variant because we need to know what the lockcount is *after* we've registered. I think lots of lockless algorithm have similar requirements. Since those are either wrappers around fetch_op or compare_swap and thus can be implemented generically I don't really see a problem with providing them. > I have a related problem, which is that some code I'm currently > working on vis-a-vis parallelism can run lock-free on platforms with > atomic 8 bit assignment but needs a spinlock or two elsewhere. So I'd > want to use pg_atomic_store_u64(), but I'd also need a clean way to > test, at compile time, whether it's available. Yes, definitely. There should be a couple of #defines that declare whether non-prerequisite options are supported. I'd guess we want at least: * 8byte math * 16byte compare_and_swap > More general question: how do we move the ball down the field in this > area? I'm willing to do some of the legwork, but I doubt I can do all > of it, and I really think we need to make some progress here soon, as > it seems that you and I and Ants are all running into the same > problems in slightly different ways. What's our strategy for getting > something done here? That's a pretty good question. I am willing to write the gcc implementation, plus the generic wrappers and provide the infrastructure to override it per-platform. I won't be able to do anything about non-linux, non-gcc platforms in that timeframe though. I was thinking of something like: include/storage/atomic.h include/storage/atomic-gcc.h include/storage/atomic-msvc.h include/storage/atomic-acc-ia64.h where atomic.h first has a list of #ifdefs including the specialized files and then lots of #ifndef providing generic variants if not already provided by the platorm specific file. We could provide not only per-compiler files but also compiler independent files for some arches so we could e.g. define the restrictions for arm once. I think whether that's useful will be visible when writing the stuff. Makes sense? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Tue, Oct 15, 2013 at 8:33 AM, Andres Freund wrote: > On 2013-10-13 16:56:12 +0200, Tom Lane wrote: >> More to the point for this specific case, it seems like our process >> ought to be >> (1) select a preferably-small set of gcc atomic intrinsics that we >> want to use. > > I suggest: > * pg_atomic_load_u32(uint32 *) > * uint32 pg_atomic_store_u32(uint32 *) We currently assume simple assignment suffices for this. > * uint32 pg_atomic_exchange_u32(uint32 *ptr, uint32 val) > * bool pg_atomic_compare_exchange_u32(uint32 *ptr, uint32 *expected, uint32 > newval) > * uint32 pg_atomic_fetch_add_u32(uint32 *ptr, uint32 add) > * uint32 pg_atomic_fetch_sub_u32(uint32 *ptr, uint32 add) > * uint32 pg_atomic_fetch_and_u32(uint32 *ptr, uint32 add) > * uint32 pg_atomic_fetch_or_u32(uint32 *ptr, uint32 add) Do we really need all of those? Compare-and-swap is clearly needed, and fetch-and-add is also very useful. I'm not sure about the rest. Not that I necessarily object to having them, but it will be a lot more work. > * u64 variants of the above > * bool pg_atomic_test_set(void *ptr) > * void pg_atomic_clear(void *ptr) What are the intended semantics here? > Ontop of that we can generically implement: > * pg_atomic_add_until_u32(uint32 *ptr, uint32 val, uint32 limit) > * pg_atomic_(add|sub|and|or)_fetch_u32() > * u64 variants of the above Are these really generally needed? > We might also want to provide a generic implementation of the math > operations based on pg_atomic_compare_exchange() to make it easier to > bring up a new architecture. +1. I have a related problem, which is that some code I'm currently working on vis-a-vis parallelism can run lock-free on platforms with atomic 8 bit assignment but needs a spinlock or two elsewhere. So I'd want to use pg_atomic_store_u64(), but I'd also need a clean way to test, at compile time, whether it's available. More general question: how do we move the ball down the field in this area? I'm willing to do some of the legwork, but I doubt I can do all of it, and I really think we need to make some progress here soon, as it seems that you and I and Ants are all running into the same problems in slightly different ways. What's our strategy for getting something done here? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-13 16:56:12 +0200, Tom Lane wrote: > More to the point for this specific case, it seems like our process > ought to be > (1) select a preferably-small set of gcc atomic intrinsics that we > want to use. I suggest: * pg_atomic_load_u32(uint32 *) * uint32 pg_atomic_store_u32(uint32 *) * uint32 pg_atomic_exchange_u32(uint32 *ptr, uint32 val) * bool pg_atomic_compare_exchange_u32(uint32 *ptr, uint32 *expected, uint32 newval) * uint32 pg_atomic_fetch_add_u32(uint32 *ptr, uint32 add) * uint32 pg_atomic_fetch_sub_u32(uint32 *ptr, uint32 add) * uint32 pg_atomic_fetch_and_u32(uint32 *ptr, uint32 add) * uint32 pg_atomic_fetch_or_u32(uint32 *ptr, uint32 add) * u64 variants of the above * bool pg_atomic_test_set(void *ptr) * void pg_atomic_clear(void *ptr) Ontop of that we can generically implement: * pg_atomic_add_until_u32(uint32 *ptr, uint32 val, uint32 limit) * pg_atomic_(add|sub|and|or)_fetch_u32() * u64 variants of the above We might also want to provide a generic implementation of the math operations based on pg_atomic_compare_exchange() to make it easier to bring up a new architecture. I think we should leave 64bit support optional for now. Opinions? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-14 09:42:46 -0400, Robert Haas wrote: > On Mon, Oct 14, 2013 at 9:41 AM, Andres Freund wrote: > > On 2013-10-14 09:40:10 -0400, Robert Haas wrote: > >> On Sun, Oct 13, 2013 at 5:34 AM, Tom Lane wrote: > >> >> - PA-RISC. I think Tom was the remaining user there? Maybe just !gcc. > >> > > >> > Until pretty recently, there was a PA-RISC machine (not mine) in the > >> > buildfarm. I don't see it in the list today though. In any case, > >> > HP's compiler has always been a PITA, so no objection to requiring gcc > >> > for this platform. > >> > >> I object to requiring gcc on that platform; we've had recent interest > >> in aCC-compiled builds on that platform. > > > > On PA-RISC or Itanic? > > Oh, sorry. I think it was Itanium. Good. Not that Itanium is nice, but it certainly supports all required atomic ops. I think somebody with access to that will have to figure out what intrinsics are provided then - I haven't found reference documentation for aCC's ia64/sys/inline.h. I think Tom's stance that people who want a platform to be supported need to help is reasonable. It's really hard to develop not entirely trivial code for platforms you don't have access to and that don't even have a buildfarm member. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Mon, Oct 14, 2013 at 9:41 AM, Andres Freund wrote: > On 2013-10-14 09:40:10 -0400, Robert Haas wrote: >> On Sun, Oct 13, 2013 at 5:34 AM, Tom Lane wrote: >> >> - PA-RISC. I think Tom was the remaining user there? Maybe just !gcc. >> > >> > Until pretty recently, there was a PA-RISC machine (not mine) in the >> > buildfarm. I don't see it in the list today though. In any case, >> > HP's compiler has always been a PITA, so no objection to requiring gcc >> > for this platform. >> >> I object to requiring gcc on that platform; we've had recent interest >> in aCC-compiled builds on that platform. > > On PA-RISC or Itanic? Oh, sorry. I think it was Itanium. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-14 09:40:10 -0400, Robert Haas wrote: > On Sun, Oct 13, 2013 at 5:34 AM, Tom Lane wrote: > >> - PA-RISC. I think Tom was the remaining user there? Maybe just !gcc. > > > > Until pretty recently, there was a PA-RISC machine (not mine) in the > > buildfarm. I don't see it in the list today though. In any case, > > HP's compiler has always been a PITA, so no objection to requiring gcc > > for this platform. > > I object to requiring gcc on that platform; we've had recent interest > in aCC-compiled builds on that platform. On PA-RISC or Itanic? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Sun, Oct 13, 2013 at 5:34 AM, Tom Lane wrote: >> - PA-RISC. I think Tom was the remaining user there? Maybe just !gcc. > > Until pretty recently, there was a PA-RISC machine (not mine) in the > buildfarm. I don't see it in the list today though. In any case, > HP's compiler has always been a PITA, so no objection to requiring gcc > for this platform. I object to requiring gcc on that platform; we've had recent interest in aCC-compiled builds on that platform. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-13 20:39:21 +0200, Tom Lane wrote: > Andres Freund writes: > > The question about platforms that simply cannot provide such atomics > > like PA-RISC, which afaics is the only one, remains tho. I am not sure > > we really want to provide codepaths that are only going to be tested > > there. > > PA-RISC is a dead architecture. According to wikipedia, HP hasn't sold > any such machines since 2008, and won't support them beyond 2013. If > that really is the only case we're worried about supporting, it's an > easy decision. Great. > What worries me more is that you mentioned several cases where the gcc > atomics exist but need kernel support. I have to think that a trap > to the kernel would make the operation so expensive as to be a serious > performance loss, not gain. So it seems to me that platforms like that > are essentially being kicked to the curb if we make this change, even > if they theoretically could still work. I think it's not that bad on most - they all seem to use some more lightweight trap mechanisms that basically just change protection and stops interrupts while doing the math. No chance of scheduling and such. > Are there any that we really care about? I don't think so. At least not if we're restricting ourselves to 32bit cmpxchg/xadd which I think will be enough for the first rounds of improvement. It's: - PA-RISC - sparc before ultrasparcs (1995) - Multi-CPU/core SuperH before SH4 (uni SH2 has some cute interrupt handling tricks, that do not require a trap) - arm before v6 Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
Andres Freund writes: > The question about platforms that simply cannot provide such atomics > like PA-RISC, which afaics is the only one, remains tho. I am not sure > we really want to provide codepaths that are only going to be tested > there. PA-RISC is a dead architecture. According to wikipedia, HP hasn't sold any such machines since 2008, and won't support them beyond 2013. If that really is the only case we're worried about supporting, it's an easy decision. What worries me more is that you mentioned several cases where the gcc atomics exist but need kernel support. I have to think that a trap to the kernel would make the operation so expensive as to be a serious performance loss, not gain. So it seems to me that platforms like that are essentially being kicked to the curb if we make this change, even if they theoretically could still work. Are there any that we really care about? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-13 16:56:12 +0200, Tom Lane wrote: > Andres Freund writes: > > That's a fair point. But all of them will use gcc, right? I've > > previously thought we'd need 4.4 because there's an incompatibility > > between 4.3 and 4.4 but I think it won't touch us, so 4.2 which added > > atomics for mips seems fine. Given there's no buildfarm animal and > > there's lots of variety out there that seems like a fair amount of > > support. > FWIW, I wouldn't have the slightest bit of difficulty with setting a > project policy that if you want some platform to be supported, you > must provide a buildfarm animal running on it. > More to the point for this specific case, it seems like our process > ought to be > (1) select a preferably-small set of gcc atomic intrinsics that we > want to use. > (2) provide macro wrappers for these so that substituting other > implementations isn't too tough. > (3) for any platform where people don't want to use recent gcc, > insist they provide substitute implementations of the macros. > > I don't think the core project has to be responsible for implementing > (3), except maybe on very-mainstream arches such as x86. We can > adopt a self-help attitude for everything else. Sounds fair to me. The question about platforms that simply cannot provide such atomics like PA-RISC, which afaics is the only one, remains tho. I am not sure we really want to provide codepaths that are only going to be tested there. I do plan to propose a set of macros for this once we know what cases we need to support. > But ... having said all that, it would be nice to see some proof of > significant performance benefits that we're going to get from kicking > those non-mainstream arches to the curb. I'm not enamored of removing > real functionality now for vague promises of benefits later. The reason I am bringing this up is that I'd like lift the patch in http://archives.postgresql.org/message-id/20130926225545.GB26663%40awork2.anarazel.de from a purely POC state to something committable. The numbers in it seem to justify causing some pain. Once the infrastructure is there we can easily get some really nice further benefits by changing buffer pinning - that's one of the most expensive things wrt. scalability after the above patch. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
Andres Freund writes: > That's a fair point. But all of them will use gcc, right? I've > previously thought we'd need 4.4 because there's an incompatibility > between 4.3 and 4.4 but I think it won't touch us, so 4.2 which added > atomics for mips seems fine. Given there's no buildfarm animal and > there's lots of variety out there that seems like a fair amount of > support. FWIW, I wouldn't have the slightest bit of difficulty with setting a project policy that if you want some platform to be supported, you must provide a buildfarm animal running on it. More to the point for this specific case, it seems like our process ought to be (1) select a preferably-small set of gcc atomic intrinsics that we want to use. (2) provide macro wrappers for these so that substituting other implementations isn't too tough. (3) for any platform where people don't want to use recent gcc, insist they provide substitute implementations of the macros. I don't think the core project has to be responsible for implementing (3), except maybe on very-mainstream arches such as x86. We can adopt a self-help attitude for everything else. But ... having said all that, it would be nice to see some proof of significant performance benefits that we're going to get from kicking those non-mainstream arches to the curb. I'm not enamored of removing real functionality now for vague promises of benefits later. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-13 14:08:59 +0200, Andres Freund wrote: > On 2013-10-13 11:34:42 +0200, Tom Lane wrote: > > Andres Freund writes: > > > I think we should remove support for the following architectures: > > > - superH > > > > This one was contributed just a year or two ago, if memory serves, > > which suggests that somebody out there cares about it. OTOH, if > > they still care, we could insist they provide whatever atomic ops > > we want to depend on. > > It was 2009 - aac3c301b5e8178841e5749b3657c1a639ba06c1 . I haven't yet > verified if gcc's atomics support is acceptable for the platform > (checkout is running for the last 3h...). > If it's supported, falling back to that seems easy enough. Ok, so the git checkout has finally finished. http://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/config/sh/sync.md;hb=HEAD makes for an interesting read... In short: On newer sh variants (SH4) true hardware support is used, on older ones the linux provides helpers. The variant has to be selected by gcc parameters unfortunately, but I'd guess that's ok (e.g. SH4 -matomic-model=hard-llcs ). > > > - PA-RISC. I think Tom was the remaining user there? Maybe just !gcc. > > > > Until pretty recently, there was a PA-RISC machine (not mine) in the > > buildfarm. I don't see it in the list today though. In any case, > > HP's compiler has always been a PITA, so no objection to requiring gcc > > for this platform. > > The reason I'd like to generally get rid of PA-RISC is that it's the > only platform that doesn't seem to have any form of compare and > swap. GCC should provide fallbacks - with some warnings - using > spinlocks instead Hm. I had misunderstood the documentation around this a bit - source code still is the only reasonable documentation :(. PA-RISC is only supported on linux using kernel helpers (which switch off interrupts for the duration of the operation...). > but I am afraid people will start doing things like > atomic operations in signal handlers that won't be noticed and will be a > PITA to debug. That's already a possibility with our memory barriers :(. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-13 11:34:42 +0200, Tom Lane wrote: > Andres Freund writes: > > I think we should remove support for the following architectures: > > - superH > > This one was contributed just a year or two ago, if memory serves, > which suggests that somebody out there cares about it. OTOH, if > they still care, we could insist they provide whatever atomic ops > we want to depend on. It was 2009 - aac3c301b5e8178841e5749b3657c1a639ba06c1 . I haven't yet verified if gcc's atomics support is acceptable for the platform (checkout is running for the last 3h...). If it's supported, falling back to that seems easy enough. > > - PA-RISC. I think Tom was the remaining user there? Maybe just !gcc. > > Until pretty recently, there was a PA-RISC machine (not mine) in the > buildfarm. I don't see it in the list today though. In any case, > HP's compiler has always been a PITA, so no objection to requiring gcc > for this platform. The reason I'd like to generally get rid of PA-RISC is that it's the only platform that doesn't seem to have any form of compare and swap. GCC should provide fallbacks - with some warnings - using spinlocks instead but I am afraid people will start doing things like atomic operations in signal handlers that won't be noticed and will be a PITA to debug. Having read a fair amount of assembler looking at this I have to say, anybody thinking LL/SC architectures are neat... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On 2013-10-12 18:35:00 -0700, Peter Geoghegan wrote: > Not so sure about these. > > - M32R (no userspace CAS afaics) I don't think M32R will hurt us/anybody much. > > - 32bit/ > - mips for anything but gcc > 4.4, using gcc's atomics support The reason I'd like to de-support mips for older GCCs is that writing assembler for them isn't trivial enough to do it blindly and I've had - for other stuff - difficulties getting my hand on them. GCC provides all the atomics for mips since 4.2, so we can just rely on that. > > - s390 for anything but gcc > 4.4, using gcc's atomics support Easier to write assembler, but still untestable and even harder to get access on. I think 4.2 should be fine as well. > I think we should think hard about removing support for MIPS. A lot of > Chinese chip manufacturers have licensed MIPS technology in just the > last couple of years, so there is plenty of it out there; I'd be > slightly concerned that the proposed restrictions on MIPS would be > onerous. Much of this is the kind of hardware that a person might > plausibly want to run Postgres on. That's a fair point. But all of them will use gcc, right? I've previously thought we'd need 4.4 because there's an incompatibility between 4.3 and 4.4 but I think it won't touch us, so 4.2 which added atomics for mips seems fine. Given there's no buildfarm animal and there's lots of variety out there that seems like a fair amount of support. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
Andres Freund writes: > I think we should remove support for the following architectures: > - superH This one was contributed just a year or two ago, if memory serves, which suggests that somebody out there cares about it. OTOH, if they still care, we could insist they provide whatever atomic ops we want to depend on. > - PA-RISC. I think Tom was the remaining user there? Maybe just !gcc. Until pretty recently, there was a PA-RISC machine (not mine) in the buildfarm. I don't see it in the list today though. In any case, HP's compiler has always been a PITA, so no objection to requiring gcc for this platform. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] removing old ports and architectures
On Sat, Oct 12, 2013 at 5:46 PM, Andres Freund wrote: > I think we should remove support the following ports: > - IRIX > - UnixWare > - Tru64 > > Neither of those are relevant. Seems reasonable. > I think we should remove support for the following architectures: > - VAX Agreed. > - univel (s_lock support remaining) > - sinix (s_lock support remaining) > - sun3 (I think it's just s_lock support remaining) > - natsemi 32k I don't know enough about these, which doesn't bode well for them. > - superH SuperH isn't dead, but it is only used for very small embedded systems, I think (mostly microcontrollers). So maybe. > - ALPHA (big pain in the ass to get right, nobody uses it anymore) Yes, for many years now ALPHA has only been useful as a way of illustrating how bad it's possible for CPU memory operation reordering considerations to get. So I quite agree. > - m86k (doesn't have a useable CAS on later iterations like coldfire) It does seem like Motorola 68k is vanishingly close to dead. > - M32R (no userspace CAS afaics) > - mips for anything but gcc > 4.4, using gcc's atomics support > - s390 for anything but gcc > 4.4, using gcc's atomics support > - 32bit/ Possibly: > - all mips > - PA-RISC. I think Tom was the remaining user there? Maybe just !gcc. I think we should think hard about removing support for MIPS. A lot of Chinese chip manufacturers have licensed MIPS technology in just the last couple of years, so there is plenty of it out there; I'd be slightly concerned that the proposed restrictions on MIPS would be onerous. Much of this is the kind of hardware that a person might plausibly want to run Postgres on. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] removing old ports and architectures
Hi, As discussed in 20130926225545.gb26...@awork2.anarazel.de and on quite some other occasions there's quite some scalability improvements we could make if we had cross platform support for atomic operations. Providing that is a fair bit of work for every architecture/compiler/OS, so I think it's a reasonable thing to remove unused things first. Alternatively we can maintain parallel code paths for atomic ops supporting platforms and for those without, but that seems like a bad idea from a complexity and testing perspective given that all even remotely current platforms should support them. I think we should remove support the following ports: - IRIX - UnixWare - Tru64 Neither of those are relevant. I think we should remove support for the following architectures: - VAX - univel (s_lock support remaining) - sinix (s_lock support remaining) - sun3 (I think it's just s_lock support remaining) - natsemi 32k - superH - ALPHA (big pain in the ass to get right, nobody uses it anymore) - m86k (doesn't have a useable CAS on later iterations like coldfire) - M32R (no userspace CAS afaics) - mips for anything but gcc > 4.4, using gcc's atomics support - s390 for anything but gcc > 4.4, using gcc's atomics support - 32bit/http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers