Re: [HACKERS] "make report"
It depends. QNX4 may be used with GCC, in which case it does have long long. I am not sure if that combination will play along with Postgres, but it should not be assumed impossible. - Original Message - From: "Peter Eisentraut" <[EMAIL PROTECTED]> To: "Thomas Lockhart" <[EMAIL PROTECTED]> Cc: "PostgreSQL Hackers" <[EMAIL PROTECTED]> Sent: Wednesday, April 24, 2002 1:14 PM Subject: Re: [HACKERS] "make report" > Thomas Lockhart writes: > > > Right. The two areas which come to mind are integer availability and the > > timezone support (as you might know we support *three* different time > > zone models). At the moment, none of the developers know the features > > supported on the platforms we claim to support. Which platforms do not > > have int8 support still? > > "Still" is the wrong word. There used to be platforms with certain areas > of trouble, and those platforms don't go away. > > But since you asked: QNX 4 and SCO OpenServer are known to lack 8 byte > integers. > > > Which do not have time zone interfaces fitting > > into the two "zonefull" styles? I'd like to know, but istm that the > > people *with* the platforms could do this much more easily than those > > without. What am I missing here?? > > I don't think polling users this way will yield reliable results. If you > really want to find out, break something and see if someone complains. > > -- > Peter Eisentraut [EMAIL PROTECTED] > > > ---(end of broadcast)--- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to [EMAIL PROTECTED] so that your > message can get through to the mailing list cleanly > ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] HEADS UP: Win32/OS2/BeOS native ports
> mlw <[EMAIL PROTECTED]> writes: > > I think that you should create a verbatim implementation of the SysV > > shared memory API in native Win32. It may have to be a pgsysvshm.dll > > or something like it, but I think it is the best possible approach. > > > Let me look at it, I may be able to have something pretty quick. > > The notion of redesigning the internal API shouldn't be forgotten, > though. I'm not so dissatisfied with the shmem API (mainly because > it's only relevant at startup; once we've created and attached the > shmem segment, we're done worrying about it). But the SysV semaphore > API is really kind of ugly, and the ugliness doesn't buy anything except > porting difficulty. Moreover, putting a cleaner API layer there would > make it easier to experiment with cheaper semaphore primitives, such > as POSIX mutexes. > > There was a thread last fall concerning redesigning that code --- I've > forgotten the guy's name, but IIRC he wanted to make a port to QNX6, That would be me. > and the sema code was getting in the way. We put the work on hold > because we were getting close to 7.2 release (or thought we were, > anyway) but the project ought to be taken up again. > Yes, I am intended to give it another spin soon. I think it is bad idea to impose SysV ugliness on systems which have better solutions. Main problem with SysV primitives is that they are 'sticky' (i.e., not cleaned up if process dies/exits by the system). So Postgres has to deal with issues like discovering leftovers, finding unused IPC keys, etc. It is inelegant and takes up lot of code. POSIX primitives are anonymous and cleaned up automatically. So you just say 'give me a semaphore' and you get it, nothing gets into your way. Performance of POSIX mutexes and semaphores (on platforms where they are implemented properly) is also better than SysV semaphores. Unfortunately some systems have rather lame POSIX support, for example semaphores and mutexes can't be shared across processes on Linux. That's basically the reason why people keep sticking to SysV. What really need to be done is new abstraction layer which would cover SysV API, POSIX and whatever native APIs are better for BeOS/OS2/Win32. I almost did it last time... -- igor ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] HEADS UP: Win32/OS2/BeOS native ports
> "Igor Kovalenko" <[EMAIL PROTECTED]> writes: > > What really need to be done is new abstraction layer which would cover SysV > > API, POSIX and whatever native APIs are better for BeOS/OS2/Win32. I almost > > did it last time... > > Yes. I just sent off a proposal for a cleaner semaphore API --- please > comment on it. > I will look. I remember from my last attempt that it actually did not involve a lot of changes in your existing abstraction layer (which already exists, just being SysV-centric). I believe only one function prototype had to be changed... Your proposal sounds like more changes will be needed... > My inclination is to stick with the SysV API for shared memory, however. > The "stickiness" is actually not a bad thing for us in the shared memory > case, because it allows a new postmaster to detect the situation where > old backends are still running: it can see that there is an old shmem > segment still present with attached processes. Without that, we have no > good defense against the scenario where an old postmaster dumped core > leaving backends still running. The backends are fine as long as they > are left to finish out their operations, or even killed with whatever > degree of prejudice the admin wants. But what we must *not* do is allow > a new postmaster to start while the old backends are still running; > that would mean two sets of backends running without contact with each > other, which would be fatal for data integrity. The SysV API lets us > detect that case, but I don't see any equally good way to do it if we > are using anonymous shared memory. It does not have to be anonymous. POSIX also defines shm_open(same arguments as open) API which will create named object in whatever location corresponds to shared memory storage on that platform (object is then grown to needed size by ftruncate() and the fd is then passed to mmap). The object will exist in name space and can be detected by subsequent calls to shm_open() with same name. It is not really different from doing open(), but more portable (mmap() on regular files may not be supported). I suggest we do IPC abstraction which would cover shared memory as well as semaphores, otherwise it will be only half of solution - platforms without SysV API would still have to emulate SysV shared memory. -- igor ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] HEADS UP: Win32/OS2/BeOS native ports
> "Marc G. Fournier" <[EMAIL PROTECTED]> writes: > > Since our default behavior (at startup) is to have TCP sockets disabled, > > how many OSs are there that don't support UD sockets? > > A quick look in the sources shows that we #undef HAVE_UNIX_SOCKETS for > QNX, BeOS, and old cygwin versions ... which are exactly the platforms > that don't have SysV shmem support, so those are exactly the guys who > we're trying to fix the problem for. Next release of QNX (6.2) will add support for UDS, but they are still not quite portable. > > I do like the idea of using a Unix socket this way where available, > though. It'd let us switch over the shmem code to using IPC_PRIVATE > shmem key, which'd simplify that code tremendously; and we could make > some progress against the dead-PID-in-lockfile problem. > > Could we get away with saying that the Unix-socket-less platforms have > weaker protection against mistakenly restarting the postmaster? We > could have a plain-vanilla lockfile instead of a socket lockfile on > those platforms, which would not catch the dead-postmaster-live-backends > case, but it'd be better than nothing. And I am not convinced that the > shmem-connection-count check should be trusted on QNX or BeOS, anyway, > so I'm not sure that they actually have a functioning check now. Why can't we use named pipe (aka FIFO file) instead of UDS? I think that is more portable... The socketpair() function also tends to be more portable than whole UDS in general... It works on QNX4 even, but not sure about BeOS. Another thought is, why can't we use bind() to the postmaster port to detect other postmasters? I might be missing something, so pardon by ignorance. But should not bind() to same port fail with EADDRINUSE unless SO_REUSEADDR is set? I don't really know if it is set in postgres or not ... -- igor ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] HEADS UP: Win32/OS2/BeOS native ports
That's what Apache does. Note, on most platforms MAP_ANON is equivalent to mmmap-ing /dev/zero. Solaris for example does not provide MAP_ANON but using fd=open(/dev/zero) mmap(fd, ...) close(fd) works just fine. - Original Message - From: "Bruce Momjian" <[EMAIL PROTECTED]> To: "Igor Kovalenko" <[EMAIL PROTECTED]> Cc: "Tom Lane" <[EMAIL PROTECTED]>; "mlw" <[EMAIL PROTECTED]>; "Marc G. Fournier" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Sunday, June 02, 2002 7:47 PM Subject: Re: [HACKERS] HEADS UP: Win32/OS2/BeOS native ports > Igor Kovalenko wrote: > > It does not have to be anonymous. POSIX also defines shm_open(same arguments > > as open) API which will create named object in whatever location corresponds > > to shared memory storage on that platform (object is then grown to needed > > size by ftruncate() and the fd is then passed to mmap). The object will > > exist in name space and can be detected by subsequent calls to shm_open() > > with same name. It is not really different from doing open(), but more > > portable (mmap() on regular files may not be supported). > > Actually, I think the best shared memory implemention would be > MAP_ANON | MAP_SHARED mmap(), which could be called from the postmaster > and passed to child processes. > > While all our platforms have mmap(), many don't have MAP_ANON, but those > that do could use it. You need MAP_ANON to prevent the shared memory > from being written to a disk file. > > -- > Bruce Momjian| http://candle.pha.pa.us > [EMAIL PROTECTED] | (610) 853-3000 > + If your life is a hard drive, | 830 Blythe Avenue > + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 > ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [HACKERS] Roadmap for a Win32 port
I might be naive here, but would not proper threading model remove the need for fork() altogether? On both Unix and Win32? Should not be too hard to come up with abstraction which encapsulates POSIX, BeOS and Win32 threads... I am not sure how universal POSIX threads are by now. Any important Unix platforms which don't support them yet? This has downside of letting any bug to kill the whole thing. On the bright side, performance should be better on some platforms (note however, Apache group still can't come up with implementation of threaded model which would provide better performance than forked or other models). The need to deal with possibility of 'alien' postmaster running along with orphaned backends would also be removed since there would be only one process. Issue of thread safety of code will come up undoubtedly and some things will probably have to be revamped. But in long term this is probably best way if you want to have efficient and uniform Unix AND Win32 implementations. I am not too familiar with Win32. Speaking about POSIX threads, it would be something like a thread pool with low & high watermarks. Main thread would handle thread pool and hand over requests to worker threads (blocked on condvar). How does that sound? -- igor - Original Message - From: "Bruce Momjian" <[EMAIL PROTECTED]> To: "PostgreSQL-development" <[EMAIL PROTECTED]> Sent: Tuesday, June 04, 2002 11:33 PM Subject: [HACKERS] Roadmap for a Win32 port > OK, I think I am now caught up on the Win32/cygwin discussion, and would > like to make some remarks. > > First, are we doing enough to support the Win32 platform? I think the > answer is clearly "no". There are 3-5 groups/companies working on Win32 > ports of PostgreSQL. We always said there would not be PostgreSQL forks > if we were doing our job to meet user needs. Well, obviously, a number > of groups see a need for a better Win32 port and we aren't meeting that > need, so they are. I believe this is one of the few cases where groups > are going out on their own because we are falling behind. > > So, there is no question in my mind we need to do more to encourage > Win32 ports. Now, on to the details. > > INSTALLER > - > > We clearly need an installer that is zero-hassle for users. We need to > decide on a direction for this. > > GUI > --- > > We need a slick GUI. pgadmin2 seems to be everyone's favorite, with > pgaccess on Win32 also an option. What else do we need here? > > BINARY > -- > > This is the big daddy. It is broken down into several sections: > > FORK() > > How do we handle fork()? Do we use the cygwin method that copies the > whole data segment, or put the global data in shared memory and copy > that small part manually after we create a new process? > > THREADING > > Related to fork(), do we implement an optionally threaded postmaster, > which eliminates CreateProcess() entirely? I don't think we will have > superior performance on Win32 without it. (This would greatly help > Solaris as well.) > > IPC > > We can use Cygwin, MinGW, Apache, or our own code for this. Are there > other options? > > ENVIRONMENT > > Lots of our code requires a unix shell and utilities. Will we continue > using cygwin for this? > > -- - > > As a roadmap, it would be good to get consensus on as many of these > items as possible so people can start working in these areas. We can > keep a web page of decisions we have made to help rally developers to > the project. > > -- > Bruce Momjian| http://candle.pha.pa.us > [EMAIL PROTECTED] | (610) 853-3000 > + If your life is a hard drive, | 830 Blythe Avenue > + Christ can be your backup.| Drexel Hill, Pennsylvania 19026 > > ---(end of broadcast)--- > TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/users-lounge/docs/faq.html > ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Roadmap for a Win32 port
I think SGI gets amazing performance because they have very good (efficient) synchronisation primitives on SGI. Some proprietary light-weight mutexes. Using threaded or mixed model just by itself is not going to do a miracle. Threads will save you some context switch time, but that will probably translate into lower CPU usage rather than performance boost. And if your mutexes are not fast or awkwardly implemented (say Linux), it might be even worse. Apache is not all that fast on Linux as on SGI, whatever model you chose. I also doubt that purely threaded model would be slower than mixed one. Now about the AIO model. It is useful when you need to do something else while I/O requests are being processed as long as platform does it in some useful way. If all you can do is to submit requests and keep sitting in select/poll then AIO does not buy you anything you can't get by just using threaded model. However, if you can tag the requests and set up notifications, then few I/O threads could handle relatively large number of requests from different clients. Note, this means you don't have any association between clients and servers at all, there is pool of generic I/O threads which serve requests from whoever they come. It saves system resources and scales very well. It also provides interesting possibilities for fault recovery - since handlers are generic all the state information would have to be kept in some kind of global context area. That area can be saved into persistent memory or dumped onto disk and *recovered* after a forced restart. Server and library could be designed in such a way that clients may continue where they left with a recoverable error. In POSIX AIO model you can tag requests and set up notifications via synchronous signals. You wait for them *synchronously* in 'waiter' thread via sigwaitinfo() and avoid the headache of asynchronous signals hitting you any time... Unfortunately on some platforms (Solaris) the depth of synchronous signal queue is fixed at magic value 32 (and not adjustable). This may not be a problem if you're sure that waiting thread will be able to drain the queue faster than it gets filled with notifications... but I'm not sure there is a portable way to guarantee that, so you need to check for overloads and handle them... that complicates things. On Solaris you also need a mile of compiler/linker switches to even get this scheme to work and I am afraid other platforms may not support it at all (but then again, they may not support AIO to begin with). And speaking about getting best of all worlds. Note how Apache spent nearly 3 years developing their portable Multi-Processing Modules scheme. What they got for that is handful of models neither of which perform noticeably better than original pre-fork() model. Trying to swallow all possible ways to handle things on all possible platforms usually does not produce very fast code. It tends to produce very complex code with mediocre performance and introduces extra complexity into configuration process. If you consider all that was done mostly to support Win32, one might doubt if it was worth the while. What I am trying to say is, extra complexity in model to squeeze few percent of performance is not a wise investment of time and efforts. On Win32 you don't really compete in terms of performance. You compete in terms of easyness and features. Spend 3 years trying to support Windows and Unix in most optimal way including all subvariants of Unix ... meanwhile MSFT will come up with some bundled SQL server. It probably will have more features since they will spend time doing features rather than inventing a model to support gazillion of platforms. Chances are, it will be faster too - due to better integration with OS and better compiler. I am not in position to tell you what to do guys. But if I was asked, I'd say supporting Win32 is only worth it if it comes as a natural result of a simple, coherent and uniform model applied to Unix. Threaded model may not have as much inherent stability as forked/mixed, but it has inherent simplicity and better Unix/Windows/BeOS portability. It can be done faster and simpler code will make work on features easier. Regards, - Igor "There are 2 ways to design an efficient system - first is to design it so complex that there are no obvious deficiencies, second is to design it so simple that there are obviously no deficiencies. Second way is much harder" (author unknown to me) - Original Message - From: "Neil Conway" <[EMAIL PROTECTED]> To: "Jon Franz" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Wednesday, June 05, 2002 7:05 PM Subject: Re: [HACKERS] Roadmap for a Win32 port > On Wed, 5 Jun 2002 18:50:46 -0400 > "Jon Franz" <[EMAIL PROTECTED]> wrote: > > One note: SGI developers discovered they could get amazing performance using > > as hybrid threaded and forked-process model with apache - we might want to > > look into this. They even have a library for network-commun
Re: [HACKERS] Native Win32/OS2/BeOS/NetWare ports
> Hello together > > i've seen a lot of discussion about a native win32/OS2/BEOS port of > PostgreSQL. > > During the last months i've ported PostgreSQL over to Novell NetWare > and i've > changed the code that I use pthreads instead of fork() now. > > I had a lot of work with the variables and cleanup but mayor parts are > done. > > I would appreciate if we could combine this work. Very nice... I have patches for QNX6 which also involved redoing shared memory and sempahores stuff. It would make very good sense to intergate, especially since you managed to do something very close to what I wanted :) > My plan was to finish this port, discuss the port with other people and > offer all the work > to the PostgreSQL source tree, but now i'm jumping in here because of > all the discussions. > > What i've done in detail: > - i've defined #USE_PTHREADS in pg_config.h to differentiate between > the forked and the > threaded backend. > - I've added several parts in postmaster.c so all functions are based > on pthreads now. > - I've changed the signal handling because signals are process based Careful here. On certain systems (on many, I suspect) POSIX semantics for signals is NOT default. Enforcing POSIX semantics requires certain compile time switches which will also change behavior of various functions. > - I've changed code in ipc.c to have a clean shutdown of threads > - I've written some functions to switch the global variables. The > globals are controled with > POSIX semaphores. > - I've written a new implementation of shared memory and semaphores- > With pthreads I don't > need real shared memory any more and i'm using POSIX semaphores now POSIX semaphores for what? I assume by the conext that you're talking about replacing SysV semaphores which are used to control access to shared memory. If that is the case, POSIX semaphores are not the best choice really. POSIX mutexes would be okay, but on SMP systems spinlocks (hardware TAS based macros or POSIX spinlocks) would probably be better anyway. Note that on most platforms spinlocks are used for that and SysV semaphores were just a 'last resort' which had unacceptable performance and so I guess it was not used at all. Do you have your patch somewhere online? -- igor ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: Fw: Fw: [HACKERS] bad performance on irix
Makes me wonder... perhaps now someone will be convinced to take a look at the POSIX IPC patch. On some platforms (not on Linux I am afraid) POSIX mutexes might be quite a bit faster than SYSV semaphores. Luis Alberto Amigo Navarro wrote: > > Hi all: > again on performance, here is an extract from an 8 read-only queries, notice > that total time is 179s and it is expending about 80secs only in semaphores > Isn't there any other way to improve ipc-locks? > thanks and regards. > > ---(end of broadcast)--- > TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/users-lounge/docs/faq.html ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: Fw: Fw: [HACKERS] bad performance on irix
I am confused to hell. I always thought MIPS does NOT have TAS instruction ;) "Robert E. Bruccoleri" wrote: > > Dear Igor, > > Igor Kovalenko writes: > > > Makes me wonder... perhaps now someone will be convinced to take a look > > at the POSIX IPC patch. On some platforms (not on Linux I am afraid) > > POSIX mutexes might be quite a bit faster than SYSV semaphores. > > Yes, but on the SGI platform, the MIPS test_and_set instructions are > really fast and should be used. > > +-++ > | Robert E. Bruccoleri, Ph.D. | email: [EMAIL PROTECTED]| > | P.O. Box 314| URL: http://www.congen.com/~bruc | > | Pennington, NJ 08534|| > +-++ ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: Fw: Fw: [HACKERS] bad performance on irix
Okay. Anyway, the semaphores are apparently used for purposes other than TAS. That can be made faster too, on platforms which support POSIX mutexes (shared between processes). "Robert E. Bruccoleri" wrote: > > Dear Igor, > > > I am confused to hell. I always thought MIPS does NOT have TAS > > instruction ;) > > On the SGI platform, there are very high speed implementations of test > and set which allow large number of processes to safely and quickly > access shared memory. SGI has a hardware team that specifies MIPS > processor variants that are used in their servers so the machines can > scale. > > I've tried to get SGI interested in putting some internal engineering > effort to improve PostgreSQL performance on operations which could > benefit from its shared memory parallel architecture (like index > creation and sorting), but without success. > > +-++ > | Robert E. Bruccoleri, Ph.D. | email: [EMAIL PROTECTED]| > | P.O. Box 314| URL: http://www.congen.com/~bruc | > | Pennington, NJ 08534|| > +-++ ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: Fw: Fw: [HACKERS] bad performance on irix
No, I've been told it is not gonna be considered for 7.2x and I shall wait till 7.3. Luis Alberto Amigo Navarro wrote: > > > Makes me wonder... perhaps now someone will be convinced to take a look > > at the POSIX IPC patch. On some platforms (not on Linux I am afraid) > > POSIX mutexes might be quite a bit faster than SYSV semaphores. > > > Is there any current patch? > Regards ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: Fw: Fw: [HACKERS] bad performance on irix
Does that mean I should redo patch for 7.3 as is, or you guys want it to go farther this time? The last version had compromises intended to make changes minimal... Also, does anyone from Darwin or BeOS camp care? You guys should not be working through emulation of SysV ugliness. If someone is listening, we could come up with a version suitable for you too... -- igor Christopher Kings-Lynne wrote: > > Just remember that patches for 7.3 are being accepted at this very moment... > > Chris > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED]]On Behalf Of Igor Kovalenko > > Sent: Friday, 22 March 2002 1:31 AM > > To: Luis Alberto Amigo Navarro > > Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] > > Subject: Re: Fw: Fw: [HACKERS] bad performance on irix > > > > > > No, I've been told it is not gonna be considered for 7.2x and I shall > > wait till 7.3. > > > > Luis Alberto Amigo Navarro wrote: > > > > > > > Makes me wonder... perhaps now someone will be convinced to > > take a look > > > > at the POSIX IPC patch. On some platforms (not on Linux I am afraid) > > > > POSIX mutexes might be quite a bit faster than SYSV semaphores. > > > > > > > Is there any current patch? > > > Regards > > > > ---(end of broadcast)--- > > TIP 4: Don't 'kill -9' the postmaster > > ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
