Re: [HACKERS] LISTEN/NOTIFY enhancement: Portable signal handling?
In case anyone's wondering why I'm replying to an article that's nearly a year old, it came up in a search while I was looking for the more recent LISTEN/NOTIFY thread... In article [EMAIL PROTECTED], Tom Lane [EMAIL PROTECTED] wrote: Sean Chittenden [EMAIL PROTECTED] writes: The async interface is nice, but not really useful to me as it requires polling, instead of unblocking when an event comes through, which would create a vastly more real time interface that should be easier on the database. BTW, this is nonsense; the backend sends a message exactly when the NOTIFY occurs. It may well be that libpq needs some work to make it easier to wait for a NOTIFY without polling, but again there isn't a reason to clutter the server-side code with it. One thing that persistently annoys me about using LISTEN in clients is that much of the time I'm not using libpq directly, but some layer above libpq that implements a generic SQL interface. These interfaces generally assume that SQL calls are synchronous and blocking unless you use some kind of escape to the lower-level driver, and in a few environments there is no such escape at all. I'd really like to have something that looks as much like an SQL statement as possible, which blocks until a NOTIFY event or a timeout occurs. If not a stand-alone SQL command then at least a function. In one case where I really needed this, I implemented a really *ugly* PL/PerlU function that did the following: open a new client connection (with caching) to the server from the server's own backend set up appropriate LISTEN commands, go to sleep on a poll() from the file descriptor, then return immediately after poll() does. Thankfully this particular application was not required to scale beyond a half-dozen clients! -- Zygo Blaxell (Laptop) [EMAIL PROTECTED] GPG = B203 9402 B0E7 0F20 3B74 B13C DFBC C916 4395 FD03 ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
[HACKERS] LISTEN/NOTIFY enhancement: Portable signal handling?
Howdy. I'm starting some work on our favorite LISTEN/NOTIFY subsystem in the hopes of more advanced functionality. Right now I'm using a bastardized (RAISE NOTICE + tailing error logs) + NOTIFY to get what should just be built in to the LISTEN/NOTIFY interface. Here's the syntax for the proposed functionality. I've updated the grammar to support the following: LISTEN name [BLOCK] [WITH] [TIMEOUT [=] ''::INTERVAL]; NOTIFY name a_expr; But am having something of a think-o when it comes to the semantics of the BLOCK keyword for the LISTEN command. Let me first explain my goals for this exercise: * Allow passing of a data type in a NOTIFY. Something like: NOTIFY 'relname' a_expr; Should be able to pass a text, INET, etc... shouldn't matter... not sure if this is possible though given that OIDs don't travel with data types... I may have to settle for a TEXT arg, which is acceptable. * Allow LISTEN to block until a value changes. LISTEN [BLOCK|WAIT] 'relname' * Allow LISTEN to have a timeout LISTEN name [BLOCK] [WITH] [TIMEOUT [=] ''::INTERVAL]; * Allow blocking LISTEN queries to update the status of the proc title while blocking. Basically I want to introduce formal support for turning PostgreSQL into a message bus. To start with, it won't be a scalable message bus that can scale to thousands of concurrent connections (that's something I'd like to do long term(tm), but I digress). The problem is with the BLOCK'ing semantics or implementation. There are two ways that I can do this, IMHO, but I'm out to solicit alternatives as I'm still getting a handle on what the best interfaces/APIs are internally to get something done. Option 1) Use sleep(3) for the given timeout and wake up on some interruptible a signal (USR2?). This is the simplest solution, but likely the least portable to win32. Given the new world order of 8.0 and it's portability headaches, it's something I'm aware of. Option 2) block on an exclusive lock. Check to see if relname has been registered. If it has, block on the existing exclusive lock (can I block on a lock for a given number of sec/ms?). If it hasn't, create an exclusive lock, but give the lock away (to the parent postmaster, a lockmgr proc, etc) so that a notify can remove the remove and unlock the exclusive lock so that all of the blocking children wake up. The async interface is nice, but not really useful to me as it requires polling, instead of unblocking when an event comes through, which would create a vastly more real time interface that should be easier on the database. Does anyone think there would be any conflicts with the use of the existing alarm code from storage/lmgr/proc.c for the LISTEN/NOTIFY interface? It looks like SIGALRM has a reserved purpose. I haven't found any global alarm handling interface that can be used to assign different meanings when an SIGALRM is received. Any other thoughts on the best way to proceed? -sc -- Sean Chittenden ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] LISTEN/NOTIFY enhancement: Portable signal handling?
Basically I want to introduce formal support for turning PostgreSQL into a message bus. To start with, it won't be a scalable message bus that can scale to thousands of concurrent connections (that's something I'd like to do long term(tm), but I digress). The problem is with the BLOCK'ing semantics or implementation. There are two ways that I can do this, IMHO, but I'm out to solicit alternatives as I'm still getting a handle on what the best interfaces/APIs are internally to get something done. Sounds great :-) Option 1) Use sleep(3) for the given timeout and wake up on some interruptible a signal (USR2?). This is the simplest solution, but likely the least portable to win32. Given the new world order of 8.0 and it's portability headaches, it's something I'm aware of. There should be no problem doing this on win32. The portability layer (in backend/port/win32) implements a sleep (pg_usleep, that is) that can be interrupted by any signal (our win32 signal implementation actually supports 256 signals, but you'd have to stick to a signal that's supported on all platforms and also not already in use. And that's going to be a lot harder unless you can piggyback on something that's already there). The async interface is nice, but not really useful to me as it requires polling, instead of unblocking when an event comes through, which would create a vastly more real time interface that should be easier on the database. As long as you're working only from the client side, you don't need to do polling. See the bottom of the page at http://developer.postgresql.org/docs/postgres/libpq-notify.html. You just need to pass the socket down to select(). I'm doing this from C and perl, don't know about other languages. That said, there are certainly a lot of cases where a blocking call at the SQL level would also be useful. Just don't kill what we have now :-) //Magnus ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] LISTEN/NOTIFY enhancement: Portable signal handling?
Sean Chittenden [EMAIL PROTECTED] writes: * Allow LISTEN to block until a value changes. LISTEN [BLOCK|WAIT] 'relname' * Allow LISTEN to have a timeout LISTEN name [BLOCK] [WITH] [TIMEOUT [=] ''::INTERVAL]; * Allow blocking LISTEN queries to update the status of the proc title while blocking. I don't believe in any of these things, at least not on the server side. You can get the same effect on the client side without cluttering the LISTEN semantics and implementation. regards, tom lane ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] LISTEN/NOTIFY enhancement: Portable signal handling?
Sean Chittenden [EMAIL PROTECTED] writes: The async interface is nice, but not really useful to me as it requires polling, instead of unblocking when an event comes through, which would create a vastly more real time interface that should be easier on the database. BTW, this is nonsense; the backend sends a message exactly when the NOTIFY occurs. It may well be that libpq needs some work to make it easier to wait for a NOTIFY without polling, but again there isn't a reason to clutter the server-side code with it. regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] LISTEN/NOTIFY enhancement: Portable signal handling?
Sean Chittenden wrote: Option 1) Use sleep(3) for the given timeout and wake up on some interruptible a signal (USR2?). This is the simplest solution, but likely the least portable to win32. Given the new world order of 8.0 and it's portability headaches, it's something I'm aware of. Option 2) block on an exclusive lock. Check to see if relname has been registered. If it has, block on the existing exclusive lock (can I block on a lock for a given number of sec/ms?). If it hasn't, create an exclusive lock, but give the lock away (to the parent postmaster, a lockmgr proc, etc) so that a notify can remove the remove and unlock the exclusive lock so that all of the blocking children wake up. The async interface is nice, but not really useful to me as it requires polling, instead of unblocking when an event comes through, which would create a vastly more real time interface that should be easier on the database. Does anyone think there would be any conflicts with the use of the existing alarm code from storage/lmgr/proc.c for the LISTEN/NOTIFY interface? It looks like SIGALRM has a reserved purpose. I haven't found any global alarm handling interface that can be used to assign different meanings when an SIGALRM is received. Any other thoughts on the best way to proceed? You can make cooperative blocking locks using some slick client side code and the swiss army knife userlock module. Since user locks pierce transaction encapsulation they can be used for these types of things. select user_write_lock(some number); if return = 1 notify some message else wait a and try again, etc. end if // release lock, etc. NOTIFY 'relname' a_expr; This would be great to have...at least I think this is what you are driving at: (adding a noiseword for readability) LISTEN system_messages; NOTIFY system_messages MESSAGE logoff; NOTIFY request_unlock MESSAGE 12345; -- for use with user locks! Etc. Another cute tidbit about the listen/notify interface is that it can be abused to create server side storage that is cleaned up by the server when a backend drops. I was using this to crate a crude pessimistic locking system before I discovered the userlock module. Note that this is not a scalable or robust approach however. listen 12345; --acquired a lock on '12345' select listenerpid from pg_listener where relname = '12345' Merlin ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] LISTEN/NOTIFY enhancement: Portable signal handling?
NOTIFY 'relname' a_expr; This would be great to have...at least I think this is what you are driving at: (adding a noiseword for readability) LISTEN system_messages; NOTIFY system_messages MESSAGE logoff; NOTIFY request_unlock MESSAGE 12345; -- for use with user locks! Hrm... the userlock module may work for what I'm after, but the module is GPL'ed, which is something I don't want to rely on. I wanted to have a backend process essentially act as a long running client that would act on notify messages to dispatch work. I was hoping to accomplish something like, LISTEN REGISTER event_name; LISTEN * BLOCK WITH TIMEOUT = 30; which would only listen to events that you've registered to listen to, but would only deliver messages when you explicitly were going to block waiting for message delivery (ie, LISTEN BLOCK). I'll settle for getting just message passing via NOTIFY/LISTEN to work and will revisit extending things further once I have this bit done. -sc -- Sean Chittenden ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [HACKERS] LISTEN/NOTIFY enhancement: Portable signal handling?
The async interface is nice, but not really useful to me as it requires polling, instead of unblocking when an event comes through, which would create a vastly more real time interface that should be easier on the database. BTW, this is nonsense; the backend sends a message exactly when the NOTIFY occurs. It may well be that libpq needs some work to make it easier to wait for a NOTIFY without polling, but again there isn't a reason to clutter the server-side code with it. It's asynchronous with regards to client delivery of the message. Sending of the NOTIFY message is synchronous, but delivery to the client is not. I don't see how it could be any other way in PostgreSQL. libpq(3) actually has a reasonable interface that relies on the developer to block on the fd as described here: http://developer.postgresql.org/docs/postgres/libpq-notify.html The problem is that this doesn't work in pl/*, which is the problem I was trying to address. *shrug* -sc -- Sean Chittenden ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
