-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
You misunderstand the requirements. LISTEN notifications are *not*
meant to survive a database crash, and never have been. However,
so long as both client and server stay up, they must be reliable.
If the client has to poll database state
On Sat, Nov 14, 2009 at 11:06 PM, Merlin Moncure mmonc...@gmail.com wrote:
The old method (measured on a 4 core high performance server) has
severe scaling issues due to table bloat (we knew that):
./pgbench -c 10 -t 1000 -n -b listen.sql -f notify.sql
run #1 tps = 1364.948079 (including
On Mon, Nov 16, 2009 at 4:41 PM, Joachim Wieland j...@mcknight.de wrote:
On Sat, Nov 14, 2009 at 11:06 PM, Merlin Moncure mmonc...@gmail.com wrote:
The old method (measured on a 4 core high performance server) has
severe scaling issues due to table bloat (we knew that):
./pgbench -c 10 -t 1000
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
old method scaled (badly) on volume of notifications and your stuff
seems to scale based on # of client's sending simultaneous
notifications. Well, you're better all day long, but it shows that
your fears regarding locking were not
On Thu, 12 Nov 2009 11:22:32 -0500
Andrew Chernow a...@esilo.com wrote:
However I share Greg's concerns that people are trying to use NOTIFY
as a message queue which it is not designed to be.
When you have an established libpq connection waiting for notifies it
is not unreasonable to
On Wed, 2009-11-11 at 22:25 +0100, Joachim Wieland wrote:
3. Every distinct notification is delivered.
Regarding performance, the slru-queue is not fsync-ed to disk
These two statements seem to be in opposition. How do you know a
notification will be delivered if the queue is
Simon Riggs si...@2ndquadrant.com writes:
On Wed, 2009-11-11 at 22:25 +0100, Joachim Wieland wrote:
3. Every distinct notification is delivered.
Regarding performance, the slru-queue is not fsync-ed to disk
These two statements seem to be in opposition. How do you know a
notification will be
On Sun, 2009-11-15 at 16:48 -0500, Tom Lane wrote:
Simon Riggs si...@2ndquadrant.com writes:
On Wed, 2009-11-11 at 22:25 +0100, Joachim Wieland wrote:
3. Every distinct notification is delivered.
Regarding performance, the slru-queue is not fsync-ed to disk
These two statements seem to
Simon Riggs si...@2ndquadrant.com writes:
On Sun, 2009-11-15 at 16:48 -0500, Tom Lane wrote:
You misunderstand the requirements. LISTEN notifications are *not*
meant to survive a database crash, and never have been. However,
so long as both client and server stay up, they must be reliable.
On Fri, Nov 13, 2009 at 11:08 AM, Tom Lane t...@sss.pgh.pa.us wrote:
(By the way, has anyone yet tried to
compare the speed of this implementation to the old code?)
I quickly hacked pgbench to take a custom script on connection (for
listen), and make pgbench'd 'notify x'; with all clients doing
Unfortunately with all that payload-length discussion, the other part
of my email regarding ACID compliant behavior got completely lost. I
would appreciate some input on that part also...
Thanks,
Joachim
On Thu, Nov 12, 2009 at 12:17 PM, Joachim Wieland j...@mcknight.de wrote:
On Thu, Nov 12,
On Fri, Nov 13, 2009 at 1:57 AM, Robert Haas robertmh...@gmail.com wrote:
I agree. We frequently reject features on the basis that someone
might do something stupid with them. It's lame and counterproductive,
and we should stop. The world contains infinite amounts of lameness,
but that's
On Fri, Nov 13, 2009 at 5:35 AM, Greg Stark gsst...@mit.edu wrote:
On Fri, Nov 13, 2009 at 1:57 AM, Robert Haas robertmh...@gmail.com wrote:
I agree. We frequently reject features on the basis that someone
might do something stupid with them. It's lame and counterproductive,
and we should
Joachim Wieland wrote:
1. Instead of placing the queue into shared memory only I propose to create a
new subdirectory pg_notify/ and make the queue slru-based, such that we do not
risk blocking. Several people here have pointed out that blocking is a true
no-go for a new listen/notify
spill to disk and need an efficient storage mechanism. The natural
implementation of this in Postgres would be a table, not the slru. If
This is what I think the people's real problem is, the implementation becomes a
more complex when including payloads (larger ones even more so). I think
On Fri, Nov 13, 2009 at 1:47 PM, Andrew Chernow a...@esilo.com wrote:
This is what I think the people's real problem is, the implementation
becomes a more complex when including payloads (larger ones even more so).
I think its a side-track to discuss queue vs condition variables. Whether
a
Calling this a creeping feature is quite a leap.
It's true that the real creep is having the payload at all rather than
not having it.
Not having the payload at all is like santa showing up without his bag
of toys. Instead, you have to drive/fly to the north pole where he just
came from to
On Fri, Nov 13, 2009 at 10:00 AM, Andrew Chernow a...@esilo.com wrote:
I think the original OP was close. The structure can still be fixed length
but maybe we can bump it to 8k (BLCKSZ)?
The problem with this (which I basically agree with) is that this will
greatly increase the size of the
Merlin Moncure mmonc...@gmail.com writes:
The problem with this (which I basically agree with) is that this will
greatly increase the size of the queue for all participants of this
feature if they use the payload or not. I think it boils down to
this: is there a reasonably effective way of
Merlin Moncure wrote:
On Fri, Nov 13, 2009 at 10:00 AM, Andrew Chernow a...@esilo.com wrote:
I think the original OP was close. The structure can still be fixed length
but maybe we can bump it to 8k (BLCKSZ)?
The problem with this (which I basically agree with) is that this will
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
This is BS. The problem is not that someone might do something stupid
with this feature. The problem is that we're making these other use
cases into
Greg Sabino Mullane g...@turnstep.com writes:
Talk of efficiency also seems silly here - using
shared memory is already way more efficient than our current listen/notify
system.
Except that the proposed implementation spills to disk. Particularly if
it has to have support for large payloads,
Tom Lane wrote:
Greg Sabino Mullane g...@turnstep.com writes:
Talk of efficiency also seems silly here - using
shared memory is already way more efficient than our current listen/notify
system.
Except that the proposed implementation spills to disk. Particularly if
it has to have support for
My original intention was to have the queue as a circular buffer where
the size of the entries was variable, but possibly bounded. Certainly
using fixed length records of large size seems somewhat wasteful.
Maybe we should do away with 'spill to disk' all together and either
hard-code an
Josh Berkus wrote:
Payloads are also quite useful even in a lossy environment, where you
understand that LISTEN is not a queue. For example, I'd like to be
using LISTEN/NOTIFY for cache invalidation for some applications; if it
misses a few, or double-counts them, it's not an issue. However,
On Thu, Nov 12, 2009 at 2:12 AM, Andrew Gierth
and...@tao11.riddles.org.uk wrote:
Does it cope with the case where a trigger is doing NOTIFY, and you do
a whole-table update, therefore dumping potentially millions of
notifications in at once?
(for example a rare maintenance operation on a
On Thu, Nov 12, 2009 at 4:25 AM, Tom Lane t...@sss.pgh.pa.us wrote:
One possible solution would be to write to the queue before committing
and adding the TransactionID. Then other backends can check if our
TransactionID has successfully committed or not. Not sure if this is
worth the overhead
2. The payload parameter is optional. A notifying client can either call
NOTIFY foo; or NOTIFY foo 'payload';. The length of the payload is
currently limited to 128 characters... Not sure if we should allow longer
payload strings...
Might be a good idea to make the max the same as the max
On Thu, Nov 12, 2009 at 8:25 AM, Andrew Chernow a...@esilo.com wrote:
2. The payload parameter is optional. A notifying client can either call
NOTIFY foo; or NOTIFY foo 'payload';. The length of the payload is
currently limited to 128 characters... Not sure if we should allow
longer
payload
What advantage is there in limiting it to a tiny size? This is a
'payload' after all...an arbitrary data block. Looking at the patch I
noticed the payload structure (AsyncQueueEntry) is fixed length and
designed to lay into QUEUE_PAGESIZE (set to) BLCKSZ sized pages.
H. Looks like the
On Thu, Nov 12, 2009 at 3:09 PM, Andrew Chernow a...@esilo.com wrote:
What advantage is there in limiting it to a tiny size? This is a
'payload' after all...an arbitrary data block. Looking at the patch I
noticed the payload structure (AsyncQueueEntry) is fixed length and
designed to lay
On Thu, Nov 12, 2009 at 3:30 PM, Merlin Moncure mmonc...@gmail.com wrote:
Couple of questions:
*) is BLCKSZ a hard requirement, that is, coming from the slru
implementation, or can QUEUE_PAGESIZE be bumped independently of block
size.
It's the size of slru's pages.
*) why not make the
--On 12. November 2009 15:48:44 + Greg Stark gsst...@mit.edu wrote:
I'm beginning to think the right solution is to reject the idea of
adding a payload to the NOTIFY mechanism and instead provide a queue
contrib module which provides the interface people want.
Isn't PgQ already the
On Thu, Nov 12, 2009 at 11:30 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Joachim Wieland j...@mcknight.de writes:
However I share Greg's concerns that people are trying to use NOTIFY
as a message queue which it is not designed to be.
Yes. Particularly those complaining that they want to have
On Thu, Nov 12, 2009 at 11:30 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Joachim Wieland j...@mcknight.de writes:
However I share Greg's concerns that people are trying to use NOTIFY
as a message queue which it is not designed to be.
Yes. Particularly those complaining that they want to have
On Thu, Nov 12, 2009 at 11:39 AM, Robert Haas robertmh...@gmail.com wrote:
On Thu, Nov 12, 2009 at 11:30 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Joachim Wieland j...@mcknight.de writes:
However I share Greg's concerns that people are trying to use NOTIFY
as a message queue which it is not
Now you might say that yeah, that's the point, we're trying to enable
using NOTIFY in a different style. The problem is that if you are
trying to use NOTIFY as a queue, you will soon realize that it has
the wrong semantics for that --- in particular, losing notifies across
a system crash or
On Thu, Nov 12, 2009 at 4:30 PM, Tom Lane t...@sss.pgh.pa.us wrote:
So while a payload string for NOTIFY has been on the to-do list since
forever, I have to think that Greg's got a good point questioning
whether it is actually a good idea.
Here's an example of why I'd like a payload (and not
Joachim Wieland j...@mcknight.de writes:
However I share Greg's concerns that people are trying to use NOTIFY
as a message queue which it is not designed to be.
Yes. Particularly those complaining that they want to have very large
payload strings --- that's pretty much a dead giveaway that
However I share Greg's concerns that people are trying to use NOTIFY
as a message queue which it is not designed to be.
When you have an established libpq connection waiting for notifies it is
not unreasonable to expect/desire a payload. ISTM, the problem is that
the initial design was
On Thu, Nov 12, 2009 at 11:40 AM, Merlin Moncure mmonc...@gmail.com wrote:
I'm sorry, the 128 character limit is simply lame (other than for
unsolvable implementation/performance complexity which I doubt is the
case here), and if that constraint is put in by the implementation,
than the
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
Tom Lane writes:
Yes. Particularly those complaining that they want to have very large
payload strings --- that's pretty much a dead giveaway that it's not
being used as a
On 11/12/09 8:30 AM, Tom Lane wrote:
So while a payload string for NOTIFY has been on the to-do list since
forever, I have to think that Greg's got a good point questioning
whether it is actually a good idea.
Sure, people will abuse it as a queue. But people abuse arrays when
they should be
On Thu, Nov 12, 2009 at 8:44 PM, Josh Berkus j...@agliodbs.com wrote:
On 11/12/09 8:30 AM, Tom Lane wrote:
So while a payload string for NOTIFY has been on the to-do list since
forever, I have to think that Greg's got a good point questioning
whether it is actually a good idea.
Sure, people
and we should stop. The world contains infinite amounts of lameness,
but that's the world's problem, not ours. There is zero evidence that
+1
this feature is only useful for stupid purposes, and some evidence
(namely, the opinions of esteemed community members) that it is useful
for at
On Nov 12, 2009, at 5:57 PM, Robert Haas wrote:
On Thu, Nov 12, 2009 at 8:44 PM, Josh Berkus j...@agliodbs.com wrote:
On 11/12/09 8:30 AM, Tom Lane wrote:
So while a payload string for NOTIFY has been on the to-do list since
forever, I have to think that Greg's got a good point questioning
On Wed, Nov 11, 2009 at 10:25:05PM +0100, Joachim Wieland wrote:
Hi,
Attached is a patch for a new listen/notify implementation.
In a few words, the patch reimplements listen/notify as an slru-based queue
which works similar to the sinval structure. Essentially it is a ring buffer
on
On Nov 11, 2009, at 4:25 PM, Joachim Wieland wrote:
Hi,
Attached is a patch for a new listen/notify implementation.
In a few words, the patch reimplements listen/notify as an slru-
based queue
which works similar to the sinval structure. Essentially it is a
ring buffer on
disk with pages
2. adds the possibility to specify a payload parameter, i.e. executing
in SQL
NOTIFY foo 'payload'; and 'payload' will be delivered to any
listening
backend.
Thank you for implementing this- LISTEN/NOTIFY without a payload has
been a major problem to work around for me.
+1
--
Andrew
/*
+ * This function is executed for every notification found in the queue in
order
+ * to check if the current backend is listening on that channel. Not sure if
we
+ * should further optimize this, for example convert to a sorted array and
+ * allow binary search on it...
+ */
+ static
On Thu, Nov 12, 2009 at 1:04 AM, Andrew Chernow a...@esilo.com wrote:
I think a bsearch would be needed. Very busy servers that make heavy use of
notifies would be quite a penalty.
In such an environment, how many relations/channels is a backend
typically listening to?
Do you have average /
Joachim Wieland wrote:
On Thu, Nov 12, 2009 at 1:04 AM, Andrew Chernow a...@esilo.com wrote:
I think a bsearch would be needed. Very busy servers that make heavy use of
notifies would be quite a penalty.
In such an environment, how many relations/channels is a backend
typically listening to?
Andrew Chernow a...@esilo.com writes:
+ * This function is executed for every notification found in the queue in
order
+ * to check if the current backend is listening on that channel. Not sure
if we
+ * should further optimize this, for example convert to a sorted array and
+ * allow
Martijn == Martijn van Oosterhout klep...@svana.org writes:
Hi,
Attached is a patch for a new listen/notify implementation.
In a few words, the patch reimplements listen/notify as an
slru-based queue which works similar to the sinval
structure. Essentially it is a ring buffer on
Premature optimization is the root of all evil ;-). Unless you've done
some profiling and can show that this is a hot spot, making it more
complicated isn't the thing to be doing now.
I'm thinking of how our system uses/abuses notifies, and began wondering
if several thousand backends
On Wed, Nov 11, 2009 at 5:48 PM, A.M. age...@themactionfaction.com wrote:
At least with this new payload, I can set the payload to the transaction ID
and be certain that all the notifications I sent are processed (and in order
even!) but could you explain why the coalescing is still necessary?
On Nov 11, 2009, at 9:28 PM, Merlin Moncure wrote:
On Wed, Nov 11, 2009 at 5:48 PM, A.M.
age...@themactionfaction.com wrote:
At least with this new payload, I can set the payload to the
transaction ID
and be certain that all the notifications I sent are processed
(and in order
even!) but
Joachim Wieland j...@mcknight.de writes:
However, if for some reason we cannot write to the slru files in the
pg_notify/
directory we might want to roll back the current transaction but with the
proposed patch we cannot because we have already committed...
I think this is a deal-breaker, and
I thought of a compromise: add the number of times a notification was
generated (coalesced count+1) to the callback data. That would satisfy
any backwards compatibility concerns and my use case too!
If you are suggesting that the server poke data into the notifier's opaque
payload, I
Andrew Chernow a...@esilo.com writes:
I thought of a compromise: add the number of times a notification was
generated (coalesced count+1) to the callback data. That would satisfy
any backwards compatibility concerns and my use case too!
If you are suggesting that the server poke data into
On Nov 11, 2009, at 10:43 PM, Tom Lane wrote:
Andrew Chernow a...@esilo.com writes:
I thought of a compromise: add the number of times a notification
was
generated (coalesced count+1) to the callback data. That would
satisfy
any backwards compatibility concerns and my use case too!
If
2. The payload parameter is optional. A notifying client can either call
NOTIFY foo; or NOTIFY foo 'payload';. The length of the payload is
currently limited to 128 characters... Not sure if we should allow longer
payload strings...
Might be a good idea to make the max the same as the max
62 matches
Mail list logo
