subject:"\[HACKERS\] Listen \/ Notify \- what to do when the queue is full"

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-18 Thread Simon Riggs

On Mon, 2010-02-15 at 15:00 -0500, Tom Lane wrote:
 Joachim Wieland j...@mcknight.de writes:
  We could probably fake this on the Hot Standby in the following way:
 
  We introduce a commit record for every notifying transaction and write
  it into the queue itself. So right before writing anything else, we
  write an entry which informs readers that the following records are
  not yet committed. Then we write the actual notifications and commit.
  In post-commit we return back to the commit record and flip its
  status.
 
 This doesn't seem likely to work --- it essentially makes commit non
 atomic.  There has to be one and only one authoritative reference as
 to whether transaction X committed.

I thought a bit more about this and don't really understand why we need
an xid at all. When we discussed this before the role of a NOTIFY was to
remind us to refresh a cache, not as a way of delivering a transactional
payload. If the cache refresh use case is still the objective why does
it matter whether we commit or not when we issue a NOTIFY? Surely, the
rare case where we actually abort right at the end of the transaction
will just cause an unnecessary cache refresh. 

 I think that having HS slave sessions issue notifies is a fairly silly
 idea anyway.  They can't write the database, so exactly what condition
 are they going to be notifying others about?

Agreed

 What *would* be useful is for HS slaves to be able to listen for notify
 messages issued by writing sessions on the master.  This patch gets rid
 of the need for LISTEN to change on-disk state, so in principle we can
 do it.  The only bit we seem to lack is WAL transmission of the messages
 (plus of course synchronization in case a slave session is too slow
 about picking up messages).  Definitely a 9.1 project at this point
 though.

OK

-- 
 Simon Riggs   www.2ndQuadrant.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-18 Thread Josh Berkus

On 2/18/10 9:58 AM, Simon Riggs wrote:
 I thought a bit more about this and don't really understand why we need
 an xid at all. When we discussed this before the role of a NOTIFY was to
 remind us to refresh a cache, not as a way of delivering a transactional
 payload. If the cache refresh use case is still the objective why does
 it matter whether we commit or not when we issue a NOTIFY? Surely, the
 rare case where we actually abort right at the end of the transaction
 will just cause an unnecessary cache refresh. 

Actually, even for that use, it doesn't wash to have notifies being sent
for transactions which have not yet committed.  Think of cases (and I
have two applications) where data is being pushed into an external
non-transactional cache or queue (like Memcached, Redis or ApacheMQ)
from PostgreSQL on the backend.  If the transaction fails, it's
important that the data not get pushed.

I guess I'm not following why HS would be different from a single server
in this regard, though?

Mind you, this is all 9.1 discussion, no?

--Josh Berkus



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-18 Thread Merlin Moncure

On Thu, Feb 18, 2010 at 12:58 PM, Simon Riggs si...@2ndquadrant.com wrote:
 On Mon, 2010-02-15 at 15:00 -0500, Tom Lane wrote:
 Joachim Wieland j...@mcknight.de writes:
  We could probably fake this on the Hot Standby in the following way:

  We introduce a commit record for every notifying transaction and write
  it into the queue itself. So right before writing anything else, we
  write an entry which informs readers that the following records are
  not yet committed. Then we write the actual notifications and commit.
  In post-commit we return back to the commit record and flip its
  status.

 This doesn't seem likely to work --- it essentially makes commit non
 atomic.  There has to be one and only one authoritative reference as
 to whether transaction X committed.

 I thought a bit more about this and don't really understand why we need
 an xid at all. When we discussed this before the role of a NOTIFY was to
 remind us to refresh a cache, not as a way of delivering a transactional
 payload. If the cache refresh use case is still the objective why does
 it matter whether we commit or not when we issue a NOTIFY? Surely, the
 rare case where we actually abort right at the end of the transaction
 will just cause an unnecessary cache refresh.

notifications serve many more purposes than cache refreshes...it's a
generic 'wake up and do something' to the client.

For example, one of those things could be for the client to shut down.
 If the server errors out of the transaction that set up the client to
shut down, you probably wouldn't want the client to shut down.  I
don't think that's a big deal really, but it conflicts with the old
behavior.

However, being able to send notifications immediately (not at end of
transaction) would be exceptionally useful in some cases.   This
happens when the notifying backend is waiting on some sort of response
from the notified client.  If you could NOTIFY IMMEDIATELY, then you
could ping the client and get the response in a single transaction
without using dblink based hacks.

merlin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-18 Thread Tom Lane

Merlin Moncure mmonc...@gmail.com writes:
 On Thu, Feb 18, 2010 at 12:58 PM, Simon Riggs si...@2ndquadrant.com wrote:
 I thought a bit more about this and don't really understand why we need
 an xid at all. When we discussed this before the role of a NOTIFY was to
 remind us to refresh a cache, not as a way of delivering a transactional
 payload. If the cache refresh use case is still the objective why does
 it matter whether we commit or not when we issue a NOTIFY? Surely, the
 rare case where we actually abort right at the end of the transaction
 will just cause an unnecessary cache refresh.

 notifications serve many more purposes than cache refreshes...it's a
 generic 'wake up and do something' to the client.

The point to my mind is that the previous implementation guaranteed that
failed transactions would not send notifies.  I don't think we can just
drop that semantic consistency statement and not break applications.

Also, as Josh notes, even for cache refresh uses it is *critical* that
the notifies not be delivered to listeners till after the sender
commits; else you have race conditions where the listeners look for
changes before they can see them.  So it's difficult to make it
much simpler than this anyhow.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-17 Thread Joachim Wieland

On Tue, Feb 16, 2010 at 11:41 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 Joachim Wieland j...@mcknight.de writes:
 [ listen/notify patch ]

 I found a number of implementation problems having to do with wraparound
 behavior and error recovery.  I think they're all fixed, but any
 remaining bugs are probably my fault not yours.

First, thanks for the rework you have done and thanks for applying this.

While I can see a lot of improvements over my version, I think the
logic in asyncQueueProcessPageEntries() needs to be reordered:

+ static bool
+ asyncQueueProcessPageEntries(QueuePosition *current,
+QueuePosition stop,
+char *page_buffer)
[...]
+   do
+   {
[...]
+   /*
+* Advance *current over this message, possibly to the next 
page.
+* As noted in the comments for asyncQueueReadAllNotifications, 
we
+* must do this before possibly failing while processing the 
message.
+*/
+   reachedEndOfPage = asyncQueueAdvance(current, qe-length);
[...]
+   if (TransactionIdDidCommit(qe-xid))
[...]
+   else if (TransactionIdDidAbort(qe-xid))
[...]
+   else
+   {
+   /*
+* The transaction has neither committed nor 
aborted so far,
+* so we can't process its message yet.  Break 
out of the loop.
+*/
+   reachedStop = true;
+   break;

In the beginning you are advancing *current but later on you could
find out that the transaction is still running. As the position in the
queue has already advanced you would miss one notification here
because you'd restart directly behind this notification in the
queue...


Joachim

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-17 Thread Tom Lane

Joachim Wieland j...@mcknight.de writes:
 While I can see a lot of improvements over my version, I think the
 logic in asyncQueueProcessPageEntries() needs to be reordered:

Hmmm ... I was intending to cover the case of a failure in
TransactionIdDidCommit too, but I can see it will take a bit more
thought.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-16 Thread Greg Sabino Mullane


-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160


 * We also discussed the idea of having a NOTIFY command that 
 would work from Primary to Standby.

Just curious, what's a use case for this?

- -- 
Greg Sabino Mullane g...@turnstep.com
End Point Corporation http://www.endpoint.com/
PGP Key: 0x14964AC8 201002161102
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-BEGIN PGP SIGNATURE-

iEYEAREDAAYFAkt6wZ4ACgkQvJuQZxSWSsjrYwCfSWvHlTBFT/fIYcBToX9C57GO
toAAoOLQhBj6NdVTayaVtRH8L7nk16qM
=LBAH
-END PGP SIGNATURE-



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-16 Thread Jeff Davis

On Tue, 2010-02-16 at 16:02 +, Greg Sabino Mullane wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: RIPEMD160
 
 
  * We also discussed the idea of having a NOTIFY command that 
  would work from Primary to Standby.
 
 Just curious, what's a use case for this?

If you have some kind of cache above the DBMS, you need to invalidate it
when a part of the database is updated. It makes sense that every reader
would want to know about the update, not just those connected to the
master.

Regards,
Jeff Davis


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-16 Thread Tom Lane

Joachim Wieland j...@mcknight.de writes:
 [ listen/notify patch ]

Applied after rather a lot of hacking.

Aside from the issues previously raised, I changed the NOTIFY syntax to
include a comma between channel name and payload.  The submitted syntax
with no comma looked odd to me, and it would have been a real nightmare
to extend if we ever decide we want to support expressions in NOTIFY.

I found a number of implementation problems having to do with wraparound
behavior and error recovery.  I think they're all fixed, but any
remaining bugs are probably my fault not yours.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-16 Thread Tom Lane

Simon Riggs si...@2ndquadrant.com writes:
 * Don't really like pg_listening() as a name. Perhaps pg_listening_to()
 or pg_listening_on() or pg_listening_for() or pg_listening_channels() or
 pg_listen_channels()

BTW, I used pg_listening_channels() for that.

 * I think it's confusing that pg_notify is both a data structure and a
 function. Suggest changing one of those to avoid issues in
 understanding. Use pg_notify might be confused by a DBA.

I didn't change that.  The data structure is PGnotify, which seems
enough different from pg_notify to not be a real serious problem.
There is a duplication with the $PGDATA subdirectory pg_notify/,
but that one is not a user-facing name, so I thought it wasn't
really an issue.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-15 Thread Simon Riggs

On Sun, 2010-02-14 at 17:22 +0100, Joachim Wieland wrote:

 New patch attached, thanks for the review.

Next set of questions

* Will this work during Hot Standby now? The barrier was that it wrote
to a table and so we could not allow that. ISTM this new version can and
should work with Hot Standby. Can you test that and if so, remove the
explicit barrier code and change tests and docs to enable it?

* We also discussed the idea of having a NOTIFY command that would work
from Primary to Standby. All this would need is some code to WAL log the
NOTIFY if not in Hot Standby and for some recovery code to send the
NOTIFY to any listeners on the standby. I would suggest that would be an
option on NOTIFY to WAL log the notification:
e.g. NOTIFY me 'with_payload' FOR STANDBY ALSO;

* Don't really like pg_listening() as a name. Perhaps pg_listening_to()
or pg_listening_on() or pg_listening_for() or pg_listening_channels() or
pg_listen_channels()

* I think it's confusing that pg_notify is both a data structure and a
function. Suggest changing one of those to avoid issues in
understanding. Use pg_notify might be confused by a DBA.

-- 
 Simon Riggs   www.2ndQuadrant.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-15 Thread Joachim Wieland

On Sun, Feb 14, 2010 at 11:44 PM, Simon Riggs si...@2ndquadrant.com wrote:
Next set of questions

* Will this work during Hot Standby now? The barrier was that it wrote
to a table and so we could not allow that. ISTM this new version can and
should work with Hot Standby. Can you test that and if so, remove the
explicit barrier code and change tests and docs to enable it?

I have tested it already. The point where it currently fails is the
following line:

qe-xid = GetCurrentTransactionId();

We record the TransactionId (of the notifying transaction) in the
notification in order to later check if this transaction has committed
successfully or not. If you tell me how we can find this out in HS, we
might be done...

The reason why we are doing all this is because we fear that we can
not write the notifications to disk once we have committed to clog...
So we write them to disk before committing to clog and therefore need
to record the TransactionId.

* We also discussed the idea of having a NOTIFY command that would work
from Primary to Standby. All this would need is some code to WAL log the
NOTIFY if not in Hot Standby and for some recovery code to send the
NOTIFY to any listeners on the standby. I would suggest that would be an
option on NOTIFY to WAL log the notification:
e.g. NOTIFY me 'with_payload' FOR STANDBY ALSO;

What should happen if you wanted to replay a NOTIFY WAL record in the
standby but cannot write to the pg_notify/ directory?

* Don't really like pg_listening() as a name. Perhaps pg_listening_to()
or pg_listening_on() or pg_listening_for() or pg_listening_channels() or
pg_listen_channels()

pg_listen_channels() sounds best to me but I leave this decision to a
native speaker.

* I think it's confusing that pg_notify is both a data structure and a
function. Suggest changing one of those to avoid issues in
understanding. Use pg_notify might be confused by a DBA.

You are talking about the libpq datastructure PGnotify I suppose... I
don't see it overly confusing but I wouldn't object changing it. There
was a previous discussion about the name, see the last paragraph of
http://archives.postgresql.org/message-id/dc7b844e1002021510i4aaa879fy8bbdd003729d2...@mail.gmail.com

Joachim

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

1 2 >

1 - 100 of 113 matches

Mail list logo