On 10/23/2012 06:47 PM, Robert Haas wrote:
On Wed, Oct 17, 2012 at 4:25 PM, Josh Berkus <j...@agliodbs.com> wrote:
...
3. Double-down on #2 in a multithreaded environment.
For an application-level queue, the base functionality is:
ADD ITEM
READ NEXT (#) ITEM(S)
LOCK ITEM
DELETE ITEM
More sophisticated an useful queues also allow:
READ NEXT UNLOCKED ITEM
LOCK NEXT UNLOCKED ITEM
UPDATE ITEM
READ NEXT (#) UNSEEN ITEM(S)
The design you describe seems to prohibit pretty much all of the above
operations after READ NEXT. This makes it completely useless as a
application-level queue.
By the above logic MVCC "prohibits" UPDATES and DELETES on table data ;)
WAL-only tables/queues "prohobit" none of what you claim above, you just
implement in a (loosely) MVCC way by keeping track of what events are
processed.
And, for that matter, if your new queue only accepts INSERTs, why not
just improve LISTEN/NOTIFY so that it's readable on replicas? What does
this design buy you that that doesn't?
I get the ability to easily keep track of which events are already acted on
and which are not.
And you really can't fall back on processing LISTEN/NOTIFY - they
come when they come.
For WAL based event stream you only need to track LSN and in case
of multiple cooperative consumers (which I think Josh meant by
"multithreaded" above) a small structure to keep track of locking
and event consumption while The WAL part takes care of consistency,
order and durability.
I've read the whole thread, but I still don't see that anyone's said
it better than this, and I agree with these comments. (I don't find
them ad hominem, either.)
It's also worth noting that in order to be useful, this feature
intrinsically requires the logical replication stuff to be committed
first.
I agree that this feature - at least if implemented as
proposed - does need some underlying features from the Logical
Replication.
Otoh it does not really _need_ to have full logical replication
integrated - just
having a special WAL type and easy way for your own WAL reader (something
like pg_basebackup cold work well a a sample).
Without WAL-based logical replication I already can do the same
thing in a bit more expensive way by having a before trigger which
logs the insert in Slony/Londiste style event table and then cancels it
on the main table.
It's not entirely clear that there's not enough time to get
logical replication committed for 9.3, and the chances of getting any
follow-on features getting committed as well seems remote. Besides
the shortness of the time, I think all experience has shown that it's
best not to rush into the design of follow-on features before we've
got the basic feature well nailed down. This certainly can't be said
of logical replication at this point. Andres seems to be making good
progress and I'm grateful for his work on it, but I think there's a
lot left to do before that one is in the bag (as I think Andres would
agree).
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
