Re: [HACKERS] logical replication - still unstable after all these months

2017-05-29 Thread Petr Jelinek
bscriber (the lines about when sync workers finished), contents of the pg_subscription_rel (with srrelid casted to regclass so we know which table is which), and pg_waldump output around the LSNs found in the logs and in the pg_subscription_rel (+ few lines before and some after to get context). It'

Re: [HACKERS] logical replication busy-waiting on a lock

2017-05-27 Thread Petr Jelinek
On 27/05/17 15:44, Petr Jelinek wrote: > On 27/05/17 01:25, Jeff Janes wrote: >> When I create a subscription in the disabled state, and then later doing >> "alter subscription sub enable;", on the master I sometimes get a tight >> loop of the deadlock detector:

Re: [HACKERS] logical replication busy-waiting on a lock

2017-05-27 Thread Petr Jelinek
he XactLockTableWait releases the lock immediately after acquiring it. In fact AFAICS everything that acquires ShareLock on xid releases it immediately after acquiring as it's only used for waits. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Su

Re: [HACKERS] ALTER SUBSCRIPTION ..SET PUBLICATION refresh is not throwing error.

2017-05-27 Thread Petr Jelinek
On 27/05/17 04:00, Euler Taveira wrote: > 2017-05-26 21:29 GMT-03:00 Petr Jelinek <mailto:petr.jeli...@2ndquadrant.com>>: > > > Actually another possibility would be to remove the REFRESH keyword > completely and just have [ WITH (...) ] and have the refres

Re: [HACKERS] ALTER SUBSCRIPTION ..SET PUBLICATION refresh is not throwing error.

2017-05-26 Thread Petr Jelinek
On 27/05/17 02:13, Euler Taveira wrote: > 2017-05-26 17:58 GMT-03:00 Peter Eisentraut > <mailto:peter.eisentr...@2ndquadrant.com>>: > > On 5/24/17 15:38, Petr Jelinek wrote: > >>> I wonder if we actually need the SKIP REFRESH syntax, there is the >

Re: [HACKERS] logical replication - still unstable after all these months

2017-05-26 Thread Petr Jelinek
ome more output from the tests (do they log some intermediary states of those md5 checksums, maybe numbers of rows etc?), description of the problems, errors from logs, etc. I for example don't get any issues from similar test as the one described in this thread so without more info it mig

Re: [HACKERS] logical replication - still unstable after all these months

2017-05-26 Thread Petr Jelinek
Hi, Hmm, I was under the impression that the changes we proposed in the snapbuild thread fixed your issues, does this mean they didn't? Or the modified versions of those that were eventually committed didn't? Or did issues reappear at some point? -- Petr Jelinek

Re: [HACKERS] Create subscription with `create_slot=false` and incorrect slot name

2017-05-25 Thread Petr Jelinek
exist, still active, no > permission, etc.). We don't want to give the hint that is effectively > "just forget about the slot then" for all of them. So we would need > some way to distinguish "you can't do this right now" from "this would > never

Re: [HACKERS] No parameter values checking while creating Alter subscription...Connection

2017-05-25 Thread Petr Jelinek
think during reconfigurations it's quite useful to be able to do so > even if the other hosts aren't reachable that second. > Yes, it's intended behavior for this very reason, we want ability to (re)configure downstream without any existing upstream. It's

[HACKERS] Walsender timeouts and large transactions

2017-05-25 Thread Petr Jelinek
rwise walsender might ignore them for too long on large transactions. Thoughts? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services 0001-Fix-walsender-timeouts-when-decoding-large-transacti.patch Description: binary/oct

Re: [HACKERS] ALTER SUBSCRIPTION ..SET PUBLICATION refresh is not throwing error.

2017-05-24 Thread Petr Jelinek
On 24/05/17 21:28, Petr Jelinek wrote: > On 24/05/17 21:21, Petr Jelinek wrote: >> On 24/05/17 21:07, Euler Taveira wrote: >>> 2017-05-23 6:00 GMT-03:00 tushar >> <mailto:tushar.ah...@enterprisedb.com>>: >>> >>> >>> s=# alter su

Re: [HACKERS] ALTER SUBSCRIPTION ..SET PUBLICATION refresh is not throwing error.

2017-05-24 Thread Petr Jelinek
On 24/05/17 21:21, Petr Jelinek wrote: > On 24/05/17 21:07, Euler Taveira wrote: >> 2017-05-23 6:00 GMT-03:00 tushar > <mailto:tushar.ah...@enterprisedb.com>>: >> >> >> s=# alter subscription s1 set publication skip refresh ; >> NOT

Re: [HACKERS] ALTER SUBSCRIPTION ..SET PUBLICATION refresh is not throwing error.

2017-05-24 Thread Petr Jelinek
ITH ... ]" when user wants to refresh, so if REFRESH is not specified, we can just behave as if SKIP REFRESH was used, it's not like there is 3rd possible behavior. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-05-24 Thread Petr Jelinek
ding issue that sync worker will keep running inside the long COPY because the invalidation messages are also not processed until it finishes but all the original issues reported here disappear for me with the attached patches applied. -- Petr Jelinek http://www.2ndQuadra

Re: [HACKERS] walsender & parallelism

2017-05-23 Thread Petr Jelinek
On 23/05/17 19:45, Andres Freund wrote: > > > On May 23, 2017 1:42:41 PM EDT, Petr Jelinek > wrote: >> Hi, >> >> so this didn't really move anywhere AFAICS, do we think the approach >> I've chosen is good or do we want to do something else he

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-05-23 Thread Petr Jelinek
to add boolean that indicates that the worker is not using specific user (this can be easily set in InitializeSessionUserIdStandalone()) and will work for multiple things. About application_name, perhaps we should just add bgw_type or bgw_group and show it as worker_type in activity and that's it? I think thi

Re: [HACKERS] walsender & parallelism

2017-05-23 Thread Petr Jelinek
Hi, so this didn't really move anywhere AFAICS, do we think the approach I've chosen is good or do we want to do something else here? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-h

Re: [HACKERS] Multiple table synchronizations are processed serially

2017-05-19 Thread Petr Jelinek
> has to wait for that. > Well IMHO this concrete example is partially effect of the snapshort builder fixes we did with Andres. Before those fixes the ondisk snapshot may have been used in this situation. Makes me think if we want to mark the exported snapshot as containing all transacti

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-05-19 Thread Petr Jelinek
er accesses however, the workers need to be able to access the catalog during clean shutdown in some situations. What we need is to block starting of new workers for that subscription so only those code paths would need to block. So I think we might want to do both finer-grained locking and decrea

Re: [HACKERS] snapbuild woes

2017-05-14 Thread Petr Jelinek
On 13/05/17 22:23, Andres Freund wrote: > On 2017-05-12 10:57:55 +0200, Petr Jelinek wrote: >> Hmm, well it helps but actually now that we don't track individual >> running transactions anymore it got much less effective (my version of >> 0005 does as well). >> >

Re: [HACKERS] snapbuild woes

2017-05-12 Thread Petr Jelinek
On 12/05/17 10:57, Petr Jelinek wrote: > On 12/05/17 03:31, Andres Freund wrote: >> On 2017-05-11 14:54:26 -0700, Andres Freund wrote: >>> On 2017-05-11 14:51:55 -0700, wrote: >>>> On 2017-05-08 00:10:12 -0700, Andres Freund wrote: >>>>> I plan to

Re: [HACKERS] If subscription to foreign table valid ?

2017-05-12 Thread Petr Jelinek
On 12/05/17 15:55, Neha Khatri wrote: > On 5/11/17, Petr Jelinek wrote: >> Hi, >> >> On 11/05/17 14:25, tushar wrote: >>> Hi, >>> >>> I observed that -we cannot publish "foreign table" in Publication >>> >>> but same thi

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-12 Thread Petr Jelinek
On 12/05/17 15:02, Peter Eisentraut wrote: > On 5/9/17 11:43, Petr Jelinek wrote: >> Here is rebased version of the other patch (the interface rework). I >> also fixed the tab completion there. > > Committed. > > One small thing I changed, to make it look more like CR

Re: [HACKERS] Time based lag tracking for logical replication

2017-05-12 Thread Petr Jelinek
sub1 CONNECTION 'dbname=postgres host=localhost > user=neha port=5432' PUBLICATION mypub; > NOTICE: synchronized table states > ERROR: could not create replication slot "sub1": ERROR: could not > load library "/home/neha/postgres/PGCurrentInstall/lib/pgoutput.so": &

Re: [HACKERS] If subscription to foreign table valid ?

2017-05-12 Thread Petr Jelinek
On 11/05/17 15:43, Petr Jelinek wrote: > Hi, > > On 11/05/17 14:25, tushar wrote: >> Hi, >> >> I observed that -we cannot publish "foreign table" in Publication >> >> postgres=# create foreign table t (n int) server db1_server options >> (t

Re: [HACKERS] snapbuild woes

2017-05-12 Thread Petr Jelinek
for session 3 to finish slot creation, takes about 20 mins on my laptop without patches, minute and half with your patches for 0004 and 0005 (or with your 0004 and my 0005) and about 2s with my original 0004 and 0005. What makes it slow is the constant qsorting IIRC. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] snapbuild woes

2017-05-11 Thread Petr Jelinek
On 11/05/17 16:33, Stas Kelvich wrote: > >> On 10 May 2017, at 11:43, Petr Jelinek wrote: >> >> On 09/05/17 22:11, Erik Rijkers wrote: >>> On 2017-05-09 21:00, Petr Jelinek wrote: >>>> On 09/05/17 19:54, Erik Rijkers wrote: >>>>> On 2017

Re: [HACKERS] If subscription to foreign table valid ?

2017-05-11 Thread Petr Jelinek
7; was foreign table during ALTER SUBSCRIPTION does not mean that it won't be something else half hour later. I think it does make sense to add check for this into CREATE/ALTER SUBSCRIBER though so that user is informed immediately about the mistake rather than by errors in the logs later

Re: [HACKERS] Time based lag tracking for logical replication

2017-05-11 Thread Petr Jelinek
On 11/05/17 15:01, Simon Riggs wrote: > On 11 May 2017 at 08:32, Noah Misch wrote: >> On Sun, Apr 23, 2017 at 01:10:32AM +0200, Petr Jelinek wrote: >>> The time based lag tracking commit [1] added interface for logging >>> progress of replication so that we can

Re: [HACKERS] Re: logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-11 Thread Petr Jelinek
y send > a status update within 24 hours, and include a date for your subsequent status > update. Refer to the policy on open item ownership: > https://www.postgresql.org/message-id/20170404140717.GA2675809%40tornado.leadboat.com > The patch Peter mentioned was committed. There

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-05-11 Thread Petr Jelinek
s for all the other workers for same subscription and shuts them down as well sounds like solution to this. [1] https://www.postgresql.org/message-id/cahgqgwhpi8ky-yanffe0sgmhktsycqltnkx07bw9s7-rn17...@mail.gmail.com -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Develo

Re: [HACKERS] Should pg_current_wal_location() become pg_current_wal_lsn()

2017-05-10 Thread Petr Jelinek
l, anybody else want to vote? > I am for standardizing (although I don't have preference of location vs lsn). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@pos

Re: [HACKERS] Function to move the position of a replication slot

2017-05-10 Thread Petr Jelinek
orrectly for logical slots so it should throw error on those at the moment (at least until we write something that works, but that's far more complex than this patch). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training &a

Re: [HACKERS] 'nocopy data' option is set in SUBSCRIPTION but still data is getting migrated

2017-05-10 Thread Petr Jelinek
copy data option is set ? > Yes, (no)copy data only affects existing data at the time of running the command, any additional data are always replicated. The "alter subscription sub refresh publication" does nothing unless you added/removed tables to/from publication. -- Pe

Re: [HACKERS] [BUGS] Concurrent ALTER SEQUENCE RESTART Regression

2017-05-10 Thread Petr Jelinek
) if we do this, did you try to run any benchmark on this? Looking at the changes to open_share_lock(), I wonder if we need to have lock level as parameter so that lastval() is not blocked. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training &

Re: [HACKERS] snapbuild woes

2017-05-10 Thread Petr Jelinek
On 09/05/17 22:11, Erik Rijkers wrote: > On 2017-05-09 21:00, Petr Jelinek wrote: >> On 09/05/17 19:54, Erik Rijkers wrote: >>> On 2017-05-09 11:50, Petr Jelinek wrote: >>> >> >> Ah okay, so this is same issue that's reported by both Masahiko Sawada &

Re: [HACKERS] snapbuild woes

2017-05-09 Thread Petr Jelinek
On 09/05/17 19:54, Erik Rijkers wrote: > On 2017-05-09 11:50, Petr Jelinek wrote: > >> I rebased the above mentioned patch to apply to the patches Andres sent, >> if you could try to add it on top of what you have and check if it still >> fails, that would be help

Re: [HACKERS] logical replication deranged sender

2017-05-09 Thread Petr Jelinek
On 09/05/17 19:13, Jeff Janes wrote: > On Tue, May 9, 2017 at 9:18 AM, Petr Jelinek > mailto:petr.jeli...@2ndquadrant.com>> wrote: > > On 08/05/17 13:47, Petr Jelinek wrote: > > On 08/05/17 01:17, Jeff Janes wrote: > >> After dropping a subscription,

Re: [HACKERS] logical replication deranged sender

2017-05-09 Thread Petr Jelinek
On 08/05/17 13:47, Petr Jelinek wrote: > On 08/05/17 01:17, Jeff Janes wrote: >> After dropping a subscription, it says it succeeded and that it dropped >> the slot on the publisher. >> >> But the publisher still has the slot, and a full-tilt process described >&

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-09 Thread Petr Jelinek
On 09/05/17 16:28, Peter Eisentraut wrote: > On 5/9/17 04:39, Petr Jelinek wrote: >>>> What we want to simulate instead is an "auto" dependency of the slot on >>>> the subscription. So you can drop the slot separately (subject to other >>>> restri

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-09 Thread Petr Jelinek
On 09/05/17 11:44, Masahiko Sawada wrote: > On Tue, May 9, 2017 at 5:57 PM, Petr Jelinek > wrote: >> On 09/05/17 10:51, Masahiko Sawada wrote: >>> On Tue, May 9, 2017 at 5:39 PM, Petr Jelinek >>> wrote: >>>> On 09/05/17 07:07, Peter Eisentraut wrote: >

Re: [HACKERS] snapbuild woes

2017-05-09 Thread Petr Jelinek
On 09/05/17 10:59, Erik Rijkers wrote: > On 2017-05-09 10:50, Petr Jelinek wrote: >> On 09/05/17 00:03, Erik Rijkers wrote: >>> On 2017-05-05 02:00, Andres Freund wrote: >>>> >>>> Could you have a look? >>> >>> Running tests with

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-09 Thread Petr Jelinek
On 09/05/17 10:51, Masahiko Sawada wrote: > On Tue, May 9, 2017 at 5:39 PM, Petr Jelinek > wrote: >> On 09/05/17 07:07, Peter Eisentraut wrote: >>> On 5/8/17 23:23, Peter Eisentraut wrote: >>>> The way this uses RESTRICT and CASCADE appears to be backwards from

Re: [HACKERS] SUBSCRIPTIONS and pg_upgrade

2017-05-09 Thread Petr Jelinek
the policy on open item ownership: >>> https://www.postgresql.org/message-id/20170404140717.GA2675809%40tornado.leadboat.com >> >> Fair enough. I have set myself a reminder to report back on May 30. > > I think that it would be nice to fix that even before beta, so >

Re: [HACKERS] snapbuild woes

2017-05-09 Thread Petr Jelinek
issue which is not yet fixed (and the original patch will not apply on top of this change). Alternatively there is a bug in this patch. Did you see high CPU usage during the test when there were those "startup" state walsenders? -- Petr Jelinek http://www.2ndQuadran

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-09 Thread Petr Jelinek
first whether the slot exists. If the >> subscription is associated with the slot, it should be there. > > Here is your patch amended for that. > I am fine with this mechanism as well. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-08 Thread Petr Jelinek
On 08/05/17 22:55, Peter Eisentraut wrote: > On 5/5/17 13:01, Petr Jelinek wrote: >> What do you think of attached. I actually did add RESTRICT/CASCADE, in a >> way that if there is slot RESTRICT will refuse to drop subscription and >> CASCADE will try to drop it. Still all

Re: [HACKERS] Logical replication - TRAP: FailedAssertion in pgstat.c

2017-05-08 Thread Petr Jelinek
On 08/05/17 18:12, Peter Eisentraut wrote: > On 5/5/17 07:13, Petr Jelinek wrote: >> Yes that's different thing that we've been discussing a bit in snapbuild >> woes thread. >> >>> the 'TRAP: FailedAssertion in pgstat.c' anymore. >>>

Re: [HACKERS] Logical replication - TRAP: FailedAssertion in pgstat.c

2017-05-08 Thread Petr Jelinek
On 08/05/17 17:52, Masahiko Sawada wrote: > On Fri, May 5, 2017 at 8:13 PM, Petr Jelinek > wrote: >> On 03/05/17 13:23, Erik Rijkers wrote: >>> On 2017-05-03 08:17, Petr Jelinek wrote: >>>> On 02/05/17 20:43, Robert Haas wrote: >>>>> O

Re: [HACKERS] logical replication deranged sender

2017-05-08 Thread Petr Jelinek
0.00 s, elapsed: 1.00 s > statement: drop subscription foobar ; > Yeah that's normal because DropSubscription holds exclusive lock on pg_subscription. I am guessing 21270 is the logical replication launcher? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Developmen

Re: [HACKERS] Get stuck when dropping a subscription during synchronizing table

2017-05-08 Thread Petr Jelinek
uld stop itself if the subscription got removed, but of course again the dependencies are an issue, so we should probably kill those explicitly as well. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent vi

Re: [HACKERS] Draft release notes for next week's back-branch releases

2017-05-08 Thread Petr Jelinek
On 06/05/17 19:38, Tom Lane wrote: > Petr Jelinek writes: >> On 06/05/17 19:15, Tom Lane wrote: >>> (Or, wait a minute. That documentation only applies to v10, but we >>> need to be writing this relnote for 9.6 users. What terminology should >>> we be

Re: [HACKERS] Draft release notes for next week's back-branch releases

2017-05-06 Thread Petr Jelinek
On 06/05/17 19:15, Tom Lane wrote: > Petr Jelinek writes: >> On 06/05/17 18:16, Tom Lane wrote: >>> Hmm, I'm hoping for something more user-oriented. Is the corruption >>> time-limited? What's an "exported snapshot" anyway, is it the same >>&

Re: [HACKERS] Draft release notes for next week's back-branch releases

2017-05-06 Thread Petr Jelinek
On 06/05/17 18:16, Tom Lane wrote: > Petr Jelinek writes: >> On 06/05/17 17:25, Tom Lane wrote: >>> OK, can you suggest better wording? > >> Something like the attached (it requires some polishing of English >> probably). > > Hmm, I'm hoping for some

Re: [HACKERS] Draft release notes for next week's back-branch releases

2017-05-06 Thread Petr Jelinek
On 06/05/17 17:25, Tom Lane wrote: > Petr Jelinek writes: >> Hmm, the 2bef06d51 and 56e19d938 are fixes for different bugs, we can >> keep them together since result of both is corrupted snapshot, but the >> description can't just mangle pieces of text from different com

Re: [HACKERS] Draft release notes for next week's back-branch releases

2017-05-06 Thread Petr Jelinek
about using ondisk snapshot (which only contain transactions changing catalogs) as full snapshots, it's more about creating multiple logical replication slots while concurrent writes happen. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Tr

Re: [HACKERS] Not getting error if ALTER SUBSCRIPTION syntax is wrong.

2017-05-06 Thread Petr Jelinek
On 05/05/17 19:51, Petr Jelinek wrote: > On 05/05/17 14:40, tushar wrote: >> Hi, >> >> While testing 'logical replication' against v10 , i encountered one >> issue where data stop migrating after ALTER PUBLICATION. >> >> X Server >> \\ Make s

Re: [HACKERS] Not getting error if ALTER SUBSCRIPTION syntax is wrong.

2017-05-05 Thread Petr Jelinek
erver > postgres=# select * from test; > n > --- > 1 > (1 row) > > I think probably syntax of alter subscription is not correct but > surprisingly it is not throwing an error. > Syntax of ALTER command is correct, syntax of the connection string is not, you are probab

Re: [HACKERS] snapbuild woes

2017-05-05 Thread Petr Jelinek
On 05/05/17 18:18, Andres Freund wrote: > On 2017-05-05 13:53:16 +0200, Petr Jelinek wrote: >> On 05/05/17 02:42, Andres Freund wrote: >>> On 2017-05-04 17:00:04 -0700, Andres Freund wrote: >>>> Attached is a prototype patch for that. >>> >> >

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-05 Thread Petr Jelinek
On 02/05/17 15:31, Tom Lane wrote: > Petr Jelinek writes: >> Let me expand, if we don't drop the slot by default when dropping >> subscription, we'll have a lot of users with dead slots laying around >> holding back WAL/catalog_xmin, that's really bad. If w

Re: [HACKERS] compiler warning with VS 2017

2017-05-05 Thread Petr Jelinek
On 05/05/17 16:56, Tom Lane wrote: > Petr Jelinek writes: >> On 05/05/17 06:50, Tom Lane wrote: >>> Actually, looking around a bit there, it's not even clear why >>> we should be booby-trapping the value of an unchanged column in >>> the first place. So

Re: [HACKERS] [patch] Build pgoutput with MSVC

2017-05-05 Thread Petr Jelinek
s in middle of line patch: Only garbage was found > in the patch input.". But I can just copy/paste it manually :) > Thanks for fixing this, I admit I have no idea how our windows build system works. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PG 10 release notes

2017-05-05 Thread Petr Jelinek
happens in core, the decoding just provides C API for plugins to consume the decoded info (ie, heap tuples and similar). The test_decoding *tests* the decoding API and the external projects use the decoding API as well but they don't really decode anything, their role is of filtering and deciding

Re: [HACKERS] PG 10 release notes

2017-05-05 Thread Petr Jelinek
json plugin > into care soon-ish. While there was some initial work towards that, it > unfortunately stalled at some point. > > Euler, are you perchance interested in working towards that? ;) > I am happy to review any work in this area BTW. -- Petr Jelinek h

Re: [HACKERS] compiler warning with VS 2017

2017-05-05 Thread Petr Jelinek
t is inadequate too. Hmm, as far as I can recollect this is just leftover debugging code that was intended to help ensure that we are checking the "changed" everywhere we are supposed to (since I changed handling of these structured quite a bit during development). Should be changed to N

Re: [HACKERS] snapbuild woes

2017-05-05 Thread Petr Jelinek
you want to store it in the ondisk snapshot as well? About better name for it what about something like oldest_full_xact? Otherwise the logic seems to be right on first read, will ponder it a bit more -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7

Re: [HACKERS] Logical replication - TRAP: FailedAssertion in pgstat.c

2017-05-05 Thread Petr Jelinek
On 03/05/17 13:23, Erik Rijkers wrote: > On 2017-05-03 08:17, Petr Jelinek wrote: >> On 02/05/17 20:43, Robert Haas wrote: >>> On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut > >>>> code path that calls CommitTransactionCommand() should have one, no? >

Re: [HACKERS] snapbuild woes

2017-05-05 Thread Petr Jelinek
On 05/05/17 02:00, Andres Freund wrote: > Hi, > > On 2017-05-02 08:55:53 +0200, Petr Jelinek wrote: >> Aah, now I understand we talked about slightly different things, I >> considered the running thing to be first step towards tracking aborted >> txes everywhere.

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-04 Thread Petr Jelinek
On 04/05/17 23:29, Tom Lane wrote: > Robert Haas writes: >> On Wed, May 3, 2017 at 12:38 AM, Petr Jelinek >> wrote: >>> Ok, Let me be clear, I actually happen to agree with your proposal. The >>> reason I am moaning is that I always seem to find myself doi

Re: [HACKERS] CTE inlining

2017-05-03 Thread Petr Jelinek
n easily figure where to add the >> MATERIALIZED option and regain the original performance. > > +1 for option 1. This change will be welcome for a large number of > queries, but forced materialization is a real need and I use it often. > This comes off as a very reasonable comp

Re: [HACKERS] snapbuild woes

2017-05-03 Thread Petr Jelinek
t's a tricky bug, but I'm worried this isn't on track to end. > What is missing to make it end? It's tricky 5 bugs, and they are quite sensitive on rare timing/concurrency events. First two are fixed, we can live with 5th being done later (as it's not correctness fix, but

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-03 Thread Petr Jelinek
On 02/05/17 16:14, Petr Jelinek wrote: > On 02/05/17 15:31, Tom Lane wrote: >> Petr Jelinek writes: >>> Let me expand, if we don't drop the slot by default when dropping >>> subscription, we'll have a lot of users with dead slots laying around >>> ho

Re: [HACKERS] Time based lag tracking for logical replication

2017-05-02 Thread Petr Jelinek
On 03/05/17 08:28, Simon Riggs wrote: > On 23 April 2017 at 01:10, Petr Jelinek wrote: >> Hi, >> >> The time based lag tracking commit [1] added interface for logging >> progress of replication so that we can report lag as time interval >> instead of just bytes. B

Re: [HACKERS] Logical replication - TRAP: FailedAssertion in pgstat.c

2017-05-02 Thread Petr Jelinek
On 02/05/17 20:43, Robert Haas wrote: > On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut > wrote: >> On 4/16/17 16:11, Petr Jelinek wrote: >>> Yeah it is, it needs to be fenced to happen only after commit, which is >>> not guaranteed at the point of cod

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 18:25, Alvaro Herrera wrote: > Petr Jelinek wrote: >> On 02/05/17 18:00, Robert Haas wrote: >>> On Tue, May 2, 2017 at 11:49 AM, Alvaro Herrera >>> wrote: >>>> Petr Jelinek wrote: >>>>> So the only way to fulfill the requiremen

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 22:40, Robert Haas wrote: > On Tue, May 2, 2017 at 3:02 PM, Petr Jelinek > wrote: >> That sounds okay. I know PeterE didn't like the lower case and >> underscore separated words for options in the original patch, so I'd >> like to hear his opini

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 19:42, Robert Haas wrote: > On Tue, May 2, 2017 at 8:45 AM, Petr Jelinek > wrote: >> I am happy to implement something different, it's quite trivial to >> change. But I am not going to propose anything different as I can't >> think of better syntax (i

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 18:00, Robert Haas wrote: > On Tue, May 2, 2017 at 11:49 AM, Alvaro Herrera > wrote: >> Petr Jelinek wrote: >>> So the only way to fulfill the requirement you stated is to just not try >>> to drop the slot, ever, on DROP SUBSCRIPTION. That makes

Re: [HACKERS] Logical replication in the same cluster

2017-05-02 Thread Petr Jelinek
On 02/05/17 16:37, Andres Freund wrote: > On 2017-05-02 09:17:27 +0200, Petr Jelinek wrote: >> Yes because otherwise we risk leaving slot on the upstream if the >> command fails downstream. > > Shouldn't temporary slots be able to solve that concern? Create it as &g

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 15:31, Tom Lane wrote: > Petr Jelinek writes: >> Let me expand, if we don't drop the slot by default when dropping >> subscription, we'll have a lot of users with dead slots laying around >> holding back WAL/catalog_xmin, that's really bad. If w

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 15:14, Petr Jelinek wrote: > On 02/05/17 15:10, Tom Lane wrote: >> Robert Haas writes: >>>> On Thu, Apr 20, 2017 at 7:46 AM, Petr Jelinek >>>> wrote: >>>>> DROP SUBSCRIPTION mysub NODROP SLOT; >> >>>> I'm prett

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 15:10, Tom Lane wrote: > Robert Haas writes: >>> On Thu, Apr 20, 2017 at 7:46 AM, Petr Jelinek >>> wrote: >>>> DROP SUBSCRIPTION mysub NODROP SLOT; > >>> I'm pretty uninspired by this choice of syntax. > > Actually, this c

Re: [HACKERS] logical replication syntax (was DROP SUBSCRIPTION, query cancellations and slot handling)

2017-05-02 Thread Petr Jelinek
On 02/05/17 14:13, Tom Lane wrote: > Robert Haas writes: >> On Thu, Apr 20, 2017 at 7:46 AM, Petr Jelinek >> wrote: >>> DROP SUBSCRIPTION mysub NODROP SLOT; > >> I'm pretty uninspired by this choice of syntax. Logical replication >> seems to have

Re: [HACKERS] Logical replication in the same cluster

2017-05-02 Thread Petr Jelinek
On 02/05/17 04:14, Peter Eisentraut wrote: > On 4/30/17 20:31, Andres Freund wrote: >> On 2017-04-26 23:41:51 +0200, Petr Jelinek wrote: >>> Yes that's result of how logical replication slots work, the transaction >>> that needs to finish is your transac

Re: [HACKERS] logical replication and PANIC during shutdown checkpoint in publisher

2017-05-02 Thread Petr Jelinek
ATE_REPLICATION_SLOT PHYSICAL will not WAL log and CREATE_REPLICATION_SLOT LOGICAL can't be run without being connected to db, neither can logical decoding and SQL queries, so that coves all. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support,

Re: [HACKERS] snapbuild woes

2017-05-01 Thread Petr Jelinek
On 01/05/17 21:14, Andres Freund wrote: > On 2017-05-01 11:09:44 +0200, Petr Jelinek wrote: >> On 01/05/17 10:03, Andres Freund wrote: >>> On 2017-05-01 03:54:49 +0200, Petr Jelinek wrote: > >>>> But, I still think we need to restart the tracking after new >&

Re: [HACKERS] snapbuild woes

2017-05-01 Thread Petr Jelinek
On 01/05/17 10:03, Andres Freund wrote: > On 2017-05-01 03:54:49 +0200, Petr Jelinek wrote: >> I agree with adding running, I think that's good thing even for the per >> transaction tracking and snapshot exports - we could use the newly added >> field to get rid of the is

Re: [HACKERS] snapbuild woes

2017-04-30 Thread Petr Jelinek
On 01/05/17 04:29, Craig Ringer wrote: > On 1 May 2017 at 09:54, Petr Jelinek wrote: > >> But, I still think we need to restart the tracking after new >> xl_running_xacts. Reason for that is afaics any of the catalog snapshots >> that we assigned to transactions at the e

Re: [HACKERS] snapbuild woes

2017-04-30 Thread Petr Jelinek
re we knew one of the supposedly running txes was actually already committed and that transaction might have done catalog changes. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hacke

Re: [HACKERS] Time based lag tracking for logical replication

2017-04-29 Thread Petr Jelinek
On 23/04/17 01:10, Petr Jelinek wrote: > Hi, > > The time based lag tracking commit [1] added interface for logging > progress of replication so that we can report lag as time interval > instead of just bytes. But the patch didn't contain patch for the > builtin logical

Re: [HACKERS] Interval for launching the table sync worker

2017-04-27 Thread Petr Jelinek
On 27/04/17 21:00, Peter Eisentraut wrote: > On 4/27/17 06:47, Petr Jelinek wrote: >> One thing I am missing in your patch however is cleanup of entries for >> relations that finished sync. I wonder if it would be enough to just >> destroy the hash when we get to empty lis

Re: [HACKERS] Interval for launching the table sync worker

2017-04-27 Thread Petr Jelinek
c. I wonder if it would be enough to just destroy the hash when we get to empty list. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chan

Re: [HACKERS] some review comments on logical rep code

2017-04-27 Thread Petr Jelinek
On 26/04/17 18:36, Fujii Masao wrote: > On Thu, Apr 27, 2017 at 1:28 AM, Fujii Masao wrote: >> On Wed, Apr 26, 2017 at 3:47 PM, Kyotaro HORIGUCHI >> wrote: >>> At Wed, 26 Apr 2017 14:31:12 +0900, Masahiko Sawada >>> wrote in >>> >>>> On W

Re: [HACKERS] Logical replication in the same cluster

2017-04-27 Thread Petr Jelinek
via SQL and I'd rather not hide even more errors in the log. Note that the workaround for all of this is not all that complex, you do same thing (create slot manually) you'd do for physical replication with slots. Thoughts? -- Petr Jelinek http://www.2ndQuadrant.c

Re: [HACKERS] Logical replication in the same cluster

2017-04-26 Thread Petr Jelinek
o finish is your transaction. It can be worked around by creating the slot manually via the SQL interface for example and create the subscription using WITH (NOCREATE SLOT, SLOT NAME = 'your slot') . -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development

Re: [HACKERS] some review comments on logical rep code

2017-04-25 Thread Petr Jelinek
>> not false wakeups, but 'possible delay of launching new workers >> by 3 minutes at most' (this is centainly a kind of false wakeups, >> though). We can live with this failure when using two-paase >> commmit, but I think it shouldn't happen silently. >> >> >> How about providing AtPrepare_ApplyLauncher(void) like the >> follows and calling it in PrepareTransaction? > > Or we should apply the attached patch and handle the 2PC case properly? > I was thinking that it's overkill more than necessary, but that seems not true > as far as I implement that. > Looks like it does not even increase size of the 2pc file, +1 for this. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Separation walsender & normal backends

2017-04-25 Thread Petr Jelinek
efulness if much lower without the initial copy. The original patch for this added new commands to replication protocol, adding generic SQL interface was result of request in the reviews. I personally don't mind moving back my original idea of special commands if that was the consensus, but p

Re: [HACKERS] tablesync patch broke the assumption that logical rep depends on?

2017-04-25 Thread Petr Jelinek
On 22/04/17 22:09, Petr Jelinek wrote: > On 21/04/17 16:31, Petr Jelinek wrote: >> On 21/04/17 16:23, Peter Eisentraut wrote: >>> On 4/21/17 10:11, Petr Jelinek wrote: >>>> On 21/04/17 16:09, Peter Eisentraut wrote: >>>>> On 4/20/17 14:29, Petr Jelinek

Re: [HACKERS] PG 10 release notes

2017-04-25 Thread Petr Jelinek
g was really more than a proof-of-concept > until now. > Huh? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your sub

Re: [HACKERS] PG 10 release notes

2017-04-24 Thread Petr Jelinek
n=database kind of connection). Or the ability of logical decoding to follow timeline switches. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

<    1   2   3   4   5   6   7   8   9   10   >