Re: [COMMITTERS] pgsql: Introduce replication progress tracking infrastructure.
On 2015-04-30 13:56:08 +0900, Michael Paquier wrote: On Thu, Apr 30, 2015 at 2:37 AM, Andres Freund and...@anarazel.de wrote: Introduce replication progress tracking infrastructure. [...] Some comments about the docs: 1) the the: + entry +Create a replication origin with the the passed in external +name, and create an internal id for it. + /entry + /row Fixed. 2) Missing markup type for oid. + entry +Lookup replication origin by name and return the internal +oid. If no corresponding replication origin is found a error +is thrown. + /entry + /row I'm not really seeing a problem in that. To me it doesn't seem helpful to refer to types when it's not about the type. Anyway, I added it here and a couple other places nonetheless, I don't really care that much. The excerpt highlighted a 'a' vs 'an' error, found a good many more :(. Should probably have either earlier or never learned about the actual rules behind this stuff. 3) Perhaps Check that a replication has been configured in the current session instead of using a question? + entry +Has a replication origin been configured in the current session? + /entry Hm, reads fine to me. 4) Missing markup type for oid? + Replication origins consist out of a name and a oid. The name, which Same as 2). 5) will persist or will be persistent, not will be persist I guess. + If that's done replication progress will be persist in a crash safe I guess I rewrote that sentence once too often... 6) The use of that, that looks weird to me. There should be only one. + system to one other, another problem can be that, that it is hard to avoid Yes, that sounds wrong. I'll just remove the comma and one 'that'. Pushed the fixups. Thanks for the check! Andres Freund -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Introduce replication progress tracking infrastructure.
On 04/29/2015 01:37 PM, Andres Freund wrote: Introduce replication progress tracking infrastructure. This appears to have broken some builds (see for example http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=bowerbirddt=2015-04-29%2019%3A31%3A06) by using UINT16_MAX instead of PG_UINT16_MAX. cheers andrew -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
[COMMITTERS] pgsql: Introduce replication progress tracking infrastructure.
Introduce replication progress tracking infrastructure. When implementing a replication solution ontop of logical decoding, two related problems exist: * How to safely keep track of replication progress * How to change replication behavior, based on the origin of a row; e.g. to avoid loops in bi-directional replication setups The solution to these problems, as implemented here, consist out of three parts: 1) 'replication origins', which identify nodes in a replication setup. 2) 'replication progress tracking', which remembers, for each replication origin, how far replay has progressed in a efficient and crash safe manner. 3) The ability to filter out changes performed on the behest of a replication origin during logical decoding; this allows complex replication topologies. E.g. by filtering all replayed changes out. Most of this could also be implemented in userspace, e.g. by inserting additional rows contain origin information, but that ends up being much less efficient and more complicated. We don't want to require various replication solutions to reimplement logic for this independently. The infrastructure is intended to be generic enough to be reusable. This infrastructure also replaces the 'nodeid' infrastructure of commit timestamps. It is intended to provide all the former capabilities, except that there's only 2^16 different origins; but now they integrate with logical decoding. Additionally more functionality is accessible via SQL. Since the commit timestamp infrastructure has also been introduced in 9.5 (commit 73c986add) changing the API is not a problem. For now the number of origins for which the replication progress can be tracked simultaneously is determined by the max_replication_slots GUC. That GUC is not a perfect match to configure this, but there doesn't seem to be sufficient reason to introduce a separate new one. Bumps both catversion and wal page magic. Author: Andres Freund, with contributions from Petr Jelinek and Craig Ringer Reviewed-By: Heikki Linnakangas, Petr Jelinek, Robert Haas, Steve Singer Discussion: 20150216002155.gi15...@awork2.anarazel.de, 20140923182422.ga15...@alap3.anarazel.de, 20131114172632.ge7...@alap2.anarazel.de Branch -- master Details --- http://git.postgresql.org/pg/commitdiff/5aa2350426c4fdb3d04568b65aadac397012bbcb Modified Files -- contrib/test_decoding/Makefile |3 +- contrib/test_decoding/expected/replorigin.out | 141 +++ contrib/test_decoding/sql/replorigin.sql| 64 + contrib/test_decoding/test_decoding.c | 28 + doc/src/sgml/catalogs.sgml | 123 ++ doc/src/sgml/filelist.sgml |1 + doc/src/sgml/func.sgml | 201 ++- doc/src/sgml/logicaldecoding.sgml | 35 +- doc/src/sgml/postgres.sgml |1 + doc/src/sgml/replication-origins.sgml | 93 ++ src/backend/access/heap/heapam.c| 19 + src/backend/access/rmgrdesc/Makefile|4 +- src/backend/access/rmgrdesc/replorigindesc.c| 61 + src/backend/access/rmgrdesc/xactdesc.c | 24 +- src/backend/access/transam/commit_ts.c | 53 +- src/backend/access/transam/rmgr.c |1 + src/backend/access/transam/xact.c | 76 +- src/backend/access/transam/xlog.c |8 + src/backend/access/transam/xloginsert.c | 27 +- src/backend/access/transam/xlogreader.c |6 + src/backend/catalog/Makefile|2 +- src/backend/catalog/catalog.c |8 +- src/backend/catalog/system_views.sql|7 + src/backend/replication/logical/Makefile|3 +- src/backend/replication/logical/decode.c| 49 +- src/backend/replication/logical/logical.c | 29 + src/backend/replication/logical/origin.c| 1485 +++ src/backend/replication/logical/reorderbuffer.c |5 +- src/backend/storage/ipc/ipci.c |3 + src/backend/utils/cache/syscache.c | 23 + src/bin/pg_resetxlog/pg_resetxlog.c |3 + src/include/access/commit_ts.h | 14 +- src/include/access/rmgrlist.h |1 + src/include/access/xact.h | 11 + src/include/access/xlog.h |1 + src/include/access/xlog_internal.h |2 +- src/include/access/xlogdefs.h |6 + src/include/access/xloginsert.h |1 + src/include/access/xlogreader.h |3 + src/include/access/xlogrecord.h |1 + src/include/catalog/catversion.h|2 +- src/include/catalog/indexing.h |6 + src/include/catalog/pg_proc.h | 36 + src/include/catalog/pg_replication_origin.h | 70 ++ src/include/replication/logical.h |2 +
Re: [COMMITTERS] pgsql: Introduce replication progress tracking infrastructure.
On 2015-04-29 15:49:10 -0400, Andrew Dunstan wrote: On 04/29/2015 01:37 PM, Andres Freund wrote: Introduce replication progress tracking infrastructure. This appears to have broken some builds (see for example http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=bowerbirddt=2015-04-29%2019%3A31%3A06) by using UINT16_MAX instead of PG_UINT16_MAX. Thanks for the notice, fixed. I unfortunately attributed this to Andrew Gierth in the commit message. I somehow managed to merge your contacts in one on my mobile phone... Greetings, Andres Freund -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Introduce replication progress tracking infrastructure.
On Thu, Apr 30, 2015 at 2:37 AM, Andres Freund and...@anarazel.de wrote: Introduce replication progress tracking infrastructure. [...] Some comments about the docs: 1) the the: + entry +Create a replication origin with the the passed in external +name, and create an internal id for it. + /entry + /row 2) Missing markup type for oid. + entry +Lookup replication origin by name and return the internal +oid. If no corresponding replication origin is found a error +is thrown. + /entry + /row 3) Perhaps Check that a replication has been configured in the current session instead of using a question? + entry +Has a replication origin been configured in the current session? + /entry 4) Missing markup type for oid? + Replication origins consist out of a name and a oid. The name, which 5) will persist or will be persistent, not will be persist I guess. + If that's done replication progress will be persist in a crash safe 6) The use of that, that looks weird to me. There should be only one. + system to one other, another problem can be that, that it is hard to avoid Regards, -- Michael -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
