[COMMITTERS] pgsql: Avoid pin scan for replay of XLOG_BTREE_VACUUM in all cases
Avoid pin scan for replay of XLOG_BTREE_VACUUM in all cases Replay of XLOG_BTREE_VACUUM during Hot Standby was previously thought to require complex interlocking that matched the requirements on the master. This required an O(N) operation that became a significant problem with large indexes, causing replication delays of seconds or in some cases minutes while the XLOG_BTREE_VACUUM was replayed. This commit skips the “pin scan” that was previously required, by observing in detail when and how it is safe to do so, with full documentation. The pin scan is skipped only in replay; the VACUUM code path on master is not touched here. No tests included. Manual tests using an additional patch to view WAL records and their timing have shown the change in WAL records and their handling has successfully reduced replication delay. This is a back-patch of commits 687f2cd7a015, 3e4b7d87988f, b60284261375 by Simon Riggs, to branches 9.4 and 9.5. No further backpatch is possible because this depends on catalog scans being MVCC. I (Álvaro) additionally updated a slight problem in the README, which explains why this touches the 9.6 and master branches. Branch -- REL9_6_STABLE Details --- http://git.postgresql.org/pg/commitdiff/f5d89443203480e39a6a15e64f1950c3b4d3c9a2 Modified Files -- src/backend/access/nbtree/README | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) -- Sent via pgsql-committers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
[COMMITTERS] pgsql: Avoid pin scan for replay of XLOG_BTREE_VACUUM in all cases
Avoid pin scan for replay of XLOG_BTREE_VACUUM in all cases Replay of XLOG_BTREE_VACUUM during Hot Standby was previously thought to require complex interlocking that matched the requirements on the master. This required an O(N) operation that became a significant problem with large indexes, causing replication delays of seconds or in some cases minutes while the XLOG_BTREE_VACUUM was replayed. This commit skips the “pin scan” that was previously required, by observing in detail when and how it is safe to do so, with full documentation. The pin scan is skipped only in replay; the VACUUM code path on master is not touched here. No tests included. Manual tests using an additional patch to view WAL records and their timing have shown the change in WAL records and their handling has successfully reduced replication delay. This is a back-patch of commits 687f2cd7a015, 3e4b7d87988f, b60284261375 by Simon Riggs, to branches 9.4 and 9.5. No further backpatch is possible because this depends on catalog scans being MVCC. I (Álvaro) additionally updated a slight problem in the README, which explains why this touches the 9.6 and master branches. Branch -- master Details --- http://git.postgresql.org/pg/commitdiff/f65b94f63962e9f7e144a469bc1750286ddaee27 Modified Files -- src/backend/access/nbtree/README | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) -- Sent via pgsql-committers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
[COMMITTERS] pgsql: Avoid pin scan for replay of XLOG_BTREE_VACUUM in all cases
Avoid pin scan for replay of XLOG_BTREE_VACUUM in all cases Replay of XLOG_BTREE_VACUUM during Hot Standby was previously thought to require complex interlocking that matched the requirements on the master. This required an O(N) operation that became a significant problem with large indexes, causing replication delays of seconds or in some cases minutes while the XLOG_BTREE_VACUUM was replayed. This commit skips the “pin scan” that was previously required, by observing in detail when and how it is safe to do so, with full documentation. The pin scan is skipped only in replay; the VACUUM code path on master is not touched here. No tests included. Manual tests using an additional patch to view WAL records and their timing have shown the change in WAL records and their handling has successfully reduced replication delay. This is a back-patch of commits 687f2cd7a015, 3e4b7d87988f, b60284261375 by Simon Riggs, to branches 9.4 and 9.5. No further backpatch is possible because this depends on catalog scans being MVCC. I (Álvaro) additionally updated a slight problem in the README, which explains why this touches the 9.6 and master branches. Branch -- REL9_5_STABLE Details --- http://git.postgresql.org/pg/commitdiff/c0db1ec2600a898ac75d14057e01fb716059a2f5 Modified Files -- src/backend/access/nbtree/README| 22 ++ src/backend/access/nbtree/nbtree.c | 15 +++ src/backend/access/nbtree/nbtxlog.c | 25 ++--- src/include/access/nbtree.h | 6 -- 4 files changed, 59 insertions(+), 9 deletions(-) -- Sent via pgsql-committers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
[COMMITTERS] pgsql: Avoid pin scan for replay of XLOG_BTREE_VACUUM in all cases
Avoid pin scan for replay of XLOG_BTREE_VACUUM in all cases Replay of XLOG_BTREE_VACUUM during Hot Standby was previously thought to require complex interlocking that matched the requirements on the master. This required an O(N) operation that became a significant problem with large indexes, causing replication delays of seconds or in some cases minutes while the XLOG_BTREE_VACUUM was replayed. This commit skips the “pin scan” that was previously required, by observing in detail when and how it is safe to do so, with full documentation. The pin scan is skipped only in replay; the VACUUM code path on master is not touched here. No tests included. Manual tests using an additional patch to view WAL records and their timing have shown the change in WAL records and their handling has successfully reduced replication delay. This is a back-patch of commits 687f2cd7a015, 3e4b7d87988f, b60284261375 by Simon Riggs, to branches 9.4 and 9.5. No further backpatch is possible because this depends on catalog scans being MVCC. I (Álvaro) additionally updated a slight problem in the README, which explains why this touches the 9.6 and master branches. Branch -- REL9_4_STABLE Details --- http://git.postgresql.org/pg/commitdiff/30e3cb307d494990cb38cebff6bfc8b4ce27d7ab Modified Files -- src/backend/access/nbtree/README| 22 ++ src/backend/access/nbtree/nbtree.c | 15 +++ src/backend/access/nbtree/nbtxlog.c | 23 +-- src/include/access/nbtree.h | 6 -- 4 files changed, 58 insertions(+), 8 deletions(-) -- Sent via pgsql-committers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
[COMMITTERS] pgsql: Re-pgindent src/bin/pg_dump/*
Re-pgindent src/bin/pg_dump/* Cleanup for recent patches --- it's not much change, but I got annoyed while re-indenting the view-rule fix I'm working on. Branch -- master Details --- http://git.postgresql.org/pg/commitdiff/fcf70e0dbca1432959be5f3557acd546d639c9ba Modified Files -- src/bin/pg_dump/pg_backup_archiver.c | 8 ++-- src/bin/pg_dump/pg_backup_archiver.h | 33 +++-- src/bin/pg_dump/pg_dump.c| 6 -- 3 files changed, 29 insertions(+), 18 deletions(-) -- Sent via pgsql-committers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
[COMMITTERS] pgsql: Improve pg_dump/pg_restore --create --if-exists logic.
Improve pg_dump/pg_restore --create --if-exists logic. Teach it not to complain if the dropStmt attached to an archive entry is actually spelled CREATE OR REPLACE VIEW, since that will happen due to an upcoming bug fix. Also, if it doesn't recognize a dropStmt, have it print a WARNING and then emit the dropStmt unmodified. That seems like a much saner behavior than Assert'ing or dumping core due to a null-pointer dereference, which is what would happen before :-(. Back-patch to 9.4 where this option was introduced. Discussion: <[email protected]> Branch -- REL9_5_STABLE Details --- http://git.postgresql.org/pg/commitdiff/a7864037d8b7fd172d870782a8024d3f96b0b17b Modified Files -- src/bin/pg_dump/pg_backup_archiver.c | 48 1 file changed, 32 insertions(+), 16 deletions(-) -- Sent via pgsql-committers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
[COMMITTERS] pgsql: Improve pg_dump/pg_restore --create --if-exists logic.
Improve pg_dump/pg_restore --create --if-exists logic. Teach it not to complain if the dropStmt attached to an archive entry is actually spelled CREATE OR REPLACE VIEW, since that will happen due to an upcoming bug fix. Also, if it doesn't recognize a dropStmt, have it print a WARNING and then emit the dropStmt unmodified. That seems like a much saner behavior than Assert'ing or dumping core due to a null-pointer dereference, which is what would happen before :-(. Back-patch to 9.4 where this option was introduced. Discussion: <[email protected]> Branch -- REL9_6_STABLE Details --- http://git.postgresql.org/pg/commitdiff/0eaa5118ae457ff9873a796e08adaea1bb8172f9 Modified Files -- src/bin/pg_dump/pg_backup_archiver.c | 48 1 file changed, 32 insertions(+), 16 deletions(-) -- Sent via pgsql-committers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
[COMMITTERS] pgsql: Improve pg_dump/pg_restore --create --if-exists logic.
Improve pg_dump/pg_restore --create --if-exists logic. Teach it not to complain if the dropStmt attached to an archive entry is actually spelled CREATE OR REPLACE VIEW, since that will happen due to an upcoming bug fix. Also, if it doesn't recognize a dropStmt, have it print a WARNING and then emit the dropStmt unmodified. That seems like a much saner behavior than Assert'ing or dumping core due to a null-pointer dereference, which is what would happen before :-(. Back-patch to 9.4 where this option was introduced. Discussion: <[email protected]> Branch -- master Details --- http://git.postgresql.org/pg/commitdiff/ac888986fc656c0badb0a4918b3581e2ebcb154a Modified Files -- src/bin/pg_dump/pg_backup_archiver.c | 48 1 file changed, 32 insertions(+), 16 deletions(-) -- Sent via pgsql-committers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
[COMMITTERS] pgsql: Improve pg_dump/pg_restore --create --if-exists logic.
Improve pg_dump/pg_restore --create --if-exists logic. Teach it not to complain if the dropStmt attached to an archive entry is actually spelled CREATE OR REPLACE VIEW, since that will happen due to an upcoming bug fix. Also, if it doesn't recognize a dropStmt, have it print a WARNING and then emit the dropStmt unmodified. That seems like a much saner behavior than Assert'ing or dumping core due to a null-pointer dereference, which is what would happen before :-(. Back-patch to 9.4 where this option was introduced. Discussion: <[email protected]> Branch -- REL9_4_STABLE Details --- http://git.postgresql.org/pg/commitdiff/e69b532be797b3fac87a34be71fc97959f8a02aa Modified Files -- src/bin/pg_dump/pg_backup_archiver.c | 50 1 file changed, 33 insertions(+), 17 deletions(-) -- Sent via pgsql-committers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
[COMMITTERS] pgsql: Fix pg_dump's handling of circular dependencies in views.
Fix pg_dump's handling of circular dependencies in views. pg_dump's traditional solution for breaking a circular dependency involving a view was to create the view with CREATE TABLE and then later issue CREATE RULE "_RETURN" ... to convert the table to a view, relying on the backend's very very ancient code that supports making views that way. We've wanted to get rid of that kluge for a long time, but the thing that finally motivates doing something about it is the recognition that this method fails with the --clean option, because it leads to issuing DROP RULE "_RETURN" followed by DROP TABLE --- and the backend won't let you drop a view's _RETURN rule. Instead, let's break circular dependencies by initially creating the view using CREATE VIEW AS SELECT NULL::columntype AS columnname, ... (so that it has the right column names and types to support external references, but no dependencies beyond the column data types), and then later dumping the ON SELECT rule using the spelling CREATE OR REPLACE VIEW. This method wasn't available when this code was originally written, but it's been possible since PG 7.3, so it seems fine to start relying on it now. To solve the --clean problem, make the dropStmt for an ON SELECT rule be CREATE OR REPLACE VIEW with the same dummy target list as above. In this way, during the DROP phase, we first reduce the view to have no extra dependencies, and then we can drop it entirely when we've gotten rid of whatever had a circular dependency on it. (Note: this should work adequately well with the --if-exists option, since the CREATE OR REPLACE VIEW will go through whether the view exists or not. It could fail if the view exists with a conflicting column set, but we don't really support --clean against a non-matching database anyway.) This allows cleaning up some other kluges inside pg_dump, notably that we don't need a notion of reloptions attached to a rule anymore. Although this is a bug fix, commit to HEAD only for now. The problem's existed for a long time and we've had relatively few complaints, so it doesn't really seem worth taking risks to fix it in the back branches. We might revisit that choice if no problems emerge. Discussion: <[email protected]> Branch -- master Details --- http://git.postgresql.org/pg/commitdiff/d8c05aff56be92dda889abc89e3f3921d1c29c30 Modified Files -- src/bin/pg_dump/pg_dump.c | 181 +++-- src/bin/pg_dump/pg_dump.h | 3 +- src/bin/pg_dump/pg_dump_sort.c | 15 +--- 3 files changed, 140 insertions(+), 59 deletions(-) -- Sent via pgsql-committers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
[COMMITTERS] pgsql: Remove or reduce verbosity of some debug messages.
Remove or reduce verbosity of some debug messages. The debug messages that merely print StartTransactionCommand, CommitTransactionCommand, ProcessUtilty, or ProcessQuery with no additional details seem to be useless. Get rid of them. The transaction status messages produced by ShowTransactionState are occasionally useful, but they are extremely verbose, producing multiple lines of log output every time they fire, which can happens multiple times per transaction. So, reduce the level to DEBUG5; avoid emitting an extra line just to explain which debug point is at issue; and tighten up the rest of the message so it doesn't use quite so much horizontal space. With these changes, it's possible to run a somewhat busy system with a log level even as high as DEBUG4, whereas previously anything above DEBUG2 would flood the log with output that probably wasn't really all that useful. Branch -- master Details --- http://git.postgresql.org/pg/commitdiff/a43f1939d5dcd02f4df1604a68392332168e4be0 Modified Files -- src/backend/access/transam/xact.c | 22 ++ src/backend/tcop/postgres.c | 6 -- src/backend/tcop/pquery.c | 4 3 files changed, 10 insertions(+), 22 deletions(-) -- Sent via pgsql-committers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
