Re: pgsql: Fix the display of UNKNOWN message type in apply worker.
On Tue, Jul 25, 2023 at 03:52:32AM +, Amit Kapila wrote: > Fix the display of UNKNOWN message type in apply worker. > > We include the message type while displaying an error context in the > apply worker. Now, while retrieving the message type string if the > message type is unknown we throw an error that will hide the original > error. So, instead, we need to simply return the string indicating an > unknown message type. koel is complaining after this commit: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=koel=2023-07-25%2004%3A49%3A03 -- Michael signature.asc Description: PGP signature
pgsql: Optimize WAL insertion lock acquisition and release with some at
Optimize WAL insertion lock acquisition and release with some atomics The WAL insertion lock variable insertingAt is currently being read and written with the help of the LWLock wait list lock to avoid any read of torn values. This wait list lock can become a point of contention on a highly concurrent write workloads. This commit switches insertingAt to a 64b atomic variable that provides torn-free reads/writes. On platforms without 64b atomic support, the fallback implementation uses spinlocks to provide the same guarantees for the values read. LWLockWaitForVar(), through LWLockConflictsWithVar(), reads the new value to check if it still needs to wait with a u64 atomic operation. LWLockUpdateVar() updates the variable before waking up the waiters with an exchange_u64 (full memory barrier). LWLockReleaseClearVar() now uses also an exchange_u64 to reset the variable. Before this commit, all these steps relied on LWLockWaitListLock() and LWLockWaitListUnlock(). This reduces contention on LWLock wait list lock and improves performance of highly-concurrent write workloads. Here are some numbers using pg_logical_emit_message() (HEAD at d6677b93) with various arbitrary record lengths and clients up to 1k on a rather-large machine (64 vCPUs, 512GB of RAM, 16 cores per sockets, 2 sockets), in terms of TPS numbers coming from pgbench: message_size_b | 16 | 64 |256 | 1024 ++++--- patch_4_clients| 83830 | 82929 | 80478 | 73131 patch_16_clients | 267655 | 264973 | 250566 | 213985 patch_64_clients | 380423 | 378318 | 356907 | 294248 patch_256_clients | 360915 | 354436 | 326209 | 263664 patch_512_clients | 332654 | 321199 | 287521 | 240128 patch_1024_clients | 288263 | 276614 | 258220 | 217063 patch_2048_clients | 252280 | 243558 | 230062 | 192429 patch_4096_clients | 212566 | 213654 | 205951 | 166955 head_4_clients | 83686 | 83766 | 81233 | 73749 head_16_clients| 266503 | 265546 | 249261 | 213645 head_64_clients| 366122 | 363462 | 341078 | 261707 head_256_clients | 132600 | 132573 | 134392 | 165799 head_512_clients | 118937 | 114332 | 116860 | 150672 head_1024_clients | 133546 | 115256 | 125236 | 151390 head_2048_clients | 137877 | 117802 | 120909 | 138165 head_4096_clients | 113440 | 115611 | 120635 | 114361 Bharath has been measuring similar improvements, where the limit of the WAL insertion lock begins to be felt when more than 256 concurrent clients are involved in this specific workload. An extra patch has been discussed to introduce a fast-exit path in LWLockUpdateVar() when there are no waiters, still this does not influence the write-heavy workload cases discussed as there are always waiters. This will be considered separately. Author: Bharath Rupireddy Reviewed-by: Nathan Bossart, Andres Freund, Michael Paquier Discussion: https://postgr.es/m/CALj2ACVF+6jLvqKe6xhDzCCkr=rfd6upagc3477pji1ke9g...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/71e4cc6b8ec6a08f81973bd387fe575134cd0bdf Modified Files -- src/backend/access/transam/xlog.c | 4 ++-- src/backend/storage/lmgr/lwlock.c | 46 ++- src/include/storage/lwlock.h | 6 ++--- 3 files changed, 26 insertions(+), 30 deletions(-)
pgsql: Fix the display of UNKNOWN message type in apply worker.
Fix the display of UNKNOWN message type in apply worker. We include the message type while displaying an error context in the apply worker. Now, while retrieving the message type string if the message type is unknown we throw an error that will hide the original error. So, instead, we need to simply return the string indicating an unknown message type. Reported-by: Ashutosh Bapat Author: Euler Taveira, Amit Kapila Reviewed-by: Ashutosh Bapat Backpatch-through: 15 Discussion: https://postgr.es/m/caexhw5suaedw-mbzt_qu4rvxwz1vl54-l+ci2zreywebpzx...@mail.gmail.com Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/ad486b0eaef4b2fcb7c672fcbf5c2f31eb99ac83 Modified Files -- src/backend/replication/logical/proto.c | 13 ++--- src/backend/replication/logical/worker.c | 2 +- src/include/replication/logicalproto.h | 2 +- 3 files changed, 12 insertions(+), 5 deletions(-)
pgsql: Fix the display of UNKNOWN message type in apply worker.
Fix the display of UNKNOWN message type in apply worker. We include the message type while displaying an error context in the apply worker. Now, while retrieving the message type string if the message type is unknown we throw an error that will hide the original error. So, instead, we need to simply return the string indicating an unknown message type. Reported-by: Ashutosh Bapat Author: Euler Taveira, Amit Kapila Reviewed-by: Ashutosh Bapat Backpatch-through: 15 Discussion: https://postgr.es/m/caexhw5suaedw-mbzt_qu4rvxwz1vl54-l+ci2zreywebpzx...@mail.gmail.com Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/2864eb977a6e49d4355b679d7e1e54f01a231b80 Modified Files -- src/backend/replication/logical/proto.c | 13 ++--- src/backend/replication/logical/worker.c | 2 +- src/include/replication/logicalproto.h | 2 +- 3 files changed, 12 insertions(+), 5 deletions(-)
pgsql: Fix the display of UNKNOWN message type in apply worker.
Fix the display of UNKNOWN message type in apply worker. We include the message type while displaying an error context in the apply worker. Now, while retrieving the message type string if the message type is unknown we throw an error that will hide the original error. So, instead, we need to simply return the string indicating an unknown message type. Reported-by: Ashutosh Bapat Author: Euler Taveira, Amit Kapila Reviewed-by: Ashutosh Bapat Backpatch-through: 15 Discussion: https://postgr.es/m/caexhw5suaedw-mbzt_qu4rvxwz1vl54-l+ci2zreywebpzx...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/d38ad8e31dee1a69f4f6d4b6bb4e007e3751a93c Modified Files -- src/backend/replication/logical/proto.c | 13 ++--- src/backend/replication/logical/worker.c | 2 +- src/include/replication/logicalproto.h | 2 +- 3 files changed, 12 insertions(+), 5 deletions(-)
pgsql: Fix off-by-one in LimitAdditionalPins()
Fix off-by-one in LimitAdditionalPins()
Due to the bug LimitAdditionalPins() could return 0, violating
LimitAdditionalPins()'s API ("One additional pin is always allowed"). This
could be hit when setting shared_buffers very low and using a fair amount of
concurrency.
This bug was introduced in 31966b151e6a.
Author: "Anton A. Melnikov"
Reported-by: "Anton A. Melnikov"
Reported-by: Victoria Shepard
Discussion: https://postgr.es/m/ae46f2fb-5586-3de0-b54b-1bb0f6410...@inbox.ru
Backpatch: 16-
Branch
--
REL_16_STABLE
Details
---
https://git.postgresql.org/pg/commitdiff/bd2f46c6559ddcafe7bae5015ac45b69bc014067
Modified Files
--
src/backend/storage/buffer/bufmgr.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
pgsql: Fix off-by-one in LimitAdditionalPins()
Fix off-by-one in LimitAdditionalPins()
Due to the bug LimitAdditionalPins() could return 0, violating
LimitAdditionalPins()'s API ("One additional pin is always allowed"). This
could be hit when setting shared_buffers very low and using a fair amount of
concurrency.
This bug was introduced in 31966b151e6a.
Author: "Anton A. Melnikov"
Reported-by: "Anton A. Melnikov"
Reported-by: Victoria Shepard
Discussion: https://postgr.es/m/ae46f2fb-5586-3de0-b54b-1bb0f6410...@inbox.ru
Backpatch: 16-
Branch
--
master
Details
---
https://git.postgresql.org/pg/commitdiff/f3bc5192889f6f02aa10ca9f24df4eab1f1493c1
Modified Files
--
src/backend/storage/buffer/bufmgr.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
pgsql: Make test_decoding ddl.out shorter
Make test_decoding ddl.out shorter Some of the test_decoding test output was extremely wide, because it deals with massive toasted values, and the aligned mode causes psql to produce 200kB of whitespace and dashes. Change to unaligned mode temporarily to avoid that behavior. Backpatch to 14, where it applies cleanly. Discussion: https://postgr.es/m/20230405103953.sxleixp3uz5lazst@alvherre.pgsql Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/d6677b93c79b19bf3772cb32fcea0a77d8520db9 Modified Files -- contrib/test_decoding/expected/ddl.out | 316 - contrib/test_decoding/sql/ddl.sql | 6 + 2 files changed, 164 insertions(+), 158 deletions(-)
pgsql: Make test_decoding ddl.out shorter
Make test_decoding ddl.out shorter Some of the test_decoding test output was extremely wide, because it deals with massive toasted values, and the aligned mode causes psql to produce 200kB of whitespace and dashes. Change to unaligned mode temporarily to avoid that behavior. Backpatch to 14, where it applies cleanly. Discussion: https://postgr.es/m/20230405103953.sxleixp3uz5lazst@alvherre.pgsql Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/b1dc946eee3d8d4fd9776ccbc21bcb4ab747b998 Modified Files -- contrib/test_decoding/expected/ddl.out | 316 - contrib/test_decoding/sql/ddl.sql | 6 + 2 files changed, 164 insertions(+), 158 deletions(-)
pgsql: Make test_decoding ddl.out shorter
Make test_decoding ddl.out shorter Some of the test_decoding test output was extremely wide, because it deals with massive toasted values, and the aligned mode causes psql to produce 200kB of whitespace and dashes. Change to unaligned mode temporarily to avoid that behavior. Backpatch to 14, where it applies cleanly. Discussion: https://postgr.es/m/20230405103953.sxleixp3uz5lazst@alvherre.pgsql Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/c5c5832600e9dfa4f690d1f4af536c3fd6d5d7e9 Modified Files -- contrib/test_decoding/expected/ddl.out | 316 - contrib/test_decoding/sql/ddl.sql | 6 + 2 files changed, 164 insertions(+), 158 deletions(-)
pgsql: Make test_decoding ddl.out shorter
Make test_decoding ddl.out shorter Some of the test_decoding test output was extremely wide, because it deals with massive toasted values, and the aligned mode causes psql to produce 200kB of whitespace and dashes. Change to unaligned mode temporarily to avoid that behavior. Backpatch to 14, where it applies cleanly. Discussion: https://postgr.es/m/20230405103953.sxleixp3uz5lazst@alvherre.pgsql Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/3bb8b9342f8abe81af47ff7e5217865107cc15b3 Modified Files -- contrib/test_decoding/expected/ddl.out | 316 - contrib/test_decoding/sql/ddl.sql | 6 + 2 files changed, 164 insertions(+), 158 deletions(-)
pgsql: Compare only major versions in AdjustUpgrade.pm
Compare only major versions in AdjustUpgrade.pm Because PostgreSQL::Version is very nuanced about development version numbers, the comparison to 16beta2 makes it think that that release is older than 16, therefore applying a database tweak that doesn't work there (the comparison is only supposed to match when run on version 15). As suggested by Andrew Dunstan, fix by having AdjustUpgrade.pm public methods create a separate PostgreSQL::Version object to use for these comparisons, that only carries the major version number. While at it, have the same methods ensure that the objects given are of the expected type. Backpatch to 16. This module goes all the way back to 9.2, but there's probably no need for this fix except where betas still live. Co-authored-by: Andrew Dunstan Discussion: https://postgr.es/m/20230719110504.zbu74o54bqqlsufb@alvherre.pgsql Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/6061adedf53cbe0f94a52dd02465abf8c7c9fb43 Modified Files -- src/test/perl/PostgreSQL/Test/AdjustUpgrade.pm | 18 ++ 1 file changed, 18 insertions(+)
pgsql: Compare only major versions in AdjustUpgrade.pm
Compare only major versions in AdjustUpgrade.pm Because PostgreSQL::Version is very nuanced about development version numbers, the comparison to 16beta2 makes it think that that release is older than 16, therefore applying a database tweak that doesn't work there (the comparison is only supposed to match when run on version 15). As suggested by Andrew Dunstan, fix by having AdjustUpgrade.pm public methods create a separate PostgreSQL::Version object to use for these comparisons, that only carries the major version number. While at it, have the same methods ensure that the objects given are of the expected type. Backpatch to 16. This module goes all the way back to 9.2, but there's probably no need for this fix except where betas still live. Co-authored-by: Andrew Dunstan Discussion: https://postgr.es/m/20230719110504.zbu74o54bqqlsufb@alvherre.pgsql Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/28ce9d51f9082318e84f2a84385871394dcf2130 Modified Files -- src/test/perl/PostgreSQL/Test/AdjustUpgrade.pm | 18 ++ 1 file changed, 18 insertions(+)
