Re: pgsql: Enhance libpq encryption negotiation tests with new GUC
At Sun, 07 Apr 2024 23:50:08 +, Heikki Linnakangas
wrote in
> Enhance libpq encryption negotiation tests with new GUC
This commit adds the following messages:
> gettext_noop("Log details of pre-authentication connection handshake."),
Similar to a nearby commit, other messages with a similar context use
the phrase "Logs ". Wouldn't it be better to align this
message with existing ones?
diff --git a/src/backend/utils/misc/guc_tables.c
b/src/backend/utils/misc/guc_tables.c
index 83e3a59d7e..4584829992 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -1227,7 +1227,7 @@ struct config_bool ConfigureNamesBool[] =
},
{
{"trace_connection_negotiation", PGC_POSTMASTER,
DEVELOPER_OPTIONS,
- gettext_noop("Log details of pre-authentication
connection handshake."),
+ gettext_noop("Logs details of pre-authentication
connection handshake."),
NULL,
GUC_NOT_IN_SAMPLE
},
regards
--
Kyotaro Horiguchi
NTT Open Source Software Center
Re: pgsql: Transform OR clauses to ANY expression
At Mon, 08 Apr 2024 14:46:57 +0900 (JST), Kyotaro Horiguchi
wrote in
> At Sun, 07 Apr 2024 22:28:06 +, Alexander Korotkov
> wrote in
> > Transform OR clauses to ANY expression
>
> This commit introduces a message like this:
>
> > gettext_noop("Set the minimum length of the list of OR clauses to attempt
> > the OR-to-ANY transformation."),
>
> Unlike the usual phrasing of similar messages in this context, which
> use the form "Sets the minimum length of...", this message uses an
> imperative form ("Set"). I believe it would be better to align the
> tone of this message with the others by changing "Set" to "Sets".
>
> regards.
>
>
> diff --git a/src/backend/utils/misc/guc_tables.c
> b/src/backend/utils/misc/guc_tables.c
> index 83e3a59d7e..a527ffe0b0 100644
> --- a/src/backend/utils/misc/guc_tables.c
> +++ b/src/backend/utils/misc/guc_tables.c
> @@ -3670,7 +3670,7 @@ struct config_int ConfigureNamesInt[] =
>
> {
> {"or_to_any_transform_limit", PGC_USERSET, QUERY_TUNING_OTHER,
> - gettext_noop("Set the minimum length of the list of OR
> clauses to attempt the OR-to-ANY transformation."),
> + gettext_noop("Sets the minimum length of the list of OR
> clauses to attempt the OR-to-ANY transformation."),
> gettext_noop("Once the limit is reached, the planner
> will try to replace expression like "
>"'x=c1 OR x=c2 ..' to the
> expression 'x = ANY(ARRAY[c1,c2,..])'"),
> GUC_EXPLAIN
Sorry for the sequential posts, but I found the following contruct in
the same patch to be quite untranslatable.
> errmsg("%s bound of partition \"%s\" is %s %s bound of split partition",
> first ? "lower" : "upper",
> relname,
> defaultPart ? (first ? "less than" : "greater than") : "not
> equals to",
> first ? "lower" : "upper"),
> parser_errposition(pstate, datum->location)));
I might be misunderstanding this, but if the expressions are correct,
the message could be divided into four fixed messages based on "first"
and "defaultPart". Alternatively, it might be better to provide
simpler explation of the situation.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
pgsql: Fix the wording of or_to_any_transform_limit description
Fix the wording of or_to_any_transform_limit description Reported-by: Kyotaro Horiguchi Discussion: https://postgr.es/m/20240408.144657.1746688590065601661.horikyota.ntt%40gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/b453a7a16a7ed2ba96522e521143bc652b74875f Modified Files -- src/backend/utils/misc/guc_tables.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
pgsql: Fix the value of or_to_any_transform_limit in postgresql.conf.sa
Fix the value of or_to_any_transform_limit in postgresql.conf.sample Reported-by: Justin Pryzby Discussion: https://postgr.es/m/ZhM8jH8gsKm5Q-9p%40pryzbyj2023 Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/2af75e1174786a02fc35755c0cb98c522d72a065 Modified Files -- src/backend/utils/misc/postgresql.conf.sample | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Re: pgsql: Transform OR clauses to ANY expression
At Sun, 07 Apr 2024 22:28:06 +, Alexander Korotkov
wrote in
> Transform OR clauses to ANY expression
This commit introduces a message like this:
> gettext_noop("Set the minimum length of the list of OR clauses to attempt the
> OR-to-ANY transformation."),
Unlike the usual phrasing of similar messages in this context, which
use the form "Sets the minimum length of...", this message uses an
imperative form ("Set"). I believe it would be better to align the
tone of this message with the others by changing "Set" to "Sets".
regards.
diff --git a/src/backend/utils/misc/guc_tables.c
b/src/backend/utils/misc/guc_tables.c
index 83e3a59d7e..a527ffe0b0 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3670,7 +3670,7 @@ struct config_int ConfigureNamesInt[] =
{
{"or_to_any_transform_limit", PGC_USERSET, QUERY_TUNING_OTHER,
- gettext_noop("Set the minimum length of the list of OR
clauses to attempt the OR-to-ANY transformation."),
+ gettext_noop("Sets the minimum length of the list of OR
clauses to attempt the OR-to-ANY transformation."),
gettext_noop("Once the limit is reached, the planner
will try to replace expression like "
"'x=c1 OR x=c2 ..' to the
expression 'x = ANY(ARRAY[c1,c2,..])'"),
GUC_EXPLAIN
--
Kyotaro Horiguchi
NTT Open Source Software Center
pgsql: Remove references to old function name
Remove references to old function name In a97bbe1f1df I accidentally referenced heapgetpage(), both in a function name and a comment. But since 44086b09753 the relevant function is named heap_prepare_pagescan(). Rename the new function to page_collect_tuples(). Reported-by: Melanie Plageman Reported-by: David Rowley Discussion: https://postgr.es/m/20240407172615.cocrsvboqm3tt...@awork3.anarazel.de Discussion: https://postgr.es/m/CAApHDvp4SniHopTrVeKWcEvNXFtdki0utAvO=5r7h6tnhtu...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/e3b69be9515979c546fbfc51f9f801c7b198 Modified Files -- src/backend/access/heap/heapam.c | 16 1 file changed, 8 insertions(+), 8 deletions(-)
pgsql: Add pg_buffercache_evict() function for testing.
Add pg_buffercache_evict() function for testing. When testing buffer pool logic, it is useful to be able to evict arbitrary blocks. This function can be used in SQL queries over the pg_buffercache view to set up a wide range of buffer pool states. Of course, buffer mappings might change concurrently so you might evict a block other than the one you had in mind, and another session might bring it back in at any time. That's OK for the intended purpose of setting up developer testing scenarios, and more complicated interlocking schemes to give stronger guararantees about that would likely be less flexible for actual testing work anyway. Superuser-only. Author: Palak Chaturvedi Author: Thomas Munro (docs, small tweaks) Reviewed-by: Nitin Jadhav Reviewed-by: Andres Freund Reviewed-by: Cary Huang Reviewed-by: Cédric Villemain Reviewed-by: Jim Nasby Reviewed-by: Maxim Orlov Reviewed-by: Thomas Munro Reviewed-by: Melanie Plageman Discussion: https://postgr.es/m/calfch19pw48zwwzuorspsav9hqt0upyabpc4boz4w+c7ff5...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/13453eedd3f692f8dcf8e334396eee84f00fdde2 Modified Files -- contrib/pg_buffercache/Makefile| 2 +- contrib/pg_buffercache/meson.build | 1 + .../pg_buffercache/pg_buffercache--1.4--1.5.sql| 6 +++ contrib/pg_buffercache/pg_buffercache.control | 2 +- contrib/pg_buffercache/pg_buffercache_pages.c | 20 +++ doc/src/sgml/pgbuffercache.sgml| 39 +++--- src/backend/storage/buffer/bufmgr.c| 63 ++ src/include/storage/bufmgr.h | 2 + 8 files changed, 127 insertions(+), 8 deletions(-)
pgsql: Fix alignment of stack variable
Fix alignment of stack variable Declare with union similar to PGAlignedBlock. Report and fix by Andres Freund Discussion: https://postgr.es/m/20240407190731.izm3mdazednrsiqk%40awork3.anarazel.de Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/0ea51bac3802dca9dcf5cbaaf7a19a1c1ae4781a Modified Files -- src/backend/access/common/tidstore.c | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-)
pgsql: Add more tab completion support for ALTER DEFAULT PRIVILEGES in
Add more tab completion support for ALTER DEFAULT PRIVILEGES in psql. This adds tab completion of "GRANT" and "REVOKE [GRANT OPTION FOR]" for ALTER DEFAULT PRIVILEGES, and adds "WITH GRANT OPTION" for ALTER DEFAULT PRIVILEGES ... GRANT ... TO role. Author: Vignesh C, with cosmetic adjustments by me Reviewed-by: Shubham Khanna, Masahiko Sawada Discussion: https://postgr.es/m/CALDaNm1aEdJb-QJi%3DGWStkfj_%2BEDUK_VtDkn%2BTjQ2z7HyU0MBw%40mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/304b6b1a6b7116bae30ea24119f04474eba7f0ed Modified Files -- src/bin/psql/tab-complete.c | 20 1 file changed, 16 insertions(+), 4 deletions(-)
pgsql: Remove redundant nbtree preprocessing assertions.
Remove redundant nbtree preprocessing assertions. One of the assertions was the subject of a false positive complaint from Coverity, but none of the assertions added much, so get rid of them. Reported-By: Tom Lane Discussion: https://postgr.es/m/3000247.1712537...@sss.pgh.pa.us Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/3b08133cd13c4ac985a9480160282ba35838 Modified Files -- src/backend/access/nbtree/nbtutils.c | 4 1 file changed, 4 deletions(-)
pgsql: simplehash: Free collisions array in SH_STAT
simplehash: Free collisions array in SH_STAT While SH_STAT() is only used for debugging, the allocated array can be large, and therefore should be freed. It's unclear why coverity started warning now. Reported-by: Tom Lane Reported-by: Coverity Discussion: https://postgr.es/m/3005248.1712538...@sss.pgh.pa.us Backpatch: 12- Branch -- REL_12_STABLE Details --- https://git.postgresql.org/pg/commitdiff/4b179a47242993fe48b1bfa21b17dc9d5e2b35a2 Modified Files -- src/include/lib/simplehash.h | 3 +++ 1 file changed, 3 insertions(+)
pgsql: simplehash: Free collisions array in SH_STAT
simplehash: Free collisions array in SH_STAT While SH_STAT() is only used for debugging, the allocated array can be large, and therefore should be freed. It's unclear why coverity started warning now. Reported-by: Tom Lane Reported-by: Coverity Discussion: https://postgr.es/m/3005248.1712538...@sss.pgh.pa.us Backpatch: 12- Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/974374dcd394c80f37e20b692b7d8b122aa8ed33 Modified Files -- src/include/lib/simplehash.h | 3 +++ 1 file changed, 3 insertions(+)
pgsql: simplehash: Free collisions array in SH_STAT
simplehash: Free collisions array in SH_STAT While SH_STAT() is only used for debugging, the allocated array can be large, and therefore should be freed. It's unclear why coverity started warning now. Reported-by: Tom Lane Reported-by: Coverity Discussion: https://postgr.es/m/3005248.1712538...@sss.pgh.pa.us Backpatch: 12- Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/25ee58e8f26df9ce3037258206eb100a3fc55103 Modified Files -- src/include/lib/simplehash.h | 3 +++ 1 file changed, 3 insertions(+)
pgsql: simplehash: Free collisions array in SH_STAT
simplehash: Free collisions array in SH_STAT While SH_STAT() is only used for debugging, the allocated array can be large, and therefore should be freed. It's unclear why coverity started warning now. Reported-by: Tom Lane Reported-by: Coverity Discussion: https://postgr.es/m/3005248.1712538...@sss.pgh.pa.us Backpatch: 12- Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/dcb7cf945c88eabce7f8350023b8184266d96a12 Modified Files -- src/include/lib/simplehash.h | 3 +++ 1 file changed, 3 insertions(+)
pgsql: simplehash: Free collisions array in SH_STAT
simplehash: Free collisions array in SH_STAT While SH_STAT() is only used for debugging, the allocated array can be large, and therefore should be freed. It's unclear why coverity started warning now. Reported-by: Tom Lane Reported-by: Coverity Discussion: https://postgr.es/m/3005248.1712538...@sss.pgh.pa.us Backpatch: 12- Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/af7e90a2778625d63add69165fb68c370509a199 Modified Files -- src/include/lib/simplehash.h | 3 +++ 1 file changed, 3 insertions(+)
pgsql: simplehash: Free collisions array in SH_STAT
simplehash: Free collisions array in SH_STAT While SH_STAT() is only used for debugging, the allocated array can be large, and therefore should be freed. It's unclear why coverity started warning now. Reported-by: Tom Lane Reported-by: Coverity Discussion: https://postgr.es/m/3005248.1712538...@sss.pgh.pa.us Backpatch: 12- Branch -- REL_13_STABLE Details --- https://git.postgresql.org/pg/commitdiff/eabf98e949e3a655494a32ceb8d146117cce0505 Modified Files -- src/include/lib/simplehash.h | 3 +++ 1 file changed, 3 insertions(+)
Re: pgsql: Send ALPN in TLS handshake, require it in direct SSL connections
On 08/04/2024 04:50, Andres Freund wrote: On 2024-04-08 01:25:40 +, Heikki Linnakangas wrote: Send ALPN in TLS handshake, require it in direct SSL connections libpq now always tries to send ALPN. With the traditional negotiated SSL connections, the server accepts the ALPN, and refuses the connection if it's not what we expect, but connecting without ALPN is still OK. With the new direct SSL connections, ALPN is mandatory. NOTE: This uses "TBD-pgsql" as the protocol ID. We must register a proper one with IANA before the release! Author: Greg Stark, Heikki Linnakangas Reviewed-by: Matthias van de Meent, Jacob Champion My compiler complains: [396/992 42 39%] Compiling C object src/backend/postgres_lib.a.p/libpq_be-secure-openssl.c.o ../../../../../home/andres/src/postgresql/src/backend/libpq/be-secure-openssl.c: In function 'alpn_cb': ../../../../../home/andres/src/postgresql/src/backend/libpq/be-secure-openssl.c:1327:69: warning: ordered comparison of pointer with integer zero [-Wextra] 1327 | if (*out == NULL || *outlen > sizeof(alpn_protos) || outlen <= 0) | ^~ [991/992 1 99%] Linking target src/bin/pg_dump/pg_dump And I think it may show why the warning is a good idea - I assume "*outlen <= 0" was intended? Yes. Fixed, thanks! -- Heikki Linnakangas Neon (https://neon.tech)
pgsql: Fix check for 'outlen' return from SSL_select_next_proto()
Fix check for 'outlen' return from SSL_select_next_proto() Fixes compiler warning reported by Andres Freund. Discusssion: https://www.postgresql.org/message-id/20240408015055.xsuahullywpfw...@awork3.anarazel.de Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/3e60e956b0017ae9d329bf242e76c4487fb39b4d Modified Files -- src/backend/libpq/be-secure-openssl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Re: pgsql: Send ALPN in TLS handshake, require it in direct SSL connections
Hi, On 2024-04-08 01:25:40 +, Heikki Linnakangas wrote: > Send ALPN in TLS handshake, require it in direct SSL connections > > libpq now always tries to send ALPN. With the traditional negotiated > SSL connections, the server accepts the ALPN, and refuses the > connection if it's not what we expect, but connecting without ALPN is > still OK. With the new direct SSL connections, ALPN is mandatory. > > NOTE: This uses "TBD-pgsql" as the protocol ID. We must register a > proper one with IANA before the release! > > Author: Greg Stark, Heikki Linnakangas > Reviewed-by: Matthias van de Meent, Jacob Champion My compiler complains: [396/992 42 39%] Compiling C object src/backend/postgres_lib.a.p/libpq_be-secure-openssl.c.o ../../../../../home/andres/src/postgresql/src/backend/libpq/be-secure-openssl.c: In function 'alpn_cb': ../../../../../home/andres/src/postgresql/src/backend/libpq/be-secure-openssl.c:1327:69: warning: ordered comparison of pointer with integer zero [-Wextra] 1327 | if (*out == NULL || *outlen > sizeof(alpn_protos) || outlen <= 0) | ^~ [991/992 1 99%] Linking target src/bin/pg_dump/pg_dump And I think it may show why the warning is a good idea - I assume "*outlen <= 0" was intended? Greetings, Andres Freund
pgsql: Silence perlcritic warnings in new libpq tests
Silence perlcritic warnings in new libpq tests Per buildfarm member 'koel'. Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/d60ab76f630921dc4deb958fb7892ef6b0e4a7ee Modified Files -- src/test/libpq_encryption/t/001_negotiate_encryption.pl | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
pgsql: Refactor libpq state machine for negotiating encryption
Refactor libpq state machine for negotiating encryption This fixes the few corner cases noted in commit 705843d294, as shown by the changes in the test. Author: Heikki Linnakangas, Matthias van de Meent Reviewed-by: Jacob Champion Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/05fd30c0e730bd5238f62d2fdfdcfaf28b16b225 Modified Files -- src/interfaces/libpq/fe-connect.c | 427 - src/interfaces/libpq/libpq-int.h | 14 +- .../libpq_encryption/t/001_negotiate_encryption.pl | 26 +- 3 files changed, 265 insertions(+), 202 deletions(-)
pgsql: Support TLS handshake directly without SSLRequest negotiation
Support TLS handshake directly without SSLRequest negotiation By skipping SSLRequest, you can eliminate one round-trip when establishing a TLS connection. It is also more friendly to generic TLS proxies that don't understand the PostgreSQL protocol. This is disabled by default in libpq, because the direct TLS handshake will fail with old server versions. It can be enabled with the sslnegotation=direct option. It will still fall back to the negotiated TLS handshake if the server rejects the direct attempt, either because it is an older version or the server doesn't support TLS at all, but the fallback can be disabled with the sslnegotiation=requiredirect option. Author: Greg Stark, Heikki Linnakangas Reviewed-by: Matthias van de Meent, Jacob Champion Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/d39a49c1e459804831302807c724fa6512e90cf0 Modified Files -- doc/src/sgml/libpq.sgml| 87 - doc/src/sgml/protocol.sgml | 36 +++ src/backend/libpq/be-secure.c | 52 ++- src/backend/libpq/pqcomm.c | 12 +- src/backend/tcop/backend_startup.c | 98 +- src/include/libpq/libpq-be.h | 13 + src/include/libpq/libpq.h | 2 +- src/interfaces/libpq/fe-connect.c | 102 +- src/interfaces/libpq/fe-secure-openssl.c | 7 +- src/interfaces/libpq/libpq-fe.h| 4 +- src/interfaces/libpq/libpq-int.h | 6 +- .../libpq_encryption/t/001_negotiate_encryption.pl | 357 + 12 files changed, 609 insertions(+), 167 deletions(-)
pgsql: Send ALPN in TLS handshake, require it in direct SSL connections
Send ALPN in TLS handshake, require it in direct SSL connections libpq now always tries to send ALPN. With the traditional negotiated SSL connections, the server accepts the ALPN, and refuses the connection if it's not what we expect, but connecting without ALPN is still OK. With the new direct SSL connections, ALPN is mandatory. NOTE: This uses "TBD-pgsql" as the protocol ID. We must register a proper one with IANA before the release! Author: Greg Stark, Heikki Linnakangas Reviewed-by: Matthias van de Meent, Jacob Champion Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/91044ae4baeac2e501e34164a69bd5d9c4976d21 Modified Files -- doc/src/sgml/libpq.sgml | 12 + src/backend/libpq/be-secure-openssl.c| 77 src/backend/tcop/backend_startup.c | 8 src/bin/psql/command.c | 7 ++- src/include/libpq/libpq-be.h | 1 + src/include/libpq/pqcomm.h | 19 src/interfaces/libpq/fe-secure-openssl.c | 35 +++ 7 files changed, 157 insertions(+), 2 deletions(-)
pgsql: Use streaming I/O in ANALYZE.
Use streaming I/O in ANALYZE. The ANALYZE command prefetches and reads sample blocks chosen by a BlockSampler algorithm. Instead of calling [Prefetch|Read]Buffer() for each block, ANALYZE now uses the streaming API introduced in b5a9b18cd0. Author: Nazir Bilal Yavuz Reviewed-by: Melanie Plageman Reviewed-by: Andres Freund Reviewed-by: Jakub Wartak Reviewed-by: Heikki Linnakangas Reviewed-by: Thomas Munro Discussion: https://postgr.es/m/flat/CAN55FZ0UhXqk9v3y-zW_fp4-WCp43V8y0A72xPmLkOM%2B6M%2BmJg%40mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/041b96802efa33d2bc9456f2ad946976b92b5ae1 Modified Files -- src/backend/access/heap/heapam_handler.c | 29 ++- src/backend/commands/analyze.c | 85 ++-- src/include/access/heapam.h | 5 +- 3 files changed, 43 insertions(+), 76 deletions(-)
pgsql: injection_points: Introduce runtime conditions
injection_points: Introduce runtime conditions This adds a new SQL function injection_points_set_local() that can be used to force injection points to be run only in the process where they are attached. This is handy for SQL tests to: - Detach automatically injection points when the process exits. - Allow tests with injection points to run concurrently with other test suites, so as such modules do not have to be marked with NO_INSTALLCHECK. Currently, the only condition that can be registered is for a PID. This could be extended to more kinds later, if required, like database names/OIDs, roles, or more concepts I did not consider. Using a single function for SQL scripts is an idea from Heikki Linnakangas. Reviewed-by: Andrey Borodin Discussion: https://postgr.es/m/zfp7ids9tvrke...@paquier.xyz Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/f587338dec87d3c35b025e131c5977930ac69077 Modified Files -- .../injection_points/expected/injection_points.out | 77 + .../injection_points/injection_points--1.0.sql | 11 ++ .../modules/injection_points/injection_points.c| 176 + .../injection_points/sql/injection_points.sql | 20 +++ src/tools/pgindent/typedefs.list | 1 + 5 files changed, 285 insertions(+)
pgsql: Enhance libpq encryption negotiation tests with new GUC
Enhance libpq encryption negotiation tests with new GUC The new "log_connection_negotiation" server option causes the server to print messages to the log when it receives a SSLRequest or GSSENCRequest packet from the client. Together with "log_connections", it gives a trace of how a connection and encryption is negotiatated. Use the option in the libpq_encryption test, to verify in more detail how libpq negotiates encryption with different gssencmode and sslmode options. This revealed a couple of cases where libpq retries encryption or authentication, when it should already know that it cannot succeed. I marked them with XXX comments in the test tables. They only happen when the connection was going to fail anyway, and only with rare combinations of options, so they're not serious. Discussion: https://www.postgresql.org/message-id/caeze2wja8vuozygcepwueicrwa4jp316k0mvjrow4pfmwp0...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/705843d294d5d3bc5cce4001596df4df5f1c8b59 Modified Files -- src/backend/tcop/backend_startup.c | 23 ++ src/backend/utils/misc/guc_tables.c| 11 + .../libpq_encryption/t/001_negotiate_encryption.pl | 380 - 3 files changed, 255 insertions(+), 159 deletions(-)
pgsql: With gssencmode='require', check credential cache before connect
With gssencmode='require', check credential cache before connecting Previously, libpq would establish the TCP connection, and then immediately disconnect if the credentials were not available. The same thing happened if you tried to use a Unix domain socket with gssencmode=require. Check those conditions before establishing the TCP connection. This is a very minor issue, but my motivation to do this now is that I'm about to add more detail to the tests for encryption negotiation. This makes the case of gssencmode=require but no credentials configured fail at the same stage as with gssencmode=require and GSSAPI support not compiled at all. That avoids having to deal with variations in expected output depending on build options. Discussion: https://www.postgresql.org/message-id/caeze2wja8vuozygcepwueicrwa4jp316k0mvjrow4pfmwp0...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/20f9b61cc1926775b1ceb25196df942efaf8bdd2 Modified Files -- src/interfaces/libpq/fe-connect.c | 32 ++-- 1 file changed, 30 insertions(+), 2 deletions(-)
pgsql: Add tests for libpq gssencmode and sslmode options
Add tests for libpq gssencmode and sslmode options Test all combinations of gssencmode, sslmode, whether the server supports SSL and/or GSSAPI encryption, and whether they are accepted by pg_hba.conf. This is in preparation for refactoring that code in libpq, and for adding a new option for "direct SSL" connections, which adds another dimension to the logic. If we add even more options in the future, testing all combinations will become unwieldy and we'll need to rethink this, but for now an exhaustive test is nice. Author: Heikki Linnakangas, Matthias van de Meent Reviewed-by: Jacob Champion Discussion: https://www.postgresql.org/message-id/a3af4070-3556-461d-aec8-a8d794f94...@iki.fi Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/1169920ff77025550718b90a5cafc6849875f43f Modified Files -- .cirrus.tasks.yml | 2 +- src/test/libpq_encryption/Makefile | 25 + src/test/libpq_encryption/README | 31 ++ src/test/libpq_encryption/meson.build | 18 + .../libpq_encryption/t/001_negotiate_encryption.pl | 548 + src/test/meson.build | 1 + 6 files changed, 624 insertions(+), 1 deletion(-)
pgsql: Move Kerberos module
Move Kerberos module So that we can reuse it in new tests. Discussion: https://www.postgresql.org/message-id/a3af4070-3556-461d-aec8-a8d794f94...@iki.fi Reviewed-by: Jacob Champion, Matthias van de Meent Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/9f899562d420fe36305badc169550673914389b7 Modified Files -- src/test/kerberos/t/001_auth.pl | 179 ++- src/test/perl/PostgreSQL/Test/Kerberos.pm | 234 ++ 2 files changed, 245 insertions(+), 168 deletions(-)
pgsql: Make GIN test using injection points repeatable
Make GIN test using injection points repeatable As written, the test would fail when run repeatedly because one of the injection points attached was not detached. This would not matter if the test is rewritten to be concurrently safe, but let's be clean and it is a good practice. Oversight in 6a1ea02c491d. Discussion: https://postgr.es/m/zfp7ids9tvrke...@paquier.xyz Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/997db123c054c597164be77b7bf7da6479471621 Modified Files -- src/test/modules/gin/expected/gin_incomplete_splits.out | 6 ++ src/test/modules/gin/sql/gin_incomplete_splits.sql | 2 ++ 2 files changed, 8 insertions(+)
pgsql: Fix incorrect KeeperBlock macro in bump.c
Fix incorrect KeeperBlock macro in bump.c The macro was missing a MAXALIGN around the sizeof(BumpContext) which would cause problems detecting the keeper block on 32-bit systems that have a MAXALIGN value of 8. Thank you to Andres Freund, Tomas Vondra and Tom Lane for investigating and testing. Reported-by: Melanie Plageman, Tomas Vondra Discussion: https://postgr.es/m/caakru_y6dzjijezghgnzp0gjar1jvq-ch7xgdqexdvhnpgd...@mail.gmail.com Discussion: https://postgr.es/m/a4a10b89-6ba8-4abd-b449-019aafff0...@enterprisedb.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/705ec0565371033082e7b1c278afcb42aa5a7421 Modified Files -- src/backend/utils/mmgr/bump.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
pgsql: Fix usage of same ListCell transform_or_to_any()'s in nested loo
Fix usage of same ListCell transform_or_to_any()'s in nested loops Discussion: https://postgr.es/m/CAAKRu_b4SXNW4GAM0bv3e6wcL5ODSXg1ZdRCn6uyLLjSPbveBg%40mail.gmail.com Author: Melanie Plageman Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/beabea6c2063e583628c59d03102dba996975b4a Modified Files -- src/backend/optimizer/prep/prepqual.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Re: pgsql: Transform OR clauses to ANY expression
On Mon, Apr 8, 2024 at 1:35 AM Melanie Plageman wrote: > On Sun, Apr 7, 2024 at 6:28 PM Alexander Korotkov > wrote: > > > > Transform OR clauses to ANY expression > > > > Replace (expr op C1) OR (expr op C2) ... with expr op ANY(ARRAY[C1, C2, > > ...]) > > on the preliminary stage of optimization when we are still working with the > > expression tree. > > > > Here Cn is a n-th constant expression, 'expr' is non-constant expression, > > 'op' > > is an operator which returns boolean result and has a commuter (for the case > > of reverse order of constant and non-constant parts of the expression, > > like 'Cn op expr'). > > > > Sometimes it can lead to not optimal plan. This is why there is a > > or_to_any_transform_limit GUC. It specifies a threshold value of length of > > arguments in an OR expression that triggers the OR-to-ANY transformation. > > Generally, more groupable OR arguments mean that transformation will be more > > likely to win than to lose. > > I'm getting this warning now > > /src/backend/optimizer/prep/prepqual.c:582:33: warning: declaration of > ‘lc__state’ shadows a previous local [-Wshadow=compatible-local] > 582 | foreach(lc, entry->consts) Thank you for catching. I'm fixing this now. -- Regards, Alexander Korotkov
Re: pgsql: Transform OR clauses to ANY expression
On Sun, Apr 7, 2024 at 6:28 PM Alexander Korotkov wrote: > > Transform OR clauses to ANY expression > > Replace (expr op C1) OR (expr op C2) ... with expr op ANY(ARRAY[C1, C2, ...]) > on the preliminary stage of optimization when we are still working with the > expression tree. > > Here Cn is a n-th constant expression, 'expr' is non-constant expression, 'op' > is an operator which returns boolean result and has a commuter (for the case > of reverse order of constant and non-constant parts of the expression, > like 'Cn op expr'). > > Sometimes it can lead to not optimal plan. This is why there is a > or_to_any_transform_limit GUC. It specifies a threshold value of length of > arguments in an OR expression that triggers the OR-to-ANY transformation. > Generally, more groupable OR arguments mean that transformation will be more > likely to win than to lose. I'm getting this warning now /src/backend/optimizer/prep/prepqual.c:582:33: warning: declaration of ‘lc__state’ shadows a previous local [-Wshadow=compatible-local] 582 | foreach(lc, entry->consts)
pgsql: Transform OR clauses to ANY expression
Transform OR clauses to ANY expression Replace (expr op C1) OR (expr op C2) ... with expr op ANY(ARRAY[C1, C2, ...]) on the preliminary stage of optimization when we are still working with the expression tree. Here Cn is a n-th constant expression, 'expr' is non-constant expression, 'op' is an operator which returns boolean result and has a commuter (for the case of reverse order of constant and non-constant parts of the expression, like 'Cn op expr'). Sometimes it can lead to not optimal plan. This is why there is a or_to_any_transform_limit GUC. It specifies a threshold value of length of arguments in an OR expression that triggers the OR-to-ANY transformation. Generally, more groupable OR arguments mean that transformation will be more likely to win than to lose. Discussion: https://postgr.es/m/567ED6CA.2040504%40sigaev.ru Author: Alena Rybakina Author: Andrey Lepikhov Reviewed-by: Peter Geoghegan Reviewed-by: Ranier Vilela Reviewed-by: Alexander Korotkov Reviewed-by: Robert Haas Reviewed-by: Jian He Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/72bd38cc99a15da6f97373fae98027c908c398ea Modified Files -- doc/src/sgml/config.sgml | 57 src/backend/nodes/queryjumblefuncs.c | 27 ++ src/backend/optimizer/prep/prepqual.c | 399 +- src/backend/utils/misc/guc_tables.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/nodes/queryjumble.h | 1 + src/include/optimizer/optimizer.h | 2 + src/test/regress/expected/create_index.out| 159 ++ src/test/regress/expected/join.out| 50 src/test/regress/expected/partition_prune.out | 37 +-- src/test/regress/sql/create_index.sql | 45 +++ src/test/regress/sql/join.sql | 11 + src/test/regress/sql/partition_prune.sql | 2 + src/tools/pgindent/typedefs.list | 2 + 14 files changed, 785 insertions(+), 20 deletions(-)
pgsql: Change debug printing to log filename
Change debug printing to log filename When restarting the cluster fails the code introduced in 33774978c78 printed the full log contents to aid debugging. For cases when the logfile is large this adds unnecessary overhead. Reduce to printing the logfile path instead. Reported-by: Andres Freund Discussion: https://postgr.es/m/20240406214439.2n4zf2w7ukhf7...@awork3.anarazel.de Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/75a47b6a0dc4a235307e1acd1a6b3845cb881e55 Modified Files -- src/test/perl/PostgreSQL/Test/Cluster.pm | 8 1 file changed, 4 insertions(+), 4 deletions(-)
pgsql: Doc: clarify behavior of boolean options in replication protocol
Doc: clarify behavior of boolean options in replication protocol commands. Same idea as ec7e053a9, but applying to the walsender commands described in protocol.sgml. Peter Smith Discussion: https://postgr.es/m/CAHut+PvwjZfdGt2R8HTXgSZft=jzkymrs8kug31ps7zqaaw...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/626603d46337ee331d7616b34ae539b1195556a7 Modified Files -- doc/src/sgml/protocol.sgml | 7 +++ 1 file changed, 7 insertions(+)
Re: pgsql: Use bump memory context for tuplesorts
Hi,
On 2024-04-07 13:52:28 -0400, Tom Lane wrote:
> Melanie Plageman writes:
> > Looks like this assert is tripping on grison [1].
>
> > running bootstrap script ... TRAP: failed Assert("total_allocated ==
> > context->mem_allocated"), File: "bump.c", Line: 808, PID: 30248
>
> The same on mamba. However, I failed to duplicate it in an x86
> (32-bit) VM. I think the critical factor on the failing machines
> is that MAXALIGN is 8 which is more than their sizeof(pointer).
For posterity: I think the above is correct, and that I found the concrete
reason this causes issues on the thread related to the patch. I posted there
because I figured it out after looking at Tomas' gdb output:
https://postgr.es/m/20240407210924.fvwfwbzms3bvtfcd%40awork3.anarazel.de
- Andres
pgsql: Remove useless duplicate call of defGetBoolean().
Remove useless duplicate call of defGetBoolean(). Seems to be a copy-and-paste error dating to dc2123400. Noted while reviewing a related documentation patch. Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/0c66a164e75163f5a850fb89d13ec0b213753d8d Modified Files -- src/backend/backup/basebackup.c | 1 - 1 file changed, 1 deletion(-)
pgsql: Doc: show how to get the equivalent of LIMIT for UPDATE/DELETE.
Doc: show how to get the equivalent of LIMIT for UPDATE/DELETE. Add examples showing use of a CTE and a self-join to perform partial UPDATEs and DELETEs. Corey Huinker, reviewed by Laurenz Albe Discussion: https://postgr.es/m/CADkLM=caneqsuwpwnfi2jr4ix99e0ejm_3jtce-yjneqc7r...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/2daeba6a4e49f2ac4cc202fc9352c2a59e928a6c Modified Files -- doc/src/sgml/ref/delete.sgml | 24 ++-- doc/src/sgml/ref/update.sgml | 40 +++- 2 files changed, 61 insertions(+), 3 deletions(-)
pgsql: Doc: update documentation about EXCLUDE constraint elements.
Doc: update documentation about EXCLUDE constraint elements. What the documentation calls an exclude_element is an index_elem according to gram.y, and it allows all the same options that a CREATE INDEX column specification does. The COLLATE patch neglected to update the CREATE/ALTER TABLE docs about that, and later the opclass-parameters patch made the same oversight. Add those options to the syntax synopses, and polish the associated text a bit. Back-patch to v13 where opclass parameters came in. We could update v12 with just the COLLATE omission, but it doesn't quite seem worth the trouble at this point. Shihao Zhong, reviewed by Daniel Vérité, Shubham Khanna and myself Discussion: https://postgr.es/m/CAGRkXqShbVyB8E3gapfdtuwiWTiK=Q67Qb9qwxu=+-w0w46...@mail.gmail.com Branch -- REL_13_STABLE Details --- https://git.postgresql.org/pg/commitdiff/5fe7bcc6766be52bf5958ac660af92da482ba0e3 Modified Files -- doc/src/sgml/ref/alter_table.sgml | 2 +- doc/src/sgml/ref/create_table.sgml | 10 +- 2 files changed, 6 insertions(+), 6 deletions(-)
pgsql: Doc: update documentation about EXCLUDE constraint elements.
Doc: update documentation about EXCLUDE constraint elements. What the documentation calls an exclude_element is an index_elem according to gram.y, and it allows all the same options that a CREATE INDEX column specification does. The COLLATE patch neglected to update the CREATE/ALTER TABLE docs about that, and later the opclass-parameters patch made the same oversight. Add those options to the syntax synopses, and polish the associated text a bit. Back-patch to v13 where opclass parameters came in. We could update v12 with just the COLLATE omission, but it doesn't quite seem worth the trouble at this point. Shihao Zhong, reviewed by Daniel Vérité, Shubham Khanna and myself Discussion: https://postgr.es/m/CAGRkXqShbVyB8E3gapfdtuwiWTiK=Q67Qb9qwxu=+-w0w46...@mail.gmail.com Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/e2663a63c3b9459651f9370a971966d4fcfe0332 Modified Files -- doc/src/sgml/ref/alter_table.sgml | 2 +- doc/src/sgml/ref/create_table.sgml | 10 +- 2 files changed, 6 insertions(+), 6 deletions(-)
pgsql: Doc: update documentation about EXCLUDE constraint elements.
Doc: update documentation about EXCLUDE constraint elements. What the documentation calls an exclude_element is an index_elem according to gram.y, and it allows all the same options that a CREATE INDEX column specification does. The COLLATE patch neglected to update the CREATE/ALTER TABLE docs about that, and later the opclass-parameters patch made the same oversight. Add those options to the syntax synopses, and polish the associated text a bit. Back-patch to v13 where opclass parameters came in. We could update v12 with just the COLLATE omission, but it doesn't quite seem worth the trouble at this point. Shihao Zhong, reviewed by Daniel Vérité, Shubham Khanna and myself Discussion: https://postgr.es/m/CAGRkXqShbVyB8E3gapfdtuwiWTiK=Q67Qb9qwxu=+-w0w46...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/1973d9fb318316df16b0da89e6fe5f2c53473b38 Modified Files -- doc/src/sgml/ref/alter_table.sgml | 2 +- doc/src/sgml/ref/create_table.sgml | 10 +- 2 files changed, 6 insertions(+), 6 deletions(-)
pgsql: Doc: update documentation about EXCLUDE constraint elements.
Doc: update documentation about EXCLUDE constraint elements. What the documentation calls an exclude_element is an index_elem according to gram.y, and it allows all the same options that a CREATE INDEX column specification does. The COLLATE patch neglected to update the CREATE/ALTER TABLE docs about that, and later the opclass-parameters patch made the same oversight. Add those options to the syntax synopses, and polish the associated text a bit. Back-patch to v13 where opclass parameters came in. We could update v12 with just the COLLATE omission, but it doesn't quite seem worth the trouble at this point. Shihao Zhong, reviewed by Daniel Vérité, Shubham Khanna and myself Discussion: https://postgr.es/m/CAGRkXqShbVyB8E3gapfdtuwiWTiK=Q67Qb9qwxu=+-w0w46...@mail.gmail.com Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/7a0541301078bdd0fb548141c43b335fbabcaf81 Modified Files -- doc/src/sgml/ref/alter_table.sgml | 2 +- doc/src/sgml/ref/create_table.sgml | 10 +- 2 files changed, 6 insertions(+), 6 deletions(-)
pgsql: Doc: update documentation about EXCLUDE constraint elements.
Doc: update documentation about EXCLUDE constraint elements. What the documentation calls an exclude_element is an index_elem according to gram.y, and it allows all the same options that a CREATE INDEX column specification does. The COLLATE patch neglected to update the CREATE/ALTER TABLE docs about that, and later the opclass-parameters patch made the same oversight. Add those options to the syntax synopses, and polish the associated text a bit. Back-patch to v13 where opclass parameters came in. We could update v12 with just the COLLATE omission, but it doesn't quite seem worth the trouble at this point. Shihao Zhong, reviewed by Daniel Vérité, Shubham Khanna and myself Discussion: https://postgr.es/m/CAGRkXqShbVyB8E3gapfdtuwiWTiK=Q67Qb9qwxu=+-w0w46...@mail.gmail.com Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/a690920015bddc9c20f93d1a828c53ac3c21e489 Modified Files -- doc/src/sgml/ref/alter_table.sgml | 2 +- doc/src/sgml/ref/create_table.sgml | 10 +- 2 files changed, 6 insertions(+), 6 deletions(-)
pgsql: Use conditional variable to wait for next MultiXact offset
Use conditional variable to wait for next MultiXact offset In one multixact.c edge case, we need a mechanism to wait for one multixact offset to be written before being allowed to read the next one. We used to handle this case by sleeping for one millisecond and retrying, but such sleeps have been reported as problematic in production cases. We can avoid the problem by using a condition variable: readers sleep on it and then every creator of multixacts broadcasts into the CV when creation is sufficiently far along. Author: Kyotaro Horiguchi Reviewed-by: Andrey Borodin Discussion: https://postgr.es/m/47a598f4-b4e7-4029-8fec-a06a6c3cb...@yandex-team.ru Discussion: https://postgr.es/m/20200515.090333.24867479329066911.horikyota.ntt Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/a0e0fb1ba56ff4323542c75ce997a41d31ad28a0 Modified Files -- src/backend/access/transam/multixact.c | 30 +++-- src/backend/utils/activity/wait_event_names.txt | 1 + 2 files changed, 29 insertions(+), 2 deletions(-)
pgsql: Avoid extra lookups with nbtree array inequalities.
Avoid extra lookups with nbtree array inequalities. nbtree index scans with SAOP inequalities (but no SAOP equalities) performed extra ORDER proc lookups for any remaining equality strategy scan keys. This could waste cycles, and caused assertion failures. Keeping around a separate ORDER proc is only necessary for a scan's non-array/non-SAOP equality scan keys when the scan has at least one other SAOP equality strategy key (a SAOP inequality shouldn't count). To fix, replace _bt_preprocess_array_keys_final's assertion with a test that makes the function return early when the scan has no SAOP equality scan keys. Oversight in commit 1b134ca5, which enhanced nbtree ScalarArrayOp execution. Reported-By: Alexander Lakhin Discussion: https://postgr.es/m/0539d3d3-a402-0a49-ed5e-26429dffc...@gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/473411fc51157e8e825ee865c2822f976e0da5e3 Modified Files -- src/backend/access/nbtree/nbtutils.c | 8 +++- 1 file changed, 7 insertions(+), 1 deletion(-)
Re: pgsql: Use bump memory context for tuplesorts
Melanie Plageman writes:
> Looks like this assert is tripping on grison [1].
> running bootstrap script ... TRAP: failed Assert("total_allocated ==
> context->mem_allocated"), File: "bump.c", Line: 808, PID: 30248
The same on mamba. However, I failed to duplicate it in an x86
(32-bit) VM. I think the critical factor on the failing machines
is that MAXALIGN is 8 which is more than their sizeof(pointer).
regards, tom lane
pgsql: Improve check in LDAP test to find the OpenLDAP installation
Improve check in LDAP test to find the OpenLDAP installation If the OpenLDAP installation directory is not found, set $setup to 0 so that the LDAP tests are skipped. The macOS checks were already doing that, but the checks on other OS's were not. While we're at it, improve the error message when the tests are skipped, to specify whether the OS is supported at all, or if we just didn't find the installation directory. This was accidentally "working" without this, i.e. we were skipping the tests if the OpenLDAP installation was not found, because of a bug in the LdapServer test module: the END block clobbered the exit code so if the script die()s before running the first subtest, the whole test script was marked as SKIPped. The next commit will fix that bug, but we need to fix the setup code first. These checks should probably go into configure/meson, but this is better than nothing and allows fixing the bug in the END block. Backpatch to all supported versions. Discussion: https://www.postgresql.org/message-id/fb898a70-3a88-4629-88e9-f23750200...@iki.fi Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/1843a27efb56eee2f679a246c47bd56ad891592c Modified Files -- src/test/ldap/t/001_auth.pl | 84 ++--- 1 file changed, 56 insertions(+), 28 deletions(-)
pgsql: Improve check in LDAP test to find the OpenLDAP installation
Improve check in LDAP test to find the OpenLDAP installation If the OpenLDAP installation directory is not found, set $setup to 0 so that the LDAP tests are skipped. The macOS checks were already doing that, but the checks on other OS's were not. While we're at it, improve the error message when the tests are skipped, to specify whether the OS is supported at all, or if we just didn't find the installation directory. This was accidentally "working" without this, i.e. we were skipping the tests if the OpenLDAP installation was not found, because of a bug in the LdapServer test module: the END block clobbered the exit code so if the script die()s before running the first subtest, the whole test script was marked as SKIPped. The next commit will fix that bug, but we need to fix the setup code first. These checks should probably go into configure/meson, but this is better than nothing and allows fixing the bug in the END block. Backpatch to all supported versions. Discussion: https://www.postgresql.org/message-id/fb898a70-3a88-4629-88e9-f23750200...@iki.fi Branch -- REL_13_STABLE Details --- https://git.postgresql.org/pg/commitdiff/9e0493445a575495b901bd8ab0d9c976ee6e4e62 Modified Files -- src/test/ldap/t/001_auth.pl | 84 ++--- 1 file changed, 56 insertions(+), 28 deletions(-)
pgsql: Improve check in LDAP test to find the OpenLDAP installation
Improve check in LDAP test to find the OpenLDAP installation If the OpenLDAP installation directory is not found, set $setup to 0 so that the LDAP tests are skipped. The macOS checks were already doing that, but the checks on other OS's were not. While we're at it, improve the error message when the tests are skipped, to specify whether the OS is supported at all, or if we just didn't find the installation directory. This was accidentally "working" without this, i.e. we were skipping the tests if the OpenLDAP installation was not found, because of a bug in the LdapServer test module: the END block clobbered the exit code so if the script die()s before running the first subtest, the whole test script was marked as SKIPped. The next commit will fix that bug, but we need to fix the setup code first. These checks should probably go into configure/meson, but this is better than nothing and allows fixing the bug in the END block. Backpatch to all supported versions. Discussion: https://www.postgresql.org/message-id/fb898a70-3a88-4629-88e9-f23750200...@iki.fi Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/854dd250ee178aca201e73923c99aac09b25ea9c Modified Files -- src/test/ldap/LdapServer.pm | 91 --- src/test/ldap/t/001_auth.pl | 3 +- src/test/ldap/t/002_bindpasswd.pl | 3 +- 3 files changed, 67 insertions(+), 30 deletions(-)
pgsql: Don't clobber test exit code at cleanup in LDAP/Kerberors tests
Don't clobber test exit code at cleanup in LDAP/Kerberors tests If the test script die()d before running the first test, the whole test was interpreted as SKIPped rather than failed. The PostgreSQL::Cluster module got this right. Backpatch to all supported versions. Discussion: https://www.postgresql.org/message-id/fb898a70-3a88-4629-88e9-f23750200...@iki.fi Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/3d5a9bb8df65ac6446ae5ad769c0adc5f332c164 Modified Files -- src/test/kerberos/t/001_auth.pl | 7 ++- src/test/ldap/t/001_auth.pl | 5 + 2 files changed, 11 insertions(+), 1 deletion(-)
pgsql: Improve check in LDAP test to find the OpenLDAP installation
Improve check in LDAP test to find the OpenLDAP installation If the OpenLDAP installation directory is not found, set $setup to 0 so that the LDAP tests are skipped. The macOS checks were already doing that, but the checks on other OS's were not. While we're at it, improve the error message when the tests are skipped, to specify whether the OS is supported at all, or if we just didn't find the installation directory. This was accidentally "working" without this, i.e. we were skipping the tests if the OpenLDAP installation was not found, because of a bug in the LdapServer test module: the END block clobbered the exit code so if the script die()s before running the first subtest, the whole test script was marked as SKIPped. The next commit will fix that bug, but we need to fix the setup code first. These checks should probably go into configure/meson, but this is better than nothing and allows fixing the bug in the END block. Backpatch to all supported versions. Discussion: https://www.postgresql.org/message-id/fb898a70-3a88-4629-88e9-f23750200...@iki.fi Branch -- REL_12_STABLE Details --- https://git.postgresql.org/pg/commitdiff/1782571f6537c1d1a3c65fbf47a1be445eb606e3 Modified Files -- src/test/ldap/t/001_auth.pl | 84 ++--- 1 file changed, 56 insertions(+), 28 deletions(-)
pgsql: Don't clobber test exit code at cleanup in LDAP/Kerberors tests
Don't clobber test exit code at cleanup in LDAP/Kerberors tests If the test script die()d before running the first test, the whole test was interpreted as SKIPped rather than failed. The PostgreSQL::Cluster module got this right. Backpatch to all supported versions. Discussion: https://www.postgresql.org/message-id/fb898a70-3a88-4629-88e9-f23750200...@iki.fi Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/a9c20c85c73a41d3736f142743fa1a01df1e04b6 Modified Files -- src/test/kerberos/t/001_auth.pl | 7 ++- src/test/ldap/t/001_auth.pl | 5 + 2 files changed, 11 insertions(+), 1 deletion(-)
pgsql: Don't clobber test exit code at cleanup in LDAP/Kerberors tests
Don't clobber test exit code at cleanup in LDAP/Kerberors tests If the test script die()d before running the first test, the whole test was interpreted as SKIPped rather than failed. The PostgreSQL::Cluster module got this right. Backpatch to all supported versions. Discussion: https://www.postgresql.org/message-id/fb898a70-3a88-4629-88e9-f23750200...@iki.fi Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/dfe402f955be8e7676e4da86517b8cdf75a072a2 Modified Files -- src/test/kerberos/t/001_auth.pl | 7 ++- src/test/ldap/LdapServer.pm | 5 + 2 files changed, 11 insertions(+), 1 deletion(-)
pgsql: Don't clobber test exit code at cleanup in LDAP/Kerberors tests
Don't clobber test exit code at cleanup in LDAP/Kerberors tests If the test script die()d before running the first test, the whole test was interpreted as SKIPped rather than failed. The PostgreSQL::Cluster module got this right. Backpatch to all supported versions. Discussion: https://www.postgresql.org/message-id/fb898a70-3a88-4629-88e9-f23750200...@iki.fi Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/a475a2fa3bc9c68eaa2aeda28d6d86529eda4673 Modified Files -- src/test/kerberos/t/001_auth.pl | 5 + src/test/ldap/LdapServer.pm | 5 + 2 files changed, 10 insertions(+)
pgsql: Don't clobber test exit code at cleanup in LDAP/Kerberors tests
Don't clobber test exit code at cleanup in LDAP/Kerberors tests If the test script die()d before running the first test, the whole test was interpreted as SKIPped rather than failed. The PostgreSQL::Cluster module got this right. Backpatch to all supported versions. Discussion: https://www.postgresql.org/message-id/fb898a70-3a88-4629-88e9-f23750200...@iki.fi Branch -- REL_13_STABLE Details --- https://git.postgresql.org/pg/commitdiff/f700e7d5710d04940df2d4e0d04080b1899a8f07 Modified Files -- src/test/kerberos/t/001_auth.pl | 7 ++- src/test/ldap/t/001_auth.pl | 5 + 2 files changed, 11 insertions(+), 1 deletion(-)
pgsql: Improve check in LDAP test to find the OpenLDAP installation
Improve check in LDAP test to find the OpenLDAP installation If the OpenLDAP installation directory is not found, set $setup to 0 so that the LDAP tests are skipped. The macOS checks were already doing that, but the checks on other OS's were not. While we're at it, improve the error message when the tests are skipped, to specify whether the OS is supported at all, or if we just didn't find the installation directory. This was accidentally "working" without this, i.e. we were skipping the tests if the OpenLDAP installation was not found, because of a bug in the LdapServer test module: the END block clobbered the exit code so if the script die()s before running the first subtest, the whole test script was marked as SKIPped. The next commit will fix that bug, but we need to fix the setup code first. These checks should probably go into configure/meson, but this is better than nothing and allows fixing the bug in the END block. Backpatch to all supported versions. Discussion: https://www.postgresql.org/message-id/fb898a70-3a88-4629-88e9-f23750200...@iki.fi Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/c8df46b65705c87ab5b76cefe1ff69aed72095bf Modified Files -- src/test/ldap/t/001_auth.pl | 83 +++-- 1 file changed, 58 insertions(+), 25 deletions(-)
pgsql: Improve check in LDAP test to find the OpenLDAP installation
Improve check in LDAP test to find the OpenLDAP installation If the OpenLDAP installation directory is not found, set $setup to 0 so that the LDAP tests are skipped. The macOS checks were already doing that, but the checks on other OS's were not. While we're at it, improve the error message when the tests are skipped, to specify whether the OS is supported at all, or if we just didn't find the installation directory. This was accidentally "working" without this, i.e. we were skipping the tests if the OpenLDAP installation was not found, because of a bug in the LdapServer test module: the END block clobbered the exit code so if the script die()s before running the first subtest, the whole test script was marked as SKIPped. The next commit will fix that bug, but we need to fix the setup code first. These checks should probably go into configure/meson, but this is better than nothing and allows fixing the bug in the END block. Backpatch to all supported versions. Discussion: https://www.postgresql.org/message-id/fb898a70-3a88-4629-88e9-f23750200...@iki.fi Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/e13b586d7c989147afd910c6997659a8412a8b84 Modified Files -- src/test/ldap/LdapServer.pm | 91 --- src/test/ldap/t/001_auth.pl | 3 +- src/test/ldap/t/002_bindpasswd.pl | 3 +- 3 files changed, 67 insertions(+), 30 deletions(-)
pgsql: Don't clobber test exit code at cleanup in LDAP/Kerberors tests
Don't clobber test exit code at cleanup in LDAP/Kerberors tests If the test script die()d before running the first test, the whole test was interpreted as SKIPped rather than failed. The PostgreSQL::Cluster module got this right. Backpatch to all supported versions. Discussion: https://www.postgresql.org/message-id/fb898a70-3a88-4629-88e9-f23750200...@iki.fi Branch -- REL_12_STABLE Details --- https://git.postgresql.org/pg/commitdiff/0d95711ae0d3b328b0a3534a546a4f848d273e08 Modified Files -- src/test/kerberos/t/001_auth.pl | 7 ++- src/test/ldap/t/001_auth.pl | 5 + 2 files changed, 11 insertions(+), 1 deletion(-)
Re: pgsql: Reduce branches in heapgetpage()'s per-tuple loop
Hi, On 2024-04-07 09:14:57 -0400, Melanie Plageman wrote: > On Sun, Apr 7, 2024 at 3:29 AM Andres Freund wrote: > > > > Reduce branches in heapgetpage()'s per-tuple loop > > > > Until now, heapgetpage()'s loop over all tuples performed some conditional > > checks for each tuple, even though condition did not change across the loop. > > I haven't reviewed exactly what this does, but heapgetpage() was > removed (or split into two functions, rather) in 44086b097537, so I > find the naming of this function, or, at least the comments, commit > message and description a bit confusing. Oops. That name was so ingrained in my head that despite looking over these lines quite a few times, I couldn't spot that. Will send an update to the thread. Greetings, Andres Freund
Re: pgsql: Use bump memory context for tuplesorts
On Sun, Apr 7, 2024 at 8:32 AM David Rowley wrote:
>
> Use bump memory context for tuplesorts
>
> 29f6a959c added a bump allocator type for efficient compact allocations.
> Here we make use of this for non-bounded tuplesorts to store tuples.
> This is very space efficient when storing narrow tuples due to bump.c
> not having chunk headers. This means we can fit more tuples in work_mem
> before spilling to disk, or perform an in-memory sort touching fewer
> cacheline.
Looks like this assert is tripping on grison [1].
running bootstrap script ... TRAP: failed Assert("total_allocated ==
context->mem_allocated"), File: "bump.c", Line: 808, PID: 30248
[1]
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=grison&dt=2024-04-07%2014%3A10%3A09
pgsql: Use streaming I/O in sequential scans.
Use streaming I/O in sequential scans. Instead of calling ReadBuffer() for each block, heap sequential scans and TID range scans now use the streaming API introduced in b5a9b18cd0. Author: Melanie Plageman Reviewed-by: Andres Freund Reviewed-by: Thomas Munro Discussion: https://postgr.es/m/flat/CAAKRu_YtXJiYKQvb5JsA2SkwrsizYLugs4sSOZh3EAjKUg%3DgEQ%40mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/b7b0f3f27241e424b7103397489464d907cef2c4 Modified Files -- src/backend/access/heap/heapam.c | 235 +++ src/include/access/heapam.h | 15 +++ 2 files changed, 177 insertions(+), 73 deletions(-)
Re: pgsql: Reduce branches in heapgetpage()'s per-tuple loop
On Sun, Apr 7, 2024 at 3:29 AM Andres Freund wrote: > > Reduce branches in heapgetpage()'s per-tuple loop > > Until now, heapgetpage()'s loop over all tuples performed some conditional > checks for each tuple, even though condition did not change across the loop. I haven't reviewed exactly what this does, but heapgetpage() was removed (or split into two functions, rather) in 44086b097537, so I find the naming of this function, or, at least the comments, commit message and description a bit confusing. - Melanie
pgsql: Use bump memory context for tuplesorts
Use bump memory context for tuplesorts 29f6a959c added a bump allocator type for efficient compact allocations. Here we make use of this for non-bounded tuplesorts to store tuples. This is very space efficient when storing narrow tuples due to bump.c not having chunk headers. This means we can fit more tuples in work_mem before spilling to disk, or perform an in-memory sort touching fewer cacheline. Author: David Rowley Reviewed-by: Nathan Bossart Reviewed-by: Matthias van de Meent Reviewed-by: Tomas Vondra Reviewed-by: John Naylor Discussion: https://postgr.es/m/CAApHDvqGSpCU95TmM=Bp=6xjl_nlys4zdzopfnywbk97xrd...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/6ed83d5fa55cf6e6c9d1be34ec10730c48eba763 Modified Files -- src/backend/utils/sort/tuplesort.c | 52 +- src/backend/utils/sort/tuplesortvariants.c | 38 +++--- src/include/utils/tuplesort.h | 21 +--- 3 files changed, 78 insertions(+), 33 deletions(-)
pgsql: Add XLogCtl->logInsertResult
Add XLogCtl->logInsertResult This tracks the position of WAL that's been fully copied into WAL buffers by all processes emitting WAL. (For some reason we call that "WAL insertion"). This is updated using atomic monotonic advance during WaitXLogInsertionsToFinish, which is not when the insertions actually occur, but it's the only place where we know where have all the insertions have completed. This value is useful in WALReadFromBuffers, which can verify that callers don't try to read past what has been inserted. (However, more infrastructure is needed in order to actually use WAL after the flush point, since it could be lost.) The value is also useful in WaitXLogInsertionsToFinish() itself, since we can now exit quickly when all WAL has been already inserted, without even having to take any locks. Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/f3ff7bf83bce11709add9a6d31d4bebe95e086e3 Modified Files -- src/backend/access/transam/xlog.c | 40 ++- src/include/port/atomics.h| 36 +++ 2 files changed, 75 insertions(+), 1 deletion(-)
pgsql: Introduce a bump memory allocator
Introduce a bump memory allocator This introduces a bump MemoryContext type. The bump context is best suited for short-lived memory contexts which require only allocations of memory and never a pfree or repalloc, which are unsupported. Memory palloc'd into a bump context has no chunk header. This makes bump a useful context type when lots of small allocations need to be done without any need to pfree those allocations. Allocation sizes are rounded up to the next MAXALIGN boundary, so with this and no chunk header, allocations are very compact indeed. Allocations are also very fast as bump does not check any freelists to try and make use of previously free'd chunks. It just checks if there is enough room on the current block, and if so it bumps the freeptr beyond this chunk and returns the value that the freeptr was previously pointing to. Simple and fast. A new block is malloc'd when there's not enough space in the current block. Code using the bump allocator must take care never to call any functions which could try to call realloc() (or variants), pfree(), GetMemoryChunkContext() or GetMemoryChunkSpace() on a bump allocated chunk. Due to lack of chunk headers, these operations are unsupported. To increase the chances of catching such issues, when compiled with MEMORY_CONTEXT_CHECKING, bump allocated chunks are given a header and any attempt to perform an unsupported operation will result in an ERROR. Without MEMORY_CONTEXT_CHECKING, code attempting an unsupported operation could result in a segfault. A follow-on commit will implement the first user of bump. Author: David Rowley Reviewed-by: Nathan Bossart Reviewed-by: Matthias van de Meent Reviewed-by: Tomas Vondra Reviewed-by: John Naylor Discussion: https://postgr.es/m/CAApHDvqGSpCU95TmM=Bp=6xjl_nlys4zdzopfnywbk97xrd...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/29f6a959cfd8ffa7d6db2c0629439c5329e2853e Modified Files -- src/backend/nodes/gen_node_support.pl | 2 +- src/backend/utils/mmgr/Makefile | 1 + src/backend/utils/mmgr/bump.c | 811 ++ src/backend/utils/mmgr/mcxt.c | 15 +- src/backend/utils/mmgr/meson.build| 1 + src/include/nodes/memnodes.h | 3 +- src/include/utils/memutils.h | 7 + src/include/utils/memutils_internal.h | 18 +- src/tools/pgindent/typedefs.list | 2 + 9 files changed, 856 insertions(+), 4 deletions(-)
pgsql: Enlarge bit-space for MemoryContextMethodID
Enlarge bit-space for MemoryContextMethodID Reserve 4 bits for MemoryContextMethodID rather than 3. 3 bits did technically allow a maximum of 8 memory context types, however, we've opted to reserve some bit patterns which left us with only 4 slots, all of which were used. Here we add another bit which frees up 8 slots for future memory context types. In passing, adjust the enum names in MemoryContextMethodID to make it more clear which ones can be used and which ones are reserved. Author: Matthias van de Meent, David Rowley Discussion: https://postgr.es/m/CAApHDvqGSpCU95TmM=Bp=6xjl_nlys4zdzopfnywbk97xrd...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/0ba8b75e7ea6b7b3090c81239ebcb866772a624b Modified Files -- src/backend/utils/mmgr/README| 21 ++-- src/backend/utils/mmgr/mcxt.c| 43 src/include/utils/memutils_internal.h| 18 + src/include/utils/memutils_memorychunk.h | 32 ++-- 4 files changed, 72 insertions(+), 42 deletions(-)
pgsql: Avoid needless large memcpys in libpq socket writing
Avoid needless large memcpys in libpq socket writing Until now, when calling pq_putmessage to write new data to a libpq socket, all writes are copied into a buffer and that buffer gets flushed when full to avoid having to perform small writes to the socket. There are cases where we must write large amounts of data to the socket, sometimes larger than the size of the buffer. In this case, it's wasteful to memcpy this data into the buffer and flush it out, instead, we can send it directly from the memory location that the data is already stored in. Here we adjust internal_putbytes() so that after having just flushed the buffer to the socket, if the remaining bytes to send is as big or bigger than the buffer size, we just send directly rather than needlessly copying into the PqSendBuffer buffer first. Examples of operations that write large amounts of data in one message are; outputting large tuples with SELECT or COPY TO STDOUT and pg_basebackup. Author: Melih Mutlu Reviewed-by: Heikki Linnakangas Reviewed-by: Jelte Fennema-Nio Reviewed-by: David Rowley Reviewed-by: Ranier Vilela Reviewed-by: Andres Freund Discussion: https://postgr.es/m/cagpvpcr15nosj0f6xe-c2h477zfr88q12e6wjeoezc8zykt...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/c4ab7da60617f020e8d75b1584d0754005d71830 Modified Files -- src/backend/libpq/pqcomm.c | 74 +- 1 file changed, 53 insertions(+), 21 deletions(-)
pgsql: Reduce branches in heapgetpage()'s per-tuple loop
Reduce branches in heapgetpage()'s per-tuple loop Until now, heapgetpage()'s loop over all tuples performed some conditional checks for each tuple, even though condition did not change across the loop. This commit fixes that by moving the loop into an inline function. By calling it with different constant arguments, the compiler can generate an optimized loop for the different conditions, at the price of two per-page checks. For cases of all-visible tables and an isolation level other than serializable, speedups of up to 25% have been measured. Reviewed-by: John Naylor Reviewed-by: Zhang Mingli Tested-by: Quan Zongliang Discussion: https://postgr.es/m/20230716015656.xjvemfbp5fysj...@awork3.anarazel.de Discussion: https://postgr.es/m/2ef7ff1b-3d18-2283-61b1-bbd25fc6c...@yeah.net Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/a97bbe1f1df9eba0b18207c321c67de80b33db16 Modified Files -- src/backend/access/heap/heapam.c | 102 --- 1 file changed, 74 insertions(+), 28 deletions(-)
