Re: Removing support for COPY FROM STDIN in protocol version 2

2021-02-04 Thread Heikki Linnakangas
On 04/02/2021 17:35, Tom Lane wrote: Alvaro Herrera writes: Yeah, the changes I was thinking about are all in libpq-int.h so that's not really a problem. But one enum in libpq-fe.h renumbers values, and I think it's better to keep the old value labelled as "unused" to avoid any changes. Oh,

Re: Removing support for COPY FROM STDIN in protocol version 2

2021-02-03 Thread Heikki Linnakangas
On 03/02/2021 18:29, Tom Lane wrote: Alvaro Herrera writes: On 2021-Feb-03, Tom Lane wrote: I have a vague recollection that JDBC users still like to use protocol 2 for some reason --- is that out of date? [ yes, since 2016 ] Then let's kill it dead, server and libpq both. Ok, works

Removing support for COPY FROM STDIN in protocol version 2

2021-02-03 Thread Heikki Linnakangas
for the old protocol altogether, but I'm not proposing that we go that far just yet. [1] https://www.postgresql.org/message-id/e7861509-3960-538a-9025-b75a61188e01%40iki.fi - Heikki >From 21ce10dd2bf43c6924645bff4d40d18663835dcf Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Wed, 3

Re: Bug in COPY FROM backslash escaping multi-byte chars

2021-02-03 Thread Heikki Linnakangas
On 03/02/2021 15:38, John Naylor wrote: On Wed, Feb 3, 2021 at 8:08 AM Heikki Linnakangas <mailto:hlinn...@iki.fi>> wrote: > > Hi, > > While playing with COPY FROM refactorings in another thread, I noticed > corner case where I think backslash escaping doesn't wor

Bug in COPY FROM backslash escaping multi-byte chars

2021-02-03 Thread Heikki Linnakangas
raightforward patch to fix that. Anyone see a problem with this? - Heikki >From 632549e79dc98cb338d7fc22888d8b6237a47d10 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Wed, 3 Feb 2021 14:00:32 +0200 Subject: [PATCH 1/1] Fix a corner-case in COPY FROM backslash processing. If a mult

Re: Improve new hash partition bound check error messages

2021-02-02 Thread Heikki Linnakangas
On 02/02/2021 12:35, Peter Eisentraut wrote: I had a bit of trouble parsing the error message "every hash partition modulus must be a factor of the next larger modulus", so I went into the code, added some comments and added errdetail messages for each case. I think it's a bit clearer now.

Re: [POC] verifying UTF-8 using SIMD instructions

2021-02-01 Thread Heikki Linnakangas
On 01/02/2021 19:32, John Naylor wrote: It makes sense to start with the ascii subset of UTF-8 for a couple reasons. First, ascii is very widespread in database content, particularly in bulk loads. Second, ascii can be validated using the simple SSE2 intrinsics that come with (I believe) any

Re: Perform COPY FROM encoding conversions in larger chunks

2021-01-28 Thread Heikki Linnakangas
On 28/01/2021 01:23, John Naylor wrote: Hi Heikki, 0001 through 0003 are straightforward, and I think they can be committed now if you like. 0004 is also pretty straightforward. The check you proposed upthread for pg_upgrade seems like the best solution to make that workable. I'll take a

Re: Perform COPY FROM encoding conversions in larger chunks

2021-01-28 Thread Heikki Linnakangas
On 28/01/2021 01:23, John Naylor wrote: Hi Heikki, 0001 through 0003 are straightforward, and I think they can be committed now if you like. Thanks for the review! I did some more rigorous microbenchmarking of patch 1 and 2. I used the attached test script, which calls convert_from()

Re: Online checksums patch - once again

2021-01-27 Thread Heikki Linnakangas
Revisiting an issue we discussed earlier: On 25/11/2020 15:20, Daniel Gustafsson wrote: On 23 Nov 2020, at 18:36, Heikki Linnakangas wrote: On 17/11/2020 10:56, Daniel Gustafsson wrote: I've reworked this in the attached such that the enable_ and disable_ functions merely call

Re: Protect syscache from bloating with negative cache entries

2021-01-27 Thread Heikki Linnakangas
001 From: Heikki Linnakangas Date: Wed, 27 Jan 2021 13:08:08 +0200 Subject: [PATCH v8 1/1] CatCache expiration feature Author: Kyotaro Horiguchi --- src/backend/access/transam/xact.c | 3 ++ src/backend/utils/cache/catcache.c | 82 +- src/backend/utils/misc/gu

Re: Online checksums patch - once again

2021-01-26 Thread Heikki Linnakangas
On 22/01/2021 14:21, Heikki Linnakangas wrote: On 22/01/2021 13:55, Heikki Linnakangas wrote: I read through the latest patch, v31-0001-Support-checksum-enable-disable-in-a-running-clu.patch. Some comments below: One more thing: In SetRelationNumChecks(), you should use SearchSysCacheCopy1

Re: automatic analyze: readahead - add "IO read time" log message

2021-01-26 Thread Heikki Linnakangas
On 13/01/2021 23:17, Stephen Frost wrote: Would be great to get a review / comments from others as to if there's any concerns. I'll admit that it seems reasonably straight-forward to me, but hey, I wrote most of it, so that's not really a fair assessment... ;) Look good overall. A few minor

Re: Protect syscache from bloating with negative cache entries

2021-01-26 Thread Heikki Linnakangas
Hi, On 19/11/2020 07:25, Kyotaro Horiguchi wrote: Performance measurement on the attached showed better result about searching but maybe worse for cache entry creation. Each time number is the mean of 10 runs. # Cacache (negative) entry creation : time(ms) (% to master) master

Re: shared tempfile was not removed on statement_timeout

2021-01-26 Thread Heikki Linnakangas
On 26/01/2021 06:46, Kyotaro Horiguchi wrote: Looking the comment of SharedFileSetOnDetach: | * everything in them. We can't raise an error on failures, because this runs | * in error cleanup paths. I feel that a function that shouldn't error-out also shouldn't be cancellable. If that's the

Re: cleaning up a few CLOG-related things

2021-01-25 Thread Heikki Linnakangas
On 25/01/2021 18:56, Robert Haas wrote: I attach a series of proposed patches to slightly improve some minor things related to the CLOG code. [patches 0001 - 0003] Makes sense. 0004 - In StartupCLOG(), correct an off-by-one error. Currently, if the nextXid is exactly a multiple of the

Snapbuild woes followup

2021-01-25 Thread Heikki Linnakangas
In SnapBuildFinalSnapshot(), we have this comment: /* * c) transition from START to BUILDING_SNAPSHOT. * * In START state, and a xl_running_xacts record with running xacts is * encountered. In that case, switch to BUILDING_SNAPSHOT state, and

Re: Polyphase merge is obsolete

2021-01-22 Thread Heikki Linnakangas
On 22/10/2020 14:48, Heikki Linnakangas wrote: On 11/09/2017 13:37, Tomas Vondra wrote: I planned to do some benchmarking on this patch, but apparently the patch no longer applies. Rebase please? Here's a rebase of this. Sorry to keep you waiting :-). Here's an updated version that fixes

Re: Some more hackery around cryptohashes (some fixes + SHA1)

2021-01-22 Thread Heikki Linnakangas
On 07/01/2021 05:41, Michael Paquier wrote: On Mon, Dec 14, 2020 at 12:48:15PM +0900, Michael Paquier wrote: This is a nice cleanup, so I have moved ahead and applied it. A rebased version of the SHA1 business is attached. Rebased version attached to address the conflicts caused by 55fe26a.

Re: pg_rewind copies

2021-01-22 Thread Heikki Linnakangas
version, with that fixed and rebased. No other changes. - Heikki >From 649ce2ffb7ef390e96dbde9bd7da27a8a3d330d4 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Fri, 22 Jan 2021 15:16:41 +0200 Subject: [PATCH v2 1/1] pg_rewind: Fetch small files according to new size. There's a r

Re: PoC Refactor AM analyse API

2021-01-22 Thread Heikki Linnakangas
On 30/12/2020 11:12, Denis Smirnov wrote: But why do you pass int natts and VacAttrStats **stats to acquire_sample_rows()? Is it of any use? It seems to break abstraction too. Yes, it is really a kluge that should be discussed. The main problem is that we don’t pass projection information to

Re: Online checksums patch - once again

2021-01-22 Thread Heikki Linnakangas
On 22/01/2021 13:55, Heikki Linnakangas wrote: I read through the latest patch, v31-0001-Support-checksum-enable-disable-in-a-running-clu.patch. Some comments below: One more thing: In SetRelationNumChecks(), you should use SearchSysCacheCopy1() to get a modifiable copy of the tuple

Re: Online checksums patch - once again

2021-01-22 Thread Heikki Linnakangas
I read through the latest patch, v31-0001-Support-checksum-enable-disable-in-a-running-clu.patch. Some comments below: On 19/01/2021 14:32, Daniel Gustafsson wrote: + /* +* Hold interrupts for the duration of xlogging to avoid the state of data +* checksums changing

Re: Additional Chapter for Tutorial - arch-dev.sgml

2021-01-22 Thread Heikki Linnakangas
On 21/01/2021 14:38, Jürgen Purtz wrote: This supervisor process is called postmaster and listens at a specified TCP/IP port for incoming connections. Whenever he detects a request for a connection, he spawns a new backend process. It sounds weird to refer to a process with "he". I left out

Re: Avoiding smgrimmedsync() during nbtree index builds

2021-01-21 Thread Heikki Linnakangas
On 21/01/2021 22:36, Andres Freund wrote: Hi, Every nbtree index build currently does an smgrimmedsync at the end: /* * Read tuples in correct sort order from tuplesort, and load them into * btree leaves. */ static void _bt_load(BTWriteState *wstate, BTSpool *btspool, BTSpool *btspool2)

Re: ResourceOwner refactoring

2021-01-21 Thread Heikki Linnakangas
On 21/01/2021 06:14, Michael Paquier wrote: On Thu, Jan 21, 2021 at 12:11:37AM +0200, Heikki Linnakangas wrote: Summary: In the the worst scenario, the patched version is still 24% slower than unpatched. But many other scenarios are now faster with the patch. Is there a reason explaining

Re: ResourceOwner refactoring

2021-01-20 Thread Heikki Linnakangas
On 19/01/2021 11:09, Heikki Linnakangas wrote: On 18/01/2021 18:11, Robert Haas wrote: On Mon, Jan 18, 2021 at 11:11 AM Robert Haas wrote: On Mon, Jan 18, 2021 at 10:19 AM Heikki Linnakangas wrote: On 18/01/2021 16:34, Alvaro Herrera wrote: So according to your performance benchmark, we're

Re: ResourceOwner refactoring

2021-01-19 Thread Heikki Linnakangas
On 18/01/2021 16:34, Alvaro Herrera wrote: On 2021-Jan-18, Heikki Linnakangas wrote: +static ResourceOwnerFuncs jit_funcs = +{ + /* relcache references */ + .name = "LLVM JIT context", + .phase = RESOURCE_RELEASE_BEFORE_LOCKS, + .Relea

Re: ResourceOwner refactoring

2021-01-19 Thread Heikki Linnakangas
On 18/01/2021 18:11, Robert Haas wrote: On Mon, Jan 18, 2021 at 11:11 AM Robert Haas wrote: On Mon, Jan 18, 2021 at 10:19 AM Heikki Linnakangas wrote: On 18/01/2021 16:34, Alvaro Herrera wrote: So according to your performance benchmark, we're willing to accept a 30% performance loss

Re: ResourceOwner refactoring

2021-01-18 Thread Heikki Linnakangas
On 18/01/2021 16:34, Alvaro Herrera wrote: So according to your performance benchmark, we're willing to accept a 30% performance loss on an allegedly common operation -- numkeep=0 numsnaps=10 becomes 49.8ns from 37.6ns. That seems a bit shocking. Maybe you can claim that these operations aren't

Re: Additional Chapter for Tutorial - arch-dev.sgml

2021-01-18 Thread Heikki Linnakangas
On 20/11/2020 23:52, Erik Rijkers wrote: (smallish) Changes to arch-dev.sgml This looks good to me. One little complaint: @@ -125,7 +122,7 @@ use a supervisor process (also master process) that spawns a new server process every time a connection is requested. This supervisor -

Re: Yet another fast GiST build

2021-01-18 Thread Heikki Linnakangas
On 18/01/2021 01:10, Peter Geoghegan wrote: On Sun, Jan 17, 2021 at 3:04 PM Peter Geoghegan wrote: I personally agree with you - it's not like there aren't other ways for superusers to crash the server (most of which seem very similar to this gist_page_items() issue, in fact). I just think

Re: ResourceOwner refactoring

2021-01-18 Thread Heikki Linnakangas
nt changes. And a third patch containing the source for the performance test. - Heikki >From ca5ef92d30d6d2d77e6758da3ae60d075451d5c1 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Wed, 13 Jan 2021 12:21:28 +0200 Subject: [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() cal

Re: Yet another fast GiST build

2021-01-17 Thread Heikki Linnakangas
On 18/01/2021 00:35, Peter Geoghegan wrote: On Sun, Jan 17, 2021 at 12:50 PM Tom Lane wrote: I noticed that gist_page_items() thinks it can hold inter_call_data->rel open across a series of calls. That's completely unsafe: the executor might not run the call series to completion (see LIMIT),

Re: ResourceOwner refactoring

2021-01-14 Thread Heikki Linnakangas
that the object to remember and forget fits in the array. When running the regression test suite, about 96% of ResourceOwnerForget() calls fit in the array. I think that's acceptable. - Heikki >From 915ef70c8e227fe1c9d403bf6414d54b6673ddae Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Dat

Re: profiling

2021-01-14 Thread Heikki Linnakangas
On 14/01/2021 12:14, Павел Еремин wrote: Hello. I am interested in the question of profiling. As far as I understand, this issue has not been resolved in postgres. And I see the urgency of this problem. Are there any postgres development plans for this functionality? What exactly do you mean

Re: Yet another fast GiST build

2021-01-13 Thread Heikki Linnakangas
On 13 January 2021 13:53:39 EET, Heikki Linnakangas wrote: >Buildfarm animal thorntail is still not happy: > >> --- >> /home/nm/farm/sparc64_deb10_gcc_64_ubsan/HEAD/pgsql.build/../pgsql/contrib/pageinspect/expected/gist.out >> 2021-01-13 13:38:09.721752365 +0300

Re: Yet another fast GiST build

2021-01-13 Thread Heikki Linnakangas
On 13 January 2021 20:04:10 EET, Heikki Linnakangas wrote: > > >On 13 January 2021 13:53:39 EET, Heikki Linnakangas wrote: >>Looks like the LSN on the page is not set to GistBuildLSN as expected. >>Weird. >> >>Thorntail is a sparc64 system, so little-endia

Re: Yet another fast GiST build

2021-01-13 Thread Heikki Linnakangas
On 13/01/2021 12:34, Heikki Linnakangas wrote: On 13/01/2021 11:46, Andrey Borodin wrote: 13 янв. 2021 г., в 13:41, Heikki Linnakangas написал(а): One more question: will bytea tests run correctly on 32bit\different-endian systems? Good question. Somehow I thought we were printing

Re: Yet another fast GiST build

2021-01-13 Thread Heikki Linnakangas
On 13/01/2021 11:46, Andrey Borodin wrote: 13 янв. 2021 г., в 13:41, Heikki Linnakangas написал(а): One more question: will bytea tests run correctly on 32bit\different-endian systems? Good question. Somehow I thought we were printing esseantilly text values as bytea. But they are Points

Re: Yet another fast GiST build

2021-01-13 Thread Heikki Linnakangas
On 12/01/2021 18:19, Andrey Borodin wrote: 12 янв. 2021 г., в 18:49, Heikki Linnakangas написал(а): Fixed the docs accordingly, and ran pgindent. New patch version attached. Thanks! Looks good to me. Pushed, thanks! One more question: will bytea tests run correctly on 32bit\different

Re: ResourceOwner refactoring

2021-01-12 Thread Heikki Linnakangas
/cryptohash_openssl.c: patch does not apply error: patch failed: src/include/utils/resowner_private.h:1 error: src/include/utils/resowner_private.h: patch does not apply Here's a rebased version. Thanks! - Heikki >From 31b1b4661823cf38b2d4c5931f96c477b6441271 Mon Sep 17 00:00:00 2001 From: Hei

Re: Yet another fast GiST build

2021-01-12 Thread Heikki Linnakangas
rom 791e45bcc86b205b372ea0679c8c036fff941f56 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Tue, 12 Jan 2021 15:43:09 +0200 Subject: [PATCH v4 1/1] Add functions to 'pageinspect' to inspect GiST indexes. Author: Andrey Borodin and me Discussion: https://www.postgresql.org/message-id/3E4F9093-A1B5-4DF8-A

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2021-01-06 Thread Heikki Linnakangas
On 07/01/2021 06:15, Michael Paquier wrote: On Wed, Jan 06, 2021 at 03:58:22PM +0200, Heikki Linnakangas wrote: contrib/pgcrypto/internal-sha2.c and src/backend/utils/adt/cryptohashfuncs.c: the call to pg_cryptohash_create() is missing check for NULL result. With your latest patch, that's OK

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2021-01-06 Thread Heikki Linnakangas
On 25/12/2020 02:57, Michael Paquier wrote: On Mon, Dec 21, 2020 at 04:28:26PM -0500, Robert Haas wrote: TBH, I think there's no point in return an error here at all, because it's totally non-specific. You have no idea what failed, just that something failed. Blech. If we want to check that ctx

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2021-01-06 Thread Heikki Linnakangas
On 06/01/2021 13:42, Michael Paquier wrote: On Sat, Dec 19, 2020 at 03:13:50PM +0900, Michael Paquier wrote: At the same time, I have taken care of your comment from upthread to return a failure if the caller passes NULL for the context, and adjusted some comments. What do you think of the

Re: pg_rewind restore_command issue in PG12

2021-01-04 Thread Heikki Linnakangas
On 03/01/2021 20:13, Amine Tengilimoglu wrote:      In a situation where pg_rewind gets an error due to a missing wall, I  have set restore_command so that the needed wals can be read from the archive (I don't want to manually copy the wal files), but I see it doesn't work. What am I missing? 

Re: Deleting older versions in unique indexes to avoid page splits

2021-01-04 Thread Heikki Linnakangas
On 02/12/2020 00:18, Peter Geoghegan wrote: On Tue, Dec 1, 2020 at 1:50 AM Heikki Linnakangas wrote: On 30/11/2020 21:50, Peter Geoghegan wrote: +} TM_IndexDelete; +} TM_IndexStatus; Is it really significantly faster to have two arrays? If you had just one array, each element would

Re: Perform COPY FROM encoding conversions in larger chunks

2020-12-22 Thread Heikki Linnakangas
On 22/12/2020 22:01, John Naylor wrote: In 0004, it seems you have some doubts about upgrade compatibility. Is that because user-defined conversions would no longer have the right signature? Exactly. If you have an extension that adds a custom conversion function and does CREATE CONVERSION,

Re: \gsetenv

2020-12-20 Thread Heikki Linnakangas
On 20/12/2020 21:05, David Fetter wrote: We have plenty of ways to spawn shells and cause havoc, and we wouldn't be able to block them all even if we decided to put a bunch of pretty onerous restrictions on psql at this very late date. We have \set, backticks, \!, and bunches of things less

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2020-12-18 Thread Heikki Linnakangas
On 18/12/2020 12:55, Heikki Linnakangas wrote: On 18/12/2020 12:10, Michael Paquier wrote: On Fri, Dec 18, 2020 at 11:35:14AM +0200, Heikki Linnakangas wrote: pg_cryptohash_create() is now susceptible to leaking memory in TopMemoryContext, if the allocations fail. I think the attached should

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2020-12-18 Thread Heikki Linnakangas
On 18/12/2020 12:10, Michael Paquier wrote: On Fri, Dec 18, 2020 at 11:35:14AM +0200, Heikki Linnakangas wrote: pg_cryptohash_create() is now susceptible to leaking memory in TopMemoryContext, if the allocations fail. I think the attached should fix it (but I haven't tested it at all). Yeah

Re: Refactor routine to check for ASCII-only case

2020-12-18 Thread Heikki Linnakangas
On 18/12/2020 05:57, Michael Paquier wrote: As mentioned in [1], there are three places where there is the same routine to check if a string is made only of ASCII characters. This makes for a small-ish but nice cleanup, as per the attached. +1 - Heikki

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2020-12-18 Thread Heikki Linnakangas
On 18/12/2020 11:35, Heikki Linnakangas wrote: BTW, looking at pg_cryptohash_ctx and pg_cryptohash_state, why do we need two structs? They're both allocated and controlled by the cryptohash implementation. It would seem simpler to have just one. Something like this. Passes regression tests

Re: Incorrect allocation handling for cryptohash functions with OpenSSL

2020-12-18 Thread Heikki Linnakangas
On 18/12/2020 09:35, Michael Paquier wrote: Hi all, As of the work done in 87ae9691, I have played with error injections in the code paths using this code, but forgot to count for cases where cascading resowner cleanups are involved. Like other resources (JIT, DSM, etc.), this requires an

Re: pgbench failed when -f option contains a char '@'

2020-12-18 Thread Heikki Linnakangas
On 18/12/2020 08:22, Wang, Shenhao wrote: Hi, hackers pgbench use -f filename[@weight] to receive a sql script file with a weight, but if I create a file contains char'@', like a...@2.sql, specify this file without weigth, pgbench will failed with error: pgbench: fatal: invalid weight

Re: ResourceOwner refactoring

2020-12-16 Thread Heikki Linnakangas
currently, but to be sure, I'm planning to grep ResourceOwnerRemember and look at each call carefullly. And perhaps we can add an assertion for this, although I'm not sure where. - Heikki >From 71a372b9a82a5b50a40386b403d256b4f6fac794 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date:

Re: Optimizing the documentation

2020-12-14 Thread Heikki Linnakangas
On 14/12/2020 21:50, Joshua Drake wrote: The community has spent a lot of time optimizing features over the years. Excellent examples include parallel query and partitioning which have been multi-year efforts to increase the quality,  performance, and extend features of the original commit. We

Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing

2020-12-14 Thread Heikki Linnakangas
On 12/12/2020 13:52, Lukas Meisegeier wrote: Thanks for the provided ideas :) I use HaProxy for my load-balancing and unfortunately I can't define that I want to listen on a port for both ssl and non ssl requests. Could you configure HaProxy to listen on separate ports for SSL and non-SSL

Re: anonymous block returning like a function

2020-12-14 Thread Heikki Linnakangas
On 11/12/2020 21:06, PegoraroF10 wrote: I would like to have an anonymous block, like DO, but having resuts, like an usual function does. I know any user can do ... create function pg_temp.run_time_bigger(numeric,numeric) returns numeric language plpgsql as $$ begin if $1 > $2 then return $1;

Re: pg_basebackup caused FailedAssertion

2020-12-14 Thread Heikki Linnakangas
On 12/12/2020 00:47, Jeff Davis wrote: On Wed, 2013-02-27 at 19:29 +0200, Heikki Linnakangas wrote: Right. I fixed that by adding WL_SOCKET_READABLE, and handling any messages that might arrive after the frontend already sent CopyEnd. The frontend shouldn't send any messages after CopyEnd

Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing

2020-12-11 Thread Heikki Linnakangas
On 11/12/2020 16:46, Lukas Meisegeier wrote: Hey Heikki, thanks for providing feedback :) The traffic between proxy and psql-server is unencrypted thats why I don't need to patch the server. Ok. I tried returning a fixed response on the first plain SSLRequest forwarding it to a psql-server

Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing

2020-12-11 Thread Heikki Linnakangas
On 10/12/2020 17:49, Lukas Meisegeier wrote: I try to host multiple postgresql-servers on the same ip and the same port through SNI-based load-balancing. Currently this is not possible because of two issues: 1. The psql client won't set the tls-sni-extension correctly

Re: pg_rewind race condition just after promotion

2020-12-09 Thread Heikki Linnakangas
On 08/12/2020 06:45, Kyotaro Horiguchi wrote: At Mon, 7 Dec 2020 20:13:25 +0200, Heikki Linnakangas wrote in I think we should fix this properly. I'm not sure if it can lead to a broken cluster, but at least it can cause pg_rewind to fail unnecessarily and in a user-unfriendly way

pg_rewind race condition just after promotion

2020-12-07 Thread Heikki Linnakangas
Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Mon, 7 Dec 2020 19:59:50 +0200 Subject: [PATCH 1/1] pg_rewind: Fix determining TLI when server was just promoted. If the source server was just promoted, and it hasn't written the checkpoint record yet, pg_rewind considered the ser

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-12-05 Thread Heikki Linnakangas
On 05/12/2020 17:10, Andy Fan wrote: Actually I can't understand this, could you explain more?  Based on my current knowledge,  when we run "SELECT DISTINCT a FROM t",  we never care about which operator to use for the unique. SortGroupClause includes 'eqop' field, which determines the

Re: Corner-case bug in pg_rewind

2020-12-04 Thread Heikki Linnakangas
On 04/12/2020 00:16, Heikki Linnakangas wrote: On 03/12/2020 16:10, Heikki Linnakangas wrote: On 02/12/2020 15:26, Ian Barwick wrote: On 02/12/2020 20:13, Heikki Linnakangas wrote: Attached are two patches. The first patch is your original patch, unmodified (except for a cosmetic rename

Re: Corner-case bug in pg_rewind

2020-12-03 Thread Heikki Linnakangas
On 03/12/2020 16:10, Heikki Linnakangas wrote: On 02/12/2020 15:26, Ian Barwick wrote: On 02/12/2020 20:13, Heikki Linnakangas wrote: Attached are two patches. The first patch is your original patch, unmodified (except for a cosmetic rename of the test file). The second patch builds

Re: Corner-case bug in pg_rewind

2020-12-03 Thread Heikki Linnakangas
On 03/12/2020 16:49, Pavel Borisov wrote: Ok, pushed and backpatched this now. Very nice! Thanks to you all! Thanks for the review, Pavel! I just realized that I forgot to credit you in the commit message. I'm sorry. - Heikki

Re: Corner-case bug in pg_rewind

2020-12-03 Thread Heikki Linnakangas
On 02/12/2020 15:26, Ian Barwick wrote: On 02/12/2020 20:13, Heikki Linnakangas wrote: Attached are two patches. The first patch is your original patch, unmodified (except for a cosmetic rename of the test file). The second patch builds on that, demonstrating and fixing the issue I mentioned

Re: Deprecate custom encoding conversions

2020-12-02 Thread Heikki Linnakangas
On 02/12/2020 18:18, Tom Lane wrote: Heikki Linnakangas writes: I propose that we add a notice to the CREATE CONVERSION docs to say that it is deprecated, and remove it in a few years. While I agree that it's probably not that useful, what would we gain by removing it? If you intend

Deprecate custom encoding conversions

2020-12-02 Thread Heikki Linnakangas
Hi, PostgreSQL allows writing custom encoding conversion functions between any character encodings, using the CREATE CONVERSION command. It's pretty flexible, you can define default and non-default conversions, and the conversions live in schemas so you can have multiple conversions

Re: Corner-case bug in pg_rewind

2020-12-02 Thread Heikki Linnakangas
+[], +[qr|rewinding from last common checkpoint at|], +'pg_rewind detects rewind needed'); + -- 2.20.1 >From de487455adae3610310101760739af178e80e446 Mon Sep 17 00:00:00 2001 From: Heikki Linnakangas Date: Wed, 2 Dec 2020 13:04:37 +0200 Subject: [PATCH v2 2/2] Correctly scan all WAL in target,

Re: Index Skip Scan (new UniqueKeys)

2020-12-01 Thread Heikki Linnakangas
On 01/12/2020 22:21, Dmitry Dolgov wrote: On Mon, Nov 30, 2020 at 04:42:20PM +0200, Heikki Linnakangas wrote: I had a quick look at this patch. I haven't been following this thread, so sorry if I'm repeating old arguments, but here we go: Thanks! - I'm surprised you need a new index AM

Re: Avoid using lcons and list_delete_first in plan_union_children()

2020-12-01 Thread Heikki Linnakangas
On 01/12/2020 12:52, Hou, Zhijie wrote: Hi, In function plan_union_children(), I found the lcons and list_delete_first here is easy to be replaced by lappend and list_delete_last. And I also found a previous commit do similar thing, so I try to improve this one. Previous commit:

Re: Deleting older versions in unique indexes to avoid page splits

2020-12-01 Thread Heikki Linnakangas
This is a wholly new concept with a lot of heuristics. It's a lot of swallow. But here are a few quick comments after a first read-through: On 30/11/2020 21:50, Peter Geoghegan wrote: +/* + * State used when calling table_index_delete_check() to perform "bottom up" + * deletion of duplicate

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-11-30 Thread Heikki Linnakangas
On 30/11/2020 16:30, Jesper Pedersen wrote: On 11/30/20 5:04 AM, Heikki Linnakangas wrote: On 26/11/2020 16:58, Andy Fan wrote: This patch has stopped moving for a while,  any suggestion about how to move on is appreciated. The question on whether UniqueKey.exprs should be a list

Re: Index Skip Scan (new UniqueKeys)

2020-11-30 Thread Heikki Linnakangas
On 24/10/2020 19:45, Dmitry Dolgov wrote: Here is a new version which doesn't require "scanstart" argument and contains few other changes to address the issues mentioned earlier. It's also based on the latest UniqueKeys patches with the valgrind issue fixed (as before they're attached also just

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-11-30 Thread Heikki Linnakangas
On 26/11/2020 16:58, Andy Fan wrote: This patch has stopped moving for a while,  any suggestion about how to move on is appreciated. The question on whether UniqueKey.exprs should be a list of EquivalenceClasses or PathKeys is unresolved. I don't have an opinion on that, but I'd suggest that

Re: obsolete comment from WITH OIDS days

2020-11-30 Thread Heikki Linnakangas
On 30/11/2020 10:21, Amit Langote wrote: I noticed a couple of comments that were obsoleted by commit 578b229718 which forgot to remove them. Attached fixes that. Applied, thanks! - Heikki

Re: Removing unneeded self joins

2020-11-29 Thread Heikki Linnakangas
On 28/11/2020 19:21, Andrey Lepikhov wrote: On 27.11.2020 21:49, Heikki Linnakangas wrote: On 31/10/2020 11:26, Andrey V. Lepikhov wrote: +    /* + * Process restrictlist to seperate out the self join quals from + * the other quals. e.g x = x goes

Re: [PATCH] LWLock self-deadlock detection

2020-11-27 Thread Heikki Linnakangas
On 26/11/2020 04:50, Tom Lane wrote: Craig Ringer writes: On Wed, Nov 25, 2020 at 9:23 PM Ashutosh Bapat wrote: I'd prefer to make the lock self deadlock check run for production builds, not just cassert builds. I'd like to register a strong objection to spending any cycles whatsoever on

Re: bitmaps and correlation

2020-11-27 Thread Heikki Linnakangas
On 06/11/2020 19:57, Justin Pryzby wrote: On Fri, Nov 06, 2020 at 01:51:26PM +, Anastasia Lubennikova wrote: The first patch is simply a refactoring and don't see any possible objections against it. The second patch also looks fine to me. The logic is understandable and the code is neat.

Re: Removing unneeded self joins

2020-11-27 Thread Heikki Linnakangas
On 31/10/2020 11:26, Andrey V. Lepikhov wrote: + /* +* Process restrictlist to seperate out the self join quals from +* the other quals. e.g x = x goes to selfjoinquals and a = b to +* otherjoinquals.

Re: parallel distinct union and aggregate support patch

2020-11-27 Thread Heikki Linnakangas
I also had a quick look at the patch and the comments made so far. Summary: 1. The performance results are promising. 2. The code needs comments. Regarding the design: Thomas Munro mentioned the idea of a "Parallel Repartition" node that would redistribute tuples like this. As I understand

Re: Setof RangeType returns

2020-11-27 Thread Heikki Linnakangas
On 26/11/2020 23:28, Patrick Handja wrote: Hello, I am currently working on Library with some specific operators to manipulate RangeType in PostGreSQL. I would like to know if it is possible to return a setof rangetype elements in Postresql in C-Language function using the suggestion like

Re: [PoC] Non-volatile WAL buffer

2020-11-26 Thread Heikki Linnakangas
On 26/11/2020 21:27, Tomas Vondra wrote: Hi, Here's the "simple patch" that I'm currently experimenting with. It essentially replaces open/close/write/fsync with pmem calls (map/unmap/memcpy/persist variants), and it's by no means committable. But it works well enough for experiments /

Re: Improving spin-lock implementation on ARM.

2020-11-26 Thread Heikki Linnakangas
On 26/11/2020 06:30, Krunal Bauskar wrote: Improving spin-lock implementation on ARM. * Spin-Lock is known to have a significant effect on performance   with increasing scalability. * Existing Spin-Lock implementation for ARM is

Re: Online checksums patch - once again

2020-11-25 Thread Heikki Linnakangas
On 25/11/2020 15:20, Daniel Gustafsson wrote: On 23 Nov 2020, at 18:36, Heikki Linnakangas wrote: What happens is if you crash between UpdateControlFile() and XlogChecksum()? Good point, that would not get the cluster to a consistent state. The XlogChecksum should be performed before

Re: Deduplicate aggregates and transition functions in planner

2020-11-24 Thread Heikki Linnakangas
On 19/11/2020 12:38, Heikki Linnakangas wrote: So barring objections, I'm going to push the attached updated patch that includes the removal of AggrefExprState, and leave CookedAggrefs or other further refactorings for the future. Done. Thanks! - Heikki

Re: Online checksums patch - once again

2020-11-23 Thread Heikki Linnakangas
On 17/11/2020 10:56, Daniel Gustafsson wrote: On 5 Oct 2020, at 13:36, Heikki Linnakangas wrote: 2. The signaling between enable_data_checksums() and the launcher process looks funny to me. The general idea seems to be that enable_data_checksums() just starts the launcher process

prion failed with ERROR: missing chunk number 0 for toast value 14334 in pg_toast_2619

2020-11-23 Thread Heikki Linnakangas
Hi, After my commit c532d15ddd to split up copy.c, buildfarm animal "prion" failed in pg_upgrade (https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion=2020-11-23%2009%3A23%3A16): Upgrade Complete Optimizer statistics are not transferred by pg_upgrade so, once you

Re: Multi Inserts in CREATE TABLE AS - revived patch

2020-11-23 Thread Heikki Linnakangas
On 23/11/2020 11:15, Bharath Rupireddy wrote: Attaching v2 patch, rebased on the latest master 17958972. I just broke this again with commit c532d15ddd to split up copy.c. Here's another rebased version. - Heikki >From dca55175c590914f6a802ec3d36e20db55a3e3c7 Mon Sep 17 00:00:00 2001 From:

Re: Split copy.c

2020-11-23 Thread Heikki Linnakangas
On 18/11/2020 08:21, Soumyadeep Chakraborty wrote: On Tue, Nov 17, 2020 at 2:38 AM Heikki Linnakangas wrote: Fixed all the other things you listed, fixed a bug in setting 'file_encoding', and trimmed down the #includes. Thanks! LGTM! Marking as Ready for Committer. Pushed, thanks

Re: bug in pageinspect's "tuple data" feature

2020-11-22 Thread Heikki Linnakangas
On 21/11/2020 21:32, Alvaro Herrera wrote: If you have a sufficiently broken data page, pageinspect throws an error when trying to examine the page: ERROR: invalid memory alloc request size 18446744073709551451 This is pretty unhelpful; it would be better not to try to print the data instead

Re: Refactor pg_rewind code and make it work against a standby

2020-11-20 Thread Heikki Linnakangas
On 20/11/2020 19:14, Andres Freund wrote: Hi, On 2020-11-20 16:19:03 +0200, Heikki Linnakangas wrote: Pushed a fix similar to your patch, but I put the wait_for_catchup() before running pg_rewind. The point of inserting the 'in A, after C was promoted' row is that it's present in B when

Re: Refactor pg_rewind code and make it work against a standby

2020-11-20 Thread Heikki Linnakangas
On 20/11/2020 02:38, Andres Freund wrote: I locally, on a heavily modified branch (AIO support), started to get consistent failures in this test. I *suspect*, but am not sure, that it's the test's fault, not the fault of modifications. As far as I can tell, after the pg_rewind call, there's no

Re: don't allocate HashAgg hash tables when running explain only

2020-11-20 Thread Heikki Linnakangas
On 20/11/2020 08:31, Michael Paquier wrote: On Thu, Nov 19, 2020 at 08:47:51AM +0200, Heikki Linnakangas wrote: Yeah, I believe it's always been like that. Yeah, arguably it should be backpatched. I felt conservative and didn't backpatch, but feel free to do it if you think it should be. +1

Re: Deduplicate aggregates and transition functions in planner

2020-11-19 Thread Heikki Linnakangas
On 29/10/2020 19:48, Andres Freund wrote: On 2020-10-29 10:17:20 +0200, Heikki Linnakangas wrote: On 28/10/2020 21:59, Andres Freund wrote: It wouldn't surprise me to see a small execution time speedup here - I've seen the load of the aggno show up in profiles. I think you'd be hard-pressed

Re: don't allocate HashAgg hash tables when running explain only

2020-11-18 Thread Heikki Linnakangas
On 19/11/2020 07:20, Greg Stark wrote: On Wed, 18 Nov 2020 at 05:40, Heikki Linnakangas wrote: On 13/11/2020 18:10, Alexey Bashtanov wrote: I would appreciate if someone could have a look at the patch attached, which makes executor skip initializing hash tables when doing explain only

<    1   2   3   4   5   6   7   8   9   10   >