Re: Different compression methods for FPI

2021-06-16 Thread Michael Paquier
On Wed, Jun 16, 2021 at 09:39:57AM +0900, Michael Paquier wrote: > From I'd like us to finish with here is one new algorithm method, able > to cover a large range of cases as mentioned upthread, from > low-CPU/low-compression to high-CPU/high-compression. It does not > seem like a good idea to be

Re: Skip partition tuple routing with constant partition key

2021-06-16 Thread Amit Langote
Hou-san, On Mon, Jun 7, 2021 at 8:38 PM houzj.f...@fujitsu.com wrote: > Thanks for the patch and explanation, I think you are right that it’s better > add > the rechecking-cached-offset code directly in get_partiiton_for_tuple(). > > And now, I think maybe it's time to try to optimize the perfor

Re: Error on pgbench logs

2021-06-16 Thread Michael Paquier
On Wed, Jun 16, 2021 at 08:58:17AM +0200, Fabien COELHO wrote: > Actually it would work if both are mixed: the code would aggregate a sample. > However it does not look very useful to do that, so it is arbitrary > forbidden. Not sure whether this is that useful to prevent this use case. Okay, note

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-06-16 Thread Dilip Kumar
On Tue, Jun 15, 2021 at 7:01 PM Andrew Dunstan wrote: > > Rather than use size, I'd be inclined to say use this if the source > database is marked as a template, and use the copydir approach for > anything that isn't. Yeah, that is possible, on the other thought wouldn't it be good to provide con

Re: detailed error message of pg_waldump

2021-06-16 Thread Masahiko Sawada
On Fri, Jun 4, 2021 at 5:35 PM Kyotaro Horiguchi wrote: > > In a very common operation of accidentally specifying a recycled > segment, pg_waldump often returns the following obscure message. > > $ pg_waldump 0001002D > pg_waldump: fatal: could not find a valid record after 0/2D000

Re: Error on pgbench logs

2021-06-16 Thread Fabien COELHO
+ * The function behaviors changes depending on sample_rate (a fraction of + * transaction is reported) and agg_interval (transactions are aggregated + * over the interval and reported once). The first part of this sentence has an incorrect grammar. Indeed. v5 attempts to improve comments.

Re: Duplicate history file?

2021-06-16 Thread Kyotaro Horiguchi
At Wed, 16 Jun 2021 12:36:19 +0800, Julien Rouhaud wrote in > On Wed, Jun 16, 2021 at 01:10:16PM +0900, Kyotaro Horiguchi wrote: > > > > I agree to Julien, however, I want to discuss (also) on what to do for > > 14 now. If we decide not to touch the document for the version. that > > discussio

Re: Different compression methods for FPI

2021-06-16 Thread Andrey Borodin
> 16 июня 2021 г., в 12:18, Michael Paquier написал(а): > Among the > remaining two I would be tempted to choose LZ4. That's consistent > with what toast can use now. I agree that allowing just lz4 - is already a huge step ahead. But I'd suggest supporting zstd as well. Currently we only com

Re: detailed error message of pg_waldump

2021-06-16 Thread Kyotaro Horiguchi
Thanks! At Wed, 16 Jun 2021 16:52:11 +0900, Masahiko Sawada wrote in > On Fri, Jun 4, 2021 at 5:35 PM Kyotaro Horiguchi > wrote: > > > > In a very common operation of accidentally specifying a recycled > > segment, pg_waldump often returns the following obscure message. > > > > $ pg_waldump 00

Re: Signed vs. Unsigned (some)

2021-06-16 Thread Peter Eisentraut
On 15.06.21 10:17, Kyotaro Horiguchi wrote: The definitions are not ((type) -1) but ((type) 0x) so actually they might be different if we forget to widen the constant when widening the types. Regarding to the compiler behavior, I think we are assuming C99[1] and C99 defines that -1 is co

Re: Different compression methods for FPI

2021-06-16 Thread Heikki Linnakangas
On 16/06/2021 11:17, Andrey Borodin wrote: 16 июня 2021 г., в 12:18, Michael Paquier написал(а): Among the remaining two I would be tempted to choose LZ4. That's consistent with what toast can use now. I agree that allowing just lz4 - is already a huge step ahead. But I'd suggest supporting

Re: Different compression methods for FPI

2021-06-16 Thread Andrey Borodin
> 16 июня 2021 г., в 13:49, Heikki Linnakangas написал(а): > > Hmm, do we currently compress each block in a WAL record separately, for > records that contain multiple full-page images? That could make a big > difference e.g. for GiST index build that WAL-logs 32 pages in each record. > If

Re: Skipping logical replication transactions on subscriber side

2021-06-16 Thread Amit Kapila
On Tue, Jun 15, 2021 at 6:13 AM Masahiko Sawada wrote: > > On Wed, Jun 2, 2021 at 3:07 PM Amit Kapila wrote: > > > > On Tue, Jun 1, 2021 at 9:05 PM Peter Eisentraut > > wrote: > > > > > > On 01.06.21 06:01, Amit Kapila wrote: > > > > But, won't that be costly in cases where we have errors in the

Re: Position of ClientAuthentication hook

2021-06-16 Thread Rafia Sabih
On Mon, 14 Jun 2021 at 21:04, Robert Haas wrote: > > On Mon, Jun 14, 2021 at 8:51 AM Rafia Sabih wrote: > > I have a doubt regarding the positioning of clientAuthentication hook > > in function ClientAuthentication. Particularly, in case of hba errors, > > i.e. cases uaReject or uaImplicitReject

Pipeline mode and PQpipelineSync()

2021-06-16 Thread Boris Kolpackov
I am trying to add bulk operation support to ODB (a C++ ORM) using the new pipeline mode added to libpq in PostgreSQL 14. However, things don't seem to be working according to the documentation (or perhaps I am misunderstanding something). Specifically, the documentation[1] makes it sound like the

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-06-16 Thread Amit Kapila
On Mon, Jun 14, 2021 at 5:33 PM osumi.takami...@fujitsu.com wrote: > > On Friday, June 11, 2021 2:13 PM vignesh C wrote: > > Attached the patch-set that addressed those two comments. > Few minor comments: 1. + + +Clustering pg_class in a transaction. Can we change above t

Issue with some calls to GetMultiXactIdMembers()

2021-06-16 Thread Greg Nancarrow
Hi, There's a couple of calls to GetMultiXactIdMembers() in heapam.c which subsequently pfree() the returned "members" pointer (pass-by-reference parameter) if it's non-NULL. However, there's an error return within GetMultiXactIdMembers() that returns -1 without NULLing out "members", and the call

Re: Unresolved repliaction hang and stop problem.

2021-06-16 Thread Ha Ka
śr., 16 cze 2021 o 12:31 Lukasz Biegaj napisał(a): > On 04.05.2021 16:35, Alvaro Herrera wrote: > > I would suggest to do that by running the problematic > > workload in the test system under "perf record -g" > We'll go with both propositions. I expect to come back to you with > results in about a

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Matthias van de Meent
On Tue, 15 Jun 2021 at 03:22, Andres Freund wrote: > > Hi, > > > @@ -4032,6 +4039,24 @@ GlobalVisTestShouldUpdate(GlobalVisState *state) > > static void > > GlobalVisUpdateApply(ComputeXidHorizonsResult *horizons) > > { > > + /* assert non-decreasing nature of horizons */ > > Thinking more

Re: Added schema level support for publication.

2021-06-16 Thread Amit Kapila
On Sat, Jan 9, 2021 at 8:08 PM Bharath Rupireddy wrote: > > And, with this feature, since there can be many huge tables inside a > schema, the initial table sync phase of the replication can take a > while. > > Say a user has created a publication for a schema with hundreds of > tables in it, at s

Re: RFC: Logging plan of the running query

2021-06-16 Thread torikoshia
On 2021-06-15 13:27, Bharath Rupireddy wrote: On Mon, Jun 14, 2021 at 5:48 PM torikoshia wrote: > 1) We could just say "Requests to log query plan of the presently > running query of a given backend along with an untruncated query > string in the server logs." > Instead of > +They will

Re: Added schema level support for publication.

2021-06-16 Thread Ajin Cherian
On Sat, Jun 5, 2021 at 11:32 PM vignesh C wrote: > Thanks for identifying and reporting this issue. I have \dn with the > equivalent query to display only the publication name. The updated > patch has the fix for the same. > The patch no longer applies, I think a recent change to tab-complete ha

Re: Signed vs. Unsigned (some)

2021-06-16 Thread Ranier Vilela
Em qua., 16 de jun. de 2021 às 05:48, Peter Eisentraut < peter.eisentr...@enterprisedb.com> escreveu: > On 15.06.21 10:17, Kyotaro Horiguchi wrote: > > The definitions are not ((type) -1) but ((type) 0x) so > > actually they might be different if we forget to widen the constant > > when wi

Re: unnesting multirange data types

2021-06-16 Thread Alexander Korotkov
On Tue, Jun 15, 2021 at 8:28 PM Alvaro Herrera wrote: > On 2021-Jun-15, Tom Lane wrote: > > > It looks to me like the proximate problem is that you should > > have taught pg_dump to skip these new auto-generated functions. > > However, I fail to see why we need auto-generated functions > > for thi

Re: unnesting multirange data types

2021-06-16 Thread Alexander Korotkov
On Tue, Jun 15, 2021 at 9:43 PM Andrew Dunstan wrote: > On 6/15/21 12:06 PM, Alexander Korotkov wrote: > > On Tue, Jun 15, 2021 at 4:49 PM Tom Lane wrote: > >> Alexander Korotkov writes: > >>> Pushed! Thanks to thread participants for raising this topic and > >>> review. I'll be around to reso

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-16 Thread Alexander Pyhalov
Tomas Vondra писал 2021-06-12 00:01: On 6/9/21 1:08 PM, Tomas Vondra wrote: On 6/9/21 12:50 PM, Bharath Rupireddy wrote: On Wed, Jun 9, 2021 at 4:00 PM Tomas Vondra wrote: Hi, Here's a v2 fixing a silly bug with reusing the same variable in two nested loops (worked for simple postgres_fdw

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-06-16 Thread houzj.f...@fujitsu.com
On Tuesday, June 15, 2021 10:01 PM Robert Haas wrote: > On Tue, Jun 15, 2021 at 7:05 AM Amit Kapila wrote: > > Okay, but I think if we go with your suggested model where whenever > > there is a change in parallel-safety of any function, we need to send > > the new invalidation then I think it won

Re: unnesting multirange data types

2021-06-16 Thread Alexander Korotkov
On Tue, Jun 15, 2021 at 8:18 PM Tom Lane wrote: > Alexander Korotkov writes: > > I did run "check-world", but it passed for me. Probably the same > > reason it passed for some buildfarm animals... > > It looks to me like the proximate problem is that you should > have taught pg_dump to skip thes

Re: [PATCH] rename column if exists

2021-06-16 Thread Yagiz Nizipli
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:tested, passed Thank you for your contribution. This is a useful feature. Alth

Re: Duplicate history file?

2021-06-16 Thread Stephen Frost
Greetings, * Julien Rouhaud (rjuju...@gmail.com) wrote: > On Wed, Jun 16, 2021 at 01:17:11AM -0400, Stephen Frost wrote: > > > Consider that, really, an archive command should refuse to allow archiving > > > > of WAL on a timeline which doesn’t have a corresponding history file in > > > the > > >

Re: Support for NSS as a libpq TLS backend

2021-06-16 Thread Daniel Gustafsson
> On 16 Jun 2021, at 01:50, Jacob Champion wrote: > I've been tracking down reference leaks in the client. These open > references prevent NSS from shutting down cleanly, which then makes it > impossible to open a new context in the future. This probably affects > other libpq clients more than it

Re: Unresolved repliaction hang and stop problem.

2021-06-16 Thread Alvaro Herrera
On 2021-Jun-16, Ha Ka wrote: > śr., 16 cze 2021 o 12:31 Lukasz Biegaj > napisał(a): > > On 04.05.2021 16:35, Alvaro Herrera wrote: > > > I would suggest to do that by running the problematic > > > workload in the test system under "perf record -g" > > We'll go with both propositions. I expect to

Re: [Proposal] Add accumulated statistics for wait event

2021-06-16 Thread Jehan-Guillaume de Rorthais
Hi Andres, On Mon, 14 Jun 2021 15:01:14 -0700 Andres Freund wrote: > On 2021-06-14 23:20:47 +0200, Jehan-Guillaume de Rorthais wrote: > > > On 2021-06-14 16:10:32 +0200, Jehan-Guillaume de Rorthais wrote: > > > > In the patch in attachment, I tried to fix this by using kind of an > > > > inter

Re: Duplicate history file?

2021-06-16 Thread Julien Rouhaud
On Wed, Jun 16, 2021 at 9:19 PM Stephen Frost wrote: > > This is exactly it. I don't agree that we can, or should, treat every > sensible thing that we realize about what the archive command or the > backup tool should be doing as some bug in our documentation that has to > be backpatched. > If y

Re: pg_stat_statements and "IN" conditions

2021-06-16 Thread Dmitry Dolgov
> On Tue, Jun 15, 2021 at 05:18:50PM +0200, Dmitry Dolgov wrote: > > On Thu, Mar 18, 2021 at 04:50:02PM +0100, Dmitry Dolgov wrote: > > > On Thu, Mar 18, 2021 at 09:38:09AM -0400, David Steele wrote: > > > On 1/5/21 10:51 AM, Zhihong Yu wrote: > > > > > > > > +   int         lastExprLenght = 0; > >

Re: Improving isolationtester's data output

2021-06-16 Thread Tom Lane
Noah Misch writes: > I'm +1 for back-patching this class of change. I've wasted time adapting a > back-patch's test case to account for non-back-patched test infrastructure > changes. Every back-patch of test infrastructure has been a strict win from > my perspective. Hearing few objections, I'

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-16 Thread Tomas Vondra
On 6/16/21 2:36 PM, Alexander Pyhalov wrote: Hi. It seems this commit commit b676ac443b6a83558d4701b2dd9491c0b37e17c4 Author: Tomas Vondra Date:   Fri Jun 11 20:19:48 2021 +0200     Optimize creation of slots for FDW bulk inserts has broken batch insert for partitions with unique indexes.

Re: Should wal receiver reply to wal sender more aggressively?

2021-06-16 Thread Paul Guo
[ Resending the mail since I found my previous email has a very bad format that is hard to read]. While working on some related issues I found that the wal receiver tries to call walrcv_receive() loop before replying the write/flush/apply LSN to wal senders in XLogWalRcvSendReply(). It is possib

Re: Duplicate history file?

2021-06-16 Thread Stephen Frost
Greetings, * Julien Rouhaud (rjuju...@gmail.com) wrote: > On Wed, Jun 16, 2021 at 9:19 PM Stephen Frost wrote: > > This is exactly it. I don't agree that we can, or should, treat every > > sensible thing that we realize about what the archive command or the > > backup tool should be doing as som

Re: Decoding speculative insert with toast leaks memory

2021-06-16 Thread Tom Lane
Amit Kapila writes: > Pushed! skink reports that this has valgrind issues: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2021-06-15%2020%3A49%3A26 2021-06-16 01:20:13.344 UTC [2198271][4/0:0] LOG: received replication command: IDENTIFY_SYSTEM 2021-06-16 01:20:13.384 UTC [21

pgbench logging broken by time logic changes

2021-06-16 Thread Gregory Smith
Back on March 10 Thomas Munro committed and wrestled multiple reworks of the pgbench code from Fabien and the crew. The feature to synchronize startup I'm looking forward to testing now that I have a packaged beta. Variations on that problem have bit me so many times I added code last year to my p

Re: Improving isolationtester's data output

2021-06-16 Thread Alvaro Herrera
On 2021-Jun-16, Tom Lane wrote: > Noah Misch writes: > > I'm +1 for back-patching this class of change. I've wasted time adapting a > > back-patch's test case to account for non-back-patched test infrastructure > > changes. Every back-patch of test infrastructure has been a strict win from > >

Re: Unresolved repliaction hang and stop problem.

2021-06-16 Thread Ha Ka
> Hello, thanks, I downloaded the files but since you sent the perf.data > files there's not much I can do to usefully interpret them. Can you > please do "perf report -g > perf_report.txt" on each subdir with a > perf.data file and upload those text files? (You don't need to rerun > the test cas

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Greg Stark
I think Andres's point earlier is the one that stands out the most for me: > I still think that's the most reasonable course. I actually like the > feature, but I don't think a better implementation of it would share > much if any of the current infrastructure. That makes me wonder whether rippin

Re: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-06-16 Thread Robert Haas
On Tue, Jun 15, 2021 at 10:41 AM Amit Kapila wrote: > > I don't think that finding the relation involved and registering an > > invalidation for the same will work properly. Suppose there is a > > concurrently-running transaction which has created a new table and > > attached a trigger function to

Re: SQLSTATE for replication connection failures

2021-06-16 Thread Tom Lane
Masahiko Sawada writes: > On Mon, Jun 14, 2021 at 6:18 PM Amit Kapila wrote: >>> Shall we just use ERRCODE_CONNECTION_FAILURE for these failures, or >>> would it be better to invent another SQLSTATE code? Arguably, >>> ERRCODE_CONNECTION_FAILURE is meant for failures of client connections; >>> b

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Tom Lane
Greg Stark writes: > Fwiw I too think the basic idea of the feature is actually awesome. > There are tons of use cases where you might have one long-lived > transaction working on a dedicated table (or even a schema) that will > never look at the rapidly mutating tables in another schema and never

Re: pgbench bug candidate: negative "initial connection time"

2021-06-16 Thread Yugo NAGATA
Hello Fabien, On Mon, 14 Jun 2021 11:30:14 +0200 (CEST) Fabien COELHO wrote: > >>> Hmmm. Possibly. Another option could be not to report anything after some > >>> errors. I'm not sure, because it would depend on the use case. I guess the > >>> command returned an error status as well. > >> > >>

Re: a path towards replacing GEQO with something better

2021-06-16 Thread Robert Haas
On Tue, Jun 15, 2021 at 2:16 PM John Naylor wrote: > > I don't quite understand the difference between the "chain" case and > > the "star" case. Can you show sample queries for each one? e.g. SELECT > > ... FROM a_1, a_2, ..., a_n WHERE ? > > SELECT * > FROMtab1, tab2, tab3, tab4 > WHERE ta

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 3:59 AM Matthias van de Meent wrote: > On Tue, 15 Jun 2021 at 03:22, Andres Freund wrote: > > > @@ -4032,6 +4039,24 @@ GlobalVisTestShouldUpdate(GlobalVisState *state) > > > static void > > > GlobalVisUpdateApply(ComputeXidHorizonsResult *horizons) > > > { > > > + /

Re: Transactions involving multiple postgres foreign servers, take 2

2021-06-16 Thread Robert Haas
On Tue, Jun 15, 2021 at 5:51 AM tsunakawa.ta...@fujitsu.com wrote: > Postgres can do that, but other implementations can not necessaily do it, I'm > afraid. But before that, the FDW interface documentation doesn't describe > anything about how to handle interrupts. Actually, odbc_fdw and possi

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Stephen Frost
Greetings, * Greg Stark (st...@mit.edu) wrote: > I think Andres's point earlier is the one that stands out the most for me: > > > I still think that's the most reasonable course. I actually like the > > feature, but I don't think a better implementation of it would share > > much if any of the cu

Re: Support for NSS as a libpq TLS backend

2021-06-16 Thread Jacob Champion
On Wed, 2021-06-16 at 15:31 +0200, Daniel Gustafsson wrote: > > On 16 Jun 2021, at 01:50, Jacob Champion wrote: > > I've been tracking down reference leaks in the client. These open > > references prevent NSS from shutting down cleanly, which then makes it > > impossible to open a new context in t

PoC: Using Count-Min Sketch for join cardinality estimation

2021-06-16 Thread Tomas Vondra
Hi, During the recent "CMU vaccination" talk given by Robert [1], a couple of the attendees (some of which were engineers working on various other database systems) asked whether PostgreSQL optimizer uses sketches. Which it does not, as far as I'm aware. Perhaps some of our statistics could b

Re: a path towards replacing GEQO with something better

2021-06-16 Thread John Naylor
On Wed, Jun 16, 2021 at 12:01 PM Robert Haas wrote: > > On Tue, Jun 15, 2021 at 2:16 PM John Naylor > wrote: > > > I don't quite understand the difference between the "chain" case and > > > the "star" case. Can you show sample queries for each one? e.g. SELECT > > > ... FROM a_1, a_2, ..., a_n WH

Re: Avoid stuck of pbgench due to skipped transactions

2021-06-16 Thread Yugo NAGATA
On Mon, 14 Jun 2021 16:06:10 +0900 Yugo NAGATA wrote: > On Mon, 14 Jun 2021 08:47:40 +0200 (CEST) > Fabien COELHO wrote: > > > I attached the fixed patch that uses return instead of break, and I > > > confirmed > > > that this made the progress reporting work property. > > > > I'm hesitating

Allowing regular identifiers in isolationtester scripts

2021-06-16 Thread Tom Lane
[ new subject, new thread, new patch ] Alvaro Herrera writes: > On 2021-Jun-16, Tom Lane wrote: >> BTW, as long as we're thinking of back-patching nontrivial specfile >> changes, I have another modest proposal. What do people think of >> removing the requirement for step/session names to be doub

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 9:03 AM Peter Geoghegan wrote: > On Wed, Jun 16, 2021 at 3:59 AM Matthias van de Meent > > So the implicit assumption in heap_page_prune that > > HeapTupleSatisfiesVacuum(OldestXmin) is always consistent with > > heap_prune_satisfies_vacuum(vacrel) has never been true. In t

Re: a path towards replacing GEQO with something better

2021-06-16 Thread Tom Lane
John Naylor writes: > On Wed, Jun 16, 2021 at 12:01 PM Robert Haas wrote: >> I feel like these are completely equivalent. Either way, the planner >> is going to deduce that all the ".col" columns are equal to each other >> via the equivalence class machinery, and then the subsequent planning >> w

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Tom Lane
Stephen Frost writes: > I've long felt that the appropriate approach to addressing that is to > improve on VACUUM and find a way to do better than just having the > conditional of 'xmax < global min' drive if we can mark a given tuple as > no longer visible to anyone. Yeah, I think this scenario

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 10:04 AM Tom Lane wrote: > I remember that Heikki was fooling with a patch that reduced snapshots > to LSNs. If we got that done, it'd be practical to expose complete > info about backends' snapshot state in a lot of cases (i.e., anytime > you had less than N live snapshot

Re: pgbench logging broken by time logic changes

2021-06-16 Thread Yugo NAGATA
On Wed, 16 Jun 2021 10:49:36 -0400 Gregory Smith wrote: > A lot of things are timed in pgbench now so I appreciate the idea. Y'all > started that whole big thread about sync on my birthday though and I didn't > follow the details of what that was reviewed against. For the logging use > case I s

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Andres Freund
Hi, On 2021-06-15 21:59:45 -0700, Noah Misch wrote: > Hackers are rather wise, but the variety of PostgreSQL use is enormous. We > see that, among other ways, when regression fixes spike in each vN.1. The > $SUBJECT feature was born in response to a user experience; a lack of hacker > interest d

Re: pgbench bug candidate: negative "initial connection time"

2021-06-16 Thread Fabien COELHO
Hello Yugo-san, When connection break while the bench has already started, maybe it makes more sense to proceed, The result would be incomplete also in this case. However, the reason why it is worth to proceed is that such information is still useful for users, or we don't want to waste the

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Andres Freund
Hi, On 2021-06-16 13:04:07 -0400, Tom Lane wrote: > Yeah, I think this scenario of a few transactions with old snapshots > and the rest with very new ones could be improved greatly if we exposed > more info about backends' snapshot state than just "oldest xmin". But > that might be expensive to d

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 11:06 AM Andres Freund wrote: > > 2) (a) Some hackers want the feature gone so they can implement changes > >without making those changes cooperate with this feature. (b) Bugs in > > this > >feature make such cooperation materially harder. > > I think the a) part

Re: pgbench logging broken by time logic changes

2021-06-16 Thread Fabien COELHO
Hello Greg, I have a lot of community oriented work backed up behind this right now, so I'm gonna be really honest. This time rework commit in its current form makes me uncomfortable at this point in the release schedule. The commit has already fought through two rounds of platform specific

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Andres Freund
Hi, On 2021-06-16 10:44:49 -0700, Peter Geoghegan wrote: > On Wed, Jun 16, 2021 at 10:04 AM Tom Lane wrote: > > Of course, there's still the question of how VACUUM could cheaply > > apply such info to decide what could be purged. > I would think that it wouldn't really matter inside VACUUM -- it

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 11:27 AM Andres Freund wrote: > 2) Modeling when it is safe to remove row versions. It is easy to remove >a tuple that was inserted and deleted within one "not needed" xid >range, but it's far less obvious when it is safe to remove row >versions where prior/late

Re: pgbench logging broken by time logic changes

2021-06-16 Thread Fabien COELHO
pg_time_now(). This uses INSTR_TIME_SET_CURRENT in it, but this macro can call clock_gettime(CLOCK_MONOTONIC[_RAW], ) instead of gettimeofday or clock_gettime(CLOCK_REALTIME, ). When CLOCK_MONOTONIC[_RAW] is used, clock_gettime doesn't return epoch time. Therefore, we can use INSTR_TIME_SET_CURR

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Andres Freund
Hi, On 2021-06-16 12:59:33 +0200, Matthias van de Meent wrote: > PFA my adapted patch that fixes this new-ish issue, and does not > include the (incorrect) assertions in GlobalVisUpdateApply. I've > tested this against the reproducing case, both with and without the > fix in GetOldestNonRemovableT

Re: pgbench logging broken by time logic changes

2021-06-16 Thread Andrew Dunstan
On 6/16/21 2:59 PM, Fabien COELHO wrote: > > Hello Greg, > >> I have a lot of community oriented work backed up behind this right >> now, so >> I'm gonna be really honest.  This time rework commit in its current form >> makes me uncomfortable at this point in the release schedule.  The >> commit

Re: snapshot too old issues, first around wraparound and then more.

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 12:06 PM Andres Freund wrote: > > I would think that it wouldn't really matter inside VACUUM -- it would > > only really need to be either an opportunistic pruning or an > > opportunistic index deletion thing -- probably both. Most of the time > > VACUUM doesn't seem to end

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Andres Freund
Hi, On 2021-06-16 09:46:07 -0700, Peter Geoghegan wrote: > On Wed, Jun 16, 2021 at 9:03 AM Peter Geoghegan wrote: > > On Wed, Jun 16, 2021 at 3:59 AM Matthias van de Meent > > > So the implicit assumption in heap_page_prune that > > > HeapTupleSatisfiesVacuum(OldestXmin) is always consistent with

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Matthias van de Meent
On Wed, 16 Jun 2021 at 21:12, Andres Freund wrote: > > Hi, > > On 2021-06-16 12:59:33 +0200, Matthias van de Meent wrote: > > PFA my adapted patch that fixes this new-ish issue, and does not > > include the (incorrect) assertions in GlobalVisUpdateApply. I've > > tested this against the reproducin

Re: Improving isolationtester's data output

2021-06-16 Thread Andres Freund
Hi, On 2021-06-15 22:44:29 -0400, Tom Lane wrote: > Here's a really quick-and-dirty patch to see what that would look > like. I haven't bothered here to update the expected-files outside > the main src/test/isolation directory, nor to fix the variant files. Neat. > + memset(&popt, 0, sizeo

Re: Improving isolationtester's data output

2021-06-16 Thread Tom Lane
Alvaro Herrera writes: > On 2021-Jun-16, Tom Lane wrote: >> Hearing few objections, I'll plan on back-patching. I'm thinking that the >> best thing to do is apply these changes after beta2 wraps, but before we >> branch v14. > Great. After checking cross-version diffs to see how painful that is

Re: Improving isolationtester's data output

2021-06-16 Thread Tom Lane
Andres Freund writes: > On 2021-06-15 22:44:29 -0400, Tom Lane wrote: >> +memset(&popt, 0, sizeof(popt)); >> +popt.header = true; >> +popt.align = true; >> +popt.fieldSep = "|"; >> +PQprint(stdout, res, &popt); > Is there an argument for not aligning because that can make diff

Re: Improving isolationtester's data output

2021-06-16 Thread Andres Freund
Hi, On Wed, Jun 16, 2021, at 12:37, Tom Lane wrote: > Andres Freund writes: > > On 2021-06-15 22:44:29 -0400, Tom Lane wrote: > >> + memset(&popt, 0, sizeof(popt)); > >> + popt.header = true; > >> + popt.align = true; > >> + popt.fieldSep = "|"; > >> + PQprint(stdout, res, &popt); > > > Is

Re: Support for NSS as a libpq TLS backend

2021-06-16 Thread Daniel Gustafsson
> On 16 Jun 2021, at 18:15, Jacob Champion wrote: > > On Wed, 2021-06-16 at 15:31 +0200, Daniel Gustafsson wrote: >>> On 16 Jun 2021, at 01:50, Jacob Champion wrote: >>> I've been tracking down reference leaks in the client. These open >>> references prevent NSS from shutting down cleanly, which

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Matthias van de Meent
On Wed, 16 Jun 2021 at 21:22, Andres Freund wrote: > > Hi, > > On 2021-06-16 09:46:07 -0700, Peter Geoghegan wrote: > > On Wed, Jun 16, 2021 at 9:03 AM Peter Geoghegan wrote: > > > On Wed, Jun 16, 2021 at 3:59 AM Matthias van de Meent > > > > So the implicit assumption in heap_page_prune that > >

Re: A qsort template

2021-06-16 Thread Zhihong Yu
On Tue, Jun 15, 2021 at 10:55 PM Thomas Munro wrote: > On Mon, Mar 15, 2021 at 1:09 PM Thomas Munro > wrote: > > On Sun, Mar 14, 2021 at 5:03 PM Zhihong Yu wrote: > > > + * Remove duplicates from an array. Return the new size. > > > + */ > > > +ST_SCOPE size_t > > > +ST_UNIQUE(ST_ELEMENT_TYPE

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-16 Thread Peter Geoghegan
On Wed, Jun 16, 2021 at 12:22 PM Andres Freund wrote: > I think it's more complicated than that - "before" isn't a guarantee when the > horizon can go backwards. Consider the case where a hot_standby_feedback=on > replica without a slot connects - that can result in the xid horizon going > backwar

Re: Python 3.10 breaks regression tests with traceback changes

2021-06-16 Thread Tom Lane
Peter Eisentraut writes: > Patch applied to master. I suppose we should backpatch this. Python 3.10 has evidently now percolated into Fedora Rawhide, thus caiman is failing in the back branches. Time for that back-patch. regards, tom lane

Re: Replication protocol doc fix

2021-06-16 Thread Jeff Davis
On Mon, 2021-06-14 at 10:51 -0400, Robert Haas wrote: > but if > > > you use the extended query protocol, then the result is a > > > hopeless > > > mess, because the protocol is badly designed: > > > After looking in more detail, I think I understand a bit better. Clients don't differentiate betw

Re: Replication protocol doc fix

2021-06-16 Thread Tom Lane
Jeff Davis writes: > One thing I don't fully understand is what would happen if the client > issued the Sync as the *first* message in an extended-protocol series. That'd cause the backend to send ReadyForQuery, which'd likely confuse the client. > But I think you're correct in saying that the d

Re: A qsort template

2021-06-16 Thread Thomas Munro
Hi Zhihong, On Thu, Jun 17, 2021 at 8:13 AM Zhihong Yu wrote: > In 0001-Add-bsearch-and-unique-templates-to-sort_template.h.patch : > > - const ST_ELEMENT_TYPE * > ST_SORT_PROTO_ARG); > + const ST_ELEMENT_TYPE > *ST_SOR

Re: postgres_fdw batching vs. (re)creating the tuple slots

2021-06-16 Thread Tomas Vondra
On 6/16/21 4:23 PM, Tomas Vondra wrote: On 6/16/21 2:36 PM, Alexander Pyhalov wrote: Hi. It seems this commit commit b676ac443b6a83558d4701b2dd9491c0b37e17c4 Author: Tomas Vondra Date:   Fri Jun 11 20:19:48 2021 +0200 Optimize creation of slots for FDW bulk inserts has broken batch ins

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-06-16 Thread Andres Freund
Hi, On 2021-06-15 16:50:24 +0530, Dilip Kumar wrote: > The patch modifies both CREATE DATABASE and ALTER DATABASE..SET > TABLESPACE to be fully WAL-logged. Generally quite a bit in favor of this - the current approach is very heavyweight, slow and I think we have a few open corner bugs related to

Re: A qsort template

2021-06-16 Thread Zhihong Yu
On Wed, Jun 16, 2021 at 2:54 PM Thomas Munro wrote: > Hi Zhihong, > > On Thu, Jun 17, 2021 at 8:13 AM Zhihong Yu wrote: > > In 0001-Add-bsearch-and-unique-templates-to-sort_template.h.patch : > > > > - const ST_ELEMENT_TYPE * > ST_SORT_PROTO_ARG); > > +

Re: a path towards replacing GEQO with something better

2021-06-16 Thread John Naylor
On Wed, Jun 16, 2021 at 12:01 PM Robert Haas wrote: > > I feel like these are completely equivalent. Either way, the planner > is going to deduce that all the ".col" columns are equal to each other > via the equivalence class machinery, and then the subsequent planning > will be absolutely identic

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-06-16 Thread Andres Freund
Hi, On 2021-06-15 18:11:23 +0530, Dilip Kumar wrote: > On Tue, Jun 15, 2021 at 5:34 PM Heikki Linnakangas wrote: > > > > On 15/06/2021 14:20, Dilip Kumar wrote: > > > Design Idea: > . Then > > > we can get the relfilenode of every file we need to copy, and prepare > > > a list of all such relfile

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-06-16 Thread Tomas Vondra
On 6/15/21 3:31 PM, Andrew Dunstan wrote: On 6/15/21 8:04 AM, Heikki Linnakangas wrote: Yeah, WAL-logging the contents of the source database would certainly be less weird than the current system. As Julien also pointed out, the question is, are there people using on "CREATE DATABASE foo TE

Re: Unresolved repliaction hang and stop problem.

2021-06-16 Thread Alvaro Herrera
On 2021-Jun-16, Ha Ka wrote: > Here is the upload with generated reports: https://easyupload.io/p38izx > passwd: johS5jeewo OK, so I downloaded that and this is the interesting entry in the profile for the broken case: # Samples: 5K of event 'cpu-clock' # Event count (approx.): 59989898390 # #

Re: Question about StartLogicalReplication() error path

2021-06-16 Thread Jeff Davis
On Tue, 2021-06-15 at 15:19 +0900, Kyotaro Horiguchi wrote: > I don't think the message is neded, but I don't oppose it as far as > the level is LOG and the messages were changed as something like > this: > > > - elog(DEBUG1, "cannot stream from %X/%X, minimum is > %X/%X, forwarding

Re: Outdated replication protocol error?

2021-06-16 Thread Jeff Davis
On Fri, 2021-01-15 at 13:55 -0800, Andres Freund wrote: > > > We should either: > > > > > > 1. Document that IDENTIFY_SYSTEM must always be run before > > > START_REPLICATION, and always issue a WARNING if that's not done > > > (an > > > ERROR might break existing applications); or > > > > > > 2.

Re: Split xlog.c

2021-06-16 Thread Andres Freund
Hi, On 2021-06-16 16:30:45 +0300, Heikki Linnakangas wrote: > xlog.c is very large. We've split off some functions from it over the years, > but it's still large and it keeps growing. > > Attached is a proposal to split functions related to WAL replay, standby > mode, fetching files from archive,

Re: Outdated replication protocol error?

2021-06-16 Thread Andres Freund
Hi, On 2021-06-16 15:59:11 -0700, Jeff Davis wrote: > [ digging up old thread ] > > It seems everyone agrees that the current behavior is strange. Yea. I don't remember the details, but I've also hit this problem since in some odd circumstance while reviewing the "logical decoding on standbys" p

Re: PoC: Using Count-Min Sketch for join cardinality estimation

2021-06-16 Thread John Naylor
On Wed, Jun 16, 2021 at 12:23 PM Tomas Vondra wrote: > The attached patch is a very simple (and perhaps naive) implementation > adding count-min sketch to pg_statistic for all attributes with a hash > function (as a new statistics slot kind), and considering it in > equijoinsel_inner. There's a G

Re: A qsort template

2021-06-16 Thread Tom Lane
Thomas Munro writes: > Hmm, well it was only recently damaged by commit def5b065, and that's > because I'd forgotten to put ST_ELEMENT_TYPE into typedefs.list, and I > was correcting that in this patch. If ST_ELEMENT_TYPE isn't recognized as a typedef by the buildfarm's typedef collectors, this s

  1   2   >