Re: problem with RETURNING and update row movement

2020-07-21 Thread Amit Langote
Hi Amit, Thanks for taking a look at this. On Mon, Jul 20, 2020 at 8:35 PM Amit Kapila wrote: > IIUC, here the problem is related to below part of code: > ExecInsert(..) > { > /* Process RETURNING if present */ > if (resultRelInfo->ri_projectReturning) > result = ExecProcessReturning(resultRelIn

RE: Implement UNLOGGED clause for COPY FROM

2020-07-21 Thread osumi.takami...@fujitsu.com
Hi. Amit-san > If you are going to suggest users not to replicate such tables then why can't > you > suggest them to create such tables as UNLOGGED in the first place? Another > idea could be that you create an 'unlogged' > table, copy the data to it. Then perform Alter Table .. SET Logged and

Re: OpenSSL randomness seeding

2020-07-21 Thread Michael Paquier
On Tue, Jul 21, 2020 at 10:00:20PM -0700, Noah Misch wrote: > These look good. I'll push them on Saturday or later. I wondered whether to > do both RAND_cleanup() and RAND_poll(), to purge all traces of the old seed on > versions supporting both. Since that would strictly (albeit negligibly) > i

Re: Parallel Seq Scan vs kernel read ahead

2020-07-21 Thread David Rowley
On Wed, 22 Jul 2020 at 16:40, k.jami...@fujitsu.com wrote: > I used the default max_parallel_workers & max_worker_proceses which is 8 by > default in postgresql.conf. > IOW, I ran all those tests with maximum of 8 processes set. But my query > planner capped both the > Workers Planned and Launch

Re: OpenSSL randomness seeding

2020-07-21 Thread Noah Misch
On Tue, Jul 21, 2020 at 02:13:32PM +0200, Daniel Gustafsson wrote: > The silver lining here is that while OpenSSL nooped RAND_cleanup, they also > changed what is mixed into seeding so we are still not sharing a sequence. To > fix this, changing the RAND_cleanup call to RAND_poll should be enough

Re: xl_heap_header alignment?

2020-07-21 Thread Antonin Houska
Tom Lane wrote: > I don't particularly want to remove the field, but we ought to > change or remove the comment. I'm not concerned about the existence of the field as well. The comment just made me worried that I might be missing some fundamental concept. Thanks for your opinion. -- Antonin Ho

Re: Parallel Seq Scan vs kernel read ahead

2020-07-21 Thread Thomas Munro
On Wed, Jul 22, 2020 at 3:57 PM Amit Kapila wrote: > Yeah, that is true but every time before the test the same amount of > data should be present in shared buffers (or OS cache) if any which > will help in getting consistent results. However, it is fine to > reboot the machine as well if that is

Re: Parallel Seq Scan vs kernel read ahead

2020-07-21 Thread Amit Kapila
On Wed, Jul 22, 2020 at 5:25 AM David Rowley wrote: > > I understand that Amit wrote: > > On Fri, 17 Jul 2020 at 21:18, Amit Kapila wrote: > > I think recreating the database and restarting the server after each > > run might help in getting consistent results. Also, you might want to > > take m

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-07-21 Thread Amit Kapila
On Mon, Jul 20, 2020 at 6:46 PM Dilip Kumar wrote: > > There was one warning in release mode in the last version in 0004 so > attaching a new version. > Today, I was reviewing patch v38-0001-WAL-Log-invalidations-at-command-end-with-wal_le and found a small problem with it. + /* + * Execute the

RE: archive status ".ready" files may be created too early

2020-07-21 Thread matsumura....@fujitsu.com
Hello, > At Mon, 13 Jul 2020 01:57:36 +, "Kyotaro Horiguchi > " wrote in > Am I missing something here? I write more detail(*). Record-A and Record-B are cross segment-border records. Record-A spans segment X and X+1. Record-B spans segment X+2 and X+3. If both records have been ins

Re: Stale external URL in doc?

2020-07-21 Thread Kyotaro Horiguchi
At Sat, 18 Jul 2020 22:48:47 +0900, Michael Paquier wrote in > On Fri, Jul 17, 2020 at 02:03:18PM +0900, Michael Paquier wrote: > > It would be better to get all that fixed and backpatched. Is somebody > > already looking into that? > > I have been through this set, and applied the changes as o

Re: OpenSSL randomness seeding

2020-07-21 Thread Michael Paquier
On Tue, Jul 21, 2020 at 10:36:53PM +0200, Daniel Gustafsson wrote: > I think the original intention was to handle older OpenSSL versions where > multiple successful RAND_poll calls were required for RAND_status to succeed, > the check working as an optimization since a failing RAND_poll would rende

Re: Comment referencing incorrect algorithm

2020-07-21 Thread Michael Paquier
On Tue, Jul 21, 2020 at 01:57:11PM +0200, Daniel Gustafsson wrote: > While poking around our crypto code, I noticed that a comment in sha2.h was > referencing sha-1 which is an algorithm not supported by the code. The > attached fixes the comment aligning it with other comments in the file. Thank

Re: v13 planner ERROR: could not determine which collation to use for string comparison

2020-07-21 Thread Tom Lane
Michael Paquier writes: > On Tue, Jul 21, 2020 at 06:25:00PM -0400, Tom Lane wrote: >> Ugh. It's clear from your stack trace that neqjoinsel() has forgotten to >> pass through collation to eqjoinsel(). Will fix. > Why didn't you include a regression test in bd0d893? Didn't really see much poin

Re: v13 planner ERROR: could not determine which collation to use for string comparison

2020-07-21 Thread Michael Paquier
On Tue, Jul 21, 2020 at 06:25:00PM -0400, Tom Lane wrote: > Ugh. It's clear from your stack trace that neqjoinsel() has forgotten to > pass through collation to eqjoinsel(). Will fix. Why didn't you include a regression test in bd0d893? -- Michael signature.asc Description: PGP signature

Re: Which SET TYPE don't actually require a rewrite

2020-07-21 Thread Michael Paquier
On Tue, Jul 21, 2020 at 04:55:37PM -0400, Bruce Momjian wrote: > I know Tom put a wink on that, but I actually do feel that the only > clean way to do this is to give users a way to issue the query in a > non-executing way that will report if a rewrite is going to happen. Yeah, when doing a schema

Re: Parallel Seq Scan vs kernel read ahead

2020-07-21 Thread David Rowley
Hi Kirk, Thank you for doing some testing on this. It's very useful to get some samples from other hardware / filesystem / os combinations. On Tue, 21 Jul 2020 at 21:38, k.jami...@fujitsu.com wrote: > Query Planner I/O Timings (ms): > | Worker | I/O READ (Master) | I/O READ (Patch) | I/O WRITE (

Re: Index Skip Scan (new UniqueKeys)

2020-07-21 Thread Peter Geoghegan
On Sat, Jul 11, 2020 at 9:10 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > + currItem = &so->currPos.items[so->currPos.lastItem]; > > > + itup = (IndexTuple) (so->currTuples + > > > currItem->tupleOffset); > > > + nextOffset = ItemPointerGetOffsetN

Re: Using Valgrind to detect faulty buffer accesses (no pin or buffer content lock held)

2020-07-21 Thread Peter Geoghegan
On Fri, Jul 17, 2020 at 5:53 PM Peter Geoghegan wrote: > Pushed the first patch just now, and intend to push the other one soon. > Thanks! Pushed the second piece of this (the nbtree patch) just now. Thanks for the review! -- Peter Geoghegan

Re: v13 planner ERROR: could not determine which collation to use for string comparison

2020-07-21 Thread Tom Lane
Justin Pryzby writes: > We hit this on v13b2 and verified it fails on today's HEAD (ac25e7b039). > explain SELECT 1 FROM sites NATURAL JOIN sectors WHERE sites.config_site_name > != sectors.sect_name ; > ERROR: could not determine which collation to use for string comparison > I can workaround

Re: Infinities in type numeric

2020-07-21 Thread Tom Lane
I wrote: > Dean Rasheed writes: >> I had a look at this, and I think it's mostly in good shape. It looks >> like everything from the first message in this thread has been >> resolved, except I don't know about the jsonpath stuff, because I >> haven't been following that. > Thanks for the careful

Re: Using Valgrind to detect faulty buffer accesses (no pin or buffer content lock held)

2020-07-21 Thread Peter Geoghegan
On Mon, Jul 6, 2020 at 1:35 AM Georgios Kokolatos wrote: > As a general overview, the series of patches in the mail thread do match > their description. The addition of the stricter, explicit use of > instrumentation does improve the design as the distinction of the use cases > requiring a pin

Re: Which SET TYPE don't actually require a rewrite

2020-07-21 Thread Bruce Momjian
On Fri, Jul 17, 2020 at 11:26:56AM -0400, Tom Lane wrote: > Magnus Hagander writes: > > As Amit mentions it is also triggered by some store parameter changes. But > > not all. So looking at it the other way, the part that the end user really > > cares about it "which ALTER TABLE operations will re

Re: Improving psql slash usage help message

2020-07-21 Thread Peter Eisentraut
On 2020-07-21 17:10, Tom Lane wrote: Hamid Akhtar writes: So are you suggesting to not fix this or do a more detailed review and assess what other psql messages can be grouped together. I was just imagining merging the entries for the commands that are implemented by listTables(). If you see

Re: OpenSSL randomness seeding

2020-07-21 Thread Daniel Gustafsson
> On 21 Jul 2020, at 22:00, David Steele wrote: > > On 7/21/20 3:44 PM, Daniel Gustafsson wrote: >>> On 21 Jul 2020, at 17:31, David Steele wrote: >>> On 7/21/20 8:13 AM, Daniel Gustafsson wrote: Another thing that stood out when reviewing this code is that we optimize for RAND_p

Re: Default setting for enable_hashagg_disk

2020-07-21 Thread Bruce Momjian
On Tue, Jul 14, 2020 at 03:49:40PM -0700, Peter Geoghegan wrote: > Maybe I missed your point here. The problem is not so much that we'll > get HashAggs that spill -- there is nothing intrinsically wrong with > that. While it's true that the I/O pattern is not as sequential as a > similar group agg

Re: OpenSSL randomness seeding

2020-07-21 Thread David Steele
On 7/21/20 3:44 PM, Daniel Gustafsson wrote: On 21 Jul 2020, at 17:31, David Steele wrote: On 7/21/20 8:13 AM, Daniel Gustafsson wrote: Another thing that stood out when reviewing this code is that we optimize for RAND_poll failing in pg_strong_random, when we already have RAND_status checkin

Re: v13 planner ERROR: could not determine which collation to use for string comparison

2020-07-21 Thread Justin Pryzby
Reproducer: postgres=# CREATE TABLE t AS SELECT ''a FROM generate_series(1,99); CREATE TABLE u AS SELECT ''a FROM generate_series(1,99) ; VACUUM ANALYZE t,u; postgres=# explain SELECT * FROM t JOIN u ON t.a!=u.a; ERROR: could not determine which collation to use for string comparison HINT: Use

Re: OpenSSL randomness seeding

2020-07-21 Thread Daniel Gustafsson
> On 21 Jul 2020, at 17:31, David Steele wrote: > On 7/21/20 8:13 AM, Daniel Gustafsson wrote: >> Another thing that stood out when reviewing this code is that we optimize for >> RAND_poll failing in pg_strong_random, when we already have RAND_status >> checking for a sufficiently seeded RNG for

Re: v13 planner ERROR: could not determine which collation to use for string comparison

2020-07-21 Thread David G. Johnston
On Tuesday, July 21, 2020, Justin Pryzby wrote: > We hit this on v13b2 and verified it fails on today's HEAD (ac25e7b039). > > explain SELECT 1 FROM sites NATURAL JOIN sectors WHERE > sites.config_site_name != sectors.sect_name ; > ERROR: could not determine which collation to use for string com

v13 planner ERROR: could not determine which collation to use for string comparison

2020-07-21 Thread Justin Pryzby
We hit this on v13b2 and verified it fails on today's HEAD (ac25e7b039). explain SELECT 1 FROM sites NATURAL JOIN sectors WHERE sites.config_site_name != sectors.sect_name ; ERROR: could not determine which collation to use for string comparison I can workaround the issue by DELETEing stats fo

Re: xl_heap_header alignment?

2020-07-21 Thread Tom Lane
Andres Freund writes: > On July 21, 2020 10:45:37 AM PDT, Antonin Houska wrote: >> I don't quite understand this part of the comment of the xl_heap_header >> structure: >> * NOTE: t_hoff could be recomputed, but we may as well store it because >> * it will come for free due to alignment considera

Re: xl_heap_header alignment?

2020-07-21 Thread Andres Freund
Hi, On July 21, 2020 10:45:37 AM PDT, Antonin Houska wrote: >I don't quite understand this part of the comment of the xl_heap_header >structure: > >* NOTE: t_hoff could be recomputed, but we may as well store it because > * it will come for free due to alignment considerations. > >What are the a

xl_heap_header alignment?

2020-07-21 Thread Antonin Houska
I don't quite understand this part of the comment of the xl_heap_header structure: * NOTE: t_hoff could be recomputed, but we may as well store it because * it will come for free due to alignment considerations. What are the alignment considerations? The WAL code does not appear to assume any a

RE: Parallel Seq Scan vs kernel read ahead

2020-07-21 Thread k.jami...@fujitsu.com
On Friday, July 17, 2020 6:18 PM (GMT+9), Amit Kapila wrote: > On Fri, Jul 17, 2020 at 11:35 AM k.jami...@fujitsu.com > wrote: > > > > On Wednesday, July 15, 2020 12:52 PM (GMT+9), David Rowley wrote: > > > > >On Wed, 15 Jul 2020 at 14:51, Amit Kapila wrote: > > >> > > >> On Wed, Jul 15, 2020 at

RE: Parallel Seq Scan vs kernel read ahead

2020-07-21 Thread k.jami...@fujitsu.com
On Tuesday, July 21, 2020 12:18 PM, Amit Kapila wrote: > On Tue, Jul 21, 2020 at 8:06 AM k.jami...@fujitsu.com > wrote: > > > > Thank you for the advice. I repeated the test as per your advice and > > average of 3 runs per worker/s planned. > > It still shows the following similar performance resu

Re: pg_subscription.subslotname is wrongly marked NOT NULL

2020-07-21 Thread Tom Lane
I wrote: > * On the other side of the ledger, if we don't fix these markings > we cannot back-patch the additional assertions I proposed at [1]. > I'm kind of leaning to committing this as shown and back-patching > the patch at [1], but certainly a case could be made in the other > direction. Tho

Re: Improve handling of pg_stat_statements handling of bind "IN" variables

2020-07-21 Thread Dmitry Dolgov
> On Thu, Oct 3, 2019 at 3:33 AM Pavel Trukhanov > wrote: > >> On Wed, Jun 26, 2019 at 11:10 PM Tom Lane wrote: >> >> Greg Stark writes: >> > Actually thinking about this for two more seconds the question is what it >> > would do with a query like >> > WHERE col = ANY '1,2,3'::integer[] >> > Or

Re: OpenSSL randomness seeding

2020-07-21 Thread David Steele
On 7/21/20 8:13 AM, Daniel Gustafsson wrote: After forking we call RAND_cleanup in fork_process.c to force a re-seed to ensure that two backends cannot share sequence. OpenSSL 1.1.0 deprecated RAND_cleanup, and contrary to how they usually leave deprecated APIs working until removed, they decide

Re: Improving psql slash usage help message

2020-07-21 Thread Tom Lane
Hamid Akhtar writes: > So are you suggesting to not fix this or do a more detailed review and > assess what other psql messages can be grouped together. I was just imagining merging the entries for the commands that are implemented by listTables(). If you see something else that would be worth d

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-07-21 Thread Dilip Kumar
On Tue, Jul 21, 2020 at 4:08 PM Dilip Kumar wrote: > > On Tue, Jul 21, 2020 at 11:00 AM Dilip Kumar wrote: > > > > On Tue, Jul 21, 2020 at 2:00 AM Andres Freund wrote: > > > > > > Hi, > > > > > > On 2020-07-17 16:16:23 +0530, Dilip Kumar wrote: > > > > The attached patch allows the vacuum to con

Re: Postgres-native method to identify if a tuple is frozen

2020-07-21 Thread Amit Kapila
On Mon, Jul 20, 2020 at 9:07 PM Lawrence Jones wrote: > > > So we hit the question: how can we identify if a tuple is frozen? I know the > tuple has both committed and aborted hint bits set, but accessing those bits > seems to require superuser functions and are unlikely to be that fast. > > Are

OpenSSL randomness seeding

2020-07-21 Thread Daniel Gustafsson
After forking we call RAND_cleanup in fork_process.c to force a re-seed to ensure that two backends cannot share sequence. OpenSSL 1.1.0 deprecated RAND_cleanup, and contrary to how they usually leave deprecated APIs working until removed, they decided to silently make this call a noop like below:

Comment referencing incorrect algorithm

2020-07-21 Thread Daniel Gustafsson
While poking around our crypto code, I noticed that a comment in sha2.h was referencing sha-1 which is an algorithm not supported by the code. The attached fixes the comment aligning it with other comments in the file. cheers ./daniel sha1_comment.diff Description: Binary data

Re: Improving psql slash usage help message

2020-07-21 Thread Hamid Akhtar
So are you suggesting to not fix this or do a more detailed review and assess what other psql messages can be grouped together. On Sun, Jul 12, 2020 at 8:15 PM Tom Lane wrote: > Hamid Akhtar writes: > > On Sun, Apr 26, 2020 at 1:03 AM David G. Johnston < > > david.g.johns...@gmail.com> wrote: >

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-07-21 Thread Dilip Kumar
On Tue, Jul 21, 2020 at 11:00 AM Dilip Kumar wrote: > > On Tue, Jul 21, 2020 at 2:00 AM Andres Freund wrote: > > > > Hi, > > > > On 2020-07-17 16:16:23 +0530, Dilip Kumar wrote: > > > The attached patch allows the vacuum to continue by emitting WARNING > > > for the corrupted tuple instead of imm

Re: Parallel Seq Scan vs kernel read ahead

2020-07-21 Thread Amit Kapila
On Tue, Jul 21, 2020 at 3:08 PM k.jami...@fujitsu.com wrote: > > On Tuesday, July 21, 2020 12:18 PM, Amit Kapila wrote: > > On Tue, Jul 21, 2020 at 8:06 AM k.jami...@fujitsu.com > > > > wrote: > > > > > > I am definitely missing something. Perhaps I think I could not > > > understand why there's

Re: Parallel copy

2020-07-21 Thread Amit Kapila
On Fri, Jul 17, 2020 at 2:09 PM vignesh C wrote: > > > > > Please find the updated patch with the fixes included. > > > > Patch 0003-Allow-copy-from-command-to-process-data-from-file-ST.patch > had few indentation issues, I have fixed and attached the patch for > the same. > Ensure to use the ver

Re: Auto-vectorization speeds up multiplication of large-precision numerics

2020-07-21 Thread Amit Khandekar
On Mon, 13 Jul 2020 at 14:27, Amit Khandekar wrote: > I tried this in utils/adt/Makefile : > + > +numeric.o: CFLAGS += ${CFLAGS_VECTOR} > + > and it works. > > CFLAGS_VECTOR also includes the -funroll-loops option, which I > believe, had showed improvements in the checksum.c runs ( [1] ). This > o

RE: Performing partition pruning using row value

2020-07-21 Thread kato-...@fujitsu.com
>So, after looking at these functions and modifying this patch, I would like to >add this patch to the next I updated this patch and registered for the next CF . https://commitfest.postgresql.org/29/2654/ regards, sho kato pruning-with-row-wise-comparison-v2.patch Description: pruning-with-r

Re: Wrong results from in_range() tests with infinite offset

2020-07-21 Thread Dean Rasheed
On Tue, 21 Jul 2020 at 03:06, Tom Lane wrote: > > Pushed, but I chickened out of back-patching. The improvement in what > happens for finite comparison values seems somewhat counterbalanced by > the possibility that someone might not like the definition we arrived > at for infinities. So, it's n

WAL segment switch on pg_start_backup()

2020-07-21 Thread @usernamedt
Hi, I am currently exploring the pg_start_backup() and pg_stop_backup() functions. In the documentation (https://www.postgresql.org/docs/9.0/functions-admin.html), it is stated that after calling pg_stop_backup() Postgres switches to the new WAL segment file. But it doesn’t say the same for pg