Re: Replacing the EDH SKIP primes

2019-06-18 Thread Michael Paquier
On Wed, Jun 19, 2019 at 07:44:46AM +0200, Daniel Gustafsson wrote: > I think this is v13 material, I’ll stick it in the next commitfest. Thanks! -- Michael signature.asc Description: PGP signature

Re: Replacing the EDH SKIP primes

2019-06-18 Thread Daniel Gustafsson
> On 19 Jun 2019, at 05:40, Michael Paquier wrote: > Fine by me. Let's stick with the 2048b-long one for now as we did in > c0a15e0. I am wondering if we should sneak that into v12, but I'd > rather just wait for v13 to open. I think this is v13 material, I’ll stick it in the next commitfest.

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-18 Thread Ian Barwick
n 6/18/19 12:41 AM, Stephen Frost wrote: > Greetings, > > * Ian Barwick (ian.barw...@2ndquadrant.com) wrote (...) >> I suggest explicitly documenting postgresql.auto.conf behaviour (and the circumstances >> where it's acceptable to modify it outside of ALTER SYSTEM calls) in the documentation

Some reloptions non-initialized when loaded

2019-06-18 Thread Michael Paquier
Hi all, While looking at this code, I have noticed that a couple of reloptions which are not toast-specific don't get properly initialized. toast_tuple_target and parallel_workers are the ones standing out. Thoughts? -- Michael diff --git a/src/backend/access/common/reloptions.c

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-18 Thread Ian Barwick
On 6/19/19 12:46 PM, Amit Kapila wrote: On Mon, Jun 17, 2019 at 8:20 PM Ian Barwick wrote: On 6/15/19 1:08 AM, Stephen Frost wrote: > * Amit Kapila (amit.kapil...@gmail.com) wrote: >> Right. I think if possible, it should use existing infrastructure to >> write to postgresql.auto.conf

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-18 Thread Amit Kapila
On Mon, Jun 17, 2019 at 8:20 PM Ian Barwick wrote: > On 6/15/19 1:08 AM, Stephen Frost wrote: > > * Amit Kapila (amit.kapil...@gmail.com) wrote: > >> Right. I think if possible, it should use existing infrastructure to > >> write to postgresql.auto.conf rather than inventing a new way to >

Re: Replacing the EDH SKIP primes

2019-06-18 Thread Michael Paquier
On Tue, Jun 18, 2019 at 02:05:00PM +0200, Daniel Gustafsson wrote: > The current hardcoded EDH parameter fallback use the old SKIP primes, for > which > the source disappeared from the web a long time ago. Referencing a known dead > source seems a bit silly, so I think we should either switch to

Re: openssl valgrind failures on skink are due to openssl issue

2019-06-18 Thread Michael Paquier
On Tue, Jun 18, 2019 at 07:44:26PM -0700, Andres Freund wrote: > Really, I can do that? Here is some of the stuff I use, just for the reference: ./Configure linux-x86_64 --prefix=$HOME/stable/openssl/1.1.1/ ./config --prefix=$HOME/stable/openssl/1.1.1 shared -- Michael signature.asc

Re: fix "Success" error messages

2019-06-18 Thread Michael Paquier
On Tue, Jun 18, 2019 at 09:13:19AM -0700, Shawn Debnath wrote: >> case SLRU_WRITE_FAILED: >> ereport(ERROR, >> (errcode_for_file_access(), >> errmsg("could not access status of >>

Re: openssl valgrind failures on skink are due to openssl issue

2019-06-18 Thread Andres Freund
On 2019-06-19 11:28:16 +0900, Michael Paquier wrote: > On Tue, Jun 18, 2019 at 04:34:07PM -0700, Andres Freund wrote: > > It's merged to both branches that contain the broken code. Now we need > > to wait for the next set of openssl releases, and then for distros to > > pick that up. Based on the

RE: [PATCH] memory leak in ecpglib

2019-06-18 Thread kuroda.hay...@fujitsu.com
Dear Zhang, Sorry for my late reply. I'm now planning to refactor this functionality: https://www.postgresql.org/message-id/osapr01mb20048298f882d25897c6ab23f5...@osapr01mb2004.jpnprd01.prod.outlook.com If DECLARE STATEMENT and other related statements are enabled only preprocessing process,

Re: openssl valgrind failures on skink are due to openssl issue

2019-06-18 Thread Michael Paquier
On Tue, Jun 18, 2019 at 04:34:07PM -0700, Andres Freund wrote: > It's merged to both branches that contain the broken code. Now we need > to wait for the next set of openssl releases, and then for distros to > pick that up. Based on the past release cadence >

Re: pg_upgrade: Improve invalid option handling

2019-06-18 Thread Michael Paquier
On Tue, Jun 18, 2019 at 10:25:44AM +0200, Daniel Gustafsson wrote: > Correct, that matches how pg_basebackup and psql does it. Perhaps you have a patch at hand? I can see four strings in pg_upgrade, two in exec.c and two in option.c, which could be improved. -- Michael signature.asc

Re: Still some references to configure-time WAL segment size option in msvc scripts

2019-06-18 Thread Michael Paquier
On Mon, Jun 17, 2019 at 04:32:28PM +0900, Michael Paquier wrote: > I have just bumped into $subject, which makes no sense now as this is > an init-time option. Any objections if I remove this code as per the > attached? And committed. -- Michael signature.asc Description: PGP signature

Re: nbtdesc.c and nbtpage.c are inconsistent with XLOG_BTREE_META_CLEANUP (11~)

2019-06-18 Thread Michael Paquier
On Sun, Jun 16, 2019 at 07:14:05PM -0700, Peter Geoghegan wrote: > The WAL record in question, XLOG_BTREE_META_CLEANUP, is certainly one > of the less common record types used by nbtree. I agree that this > should have been tested when it went in, but I'm not surprised that > the bug remained

Tweaking DSM and DSA limits

2019-06-18 Thread Thomas Munro
Hello, If you run a lot of parallel queries that use big parallel hash joins simultaneously, you can run out of DSM slots (for example, when testing many concurrent parallel queries). That's because we allow 64 slots + 2 * MaxBackends, but allocating seriously large amounts of dynamic shared

Re: ANALYZE: ERROR: tuple already updated by self

2019-06-18 Thread Andres Freund
Hi, On June 18, 2019 5:38:34 PM PDT, Justin Pryzby wrote: >On Tue, Jun 18, 2019 at 06:48:58PM -0500, Justin Pryzby wrote: >> On Tue, Jun 18, 2019 at 06:12:33PM -0500, Justin Pryzby wrote: >> > A customers DB crashed due to OOM. While investigating the issue >in our >> > report, I created MV

Re: ANALYZE: ERROR: tuple already updated by self

2019-06-18 Thread Justin Pryzby
On Tue, Jun 18, 2019 at 06:48:58PM -0500, Justin Pryzby wrote: > On Tue, Jun 18, 2019 at 06:12:33PM -0500, Justin Pryzby wrote: > > A customers DB crashed due to OOM. While investigating the issue in our > > report, I created MV stats, which causes this error: > > > > ts=# CREATE STATISTICS

Re: ANALYZE: ERROR: tuple already updated by self

2019-06-18 Thread Peter Geoghegan
On Tue, Jun 18, 2019 at 5:09 PM Andres Freund wrote: > > It might be interesting to set a breakpoint within heap_update(), > > which is called by simple_heap_update() --technically, this is where > > the reported failure occurs. From there, you could send an image of > > the page to the list by

Re: ANALYZE: ERROR: tuple already updated by self

2019-06-18 Thread Tom Lane
Andres Freund writes: > I think the problem is pretty plainly that for inheritance tables we'll > try to store extended statistics twice. And thus end up updating the > same row twice. They shouldn't be the same row though. If we're to try to capture ext-stats on inheritance trees --- and I

Re: ANALYZE: ERROR: tuple already updated by self

2019-06-18 Thread Andres Freund
Hi, On 2019-06-18 17:00:09 -0700, Peter Geoghegan wrote: > On Tue, Jun 18, 2019 at 4:49 PM Justin Pryzby wrote: > > Sure: > > > > (gdb) bt > > #0 errfinish (dummy=0) at elog.c:414 > > #1 0x0085e834 in elog_finish (elevel=, > > fmt=) at elog.c:1376 > > #2 0x004b93bd in

Re: ANALYZE: ERROR: tuple already updated by self

2019-06-18 Thread Andres Freund
Hi, On 2019-06-18 18:48:58 -0500, Justin Pryzby wrote: > Ah: the table is an inheritence parent. If I uninherit its child, there's no > error during ANALYZE. MV stats on the child are ok: It's a "classical" inheritance parent, not a builtin-partitioning type of parent, right? And it contains

Re: ANALYZE: ERROR: tuple already updated by self

2019-06-18 Thread Peter Geoghegan
On Tue, Jun 18, 2019 at 4:49 PM Justin Pryzby wrote: > Sure: > > (gdb) bt > #0 errfinish (dummy=0) at elog.c:414 > #1 0x0085e834 in elog_finish (elevel=, > fmt=) at elog.c:1376 > #2 0x004b93bd in simple_heap_update (relation=0x7fee161700c8, > otid=0x1fb7f44, tup=0x1fb7f40) at

Re: ANALYZE: ERROR: tuple already updated by self

2019-06-18 Thread Justin Pryzby
On Tue, Jun 18, 2019 at 06:48:58PM -0500, Justin Pryzby wrote: > On Tue, Jun 18, 2019 at 06:12:33PM -0500, Justin Pryzby wrote: > > ts=# ANALYZE sectors; > > ERROR: XX000: tuple already updated by self > > LOCATION: simple_heap_update, heapam.c:4613 > Ah: the table is an inheritence parent. If

Re: ANALYZE: ERROR: tuple already updated by self

2019-06-18 Thread Justin Pryzby
On Tue, Jun 18, 2019 at 06:12:33PM -0500, Justin Pryzby wrote: > A customers DB crashed due to OOM. While investigating the issue in our > report, I created MV stats, which causes this error: > > ts=# CREATE STATISTICS sectors_stats (dependencies) ON site_id,sect_id FROM > sectors; > CREATE

Re: ANALYZE: ERROR: tuple already updated by self

2019-06-18 Thread Andres Freund
Hi, On 2019-06-18 18:12:33 -0500, Justin Pryzby wrote: > A customers DB crashed due to OOM. While investigating the issue in our > report, I created MV stats, which causes this error: > > ts=# CREATE STATISTICS sectors_stats (dependencies) ON site_id,sect_id FROM > sectors; > CREATE STATISTICS

Re: openssl valgrind failures on skink are due to openssl issue

2019-06-18 Thread Tom Lane
Andres Freund writes: > What we could do is add a suppression like: > { >broken-openssl-accesses-random >Memcheck:Cond >... >fun:pg_strong_random >fun:InitProcessGlobals >fun:PostmasterMain >fun:main > } > (alternatively one suppression for each RAND_status,

ANALYZE: ERROR: tuple already updated by self

2019-06-18 Thread Justin Pryzby
A customers DB crashed due to OOM. While investigating the issue in our report, I created MV stats, which causes this error: ts=# CREATE STATISTICS sectors_stats (dependencies) ON site_id,sect_id FROM sectors; CREATE STATISTICS ts=# ANALYZE sectors; ERROR: XX000: tuple already updated by self

Re: openssl valgrind failures on skink are due to openssl issue

2019-06-18 Thread Andres Freund
Hi, On 2019-06-11 14:07:29 -0700, Andres Freund wrote: > On 2019-06-11 16:55:28 -0400, Tom Lane wrote: > > Andres Freund writes: > > > I can't think of a better way to fix skink for now than just disabling > > > openssl for skink, until 1.1.1d is released. > > > > Couldn't you install a local

Re: PG 12 beta 1 segfault during analyze

2019-06-18 Thread Andres Freund
Hi, On 2019-06-17 21:46:02 -0400, Steve Singer wrote: > On 6/15/19 10:18 PM, Tom Lane wrote: > > Steve Singer writes: > > > I encountered the following segfault when running against a PG 12 beta1 > > > during a analyze against a table. > > Nobody else has reported this, so you're going to have

Re: pgsql: Avoid spurious deadlocks when upgrading a tuple lock

2019-06-18 Thread Alvaro Herrera
On 2019-Jun-18, Oleksii Kliukin wrote: > Sorry, I was confused, as I was looking only at > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=de87a084c0a5ac927017cd0834b33a932651cfc9 > > without taking your subsequent commit that silences compiler warnings at >

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-06-18 Thread Melanie Plageman
On Thu, Jun 13, 2019 at 7:10 AM Robert Haas wrote: > On Tue, Jun 11, 2019 at 2:35 PM Melanie Plageman > wrote: > > Let me try to articulate what I think the bitmap implementation would > look > > like: > > > > Before doing chunked hashloop join for any batch, we would need to > > know how many

PL/Python fails on new NetBSD/PPC 8.0 install

2019-06-18 Thread Tom Lane
I noticed that the old NetBSD 5.1.5 installation I had on my G4 Mac was no longer passing our regression tests, because it has a strtof() that is sloppy about underflow. Rather than fight with that I decided to update it to something shinier (well, as shiny as you can get on hardware that's old

O(N^2) when building multi-column MCV lists

2019-06-18 Thread Tomas Vondra
Hi, When experimenting with multi-column MCV lists with statistic target set to high value (e.g. 10k), I've realized there's an O(N^2) issue in statext_mcv_build() when computing base frequencies. We do this: for (i = 0; i < nitems; i++) { ... item->base_frequency = 1.0;

Re: New EXPLAIN option: ALL

2019-06-18 Thread Peter Eisentraut
On 2019-05-15 19:58, Andres Freund wrote: > On 2019-05-15 13:53:26 -0400, Tom Lane wrote: >> FWIW, given the conflict against "EXPLAIN EXECUTE prepared_stmt_name", >> we should probably just drop the whole idea. It seemed like a great >> idea at the time, but it's going to confuse people not just

Choosing values for multivariate MCV lists

2019-06-18 Thread Tomas Vondra
Hi, The current implementation of multi-column MCV lists (added in this cycle) uses a fairly simple algorithm to pick combinations to include in the MCV list. We just compute a minimum number of occurences, and then include all entries sampled more often. See get_mincount_for_mcv_list(). By

Re: initdb recommendations

2019-06-18 Thread Peter Eisentraut
On 2019-05-23 18:54, Peter Eisentraut wrote: > To recap, the idea here was to change the default authentication methods > that initdb sets up, in place of "trust". > > I think the ideal scenario would be to use "peer" for local and some > appropriate password method (being discussed elsewhere)

Re: Valgrind failures in Apply Launcher's bgworker_quickdie() exit

2019-06-18 Thread Andres Freund
Hi, On 2018-12-17 15:35:01 -0800, Andres Freund wrote: > On 2018-12-16 13:48:00 -0800, Andres Freund wrote: > > On 2018-12-17 08:25:38 +1100, Thomas Munro wrote: > > > On Mon, Dec 17, 2018 at 7:57 AM Andres Freund wrote: > > > > The interesting bit is that if I replace the _exit(2) in > > > >

Re: idea: log_statement_sample_rate - bottom limit for sampling

2019-06-18 Thread Pavel Stehule
út 18. 6. 2019 v 14:03 odesílatel Adrien Nayrat napsal: > Hi, > > I tried the patch, here my comment: > > > gettext_noop("Zero effective disables sampling. " > > "-1 use sampling every time (without limit)."), > > I do not agree with the zero case. In fact, sampling is

Re: pgsql: Avoid spurious deadlocks when upgrading a tuple lock

2019-06-18 Thread Oleksii Kliukin
Alvaro Herrera wrote: > On 2019-Jun-16, Oleksii Kliukin wrote: > >> Alvaro Herrera wrote: >> >>> On 2019-Jun-14, Alvaro Herrera wrote: >>> I think there are worse problems here. I tried the attached isolation spec. Note that the only difference in the two permutations is that s0

Re: POC: Cleaning up orphaned files using undo logs

2019-06-18 Thread Robert Haas
On Tue, Jun 18, 2019 at 7:31 AM Amit Kapila wrote: > [ new patches ] I tried writing some code that throws an error from an undo log handler and the results were not good. It appears that the code will retry in a tight loop: 2019-06-18 13:58:53.262 EDT [42803] ERROR: robert_undo 2019-06-18

Re: Avoiding possible future conformance headaches in JSON work

2019-06-18 Thread Chapman Flack
On 6/18/19 12:51 PM, Oleg Bartunov wrote: >> have 'pg lax $.map(x => x + 10)'. > > This is exactly what we were thinking about ! Perfect! >> specify a POSIX re. Or, as like_regex has a named-parameter-like >> syntax--like_regex("abc" flag "i")--perhaps 'posix' should just be >> an extra keyword

Re: Avoiding possible future conformance headaches in JSON work

2019-06-18 Thread Oleg Bartunov
On Sat, 1 Jun 2019, 16:41 Chapman Flack, wrote: > Hi, > > We had a short conversation about this on Friday but I didn't have time > to think of a constructive suggestion, and now I've had more time to > think about it. > > Regarding the proposed PG 13 jsonpath extensions (array, map, and >

Re: pgsql: Avoid spurious deadlocks when upgrading a tuple lock

2019-06-18 Thread Alvaro Herrera
On 2019-Jun-16, Alvaro Herrera wrote: > So, I'm too lazy today to generate a case that fully reproduces the > deadlock, because you need to stall 's2' a little bit using the > well-known advisory lock trick, but this one hits the code that would > re-initialize the variable. Here's such a case.

Re: Avoiding possible future conformance headaches in JSON work

2019-06-18 Thread Alvaro Herrera
On 2019-Jun-01, Chapman Flack wrote: > In either case, perhaps we should immediately add a way to identify a > jsonpath as being PostgreSQL-extended. Maybe a keyword 'pg' that can > be accepted at the start in addition to any lax/strict, so you could > have 'pg lax $.map(x => x + 10)'. > > If we

Re: PG 12 beta 1 segfault during analyze

2019-06-18 Thread Tom Lane
Andres Freund writes: > On 2019-06-18 00:32:17 -0400, Tom Lane wrote: >> Hmm, that's a pretty obvious mistake :-( but after some fooling around >> I've not been able to cause a crash with it. I wonder what test case >> you were using, on what platform? > I suspect that's because the bug is

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-18 Thread Magnus Hagander
On Tue, Jun 18, 2019 at 3:37 PM Stephen Frost wrote: > Greetings, > > * Magnus Hagander (mag...@hagander.net) wrote: > > On Mon, Jun 17, 2019 at 5:41 PM Stephen Frost > wrote: > > > I'd further say something along the lines of 'utilities should not > > > modify a postgresql.auto.conf that's in

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-18 Thread Stephen Frost
Greetings, * Magnus Hagander (mag...@hagander.net) wrote: > On Mon, Jun 17, 2019 at 5:41 PM Stephen Frost wrote: > > I'd further say something along the lines of 'utilities should not > > modify a postgresql.auto.conf that's in place under a running PostgreSQL > > cluster'. > > Do we need to

fix "Success" error messages

2019-06-18 Thread Peter Eisentraut
In a case of a corrupted database, I saw an error message like Could not read from file ...: Success. from the SLRU module. This is because it checks that it reads or writes exactly BLCKSZ, and else goes to the error path. The attached patch gives a different error message in this case.

Replacing the EDH SKIP primes

2019-06-18 Thread Daniel Gustafsson
The current hardcoded EDH parameter fallback use the old SKIP primes, for which the source disappeared from the web a long time ago. Referencing a known dead source seems a bit silly, so I think we should either switch to a non-dead source of MODP primes or use an archive.org link for SKIP.

Re: idea: log_statement_sample_rate - bottom limit for sampling

2019-06-18 Thread Adrien Nayrat
Hi, I tried the patch, here my comment: > gettext_noop("Zero effective disables sampling. " > "-1 use sampling every time (without limit)."), I do not agree with the zero case. In fact, sampling is disabled as soon as setting is less than log_min_duration_statements.

Re: PG 12 beta 1 segfault during analyze

2019-06-18 Thread Steve Singer
On 6/18/19 12:32 AM, Tom Lane wrote: Steve Singer writes: The attached patch fixes the issue. Hmm, that's a pretty obvious mistake :-( but after some fooling around I've not been able to cause a crash with it. I wonder what test case you were using, on what platform?

Re: Batch insert in CTAS/MatView code

2019-06-18 Thread Paul Guo
On Mon, Jun 17, 2019 at 8:53 PM Paul Guo wrote: > Hi all, > > I've been working other things until recently I restarted the work, > profiling & refactoring the code. > It's been a long time since the last patch was proposed. The new patch has > now been firstly refactored due to >

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-06-18 Thread Magnus Hagander
On Mon, Jun 17, 2019 at 5:41 PM Stephen Frost wrote: > > * Ian Barwick (ian.barw...@2ndquadrant.com) wrote: > > On 6/15/19 1:08 AM, Stephen Frost wrote: > > > * Ian Barwick (ian.barw...@2ndquadrant.com) wrote: > > > >> Attached patch attempts to rectify this situation by having >

Re: pg_upgrade: Improve invalid option handling

2019-06-18 Thread Daniel Gustafsson
> On 18 Jun 2019, at 10:15, Michael Paquier wrote: > > On Fri, Jun 14, 2019 at 12:34:36PM +0200, Daniel Gustafsson wrote: >> +if ((log_opts.internal = fopen_priv(INTERNAL_LOG_FILE, "a")) == NULL) >> +pg_fatal("could not write to log file \"%s\"\n", >> INTERNAL_LOG_FILE); >> >>

Re: pg_upgrade: Improve invalid option handling

2019-06-18 Thread Michael Paquier
On Fri, Jun 14, 2019 at 12:34:36PM +0200, Daniel Gustafsson wrote: > + if ((log_opts.internal = fopen_priv(INTERNAL_LOG_FILE, "a")) == NULL) > + pg_fatal("could not write to log file \"%s\"\n", > INTERNAL_LOG_FILE); > > While we’re at it, should we change this to “could not open

Re: Generating partitioning tuple conversion maps faster

2019-06-18 Thread Michael Paquier
On Mon, Jun 17, 2019 at 12:03:03PM +0200, didier wrote: > cherry-pick apply cleanly and with a 100 columns table it improves > performance nicely (20%). 42f70cd is a performance improvement, and not a bug fix. -- Michael signature.asc Description: PGP signature

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2019-06-18 Thread shawn wang
Masahiko Sawada 于2019年6月17日周一 下午8:30写道: > On Fri, Jun 14, 2019 at 7:41 AM Tomas Vondra > wrote: > > I personally find the idea of encrypting tablespaces rather strange. > > Tablespaces are meant to define hysical location for objects, but this > > would also use them to "mark" objects as

Re: PG 12 beta 1 segfault during analyze

2019-06-18 Thread Andres Freund
Hi, On 2019-06-18 00:32:17 -0400, Tom Lane wrote: > Steve Singer writes: > > The attached patch fixes the issue. > > Hmm, that's a pretty obvious mistake :-( but after some fooling around > I've not been able to cause a crash with it. I wonder what test case > you were using, on what platform?

Re: Hash join explain is broken

2019-06-18 Thread Andres Freund
Hi, On 2019-06-13 16:23:34 -0700, Andres Freund wrote: > On June 13, 2019 3:38:47 PM PDT, Tom Lane wrote: > >Andres Freund writes: > >> I am too tired to look further into this. I suspect the only reason > >we > >> didn't previously run into trouble with the executor stashing > >hashkeys > >>