Re: Patch to clean Query after rewrite-and-analyze - reduces memusage up to 50% - increases TPS by up to 50%

2019-08-03 Thread Daniel Migowski
Am 03.08.2019 um 18:38 schrieb Tom Lane: Daniel Migowski writes: While examining the reasons for excessive memory usage in prepared statements I noticed that RTE_JOIN-kind RTEs contain a bunch of columnNames and joinaliasvars, that are irrelevant after the Query after has been rewritten. Uh,

Re: concerns around pg_lsn

2019-08-03 Thread Alvaro Herrera
On 2019-Aug-04, Jeevan Ladhe wrote: > Sure Michael, in the attached patch I have reverted the checks from > pg_lsn_in_internal() and added Assert() per my original patch. Can we please change the macro definition so that have_error is one of the arguments? Having the variable be used inside the

Re: concerns around pg_lsn

2019-08-03 Thread Jeevan Ladhe
Sure Michael, in the attached patch I have reverted the checks from pg_lsn_in_internal() and added Assert() per my original patch. Regards, Jeevan Ladhe 0001-Make-have_error-initialization-more-defensive-v2.patch Description: Binary data

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-03 Thread Peter Geoghegan
On Sat, Aug 3, 2019 at 7:48 PM Michael Paquier wrote: > Why choosing a random position within [8000,]? This leads to the > following messages for example with multiple runs, which is confusing: > Suggested random unused OID: 9473 (527 consecutive OID(s) available > Suggested random unused

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-03 Thread Michael Paquier
On Sat, Aug 03, 2019 at 11:40:24AM +0200, Julien Rouhaud wrote: > I probably write less perl than Michael, but it looks just fine to me. Indentation with pgperltidy complains with the attached diff (based on top of v4). +printf "Patches should use a more-or-less consecutive range of OIDs.\n";

Re: More refactoring for BuildIndexInfo

2019-08-03 Thread Michael Paquier
On Thu, Aug 01, 2019 at 01:13:22PM +0900, Michael Paquier wrote: > Any thoughts or objections? Hearing nothing, done. -- Michael signature.asc Description: PGP signature

Re: pglz performance

2019-08-03 Thread Petr Jelinek
Hi, On 02/08/2019 21:48, Tomas Vondra wrote: On Fri, Aug 02, 2019 at 11:20:03AM -0700, Andres Freund wrote: Another question is whether we'd actually want to include the code in core directly, or use system libraries (and if some packagers might decide to disable that, for whatever reason).

Re: Redacting information from logs

2019-08-03 Thread Tom Lane
Andres Freund writes: > On 2019-07-30 11:54:55 -0700, Jeff Davis wrote: >> My proposal is: >> * redact every '%s' in an ereport by having a special mode for >> snprintf.c (this is possible because we now own snprintf) > I'm extremely doubtful this is a sane approach. Yeah, it's really hard to

Re: Redacting information from logs

2019-08-03 Thread Chapman Flack
On 7/30/19 2:54 PM, Jeff Davis wrote: > Logs are important to diagnose problems or monitor operations, but logs > can contain sensitive information which is often unnecessary for these > purposes. Redacting the sensitive information would enable easier > access and simpler integration with

Re: Redacting information from logs

2019-08-03 Thread Andres Freund
Hi, On 2019-07-30 11:54:55 -0700, Jeff Davis wrote: > My proposal is: > > * redact every '%s' in an ereport by having a special mode for > snprintf.c (this is possible because we now own snprintf) I'm extremely doubtful this is a sane approach. We use snprintf for a heck of a lot of things. The

Re: A couple of random BF failures in kerberosCheck

2019-08-03 Thread Tom Lane
I wrote: > * kerberos/t/001_auth.pl just blithely assumes that it can pick > any random port above 48K and that's guaranteed to be free. > Maybe we should split out the code in get_new_node for finding > a free TCP port, so we can call it here? I've confirmed that the reason it's failing on my

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2019-08-03 Thread Julien Rouhaud
Hi, On Sat, Aug 3, 2019 at 1:21 AM Andres Freund wrote: > > On 2019-08-02 10:54:35 +0200, Julien Rouhaud wrote: > > However having the nested queryid in > > pg_stat_activity would be convenient to track what is a long stored > > functions currently doing. Maybe we could expose something like >

Re: Redacting information from logs

2019-08-03 Thread Tomas Vondra
On Tue, Jul 30, 2019 at 11:54:55AM -0700, Jeff Davis wrote: Logs are important to diagnose problems or monitor operations, but logs can contain sensitive information which is often unnecessary for these purposes. Redacting the sensitive information would enable easier access and simpler

Re: A couple of random BF failures in kerberosCheck

2019-08-03 Thread Tom Lane
Thomas Munro writes: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=elver=2019-07-24%2003%3A22%3A17 > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake=2019-08-02%2007%3A17%3A25 > I wondered if this might be like the recently fixed problem with slapd > not being ready to

Re: partition routing layering in nodeModifyTable.c

2019-08-03 Thread Etsuro Fujita
Hi, On Sun, Aug 4, 2019 at 3:03 AM Andres Freund wrote: > On 2019-08-03 13:48:01 -0400, Tom Lane wrote: > > Andres Freund writes: > > > On 2019-08-03 19:41:55 +0900, Etsuro Fujita wrote: > > >> What API does that function break? > > > > > You need to call it, whereas previously you did not need

Re: Unused header file inclusion

2019-08-03 Thread Andres Freund
Hi, On 2019-07-31 19:25:01 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2019-07-31 16:55:31 -0400, Tom Lane wrote: > >> Yeah. I seem to recall a proposal that nodes.h should contain > >> > >> typedef struct Foo Foo; > >> > >> for every node type Foo, and then the other headers would

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-03 Thread Tom Lane
Ian Barwick writes: > On 8/3/19 7:27 AM, Tom Lane wrote: >> Tomas Vondra writes: >>> The main issue however is that no code was written yet. >> Seems like it ought to be relatively simple ... but I didn't look. > The patch I originally sent does exactly this. Ah, you did send a patch, but

Re: partition routing layering in nodeModifyTable.c

2019-08-03 Thread Andres Freund
Hi, On 2019-08-03 13:48:01 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2019-08-03 19:41:55 +0900, Etsuro Fujita wrote: > >> What API does that function break? > > > You need to call it, whereas previously you did not need to call it. The > > effort to change an FDW to get one more

Re: partition routing layering in nodeModifyTable.c

2019-08-03 Thread Tom Lane
Andres Freund writes: > On 2019-08-03 19:41:55 +0900, Etsuro Fujita wrote: >> What API does that function break? > You need to call it, whereas previously you did not need to call it. The > effort to change an FDW to get one more parameter, or to call that > function is about the same. If those

Re: partition routing layering in nodeModifyTable.c

2019-08-03 Thread Andres Freund
Hi, On 2019-08-03 19:41:55 +0900, Etsuro Fujita wrote: > > I don't like > > ExecFindResultRelInfo at all. What's the point of it? It's introduction > > is still an API break - I don't understand why that break is better than > > just passing the ResultRelInfo directly to BeginDirectModify()? > >

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-03 Thread Tom Lane
Tomas Vondra writes: > On Fri, Aug 02, 2019 at 06:08:02PM -0700, Andres Freund wrote: >> We're WAY WAY past feature freeze. This isn't the time to rewrite guc.c, >> guc-file.l to be suitable for running outside of a backend environment. > Right. And even if we had the code, it's not quite

Re: [Patch] Adding CORRESPONDING/CORRESPONDING BY to set operation

2019-08-03 Thread David Fetter
On Sat, Aug 03, 2019 at 05:56:04PM +0200, David Fetter wrote: > On Tue, Jul 30, 2019 at 02:43:05PM -0700, 毛瑞嘉 wrote: > > Hi, > > > > > > I wrote a patch for adding CORRESPONDING/CORRESPONDING BY to set operation. > > It is a task in the todo list. This is how the patch works: > > > > > > I

Re: Fix XML handling with DOCTYPE

2019-08-03 Thread Alvaro Herrera
On 2019-Aug-03, Chapman Flack wrote: > I don't know if it's too late to get in the upcoming minor releases, > but maybe it can, if it looks ok, or the next ones, if that's too rushed. Hmm, I'm travelling back home from a conference the weekend, so yeah I think it would be rushed for me to handle

Re: [proposal] de-TOAST'ing using a iterator

2019-08-03 Thread Binguo Bao
John Naylor 于2019年8月2日周五 下午3:12写道: > On Tue, Jul 30, 2019 at 8:20 PM Binguo Bao wrote: > > > > John Naylor 于2019年7月29日周一 上午11:49写道: > >> > >> 1). For every needle comparison, text_position_next_internal() > >> calculates how much of the value is needed and passes that to > >>

Re: [Patch] Adding CORRESPONDING/CORRESPONDING BY to set operation

2019-08-03 Thread David Fetter
On Tue, Jul 30, 2019 at 02:43:05PM -0700, 毛瑞嘉 wrote: > Hi, > > > I wrote a patch for adding CORRESPONDING/CORRESPONDING BY to set operation. > It is a task in the todo list. This is how the patch works: > > > I modified transformSetOperationStmt() to get an intersection target list > which is

Patch to clean Query after rewrite-and-analyze - reduces memusage up to 50% - increases TPS by up to 50%

2019-08-03 Thread Daniel Migowski
Hello, While examining the reasons for excessive memory usage in prepared statements I noticed that RTE_JOIN-kind RTEs contain a bunch of columnNames and joinaliasvars, that are irrelevant after the Query after has been rewritten. I have some queries that join about 20 tables and select only

Re: Fix XML handling with DOCTYPE

2019-08-03 Thread Chapman Flack
On 04/01/19 17:34, Alvaro Herrera wrote: > I think there were some outright bugs in the docs, at least for > XMLTABLE, that maybe we should backpatch. If you have the energy to > cherry-pick a minimal doc update to 10/11, I offer to back-patch it. I don't know if this fits your intention for

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-03 Thread Tomas Vondra
On Fri, Aug 02, 2019 at 06:08:02PM -0700, Andres Freund wrote: Hi, On 2019-08-02 20:57:20 -0400, Stephen Frost wrote: No, I’m saying that we already *have* a library and we can add a few functions to it and if people want to leverage those functions then they can write glue code to do so, just

Re: partition routing layering in nodeModifyTable.c

2019-08-03 Thread Etsuro Fujita
Hi, On Sat, Aug 3, 2019 at 5:31 AM Andres Freund wrote: > On 2019-08-03 05:20:35 +0900, Etsuro Fujita wrote: > > On Sat, Aug 3, 2019 at 3:01 AM Andres Freund wrote: > > > On 2019-07-31 17:04:38 +0900, Amit Langote wrote: > > > > I looked into trying to do the things I mentioned above and it

Re: The unused_oids script should have a reminder to use the 8000-8999 OID range

2019-08-03 Thread Julien Rouhaud
On Sat, Aug 3, 2019 at 2:40 AM Peter Geoghegan wrote: > > On Fri, Aug 2, 2019 at 3:52 PM Tom Lane wrote: > > Better ... but I'm the world's second worst Perl programmer, > > so I have little to say about whether it's idiomatic. > > Perhaps Michael can weigh in here? I'd rather hear a second

Re: pgbench - implement strict TPC-B benchmark

2019-08-03 Thread Fabien COELHO
Hello Andres, Using pgbench -Mprepared -n -c 8 -j 8 -S pgbench_100 -T 10 -r -P1 e.g. shows pgbench to use 189% CPU in my 4/8 core/thread laptop. That's a pretty significant share. Fine, but what is the corresponding server load? 211%? 611%? And what actual time is spent in pgbench itself, vs

Re[2]: jsonb_plperl bug

2019-08-03 Thread Ivan Panchenko
> Tom Lane : > >=?UTF-8?B?SXZhbiBQYW5jaGVua28=?= < w...@mail.ru > writes: >> I have found a bug in jsonb_plperl extension. A possible fix is proposed >> below. >> ... >> +/* SVt_PV without POK flag is also NULL */ >> +if(SvTYPE(in) ==