Re: [HACKERS] Curing plpgsql's memory leaks for statement-lifespan values

On 8/19/16 10:42 AM, Tom Lane wrote: It appeared to me after collecting some stats about the functions present in the regression tests that for larger functions, the extra space eaten is just some small multiple (like 2x-3x) of the function body string length. So it's not *that* much data, even

Re: [HACKERS] Exporting more function in libpq

> I do not think this is a good idea. If the purpose of libpq is not > to abstract away the wire-level protocol, then what is its purpose? IMHO what currently libpq API does is actually dealing with limited use cases, not abstraction of the protocol. > And how could such a tool avoid breaking

Re: [HACKERS] errno clobbering in reorderbuffer

Andres Freund wrote: > Not at all, thanks. Done, thanks both for the look-over. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol

On Fri, Aug 19, 2016 at 1:51 AM, Heikki Linnakangas wrote: > On 08/18/2016 03:45 PM, Michael Paquier wrote: >> >> On Thu, Aug 18, 2016 at 9:28 PM, Heikki Linnakangas >> wrote: >> For the current ip.c, I don't have a better idea than putting in >> src/common/ip.c

Re: [HACKERS] Slowness of extended protocol

> Tatsuo>understanding it always uses unnamed portal even if the SQL is like > "BEGIN" or "COMMIT" (no parameters). They are too often used. Why not > doing like this? > > Does it actually work? > > The documentation says named portals last till the end of the transaction: > >

Re: [HACKERS] pg_basebackup wish list

On Fri, Aug 19, 2016 at 2:04 PM, Masahiko Sawada wrote: > I agree with adding this as an option and removing directory by default. > And it looks good to me except for missing new line in usage output. > > printf(_(" -l, --label=LABEL set backup label\n")); >

Re: [HACKERS] Slowness of extended protocol

Tatsuo>understanding it always uses unnamed portal even if the SQL is like "BEGIN" or "COMMIT" (no parameters). They are too often used. Why not doing like this? Does it actually work? The documentation says named portals last till the end of the transaction:

[HACKERS] Logical decoding restart problems

Hi, We are using logical decoding in multimaster and we are faced with the problem that inconsistent transactions are sent to replica. Briefly, multimaster is using logical decoding in this way: 1. Each multimaster node is connected with each other using logical decoding channel and so each

Re: [HACKERS] synchronous_commit = remote_flush

On Fri, Aug 19, 2016 at 5:25 AM, Robert Haas wrote: > On Thu, Aug 18, 2016 at 12:22 AM, Thomas Munro > wrote: >> To do something about the confusion I keep seeing about what exactly >> "on" means, I've often wished we had "remote_flush". But

[HACKERS] patch: function xmltable

Hi I am sending implementation of xmltable function. The code should to have near to final quality and it is available for testing. I invite any help with documentation and testing. Regards Pavel diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 169a385..a6334b6 100644 ---

[HACKERS] Exporting more function in libpq

I would like to proppse to export these functions in libpq. pqPutMsgStart pqPutMsgEnd pqPutc pqPuts pqPutInt pqPutnchar pqFlush pqHandleSendFailure I think this would be useful to create a tool/library which needs to handle frontend/backend protocol messages in detail. Best regards, -- Tatsuo

Re: [HACKERS] Slowness of extended protocol

BTW, there seem to be a room to enhance JDBC driver performance. In my understanding it always uses unnamed portal even if the SQL is like "BEGIN" or "COMMIT" (no parameters). They are too often used. Why not doing like this? Prepare(stmt=S1,query="BEGIN") Bind(stmt=S1,portal=P1)

[HACKERS] Should we cacheline align PGXACT?

Hackers, originally this idea was proposed by Andres Freund while experimenting with lockfree Pin/UnpinBuffer [1]. The patch is attached as well as results of pgbench -S on 72-cores machine. As before it shows huge benefit in this case. For sure, we should validate that it doesn't cause

Re: [HACKERS] LWLocks in DSM memory

On Fri, Aug 19, 2016 at 4:44 PM, Amit Kapila wrote: >Can you specify the m/c details as Andres wants tests to be conducted on some high socket m/c? As I have specified at the last line of my mail it is a 8 socket intel machine. available: 8 nodes (0-7) node 0 cpus: 0 65

Re: [HACKERS] Should we cacheline align PGXACT?

On Fri, Aug 19, 2016 at 11:42 AM, Alexander Korotkov wrote: > Hackers, > > originally this idea was proposed by Andres Freund while experimenting with > lockfree Pin/UnpinBuffer [1]. > The patch is attached as well as results of pgbench -S on 72-cores machine. > As

Re: [HACKERS] [PATCH] bigint txids vs 'xid' type, new txid_recent(bigint) => xid

On 8/18/16 9:20 PM, Craig Ringer wrote: > On 19 August 2016 at 02:35, Jim Nasby > wrote: > I think we need to either add real types for handling XID/epoch/TXID > or finally create uint types. It's *way* too easy to screw things

Re: [HACKERS] Should we cacheline align PGXACT?

Alexander Korotkov writes: > originally this idea was proposed by Andres Freund while experimenting with > lockfree Pin/UnpinBuffer [1]. > The patch is attached as well as results of pgbench -S on 72-cores > machine. As before it shows huge benefit in this case.

Re: [HACKERS] LSN as a recovery target

On 06/09/2016 02:33 PM, Michael Paquier wrote: > On Wed, May 25, 2016 at 1:32 AM, Michael Paquier > wrote: >> On Tue, May 24, 2016 at 9:29 AM, Alvaro Herrera >> wrote: >>> Christoph Berg wrote: Re: Michael Paquier 2016-05-24

Re: [HACKERS] WIP: About CMake v2

Christian Convey wrote: I'm interested in helping with your CMake effort. I don't have any experience contributing to PG, but I do have some free time at the moment. Please let me know if I can help. I glad to hear it. I suppose you can just try build postgres and send all problems to github

Re: [HACKERS] WIP: About CMake v2

Stefan Kaltenbrunner wrote: On 08/18/2016 09:30 PM, Christian Convey wrote: Hi Karl, I'll need to let Yury answer your original question regarding the best way to report CMake-related bugs. Regarding the errors you're getting... I just looked at CMake's online documentation regarding your

Re: [HACKERS] Exporting more function in libpq

Tatsuo Ishii writes: > I would like to proppse to export these functions in libpq. > pqPutMsgStart > pqPutMsgEnd > pqPutc > pqPuts > pqPutInt > pqPutnchar > pqFlush > pqHandleSendFailure > I think this would be useful to create a tool/library which needs to > handle

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

On Thu, Aug 18, 2016 at 5:04 PM, Claudio Freire wrote: > But the choice of split point may make a difference for future > insertions, so I'll look into that. A database product I worked on a long time ago had an interesting optimization for indexes that had multiple

Re: [HACKERS] WIP: About CMake v2

Stefan Kaltenbrunner wrote: hmm how do you actually want reports on how it works? I just played with it on spoonbill (OpenBSD 5.3/sparc64) and got this: CMake Error at CMakeLists.txt:1250 (file): file does not recognize sub-command GENERATE CMake Error at src/port/CMakeLists.txt:156

Re: [HACKERS] Why we lost Uber as a user

On Wed, Aug 17, 2016 at 5:18 PM, Jim Nasby wrote: > On 8/17/16 2:51 PM, Simon Riggs wrote: >> On 17 August 2016 at 12:19, Greg Stark wrote: >>> Yes, this is exactly what it should be doing and exactly why it's >>> useful. Physical replication accurately

Re: [HACKERS] WIP: About CMake v2

Andres Freund wrote: The benefit cmake brings to the table, from my pov, is that it allows to get rid of somewhat a parallel buildsystem (msvc / windows, which sources most of its information from the makefiles). If we continue to have two, especially if they're entirely separate, I see little

Re: [HACKERS] patch proposal

On Thu, Aug 18, 2016 at 7:20 PM, Stephen Frost wrote: > * Venkata B Nagothi (nag1...@gmail.com) wrote: > > On Wed, Aug 17, 2016 at 11:27 PM, Stephen Frost > wrote: > > > * Venkata B Nagothi (nag1...@gmail.com) wrote: > > > > Agreed. Additional option like

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

Heikki, Peter, thanks a lot for code review! > What's going on here? Surely pg_atomic_init_u64() should initialize > the value? It's because of how pg_atomic_exchange_u64_impl is implemented: ``` while (true) { old = ptr->value; /* <-- reading of uninitialized value! */ if

Re: [HACKERS] LWLocks in DSM memory

On Fri, Aug 19, 2016 at 4:29 PM, Mithun Cy wrote: > On Wed, Aug 17, 2016 at 9:12 PM, Andres Freund wrote: > >> >Yes. I want a long wait list, modified in bulk - which should be the >> >case with the above. >> > > I ran some pgbench. And, I do not

[HACKERS] Push down more full joins in postgres_fdw

Hi, The postgres_fdw join pushdown in 9.6 is great, but it can't handle full joins on relations with restrictions. The reason for that is, postgres_fdw can't support deparsing subqueries when creating a remote join query. So, by adding the deparsing logic to it, I removed that

Re: [HACKERS] LWLocks in DSM memory

On Wed, Aug 17, 2016 at 9:12 PM, Andres Freund wrote: > >Yes. I want a long wait list, modified in bulk - which should be the > >case with the above. > I ran some pgbench. And, I do not see much difference in performance, small variance in perf can be attributed to variance

Re: [HACKERS] WIP: About CMake v2

Stefan Kaltenbrunner wrote: well we have for example a NetBSD 5.1 boxe (coypu) on the buildfarm that have a software stack that is basically 2008/2009ish... So 2.8.0-2.8.3 seems like a realistic target to me still You can install fresh CMake to NetBSD without big problems from source. --

Re: [HACKERS] Should we cacheline align PGXACT?

On Fri, Aug 19, 2016 at 4:46 PM, Tom Lane wrote: > Alexander Korotkov writes: > > originally this idea was proposed by Andres Freund while experimenting > with > > lockfree Pin/UnpinBuffer [1]. > > The patch is attached as well as results of

Re: [HACKERS] Should we cacheline align PGXACT?

On Fri, Aug 19, 2016 at 3:19 PM, Amit Kapila wrote: > On Fri, Aug 19, 2016 at 11:42 AM, Alexander Korotkov > wrote: > > Hackers, > > > > originally this idea was proposed by Andres Freund while experimenting > with > > lockfree Pin/UnpinBuffer

[HACKERS] Make better use of existing enums in plpgsql

plpgsql.h defines a number of enums, but most of the code passes them around as ints. The attached patch updates structs and function prototypes to take enum types instead. This clarifies the struct definitions in plpgsql.h in particular. I didn't deal with the PLPGSQL_RC_* symbols, since they

Re: [HACKERS] [PATCH] add option to pg_dumpall to exclude tables from the dump

On 8/18/16 5:01 PM, Tom Lane wrote: I agree, but I think mandating a database name (which I suppose could be > *) with the specifiers would solve that issue. Hmm, something like "-T dbname1:pattern1 -T dbname2:pattern2" ? Bingo. Hopefully there'd be some way to consolidate the code between

Re: [HACKERS] anyelement -> anyrange

On 8/18/16 6:02 PM, Corey Huinker wrote: I'd be happy to roll your code into the extension, and make it marked more stable. Yeah, I've been meaning to look at submitting a pull request; hopefully will get to it today. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in

Re: [HACKERS] Curing plpgsql's memory leaks for statement-lifespan values

Jim Nasby writes: > On 7/25/16 1:50 PM, Tom Lane wrote: >> There's a glibc-dependent hack in aset.c that reports any >> plpgsql-driven palloc or pfree against a context named "SPI Proc", as >> well as changes in pl_comp.c so that transient junk created during initial >>

Re: [HACKERS] Should we cacheline align PGXACT?

Alexander Korotkov writes: > On Fri, Aug 19, 2016 at 4:46 PM, Tom Lane wrote: >> That's one mighty ugly patch. Can't you do it without needing to >> introduce the additional layer of struct nesting? > That's worrying me too. > We could use

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

On August 19, 2016 2:50:30 AM PDT, Aleksander Alekseev wrote: >Heikki, Peter, thanks a lot for code review! > >> What's going on here? Surely pg_atomic_init_u64() should initialize >> the value? > >It's because of how pg_atomic_exchange_u64_impl is implemented: >

Re: [HACKERS] Logical decoding restart problems

On 19/08/16 09:34, konstantin knizhnik wrote: We are using logical decoding in multimaster and we are faced with the problem that inconsistent transactions are sent to replica. Briefly, multimaster is using logical decoding in this way: 1. Each multimaster node is connected with each other

Re: [HACKERS] sslmode=require fallback

On Sat, Jul 30, 2016 at 11:18 AM, Bruce Momjian wrote: > On Fri, Jul 29, 2016 at 11:27:06AM -0400, Peter Eisentraut wrote: > > On 7/29/16 11:13 AM, Bruce Momjian wrote: > > > Yes, I am thinking of a case where Postgres is down but a malevolent > > > user starts a Postgres

Re: [HACKERS] Logical decoding restart problems

On 19.08.2016 19:06, Petr Jelinek wrote: I don't think this will work correctly, there will be gap between when the new slot starts to decode and the drop of the old one as the new slot first needs to make snapshot. Do I understand correctly that you are not using replication origins?

Re: [HACKERS] anyelement -> anyrange

On Fri, Aug 19, 2016 at 11:40 AM, Jim Nasby wrote: > On 8/18/16 6:02 PM, Corey Huinker wrote: > >> I'd be happy to roll your code into the extension, and make it marked >> more stable. >> > > Yeah, I've been meaning to look at submitting a pull request; hopefully > will

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

On 2016-08-19 17:55:25 -0400, Tom Lane wrote: > It'd be useful also to figure out why our existing valgrind testing has > not caught this already. The example you give looks like it surely > ought to be replicated well enough in the standard regression tests. The valgrind suppression file

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

Andres Freund writes: > On 2016-08-19 17:55:25 -0400, Tom Lane wrote: >> It'd be useful also to figure out why our existing valgrind testing has >> not caught this already. The example you give looks like it surely >> ought to be replicated well enough in the standard

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

On Thu, Aug 18, 2016 at 6:26 PM, Claudio Freire wrote: > On Thu, Aug 18, 2016 at 6:15 PM, Peter Geoghegan wrote: >> On Thu, Aug 18, 2016 at 1:41 PM, Claudio Freire >> wrote: >>> In fact, that's why non-leaf index tuples need a

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

Piotr Stefaniak writes: > On 2016-08-18 20:02, Heikki Linnakangas wrote: >>> -block = (AllocBlock) malloc(blksize); >>> +block = (AllocBlock) calloc(1, blksize); >> I think this goes too far. You're zeroing all palloc'd memory, even if >> it's going

[HACKERS] Bug in abbreviated keys abort handling (found with amcheck)

I found another bug as a result of using amcheck on Heroku customer databases. This time, the bug is in core Postgres. It's one of mine. There was a thinko in tuplesort's abbreviation abort logic, causing certain SortTuples to be spuriously marked NULL (and so, subsequently sorted as a NULL

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

On 2016-08-19 23:55, Tom Lane wrote: > I think you are failing to understand Heikki's point. There is no way > we are committing the change depicted above, because (1) it will mask more > bugs than it fixes; (2) it's an enormously expensive way to fix anything; > and (3) it will effectively

Re: [HACKERS] EXLCUDE constraints and Hash indexes

On 8/17/16 8:12 AM, Andrew Gierth wrote: I also recently found a case where using btree exclusion constraints was useful: a unique index on an expression can't be marked deferrable, but the equivalent exclusion constraint can be. That seems well worth documenting... -- Jim Nasby, Data

Re: [HACKERS] pgbench - minor doc improvements

Fabien COELHO writes: > Minor pgbench documentation improvements so that the description is more > precise: > - a pgbench script may not contain SQL commands, it only needs not to be > empty. > - point out explicitely variable setting meta commands. > - the formula

Re: [HACKERS] Patch: initdb: "'" for QUOTE_PATH (non-windows)

Here is another version of my initdb shell quoting patch. I have removed the unnecessary {} block. I also ran pgindent on the code prior to creating the patch. On Thu, Aug 18, 2016 at 3:50 PM, Ryan Murphy wrote: > > > On Thu, Aug 18, 2016 at 3:44 PM, Tom Lane

Re: [HACKERS] Re: PROPOSAL: make PostgreSQL sanitizers-friendly (and prevent information disclosure)

On 2016-08-18 20:02, Heikki Linnakangas wrote: > On 03/22/2016 03:27 PM, Aleksander Alekseev wrote: >> diff --git a/src/backend/utils/mmgr/aset.c >> b/src/backend/utils/mmgr/aset.c >> index d26991e..46ab8a2 100644 >> --- a/src/backend/utils/mmgr/aset.c >> +++ b/src/backend/utils/mmgr/aset.c >> @@

Re: [HACKERS] pgbench - allow empty queries

Fabien COELHO writes: > I wanted to test overheads on an empty query, but pgbench does not allow > it. I do not see why not. I'm inclined to think this was probably a good thing before 9.6, as a guard against accidentally writing an empty query into a script and thus

Re: [HACKERS] standalone backend PANICs during recovery

[ got around to looking at this finally ] Alvaro Herrera writes: > Bernd Helmle wrote: >> While investigating a problem on a streaming hot standby instance at a >> customer site, i got the following when using a standalone backend: >> >> PANIC:

Re: [HACKERS] Use %u to print user mapping's umid and userid

Etsuro Fujita writes: > On 2016/05/13 3:53, Tom Lane wrote: >> Robert Haas writes: >>> Regardless of what approach we take, I disagree that this needs to be >>> fixed in 9.6. >> Agreed. This is only a cosmetic issue, and it's only

Re: [HACKERS] Show dropped users' backends in pg_stat_activity

Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> I concur. Let's put the left join(s) into those views and call it >> good. > I'd suggest we also add some notes to the documentation that the correct > approach to dropping users is to disallow access first,

Re: [HACKERS] Logical decoding restart problems

On 19/08/16 18:36, Konstantin Knizhnik wrote: On 19.08.2016 19:06, Petr Jelinek wrote: I don't think this will work correctly, there will be gap between when the new slot starts to decode and the drop of the old one as the new slot first needs to make snapshot. Do I understand correctly

[HACKERS] CREATE POLICY bug ?

Hello, i'm testing the new row security level functionality in postgresql 9.5. To do that i run this script: ---cut here -- CREATE TABLE public.policy_tab ( id bigint NOT NULL, description character varying(160) NOT NULL, usr name NOT NULL, CONSTRAINT

Re: [HACKERS] LSN as a recovery target

On Fri, Aug 19, 2016 at 10:47 PM, Adrien Nayrat wrote: > I reviewed this patch rebased to deal with > f6ced51f9188ad5806219471a0b40a91dde923aa, and minor adjustment (see below) Thanks! > It do the job. However if you use an incorrect recovery_target_lsn you > get this

Re: [HACKERS] Should we cacheline align PGXACT?

On Fri, Aug 19, 2016 at 8:24 PM, Alexander Korotkov wrote: > On Fri, Aug 19, 2016 at 3:19 PM, Amit Kapila > wrote: >> >> On Fri, Aug 19, 2016 at 11:42 AM, Alexander Korotkov >> wrote: >> > Hackers, >> > >> >

Re: [HACKERS] LSN as a recovery target

On 20/08/16 02:13, Michael Paquier wrote: On Fri, Aug 19, 2016 at 10:47 PM, Adrien Nayrat wrote: I reviewed this patch rebased to deal with f6ced51f9188ad5806219471a0b40a91dde923aa, and minor adjustment (see below) Thanks! It do the job. However if you use an

[HACKERS] TODO list updated for PG 10

I have done my yearly TODO list cleanup, and I did more extensive item removal this time. There were a number of entries I could not figure out so if people want to review what is left and remove items that are undesired or done, please do that. Thanks.

Re: [HACKERS] replication slots replicated to standbys?

On Sat, Aug 20, 2016 at 1:39 PM, Bruce Momjian wrote: > Someone reported that a replication slot that existed at the time a base > backup was done on the master was copied to the standby. Because they > didn't realize it, their WAL was not being recycled on the standby. > > Is

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

On Sat, Aug 20, 2016 at 1:53 AM, Claudio Freire wrote: > All uniqueness checks will try to read-lock nbuf > unless the uniqueness check for that insertion is done. That should read "all uniqueness checks will try to read-lock the buf unless the uniqueness check for that

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

On Thu, Aug 18, 2016 at 8:24 AM, Claudio Freire wrote: > > A couple of points make me uneasy about this patch, yet I can think of > no better alternative, so I seek feedback: > > - introducing a different format for inner index tuples makes for an > invasive patch and

Re: [HACKERS] [WIP] [B-Tree] Keep indexes sorted by heap physical location

On Sat, Aug 20, 2016 at 1:05 AM, Amit Kapila wrote: > On Thu, Aug 18, 2016 at 8:24 AM, Claudio Freire > wrote: >> >> A couple of points make me uneasy about this patch, yet I can think of >> no better alternative, so I seek feedback: >> >> -

[HACKERS] replication slots replicated to standbys?

Someone reported that a replication slot that existed at the time a base backup was done on the master was copied to the standby. Because they didn't realize it, their WAL was not being recycled on the standby. Is that possible? Is it a known behavior? I don't see it documented. -- Bruce

Re: [HACKERS] CLUSTER, reform_and_rewrite_tuple(), and parallelism

On Thu, Aug 18, 2016 at 4:42 AM, Peter Geoghegan wrote: > > Does anyone have any ideas on how to: > > 1). Directly address the reform_and_rewrite_tuple() bottleneck. > > and/or: > > 2). Push down some or all of the reform_and_rewrite_tuple() work till > before tuples are passed

Re: [HACKERS] WIP: About CMake v2

Hi Yury, On Fri, Aug 19, 2016 at 9:46 AM, Yury Zhuravlev wrote: > Christian Convey wrote: >> >> I'm interested in helping with your CMake effort. I don't have any >> experience contributing to PG, but I do have some free time at the >> moment. Please let me know if