Re: [HACKERS] BUG: pg_stat_statements query normalization issues with combined queries

2016-12-21 Thread Fabien COELHO
Hello Kyotaro-san, [...] Agreed that copying statement string would be too much. But the new *Stmt node should somehow have also the end location of the statement since the user of a parse tree cannot look into another one. Yes. I was thinking of adding a "length" field next to "location", w

Re: [HACKERS] [bug fix] Trivial ecpg bug which can cause memory overrun

2016-12-21 Thread Michael Meskes
Hi, > While investigating some other issue, we found a trivial bug of > ecpg.  The attached is a fix for that. > ... Thanks for spotting and fixing. Committed. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Meskes at (Debian|Postgresql) dot Org Ja

[HACKERS] Ilegal type cast in _hash_doinsert()

2016-12-21 Thread Mithun Cy
There seems to be some base bug in _hash_doinsert(). +* XXX this is useless code if we are only storing hash keys. + */ +if (itemsz > HashMaxItemSize((Page) metap)) +ereport(ERROR, +(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("index row

Re: [HACKERS] BUG: pg_stat_statements query normalization issues with combined queries

2016-12-21 Thread Kyotaro HORIGUCHI
At Wed, 21 Dec 2016 09:28:58 +0100 (CET), Fabien COELHO wrote in > > Hello Robert & Kyotaro, > > >> I'm a little doubtful about your proposed fix. However, I haven't > >> studied the code, so I don't know what other approach might be better. > > That is indeed the question! > > The key poin

Re: [HACKERS] Cache Hash Index meta page.

2016-12-21 Thread Amit Kapila
On Wed, Dec 21, 2016 at 9:26 PM, Robert Haas wrote: > On Tue, Dec 20, 2016 at 2:25 PM, Mithun Cy wrote: >> -- I think if it is okay, I can document same for each member of >> HashMetaPageData whether to read from cached from meta page or directly from >> current meta page. Below briefly I have

Re: [HACKERS] Logical decoding on standby

2016-12-21 Thread Craig Ringer
On 22 December 2016 at 13:43, Michael Paquier wrote: > So, for 0001: > --- a/src/test/perl/PostgresNode.pm > +++ b/src/test/perl/PostgresNode.pm > @@ -93,6 +93,7 @@ use RecursiveCopy; > use Socket; > use Test::More; > use TestLib (); > +use pg_lsn qw(parse_lsn); > use Scalar::Util qw(blessed)

Re: [HACKERS] WAL consistency check facility

2016-12-21 Thread Kuntal Ghosh
On Wed, Dec 21, 2016 at 10:53 PM, Robert Haas wrote: > On Mon, Nov 28, 2016 at 11:31 PM, Michael Paquier > wrote: >> Moved to CF 2017-01, as no committers have showed up yet :( > > Seeing no other volunteers, here I am. > Thanks Robert for looking into the patch. > On a first read-through of thi

[HACKERS] [bug fix] Trivial ecpg bug which can cause memory overrun

2016-12-21 Thread Tsunakawa, Takayuki
Hello, While investigating some other issue, we found a trivial bug of ecpg. The attached is a fix for that. If you specify an input file which ends with "." (e.g. run "ecpg file."), ecpg writes one byte past the end of the allocated memory. In addition, the following statement is misleading.

Re: [HACKERS] Getting rid of "unknown error" in dblink and postgres_fdw

2016-12-21 Thread Joe Conway
On 12/21/2016 09:20 PM, Tom Lane wrote: > Joe Conway writes: >> On 12/21/2016 04:22 PM, Tom Lane wrote: >>> In short, yes, please copy that bit into dblink. > >> The attached should do the trick I think. > > I see that you need to pass the PGconn into dblink_res_error() now, but > what's the poi

Re: [HACKERS] Logical decoding on standby

2016-12-21 Thread Michael Paquier
On Tue, Dec 20, 2016 at 4:03 PM, Petr Jelinek wrote: > That's about approach, but since there are prerequisite patches in the > patchset that don't really depend on the approach I will comment about > them as well. > > 0001 and 0002 add testing infrastructure and look fine to me, possibly > commit

Re: [HACKERS] Getting rid of "unknown error" in dblink and postgres_fdw

2016-12-21 Thread Tom Lane
Joe Conway writes: > On 12/21/2016 04:22 PM, Tom Lane wrote: >> In short, yes, please copy that bit into dblink. > The attached should do the trick I think. I see that you need to pass the PGconn into dblink_res_error() now, but what's the point of the new "bool fail" parameter? > You think it

Re: [HACKERS] [PATCH] Transaction traceability - txid_status(bigint)

2016-12-21 Thread Craig Ringer
On 22 December 2016 at 09:55, Craig Ringer wrote: > Updated. > > If you think it's better to just take XidGenLock again, let me know. Here's a further update that merges in Robert's changes from the patch posted upthread. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreS

Re: [HACKERS] Parallel Index Scans

2016-12-21 Thread Amit Kapila
On Thu, Dec 22, 2016 at 9:49 AM, Amit Kapila wrote: > On Wed, Dec 21, 2016 at 8:46 PM, Anastasia Lubennikova > wrote: >> The following review has been posted through the commitfest application: >> make installcheck-world: tested, passed >> Implements feature: tested, passed >> Spec complia

Re: [HACKERS] multi-level partitions and partition-wise joins

2016-12-21 Thread Ashutosh Bapat
On Wed, Dec 21, 2016 at 10:25 PM, Robert Haas wrote: > On Wed, Dec 21, 2016 at 6:36 AM, Ashutosh Bapat > wrote: >> I am starting this as a separate thread for this since the declarative >> partitioning thread has many issues reported and it's better to keep >> this discussion separate from the is

Re: [HACKERS] Parallel Index Scans

2016-12-21 Thread Amit Kapila
On Wed, Dec 21, 2016 at 8:46 PM, Anastasia Lubennikova wrote: > The following review has been posted through the commitfest application: > make installcheck-world: tested, passed > Implements feature: tested, passed > Spec compliant: tested, passed > Documentation:test

Re: [HACKERS] Hang in pldebugger after git commit : 98a64d0

2016-12-21 Thread Amit Kapila
On Wed, Dec 21, 2016 at 9:46 PM, Robert Haas wrote: > On Tue, Dec 20, 2016 at 11:32 PM, Amit Kapila wrote: >> Ashutosh Sharma has helped to test that pldebugger issue is fixed with >> attached version. > > Committed. > Thanks! > Patch by Amit Kapial. Reported and tested by Ashutosh Sharma, who

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-12-21 Thread Michael Paquier
On Thu, Dec 22, 2016 at 6:41 AM, David Steele wrote: > On 12/21/16 4:28 PM, Andres Freund wrote: > >> Working on committing this (tomorrow morning, not tonight). There's >> some relatively minor things I want to change: Thanks for looking at this patch. >> - I don't like the name XLogSetFlags()

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-12-21 Thread Michael Paquier
On Thu, Dec 22, 2016 at 6:28 AM, Andres Freund wrote: > A mime-type of invalid/octet-stream? That's an, uh, odd choice. Indeed. I am not sure what kind of accident happened here. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscript

Re: [HACKERS] [PATCH] Transaction traceability - txid_status(bigint)

2016-12-21 Thread Craig Ringer
On 22 December 2016 at 07:49, Craig Ringer wrote: > On 22 December 2016 at 00:30, Robert Haas wrote: > >> That makes everything that happens between when we acquire that lock >> and when we release it non-interruptible, which seems undesirable. I >> think that extra copy of oldestXid is a nicer

[HACKERS] Potential data loss of 2PC files

2016-12-21 Thread Michael Paquier
Hi all, 2PC files are created using RecreateTwoPhaseFile() in two places currently: - at replay on a XLOG_XACT_PREPARE record. - At checkpoint with CheckPointTwoPhase(). Now RecreateTwoPhaseFile() is careful to call pg_fsync() to be sure that the 2PC files find their way into disk. But one piece

Re: [HACKERS] Getting rid of "unknown error" in dblink and postgres_fdw

2016-12-21 Thread Joe Conway
On 12/21/2016 04:22 PM, Tom Lane wrote: > Joe Conway writes: >> I did notice that postgres_fdw has the following stanza that I don't see >> in dblink: > >> 8<-- >> /* >> * If we don't get a message from the PGresult, try the PGconn. This >> * is needed because for connection-le

Re: [HACKERS] [COMMITTERS] pgsql: Implement table partitioning.

2016-12-21 Thread Amit Langote
On 2016/12/22 0:31, Robert Haas wrote: > On Tue, Dec 20, 2016 at 12:22 PM, Alvaro Herrera > wrote: >> Robert Haas wrote: >>> Implement table partitioning. >> >> I thought it was odd to use rd_rel->reloftype as a boolean in >> ATExecAttachPartition, but apparently we do it elsewhere too, so let's >

Re: [HACKERS] SET NOT NULL [NOT VALID / CONCURRENTLY]?

2016-12-21 Thread Joel Jacobson
Attached is the function SET_NOT_NULL(_Schema name, _Table name, _Column name) which does the following: 1. LOCK TABLE %I.%I IN ACCESS EXCLUSIVE MODE just like the normal DDL commands would do 2. SELECT EXISTS (SELECT 1 FROM %I.%I WHERE %I IS NULL) which is fast if there is an index on the column

Re: [HACKERS] Getting rid of "unknown error" in dblink and postgres_fdw

2016-12-21 Thread Tom Lane
Joe Conway writes: > I did notice that postgres_fdw has the following stanza that I don't see > in dblink: > 8<-- > /* > * If we don't get a message from the PGresult, try the PGconn. This > * is needed because for connection-level failures, PQexec may just > * return NULL, no

Re: [HACKERS] Getting rid of "unknown error" in dblink and postgres_fdw

2016-12-21 Thread Joe Conway
On 12/21/2016 10:08 AM, Tom Lane wrote: > I wrote: I propose that we should change that string to "could not obtain message string for error on connection "foo"", or something along that line. > > BTW, looking closer, I notice that the dblink case already has > > errcontext("E

Re: [HACKERS] Why does plpython delay composite type resolution?

2016-12-21 Thread Jim Nasby
On 12/21/16 8:39 AM, Tom Lane wrote: Jim Nasby writes: On 12/21/16 1:55 AM, Andreas Karlsson wrote: Does your patch handle "ALTER TYPE name ADD ATTRIBUTE ..."? My immediate guess would be that it could be a cache invalidation thing. Won't that only happen at end of transaction? No.

Re: [HACKERS] [PATCH] Transaction traceability - txid_status(bigint)

2016-12-21 Thread Craig Ringer
On 22 December 2016 at 00:30, Robert Haas wrote: > That makes everything that happens between when we acquire that lock > and when we release it non-interruptible, which seems undesirable. I > think that extra copy of oldestXid is a nicer approach. That's a side-effect I didn't realise. Given t

Re: [HACKERS] Speedup twophase transactions

2016-12-21 Thread Michael Paquier
On Wed, Dec 21, 2016 at 10:37 PM, Stas Kelvich wrote: > On 21 Dec 2016, at 19:56, Michael Paquier wrote: >> That's indeed way simpler than before. Have you as well looked at the >> most simple approach discussed? That would be just roughly replacing >> the pg_fsync() calls currently in RecreateTw

Re: [HACKERS] [BUG?] pg_event_trigger_ddl_commands() error with ALTER TEXT SEARCH CONFIGURATION

2016-12-21 Thread Artur Zakirov
Thank you for your comments, Stephen. 2016-12-21 20:34 GMT+03:00 Stephen Frost : > > Did you happen to look at adding a regression test for this to > test_ddl_deparse? Of course. I updated the patch. > >> This patch only fixes the bug. But I think I also can do a patch which >> will give pg_ts_c

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-12-21 Thread Peter Geoghegan
On Wed, Dec 21, 2016 at 10:21 AM, Peter Geoghegan wrote: > On Wed, Dec 21, 2016 at 6:00 AM, Robert Haas wrote: >> 3. Just live with the waste of space. > > I am loathe to create a special case for the parallel interface too, > but I think it's possible that *no* caller will ever actually need to

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-12-21 Thread David Steele
On 12/21/16 4:40 PM, Andres Freund wrote: On 2016-12-21 16:35:28 -0500, Robert Haas wrote: What I think "progress LSN" is getting at -- actually fairly well -- is whether we're getting anything *important* done, not whether we are consistent. I don't mind changing the name, but not to consist

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-12-21 Thread David G. Johnston
On Wed, Dec 21, 2016 at 2:40 PM, Andres Freund wrote: > That's imo pretty much what progress LSN currently describes. Have there > been any records which are important for durability/consistency and > hence need to be archived and such. > The above, to me, describes a "milestone LSN"...​ David

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-12-21 Thread David Steele
Hi Andres, On 12/21/16 4:28 PM, Andres Freund wrote: Working on committing this (tomorrow morning, not tonight). There's some relatively minor things I want to change: - I don't like the name XLogSetFlags() - it's completely unclear what that those flags refer to - it could just as well be

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-12-21 Thread Andres Freund
On 2016-12-21 16:35:28 -0500, Robert Haas wrote: > On Wed, Dec 21, 2016 at 4:28 PM, Andres Freund wrote: > > - Similarly I don't like the name "progress LSN" much. What does > > "progress" really mean in that". Maybe "consistency LSN"? > > Whoa. -1 from me for "consistency LSN". Consistency h

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-12-21 Thread Robert Haas
On Wed, Dec 21, 2016 at 4:28 PM, Andres Freund wrote: > - Similarly I don't like the name "progress LSN" much. What does > "progress" really mean in that". Maybe "consistency LSN"? Whoa. -1 from me for "consistency LSN". Consistency has to with whether the cluster has recovered up to the mini

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-12-21 Thread Andres Freund
Hi, A mime-type of invalid/octet-stream? That's an, uh, odd choice. Working on committing this (tomorrow morning, not tonight). There's some relatively minor things I want to change: - I don't like the name XLogSetFlags() - it's completely unclear what that those flags refer to - it could jus

Re: [HACKERS] Measuring replay lag

2016-12-21 Thread Thomas Munro
On Thu, Dec 22, 2016 at 2:14 AM, Fujii Masao wrote: > I agree that the capability to measure the remote_apply lag is very useful. > Also I want to measure the remote_write and remote_flush lags, for example, > in order to diagnose the cause of replication lag. Good idea. I will think about how t

Re: [HACKERS] Minor correction in alter_table.sgml

2016-12-21 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> So maybe something like >> >> All the forms of ALTER TABLE that act on a single table, >> except RENAME and SET SCHEMA, can be combined into a >> list of multiple alterations to be applied together. > Committed and back-patch'd th

Re: [HACKERS] Minor correction in alter_table.sgml

2016-12-21 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > So maybe something like > > All the forms of ALTER TABLE that act on a single table, > except RENAME and SET SCHEMA, can be combined into a > list of multiple alterations to be applied together. Committed and back-patch'd that way. Thank

Re: [HACKERS] pg_dump vs. TRANSFORMs

2016-12-21 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > That's a good point, we might be doing things wrong in other places in > > the code by using FirstNormalObjectId on pre-8.1 servers. > > > What I suggest then is an independent patch which uses a different > > variable than FirstN

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-12-21 Thread Peter Geoghegan
On Wed, Dec 21, 2016 at 6:00 AM, Robert Haas wrote: > 3. Just live with the waste of space. I am loathe to create a special case for the parallel interface too, but I think it's possible that *no* caller will ever actually need to live with this restriction at any time in the future. I am strongl

Re: [HACKERS] Getting rid of "unknown error" in dblink and postgres_fdw

2016-12-21 Thread Tom Lane
I wrote: >>> I propose that we should change that string to "could not obtain message >>> string for error on connection "foo"", or something along that line. BTW, looking closer, I notice that the dblink case already has errcontext("Error occurred on dblink connection named \"%s\": %s.

Re: [HACKERS] pg_background contrib module proposal

2016-12-21 Thread Andrew Borodin
2016-12-21 20:42 GMT+05:00 Robert Haas : > This whole subthread seems like a distraction to me. I find it hard > to believe that this test case would be stable enough to survive the > buildfarm where, don't forget, we have things like > CLOBBER_CACHE_ALWAYS machines where queries take 100x longer

Re: [HACKERS] pg_background contrib module proposal

2016-12-21 Thread David Fetter
On Wed, Dec 21, 2016 at 10:42:18AM -0500, Robert Haas wrote: > On Wed, Dec 21, 2016 at 10:29 AM, David Fetter wrote: > > On Wed, Dec 21, 2016 at 06:31:52PM +0800, Craig Ringer wrote: > >> On 21 December 2016 at 14:26, Andrew Borodin wrote: > >> > >> > I'm not sure every platform supports microsec

Re: [HACKERS] Getting rid of "unknown error" in dblink and postgres_fdw

2016-12-21 Thread Joe Conway
On 12/21/2016 09:27 AM, Tom Lane wrote: > Joe Conway writes: >> On 12/21/2016 08:49 AM, Tom Lane wrote: >>> I propose that we should change that string to "could not obtain message >>> string for error on connection "foo"", or something along that line. >>> >>> postgres_fdw has the same disease.

Re: [HACKERS] [ patch ] pg_dump: new --custom-fetch-table and --custom-fetch-value parameters

2016-12-21 Thread Stephen Frost
Andrea, * Andrea Urbani (matfan...@mail.com) wrote: >I had a problem with a Postgresql 9.3.5 on 32 bit linux, old 2.6.26 >kernel: Ok, though, to be clear, this is a feature request, so we wouldn't back-patch adding this to pg_dump. >I have solve it adding two new parameters, --custom

Re: [HACKERS] [BUG?] pg_event_trigger_ddl_commands() error with ALTER TEXT SEARCH CONFIGURATION

2016-12-21 Thread Stephen Frost
Artur, * Artur Zakirov (a.zaki...@postgrespro.ru) wrote: > 2016-11-19 21:28 GMT+03:00 Michael Paquier : > > On Thu, Nov 17, 2016 at 1:17 PM, Alvaro Herrera > > wrote: > >> It's a bug. You're right that we need to handle the object class > >> somewhere. Perhaps I failed to realize that tsconfigs

Re: [HACKERS] Getting rid of "unknown error" in dblink and postgres_fdw

2016-12-21 Thread Tom Lane
Joe Conway writes: > On 12/21/2016 08:49 AM, Tom Lane wrote: >> I propose that we should change that string to "could not obtain message >> string for error on connection "foo"", or something along that line. >> >> postgres_fdw has the same disease. It wouldn't have the notion of a named >> conne

Re: [HACKERS] WAL consistency check facility

2016-12-21 Thread Robert Haas
On Mon, Nov 28, 2016 at 11:31 PM, Michael Paquier wrote: > Moved to CF 2017-01, as no committers have showed up yet :( Seeing no other volunteers, here I am. On a first read-through of this patch -- I have not studied it in detail yet -- this looks pretty good to me. One concern is that this pa

Re: [HACKERS] Getting rid of "unknown error" in dblink and postgres_fdw

2016-12-21 Thread Joe Conway
On 12/21/2016 08:49 AM, Tom Lane wrote: > We have a report in pgsql-general of a dblink query failing with > ERROR: unknown error > This is, to say the least, unhelpful. And it violates our error > message style guidelines. > > Where that is coming from is a situation where we've failed to

[HACKERS] [ patch ] pg_dump: new --custom-fetch-table and --custom-fetch-value parameters

2016-12-21 Thread Andrea Urbani
Hello to everybody, I had a problem with a Postgresql 9.3.5 on 32 bit linux, old 2.6.26 kernel: Program: pg_dump Problem: if you have tables with big blob fields and you try to dump them with --inserts you could get errors like     pg_dump: [archiver (db)] query failed: lost synchronization with

Re: [HACKERS] Getting rid of "unknown error" in dblink and postgres_fdw

2016-12-21 Thread Tom Lane
"David G. Johnston" writes: > On Wed, Dec 21, 2016 at 9:49 AM, Tom Lane wrote: >> A possible objection is that if we really are on the hairy edge of OOM, >> trying to construct such error strings might push us over the edge > What am I missing here? Constructing said string occurs on the local

Re: [HACKERS] Getting rid of "unknown error" in dblink and postgres_fdw

2016-12-21 Thread Tom Lane
I wrote: > We have a report in pgsql-general of a dblink query failing with > ERROR: unknown error Er, fingers faster than brain this morning. Actually that report is in pgsql-bugs, specifically bug #14471: https://www.postgresql.org/message-id/20161221094443.25614.47807%40wrigleys.postgres

Re: [HACKERS] Getting rid of "unknown error" in dblink and postgres_fdw

2016-12-21 Thread David G. Johnston
On Wed, Dec 21, 2016 at 9:49 AM, Tom Lane wrote: > A possible objection is that if we really are on the hairy edge of OOM, > trying to construct such error strings might push us over the edge What am I missing here? Constructing said string occurs on the local end of the connection but the mem

Re: [HACKERS] multi-level partitions and partition-wise joins

2016-12-21 Thread Robert Haas
On Wed, Dec 21, 2016 at 6:36 AM, Ashutosh Bapat wrote: > I am starting this as a separate thread for this since the declarative > partitioning thread has many issues reported and it's better to keep > this discussion separate from the issues reported on that thread. > > While expanding inheritance

Re: [HACKERS] Declarative partitioning - another take

2016-12-21 Thread Robert Haas
On Wed, Dec 21, 2016 at 5:33 AM, Amit Langote wrote: > Breaking changes into multiple commits/patches does not seem to work for > adding regression tests. So, I've combined multiple patches into a single > patch which is now patch 0002 in the attached set of patches. Ugh, seriously? It's fine t

[HACKERS] Getting rid of "unknown error" in dblink and postgres_fdw

2016-12-21 Thread Tom Lane
We have a report in pgsql-general of a dblink query failing with ERROR: unknown error This is, to say the least, unhelpful. And it violates our error message style guidelines. Where that is coming from is a situation where we've failed to extract any primary message from a remote error.

Re: [HACKERS] [PATCH] Transaction traceability - txid_status(bigint)

2016-12-21 Thread Robert Haas
On Wed, Dec 21, 2016 at 3:02 AM, Craig Ringer wrote: > Instead, I've added a new LWLock, ClogTruncationLock, for that > purpose. vac_truncate_clog() takes it if it decides to attempt clog > truncation. This lock is held throughout the whole process of clog > truncation and oldestXid advance, so th

Re: [HACKERS] Logical decoding on standby

2016-12-21 Thread Robert Haas
On Tue, Dec 20, 2016 at 10:06 PM, Craig Ringer wrote: > On 20 December 2016 at 15:03, Petr Jelinek > wrote: >>> The biggest change in this patch, and the main intrusive part, is that >>> procArray->replication_slot_catalog_xmin is no longer directly used by >>> vacuum. Instead, a new ShmemVariab

Re: [HACKERS] Hang in pldebugger after git commit : 98a64d0

2016-12-21 Thread Robert Haas
On Tue, Dec 20, 2016 at 11:32 PM, Amit Kapila wrote: > Ashutosh Sharma has helped to test that pldebugger issue is fixed with > attached version. Committed. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsq

Re: [HACKERS] postgres_fdw bug in 9.6

2016-12-21 Thread Ashutosh Bapat
Some review comments 1. postgres_fdw doesn't push down semi and anti joins so you may want to discount these two too. + jointype == JOIN_SEMI || + jointype == JOIN_ANTI); 2. We should try to look for other not-so-cheap paths if the cheapest one is paramterized. You might want

Re: [HACKERS] Cache Hash Index meta page.

2016-12-21 Thread Robert Haas
On Tue, Dec 20, 2016 at 2:25 PM, Mithun Cy wrote: > -- I think if it is okay, I can document same for each member of > HashMetaPageData whether to read from cached from meta page or directly from > current meta page. Below briefly I have commented for each member. If you > suggest I can go with

Re: [HACKERS] Proposal : Parallel Merge Join

2016-12-21 Thread Dilip Kumar
On Wed, Dec 21, 2016 at 8:39 PM, Robert Haas wrote: > Committed the refactoring patch with some mild cosmetic adjustments. Thanks.. > > As to the second patch: > > +if (jointype == JOIN_UNIQUE_INNER) > +jointype = JOIN_INNER; > > Isn't this dead code? save_jointype might that

Re: [HACKERS] pg_background contrib module proposal

2016-12-21 Thread Robert Haas
On Wed, Dec 21, 2016 at 10:29 AM, David Fetter wrote: > On Wed, Dec 21, 2016 at 06:31:52PM +0800, Craig Ringer wrote: >> On 21 December 2016 at 14:26, Andrew Borodin wrote: >> >> > I'm not sure every platform supports microsecond sleeps >> >> Windows at least doesn't by default, unless that chang

Re: [HACKERS] Parallel Index Scans

2016-12-21 Thread Robert Haas
Thanks for reviewing! A few quick thoughts from me since I write a bunch of the design for this patch. On Wed, Dec 21, 2016 at 10:16 AM, Anastasia Lubennikova wrote: > 1. Can't we simply use "if (scan->parallel_scan != NULL)" instead of > xs_temp_snap flag? > > + if (scan->xs_temp_snap) >

Re: [HACKERS] [COMMITTERS] pgsql: Implement table partitioning.

2016-12-21 Thread Robert Haas
On Tue, Dec 20, 2016 at 12:22 PM, Alvaro Herrera wrote: > Robert Haas wrote: >> Implement table partitioning. > > I thought it was odd to use rd_rel->reloftype as a boolean in > ATExecAttachPartition, but apparently we do it elsewhere too, so let's > leave that complaint for another day. Ugh. I

Re: [HACKERS] pg_background contrib module proposal

2016-12-21 Thread David Fetter
On Wed, Dec 21, 2016 at 06:31:52PM +0800, Craig Ringer wrote: > On 21 December 2016 at 14:26, Andrew Borodin wrote: > > > I'm not sure every platform supports microsecond sleeps > > Windows at least doesn't by default, unless that changed in Win2k12 > and Win8 with the same platform/kernel impro

Re: [HACKERS] Parallel Index Scans

2016-12-21 Thread Anastasia Lubennikova
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed Hi, thank you for the patch. Results are very promising. Do y

Re: [HACKERS] Proposal : Parallel Merge Join

2016-12-21 Thread Robert Haas
On Tue, Dec 13, 2016 at 10:04 AM, Dilip Kumar wrote: > On Tue, Dec 13, 2016 at 6:42 AM, Dilip Kumar wrote: >>> This patch is hard to read because it is reorganizing a bunch of code >>> as well as adding new functionality. Perhaps you could separate it >>> into two patches, one with the refactori

Re: [HACKERS] Why does plpython delay composite type resolution?

2016-12-21 Thread Tom Lane
Jim Nasby writes: > On 12/21/16 1:55 AM, Andreas Karlsson wrote: >> Does your patch handle "ALTER TYPE name ADD ATTRIBUTE ..."? My immediate >> guess would be that it could be a cache invalidation thing. > Won't that only happen at end of transaction? No. BEGIN; SELECT plpython_

Re: [HACKERS] Rethinking our fulltext phrase-search implementation

2016-12-21 Thread Tom Lane
Artur Zakirov writes: > Otherwise it seems that queries like 'a <-> (b & c)' will always return > false. Then we need maybe some warning message. Well, the query as written is pointless, but it could be useful with something other than "b" and "c" as the AND-ed terms. In this usage "&" is equiv

Re: [HACKERS] Why does plpython delay composite type resolution?

2016-12-21 Thread Jim Nasby
On 12/21/16 1:55 AM, Andreas Karlsson wrote: On 12/21/2016 04:14 AM, Jim Nasby wrote: Why do functions that accept composite types delay type resolution until execution? I have a naive patch that speeds up plpy.execute() by 8% by caching interred python strings for the dictionary key names (whic

Re: [HACKERS] Faster methods for getting SPI results

2016-12-21 Thread Jim Nasby
On 12/20/16 10:14 PM, Jim Nasby wrote: It would be a lot more efficient if we could just grab datums from the executor and make a single copy into plpython (or R), letting the PL deal with all the memory management overhead. I briefly looked at using SPI cursors to do just that, but that looks e

Re: [HACKERS] pgstattuple documentation clarification

2016-12-21 Thread Andrew Dunstan
On 12/20/2016 11:41 PM, Robert Haas wrote: On Tue, Dec 20, 2016 at 10:01 AM, Tom Lane wrote: Andrew Dunstan writes: Recently a client was confused because there was a substantial difference between the reported table_len of a table and the sum of the corresponding tuple_len, dead_tuple_len

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-12-21 Thread Robert Haas
On Wed, Dec 21, 2016 at 7:04 AM, Heikki Linnakangas wrote: >> If the worker is always completely finished with the tape before the >> leader touches it, couldn't the leader's LogicalTapeSet just "adopt" >> the tape and overwrite it like any other? > > Currently, the logical tape code assumes that

[HACKERS] Add pgstathashindex() to get hash index table statistics.

2016-12-21 Thread Ashutosh Sharma
Hi All, I have introduced a new function 'pgstathashindex()' inside pgstatuple extension to view the statistics related to hash index table. I could have used 'pgstattuple()' function to view hash index stats instead of adding this new function but there are certain limitations when using pgstattu

Re: [HACKERS] Replication slot xmin is not reset if HS feedback is turned off while standby is shut down

2016-12-21 Thread Craig Ringer
On 21 December 2016 at 21:23, Simon Riggs wrote: > Valid bug, but this still ain't right. We don't want to turn feedback > on until HS is active, but we do want to turn it off once whether or > not HS is active yet. I just posted an update, though I forgot to add you to the direct Cc list. It pa

Re: [HACKERS] Speedup twophase transactions

2016-12-21 Thread Stas Kelvich
> On 21 Dec 2016, at 19:56, Michael Paquier wrote: > > That's indeed way simpler than before. Have you as well looked at the > most simple approach discussed? That would be just roughly replacing > the pg_fsync() calls currently in RecreateTwoPhaseFile() by a save > into a list as you are doing,

Re: [HACKERS] Replication slot xmin is not reset if HS feedback is turned off while standby is shut down

2016-12-21 Thread Craig Ringer
On 21 December 2016 at 21:03, Ants Aasma wrote: > There was a !HotStandbyActive() check there previously, I was not sure > if it was only to avoid sending useless messages or was necessary > because something was not initialized otherwise. Looks like it is not > needed and can be removed. Revised

Re: [HACKERS] Logical tape pause/resume

2016-12-21 Thread Heikki Linnakangas
On 12/21/2016 03:22 PM, Robert Haas wrote: On Wed, Dec 21, 2016 at 7:25 AM, Heikki Linnakangas wrote: For now, barring objections, I'm going to commit the first patch. It seems like a worthwhile simplification in any case, especially for Peter's Parallel tuplesort patch set. Well, it removes

Re: [HACKERS] Replication slot xmin is not reset if HS feedback is turned off while standby is shut down

2016-12-21 Thread Simon Riggs
On 21 December 2016 at 13:03, Ants Aasma wrote: > On Wed, Dec 21, 2016 at 2:09 PM, Craig Ringer wrote: >> On 21 December 2016 at 15:40, Ants Aasma wrote: >> So -1 on this part of the patch, unless there's something I've misunderstood. >>> >>> Currently there was no feedback sent if ho

Re: [HACKERS] Logical tape pause/resume

2016-12-21 Thread Robert Haas
On Wed, Dec 21, 2016 at 7:25 AM, Heikki Linnakangas wrote: > For now, barring objections, I'm going to commit the first patch. It seems > like a worthwhile simplification in any case, especially for Peter's > Parallel tuplesort patch set. Well, it removes more code than it adds. That's definitel

Re: [HACKERS] Measuring replay lag

2016-12-21 Thread Fujii Masao
On Mon, Dec 19, 2016 at 8:13 PM, Thomas Munro wrote: > On Mon, Dec 19, 2016 at 4:03 PM, Peter Eisentraut > wrote: >> On 11/22/16 4:27 AM, Thomas Munro wrote: >>> Thanks very much for testing! New version attached. I will add this >>> to the next CF. >> >> I don't see it there yet. > > Thanks fo

Re: [HACKERS] Replication slot xmin is not reset if HS feedback is turned off while standby is shut down

2016-12-21 Thread Ants Aasma
On Wed, Dec 21, 2016 at 2:09 PM, Craig Ringer wrote: > On 21 December 2016 at 15:40, Ants Aasma wrote: > >>> So -1 on this part of the patch, unless there's something I've >>> misunderstood. >> >> Currently there was no feedback sent if hot standby was not active. I >> was not sure if it was saf

Re: [HACKERS] [COMMITTERS] pgsql: Simplify LWLock tranche machinery by removing array_base/array_s

2016-12-21 Thread Pavel Stehule
2016-12-21 13:54 GMT+01:00 Robert Haas : > On Wed, Dec 21, 2016 at 2:14 AM, Pavel Stehule > wrote: > > Is there some help for extensions developers, how to fix extensions after > > this change? > > > > Orafce hits this change. > > array_base and array_stride are gone; don't pass them. The second

Re: [HACKERS] Speedup twophase transactions

2016-12-21 Thread Michael Paquier
On Fri, Dec 16, 2016 at 8:00 PM, Stas Kelvich wrote: > So, here is brand new implementation of the same thing. > > Now instead of creating pgproc entry for prepared transaction during > recovery, > I just store recptr/xid correspondence in separate 2L-list and deleting > entries in that > list if

Re: [HACKERS] [COMMITTERS] pgsql: Simplify LWLock tranche machinery by removing array_base/array_s

2016-12-21 Thread Robert Haas
On Wed, Dec 21, 2016 at 2:14 AM, Pavel Stehule wrote: > Is there some help for extensions developers, how to fix extensions after > this change? > > Orafce hits this change. array_base and array_stride are gone; don't pass them. The second argument to LWLockRegisterTranche() is now the tranche n

Re: [HACKERS] postgres_fdw bug in 9.6

2016-12-21 Thread Etsuro Fujita
On 2016/12/20 0:37, Tom Lane wrote: Etsuro Fujita writes: On 2016/12/17 1:13, Tom Lane wrote: So I think the rule could be "When first asked to produce a path for a given foreign joinrel, collect the cheapest paths for its left and right inputs, and make a nestloop path (or hashjoin path, i

Re: [HACKERS] Rethinking our fulltext phrase-search implementation

2016-12-21 Thread Artur Zakirov
Hello Tom, On 17.12.2016 21:36, Tom Lane wrote: 4. The transformations are wrong anyway. The OR case I showed above is all right, but as I argued in <24331.1480199...@sss.pgh.pa.us>, the AND case is not: regression=# select 'a <-> (b & c)'::tsquery; tsquery -

Re: [HACKERS] Logical tape pause/resume

2016-12-21 Thread Heikki Linnakangas
On 10/12/2016 05:30 PM, Heikki Linnakangas wrote: On 10/10/2016 10:31 PM, Peter Geoghegan wrote: On Sun, Oct 9, 2016 at 11:52 PM, Heikki Linnakangas wrote: Ok, good. I think we're in agreement on doing this, then, even if we don't agree on the justification :-). Agreed. :-) Attached are ne

Re: [HACKERS] Replication slot xmin is not reset if HS feedback is turned off while standby is shut down

2016-12-21 Thread Craig Ringer
On 21 December 2016 at 15:40, Ants Aasma wrote: >> So -1 on this part of the patch, unless there's something I've misunderstood. > > Currently there was no feedback sent if hot standby was not active. I > was not sure if it was safe to call GetOldestXmin() in that case. > However I did not consid

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-12-21 Thread Heikki Linnakangas
On 12/21/2016 12:53 AM, Robert Haas wrote: That leaves one problem, though: reusing space in the final merge phase. If the tapes being merged belong to different LogicalTapeSets, and create one new tape to hold the result, the new tape cannot easily reuse the space of the input tapes because they

Re: [HACKERS] SET NOT NULL [NOT VALID / CONCURRENTLY]?

2016-12-21 Thread Joel Jacobson
If you are fully confident you have no NULL values, e.g. if you have all your logics in db functions and you validate all INSERTs to a table won't pass any NULL values, and you have checked all the rows in a table are NOT NULL for the column, would it be completely crazy to just set pg_attribute.at

Re: [HACKERS] SET NOT NULL [NOT VALID / CONCURRENTLY]?

2016-12-21 Thread Craig Ringer
On 21 December 2016 at 19:01, Joel Jacobson wrote: > Similar to what we (Trustly) did when we sponsored the FOR KEY LOCK > feature to improve concurrency, > we would be very interested in also sponsoring this feature, as it > would mean a great lot to us. > I don't know if this is the right forum

[HACKERS] multi-level partitions and partition-wise joins

2016-12-21 Thread Ashutosh Bapat
Hi, I am starting this as a separate thread for this since the declarative partitioning thread has many issues reported and it's better to keep this discussion separate from the issues reported on that thread. While expanding inheritance, any inheritance hierarchy is flattened out including partit

Re: [HACKERS] invalid combination of options "-D - -F t -X stream" in pg_basebackup

2016-12-21 Thread Fujii Masao
On Tue, Dec 20, 2016 at 9:22 PM, Magnus Hagander wrote: > > > On Tue, Dec 20, 2016 at 6:56 AM, Fujii Masao wrote: >> >> On Tue, Dec 20, 2016 at 1:43 AM, Magnus Hagander >> wrote: >> > >> > >> > On Mon, Dec 19, 2016 at 5:39 PM, Fujii Masao >> > wrote: >> >> >> >> Hi, >> >> >> >> Isn't it better

Re: [HACKERS] SET NOT NULL [NOT VALID / CONCURRENTLY]?

2016-12-21 Thread Joel Jacobson
On Wed, Dec 21, 2016 at 4:24 PM, Craig Ringer wrote: >> Is anyone working on fixing this for PostgreSQL 10? > > Not as far as I know. > > IMO this and other similar cases should all be handled the same way: > create the constraint NOT VALID, then VALIDATE it while holding a weak > lock that only b

Re: [HACKERS] Declarative partitioning - another take

2016-12-21 Thread Amit Langote
On 2016/12/21 1:53, Robert Haas wrote: > On Mon, Dec 19, 2016 at 10:59 PM, Robert Haas wrote: >> On Sun, Dec 18, 2016 at 10:00 PM, Amit Langote >> wrote: >>> Here are updated patches including the additional information. >> >> Thanks. Committed 0001. Will have to review the others when I'm less

Re: [HACKERS] pg_background contrib module proposal

2016-12-21 Thread Craig Ringer
On 21 December 2016 at 14:26, Andrew Borodin wrote: > I'm not sure every platform supports microsecond sleeps Windows at least doesn't by default, unless that changed in Win2k12 and Win8 with the same platform/kernel improvements that delivered https://msdn.microsoft.com/en-us/library/hh706895(v

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-12-21 Thread Fujii Masao
On Wed, Dec 21, 2016 at 10:39 AM, Kyotaro HORIGUCHI wrote: > At Tue, 20 Dec 2016 23:47:22 +0900, Fujii Masao wrote > in >> On Tue, Dec 20, 2016 at 2:46 PM, Michael Paquier >> wrote: >> > On Tue, Dec 20, 2016 at 2:31 PM, Masahiko Sawada >> > wrote: >> >> Do we need to consider the sorting met

  1   2   >