Re: [HACKERS] OK, so culicidae is *still* broken

2017-04-18 Thread Amit Kapila
On Sun, Apr 16, 2017 at 3:04 AM, Tom Lane wrote: > Andres Freund writes: >> On 2017-04-15 17:24:54 -0400, Tom Lane wrote: >>> I wonder whether we could work around that by just destroying the created >>> process and trying again if we get a collision. It'd be a tad >>> inefficient, but hopefully

Re: [HACKERS] Other formats in pset like markdown, rst, mediawiki

2017-04-18 Thread Jan Michálek
2017-04-18 23:06 GMT+02:00 Fabien COELHO : > > Hello, > > There are different flavour of markdown, maybe you should document which >>> one is targetted. Should it be CommonMark? Another variant? Why? >>> >> >> This should be pandoc pipe table. It's because it is similar to aligned >> format. I nee

Re: [HACKERS] Assertion failure in REL9_5_STABLE

2017-04-18 Thread Pavan Deolasee
On Wed, Apr 19, 2017 at 11:19 AM, Andrew Gierth wrote: > > "Pavan" == Pavan Deolasee writes: > > Pavan> I am attaching a patch that throws a similar ERROR during > Pavan> planning even for 9.5. AFAICS in presence of grouping sets, we > Pavan> always decide to use sort-based implementation

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2017-04-18 Thread Michael Paquier
On Wed, Apr 19, 2017 at 1:52 PM, Masahiko Sawada wrote: > On Wed, Apr 19, 2017 at 12:34 PM, Noah Misch wrote: >> On Sun, Apr 16, 2017 at 07:25:28PM +0900, Fujii Masao wrote: >>> On Sun, Apr 16, 2017 at 1:19 PM, Noah Misch wrote: >>> > On Fri, Apr 14, 2017 at 11:58:23PM -0400, Noah Misch wrote: >

Re: [HACKERS] Assertion failure in REL9_5_STABLE

2017-04-18 Thread Andrew Gierth
> "Pavan" == Pavan Deolasee writes: Pavan> I am attaching a patch that throws a similar ERROR during Pavan> planning even for 9.5. AFAICS in presence of grouping sets, we Pavan> always decide to use sort-based implementation for grouping, but Pavan> do not check if the columns support ord

Re: [HACKERS] logical replication and PANIC during shutdown checkpoint in publisher

2017-04-18 Thread Michael Paquier
On Tue, Apr 18, 2017 at 3:27 AM, Peter Eisentraut wrote: > I'd imagine the postmaster would tell the walsender that it has started > shutdown, and then the walsender would reject $certain_things. But I > don't see an existing way for the walsender to know that shutdown has > been initiated. SIGI

[HACKERS] Assertion failure in REL9_5_STABLE

2017-04-18 Thread Pavan Deolasee
Hi All, I stumbled upon this test case from the master that sets an assertion failure (see stack trace at the end) on REL9_5_STABLE. create temp table gstest4(id integer, v integer, unhashable_col bit(4), unsortable_col xid); insert into gstest4 values (1,1,b'','1'),

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2017-04-18 Thread Masahiko Sawada
On Wed, Apr 19, 2017 at 12:34 PM, Noah Misch wrote: > On Sun, Apr 16, 2017 at 07:25:28PM +0900, Fujii Masao wrote: >> On Sun, Apr 16, 2017 at 1:19 PM, Noah Misch wrote: >> > On Fri, Apr 14, 2017 at 11:58:23PM -0400, Noah Misch wrote: >> >> On Wed, Apr 05, 2017 at 09:51:02PM -0400, Noah Misch wrot

Re: [HACKERS] Should pg_current_wal_location() become pg_current_wal_lsn()

2017-04-18 Thread Michael Paquier
On Wed, Apr 19, 2017 at 12:36 PM, David Rowley wrote: > In favour of "location" -> "lsn": Tom, Stephen, David Steel > In favour of "lsn" -> "location": Peter, Kyotaro I vote for "location" -> "lsn". I would expect complains about the current inconsistency at some point, and the function names hav

[HACKERS] Fixup some misusage of appendStringInfo and friends

2017-04-18 Thread David Rowley
The attached cleans up a few small misusages of appendStringInfo and related functions. Some similar work was done in, f92d6a540ac443f85f0929b284edff67da14687a d02f16470f117db3038dbfd87662d5f0eb5a2a9b cacbdd78106526d7c4f11f90b538f96ba8696fb0 so the majority of these should all be new misuseages.

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-18 Thread Tom Lane
Amit Kapila writes: > On Wed, Apr 19, 2017 at 1:27 AM, Tom Lane wrote: >> I think leaving that sort of thing out is just creating a latent bug >> that is certain to bite you later. It's true that as long as the args >> list contains only Vars, it would never be parallel-unsafe --- but >> primnod

Re: [HACKERS] Should pg_current_wal_location() become pg_current_wal_lsn()

2017-04-18 Thread David Rowley
On 19 April 2017 at 15:31, Tom Lane wrote: > David Rowley writes: >> OK, so I've read over this thread again and I think it's time to >> summarise the votes: >> ... >> In favour of "location" -> "lsn": Stephen, David Steel, >> In favour of "lsn" -> "location": Peter, Tom, Kyotaro > > FWIW, I was

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2017-04-18 Thread Noah Misch
On Sun, Apr 16, 2017 at 07:25:28PM +0900, Fujii Masao wrote: > On Sun, Apr 16, 2017 at 1:19 PM, Noah Misch wrote: > > On Fri, Apr 14, 2017 at 11:58:23PM -0400, Noah Misch wrote: > >> On Wed, Apr 05, 2017 at 09:51:02PM -0400, Noah Misch wrote: > >> > On Thu, Apr 06, 2017 at 12:48:56AM +0900, Fujii

Re: [HACKERS] Should pg_current_wal_location() become pg_current_wal_lsn()

2017-04-18 Thread Tom Lane
David Rowley writes: > OK, so I've read over this thread again and I think it's time to > summarise the votes: > ... > In favour of "location" -> "lsn": Stephen, David Steel, > In favour of "lsn" -> "location": Peter, Tom, Kyotaro FWIW, I was not voting in favor of "location"; I was just saying t

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-18 Thread Amit Kapila
On Wed, Apr 19, 2017 at 1:27 AM, Tom Lane wrote: > Amit Kapila writes: >> I have ended up doing something along the lines suggested by you (or >> at least what I have understood from your e-mail). Basically, pass >> the safe-param-ids list to parallel safety function and decide based >> on that

Re: [HACKERS] Should pg_current_wal_location() become pg_current_wal_lsn()

2017-04-18 Thread David Rowley
On 19 April 2017 at 03:33, David Steele wrote: > +1, and I'm in favor of using "lsn" wherever applicable. It seems to me > that if a user calls a function (or queries a table) that returns an lsn > then they should be aware of what an lsn is. OK, so I've read over this thread again and I think i

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Michael Paquier
On Wed, Apr 19, 2017 at 11:09 AM, Jeff Janes wrote: > Would this bug have been seen in a replica server in the absence of crashes, > or was it only vulnerable during crash recovery rather than streaming > replication? This issue could have been seen on a streaming standby as well, letting around

[HACKERS] Use sync commit for logical replication apply in TAP tests

2017-04-18 Thread Petr Jelinek
Hi, The commit 887227a1c changed the defaults for subscriptions to do async commit. But since the tests often wait for disk flush and there is no concurrent activity this has increased the amount of time needed for each test. So the attached patch changes the subscriptions create in tab tests to u

Re: [HACKERS] some review comments on logical rep code

2017-04-18 Thread Petr Jelinek
On 18/04/17 19:27, Fujii Masao wrote: > On Wed, Apr 19, 2017 at 1:35 AM, Petr Jelinek > wrote: >> Thank you for working on this! >> >> On 18/04/17 10:16, Masahiko Sawada wrote: >>> On Tue, Apr 18, 2017 at 12:24 PM, Kyotaro HORIGUCHI >>> wrote: >> 10. > > SpinLockA

Re: [HACKERS] tablesync patch broke the assumption that logical rep depends on?

2017-04-18 Thread Petr Jelinek
On 17/04/17 20:09, Peter Eisentraut wrote: > On 4/16/17 22:01, Noah Misch wrote: >> This PostgreSQL 10 open item is past due for your status update. Kindly send >> a status update within 24 hours, and include a date for your subsequent >> status >> update. Refer to the policy on open item owners

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Jeff Janes
On Tue, Apr 18, 2017 at 1:17 AM, Nikhil Sontakke wrote: > Hi, > > There was a bug in the redo 2PC remove code path. Because of which, > autovac would think that the 2PC is gone and cause removal of the > corresponding clog entry earlier than needed. > > Please find attached, the bug fix: 2pc_redo

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Jeff Janes
On Tue, Apr 18, 2017 at 5:38 AM, Simon Riggs wrote: > On 18 April 2017 at 13:12, Michael Paquier > wrote: > > On Tue, Apr 18, 2017 at 7:54 PM, Simon Riggs > wrote: > >> Yeh, this is better. Pushed. > > > > I have been outraced on this one, the error is obvious once you see it ;) > > Didn't real

[HACKERS] [PATCH] Push limit to sort through a subquery

2017-04-18 Thread Douglas Doole
We've hit a case where pass_down_bound() isn't pushing the row count limit from limit into sort. The issue is that we're getting a subquery scan node between the limit and the sort. The subquery is only doing column projection and has no quals or SRFs so it should be safe to push the limit into the

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Michael Paquier
On Wed, Apr 19, 2017 at 12:48 AM, Tom Lane wrote: > Alvaro Herrera writes: >> Tom Lane wrote: >>> FWIW, I'm a bit suspicious of relocating the temp stats directory as >>> being a reliable fix for this. > >> It's an SD card (the kind typically used in cameras and phones), not SSD. >> Saying it's s

Re: [HACKERS] Other formats in pset like markdown, rst, mediawiki

2017-04-18 Thread Fabien COELHO
Hello, There are different flavour of markdown, maybe you should document which one is targetted. Should it be CommonMark? Another variant? Why? This should be pandoc pipe table. It's because it is similar to aligned format. I need add this to documentation (i have it in recent TODO) I stil

Re: [HACKERS] Passing values to a dynamic background worker

2017-04-18 Thread Keith Fiske
On Tue, Apr 18, 2017 at 12:34 PM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 4/17/17 16:19, Keith Fiske wrote: > > I've reached a roadblock in that bgw_main_arg can only accept a single > > argument that must be passed by value for a dynamic bgw. I already > > worked around t

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-18 Thread Tom Lane
Amit Kapila writes: > I have ended up doing something along the lines suggested by you (or > at least what I have understood from your e-mail). Basically, pass > the safe-param-ids list to parallel safety function and decide based > on that if Param node reference in input expression is safe. I

Re: [HACKERS] Quals not pushed down into lateral

2017-04-18 Thread David Fetter
On Thu, Apr 13, 2017 at 01:39:07PM -0700, Andres Freund wrote: > On 2017-04-13 16:34:12 -0400, Robert Haas wrote: > > On Thu, Mar 16, 2017 at 4:45 AM, Andres Freund wrote: > > > During citus development we noticed that restrictions aren't pushed down > > > into lateral subqueries, even if they sem

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-18 Thread Petr Jelinek
On 18/04/17 19:18, Tom Lane wrote: > Petr Jelinek writes: >> On 18/04/17 18:24, Peter Eisentraut wrote: >>> I don't see why we need to do that. It is showing the correct >>> information, isn't it? > >> It does, but it's also one of the things Tom complained about and I >> think he is right in th

Re: [HACKERS] [PATCH] New command to monitor progression of long running queries

2017-04-18 Thread Maksim Milyutin
On 18.04.2017 17:39, Remi Colinet wrote: Hello Maksim, The core implementation I suggested for the new PROGRESS command uses different functions from the one used by EXPLAIN for its core implementation. Some source code is shared with EXPLAIN command. But this shared code is only related to qual

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2017-04-18 Thread Fujii Masao
On Tue, Apr 18, 2017 at 7:02 PM, Masahiko Sawada wrote: > On Tue, Apr 18, 2017 at 6:40 PM, Kyotaro HORIGUCHI > wrote: >> At Tue, 18 Apr 2017 14:58:50 +0900, Masahiko Sawada >> wrote in >>> On Tue, Apr 18, 2017 at 3:04 AM, Fujii Masao wrote: >>> > On Wed, Apr 12, 2017 at 2:36 AM, Masahiko Sawa

[HACKERS] SASL minor docs typo

2017-04-18 Thread Jaime Casanova
Hi, reading SASL docs found this typo: in protocol.sgml:1356 """ To begin a SASL authentication exchange, the server an AuthenticationSASL message. """ I guess it should say "the server sends an AuthenticationSASL message" -- Jaime Casanova www.2ndQuadrant.com PostgreSQL

Re: [HACKERS] Re: Query fails when SRFs are part of FROM clause (Commit id: 69f4b9c85f)

2017-04-18 Thread Tom Lane
Andres Freund writes: > On 2017-04-17 19:26:11 -0400, Tom Lane wrote: >> If we are going to go down this road, I think it would be a good idea >> to try to provide a cursor position for the "can't accept a set" error >> message, because otherwise it will be really unclear what's wrong with >> some

Re: [HACKERS] Other formats in pset like markdown, rst, mediawiki

2017-04-18 Thread Jan Michálek
2017-04-18 12:06 GMT+02:00 Fabien COELHO : > > Hello Jan, > > It seems that the patch does not apply anymore on head due to changes in >>> psql non regression tests. Could you rebase? >>> >> >> This should work on current master (all test passed). >> > > Patch applies, compiles and make check is o

Re: [HACKERS] some review comments on logical rep code

2017-04-18 Thread Fujii Masao
On Wed, Apr 19, 2017 at 1:35 AM, Petr Jelinek wrote: > Thank you for working on this! > > On 18/04/17 10:16, Masahiko Sawada wrote: >> On Tue, Apr 18, 2017 at 12:24 PM, Kyotaro HORIGUCHI >> wrote: >>> >>> 3. ApplyLauncherWakeup() should be static function. >>> >>> At

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-18 Thread Tom Lane
Petr Jelinek writes: > On 18/04/17 18:24, Peter Eisentraut wrote: >> I don't see why we need to do that. It is showing the correct >> information, isn't it? > It does, but it's also one of the things Tom complained about and I > think he is right in that at least values for launcher should be >

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-18 Thread Petr Jelinek
On 18/04/17 18:24, Peter Eisentraut wrote: > On 4/18/17 12:13, Petr Jelinek wrote: >> We can definitely easily detect that the bgworker is internal one by >> library_name equals 'postgres' so we can easily remove the usesysid and >> usename based on that. > > I don't see why we need to do that. I

Re: [HACKERS] Interval for launching the table sync worker

2017-04-18 Thread Petr Jelinek
On 18/04/17 18:14, Peter Eisentraut wrote: > On 4/18/17 11:59, Petr Jelinek wrote: >> Hmm if we create hashtable for this, I'd say create hashtable for the >> whole table_states then. The reason why it's list now was that it seemed >> unnecessary to have hashtable when it will be empty almost alway

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-18 Thread Tom Lane
Peter Eisentraut writes: > I think showing bgw_name as backend_type always sounds reasonable. No > need to treat external implementations differently. That's definitely an approach we could use. It would encourage people to use short bgw_names, which is a constraint that wasn't especially appar

Re: [HACKERS] some review comments on logical rep code

2017-04-18 Thread Petr Jelinek
Thank you for working on this! On 18/04/17 10:16, Masahiko Sawada wrote: > On Tue, Apr 18, 2017 at 12:24 PM, Kyotaro HORIGUCHI > wrote: >> >> 3. >>> >>> ApplyLauncherWakeup() should be static function. >> >> Attached 003 patch fixes it (and also fixes #5 review comment). >

Re: [HACKERS] Passing values to a dynamic background worker

2017-04-18 Thread Peter Eisentraut
On 4/17/17 16:19, Keith Fiske wrote: > I've reached a roadblock in that bgw_main_arg can only accept a single > argument that must be passed by value for a dynamic bgw. I already > worked around this for passing the database name to the my existing use > of a bgw with doing partition maintenance (p

Re: [HACKERS] Logical replication and synchronous replication

2017-04-18 Thread Masahiko Sawada
On Tue, Apr 18, 2017 at 9:05 PM, Craig Ringer wrote: > On 18 April 2017 at 18:55, Masahiko Sawada wrote: >> Hi, >> >> As doc of logical decoding said as a note[1], logical replication can >> support the synchronous replication with some restriction. But In >> addition to this, IIUC in logical rep

Re: [HACKERS] Logical replication launcher uses wal_retrieve_retry_interval

2017-04-18 Thread Peter Eisentraut
On 4/18/17 12:00, Petr Jelinek wrote: > As for apply_worker_launch_interval, I think we want different > name so that it can be used for tablesync rate limiting as well. But that's a mechanism we don't have yet, so maybe we should design that when we get there? -- Peter Eisentraut h

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-18 Thread Peter Eisentraut
On 4/18/17 12:13, Petr Jelinek wrote: > We can definitely easily detect that the bgworker is internal one by > library_name equals 'postgres' so we can easily remove the usesysid and > usename based on that. I don't see why we need to do that. It is showing the correct information, isn't it? > B

Re: [HACKERS] scram and \password

2017-04-18 Thread Simon Riggs
On 18 April 2017 at 09:25, Heikki Linnakangas wrote: > On 04/18/2017 11:15 AM, Simon Riggs wrote: >> >> As a potential open item, if we treat "md5" as ">= md5" >> should we not also treat "password" as ">=password"? >> >> It seems strange that we still support "password" and yet tell >> everyoneno

[HACKERS] logical replication fixes

2017-04-18 Thread Euler Taveira
Hi, While inspecting the logical replication code, I found a bug that could pick the wrong remote relation if they have the same name but different schemas. Also, I did some spelling/cosmetic changes and fixed an oversight in the ALTER SUBSCRIPTION documentation. Patches attached. -- Euler T

Re: [HACKERS] Interval for launching the table sync worker

2017-04-18 Thread Peter Eisentraut
On 4/18/17 11:59, Petr Jelinek wrote: > Hmm if we create hashtable for this, I'd say create hashtable for the > whole table_states then. The reason why it's list now was that it seemed > unnecessary to have hashtable when it will be empty almost always but > there is no need to have both hashtable

Re: [HACKERS] Interval for launching the table sync worker

2017-04-18 Thread Masahiko Sawada
On Tue, Apr 18, 2017 at 11:22 PM, Peter Eisentraut wrote: > On 4/13/17 06:23, Masahiko Sawada wrote: >> Attached the latest patch. It didn't actually necessary to change >> GetSubscriptionNotReadyRelations. I just changed the logic refreshing >> the sync table state list. > > I think this was the

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-18 Thread Petr Jelinek
On 16/04/17 22:27, Petr Jelinek wrote: > On 16/04/17 18:47, Tom Lane wrote: >> Craig Ringer writes: >>> On 12 April 2017 at 13:34, Kuntal Ghosh wrote: For backend_type=background worker, application_name shows the name of the background worker (BackgroundWorker->bgw_name). I think we ne

[HACKERS] dtrace probes

2017-04-18 Thread Jesper Pedersen
Hi, The lwlock dtrace probes define LWLockMode as int, and the TRACE_POSTGRESQL_LWLOCK methods are called using both a variable and constant definition. This leads to a mix of argument definitions depending on the call site, as seen in probes.txt file. A fix is to explicit cast 'mode' to i

Re: [HACKERS] Logical replication launcher uses wal_retrieve_retry_interval

2017-04-18 Thread Petr Jelinek
On 18/04/17 16:24, Peter Eisentraut wrote: > On 4/16/17 22:40, Masahiko Sawada wrote: >> Attached two patches add new GUCs apply_worker_timeout and >> apply_worker_launch_interval which are used instead of >> wal_receiver_timeout and wal_retrieve_retry_timeout. These new >> parameters are not setta

Re: [HACKERS] Interval for launching the table sync worker

2017-04-18 Thread Petr Jelinek
On 18/04/17 16:22, Peter Eisentraut wrote: > On 4/13/17 06:23, Masahiko Sawada wrote: >> Attached the latest patch. It didn't actually necessary to change >> GetSubscriptionNotReadyRelations. I just changed the logic refreshing >> the sync table state list. > > I think this was the right direction

Re: [HACKERS] identity columns

2017-04-18 Thread Peter Eisentraut
On 4/18/17 02:07, Noah Misch wrote: > This PostgreSQL 10 open item is past due for your status update. Kindly send > a status update within 24 hours, and include a date for your subsequent status > update. Refer to the policy on open item ownership: > https://www.postgresql.org/message-id/2017040

Re: [HACKERS] identity columns

2017-04-18 Thread Peter Eisentraut
On 4/7/17 01:26, Vitaly Burovoy wrote: > I've implement SET GENERATED ... IF NOT EXISTS. It must be placed > before other SET options but fortunately it conforms with the > standard. > Since that form always changes the sequence behind the column, I > decided to explicitly write "[NO] CACHE" in pg_

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Tom Lane
Alvaro Herrera writes: > Tom Lane wrote: >> FWIW, I'm a bit suspicious of relocating the temp stats directory as >> being a reliable fix for this. > It's an SD card (the kind typically used in cameras and phones), not SSD. > Saying it's slow is an understatement. It's *excruciatingly* slow. Oh,

Re: [HACKERS] Should pg_current_wal_location() become pg_current_wal_lsn()

2017-04-18 Thread David Steele
On 4/15/17 12:56 PM, Robert Haas wrote: > On Fri, Apr 14, 2017 at 6:52 PM, Tom Lane wrote: >> Peter Eisentraut writes: >>> If we're talking about making things easier to understand, wouldn't a >>> random user rather know what a WAL "location" is instead of a WAL "LSN"? >> I wouldn't object to sta

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Alvaro Herrera
Tom Lane wrote: > Michael Paquier writes: > > That's the point I am trying to make upthread: slow buildfarm animals > > should have minimal impact on core code modifications. We could for > > example have one environment variable that lists all the parameters to > > modify in a single string and a

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Tom Lane
Michael Paquier writes: > That's the point I am trying to make upthread: slow buildfarm animals > should have minimal impact on core code modifications. We could for > example have one environment variable that lists all the parameters to > modify in a single string and appends them at the end of

Re: [HACKERS] scram and \password

2017-04-18 Thread Heikki Linnakangas
On 04/18/2017 08:44 AM, Noah Misch wrote: On Tue, Apr 11, 2017 at 10:07:12PM +0300, Heikki Linnakangas wrote: Thanks! I've been busy on the other thread on future-proofing the protocol with negotiating the SASL mechanism, I'll come back to this once we get that settled. By the end of the week, I

Re: [HACKERS] [PATCH] New command to monitor progression of long running queries

2017-04-18 Thread Remi Colinet
Hello Maksim, The core implementation I suggested for the new PROGRESS command uses different functions from the one used by EXPLAIN for its core implementation. Some source code is shared with EXPLAIN command. But this shared code is only related to quals, properties, children, subPlans and few o

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-18 Thread Amit Kapila
On Mon, Apr 17, 2017 at 10:54 AM, Tom Lane wrote: > Amit Kapila writes: >> On Sun, Apr 16, 2017 at 9:30 PM, Tom Lane wrote: >>> FYI, I have this on my to-look-at list, and expect to fix it before Robert >>> returns from vacation. > >> Let me know if an initial patch by someone else can be helpfu

Re: [HACKERS] Logical replication launcher uses wal_retrieve_retry_interval

2017-04-18 Thread Peter Eisentraut
On 4/16/17 22:40, Masahiko Sawada wrote: > Attached two patches add new GUCs apply_worker_timeout and > apply_worker_launch_interval which are used instead of > wal_receiver_timeout and wal_retrieve_retry_timeout. These new > parameters are not settable at worker-level so far. Under what circumsta

Re: [HACKERS] Interval for launching the table sync worker

2017-04-18 Thread Peter Eisentraut
On 4/13/17 06:23, Masahiko Sawada wrote: > Attached the latest patch. It didn't actually necessary to change > GetSubscriptionNotReadyRelations. I just changed the logic refreshing > the sync table state list. I think this was the right direction, but then I got worried about having a loop within

Re: [HACKERS] Passing values to a dynamic background worker

2017-04-18 Thread Keith Fiske
On Tue, Apr 18, 2017 at 5:40 AM, Amit Langote wrote: > On 2017/04/18 18:12, Kyotaro HORIGUCHI wrote: > > At Mon, 17 Apr 2017 16:19:13 -0400, Keith Fiske wrote: > >> So after reading a recent thread on the steep learning curve for PG > >> internals [1], I figured I'd share where I've gotten stuck

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Andrew Dunstan
On 04/18/2017 08:59 AM, Michael Paquier wrote: >> Lets's say we have a bunch of possible environment settings with names >> that all begin with "PG_TAP_" PostgresNode.pm could check for the >> existence of these and take action accordingly, and you could set them >> on a buildfarm animal in the c

Re: [HACKERS] [PATCH] New command to monitor progression of long running queries

2017-04-18 Thread Maksim Milyutin
Hi! On 17.04.2017 15:09, Remi Colinet wrote: Hello, I've implemented a new command named PROGRESS to monitor progression of long running SQL queries in a backend process. Use case === A use case is shown in the below example based on a table named t_10m with 10 millions rows. The table

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Michael Paquier
On Tue, Apr 18, 2017 at 9:35 PM, Andrew Dunstan wrote: > > > On 04/18/2017 08:23 AM, Michael Paquier wrote: >> On Tue, Apr 18, 2017 at 9:13 PM, Andrew Dunstan >> wrote: >>> Yeah, but the way you have done it could also to lead to errors unless >>> you're very careful, as I found on axolotl (which

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Simon Riggs
On 18 April 2017 at 13:12, Michael Paquier wrote: > On Tue, Apr 18, 2017 at 7:54 PM, Simon Riggs wrote: >> Yeh, this is better. Pushed. > > I have been outraced on this one, the error is obvious once you see it ;) Didn't realise you were working on it, nothing competitive about it. It's clear t

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Andrew Dunstan
On 04/18/2017 08:23 AM, Michael Paquier wrote: > On Tue, Apr 18, 2017 at 9:13 PM, Andrew Dunstan > wrote: >> Yeah, but the way you have done it could also to lead to errors unless >> you're very careful, as I found on axolotl (which died recently, >> unfortunately). There I had to set the stats_

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Andrew Dunstan
On 04/18/2017 03:54 AM, Craig Ringer wrote: >> But almost nothing about The Internals of PostgreSQL: > Not surprising. They'd go out of date fast, be a huge effort to write > and maintain, and sell poorly given the small audience. > > Print books probably aren't the way forward here. > Agreed,

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Andrew Dunstan
On 04/18/2017 03:41 AM, Kang Yuzhe wrote: > > > But almost nothing about The Internals of PostgreSQL: > 1. The Internals of PostgreSQL: > http://www.interdb.jp/pg/index.html translated from Japanese Book > 2. PostgreSQL数据库内核分析(Chinese) Book on the Internals of PostgreSQL: > 3. PG Docs/site > 4.

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Michael Paquier
On Tue, Apr 18, 2017 at 9:13 PM, Andrew Dunstan wrote: > Yeah, but the way you have done it could also to lead to errors unless > you're very careful, as I found on axolotl (which died recently, > unfortunately). There I had to set the stats_temp directory to a > branch-specific name so a crash on

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Andrew Dunstan
On 04/18/2017 03:07 AM, Michael Paquier wrote: > Hi all, > > Some of you may have noticed that hamster is heavily red on the > buildfarm. I have done a bit of investigation, and I am able to > reproduce the failure manually. But actually after looking at the logs > the error has obviously showed

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Michael Paquier
On Tue, Apr 18, 2017 at 7:54 PM, Simon Riggs wrote: > Yeh, this is better. Pushed. I have been outraced on this one, the error is obvious once you see it ;) Thanks for the investigation and the fix! I have spent a couple of hours reviewing the interactions between the shmem entries of 2PC state

Re: [HACKERS] Logical replication and synchronous replication

2017-04-18 Thread Craig Ringer
On 18 April 2017 at 18:55, Masahiko Sawada wrote: > Hi, > > As doc of logical decoding said as a note[1], logical replication can > support the synchronous replication with some restriction. But In > addition to this, IIUC in logical replication decoded data is sent to > subscribers only when the

Re: [HACKERS] SCRAM authentication, take three

2017-04-18 Thread Magnus Hagander
On Tue, Apr 18, 2017 at 1:52 PM, Heikki Linnakangas wrote: > On 04/14/2017 10:33 PM, Peter Eisentraut wrote: > >> On 4/11/17 01:10, Heikki Linnakangas wrote: >> >>> That question won't arise in practice. Firstly, if the server can do >>> scram-sha-256-plus, it presumably can also do scram-sha-512

Re: [HACKERS] SCRAM authentication, take three

2017-04-18 Thread Heikki Linnakangas
On 04/14/2017 10:33 PM, Peter Eisentraut wrote: On 4/11/17 01:10, Heikki Linnakangas wrote: That question won't arise in practice. Firstly, if the server can do scram-sha-256-plus, it presumably can also do scram-sha-512-plus. Unless there's a change in the way the channel binding works, such th

[HACKERS] Logical replication ApplyContext bloat

2017-04-18 Thread Stas Kelvich
Hi, currently logical replication worker uses ApplyContext to decode received data and that context is never freed during transaction processing. Hence if publication side is performing something like 10M row inserts in single transaction, then subscription worker will occupy more than 10G of ram

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-04-18 Thread Maksim Milyutin
On 18.04.2017 13:08, Amit Langote wrote: Hi, Hi, Amit! On 2017/04/17 23:00, Maksim Milyutin wrote: Ok, thanks for the note. But I want to discuss the relevancy of introduction of a new relkind for partitioned index. I could to change the control flow in partitioned index creation (specify

[HACKERS] Logical replication and synchronous replication

2017-04-18 Thread Masahiko Sawada
Hi, As doc of logical decoding said as a note[1], logical replication can support the synchronous replication with some restriction. But In addition to this, IIUC in logical replication decoded data is sent to subscribers only when the commit WAL record is decoded (calls ReorderBufferCommit) . It

Re: [HACKERS] Partition-wise join for join between (declaratively) partitioned tables

2017-04-18 Thread Ashutosh Bapat
On Wed, Apr 5, 2017 at 8:39 AM, Robert Haas wrote: > On Tue, Apr 4, 2017 at 10:22 AM, Ashutosh Bapat > wrote: >> Yes, I agree. For an inner join, the partition key types need to "shrink" >> and for outer join they need to be "widened". I don't know if there is a way >> to know "wider" or "shorter

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Simon Riggs
On 18 April 2017 at 09:57, Nikhil Sontakke wrote: > Please find attached a second version of my bug fix which is stylistically > better and clearer than the first one. Yeh, this is better. Pushed. The bug was that the loop set gxact to be the last entry in the array, causing the exit condition

Re: [HACKERS] PANIC in pg_commit_ts slru after crashes

2017-04-18 Thread Michael Paquier
On Tue, Apr 18, 2017 at 7:05 PM, Simon Riggs wrote: > I've added a recheck in ProcessTwoPhaseBuffer() after we acquire the lock. > > If its worth acquiring the lock its worth checking we don't have a race. I see. No objections to that. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Michael Paquier
On Tue, Apr 18, 2017 at 4:15 PM, Andres Freund wrote: > Hi, > > On 2017-04-18 16:07:38 +0900, Michael Paquier wrote: >> Some of you may have noticed that hamster is heavily red on the >> buildfarm. I have done a bit of investigation, and I am able to >> reproduce the failure manually. But actually

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-04-18 Thread Amit Langote
Hi, On 2017/04/17 23:00, Maksim Milyutin wrote: > On 10.04.2017 14:20, Robert Haas wrote: >> On Tue, Apr 4, 2017 at 12:10 PM, Maksim Milyutin >> wrote: >>> 1. I have added a new relkind for local indexes named RELKIND_LOCAL_INDEX >>> (literal 'l'). >> >> Seems like it should maybe be RELKIND_PART

Re: [HACKERS] Other formats in pset like markdown, rst, mediawiki

2017-04-18 Thread Fabien COELHO
Hello Jan, It seems that the patch does not apply anymore on head due to changes in psql non regression tests. Could you rebase? This should work on current master (all test passed). Patch applies, compiles and make check is ok. There are different flavour of markdown, maybe you should doc

Re: [HACKERS] PANIC in pg_commit_ts slru after crashes

2017-04-18 Thread Simon Riggs
On 18 April 2017 at 09:51, Simon Riggs wrote: > On 17 April 2017 at 16:33, Jeff Janes wrote: >> On Sun, Apr 16, 2017 at 6:59 PM, Michael Paquier >> wrote: >>> >>> >>> >>> Jeff, does this patch make the situation better? The fix is rather >>> simple as it just makes sure that the next XID never g

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2017-04-18 Thread Masahiko Sawada
On Tue, Apr 18, 2017 at 6:40 PM, Kyotaro HORIGUCHI wrote: > At Tue, 18 Apr 2017 14:58:50 +0900, Masahiko Sawada > wrote in >> On Tue, Apr 18, 2017 at 3:04 AM, Fujii Masao wrote: >> > On Wed, Apr 12, 2017 at 2:36 AM, Masahiko Sawada >> > wrote: >> >> On Thu, Apr 6, 2017 at 4:17 PM, Masahiko S

Re: [HACKERS] Comments not allowed on partitioned table columns

2017-04-18 Thread Amit Langote
On 2017/04/18 18:45, Simon Riggs wrote: > On 18 April 2017 at 06:57, Amit Langote wrote: >> On 2017/04/18 14:50, Amit Langote wrote: >>> Attached patch fixes the oversight that COMMENT ON COLUMN is not allowed >>> on partitioned tables columns. >> >> Forgot to mention that I added this to the open

Re: [HACKERS] Comments not allowed on partitioned table columns

2017-04-18 Thread Simon Riggs
On 18 April 2017 at 06:57, Amit Langote wrote: > On 2017/04/18 14:50, Amit Langote wrote: >> Attached patch fixes the oversight that COMMENT ON COLUMN is not allowed >> on partitioned tables columns. > > Forgot to mention that I added this to the open items list. Pushed, thanks. -- Simon Riggs

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2017-04-18 Thread Kyotaro HORIGUCHI
At Tue, 18 Apr 2017 14:58:50 +0900, Masahiko Sawada wrote in > On Tue, Apr 18, 2017 at 3:04 AM, Fujii Masao wrote: > > On Wed, Apr 12, 2017 at 2:36 AM, Masahiko Sawada > > wrote: > >> On Thu, Apr 6, 2017 at 4:17 PM, Masahiko Sawada > >> wrote: > >>> On Thu, Apr 6, 2017 at 10:51 AM, Noah Mis

Re: [HACKERS] Passing values to a dynamic background worker

2017-04-18 Thread Amit Langote
On 2017/04/18 18:12, Kyotaro HORIGUCHI wrote: > At Mon, 17 Apr 2017 16:19:13 -0400, Keith Fiske wrote: >> So after reading a recent thread on the steep learning curve for PG >> internals [1], I figured I'd share where I've gotten stuck with this in a >> new thread vs hijacking that one. >> >> One o

Re: [HACKERS] Cutting initdb's runtime (Perl question embedded)

2017-04-18 Thread Dagfinn Ilmari Mannsåker
Tom Lane writes: > Andres Freund writes: >> On 2017-04-17 17:49:54 +0100, Dagfinn Ilmari Mannsåker wrote: >>> I threw Devel::NYTProf at it and picked some more low-hanging fruit. > >> I'm a bit doubtful about improving the performance of genbki at the cost >> of any sort of complication - it's o

Re: [HACKERS] Passing values to a dynamic background worker

2017-04-18 Thread Kyotaro HORIGUCHI
Hello, At Mon, 17 Apr 2017 16:19:13 -0400, Keith Fiske wrote in > So after reading a recent thread on the steep learning curve for PG > internals [1], I figured I'd share where I've gotten stuck with this in a > new thread vs hijacking that one. > > One of the goals I had with pg_partman was t

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Nikhil Sontakke
Please find attached a second version of my bug fix which is stylistically better and clearer than the first one. Regards, Nikhils On 18 April 2017 at 13:47, Nikhil Sontakke wrote: > Hi, > > There was a bug in the redo 2PC remove code path. Because of which, > autovac would think that the 2PC i

Re: [HACKERS] CREATE TRIGGER document typo

2017-04-18 Thread Heikki Linnakangas
On 04/17/2017 12:09 PM, Yugo Nagata wrote: Hi, Attached is a patch fixing simple typos in the CREATE TRIGGER document. Applied, thanks! - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pg

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Kang Yuzhe
Hello Amit, Thanks gain for being patient with me. YES, I am working with the PostgreSQL source git repository but I don't think I updated my local forked/cloned branch. I am also working on standalone PG 9.6.2 source code as well. I will try to fetch/pull the PG master content to my forked/cloned

Re: [HACKERS] PANIC in pg_commit_ts slru after crashes

2017-04-18 Thread Simon Riggs
On 17 April 2017 at 16:33, Jeff Janes wrote: > On Sun, Apr 16, 2017 at 6:59 PM, Michael Paquier > wrote: >> >> >> >> Jeff, does this patch make the situation better? The fix is rather >> simple as it just makes sure that the next XID never gets updated if >> there are no 2PC files. > > > Yes, tha

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Kang Yuzhe
Thanks Craig for teaching me a lot of things. I am just learning a lot why PG hacking/development is the way it is. Regarding interest and enthusiasm, no problem. Whats is lacking is the skill-sets and I believe having interest and enthusiasm and with your support, we will expand PG hacking/devs/u

Re: [HACKERS] scram and \password

2017-04-18 Thread Heikki Linnakangas
On 04/18/2017 11:15 AM, Simon Riggs wrote: As a potential open item, if we treat "md5" as ">= md5" should we not also treat "password" as ">=password"? It seems strange that we still support "password" and yet tell everyonenot to use it. I'd like PG10 to be the version where I don't have to tel

  1   2   >