[HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Michael Paquier
Hi all, Some of you may have noticed that hamster is heavily red on the buildfarm. I have done a bit of investigation, and I am able to reproduce the failure manually. But actually after looking at the logs the error has obviously showed up: 2017-04-16 05:07:19.650 JST [18282] LOG: database syste

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Andres Freund
Hi, On 2017-04-18 16:07:38 +0900, Michael Paquier wrote: > Some of you may have noticed that hamster is heavily red on the > buildfarm. I have done a bit of investigation, and I am able to > reproduce the failure manually. But actually after looking at the logs > the error has obviously showed up:

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Kang Yuzhe
Thanks Simon for taking your time and trying to tell and warn me the harsh reality truth:there is no shortcut to expertise. One has to fail and rise towards any journey to expertise. Overall, you are right. But I do believe that there is a way(some techniques) to speed up any journey to expertise.

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Craig Ringer
On 18 April 2017 at 01:29, Alvaro Herrera wrote: > Craig Ringer wrote: > >> Personally I have to agree that the learning curve is very steep. Some >> of the docs and presentations help, but there's a LOT to understand. > > There is a wiki page "Developer_FAQ" which is supposed to help answer > the

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Craig Ringer
On 18 April 2017 at 15:41, Kang Yuzhe wrote: > Thanks Simon for taking your time and trying to tell and warn me the harsh > reality truth:there is no shortcut to expertise. One has to fail and rise > towards any journey to expertise. Yeah, just because Pg is hard doesn't mean it's notably bad or

Re: [HACKERS] Tuplesort merge pre-reading

2017-04-18 Thread Heikki Linnakangas
On 04/14/2017 08:19 AM, Peter Geoghegan wrote: BTW, I'm skeptical of the idea of Heikki's around killing polyphase merge itself at this point. I think that keeping most tapes active per pass is useful now that our memory accounting involves handing over an even share to each maybe-active tape for

Re: [HACKERS] Other formats in pset like markdown, rst, mediawiki

2017-04-18 Thread Jan Michálek
2017-04-18 2:27 GMT+02:00 Fabien COELHO : > > Hello Jan, > > Corrected problem with \pset linestyle when format is set to markdown, or >> rst. >> >> Corrected tuples only for markdown and rst (normal and expanded) >> > > It seems that the patch does not apply anymore on head due to changes in > ps

Re: [HACKERS] scram and \password

2017-04-18 Thread Simon Riggs
On 14 March 2017 at 15:40, Tom Lane wrote: >> I was also thinking about that. Basically a primary method and a >> fallback. If that were the case, a gradual transition could happen, and >> if we want \password to enforce best practice it would be ok. > > Why exactly would anyone want "md5 only"?

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Nikhil Sontakke
Hi, There was a bug in the redo 2PC remove code path. Because of which, autovac would think that the 2PC is gone and cause removal of the corresponding clog entry earlier than needed. Please find attached, the bug fix: 2pc_redo_remove_bug.patch. I have been testing this on top of Michael's 2pc-r

Re: [HACKERS] some review comments on logical rep code

2017-04-18 Thread Masahiko Sawada
On Tue, Apr 18, 2017 at 12:24 PM, Kyotaro HORIGUCHI wrote: > Hi, > > Thank you for the revised version. > > At Mon, 17 Apr 2017 23:29:28 +0900, Masahiko Sawada > wrote in >> On Mon, Apr 17, 2017 at 9:13 PM, Masahiko Sawada >> wrote: >> > On Mon, Apr 17, 2017 at 7:39 PM, Kyotaro HORIGUCHI >> >

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-04-18 Thread Pavan Deolasee
On Fri, Apr 14, 2017 at 9:21 PM, Jaime Casanova wrote: > > > Hi Pavan, > > I run a test on current warm patchset, i used pgbench with a scale of > 20 and a fillfactor of 90 and then start the pgbench run with 6 > clients in parallel i also run sqlsmith on it. > > And i got a core dump after somet

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Amit Langote
Hi, On 2017/04/18 16:54, Kang Yuzhe wrote: > Thanks Amit for taking your time and pointing to some useful stuff on the > Internals of PostgreSQL. > > > One thing I have learned is that PG community is not as hostile/harsh as I > imagined to newbies. Rather, its the reverse. > I am feeling at hom

Re: [HACKERS] scram and \password

2017-04-18 Thread Heikki Linnakangas
On 04/18/2017 11:15 AM, Simon Riggs wrote: As a potential open item, if we treat "md5" as ">= md5" should we not also treat "password" as ">=password"? It seems strange that we still support "password" and yet tell everyonenot to use it. I'd like PG10 to be the version where I don't have to tel

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Kang Yuzhe
Thanks Craig for teaching me a lot of things. I am just learning a lot why PG hacking/development is the way it is. Regarding interest and enthusiasm, no problem. Whats is lacking is the skill-sets and I believe having interest and enthusiasm and with your support, we will expand PG hacking/devs/u

Re: [HACKERS] PANIC in pg_commit_ts slru after crashes

2017-04-18 Thread Simon Riggs
On 17 April 2017 at 16:33, Jeff Janes wrote: > On Sun, Apr 16, 2017 at 6:59 PM, Michael Paquier > wrote: >> >> >> >> Jeff, does this patch make the situation better? The fix is rather >> simple as it just makes sure that the next XID never gets updated if >> there are no 2PC files. > > > Yes, tha

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Kang Yuzhe
Hello Amit, Thanks gain for being patient with me. YES, I am working with the PostgreSQL source git repository but I don't think I updated my local forked/cloned branch. I am also working on standalone PG 9.6.2 source code as well. I will try to fetch/pull the PG master content to my forked/cloned

Re: [HACKERS] CREATE TRIGGER document typo

2017-04-18 Thread Heikki Linnakangas
On 04/17/2017 12:09 PM, Yugo Nagata wrote: Hi, Attached is a patch fixing simple typos in the CREATE TRIGGER document. Applied, thanks! - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pg

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Nikhil Sontakke
Please find attached a second version of my bug fix which is stylistically better and clearer than the first one. Regards, Nikhils On 18 April 2017 at 13:47, Nikhil Sontakke wrote: > Hi, > > There was a bug in the redo 2PC remove code path. Because of which, > autovac would think that the 2PC i

Re: [HACKERS] Passing values to a dynamic background worker

2017-04-18 Thread Kyotaro HORIGUCHI
Hello, At Mon, 17 Apr 2017 16:19:13 -0400, Keith Fiske wrote in > So after reading a recent thread on the steep learning curve for PG > internals [1], I figured I'd share where I've gotten stuck with this in a > new thread vs hijacking that one. > > One of the goals I had with pg_partman was t

Re: [HACKERS] Cutting initdb's runtime (Perl question embedded)

2017-04-18 Thread Dagfinn Ilmari Mannsåker
Tom Lane writes: > Andres Freund writes: >> On 2017-04-17 17:49:54 +0100, Dagfinn Ilmari Mannsåker wrote: >>> I threw Devel::NYTProf at it and picked some more low-hanging fruit. > >> I'm a bit doubtful about improving the performance of genbki at the cost >> of any sort of complication - it's o

Re: [HACKERS] Passing values to a dynamic background worker

2017-04-18 Thread Amit Langote
On 2017/04/18 18:12, Kyotaro HORIGUCHI wrote: > At Mon, 17 Apr 2017 16:19:13 -0400, Keith Fiske wrote: >> So after reading a recent thread on the steep learning curve for PG >> internals [1], I figured I'd share where I've gotten stuck with this in a >> new thread vs hijacking that one. >> >> One o

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2017-04-18 Thread Kyotaro HORIGUCHI
At Tue, 18 Apr 2017 14:58:50 +0900, Masahiko Sawada wrote in > On Tue, Apr 18, 2017 at 3:04 AM, Fujii Masao wrote: > > On Wed, Apr 12, 2017 at 2:36 AM, Masahiko Sawada > > wrote: > >> On Thu, Apr 6, 2017 at 4:17 PM, Masahiko Sawada > >> wrote: > >>> On Thu, Apr 6, 2017 at 10:51 AM, Noah Mis

Re: [HACKERS] Comments not allowed on partitioned table columns

2017-04-18 Thread Simon Riggs
On 18 April 2017 at 06:57, Amit Langote wrote: > On 2017/04/18 14:50, Amit Langote wrote: >> Attached patch fixes the oversight that COMMENT ON COLUMN is not allowed >> on partitioned tables columns. > > Forgot to mention that I added this to the open items list. Pushed, thanks. -- Simon Riggs

Re: [HACKERS] Comments not allowed on partitioned table columns

2017-04-18 Thread Amit Langote
On 2017/04/18 18:45, Simon Riggs wrote: > On 18 April 2017 at 06:57, Amit Langote wrote: >> On 2017/04/18 14:50, Amit Langote wrote: >>> Attached patch fixes the oversight that COMMENT ON COLUMN is not allowed >>> on partitioned tables columns. >> >> Forgot to mention that I added this to the open

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2017-04-18 Thread Masahiko Sawada
On Tue, Apr 18, 2017 at 6:40 PM, Kyotaro HORIGUCHI wrote: > At Tue, 18 Apr 2017 14:58:50 +0900, Masahiko Sawada > wrote in >> On Tue, Apr 18, 2017 at 3:04 AM, Fujii Masao wrote: >> > On Wed, Apr 12, 2017 at 2:36 AM, Masahiko Sawada >> > wrote: >> >> On Thu, Apr 6, 2017 at 4:17 PM, Masahiko S

Re: [HACKERS] PANIC in pg_commit_ts slru after crashes

2017-04-18 Thread Simon Riggs
On 18 April 2017 at 09:51, Simon Riggs wrote: > On 17 April 2017 at 16:33, Jeff Janes wrote: >> On Sun, Apr 16, 2017 at 6:59 PM, Michael Paquier >> wrote: >>> >>> >>> >>> Jeff, does this patch make the situation better? The fix is rather >>> simple as it just makes sure that the next XID never g

Re: [HACKERS] Other formats in pset like markdown, rst, mediawiki

2017-04-18 Thread Fabien COELHO
Hello Jan, It seems that the patch does not apply anymore on head due to changes in psql non regression tests. Could you rebase? This should work on current master (all test passed). Patch applies, compiles and make check is ok. There are different flavour of markdown, maybe you should doc

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-04-18 Thread Amit Langote
Hi, On 2017/04/17 23:00, Maksim Milyutin wrote: > On 10.04.2017 14:20, Robert Haas wrote: >> On Tue, Apr 4, 2017 at 12:10 PM, Maksim Milyutin >> wrote: >>> 1. I have added a new relkind for local indexes named RELKIND_LOCAL_INDEX >>> (literal 'l'). >> >> Seems like it should maybe be RELKIND_PART

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Michael Paquier
On Tue, Apr 18, 2017 at 4:15 PM, Andres Freund wrote: > Hi, > > On 2017-04-18 16:07:38 +0900, Michael Paquier wrote: >> Some of you may have noticed that hamster is heavily red on the >> buildfarm. I have done a bit of investigation, and I am able to >> reproduce the failure manually. But actually

Re: [HACKERS] PANIC in pg_commit_ts slru after crashes

2017-04-18 Thread Michael Paquier
On Tue, Apr 18, 2017 at 7:05 PM, Simon Riggs wrote: > I've added a recheck in ProcessTwoPhaseBuffer() after we acquire the lock. > > If its worth acquiring the lock its worth checking we don't have a race. I see. No objections to that. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Simon Riggs
On 18 April 2017 at 09:57, Nikhil Sontakke wrote: > Please find attached a second version of my bug fix which is stylistically > better and clearer than the first one. Yeh, this is better. Pushed. The bug was that the loop set gxact to be the last entry in the array, causing the exit condition

Re: [HACKERS] Partition-wise join for join between (declaratively) partitioned tables

2017-04-18 Thread Ashutosh Bapat
On Wed, Apr 5, 2017 at 8:39 AM, Robert Haas wrote: > On Tue, Apr 4, 2017 at 10:22 AM, Ashutosh Bapat > wrote: >> Yes, I agree. For an inner join, the partition key types need to "shrink" >> and for outer join they need to be "widened". I don't know if there is a way >> to know "wider" or "shorter

[HACKERS] Logical replication and synchronous replication

2017-04-18 Thread Masahiko Sawada
Hi, As doc of logical decoding said as a note[1], logical replication can support the synchronous replication with some restriction. But In addition to this, IIUC in logical replication decoded data is sent to subscribers only when the commit WAL record is decoded (calls ReorderBufferCommit) . It

Re: [HACKERS] Proposal: Local indexes for partitioned table

2017-04-18 Thread Maksim Milyutin
On 18.04.2017 13:08, Amit Langote wrote: Hi, Hi, Amit! On 2017/04/17 23:00, Maksim Milyutin wrote: Ok, thanks for the note. But I want to discuss the relevancy of introduction of a new relkind for partitioned index. I could to change the control flow in partitioned index creation (specify

[HACKERS] Logical replication ApplyContext bloat

2017-04-18 Thread Stas Kelvich
Hi, currently logical replication worker uses ApplyContext to decode received data and that context is never freed during transaction processing. Hence if publication side is performing something like 10M row inserts in single transaction, then subscription worker will occupy more than 10G of ram

Re: [HACKERS] SCRAM authentication, take three

2017-04-18 Thread Heikki Linnakangas
On 04/14/2017 10:33 PM, Peter Eisentraut wrote: On 4/11/17 01:10, Heikki Linnakangas wrote: That question won't arise in practice. Firstly, if the server can do scram-sha-256-plus, it presumably can also do scram-sha-512-plus. Unless there's a change in the way the channel binding works, such th

Re: [HACKERS] SCRAM authentication, take three

2017-04-18 Thread Magnus Hagander
On Tue, Apr 18, 2017 at 1:52 PM, Heikki Linnakangas wrote: > On 04/14/2017 10:33 PM, Peter Eisentraut wrote: > >> On 4/11/17 01:10, Heikki Linnakangas wrote: >> >>> That question won't arise in practice. Firstly, if the server can do >>> scram-sha-256-plus, it presumably can also do scram-sha-512

Re: [HACKERS] Logical replication and synchronous replication

2017-04-18 Thread Craig Ringer
On 18 April 2017 at 18:55, Masahiko Sawada wrote: > Hi, > > As doc of logical decoding said as a note[1], logical replication can > support the synchronous replication with some restriction. But In > addition to this, IIUC in logical replication decoded data is sent to > subscribers only when the

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Michael Paquier
On Tue, Apr 18, 2017 at 7:54 PM, Simon Riggs wrote: > Yeh, this is better. Pushed. I have been outraced on this one, the error is obvious once you see it ;) Thanks for the investigation and the fix! I have spent a couple of hours reviewing the interactions between the shmem entries of 2PC state

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Andrew Dunstan
On 04/18/2017 03:07 AM, Michael Paquier wrote: > Hi all, > > Some of you may have noticed that hamster is heavily red on the > buildfarm. I have done a bit of investigation, and I am able to > reproduce the failure manually. But actually after looking at the logs > the error has obviously showed

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Michael Paquier
On Tue, Apr 18, 2017 at 9:13 PM, Andrew Dunstan wrote: > Yeah, but the way you have done it could also to lead to errors unless > you're very careful, as I found on axolotl (which died recently, > unfortunately). There I had to set the stats_temp directory to a > branch-specific name so a crash on

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Andrew Dunstan
On 04/18/2017 03:41 AM, Kang Yuzhe wrote: > > > But almost nothing about The Internals of PostgreSQL: > 1. The Internals of PostgreSQL: > http://www.interdb.jp/pg/index.html translated from Japanese Book > 2. PostgreSQL数据库内核分析(Chinese) Book on the Internals of PostgreSQL: > 3. PG Docs/site > 4.

Re: [HACKERS] On How To Shorten the Steep Learning Curve Towards PG Hacking...

2017-04-18 Thread Andrew Dunstan
On 04/18/2017 03:54 AM, Craig Ringer wrote: >> But almost nothing about The Internals of PostgreSQL: > Not surprising. They'd go out of date fast, be a huge effort to write > and maintain, and sell poorly given the small audience. > > Print books probably aren't the way forward here. > Agreed,

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Andrew Dunstan
On 04/18/2017 08:23 AM, Michael Paquier wrote: > On Tue, Apr 18, 2017 at 9:13 PM, Andrew Dunstan > wrote: >> Yeah, but the way you have done it could also to lead to errors unless >> you're very careful, as I found on axolotl (which died recently, >> unfortunately). There I had to set the stats_

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Simon Riggs
On 18 April 2017 at 13:12, Michael Paquier wrote: > On Tue, Apr 18, 2017 at 7:54 PM, Simon Riggs wrote: >> Yeh, this is better. Pushed. > > I have been outraced on this one, the error is obvious once you see it ;) Didn't realise you were working on it, nothing competitive about it. It's clear t

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Michael Paquier
On Tue, Apr 18, 2017 at 9:35 PM, Andrew Dunstan wrote: > > > On 04/18/2017 08:23 AM, Michael Paquier wrote: >> On Tue, Apr 18, 2017 at 9:13 PM, Andrew Dunstan >> wrote: >>> Yeah, but the way you have done it could also to lead to errors unless >>> you're very careful, as I found on axolotl (which

Re: [HACKERS] [PATCH] New command to monitor progression of long running queries

2017-04-18 Thread Maksim Milyutin
Hi! On 17.04.2017 15:09, Remi Colinet wrote: Hello, I've implemented a new command named PROGRESS to monitor progression of long running SQL queries in a backend process. Use case === A use case is shown in the below example based on a table named t_10m with 10 millions rows. The table

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Andrew Dunstan
On 04/18/2017 08:59 AM, Michael Paquier wrote: >> Lets's say we have a bunch of possible environment settings with names >> that all begin with "PG_TAP_" PostgresNode.pm could check for the >> existence of these and take action accordingly, and you could set them >> on a buildfarm animal in the c

Re: [HACKERS] Passing values to a dynamic background worker

2017-04-18 Thread Keith Fiske
On Tue, Apr 18, 2017 at 5:40 AM, Amit Langote wrote: > On 2017/04/18 18:12, Kyotaro HORIGUCHI wrote: > > At Mon, 17 Apr 2017 16:19:13 -0400, Keith Fiske wrote: > >> So after reading a recent thread on the steep learning curve for PG > >> internals [1], I figured I'd share where I've gotten stuck

Re: [HACKERS] Interval for launching the table sync worker

2017-04-18 Thread Peter Eisentraut
On 4/13/17 06:23, Masahiko Sawada wrote: > Attached the latest patch. It didn't actually necessary to change > GetSubscriptionNotReadyRelations. I just changed the logic refreshing > the sync table state list. I think this was the right direction, but then I got worried about having a loop within

Re: [HACKERS] Logical replication launcher uses wal_retrieve_retry_interval

2017-04-18 Thread Peter Eisentraut
On 4/16/17 22:40, Masahiko Sawada wrote: > Attached two patches add new GUCs apply_worker_timeout and > apply_worker_launch_interval which are used instead of > wal_receiver_timeout and wal_retrieve_retry_timeout. These new > parameters are not settable at worker-level so far. Under what circumsta

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-18 Thread Amit Kapila
On Mon, Apr 17, 2017 at 10:54 AM, Tom Lane wrote: > Amit Kapila writes: >> On Sun, Apr 16, 2017 at 9:30 PM, Tom Lane wrote: >>> FYI, I have this on my to-look-at list, and expect to fix it before Robert >>> returns from vacation. > >> Let me know if an initial patch by someone else can be helpfu

Re: [HACKERS] [PATCH] New command to monitor progression of long running queries

2017-04-18 Thread Remi Colinet
Hello Maksim, The core implementation I suggested for the new PROGRESS command uses different functions from the one used by EXPLAIN for its core implementation. Some source code is shared with EXPLAIN command. But this shared code is only related to quals, properties, children, subPlans and few o

Re: [HACKERS] scram and \password

2017-04-18 Thread Heikki Linnakangas
On 04/18/2017 08:44 AM, Noah Misch wrote: On Tue, Apr 11, 2017 at 10:07:12PM +0300, Heikki Linnakangas wrote: Thanks! I've been busy on the other thread on future-proofing the protocol with negotiating the SASL mechanism, I'll come back to this once we get that settled. By the end of the week, I

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Tom Lane
Michael Paquier writes: > That's the point I am trying to make upthread: slow buildfarm animals > should have minimal impact on core code modifications. We could for > example have one environment variable that lists all the parameters to > modify in a single string and appends them at the end of

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Alvaro Herrera
Tom Lane wrote: > Michael Paquier writes: > > That's the point I am trying to make upthread: slow buildfarm animals > > should have minimal impact on core code modifications. We could for > > example have one environment variable that lists all the parameters to > > modify in a single string and a

Re: [HACKERS] Should pg_current_wal_location() become pg_current_wal_lsn()

2017-04-18 Thread David Steele
On 4/15/17 12:56 PM, Robert Haas wrote: > On Fri, Apr 14, 2017 at 6:52 PM, Tom Lane wrote: >> Peter Eisentraut writes: >>> If we're talking about making things easier to understand, wouldn't a >>> random user rather know what a WAL "location" is instead of a WAL "LSN"? >> I wouldn't object to sta

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Tom Lane
Alvaro Herrera writes: > Tom Lane wrote: >> FWIW, I'm a bit suspicious of relocating the temp stats directory as >> being a reliable fix for this. > It's an SD card (the kind typically used in cameras and phones), not SSD. > Saying it's slow is an understatement. It's *excruciatingly* slow. Oh,

Re: [HACKERS] identity columns

2017-04-18 Thread Peter Eisentraut
On 4/7/17 01:26, Vitaly Burovoy wrote: > I've implement SET GENERATED ... IF NOT EXISTS. It must be placed > before other SET options but fortunately it conforms with the > standard. > Since that form always changes the sequence behind the column, I > decided to explicitly write "[NO] CACHE" in pg_

Re: [HACKERS] identity columns

2017-04-18 Thread Peter Eisentraut
On 4/18/17 02:07, Noah Misch wrote: > This PostgreSQL 10 open item is past due for your status update. Kindly send > a status update within 24 hours, and include a date for your subsequent status > update. Refer to the policy on open item ownership: > https://www.postgresql.org/message-id/2017040

Re: [HACKERS] Interval for launching the table sync worker

2017-04-18 Thread Petr Jelinek
On 18/04/17 16:22, Peter Eisentraut wrote: > On 4/13/17 06:23, Masahiko Sawada wrote: >> Attached the latest patch. It didn't actually necessary to change >> GetSubscriptionNotReadyRelations. I just changed the logic refreshing >> the sync table state list. > > I think this was the right direction

Re: [HACKERS] Logical replication launcher uses wal_retrieve_retry_interval

2017-04-18 Thread Petr Jelinek
On 18/04/17 16:24, Peter Eisentraut wrote: > On 4/16/17 22:40, Masahiko Sawada wrote: >> Attached two patches add new GUCs apply_worker_timeout and >> apply_worker_launch_interval which are used instead of >> wal_receiver_timeout and wal_retrieve_retry_timeout. These new >> parameters are not setta

[HACKERS] dtrace probes

2017-04-18 Thread Jesper Pedersen
Hi, The lwlock dtrace probes define LWLockMode as int, and the TRACE_POSTGRESQL_LWLOCK methods are called using both a variable and constant definition. This leads to a mix of argument definitions depending on the call site, as seen in probes.txt file. A fix is to explicit cast 'mode' to i

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-18 Thread Petr Jelinek
On 16/04/17 22:27, Petr Jelinek wrote: > On 16/04/17 18:47, Tom Lane wrote: >> Craig Ringer writes: >>> On 12 April 2017 at 13:34, Kuntal Ghosh wrote: For backend_type=background worker, application_name shows the name of the background worker (BackgroundWorker->bgw_name). I think we ne

Re: [HACKERS] Interval for launching the table sync worker

2017-04-18 Thread Masahiko Sawada
On Tue, Apr 18, 2017 at 11:22 PM, Peter Eisentraut wrote: > On 4/13/17 06:23, Masahiko Sawada wrote: >> Attached the latest patch. It didn't actually necessary to change >> GetSubscriptionNotReadyRelations. I just changed the logic refreshing >> the sync table state list. > > I think this was the

Re: [HACKERS] Interval for launching the table sync worker

2017-04-18 Thread Peter Eisentraut
On 4/18/17 11:59, Petr Jelinek wrote: > Hmm if we create hashtable for this, I'd say create hashtable for the > whole table_states then. The reason why it's list now was that it seemed > unnecessary to have hashtable when it will be empty almost always but > there is no need to have both hashtable

[HACKERS] logical replication fixes

2017-04-18 Thread Euler Taveira
Hi, While inspecting the logical replication code, I found a bug that could pick the wrong remote relation if they have the same name but different schemas. Also, I did some spelling/cosmetic changes and fixed an oversight in the ALTER SUBSCRIPTION documentation. Patches attached. -- Euler T

Re: [HACKERS] scram and \password

2017-04-18 Thread Simon Riggs
On 18 April 2017 at 09:25, Heikki Linnakangas wrote: > On 04/18/2017 11:15 AM, Simon Riggs wrote: >> >> As a potential open item, if we treat "md5" as ">= md5" >> should we not also treat "password" as ">=password"? >> >> It seems strange that we still support "password" and yet tell >> everyoneno

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-18 Thread Peter Eisentraut
On 4/18/17 12:13, Petr Jelinek wrote: > We can definitely easily detect that the bgworker is internal one by > library_name equals 'postgres' so we can easily remove the usesysid and > usename based on that. I don't see why we need to do that. It is showing the correct information, isn't it? > B

Re: [HACKERS] Logical replication launcher uses wal_retrieve_retry_interval

2017-04-18 Thread Peter Eisentraut
On 4/18/17 12:00, Petr Jelinek wrote: > As for apply_worker_launch_interval, I think we want different > name so that it can be used for tablesync rate limiting as well. But that's a mechanism we don't have yet, so maybe we should design that when we get there? -- Peter Eisentraut h

Re: [HACKERS] Logical replication and synchronous replication

2017-04-18 Thread Masahiko Sawada
On Tue, Apr 18, 2017 at 9:05 PM, Craig Ringer wrote: > On 18 April 2017 at 18:55, Masahiko Sawada wrote: >> Hi, >> >> As doc of logical decoding said as a note[1], logical replication can >> support the synchronous replication with some restriction. But In >> addition to this, IIUC in logical rep

Re: [HACKERS] Passing values to a dynamic background worker

2017-04-18 Thread Peter Eisentraut
On 4/17/17 16:19, Keith Fiske wrote: > I've reached a roadblock in that bgw_main_arg can only accept a single > argument that must be passed by value for a dynamic bgw. I already > worked around this for passing the database name to the my existing use > of a bgw with doing partition maintenance (p

Re: [HACKERS] some review comments on logical rep code

2017-04-18 Thread Petr Jelinek
Thank you for working on this! On 18/04/17 10:16, Masahiko Sawada wrote: > On Tue, Apr 18, 2017 at 12:24 PM, Kyotaro HORIGUCHI > wrote: >> >> 3. >>> >>> ApplyLauncherWakeup() should be static function. >> >> Attached 003 patch fixes it (and also fixes #5 review comment). >

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-18 Thread Tom Lane
Peter Eisentraut writes: > I think showing bgw_name as backend_type always sounds reasonable. No > need to treat external implementations differently. That's definitely an approach we could use. It would encourage people to use short bgw_names, which is a constraint that wasn't especially appar

Re: [HACKERS] Interval for launching the table sync worker

2017-04-18 Thread Petr Jelinek
On 18/04/17 18:14, Peter Eisentraut wrote: > On 4/18/17 11:59, Petr Jelinek wrote: >> Hmm if we create hashtable for this, I'd say create hashtable for the >> whole table_states then. The reason why it's list now was that it seemed >> unnecessary to have hashtable when it will be empty almost alway

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-18 Thread Petr Jelinek
On 18/04/17 18:24, Peter Eisentraut wrote: > On 4/18/17 12:13, Petr Jelinek wrote: >> We can definitely easily detect that the bgworker is internal one by >> library_name equals 'postgres' so we can easily remove the usesysid and >> usename based on that. > > I don't see why we need to do that. I

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-18 Thread Tom Lane
Petr Jelinek writes: > On 18/04/17 18:24, Peter Eisentraut wrote: >> I don't see why we need to do that. It is showing the correct >> information, isn't it? > It does, but it's also one of the things Tom complained about and I > think he is right in that at least values for launcher should be >

Re: [HACKERS] some review comments on logical rep code

2017-04-18 Thread Fujii Masao
On Wed, Apr 19, 2017 at 1:35 AM, Petr Jelinek wrote: > Thank you for working on this! > > On 18/04/17 10:16, Masahiko Sawada wrote: >> On Tue, Apr 18, 2017 at 12:24 PM, Kyotaro HORIGUCHI >> wrote: >>> >>> 3. ApplyLauncherWakeup() should be static function. >>> >>> At

Re: [HACKERS] Other formats in pset like markdown, rst, mediawiki

2017-04-18 Thread Jan Michálek
2017-04-18 12:06 GMT+02:00 Fabien COELHO : > > Hello Jan, > > It seems that the patch does not apply anymore on head due to changes in >>> psql non regression tests. Could you rebase? >>> >> >> This should work on current master (all test passed). >> > > Patch applies, compiles and make check is o

Re: [HACKERS] Re: Query fails when SRFs are part of FROM clause (Commit id: 69f4b9c85f)

2017-04-18 Thread Tom Lane
Andres Freund writes: > On 2017-04-17 19:26:11 -0400, Tom Lane wrote: >> If we are going to go down this road, I think it would be a good idea >> to try to provide a cursor position for the "can't accept a set" error >> message, because otherwise it will be really unclear what's wrong with >> some

[HACKERS] SASL minor docs typo

2017-04-18 Thread Jaime Casanova
Hi, reading SASL docs found this typo: in protocol.sgml:1356 """ To begin a SASL authentication exchange, the server an AuthenticationSASL message. """ I guess it should say "the server sends an AuthenticationSASL message" -- Jaime Casanova www.2ndQuadrant.com PostgreSQL

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2017-04-18 Thread Fujii Masao
On Tue, Apr 18, 2017 at 7:02 PM, Masahiko Sawada wrote: > On Tue, Apr 18, 2017 at 6:40 PM, Kyotaro HORIGUCHI > wrote: >> At Tue, 18 Apr 2017 14:58:50 +0900, Masahiko Sawada >> wrote in >>> On Tue, Apr 18, 2017 at 3:04 AM, Fujii Masao wrote: >>> > On Wed, Apr 12, 2017 at 2:36 AM, Masahiko Sawa

Re: [HACKERS] [PATCH] New command to monitor progression of long running queries

2017-04-18 Thread Maksim Milyutin
On 18.04.2017 17:39, Remi Colinet wrote: Hello Maksim, The core implementation I suggested for the new PROGRESS command uses different functions from the one used by EXPLAIN for its core implementation. Some source code is shared with EXPLAIN command. But this shared code is only related to qual

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-18 Thread Petr Jelinek
On 18/04/17 19:18, Tom Lane wrote: > Petr Jelinek writes: >> On 18/04/17 18:24, Peter Eisentraut wrote: >>> I don't see why we need to do that. It is showing the correct >>> information, isn't it? > >> It does, but it's also one of the things Tom complained about and I >> think he is right in th

Re: [HACKERS] Quals not pushed down into lateral

2017-04-18 Thread David Fetter
On Thu, Apr 13, 2017 at 01:39:07PM -0700, Andres Freund wrote: > On 2017-04-13 16:34:12 -0400, Robert Haas wrote: > > On Thu, Mar 16, 2017 at 4:45 AM, Andres Freund wrote: > > > During citus development we noticed that restrictions aren't pushed down > > > into lateral subqueries, even if they sem

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-18 Thread Tom Lane
Amit Kapila writes: > I have ended up doing something along the lines suggested by you (or > at least what I have understood from your e-mail). Basically, pass > the safe-param-ids list to parallel safety function and decide based > on that if Param node reference in input expression is safe. I

Re: [HACKERS] Passing values to a dynamic background worker

2017-04-18 Thread Keith Fiske
On Tue, Apr 18, 2017 at 12:34 PM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 4/17/17 16:19, Keith Fiske wrote: > > I've reached a roadblock in that bgw_main_arg can only accept a single > > argument that must be passed by value for a dynamic bgw. I already > > worked around t

Re: [HACKERS] Other formats in pset like markdown, rst, mediawiki

2017-04-18 Thread Fabien COELHO
Hello, There are different flavour of markdown, maybe you should document which one is targetted. Should it be CommonMark? Another variant? Why? This should be pandoc pipe table. It's because it is similar to aligned format. I need add this to documentation (i have it in recent TODO) I stil

Re: [HACKERS] Continuous buildfarm failures on hamster with bin-check

2017-04-18 Thread Michael Paquier
On Wed, Apr 19, 2017 at 12:48 AM, Tom Lane wrote: > Alvaro Herrera writes: >> Tom Lane wrote: >>> FWIW, I'm a bit suspicious of relocating the temp stats directory as >>> being a reliable fix for this. > >> It's an SD card (the kind typically used in cameras and phones), not SSD. >> Saying it's s

[HACKERS] [PATCH] Push limit to sort through a subquery

2017-04-18 Thread Douglas Doole
We've hit a case where pass_down_bound() isn't pushing the row count limit from limit into sort. The issue is that we're getting a subquery scan node between the limit and the sort. The subquery is only doing column projection and has no quals or SRFs so it should be safe to push the limit into the

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Jeff Janes
On Tue, Apr 18, 2017 at 5:38 AM, Simon Riggs wrote: > On 18 April 2017 at 13:12, Michael Paquier > wrote: > > On Tue, Apr 18, 2017 at 7:54 PM, Simon Riggs > wrote: > >> Yeh, this is better. Pushed. > > > > I have been outraced on this one, the error is obvious once you see it ;) > > Didn't real

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Jeff Janes
On Tue, Apr 18, 2017 at 1:17 AM, Nikhil Sontakke wrote: > Hi, > > There was a bug in the redo 2PC remove code path. Because of which, > autovac would think that the 2PC is gone and cause removal of the > corresponding clog entry earlier than needed. > > Please find attached, the bug fix: 2pc_redo

Re: [HACKERS] tablesync patch broke the assumption that logical rep depends on?

2017-04-18 Thread Petr Jelinek
On 17/04/17 20:09, Peter Eisentraut wrote: > On 4/16/17 22:01, Noah Misch wrote: >> This PostgreSQL 10 open item is past due for your status update. Kindly send >> a status update within 24 hours, and include a date for your subsequent >> status >> update. Refer to the policy on open item owners

Re: [HACKERS] some review comments on logical rep code

2017-04-18 Thread Petr Jelinek
On 18/04/17 19:27, Fujii Masao wrote: > On Wed, Apr 19, 2017 at 1:35 AM, Petr Jelinek > wrote: >> Thank you for working on this! >> >> On 18/04/17 10:16, Masahiko Sawada wrote: >>> On Tue, Apr 18, 2017 at 12:24 PM, Kyotaro HORIGUCHI >>> wrote: >> 10. > > SpinLockA

[HACKERS] Use sync commit for logical replication apply in TAP tests

2017-04-18 Thread Petr Jelinek
Hi, The commit 887227a1c changed the defaults for subscriptions to do async commit. But since the tests often wait for disk flush and there is no concurrent activity this has increased the amount of time needed for each test. So the attached patch changes the subscriptions create in tab tests to u

Re: [HACKERS] Failed recovery with new faster 2PC code

2017-04-18 Thread Michael Paquier
On Wed, Apr 19, 2017 at 11:09 AM, Jeff Janes wrote: > Would this bug have been seen in a replica server in the absence of crashes, > or was it only vulnerable during crash recovery rather than streaming > replication? This issue could have been seen on a streaming standby as well, letting around

Re: [HACKERS] Should pg_current_wal_location() become pg_current_wal_lsn()

2017-04-18 Thread David Rowley
On 19 April 2017 at 03:33, David Steele wrote: > +1, and I'm in favor of using "lsn" wherever applicable. It seems to me > that if a user calls a function (or queries a table) that returns an lsn > then they should be aware of what an lsn is. OK, so I've read over this thread again and I think i

Re: [HACKERS] Inadequate parallel-safety check for SubPlans

2017-04-18 Thread Amit Kapila
On Wed, Apr 19, 2017 at 1:27 AM, Tom Lane wrote: > Amit Kapila writes: >> I have ended up doing something along the lines suggested by you (or >> at least what I have understood from your e-mail). Basically, pass >> the safe-param-ids list to parallel safety function and decide based >> on that

Re: [HACKERS] Should pg_current_wal_location() become pg_current_wal_lsn()

2017-04-18 Thread Tom Lane
David Rowley writes: > OK, so I've read over this thread again and I think it's time to > summarise the votes: > ... > In favour of "location" -> "lsn": Stephen, David Steel, > In favour of "lsn" -> "location": Peter, Tom, Kyotaro FWIW, I was not voting in favor of "location"; I was just saying t

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2017-04-18 Thread Noah Misch
On Sun, Apr 16, 2017 at 07:25:28PM +0900, Fujii Masao wrote: > On Sun, Apr 16, 2017 at 1:19 PM, Noah Misch wrote: > > On Fri, Apr 14, 2017 at 11:58:23PM -0400, Noah Misch wrote: > >> On Wed, Apr 05, 2017 at 09:51:02PM -0400, Noah Misch wrote: > >> > On Thu, Apr 06, 2017 at 12:48:56AM +0900, Fujii

  1   2   >