Re: Function to promote standby servers

2018-10-22 Thread Michael Paquier
On Sat, Oct 20, 2018 at 01:48:56PM +0900, Michael Paquier wrote: > On Sat, Oct 20, 2018 at 06:24:28AM +0200, Laurenz Albe wrote: >> Here is another version, with a fix in pg_proc.dat, an improved comment >> and "wait_seconds" exercised in the regression test. > > Thanks for the new version. This

Re: Side effect of CVE-2017-7484 fix?

2018-10-22 Thread Amit Langote
Hi, On 2018/10/22 14:41, Stephen Frost wrote: > Greetings, > > * Dilip Kumar (dilipbal...@gmail.com) wrote: >> As part of the security fix >> (e2d4ef8de869c57e3bf270a30c12d48c2ce4e00c), we have restricted the >> users from accessing the statistics of the table if the user doesn't >> have

Re: Function to promote standby servers

2018-10-22 Thread Masahiko Sawada
On Mon, Oct 22, 2018 at 3:01 PM Michael Paquier wrote: > > On Sat, Oct 20, 2018 at 01:48:56PM +0900, Michael Paquier wrote: > > On Sat, Oct 20, 2018 at 06:24:28AM +0200, Laurenz Albe wrote: > >> Here is another version, with a fix in pg_proc.dat, an improved comment > >> and "wait_seconds"

Re: Pluggable Storage - Andres's take

2018-10-22 Thread Haribabu Kommi
On Thu, Oct 18, 2018 at 1:04 PM Haribabu Kommi wrote: > On Tue, Oct 9, 2018 at 1:46 PM Haribabu Kommi > wrote: > >> >> I also observed the failure of aggregates.sql, will look into it. >> > > The random failure of aggregates.sql is as follows > > SELECT avg(a) AS avg_32 FROM aggtest WHERE a <

Re: Buildfarm failures for hash indexes: buffer leaks

2018-10-22 Thread Fabien COELHO
Hello Michaël, The first failure is unrelated to the involved commits, as they touched completely different areas of the code: INSERT INTO hash_split_heap SELECT a/2 FROM generate_series(1, 25000) a; + WARNING: buffer refcount leak: [6481] (rel=base/16384/32349, blockNum=156,

Re: WIP: Avoid creation of the free space map for small tables

2018-10-22 Thread John Naylor
On 10/16/18, Amit Kapila wrote: > I think we can avoid using prevBlockNumber and try_every_block, if we > maintain a small cache which tells whether the particular block is > tried or not. What I am envisioning is that while finding the block > with free space, if we came to know that the

Buildfarm failures for hash indexes: buffer leaks

2018-10-22 Thread Michael Paquier
Hi all, Looking at the buildfarm, serinus and moonjelly have just complained about the failure of $subject: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=serinus=2018-10-22%2006%3A34%3A02 https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=moonjelly=2018-10-20%2015%3A17%3A02 The

Re: View to get all the extension control file details

2018-10-22 Thread Kyotaro HORIGUCHI
Hello. At Wed, 17 Oct 2018 18:38:05 +1100, Haribabu Kommi wrote in > On Sat, Oct 13, 2018 at 3:57 AM Robert Haas wrote: > > > On Wed, Oct 10, 2018 at 8:27 AM Haribabu Kommi > > wrote: > > > Here is the patch as per the above discussion. > > > > One potential problem with this is that we

Re: WAL archive (archive_mode = always) ?

2018-10-22 Thread Adelino Silva
Hello Takayuki, Sorry can you explain how we can same network bandwidth by not sending the WAL archive from the primary to the standby(s). I possible scenario is have to multiple standby servers in same host for same master. or other scenarios exists ? Many thanks in advance, Adelino. On Mon,

Changes to error handling for background worker initialization?

2018-10-22 Thread Jeremy Finzel
Hello - I have an extension that uses background workers. I pass a database oid as an argument in order to launch the worker using function BackgroundWorkerInitializeConnectionByOid. In one of my regression tests that was written, I intentionally launch the worker with an invalid oid. In

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-10-22 Thread Alvaro Herrera
On 2018-Oct-14, Andres Freund wrote: > On 2018-10-14 13:26:24 +, Michael Paquier wrote: > > Avoid duplicate XIDs at recovery when building initial snapshot > I'm unhappy this approach was taken over objections. Without a real > warning. Even leaving the crummyness aside, did you check

Re: Side effect of CVE-2017-7484 fix?

2018-10-22 Thread David Fetter
On Mon, Oct 22, 2018 at 04:43:52PM +0530, Dilip Kumar wrote: > On Mon, Oct 22, 2018 at 11:22 AM David Fetter wrote: > > > > On Mon, Oct 22, 2018 at 11:10:09AM +0530, Dilip Kumar wrote: > > > As part of the security fix > > > (e2d4ef8de869c57e3bf270a30c12d48c2ce4e00c), we have restricted the > > >

Re: [Patch] pg_rewind: options to use restore_command from recovery.conf or command line

2018-10-22 Thread Alexey Kondratov
Hi Andrey, Thank you for your reply. I think it is better to load restore_command from recovery.conf. Yes, it seems to be the most native way. That's why I needed this rewritten (mostly copy-pasted) frontend-safe version of parser (guc-file.l). I didn't actually try patch yet, but the idea

Re: Function to promote standby servers

2018-10-22 Thread Laurenz Albe
Masahiko Sawada wrote: > On Mon, Oct 22, 2018 at 3:01 PM Michael Paquier wrote: > > Regarding the documentation, wouldn't it be more adapted to list the new > > function under the section "Recovery Control Functions"? Not only does > > the new function signal the postmaster, but it also creates

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-10-22 Thread Masahiko Sawada
On Thu, Sep 13, 2018 at 6:30 PM Kyotaro HORIGUCHI wrote: > > Hello. > > Thank you for the comments, Sawada-san, Peter. > > At Mon, 10 Sep 2018 19:52:24 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20180910.195224.22629595.horiguchi.kyot...@lab.ntt.co.jp> > > At Thu, 6 Sep 2018

Re: Side effect of CVE-2017-7484 fix?

2018-10-22 Thread Tom Lane
Dilip Kumar writes: > As part of the security fix > (e2d4ef8de869c57e3bf270a30c12d48c2ce4e00c), we have restricted the > users from accessing the statistics of the table if the user doesn't > have privileges on the table and the function is not leakproof. Now, > as a side effect of this, if the

Re: relhassubclass and partitioned indexes

2018-10-22 Thread Alvaro Herrera
On 2018-Oct-22, Amit Langote wrote: > Hi, > > On 2018/10/22 11:09, Michael Paquier wrote: > > On Fri, Oct 19, 2018 at 06:46:15PM +0900, Amit Langote wrote: > >> Thanks. Attached a patch to set relhassubclass when an index partition is > >> added to a partitioned index. > > > > Thanks,

Re: CVE-2017-7484-induced bugs, or, btree cmp functions are not leakproof?

2018-10-22 Thread Tom Lane
Amit Langote writes: > On Sat, Jul 14, 2018 at 11:29 PM Tom Lane wrote: >> Peter Eisentraut writes: >>> ... For >>> partitioning, we can rely on all the columns being inherited, but not >>> for plain inheritance. >> Uh, what? > But maybe for the case under question, that's irrelevant, because

Re: [HACKERS] Fix number skipping in to_number

2018-10-22 Thread Ioseph Kim
Hi. I'm checking release note for version 11. in that. |"L| and |TH| now only consume characters that are not digits, positive/negative signs, decimal points, or commas." postgres@postgres=# select to_number('1234', '+');  to_number ---    234 Is this right? Regards,

Re: [HACKERS] Transactions involving multiple postgres foreign servers, take 2

2018-10-22 Thread Kyotaro HORIGUCHI
Hello. # It took a long time to come here.. At Fri, 19 Oct 2018 21:38:35 +0900, Masahiko Sawada wrote in > On Wed, Oct 10, 2018 at 1:34 PM Masahiko Sawada wrote: ... > * Updated docs, added the new section "Distributed Transaction" at > Chapter 33 to explain the concept to users > > * Moved

Re: Speeding up text_position_next with multibyte encodings

2018-10-22 Thread Kyotaro HORIGUCHI
Hello. At Fri, 19 Oct 2018 15:52:59 +0300, Heikki Linnakangas wrote in <3173d989-bc1c-fc8a-3b69-f24246f73...@iki.fi> > Attached is a patch to speed up text_position_setup/next(), in some > common cases with multibyte encodings. > > text_position_next() uses the Boyer-Moore-Horspool search

Re: [HACKERS] PATCH: Keep one postmaster monitoring pipe per process

2018-10-22 Thread Thomas Munro
On Thu, Sep 6, 2018 at 9:57 PM Kyotaro HORIGUCHI wrote: > In sysloger.c, cur_flags is (just set but) no longer used. Right. Fixed. > === > In latch.c, > > - The parentheses around the symbols don't seem to be needed. > | (wakeEvents & (WL_EXIT_ON_PM_DEATH)) != 0 || > |

Timeout parameters

2018-10-22 Thread Nagaura, Ryohei
Hi, all. I'd like to suggest introducing two parameters to handle client-server communication timeouts. That is "tcp_user_timeout" and "socket_timeout" parameter. I implemented "tcp_user_timeout" parameter in both backend and frontend side. This parameter enables us to use TCP_USER_TIMEOUT

Re: Synchronous replay take III

2018-10-22 Thread Thomas Munro
On Mon, Sep 24, 2018 at 10:39 AM Thomas Munro wrote: > On Mon, Jul 2, 2018 at 12:39 PM Thomas Munro > wrote: > > On Sat, Mar 3, 2018 at 2:11 AM, Adam Brusselback > > wrote: > > > Thanks Thomas, appreciate the rebase and the work you've done on this. > > > I should have some time to test this

Re: [PATCH] XLogReadRecord returns pointer to currently read page

2018-10-22 Thread Andrey Lepikhov
On 23.10.2018 0:53, Heikki Linnakangas wrote: I'd expect the decompression to read from the on-disk buffer, and unpack to readRecordBuf, I still don't see a need to copy the packed record to readRecordBuf. If there is a need for that, though, the patch that implements the packing or

RE: WAL archive (archive_mode = always) ?

2018-10-22 Thread Tsunakawa, Takayuki
From: Narayanan V [mailto:vnarayanan.em...@gmail.com] > I think what Takayuki is trying to say is that streaming replication works > by sending the contents of the WAL archives to the standbys. If archive_mode > was NOT set to always, and if you wanted to archive WAL logs in the standby > you

Re: Restore CurrentUserId only if 'prevUser' is valid when abort transaction

2018-10-22 Thread Michael Paquier
On Fri, Oct 12, 2018 at 02:28:36PM +0800, Richard Guo wrote: > I think it is a better idea to avoid adjusting the state to TRANS_INPROGRESS > from TRANS_START when aborting a transaction, as your patch does, since its > only purpose is to suppress warning message. Actually, as StartSubTransaction

Re: Side effect of CVE-2017-7484 fix?

2018-10-22 Thread Dilip Kumar
On Mon, Oct 22, 2018 at 11:22 AM David Fetter wrote: > > On Mon, Oct 22, 2018 at 11:10:09AM +0530, Dilip Kumar wrote: > > As part of the security fix > > (e2d4ef8de869c57e3bf270a30c12d48c2ce4e00c), we have restricted the > > users from accessing the statistics of the table if the user doesn't > >

Re: Side effect of CVE-2017-7484 fix?

2018-10-22 Thread Dilip Kumar
On Mon, Oct 22, 2018 at 12:05 PM Amit Langote wrote: > > Hi, > > On 2018/10/22 14:41, Stephen Frost wrote: > > Greetings, > > > > * Dilip Kumar (dilipbal...@gmail.com) wrote: > >> As part of the security fix > >> (e2d4ef8de869c57e3bf270a30c12d48c2ce4e00c), we have restricted the > >> users from

Re: WAL archive (archive_mode = always) ?

2018-10-22 Thread Narayanan V
Hi, I think what Takayuki is trying to say is that streaming replication works by sending the contents of the WAL archives to the standbys. If archive_mode was NOT set to always, and if you wanted to archive WAL logs in the standby you would need to rely on the process_command and make it ship

Re: CVE-2017-7484-induced bugs, or, btree cmp functions are not leakproof?

2018-10-22 Thread Amit Langote
On Sat, Jul 14, 2018 at 11:29 PM Tom Lane wrote: > > Peter Eisentraut writes: > > ... For > > partitioning, we can rely on all the columns being inherited, but not > > for plain inheritance. > > Uh, what? Maybe he meant that partitioning doesn't allow locally defined columns in children, but

Re: Contribution to postgresql

2018-10-22 Thread Laurenz Albe
Nicky Larson wrote: > I am using postgresql at work, and I would like to contribute. > > From the todo list, I chose: > > Allow log_min_messages to be specified on a per-module basis > > Is this feature always wanted ? That would be my first contribution to > postgresql, is it an easy one ? I

Re: [Patch] pg_rewind: options to use restore_command from recovery.conf or command line

2018-10-22 Thread Andrey Borodin
Hi, Alexey! > 19 окт. 2018 г., в 22:49, Alexey Kondratov > написал(а): > I expect, that it will be a good idea to allow pg_rewind to look for a > restore_command > +1 Normally you do not expect huge progress on failed master. But you still can get a lot of WAL if you have network partition

Multiple Wait Events for extensions

2018-10-22 Thread legrand legrand
Hello, I'm playing with adding into my pg_stat_statements extension a wait event for pgss time duration (pgss_store) Adding pgstat_report_wait_start(PG_WAIT_EXTENSION) gives wait type = "Extension" / event name "Extension" and that's perfect. Now I would like to add a second wait event (for

Re: [PATCH] XLogReadRecord returns pointer to currently read page

2018-10-22 Thread Heikki Linnakangas
On 22/10/2018 20:54, Andrey Lepikhov wrote: On 22.10.2018 2:06, Heikki Linnakangas wrote: On 17/08/2018 06:47, Andrey Lepikhov wrote: I propose the patch for fix one small code defect. The XLogReadRecord() function reads the pages of a WAL segment that contain a WAL-record. Then it creates a

Re: [PATCH] XLogReadRecord returns pointer to currently read page

2018-10-22 Thread Andrey Lepikhov
On 22.10.2018 2:06, Heikki Linnakangas wrote: On 17/08/2018 06:47, Andrey Lepikhov wrote: I propose the patch for fix one small code defect. The XLogReadRecord() function reads the pages of a WAL segment that contain a WAL-record. Then it creates a readRecordBuf buffer in private memory of a

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-10-22 Thread David Rowley
On 15 October 2018 at 23:04, Krzysztof Nienartowicz wrote: > We see quite prohibitive 5-6x slowdown with native partitioning on in > comparison to trigger based in PG9.5. > This is clearly visible with highly parallel inserts (Can share > flamediagrams comparing the two). Does the 0001 patch

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-10-22 Thread Krzysztof Nienartowicz
In the end we hacked the code to re-enable triggers on partitioned tables and switch off native insert code on partitioned tables. Quite hackish and would be nice to have it fixed in a more natural manner. Yes, it looked like locking but not only - ExecSetupPartitionTupleRouting:

Re: Small performance tweak to run-time partition pruning

2018-10-22 Thread David Rowley
On 18 October 2018 at 16:13, Imai, Yoshikazu wrote: > The patch improves the performance about 1.3% which is less than David's > result, but it seems still improves the performance. Thanks for doing these benchmarks. The speedup is small, but it becomes much more significant once other

Re: Multiple Wait Events for extensions

2018-10-22 Thread Michael Paquier
On Mon, Oct 22, 2018 at 10:28:14AM -0700, legrand legrand wrote: > Could this be changed to offer an extension the ability to log multiple > events and many extensions to work together? I recall that this issue has been discussed when adding new wait events, and we discarded it for simplicity as

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-10-22 Thread Alvaro Herrera
On 2018-Oct-22, Andres Freund wrote: > Hi, > > On 2018-10-22 12:36:25 -0300, Alvaro Herrera wrote: > > On 2018-Oct-14, Andres Freund wrote: > > > > > On 2018-10-14 13:26:24 +, Michael Paquier wrote: > > > > Avoid duplicate XIDs at recovery when building initial snapshot > > > > > I'm

Re: removing unnecessary get_att*() lsyscache functions

2018-10-22 Thread Peter Eisentraut
On 19/10/2018 16:00, Michael Paquier wrote: >> - if (get_attidentity(RelationGetRelid(rel), attnum)) >> + if (TupleDescAttr(RelationGetDescr(rel), attnum - 1)->attidentity) > I find this style heavy, saving Form_pg_attribute into a different > variable would be more readable in my

Re: [Patch] pg_rewind: options to use restore_command from recovery.conf or command line

2018-10-22 Thread Alvaro Herrera
On 2018-Oct-22, Alexey Kondratov wrote: > > I didn't actually try patch yet, but the idea seems interesting. Will > > you add it to the commitfest? > I am willing to add it to the November commitfest, but I have some concerns > regarding frontend version of GUC parser. Probably, it is possible to

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-10-22 Thread Andres Freund
Hi, On 2018-10-22 12:03:26 +0900, Michael Paquier wrote: > (moving to -hackers) > > On Sun, Oct 14, 2018 at 10:42:40AM -0700, Andres Freund wrote: > > I'm unhappy this approach was taken over objections. Without a real > > warning. > > Oops, that was not clear to me. Sorry about that! I did

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-10-22 Thread Andres Freund
Hi, On 2018-10-22 12:36:25 -0300, Alvaro Herrera wrote: > On 2018-Oct-14, Andres Freund wrote: > > > On 2018-10-14 13:26:24 +, Michael Paquier wrote: > > > Avoid duplicate XIDs at recovery when building initial snapshot > > > I'm unhappy this approach was taken over objections. Without a

Re: Function to promote standby servers

2018-10-22 Thread Michael Paquier
On Mon, Oct 22, 2018 at 11:45:30AM +0200, Laurenz Albe wrote: > Masahiko Sawada wrote: >> Thank you for workig on this. There is one review comment for the latest >> patch. >> >> + if (FreeFile(promote_file)) >> + ereport(ERROR, >> +

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-10-22 Thread Michael Paquier
On Mon, Oct 22, 2018 at 07:15:38PM -0300, Alvaro Herrera wrote: > On 2018-Oct-22, Andres Freund wrote: >> Hm? My point is that this fix just puts a band-aid onto *one* of the >> places that read a XLOG_RUNNING_XACTS. Which still leaves the contents >> of WAL record corrupted. There's not even a

Re: removing unnecessary get_att*() lsyscache functions

2018-10-22 Thread Michael Paquier
On Mon, Oct 22, 2018 at 07:12:28PM +0200, Peter Eisentraut wrote: > OK, slightly reworked version attached. + attTup = (Form_pg_attribute) GETSTRUCT(tuple); attnum = ((Form_pg_attribute) GETSTRUCT(tuple))->attnum; No need to call twice GETSTRUCT here.. The rest looks fine. -- Michael

Re: relhassubclass and partitioned indexes

2018-10-22 Thread Amit Langote
On 2018/10/23 0:45, Alvaro Herrera wrote: > On 2018-Oct-22, Amit Langote wrote: > >> Hi, >> >> On 2018/10/22 11:09, Michael Paquier wrote: >>> On Fri, Oct 19, 2018 at 06:46:15PM +0900, Amit Langote wrote: Thanks. Attached a patch to set relhassubclass when an index partition is added

Re: removing unnecessary get_att*() lsyscache functions

2018-10-22 Thread Andres Freund
Hi, On 2018-10-23 09:11:17 +0900, Michael Paquier wrote: > On Mon, Oct 22, 2018 at 07:12:28PM +0200, Peter Eisentraut wrote: > > OK, slightly reworked version attached. > > + attTup = (Form_pg_attribute) GETSTRUCT(tuple); > attnum = ((Form_pg_attribute) GETSTRUCT(tuple))->attnum; > > No