Re: Copy function for logical replication slots

2018-08-13 Thread Masahiko Sawada
On Thu, Jul 12, 2018 at 9:28 PM, Masahiko Sawada wrote: > On Mon, Jul 9, 2018 at 10:34 AM, Michael Paquier wrote: >> On Mon, Jul 09, 2018 at 10:06:00AM +0900, Masahiko Sawada wrote: >>> I think that this patch might be splitted but I will be able to send >>> an updated patch in the next week. As

Re: Upper limit arguments of pg_logical_slot_xxx_changes functions accept invalid values

2018-08-13 Thread Masahiko Sawada
On Sat, Jul 28, 2018 at 2:00 AM, Robert Haas wrote: > On Wed, Jul 11, 2018 at 8:58 PM, Masahiko Sawada > wrote: >> While reading the replication slot codes, I found a wrong assignment >> in pg_logical_slot_get_changes_guts() function as follows. >> >> if (PG_ARGISNULL(2)) >>

Re: NetBSD vs libxml2

2018-08-13 Thread Nico Williams
On Mon, Aug 13, 2018 at 11:16:23AM -0400, Tom Lane wrote: > Well, the issue is that new kinds of switches introduce new potential > for bugs. In the case of -Wl,-R..., I'm not even sure that you can > write that more than once per link, so absorbing one from xml2-config > might well break things

Re: Undocumented(?) limits on regexp functions

2018-08-13 Thread Tom Lane
Andrew Gierth writes: > All the regexp functions blow up with "invalid memory alloc request" > errors when the input string exceeds 256MB in length. This restriction > does not seem to be documented anywhere that I could see. > (Also for regexp_split* and regexp_matches, there's a limit of 64M

Undocumented(?) limits on regexp functions

2018-08-13 Thread Andrew Gierth
All the regexp functions blow up with "invalid memory alloc request" errors when the input string exceeds 256MB in length. This restriction does not seem to be documented anywhere that I could see. (Also for regexp_split* and regexp_matches, there's a limit of 64M total matches, which also

Pre-v11 appearances of the word "procedure" in v11 docs

2018-08-13 Thread Peter Geoghegan
I noticed that the word "procedure" appears in at least a few places in the v11 docs as a not-quite-apt synonym of "function". For example, https://www.postgresql.org/docs/11/static/plpgsql-trigger.html talks about "PL/pgSQL Trigger Procedures" which are actually all functions in practice. I

Re: [HACKERS] pgbench - allow to store select results into variables

2018-08-13 Thread Fabien COELHO
Hello Andrew, Attached is v18, another basic rebase after some perl automatic reindentation. This patch contains CRLF line endings Alas, not according to "file" nor "hexdump" (only 0A, no 0D) on my local version, AFAICS. What happens on the path and what is done by mail clients

Re: libpq should append auth failures, not overwrite

2018-08-13 Thread Tom Lane
Fabien COELHO writes: >> As I explained in my comments, the reason I did not do these things >> is that I didn't want to change the output for cases in which just a >> single host name is given. I still don't. > Ok, I get your argument when there is just one target server (cluster), > which is

Re: [HACKERS] pgbench - allow to store select results into variables

2018-08-13 Thread Andrew Dunstan
On 04/27/2018 12:28 PM, Fabien COELHO wrote: Hello Stephen, Attached is v18, another basic rebase after some perl automatic reindentation. This patch contains CRLF line endings - and in any case it doesn't apply any more. Please fix those things. cheers andrew -- Andrew Dunstan

Re: FailedAssertion on partprune

2018-08-13 Thread Robert Haas
On Sat, Aug 11, 2018 at 9:16 AM, David Rowley wrote: > I started debugging this to see where things go wrong. I discovered > that add_paths_to_append_rel() is called yet again from > apply_scanjoin_target_to_paths() and this is where it's all going > wrong. The problem is that the gather paths

Re: Some pgq table rewrite incompatibility with logical decoding?

2018-08-13 Thread Jeremy Finzel
> > I don't think that's true, for two reasons. > > Firstly, I don't think pgq updates catalogs directly, it simply truncates > the queue tables when rotating them (which updates the relfilenode in > pg_class, of course). > > Secondly, we're occasionally seeing this on systems that do not use pgq,

Re: libpq compression

2018-08-13 Thread Andrew Dunstan
On 08/13/2018 02:47 PM, Robert Haas wrote: On Fri, Aug 10, 2018 at 5:55 PM, Andrew Dunstan wrote: This thread appears to have gone quiet. What concerns me is that there appears to be substantial disagreement between the author and the reviewers. Since the last thing was this new patch it

Re: libpq should not look up all host addresses at once

2018-08-13 Thread Tom Lane
Fabien COELHO writes: > Patch compiles, global "make check" ok, although I'm unsure whether the > feature is actually tested somewhere. I think not:-( Yeah, it's hard to test this stuff without either opening up security hazards or making unwarranted assumptions about the local network setup. I

Re: FailedAssertion on partprune

2018-08-13 Thread Robert Haas
On Wed, Aug 8, 2018 at 11:33 PM, Tom Lane wrote: > Oh ... never mind that last. The parent Append will run its children > sequentially, so that the Gathers execute one at a time, and at no > point will more than 2 workers be active. Yep. > Nonetheless, it's a damfool query plan, because we'll

Re: libpq compression

2018-08-13 Thread Robert Haas
On Fri, Aug 10, 2018 at 5:55 PM, Andrew Dunstan wrote: > This thread appears to have gone quiet. What concerns me is that there > appears to be substantial disagreement between the author and the reviewers. > Since the last thing was this new patch it should really have been put back > into

Re: Improve behavior of concurrent TRUNCATE

2018-08-13 Thread Robert Haas
On Fri, Aug 10, 2018 at 5:05 PM, Alvaro Herrera wrote: > I was actually thinking in applying to all back-branches, not just pg11, > considering it a fix for a pretty serious bug. But checking the > history, it seems that Robert patched this is 9.2 as new development > (2ad36c4e4, 1489e2f26,

Re: libpq connection timeout mismanagement

2018-08-13 Thread Robert Haas
On Thu, Aug 9, 2018 at 11:23 AM, Tom Lane wrote: > The patch that taught libpq about allowing multiple target hosts > modified connectDBComplete() with the intent of making the > connect_timeout (if specified) apply per-host, not to the complete > connection attempt. It did not do a very good

Re: libpq connection timeout mismanagement

2018-08-13 Thread Tom Lane
Fabien COELHO writes: > Patch does not "git apply", but is ok with "patch -p1". Compiles. Yeah, as far as I can tell, "git apply" is very intolerant. > The code suggests that timeout is always 2 or more > if (timeout < 2) timeout = 2; > but doc says "It is not recommended to use a timeout

Re: Temporary tables prevent autovacuum, leading to XID wraparound

2018-08-13 Thread Alvaro Herrera
If writing an OID were not atomic, the assignment would be really dangerous. I don't think your proposed update is good. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Temporary tables prevent autovacuum, leading to XID wraparound

2018-08-13 Thread Michael Paquier
On Mon, Aug 13, 2018 at 02:56:16AM -0700, Andres Freund wrote: > On 2018-08-09 18:50:47 +0200, Michael Paquier wrote: > I don't think that comment, nor the comment that you ended up > committing: > + > + /* > +* Reset the temporary namespace flag in MyProc. We assume that >

Re: Temporary tables prevent autovacuum, leading to XID wraparound

2018-08-13 Thread Alvaro Herrera
>From 1e9ba9fa9b172bda1ea54b1f3be1b930973ff45f Mon Sep 17 00:00:00 2001 From: Michael Paquier Date: Wed, 8 Aug 2018 19:45:31 +0200 Subject: [PATCH] Make autovacuum more aggressive to remove orphaned temp tables I was here to complain about this piece: > @@ -3975,6 +4033,15 @@

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-08-13 Thread Robert Haas
On Sun, Aug 12, 2018 at 9:05 AM, David Rowley wrote: > This is not a fully baked idea, but I'm wondering if a better way to > do this, instead of having this PartitionIsValid macro to determine if > the partition should be visible to the current transaction ID, we > could, when we invalidate a

Re: WIP: "More fair" LWLocks

2018-08-13 Thread Dmitry Dolgov
> On Mon, 13 Aug 2018 at 17:36, Alexander Korotkov > wrote: > > Hi! > > I run pgbench (read-write and read-only benchmarks) on Amazon > c5d.18xlarge virtual machine, which has 72 VCPU (approximately same > power as 36 physical cores). The results are attached > (lwlock-fair-ro.png and

Re: NetBSD vs libxml2

2018-08-13 Thread Nico Williams
[Quoting out of order.] On Mon, Aug 13, 2018 at 11:16:23AM -0400, Tom Lane wrote: > Robert Haas writes: > > I'm not, personally, eager to do that work for a requirement which > somehow hasn't surfaced on any other platform, nor on any previous > NetBSD release. I think NetBSD is way out in left

Re: libpq should not look up all host addresses at once

2018-08-13 Thread Fabien COELHO
The Fatal error does not really say for which host/ip the password fail. Yup, but that's not the province of this patch to improve. See https://www.postgresql.org/message-id/25918.1533918...@sss.pgh.pa.us for one that is trying to improve it. Yep, I gathered that afterwards. -- Fabien.

WIP: "More fair" LWLocks

2018-08-13 Thread Alexander Korotkov
Hi! This subject was already raised multiple times [1], [2], [3]. In short, our LWLock implementation has pathological behavior when shared lockers constitute a continuous flood. In this case exclusive lock waiters are not guaranteed to eventually get the lock. When shared lock is held, other

Re: libpq should append auth failures, not overwrite

2018-08-13 Thread Fabien COELHO
Hello Tom, ISTM that both the hostname and ip should be shown to avoid confusion about hosts with multiple ips, esp. as ips are given in any order by the dns. ... Also for homogeneity, I'd suggest to always add the server line. If the server introduction is inserted in all cases, including

Re: NetBSD vs libxml2

2018-08-13 Thread Tom Lane
Robert Haas writes: > On Sat, Aug 11, 2018 at 1:18 PM, Tom Lane wrote: >> We could fix this by teaching configure to absorb -Wl,-R... switches >> into LDFLAGS from xml2-config's output, and that seems to make things >> work, but I wonder whether we should or not. This seems like a new height >>

Re: libpq should append auth failures, not overwrite

2018-08-13 Thread Tom Lane
Fabien COELHO writes: > ISTM that both the hostname and ip should be shown to avoid confusion > about hosts with multiple ips, esp. as ips are given in any order by the > dns. > ... > Also for homogeneity, I'd suggest to always add the server line. If the > server introduction is inserted in

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-13 Thread Andres Freund
On 2018-08-13 11:50:41 -0300, Alvaro Herrera wrote: > On 2018-Aug-11, Andres Freund wrote: > > > On 2018-08-11 00:46:25 -0700, Andres Freund wrote: > > > Below you can find the bt full showing a bunch of nested invalidations. > > > Looking. > > > > Hm. I wonder if the attached fixes the issue

Re: Repeatable Read Isolation in SQL running via background worker

2018-08-13 Thread Jeremy Finzel
On Thu, Aug 9, 2018 at 4:34 PM, Jeremy Finzel wrote: > I am using worker_spi as a model to run a SQL statement inside a > background worker. From my browsing of the Postgres library, I believe > that if I want repeatable read isolation level, the proper way for me to > attain this is to add

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-13 Thread Alvaro Herrera
On 2018-Aug-11, Andres Freund wrote: > On 2018-08-11 00:46:25 -0700, Andres Freund wrote: > > Below you can find the bt full showing a bunch of nested invalidations. > > Looking. > > Hm. I wonder if the attached fixes the issue for you. If it's this I > don't understand why this doesn't occur

Re: NetBSD vs libxml2

2018-08-13 Thread Robert Haas
On Sat, Aug 11, 2018 at 1:18 PM, Tom Lane wrote: > We could fix this by teaching configure to absorb -Wl,-R... switches > into LDFLAGS from xml2-config's output, and that seems to make things > work, but I wonder whether we should or not. This seems like a new height > of unfriendliness to

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-13 Thread Andres Freund
Hi, On 2018-08-13 11:46:30 -0300, Alvaro Herrera wrote: > On 2018-Aug-11, Tomas Vondra wrote: > > > Hmmm, it's difficult to compare "bt full" output, but my backtraces look > > somewhat different (and all the backtraces I'm seeing are 100% exactly > > the same). Attached for comparison. > >

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-13 Thread Alvaro Herrera
On 2018-Aug-11, Tomas Vondra wrote: > Hmmm, it's difficult to compare "bt full" output, but my backtraces look > somewhat different (and all the backtraces I'm seeing are 100% exactly > the same). Attached for comparison. Hmm, looks similar enough to me -- at the bottom you have the executor

Re: Creating extensions for non-superusers

2018-08-13 Thread Robert Haas
On Fri, Aug 10, 2018 at 11:11 AM, Stephen Frost wrote: > For my 2c, I'd like something along these lines when it comes to a > capability but it's just not that simple. It seems pretty simple to me. We have a bunch of other predefined roles that allow otherwise-superuser-only operations to be

Re: libpq should not look up all host addresses at once

2018-08-13 Thread Tom Lane
Fabien COELHO writes: > In the same vein on a wrong password: > sh> psql "host=no-such-host,local2.coelho.net,local3.coelho.net" > Password for user fabien: > psql: could not translate host name "no-such-host" to address: Name or > service not known > could not connect to server:

Re: libpq should not look up all host addresses at once

2018-08-13 Thread Tom Lane
Fabien COELHO writes: > About the behavior from psql point of view: > * if dns works, error messages are only printed if all attempts failed: > But nothing shows if one succeeds at some point. I understand that libpq > is doing its job, but I'm wondering whether this is the best behavior.

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2018-08-13 Thread Marina Polyakova
On 12-08-2018 12:14, Fabien COELHO wrote: HEllo Marina, Hello, Fabien! v10-0003-Pgbench-errors-use-the-Variables-structure-for-c.patch - a patch for the Variables structure (this is used to reset client variables during the repeating of transactions after serialization/deadlock failures).

Re: Expression errors with "FOR UPDATE" and postgres_fdw with partition wise join enabled.

2018-08-13 Thread Etsuro Fujita
(2018/08/13 11:57), Robert Haas wrote: On Mon, Aug 6, 2018 at 8:30 AM, Etsuro Fujita wrote: In the above I used the test whether the relation's reloptkind is RELOPT_BASEREL or not, but I noticed that I had overlooked the case of a multi-level partitioned table. So I fixed that and added

Re: Get funcid when create function

2018-08-13 Thread Robert Haas
On Fri, Aug 10, 2018 at 5:50 AM, 王翔宇 wrote: > I'm developing a extension for pg. Now I have create a event trigger on > ddl_command_end, and this function will be called after I enter create > function statement. In this function I can only get parseTree. In pg source > code, I found a function

Re: [HACKERS] possible self-deadlock window after bad ProcessStartupPacket

2018-08-13 Thread Heikki Linnakangas
On 08/08/18 19:19, Heikki Linnakangas wrote: So, with this commit, the various SIGQUIT quickdie handlers are in a better shape. The regular backend's quickdie() handler still calls ereport(), which is not safe, but it's a step in the right direction. And we haven't addressed the original

Re: Temporary tables prevent autovacuum, leading to XID wraparound

2018-08-13 Thread Andres Freund
On 2018-08-09 18:50:47 +0200, Michael Paquier wrote: > On Thu, Aug 09, 2018 at 02:29:54AM -0700, Andres Freund wrote: > + /* > +* Mark MyProc as owning this namespace which other processes can use to > +* decide if a temporary namespace is in use or not. We assume that > +*

Re: Temporary tables prevent autovacuum, leading to XID wraparound

2018-08-13 Thread Michael Paquier
On Thu, Aug 09, 2018 at 06:50:47PM +0200, Michael Paquier wrote: > Better ideas are of course welcome. I have gone back-and-forth on this patch for the last couple of days, reworded the comment blocks to outline the point Andres has been making, and I have finally been able to push it and

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2018-08-13 Thread Antonin Houska
Toshi Harada wrote: > Hi. > > If "data-at-rest-encryption-wip-2018.07.25.patch" is applied to PostgreSQL > 11-beta3 released last week, > patch execution will fail as follows. > > patching file src/backend/replication/logical/reorderbuffer.c > Hunk #9 FAILED at 2464. > > 1 out of 7

"WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2018-08-13 Thread Toshi Harada
Hi. If "data-at-rest-encryption-wip-2018.07.25.patch" is applied to PostgreSQL 11-beta3 released last week, patch execution will fail as follows. patching file src/backend/replication/logical/reorderbuffer.c Hunk #9 FAILED at 2464. 1 out of 7 hunks FAILED -- saving rejects to file