Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-06 Thread Simon Riggs
On 5 September 2016 at 21:58, Claudio Freire wrote: How long does that part ever take? Is there any substantial gain from this? > Btw, without a further patch to prefetch pages on the backward scan > for truncate, however, my patience ran out before it finished >

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-09-06 Thread Abhijit Menon-Sen
At 2016-09-06 14:40:54 +0900, michael.paqu...@gmail.com wrote: > > my best advice here is to make all those recovery_target_* parameters > PGC_POSTMASTER so as they are loaded only once when the server starts, > and then we define the recovery target type used in the startup > process instead of

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-06 Thread Heikki Linnakangas
I'm reviewing patches 1-3 in this series, i.e. those patches that are not directly related to parallelism, but are independent improvements to merging. Let's begin with patch 1: On 08/02/2016 01:18 AM, Peter Geoghegan wrote: Cap the number of tapes used by external sorts Commit df700e6b set

Re: [HACKERS] Notice lock waits

2016-09-06 Thread Pavan Deolasee
On Fri, Aug 5, 2016 at 10:30 PM, Jeff Janes wrote: > > > A general facility for promoting selected LOG messages to NOTICE would > be nice, but I don't know how to design or implement that. This is > much easier, and I find it quite useful. > > IMHO that's what we need and

Re: [HACKERS] Speedup twophase transactions

2016-09-06 Thread Simon Riggs
On 6 September 2016 at 02:41, Michael Paquier wrote: > After review the result is attached. Perhaps a committer could get a look at > it? Yes, will do, but it will be a few more days yet. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-06 Thread Heikki Linnakangas
On 08/16/2016 03:33 AM, Peter Geoghegan wrote: I attach a patch that changes how we maintain the heap invariant during tuplesort merging. I already mentioned this over on the "Parallel tuplesort, partitioning, merging, and the future" thread. As noted already on that thread, this patch makes

Re: [HACKERS] pgsql: Add putenv support for msvcrt from Visual Studio 2013

2016-09-06 Thread Michael Paquier
On Thu, Sep 1, 2016 at 4:03 PM, Christian Ullrich wrote: > * Michael Paquier wrote: > >> On Wed, Apr 27, 2016 at 2:39 AM, Christian Ullrich >> wrote: > >>> * Christian Ullrich wrote: > >> And actually, by looking at those patches, isn't it a dangerous

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Ashutosh Bapat
I found a server crash while running make check in regress folder. with this set of patches. Problem is RelationBuildPartitionKey() partexprsrc may be used uninitialized. Initializing it with NIL fixes the crash. Here's patch to fix it. Came up with the fix after discussion with Amit. On Fri, Aug

Re: [HACKERS] make async slave to wait for lsn to be replayed

2016-09-06 Thread Ivan Kartyshov
On 08/31/2016 05:54 PM, Craig Ringer wrote: How do you get the commit LSN to watch for? Grab pg_current_xlog_insert_location() just after the commit and figure that replaying to that point guarantees you get the commit? That's the point, it was created in order to provide the cosistent view

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-09-06 Thread Petr Jelinek
On 06/09/16 07:18, Abhijit Menon-Sen wrote: One open issue is the various assign_recovery_target_xxx functions, which Michael noted in his earlier review: +static void +assign_recovery_target_xid(const char *newval, void *extra) +{ + recovery_target_xid = *((TransactionId *) extra); + +

Re: [HACKERS] Speedup twophase transactions

2016-09-06 Thread Stas Kelvich
> On 06 Sep 2016, at 04:41, Michael Paquier wrote: > > On Sat, Sep 3, 2016 at 10:26 PM, Michael Paquier > wrote: >> On Fri, Sep 2, 2016 at 5:06 AM, Simon Riggs wrote: >>> On 13 April 2016 at 15:31, Stas Kelvich

[HACKERS] Re: [COMMITTERS] pgsql: Add putenv support for msvcrt from Visual Studio 2013

2016-09-06 Thread Craig Ringer
On 6 Sep. 2016 15:12, "Michael Paquier" wrote: > > On Thu, Sep 1, 2016 at 4:03 PM, Christian Ullrich wrote: > > > That said, introducing this requirement would be a very significant change. > > I'm not sure how many independently maintained

Re: [HACKERS] Speedup twophase transactions

2016-09-06 Thread Michael Paquier
On Tue, Sep 6, 2016 at 5:58 PM, Stas Kelvich wrote: >> On 06 Sep 2016, at 04:41, Michael Paquier wrote: >> This overlaps with TwoPhaseGetGXact but I'd rather keep both things >> separated: it does not seem worth complicating the existing

Re: [HACKERS] Speedup twophase transactions

2016-09-06 Thread Simon Riggs
On 6 September 2016 at 09:58, Stas Kelvich wrote: >> On 06 Sep 2016, at 04:41, Michael Paquier wrote: >> >> On Sat, Sep 3, 2016 at 10:26 PM, Michael Paquier >> wrote: >>> On Fri, Sep 2, 2016 at 5:06 AM, Simon Riggs

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-09-06 Thread Abhijit Menon-Sen
At 2016-09-06 10:56:53 +0200, p...@2ndquadrant.com wrote: > > So +1 on the recovery_target = 'xid:xxx' idea. OK, I'll make it work that way. New patch in a couple of days. -- Abhijit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

[HACKERS] Override compile time log levels of specific messages/modules

2016-09-06 Thread Pavan Deolasee
While reviewing Jeff's notice_lock_waits patch, I came across his comment about having a general facility for promoting selected LOG messages. So I thought I should post it here, even though the patch is probably far from being accepted in Postgres. I recently wrote a patch for Postgres-XL to do

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-06 Thread Peter Geoghegan
On Tue, Sep 6, 2016 at 12:34 AM, Heikki Linnakangas wrote: > I'm reviewing patches 1-3 in this series, i.e. those patches that are not > directly related to parallelism, but are independent improvements to > merging. That's fantastic! Thanks! I'm really glad you're picking

Re: [HACKERS] Let file_fdw access COPY FROM PROGRAM

2016-09-06 Thread Craig Ringer
On 7 Sep. 2016 02:14, "Corey Huinker" wrote: > > Having regression tests for this is extremely problematic, because the program invoked would need to be an invokable command on any architecture supported by postgres. I'm pretty sure no such command exists. Your best bet

Re: [HACKERS] kqueue

2016-09-06 Thread Marko Tiikkaja
On 2016-06-03 01:45, Thomas Munro wrote: On Fri, Jun 3, 2016 at 4:02 AM, Alvaro Herrera wrote: Tom Lane wrote: Andres Freund writes: pg_strtoi? I think that's what Thomas did upthread. Are you taking this one then? I'd go with just

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-06 Thread Peter Geoghegan
On Tue, Sep 6, 2016 at 2:46 PM, Peter Geoghegan wrote: > Feel free to make a counter-proposal for a cap. I'm not attached to > 500. I'm mostly worried about blatant waste with very large workMem > sizings. Tens of thousands of tapes is just crazy. The amount of data > that you

Re: [HACKERS] improved DefElem list processing

2016-09-06 Thread Tom Lane
Peter Eisentraut writes: > Here are two WIP patches to improve the DefElem list processing that is > used by many utility commands. > One factors out the duplicate checks, which are currently taking up a > lot of space with duplicate code. I haven't applied

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2016-09-06 Thread Peter Geoghegan
On Tue, Sep 6, 2016 at 12:57 PM, Claudio Freire wrote: > Patch lacks any new tests, but the changed code paths seem covered > sufficiently by existing tests. A little bit of fuzzing on the patch > itself, like reverting some key changes, or flipping some key > comparisons,

Re: [HACKERS] Let file_fdw access COPY FROM PROGRAM

2016-09-06 Thread Michael Paquier
On Wed, Sep 7, 2016 at 7:53 AM, Craig Ringer wrote: > On 7 Sep. 2016 02:14, "Corey Huinker" wrote: >> > >> Having regression tests for this is extremely problematic, because the >> program invoked would need to be an invokable command on any

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2016-09-06 Thread Daniel Verite
Craig Ringer wrote: > Updated patch attached. Please find attached a couple fixes for typos I've came across in the doc part. Also it appears that PQqueriesInBatch() doesn't work as documented. It says: "Returns the number of queries still in the queue for this batch" but in fact it's

Re: [HACKERS] Suggestions for first contribution?

2016-09-06 Thread Aleksander Alekseev
Hello, Christian > Can anyone suggest a project for my first PG contribution? In my experience good starting points are: * Doing code review. It's really important for PostgreSQL project. Everyone likes to write code, no one likes to read or test it :) * Finding typos, there are a lot of

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-09-06 Thread Simon Riggs
On 6 September 2016 at 08:06, Abhijit Menon-Sen wrote: > At 2016-09-06 14:40:54 +0900, michael.paqu...@gmail.com wrote: >> >> my best advice here is to make all those recovery_target_* parameters >> PGC_POSTMASTER so as they are loaded only once when the server starts, >>

Re: [HACKERS] pgsql: Add putenv support for msvcrt from Visual Studio 2013

2016-09-06 Thread Christian Ullrich
* Michael Paquier wrote: On Thu, Sep 1, 2016 at 4:03 PM, Christian Ullrich wrote: My conclusion from April stands: The fact that master looks like it does means that there have not been many (or any) complaints about missing cross-module environment variables. If

Re: [HACKERS] PATCH: Exclude additional directories in pg_basebackup

2016-09-06 Thread Michael Paquier
On Wed, Sep 7, 2016 at 12:16 AM, David Steele wrote: > On 9/1/16 9:53 AM, Peter Eisentraut wrote: >> >> On 8/15/16 3:39 PM, David Steele wrote: >>> >>> That patch got me thinking about what else could be excluded and after >>> some investigation I found the following:

Re: [HACKERS] Suggestions for first contribution?

2016-09-06 Thread Noah Misch
On Tue, Sep 06, 2016 at 01:32:10PM -0400, Christian Convey wrote: > It sounds like the most useful thing I can do at the moment is perform > code reviews. I assumed I'd need more experience with the PG code > base, but I keep on reading that newcomers' reviews are welcome. > Unless someone has a

[HACKERS] Bug in VACUUM_TRUNCATE_LOCK_WAIT_INTERVAL

2016-09-06 Thread Simon Riggs
In vacuumlazy.c, VACUUM_TRUNCATE_LOCK_WAIT_INTERVAL is described as being in ms on line 85, yet it is used on line 1759 in a call to pg_usleep, so is treated as microseconds rather than milliseconds. As a result, the timeout during lazy_truncate_heap() is actually only 5ms long, not 5s long. So

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2016-09-06 Thread Craig Ringer
On 6 September 2016 at 16:10, Daniel Verite wrote: > Craig Ringer wrote: > >> Updated patch attached. > > Please find attached a couple fixes for typos I've came across in > the doc part. Thanks, will apply and post a rebased patch soon, or if someone picks this

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-09-06 Thread Robert Haas
On Wed, Aug 31, 2016 at 9:45 PM, Simon Riggs wrote: > This is a summary of proposed changes to the recovery.conf API for > v10. These are based in part on earlier discussions, and represent a > minimal modification to current usage. This looks great. > Proposed changes

Re: [HACKERS] condition variables

2016-09-06 Thread Amit Kapila
On Tue, Sep 6, 2016 at 5:29 PM, Robert Haas wrote: > On Mon, Sep 5, 2016 at 3:17 PM, Amit Kapila wrote: >> On Mon, Aug 15, 2016 at 10:35 PM, Robert Haas wrote: > Don't you need to set proc->cvSleeping = false in

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Rajkumar Raghuwanshi
Hi, I have applied updated patches given by you, and observe below. here in the given example, t6_p3 partition is not allowed to have null, but I am able to insert it, causing two nulls in the table. --create a partition table create table t6 (a int, b varchar) partition by list(a); create

Re: [HACKERS] Postgres abort found in 9.3.11

2016-09-06 Thread Tom Lane
"K S, Sandhya (Nokia - IN/Bangalore)" writes: > I was able to find a patch file where there is a call to ExitPostmaster() in > postmaster.c . > @@ -3081,6 +3081,11 @@ > shmem_exit(1); > reset_shared(PostPortNumber); > + /* recovery termination */

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Amit Langote
On Tue, Sep 6, 2016 at 9:19 PM, Robert Haas wrote: > On Wed, Aug 31, 2016 at 1:05 PM, Amit Langote > wrote: >>> However, it seems a lot better to make it a property of the parent >>> from a performance point of view. Suppose there are 1000

Re: [HACKERS] pgsql: Add putenv support for msvcrt from Visual Studio 2013

2016-09-06 Thread Christian Ullrich
* Michael Paquier wrote: On Tue, Sep 6, 2016 at 5:36 PM, Christian Ullrich wrote: * Michael Paquier wrote: In order to avoid any problems with the load and unload windows, my bet goes for 0001, 0002 and 0003, with the last two patches merged together, 0001 being only

Re: [HACKERS] condition variables

2016-09-06 Thread Robert Haas
On Mon, Sep 5, 2016 at 3:17 PM, Amit Kapila wrote: > On Mon, Aug 15, 2016 at 10:35 PM, Robert Haas wrote: Don't you need to set proc->cvSleeping = false in ConditionVariableSignal? >>> >>> I poked at this a bit... OK, a lot... and have some

Re: [HACKERS] pgsql: Add putenv support for msvcrt from Visual Studio 2013

2016-09-06 Thread Michael Paquier
(Just remembered to remove pgsql-committers here). On Tue, Sep 6, 2016 at 9:21 PM, Christian Ullrich wrote: > * Michael Paquier wrote: > >> On Tue, Sep 6, 2016 at 5:36 PM, Christian Ullrich >> wrote: >>> >>> * Michael Paquier wrote: > In order to

[HACKERS] Push down more UPDATEs/DELETEs in postgres_fdw

2016-09-06 Thread Etsuro Fujita
Hi, Attached is a WIP patch extending the postgres_fdw DML pushdown in 9.6 so that it can perform an update/delete on a join remotely. An example is shown below: * without the patch: postgres=# explain verbose delete from ft1 using ft2 where ft1.a = ft2.a; QUERY PLAN

Re: [HACKERS] Push down more full joins in postgres_fdw

2016-09-06 Thread Ashutosh Bapat
On Fri, Sep 2, 2016 at 3:55 PM, Etsuro Fujita wrote: > Hi Ashutosh, > > On 2016/08/22 15:49, Ashutosh Bapat wrote: > >> 1. deparsePlaceHolderVar looks odd - each of the deparse* function is >> named as deparse + > into>. PlaceHolderVar is not a parser node, so no

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Ashutosh Bapat
This patch uses RangeBound structure. There's also a structure defined with the same name in rangetypes.h with some slight differences. Should we rename the one in partition.c as PartRangeBound or something like that to avoid the confusion? -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-09-06 Thread David Steele
On 9/6/16 4:56 AM, Petr Jelinek wrote: On 06/09/16 07:18, Abhijit Menon-Sen wrote: Do we want something like this (easy to implement and document, perhaps not especially convenient to use): recovery_target = 'xid' # or 'time'/'name'/'lsn'/'immediate' recovery_target_xid = xxx? #

[HACKERS] Tuplesort merge pre-reading

2016-09-06 Thread Heikki Linnakangas
While reviewing Peter's latest round of sorting patches, and trying to understand the new "batch allocation" mechanism, I started to wonder how useful the pre-reading in the merge stage is in the first place. I'm talking about the code that reads a bunch of from each tape, loading them into

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Robert Haas
On Wed, Aug 31, 2016 at 1:05 PM, Amit Langote wrote: >> However, it seems a lot better to make it a property of the parent >> from a performance point of view. Suppose there are 1000 partitions. >> Reading one toasted value for pg_class and running stringToNode()

Re: [HACKERS] pgsql: Add putenv support for msvcrt from Visual Studio 2013

2016-09-06 Thread Michael Paquier
On Tue, Sep 6, 2016 at 5:36 PM, Christian Ullrich wrote: > * Michael Paquier wrote: >> In order to avoid any problems with the load and unload windows, my >> bet goes for 0001, 0002 and 0003, with the last two patches merged >> together, 0001 being only a set of independent

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-09-06 Thread Petr Jelinek
On 06/09/16 13:52, David Steele wrote: On 9/6/16 4:56 AM, Petr Jelinek wrote: On 06/09/16 07:18, Abhijit Menon-Sen wrote: Do we want something like this (easy to implement and document, perhaps not especially convenient to use): recovery_target = 'xid' # or

Re: [HACKERS] Bug in 9.6 tuplesort batch memory growth logic

2016-09-06 Thread Tom Lane
Peter Geoghegan writes: > While working on my parallel CREATE INDEX patch, I came across a > problem. I took a quick look at this. I do not follow the logic in this new bit: + if (availMemLessRefund <= + (int64) state->activeTapes *

Re: [HACKERS] Bug in VACUUM_TRUNCATE_LOCK_WAIT_INTERVAL

2016-09-06 Thread Robert Haas
On Tue, Sep 6, 2016 at 4:00 PM, Simon Riggs wrote: > I'm also wondering why we don't use lock_timeout when the user sets it? > Not a bug, but patch attached anyway. > vacuum_truncate_use_lock_timeout.v1.patch This part seems fairly random. I don't think it makes sense to

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Amit Langote
Hi, On Tue, Sep 6, 2016 at 8:15 PM, Rajkumar Raghuwanshi wrote: > Hi, > > I have applied updated patches given by you, and observe below. > > here in the given example, t6_p3 partition is not allowed to have null, but > I am able to insert it, causing two

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-09-06 Thread David Steele
On 9/6/16 8:07 AM, Robert Haas wrote: On Wed, Aug 31, 2016 at 9:45 PM, Simon Riggs wrote: > Related cleanup * Promotion signal file is now called "promote.trigger" rather than just "promote" * Remove user configurable "trigger_file" mechanism - use "promote.trigger"

Re: [HACKERS] ICU integration

2016-09-06 Thread Peter Eisentraut
On 8/31/16 4:24 PM, Doug Doole wrote: > ICU explicitly does not provide stability in their locales and collations. We > pushed them hard to provide this, but between changes to the CLDR data and > changes to the ICU code it just wasn’t feasible for them to provide version > to version

Re: [HACKERS] ICU integration

2016-09-06 Thread Peter Eisentraut
On 8/31/16 1:32 PM, Peter Geoghegan wrote: > ICU is happy to have multiple versions > of a collation at a time, and you'll probably retain the old collation > version in ICU. > > Even if your old collation version isn't available in a new ICU > release (which I think is unlikely in practice) I

Re: [HACKERS] ICU integration

2016-09-06 Thread Peter Eisentraut
On 8/31/16 12:32 AM, Michael Paquier wrote: > On Wed, Aug 31, 2016 at 1:12 PM, Peter Eisentraut > wrote: >> On 8/30/16 11:27 PM, Craig Ringer wrote: >>> Speaking of which, have you had a chance to try it on Windows yet? >> >> nope > > +SELECT a, b FROM

Re: [HACKERS] Proposal for changes to recovery.conf API

2016-09-06 Thread Michael Paquier
On Tue, Sep 6, 2016 at 10:01 PM, Petr Jelinek wrote: > That's also reasonable solution, I don't really have preference between > those. My main point was to get rid of the 5 or so variables where only one > will actually be used in the end. And no need to worry about the

Re: [HACKERS] PATCH: Exclude additional directories in pg_basebackup

2016-09-06 Thread David Steele
On 9/1/16 9:53 AM, Peter Eisentraut wrote: On 8/15/16 3:39 PM, David Steele wrote: That patch got me thinking about what else could be excluded and after some investigation I found the following: pg_notify, pg_serial, pg_snapshots, pg_subtrans. These directories are all cleaned, zeroed, or

Re: [HACKERS] Bug in 9.6 tuplesort batch memory growth logic

2016-09-06 Thread Peter Geoghegan
On Tue, Sep 6, 2016 at 6:17 AM, Tom Lane wrote: > It doesn't seem to me that this limit has anything to do with anything, > and the comment claiming only that it's "noncritical" isn't helping. > If the point is to prevent the later LACKMEM check from failing, then > certainly

Re: [HACKERS] Bug in VACUUM_TRUNCATE_LOCK_WAIT_INTERVAL

2016-09-06 Thread Simon Riggs
On 6 September 2016 at 11:30, Simon Riggs wrote: > In vacuumlazy.c, VACUUM_TRUNCATE_LOCK_WAIT_INTERVAL is described as > being in ms on line 85, yet it is used on line 1759 in a call to > pg_usleep, so is treated as microseconds rather than milliseconds. > > As a result,

Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan

2016-09-06 Thread Jeevan Chalke
Hi, Changes look good to me. However there are couple of minor issues need to be fixed. 1. "under" repeated on second line. Please remove. +if and when CustomScanState is located under +under LimitState; which implies the underlying node is not 2. Typo: dicsussion => discussion Please

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-09-06 Thread Simon Riggs
On 29 August 2016 at 14:52, Fujii Masao wrote: > On Sat, Aug 6, 2016 at 6:36 PM, Petr Jelinek wrote: >> On 04/08/16 06:40, Masahiko Sawada wrote: >>> >>> On Wed, Aug 3, 2016 at 3:05 PM, Michael Paquier >>> wrote:

Re: [HACKERS] Refactoring of heapam code.

2016-09-06 Thread Anastasia Lubennikova
06.09.2016 07:44, Pavan Deolasee: On Mon, Aug 8, 2016 at 3:13 PM, Anastasia Lubennikova > wrote: Thank you for the review, I'll fix these problems in final version. Posting the first message I intended to raise

Re: [HACKERS] Postgres abort found in 9.3.11

2016-09-06 Thread K S, Sandhya (Nokia - IN/Bangalore)
Hello, I was able to find a patch file where there is a call to ExitPostmaster() in postmaster.c . @@ -3081,6 +3081,11 @@ shmem_exit(1); reset_shared(PostPortNumber); + /* recovery termination */ + ereport(FATAL, + (errmsg("recovery termination due to

Re: [HACKERS] Optimization for lazy_scan_heap

2016-09-06 Thread Robert Haas
On Mon, Sep 5, 2016 at 8:57 PM, Masahiko Sawada wrote: >> What performance difference does this make, in a realistic use case? > > I have yet to measure performance effect but it would be effect for > very large frozen table. I think if you are proposing this patch because

Re: [HACKERS] WIP: Covering + unique indexes.

2016-09-06 Thread Anastasia Lubennikova
28.08.2016 09:13, Amit Kapila: On Mon, Aug 15, 2016 at 8:15 PM, Anastasia Lubennikova wrote: @@ -590,7 +622,14 @@ _bt_buildadd(BTWriteState *wstate, BTPageState *state, IndexTuple itup) if (last_off == P_HIKEY) { Assert(state->btps_minkey == NULL); -

Re: [HACKERS] Re: [COMMITTERS] pgsql: Make initdb's suggested "pg_ctl start" command line more reliabl

2016-09-06 Thread Andres Freund
On 2016-09-06 13:08:51 -0400, Tom Lane wrote: > Dash is considered a character that needs quoting. It might be possible > to avoid that if we could be certain that appendShellString's output would > never be placed in a spot where it could be taken for a switch, but that > seems like a large

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-06 Thread Robert Haas
On Sat, Sep 3, 2016 at 8:55 AM, Claudio Freire wrote: > The attached patch allows setting maintainance_work_mem or > autovacuum_work_mem higher than 1GB (and be effective), by turning the > allocation of the dead_tuples into a huge allocation. > > This results in fewer

Re: [HACKERS] Bug in 9.6 tuplesort batch memory growth logic

2016-09-06 Thread Tom Lane
Peter Geoghegan writes: > On Tue, Sep 6, 2016 at 6:17 AM, Tom Lane wrote: >> It doesn't seem to me that this limit has anything to do with anything, >> and the comment claiming only that it's "noncritical" isn't helping. > You've not understood the problem

Re: [HACKERS] GiST penalty functions [PoC]

2016-09-06 Thread Heikki Linnakangas
On 08/29/2016 09:04 AM, Andrew Borodin wrote: In this message I address only first problem. I want to construct a penalty function that will choose: 1.Entry with a zero volume and smallest margin, that can accommodate insertion tuple without extension, if one exists; 2.Entry with

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-06 Thread Claudio Freire
On Sun, Sep 4, 2016 at 8:10 PM, Jim Nasby wrote: > On 9/4/16 1:46 AM, Simon Riggs wrote: >>> >>> > The patch also makes vacuum free the dead_tuples before starting >>> > truncation. It didn't seem necessary to hold onto it beyond that >>> > point, and it might help give

Re: [HACKERS] SELECT FOR UPDATE regression in 9.5

2016-09-06 Thread Marko Tiikkaja
On 2016-09-06 6:02 PM, Marti Raudsepp wrote: This issue is also reproducible on the current master branch. In an assertions-enabled build, it traps an assertion in HeapTupleHeaderGetCmax called by heap_lock_tuple. The following test case demonstrates the issue... I think you found a

Re: [HACKERS] SELECT FOR UPDATE regression in 9.5

2016-09-06 Thread Alvaro Herrera
Marko Tiikkaja wrote: > On 2016-09-06 6:02 PM, Marti Raudsepp wrote: > >This issue is also reproducible on the current master branch. In an > >assertions-enabled build, it traps an assertion in HeapTupleHeaderGetCmax > >called by heap_lock_tuple. The following test case demonstrates the issue... >

Re: [HACKERS] Fun fact about autovacuum and orphan temp tables

2016-09-06 Thread Jim Nasby
On 9/5/16 12:14 PM, Bruce Momjian wrote: > I have certainly faced my fair share of customers with dangling temp > tables, and would like to see this changed in some way or another. I don't think we look at those temp tables frequently enough to justify keeping them around for all users. Plus,

Re: [HACKERS] Re: [COMMITTERS] pgsql: Make initdb's suggested "pg_ctl start" command line more reliabl

2016-09-06 Thread Tom Lane
Peter Eisentraut writes: > On 8/20/16 3:05 PM, Tom Lane wrote: >> Make initdb's suggested "pg_ctl start" command line more reliable. > A couple of problems with this: > The not-quoting-if-not-needed doesn't appear to do anything useful for me: >

Re: [HACKERS] Forbid use of LF and CR characters in database and role names

2016-09-06 Thread Peter Eisentraut
On 8/11/16 9:12 PM, Michael Paquier wrote: > Note that pg_dump[all] and pg_upgrade already have safeguards against > those things per the same routines putting quotes for execution as > commands into psql and shell. So attached is a patch to implement this > restriction in the backend, and I am

[HACKERS] Re: [COMMITTERS] pgsql: Make initdb's suggested "pg_ctl start" command line more reliabl

2016-09-06 Thread Peter Eisentraut
On 8/20/16 3:05 PM, Tom Lane wrote: > Make initdb's suggested "pg_ctl start" command line more reliable. > > The original coding here was not nearly careful enough about quoting > special characters, and it didn't get corner cases right for constructing > the pg_ctl path either. Use

Re: [HACKERS] (Comment)Bug in CteScanNext

2016-09-06 Thread Tom Lane
Jim Nasby writes: > On 9/3/16 1:30 PM, Tom Lane wrote: >> Or we could add something like "But first, we must deal with the special >> case of reversing direction after reaching EOF." > I'm working on that, but one thing isn't clear to me... why do we only > skip past

Re: [HACKERS] Re: [COMMITTERS] pgsql: Make initdb's suggested "pg_ctl start" command line more reliabl

2016-09-06 Thread Claudio Freire
On Tue, Sep 6, 2016 at 2:08 PM, Tom Lane wrote: >> The not-quoting-if-not-needed doesn't appear to do anything useful for me: >> 'pg-install/bin/pg_ctl' -D 'pg-install/var/data' -l logfile start > > Dash is considered a character that needs quoting. It might be possible >

[HACKERS] SELECT FOR UPDATE regression in 9.5

2016-09-06 Thread Marti Raudsepp
Hello list While testing an application with PostgreSQL 9.5, we experienced an issue involving aborted subtransactions and SELECT FOR UPDATE. In certain situations, a locking query doesn't return rows that should be visible and already locked by the current transaction. I bisected this down to

Re: [HACKERS] Obsolete TODO item "-Wcast-align" ?

2016-09-06 Thread Jim Nasby
On 9/4/16 5:10 PM, Christian Convey wrote: Thanks for the response. I'm unclear about how the TODO list is curated. Is there someone whose attention I should direct to this thread? There's a number of people that can edit the TODO list. I've removed that item. -- Jim Nasby, Data Architect,

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-06 Thread Jim Nasby
On 9/4/16 1:46 AM, Simon Riggs wrote: > The patch also makes vacuum free the dead_tuples before starting > truncation. It didn't seem necessary to hold onto it beyond that > point, and it might help give the OS more cache, especially if work > mem is configured very high to avoid multiple index

Re: [HACKERS] Quorum commit for multiple synchronous replication.

2016-09-06 Thread Masahiko Sawada
On Tue, Sep 6, 2016 at 11:08 PM, Simon Riggs wrote: > On 29 August 2016 at 14:52, Fujii Masao wrote: >> On Sat, Aug 6, 2016 at 6:36 PM, Petr Jelinek wrote: >>> On 04/08/16 06:40, Masahiko Sawada wrote: On Wed, Aug 3,

Re: [HACKERS] (Comment)Bug in CteScanNext

2016-09-06 Thread Jim Nasby
On 9/3/16 1:30 PM, Tom Lane wrote: Or we could add something like "But first, we must deal with the special case of reversing direction after reaching EOF." I'm working on that, but one thing isn't clear to me... why do we only skip past the last tuple if (!node->leader->eof_cte)? Even if

Re: [HACKERS] Re: [COMMITTERS] pgsql: Make initdb's suggested "pg_ctl start" command line more reliabl

2016-09-06 Thread Tom Lane
Claudio Freire writes: > On Tue, Sep 6, 2016 at 2:08 PM, Tom Lane wrote: >> Dash is considered a character that needs quoting. It might be possible >> to avoid that if we could be certain that appendShellString's output would >> never be placed in a

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-06 Thread Robert Haas
On Tue, Sep 6, 2016 at 2:00 PM, Tom Lane wrote: > Robert Haas writes: >> Yeah, but I've seen actual breakage from exactly this issue on >> customer systems even with the 1GB limit, and when we start allowing >> 100GB it's going to get a whole lot worse.

Re: [HACKERS] Logical Replication WIP

2016-09-06 Thread Peter Eisentraut
On 9/3/16 5:14 AM, Petr Jelinek wrote: >> What are the BKI_ROWTYPE_OID assignments for? Are they necessary >> > here? (Maybe this was just copied from pg_subscription?) >> > > Yes they are. Please explain/document why. It does not match other catalogs, which either use it for relcache

Re: [HACKERS] \timing interval

2016-09-06 Thread Corey Huinker
On Sun, Sep 4, 2016 at 7:05 PM, Jim Nasby wrote: > On 9/3/16 2:35 PM, Tom Lane wrote: > >> I pushed the patch using this: >> >> Time: 176460001.200 ms (2 d 01:01:00.001) >> >> and all else as before. >> > > I'd find this useful in the final output of EXPLAIN ANALYZE as

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-06 Thread Simon Riggs
On 6 September 2016 at 19:23, Robert Haas wrote: > On Tue, Sep 6, 2016 at 2:16 PM, Simon Riggs wrote: >> What occurs to me is that we can exactly predict how many tuples we >> are going to get when we autovacuum, since we measure that and we know >>

Re: [HACKERS] Logical Replication WIP

2016-09-06 Thread Petr Jelinek
On 01/09/16 08:29, Erik Rijkers wrote: On 2016-09-01 01:04, Erik Rijkers wrote: On 2016-08-31 22:51, Petr Jelinek wrote: Here are some small changes to logical-replication.sgml ... and other .sgml files. Thanks I'll integrate these into next iteration of the patch, -- Petr Jelinek

Re: [HACKERS] Forbid use of LF and CR characters in database and role names

2016-09-06 Thread Robert Haas
On Tue, Sep 6, 2016 at 9:43 PM, Peter Eisentraut wrote: > On 8/11/16 9:12 PM, Michael Paquier wrote: >> Note that pg_dump[all] and pg_upgrade already have safeguards against >> those things per the same routines putting quotes for execution as >> commands into

Re: [HACKERS] [PATCH] Alter or rename enum value

2016-09-06 Thread Robert Haas
On Mon, Sep 5, 2016 at 11:40 PM, Tom Lane wrote: > The opportunity cost here is potential user confusion. The only > closely parallel rename operation we have is ALTER TABLE RENAME COLUMN, > and that doesn't have a column-level IF EXISTS option; it has a > table-level IF

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-06 Thread Robert Haas
On Tue, Sep 6, 2016 at 11:22 PM, Claudio Freire wrote: > CREATE INDEX could also allocate 218GB, you just need to index enough > columns and you'll get that. > > Aside from the fact that CREATE INDEX will only allocate what is going > to be used and VACUUM will

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-06 Thread Simon Riggs
On 6 September 2016 at 19:00, Tom Lane wrote: > Robert Haas writes: >> Yeah, but I've seen actual breakage from exactly this issue on >> customer systems even with the 1GB limit, and when we start allowing >> 100GB it's going to get a whole lot worse. >

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-06 Thread Robert Haas
On Tue, Sep 6, 2016 at 2:16 PM, Simon Riggs wrote: > What occurs to me is that we can exactly predict how many tuples we > are going to get when we autovacuum, since we measure that and we know > what the number is when we trigger it. > > So there doesn't need to be any

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-06 Thread Robert Haas
On Tue, Sep 6, 2016 at 2:51 PM, Simon Riggs wrote: > On 6 September 2016 at 19:23, Robert Haas wrote: >> On Tue, Sep 6, 2016 at 2:16 PM, Simon Riggs wrote: >>> What occurs to me is that we can exactly predict how many tuples

Re: [HACKERS] ICU integration

2016-09-06 Thread Doug Doole
> The ICU ABI (not API) is also versioned. The way that this is done is > that all functions are actually macros to a versioned symbol. So > ucol_open() is actually a macro that expands to, say, ucol_open_57() in > ICU version 57. (They also got rid of a dot in their versions a while > ago.)

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-06 Thread Tom Lane
Robert Haas writes: > Yeah, but I've seen actual breakage from exactly this issue on > customer systems even with the 1GB limit, and when we start allowing > 100GB it's going to get a whole lot worse. While it's not necessarily a bad idea to consider these things, I think

Re: [HACKERS] Let file_fdw access COPY FROM PROGRAM

2016-09-06 Thread Corey Huinker
On Fri, Sep 2, 2016 at 5:07 AM, Amit Langote wrote: > > I am not familiar with win32 stuff too, so I don't have much to say about > that. Maybe someone else can chime in to help with that. > The regressions basically *can't* test this because we'd need a shell

Re: [HACKERS] \timing interval

2016-09-06 Thread Tom Lane
Corey Huinker writes: > On Sun, Sep 4, 2016 at 7:05 PM, Jim Nasby wrote: >> I'd find this useful in the final output of EXPLAIN ANALYZE as well; any >> objections to adding it? > It's sorta out of my hands now, but what Tom said earlier is that

Re: [HACKERS] Showing parallel status in \df+

2016-09-06 Thread Pavel Stehule
Hi 2016-09-06 0:05 GMT+02:00 Tom Lane : > I wrote: > > Pavel Stehule writes: > >> Using footer for this purpose is little bit strange. What about > following > >> design? > >> 1. move out source code of PL functions from \df+ > >> 2. allow not unique

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2016-09-06 Thread Robert Haas
On Tue, Sep 6, 2016 at 10:28 PM, Claudio Freire wrote: >> The problem with this is that we allocate the entire amount of >> maintenance_work_mem even when the number of actual dead tuples turns >> out to be very small. That's not so bad if the amount of memory we're >>

  1   2   >