Re: [HACKERS] Automatic cleanup of oldest WAL segments with pg_receivexlog

2017-02-23 Thread Jim Nasby
On 2/23/17 8:47 PM, Michael Paquier wrote: Anything else than measured in bytes either requires a lookup at the file timestamp, which is not reliable with noatime or a lookup at WAL itself to decide when is the commit timestamp that matches the oldest point in time of the backup policy. An indi

Re: [HACKERS] Automatic cleanup of oldest WAL segments with pg_receivexlog

2017-02-23 Thread Jim Nasby
On 2/23/17 8:52 PM, Michael Paquier wrote: OK, I forgot a bit about this past discussion. So let's say that we have a command, why not also allow users to use at will a marker %f to indicate the file name just completed? One use case here is to scan the file for the oldest and/or newest timestamp

Re: [HACKERS] btree_gin and btree_gist for enums

2017-02-23 Thread Andrew Dunstan
On 02/23/2017 08:34 PM, Tom Lane wrote: > Andrew Dunstan writes: >> On 02/23/2017 04:41 PM, Tom Lane wrote: >>> BTW, while I'm looking at this ... isn't gin_enum_cmp broken on its face? >> Yes, that's what I'm trying to fix. > I'd forgotten where this thread started. For a minute there I though

Re: [HACKERS] Automatic cleanup of oldest WAL segments with pg_receivexlog

2017-02-23 Thread Michael Paquier
On Fri, Feb 24, 2017 at 11:56 AM, Jim Nasby wrote: > On 2/23/17 8:47 PM, Michael Paquier wrote: >> >> Anything else than measured in bytes either requires a lookup at the >> file timestamp, which is not reliable with noatime or a lookup at WAL >> itself to decide when is the commit timestamp that

Re: [HACKERS] Idea on how to simplify comparing two sets

2017-02-23 Thread Jim Nasby
On 2/23/17 3:33 PM, Corey Huinker wrote: I've had to do it with temp tables any time the environment is different between control/experiment, which is the case when you're developing a drop-in replacement for an SRF or view or whatever. FWIW I'd recommend temp views, to give the planner the mos

Re: [HACKERS] Automatic cleanup of oldest WAL segments with pg_receivexlog

2017-02-23 Thread Jim Nasby
On 2/23/17 9:01 PM, Michael Paquier wrote: An idea here would be to add in the long header of the segment a timestamp of when it was created. This is inherent to only the server generating the WAL. ISTM it'd be reasonable (maybe even wise) for WAL files to contain info about the first and last

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-02-23 Thread Amit Kapila
On Thu, Feb 23, 2017 at 5:15 PM, Masahiko Sawada wrote: > On Wed, Feb 22, 2017 at 12:01 AM, Amit Kapila wrote: > >> I understand that there could be some delay in reclaiming dead pages >> but do you think it is such a big deal that we completely scan the >> index for such cases or even try to cha

Re: [HACKERS] A typo in mcxt.c

2017-02-23 Thread Andres Freund
On 2017-02-23 14:26:07 -0600, Jim Nasby wrote: > On 2/23/17 6:38 AM, Thomas Munro wrote: > > > I'm not so confident, but the "'tis" seems to me to be a typo of > > > "it's". > > That is an archaic way of contracting the same words differently: > > Given the number of non-native English speakers we

[HACKERS] timeouts in PostgresNode::psql

2017-02-23 Thread Craig Ringer
Hi all While writing some tests I noticed that in newer IPC::Run or Perl versions (haven't dug extensively to find out which), perl appends the location to the extension, so 'ne' doesn't match the passed exception string. Pattern-match the exception string to handle this. Bugfix, should be appli

Re: [HACKERS] A typo in mcxt.c

2017-02-23 Thread Tom Lane
Andres Freund writes: > On 2017-02-23 14:26:07 -0600, Jim Nasby wrote: >> On 2/23/17 6:38 AM, Thomas Munro wrote: >>> That is an archaic way of contracting the same words differently: >> Given the number of non-native English speakers we have, it's probably worth >> changing it... > I'm a non-na

Re: [HACKERS] Proposal: GetOldestXminExtend for ignoring arbitrary vacuum flags

2017-02-23 Thread Seki, Eiji
On 2017-02-15 17:27:11 Robert Haas wrote: > On Tue, Feb 14, 2017 at 1:38 PM, Jim Nasby > wrote: > > On 2/14/17 3:13 AM, Seki, Eiji wrote: > >> +extern TransactionId GetOldestXmin(Relation rel, uint8 ignoreFlags); > > > > > > My impression is that most other places that do this sort of thing ju

Re: [HACKERS] Documentation improvements for partitioning

2017-02-23 Thread Simon Riggs
On 24 February 2017 at 02:36, Robert Haas wrote: >> While its true that the patch had syntax documentation, there was no >> user design documentation which explained how it worked to allow >> objective review. Had I been able to provide input without reading >> every email message, I would have d

Re: [HACKERS] Enabling parallelism for queries coming from SQL or other PL functions

2017-02-23 Thread Amit Kapila
On Thu, Feb 23, 2017 at 8:58 PM, Dilip Kumar wrote: > On Thu, Feb 23, 2017 at 12:11 PM, Rafia Sabih > wrote: > > 2. How are you protecting, if the outer select is running in parallel, > then the function called from there should not run anything in > parallel? This may allow worker launching anot

Re: [HACKERS] Proposal: GetOldestXminExtend for ignoring arbitrary vacuum flags

2017-02-23 Thread Simon Riggs
On 16 February 2017 at 05:24, Seki, Eiji wrote: > Simon Riggs wrote: >> Please persuade us with measurements that allowing this impact on >> ANALYZE would really improve performance at least in your case, and >> also examine the effect of this on the accuracy and usefulness of the >> gathered sta

Re: [HACKERS] Should we cacheline align PGXACT?

2017-02-23 Thread Ashutosh Sharma
Okay. As suggested by Alexander, I have changed the order of reading and doing initdb for each pgbench run. With these changes, I got following results at 300 scale factor with 8GB of shared buffer. *pgbench settings:* pgbench -i -s 300 postgres pgbench -M prepared -c $thread -j $thread -T $time_f

Re: [HACKERS] Enabling parallelism for queries coming from SQL or other PL functions

2017-02-23 Thread Amit Kapila
On Thu, Feb 23, 2017 at 9:20 PM, Dilip Kumar wrote: > On Thu, Feb 23, 2017 at 8:58 PM, Dilip Kumar wrote: >> Few more comments. >> 1.I don't see any check in the code which will prevent the parallel >> execution of the query inside a function if its called from a DML >> statement. >> e.g. If we u

Re: [HACKERS] Range Partitioning behaviour - query

2017-02-23 Thread Venkata B Nagothi
On Fri, Feb 24, 2017 at 1:01 PM, Amit Langote wrote: > On 2017/02/24 10:38, David G. Johnston wrote: > > On Thu, Feb 23, 2017 at 6:17 PM, Amit Langote < > langote_amit...@lab.ntt.co.jp > >> wrote: > > > >> On 2017/02/24 8:38, Venkata B Nagothi wrote: > >>> On Thu, Feb 23, 2017 at 3:14 PM, Amit La

Re: [HACKERS] Should we cacheline align PGXACT?

2017-02-23 Thread Simon Riggs
On 24 February 2017 at 04:41, Ashutosh Sharma wrote: > Okay. As suggested by Alexander, I have changed the order of reading and > doing initdb for each pgbench run. With these changes, I got following > results at 300 scale factor with 8GB of shared buffer. > Would you be able to test my patch a

Re: [HACKERS] Range Partitioning behaviour - query

2017-02-23 Thread Venkata B Nagothi
On Fri, Feb 24, 2017 at 12:38 PM, David G. Johnston < david.g.johns...@gmail.com> wrote: > On Thu, Feb 23, 2017 at 6:17 PM, Amit Langote < > langote_amit...@lab.ntt.co.jp> wrote: > >> On 2017/02/24 8:38, Venkata B Nagothi wrote: >> > On Thu, Feb 23, 2017 at 3:14 PM, Amit Langote wrote: >> >> Upper

Re: [HACKERS] Enabling parallelism for queries coming from SQL or other PL functions

2017-02-23 Thread Dilip Kumar
On Fri, Feb 24, 2017 at 10:06 AM, Amit Kapila wrote: > We have a below check in standard_planner() (!IsParallelWorker()) > which should prohibit generating parallel plan inside worker, if that > is what you are seeing, then we might need a similar check at other > places. > > if ((cursorOptions &

[HACKERS] utility commands benefiting from parallel plan

2017-02-23 Thread Haribabu Kommi
Hi Hackers, Here I attached an implementation patch that allows utility statements that have queries underneath such as CREATE TABLE AS, CREATE MATERIALIZED VIEW and REFRESH commands to benefit from parallel plan. These write operations not performed concurrently by the parallel workers, but the

Re: [HACKERS] Documentation improvements for partitioning

2017-02-23 Thread Robert Haas
On Fri, Feb 24, 2017 at 9:53 AM, Simon Riggs wrote: > The good news is that logical replication DOES work with partitioning, > but only for a Publication with PUBLISH INSERT, pushing from a normal > table to a partitioned one. Which is useful enough for first release. > > The work on having UPDATE

Re: [HACKERS] case_preservation_and_insensitivity = on

2017-02-23 Thread Robert Haas
On Thu, Feb 23, 2017 at 6:59 PM, Tom Lane wrote: > I think these are straw-man arguments, really. Consider the actual use > case for such a feature: it's for porting some application that was not > written against Postgres to begin with. I'm not sure that's totally true. I think at least some r

Re: [HACKERS] Proposal: GetOldestXminExtend for ignoring arbitrary vacuum flags

2017-02-23 Thread Craig Ringer
On 14 February 2017 at 14:19, Seki, Eiji wrote: > Hi all, > > I propose the patch that adds a function GetOldestXminExtended that is like > GetOldestXmin but can ignore arbitrary vacuum flags. And then, rewrite > GetOldestXmin to use it. Note that this is done so as not to change the > behavior

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-23 Thread Andres Freund
On 2017-02-23 17:28:26 -0500, Tom Lane wrote: > Jeff Janes writes: > > The number of new chunks can be almost as as large as the number of old > > chunks, especially if there is a very popular value. The problem is that > > every time an old chunk is freed, the code in aset.c around line 968 has

Re: [HACKERS] Should we cacheline align PGXACT?

2017-02-23 Thread Ashutosh Sharma
On Fri, Feb 24, 2017 at 10:41 AM, Simon Riggs wrote: > On 24 February 2017 at 04:41, Ashutosh Sharma wrote: >> >> Okay. As suggested by Alexander, I have changed the order of reading and >> doing initdb for each pgbench run. With these changes, I got following >> results at 300 scale factor with

Re: [HACKERS] \if, \elseif, \else, \endif (was Re: PSQL commands: \quit_if, \quit_unless)

2017-02-23 Thread Fabien COELHO
About v17: Patch applies, "make check" & psql "make check" ok. ... '@' [...] I noticed that it takes precedence over '!'. [...] My reasoning was this: if you're in a false block, and you're not connected to a db, the \c isn't going to work for you until you get out of the false block, so rig

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-23 Thread Tom Lane
Andres Freund writes: > On 2017-02-23 17:28:26 -0500, Tom Lane wrote: >> Maybe it's time to convert that to a doubly-linked list. > Yes, I do think so. Given that we only have that for full blocks, not > for small chunks, the cost seems neglegible. > That would also, partially, address the perfor

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-23 Thread Thomas Munro
On Fri, Feb 24, 2017 at 12:17 PM, Andres Freund wrote: > Jeff, do you have a handy demonstrator? If you want to hit ExecHashIncreaseNumBatches a bunch of times, see the "ugly" example in the attached. -- Thomas Munro http://www.enterprisedb.com hj-test-queries.sql Description: Binary data --

Re: [HACKERS] UPDATE of partition key

2017-02-23 Thread Robert Haas
On Mon, Feb 20, 2017 at 2:58 PM, Amit Khandekar wrote: > I am inclined to at least have some option for the user to decide the > behaviour. In the future we can even consider support for walking > through the ctid chain across multiple relfilenodes. But till then, we > need to decide what default

Re: [HACKERS] Make subquery alias optional in FROM clause

2017-02-23 Thread Robert Haas
On Thu, Feb 23, 2017 at 7:50 PM, Tom Lane wrote: > Maybe the best answer is to not do it immediately when transforming the > subselect's RTE, but to go back after we've finished transforming the > entire FROM clause and add aliases to any RTEs that lack them. I think > probably at that point you

Re: [HACKERS] increasing the default WAL segment size

2017-02-23 Thread Beena Emerson
On Mon, Feb 20, 2017 at 2:47 PM, Beena Emerson wrote: > Hello, > > On Thu, Feb 16, 2017 at 3:37 PM, Kuntal Ghosh > wrote: > >> On Mon, Feb 6, 2017 at 11:09 PM, Beena Emerson >> wrote: >> > >> > Hello, >> > PFA the updated patches. >> I've started reviewing the patches. >> 01-add-XLogSegmentOffs

Re: [HACKERS] Poor memory context performance in large hash joins

2017-02-23 Thread Andres Freund
On 2017-02-24 01:59:01 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2017-02-23 17:28:26 -0500, Tom Lane wrote: > >> Maybe it's time to convert that to a doubly-linked list. > > > Yes, I do think so. Given that we only have that for full blocks, not > > for small chunks, the cost seems neg

Re: [HACKERS] Make subquery alias optional in FROM clause

2017-02-23 Thread Robert Haas
On Thu, Feb 23, 2017 at 4:08 PM, Pantelis Theodosiou wrote: > Question: Will the patch be removed if and when Oracle decides to be > compatible with the standard and forbids non-aliased derived tables? > > (I know it's a rather theoretical question. Unlikely that Oracle breaks > backwards compatib

Re: [HACKERS] Make subquery alias optional in FROM clause

2017-02-23 Thread Craig Ringer
On 23 February 2017 at 22:20, Tom Lane wrote: >> * Don't force/generate an alias at all. > >> I've no idea for this yet and Tom already was concerned what this might >> break. There are several places in the transform phase where the >> refnames are required (e.g. isLockedRefname()). > > Yeah. T

Re: [HACKERS] FYI: git worktrees as replacement for "rsync the CVSROOT"

2017-02-23 Thread Craig Ringer
On 16 January 2017 at 05:01, Jim Nasby wrote: > Not sure how many people still use [1], as referenced by our git wiki[2], > but it appears git worktrees are a viable replacement for that technique. In > short, if you're already in your checkout: > > git worktree add ../9.6 REL9_6_STABLE > > would

Re: [HACKERS] UPDATE of partition key

2017-02-23 Thread David G. Johnston
On Friday, February 24, 2017, Robert Haas wrote: > On Mon, Feb 20, 2017 at 2:58 PM, Amit Khandekar > wrote: > > I am inclined to at least have some option for the user to decide the > > behaviour. In the future we can even consider support for walking > > through the ctid chain across multiple r

<    1   2