Re: [HACKERS] [NOVICE] psql readline Tab insert tab

2015-06-02 Thread Albe Laurenz
Peter Eisentraut wrote: On 6/1/15 7:00 AM, Albe Laurenz wrote: Hans Ginzel wrote: how to make psql (readline) to insert tab when Tab is pressed? E.g. for pasting. I know, there is -n option. But then the history is not accessible. It could be done by adding the following lines to your

Re: [HACKERS] auto_explain sample rate

2015-06-02 Thread Craig Ringer
On 29 May 2015 at 11:35, Tom Lane t...@sss.pgh.pa.us wrote: Craig Ringer cr...@2ndquadrant.com writes: It's sometimes desirable to collect auto_explain data with ANALYZE in order to track down hard-to-reproduce issues, but the performance impacts can be pretty hefty on the DB. I'm

Re: [HACKERS] pg_xlog - pg_xjournal?

2015-06-02 Thread Joel Jacobson
On Tue, Jun 2, 2015 at 6:35 AM, Michael Nolan htf...@gmail.com wrote: Why not take a simpler approach and create a zero length file in directories that should not be fiddled with by non-experts using a file name something like DO.NOT.DELETE.THESE.FILES? Then the smart sysadmin will say but I

Re: [HACKERS] [GENERAL] psql weird behaviour with charset encodings

2015-06-02 Thread Michael Paquier
On Sun, May 24, 2015 at 2:43 AM, Noah Misch n...@leadboat.com wrote: On Sat, May 08, 2010 at 09:24:45PM -0400, Tom Lane wrote: hgonza...@gmail.com writes: http://sources.redhat.com/bugzilla/show_bug.cgi?id=649 The last explains why they do not consider it a bug: ISO C99 requires for %.*s

Re: [HACKERS] pg_xlog - pg_xjournal?

2015-06-02 Thread Guillaume Lelarge
Le 2 juin 2015 6:37 AM, Michael Nolan htf...@gmail.com a écrit : Why not take a simpler approach and create a zero length file in directories that should not be fiddled with by non-experts using a file name something like DO.NOT.DELETE.THESE.FILES? No, it won't prevent the incredibly stupid

Re: [HACKERS] checkpointer continuous flushing

2015-06-02 Thread Amit Langote
Hi Fabien, On 2015-06-01 PM 08:40, Fabien COELHO wrote: Turning checkpoint_flush_to_disk = on reduces significantly the number of late transactions. These late transactions are not uniformly distributed, but are rather clustered around times when pg is stalled, i.e. more or less

Re: [HACKERS] auto_explain sample rate

2015-06-02 Thread Pavel Stehule
2015-06-02 9:07 GMT+02:00 Craig Ringer cr...@2ndquadrant.com: On 29 May 2015 at 11:35, Tom Lane t...@sss.pgh.pa.us wrote: Craig Ringer cr...@2ndquadrant.com writes: It's sometimes desirable to collect auto_explain data with ANALYZE in order to track down hard-to-reproduce issues, but the

Re: [HACKERS] pg_xlog - pg_xjournal?

2015-06-02 Thread Mark Kirkwood
On 01/06/15 05:29, Joel Jacobson wrote: While anyone who is familiar with postgres would never do something as stupid as to delete pg_xlog, according to Google, there appears to be a fair amount of end-users out there having made the irrevocable mistake of deleting the precious directory, a

Re: [HACKERS] Re: [COMMITTERS] pgsql: Map basebackup tablespaces using a tablespace_map file

2015-06-02 Thread Amit Kapila
On Tue, Jun 2, 2015 at 10:26 PM, Andrew Dunstan and...@dunslane.net wrote: On 05/15/2015 02:21 AM, Amit Kapila wrote: Find the patch which gets rid of rmtree usage. I have made it as a separate function because the same code is used from create_tablespace_directories() as well. I thought

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-02 Thread Thomas Munro
On Tue, Jun 2, 2015 at 9:30 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: My guess is that the file existed, and perhaps had one or more pages, but the wanted page doesn't exist, so we tried to read but got 0 bytes back. read() returns 0 in this case but doesn't set errno. I didn't find

Re: [HACKERS] nested loop semijoin estimates

2015-06-02 Thread Tom Lane
Tomas Vondra tomas.von...@2ndquadrant.com writes: On 06/02/15 23:27, Tom Lane wrote: Do we have instructions around here anyplace on how to set up/use TPC-DS? I couldn't find anything about it on the wiki ... Not that I'm aware of, but it's not really all that difficult. [ instructions... ]

Re: [HACKERS] why does txid_current() assign new transaction-id?

2015-06-02 Thread Fujii Masao
On Wed, Jun 3, 2015 at 9:04 AM, Naoya Anzai nao-an...@xc.jp.nec.com wrote: diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 89a609f..6485b42 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -16233,7 +16233,7 @@ SELECT collation for ('foo' COLLATE

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-02 Thread Alvaro Herrera
Thomas Munro wrote: On Tue, Jun 2, 2015 at 9:30 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: My guess is that the file existed, and perhaps had one or more pages, but the wanted page doesn't exist, so we tried to read but got 0 bytes back. read() returns 0 in this case but doesn't

Re: [HACKERS] checkpointer continuous flushing

2015-06-02 Thread Amit Kapila
On Tue, Jun 2, 2015 at 6:45 PM, Fabien COELHO coe...@cri.ensmp.fr wrote: Hello Amit, [...] The objective is to help avoid PG stalling when fsyncing on checkpoints, and in general to get better latency-bound performance. Won't this lead to more-unsorted writes (random I/O) as the

Re: [HACKERS] checkpointer continuous flushing

2015-06-02 Thread Fabien COELHO
That might be the case in a database with a single small table; i.e. where all the writes go to a single file. But as soon as you have large tables (i.e. many segments) or multiple tables, a significant part of the writes issued independently from checkpointing will be outside the processing

Re: [HACKERS] checkpointer continuous flushing

2015-06-02 Thread Fabien COELHO
Hello Amit, Not that the GUC naming is the most pressing issue here, but do you think *_flush_on_write describes what the patch does? It is currently *_flush_to_disk. In Andres Freund version the name is sync_on_checkpoint_flush, but I did not found it very clear. Using *_flush_on_write

Re: [HACKERS] checkpointer continuous flushing

2015-06-02 Thread Andres Freund
Hi, It's nice to see the topic being picked up. If I see correctly you picked up the version without sorting durch checkpoints. I think that's not going to work - there'll be too many situations where the new behaviour will be detrimental. Did you consider combining both approaches? Greetings,

Re: [HACKERS] checkpointer continuous flushing

2015-06-02 Thread Amit Kapila
On Mon, Jun 1, 2015 at 5:10 PM, Fabien COELHO coe...@cri.ensmp.fr wrote: Hello pg-devs, This patch is a simplified and generalized version of Andres Freund's August 2014 patch for flushing while writing during checkpoints, with some documentation and configuration warnings added. For the

Re: [HACKERS] nested loop semijoin estimates

2015-06-02 Thread Tom Lane
Tomas Vondra tomas.von...@2ndquadrant.com writes: OK, so I did the testing today - with TPC-H and TPC-DS benchmarks. The results are good, IMHO. With TPC-H, I've used 1GB and 4GB datasets, and I've seen no plan changes at all. I don't plan to run the tests on larger data sets, I do expect

Re: [HACKERS] pg_xlog - pg_xjournal?

2015-06-02 Thread Abhijit Menon-Sen
At 2015-06-01 23:35:23 -0500, htf...@gmail.com wrote: No, it won't prevent the incredibly stupid from doing incredibly stupid things, nothing will. I hate to speechify, but I think we should try hard to avoid framing such questions in terms of incredibly stupid people and the things they might

Re: [HACKERS] checkpointer continuous flushing

2015-06-02 Thread Andres Freund
On 2015-06-02 15:42:14 +0200, Fabien COELHO wrote: This version seems already quite effective and very light. ISTM that adding a sort phase would mean reworking significantly how the checkpointer processes pages. Meh. The patch for that wasn't that big. Hmmm. I think it should be

[HACKERS] [PATCH] Add error handling to byteaout.

2015-06-02 Thread Andreas Seltenreich
Hi, when dealing with bytea values that are below the 1GB limit, but too large to be escaped, the error messages emitted are not very helpful for users. Especially if they appear in an unrelated context such as during pg_dump. I've attached a patch that adds ereport()ing that would have

Re: [HACKERS] pg_xlog - pg_xjournal?

2015-06-02 Thread Joshua D. Drake
On 06/01/2015 09:35 PM, Michael Nolan wrote: Why not take a simpler approach and create a zero length file in directories that should not be fiddled with by non-experts using a file name something like DO.NOT.DELETE.THESE.FILES? +1 No, it won't prevent the incredibly stupid from doing

Re: [HACKERS] pg_xlog - pg_xjournal?

2015-06-02 Thread Bruce Momjian
On Tue, Jun 2, 2015 at 07:33:19PM +0530, Abhijit Menon-Sen wrote: At 2015-06-01 23:35:23 -0500, htf...@gmail.com wrote: No, it won't prevent the incredibly stupid from doing incredibly stupid things, nothing will. I hate to speechify, but I think we should try hard to avoid framing

Re: [HACKERS] nested loop semijoin estimates

2015-06-02 Thread Tomas Vondra
On 06/02/15 01:47, Josh Berkus wrote: On 06/01/2015 03:22 PM, Tomas Vondra wrote: On 06/01/15 23:47, Josh Berkus wrote: On 06/01/2015 02:18 PM, Tom Lane wrote: Anybody else want to speak for or against back-patching the patch as posted? I intentionally didn't push it in before today's

Re: [HACKERS] pg_xlog - pg_xjournal?

2015-06-02 Thread Amit Langote
On Tue, Jun 2, 2015 at 11:03 PM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2015-06-01 23:35:23 -0500, htf...@gmail.com wrote: No, it won't prevent the incredibly stupid from doing incredibly stupid things, nothing will. I hate to speechify, but I think we should try hard to avoid

Re: [HACKERS] checkpointer continuous flushing

2015-06-02 Thread Andres Freund
On 2015-06-02 15:15:39 +0200, Fabien COELHO wrote: Won't this lead to more-unsorted writes (random I/O) as the FlushBuffer requests (by checkpointer or bgwriter) are not sorted as per files or order of blocks on disk? Yep, probably. Under moderate load this is not an issue. The io-scheduler

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-02 Thread Andres Freund
On 2015-06-01 14:22:32 -0400, Robert Haas wrote: On Mon, Jun 1, 2015 at 4:58 AM, Andres Freund and...@anarazel.de wrote: The lack of WAL logging actually has caused problems in the 9.3.3 (?) era, where we didn't do any truncation during recovery... Right, but now we're piggybacking on the

[HACKERS] Streaming replication for psycopg2

2015-06-02 Thread Shulgin, Oleksandr
Hello, I've submitted a patch to psycopg2 to support streaming replication protocol (COPY_BOTH): https://github.com/psycopg/psycopg2/pull/322 It would be great if more people had a chance to take a look and provide feedback about it. In particular, please see example usage at this github

Re: [HACKERS] checkpointer continuous flushing

2015-06-02 Thread Fabien COELHO
Hello Andres, If I see correctly you picked up the version without sorting durch checkpoints. I think that's not going to work - there'll be too many situations where the new behaviour will be detrimental. Did you consider combining both approaches? Ja, I thought that it was a more complex

Re: [HACKERS] checkpointer continuous flushing

2015-06-02 Thread Fabien COELHO
Hello Amit, [...] The objective is to help avoid PG stalling when fsyncing on checkpoints, and in general to get better latency-bound performance. Won't this lead to more-unsorted writes (random I/O) as the FlushBuffer requests (by checkpointer or bgwriter) are not sorted as per files or

Re: [HACKERS] pg_xlog - pg_xjournal?

2015-06-02 Thread Andrew Dunstan
On 06/02/2015 03:06 AM, Joel Jacobson wrote: On Tue, Jun 2, 2015 at 6:35 AM, Michael Nolan htf...@gmail.com wrote: Why not take a simpler approach and create a zero length file in directories that should not be fiddled with by non-experts using a file name something like

Re: [HACKERS] checkpointer continuous flushing

2015-06-02 Thread Fabien COELHO
Hello Andres, I would rather separate them, unless this is a blocker. I think it is a blocker. Hmmm. This is an argument... This version seems already quite effective and very light. ISTM that adding a sort phase would mean reworking significantly how the checkpointer processes pages.

[HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-02 Thread Robert Haas
On Tue, Jun 2, 2015 at 1:21 AM, Noah Misch n...@leadboat.com wrote: On Mon, Jun 01, 2015 at 02:06:05PM -0400, Robert Haas wrote: On Mon, Jun 1, 2015 at 12:46 AM, Noah Misch n...@leadboat.com wrote: On Fri, May 29, 2015 at 03:08:11PM -0400, Robert Haas wrote: SetMultiXactIdLimit() bracketed

Re: [HACKERS] checkpointer continuous flushing

2015-06-02 Thread Andres Freund
On 2015-06-02 17:01:50 +0200, Fabien COELHO wrote: The actual problem is sorting fsyncing in a way that deals efficiently with tablespaces, i.e. doesn't write to tablespaces one-by-one. Not impossible, but it requires some thought. Hmmm... I would have neglected this point in a first

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-02 Thread Andres Freund
On 2015-06-02 11:29:24 -0400, Robert Haas wrote: On Tue, Jun 2, 2015 at 8:56 AM, Andres Freund and...@anarazel.de wrote: But what *definitely* looks wrong to me is that a TruncateMultiXact() in this scenario now (since a couple weeks ago) does a SimpleLruReadPage_ReadOnly() in the members

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-02 Thread Robert Haas
On Tue, Jun 2, 2015 at 11:27 AM, Andres Freund and...@anarazel.de wrote: On 2015-06-02 11:16:22 -0400, Robert Haas wrote: I'm having trouble figuring out what to do about this. I mean, the essential principle of this patch is that if we can't count on relminmxid, datminmxid, or the control

Re: [HACKERS] [PATCH] Add error handling to byteaout.

2015-06-02 Thread Tom Lane
Andreas Seltenreich andreas.seltenre...@credativ.de writes: The scary one is due to an integer overflow the attached patch also fixes. s/int/Size/ doesn't fix anything on 32-bit machines. regards, tom lane -- Sent via pgsql-hackers mailing list

Re: [HACKERS] checkpointer continuous flushing

2015-06-02 Thread Fabien COELHO
Hmmm. I think it should be implemented as Tom suggested, that is per chunks of shared buffers, in order to avoid allocating a large memory. I don't necessarily agree. But that's really just a minor implementation detail. Probably. The actual problem is sorting fsyncing in a way that deals

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-02 Thread Andres Freund
On 2015-06-02 11:16:22 -0400, Robert Haas wrote: I'm having trouble figuring out what to do about this. I mean, the essential principle of this patch is that if we can't count on relminmxid, datminmxid, or the control file to be accurate, we can at least look at what is present on the disk.

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-02 Thread Robert Haas
On Tue, Jun 2, 2015 at 8:56 AM, Andres Freund and...@anarazel.de wrote: But what *definitely* looks wrong to me is that a TruncateMultiXact() in this scenario now (since a couple weeks ago) does a SimpleLruReadPage_ReadOnly() in the members slru via find_multixact_start(). That just won't work

[HACKERS] [PATCH] Fix documentation bug in how to calculate the quasi-unique pg_log session_id

2015-06-02 Thread Joel Jacobson
Fix documentation bug in how to calculate the quasi-unique pg_log session_id While the code is truncating the backend_start time, the query example in the documentation is rouding. Fix by doing the same thing in the documentation as in, i.e. truncating the backend_start. elog.c:

Re: [HACKERS] nested loop semijoin estimates

2015-06-02 Thread Tomas Vondra
On 06/02/15 16:37, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: OK, so I did the testing today - with TPC-H and TPC-DS benchmarks. The results are good, IMHO. I'm a bit disturbed by that, because AFAICS from the plans, these queries did not involve any semi or anti

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-02 Thread Robert Haas
On Tue, Jun 2, 2015 at 11:36 AM, Andres Freund and...@anarazel.de wrote: That would be a departure from the behavior of every existing release that includes this code based on, to my knowledge, zero trouble reports. On the other hand we're now at about bug #5 attributeable to the odd way

Re: [HACKERS] WIP: Enhanced ALTER OPERATOR

2015-06-02 Thread Robert Haas
On Fri, May 29, 2015 at 4:28 AM, Alexander Korotkov a.korot...@postgrespro.ru wrote: On Thu, May 28, 2015 at 6:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: Alexander Korotkov a.korot...@postgrespro.ru writes: Could we address both this problems by denying changing existing commutators and

[HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-02 Thread Noah Misch
On Tue, Jun 02, 2015 at 11:16:22AM -0400, Robert Haas wrote: On Tue, Jun 2, 2015 at 1:21 AM, Noah Misch n...@leadboat.com wrote: On Mon, Jun 01, 2015 at 02:06:05PM -0400, Robert Haas wrote: Granted. Would it be better to update both functions at the same time, and perhaps to make that a

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-02 Thread Andres Freund
On 2015-06-02 11:37:02 -0400, Robert Haas wrote: The exact circumstances under which we're willing to replace a relminmxid with a newly-computed one that differs are not altogether clear to me, but there's an if statement protecting that logic, so there are some circumstances in which we'll

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-02 Thread Robert Haas
On Tue, Jun 2, 2015 at 11:44 AM, Andres Freund and...@anarazel.de wrote: On 2015-06-02 11:37:02 -0400, Robert Haas wrote: The exact circumstances under which we're willing to replace a relminmxid with a newly-computed one that differs are not altogether clear to me, but there's an if statement

Re: [HACKERS] nested loop semijoin estimates

2015-06-02 Thread Tom Lane
Tomas Vondra tomas.von...@2ndquadrant.com writes: On 06/02/15 16:37, Tom Lane wrote: It's possible that the change was due to random variation in ANALYZE statistics, in which case it was just luck. I don't think so. I simply loaded the data, ran ANALYZE, and then simply started either

Re: [HACKERS] [PATCH] Add error handling to byteaout.

2015-06-02 Thread Andreas Seltenreich
Tom Lane t...@sss.pgh.pa.us writes: Andreas Seltenreich andreas.seltenre...@credativ.de writes: The scary one is due to an integer overflow the attached patch also fixes. s/int/Size/ doesn't fix anything on 32-bit machines. Well, it changes the signedness of the computation on 32-bit, and

Re: [HACKERS] Missing -i / --ignore-version in pg_dump help

2015-06-02 Thread Fujii Masao
On Fri, May 22, 2015 at 11:01 PM, Tom Lane t...@sss.pgh.pa.us wrote: Fujii Masao masao.fu...@gmail.com writes: On Fri, May 22, 2015 at 8:59 PM, Fabrízio de Royes Mello fabriziome...@gmail.com wrote: There are some reason to -i, --ignore-version option doesn't appear in pg_dump help? Because

Re: [HACKERS] RFC: Remove contrib entirely

2015-06-02 Thread David Fetter
On Mon, Jun 01, 2015 at 05:48:16PM -0300, Alvaro Herrera wrote: Peter Eisentraut wrote: But it is a valid point that we'd need to build up more extension API tests before we start cutting back significantly on the maybe-example-maybe-real extensions that we ship in contrib. We need to

Re: [HACKERS] Re: [COMMITTERS] pgsql: Map basebackup tablespaces using a tablespace_map file

2015-06-02 Thread Andrew Dunstan
On 05/15/2015 02:21 AM, Amit Kapila wrote: On Thu, May 14, 2015 at 10:29 PM, Andrew Dunstan and...@dunslane.net mailto:and...@dunslane.net wrote: On 05/14/2015 10:52 AM, Robert Haas wrote: On Thu, May 14, 2015 at 12:12 AM, Amit Kapila amit.kapil...@gmail.com