Re: [HACKERS] Minmax indexes

2013-09-26 Thread Erik Rijkers
On Thu, September 26, 2013 00:34, Erik Rijkers wrote: On Wed, September 25, 2013 22:34, Alvaro Herrera wrote: [minmax-5.patch] I have the impression it's not quite working correctly. The attached program returns different results for different values of enable_bitmapscan (consistently).

Re: [HACKERS] [PATCH] bitmap indexes

2013-09-26 Thread Antonin Houska
On 09/26/2013 12:20 AM, Alvaro Herrera wrote: This led me to research how these indexes are stored. I note that what we're doing here is to create another regular table and a btree index on top of it, and those are the ones that actually store the index data. This seems grotty and completely

Re: [HACKERS] pgbench progress report improvements - split 3 v2

2013-09-26 Thread Fabien COELHO
My feelings on the patch split haven't changed; your three bullet points call for four separate patches. Conflicting patches are bad, but dependent patches are okay; Indeed, this is the only solution if you do not want one patch. Note that it will not possible to backtrack one of the patch

[HACKERS] pgbench filler columns

2013-09-26 Thread Pavan Deolasee
While looking at the compressibility of WAL files generated by pgbench, which is close to 90%, I first thought its because of the filler column in the accounts table. But a comment in pgbench.c says: /* * Note: TPC-B requires at least 100 bytes per row, and the filler * fields in

Re: [HACKERS] [PATCH] bitmap indexes

2013-09-26 Thread Abhijit Menon-Sen
At 2013-09-25 19:20:17 -0300, alvhe...@2ndquadrant.com wrote: Here are some quick items while skimming this patch. Great, that gives me plenty to work on. At this point, I think it's appropriate to mark this patch as returned with feedback (which I will do), since the changes needed seem

Re: [HACKERS] SSI freezing bug

2013-09-26 Thread Heikki Linnakangas
On 23.09.2013 01:07, Hannu Krosing wrote: On 09/20/2013 12:55 PM, Heikki Linnakangas wrote: Hi, Prompted by Andres Freund's comments on my Freezing without Write I/O patch, I realized that there's there's an existing bug in the way predicate locking handles freezing (or rather, it doesn't

Re: [HACKERS] pgbench filler columns

2013-09-26 Thread Pavan Deolasee
On Thu, Sep 26, 2013 at 2:05 PM, Pavan Deolasee pavan.deola...@gmail.comwrote: While looking at the compressibility of WAL files generated by pgbench, which is close to 90%, I first thought its because of the filler column in the accounts table. But a comment in pgbench.c says: /*

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Andres Freund
On 2013-09-26 12:13:30 +0900, Michael Paquier wrote: 2) I don't think the drop algorithm used now is correct. Your index_concurrent_set_dead() sets both indisvalid = false and indislive = false at the same time. It does so after doing a WaitForVirtualLocks() - but that's not sufficient.

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Michael Paquier
On Thu, Sep 26, 2013 at 7:34 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-09-26 12:13:30 +0900, Michael Paquier wrote: 2) I don't think the drop algorithm used now is correct. Your index_concurrent_set_dead() sets both indisvalid = false and indislive = false at the same time. It

Re: [HACKERS] pgbench - exclude pthread_create() from connection start timing

2013-09-26 Thread Fabien COELHO
pgbench changes, when adding the throttling stuff. Having the start time taken when the thread really starts is just sanity, and I needed that just to rule out that it was the source of the strange measures. I don't get it; why is taking the time just after pthread_create() more sane than

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Andres Freund
On 2013-09-26 20:40:40 +0900, Michael Paquier wrote: On Thu, Sep 26, 2013 at 7:34 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-09-26 12:13:30 +0900, Michael Paquier wrote: 2) I don't think the drop algorithm used now is correct. Your index_concurrent_set_dead() sets both

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Bruce Momjian
On Wed, Sep 25, 2013 at 08:48:11PM -0700, Peter Geoghegan wrote: On Wed, Sep 25, 2013 at 8:19 PM, Bruce Momjian br...@momjian.us wrote: This thread had a lot of discussion about bloating. I wonder, does the code check to see if there is a matching row _before_ adding any data? That's

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Michael Paquier
On Thu, Sep 26, 2013 at 8:43 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-09-26 20:40:40 +0900, Michael Paquier wrote: On Thu, Sep 26, 2013 at 7:34 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-09-26 12:13:30 +0900, Michael Paquier wrote: 2) I don't think the drop

Re: [HACKERS] Patch for fail-back without fresh backup

2013-09-26 Thread Pavan Deolasee
On Thu, Sep 19, 2013 at 4:02 PM, Fujii Masao masao.fu...@gmail.com wrote: Hmm... when synchronous_transfer is set to data_flush, IMO the intuitive behaviors are (1) synchronous_commit = on A data flush should wait for the corresponding WAL to be flushed in the standby (2)

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Andres Freund
On 2013-09-26 20:47:33 +0900, Michael Paquier wrote: On Thu, Sep 26, 2013 at 8:43 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-09-26 20:40:40 +0900, Michael Paquier wrote: On Thu, Sep 26, 2013 at 7:34 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-09-26 12:13:30

[HACKERS] setting separate values of replication parameters to each standby to provide more granularity

2013-09-26 Thread Samrat Revagade
Hi, How about providing more granularity to replication, by setting separate values of replication parameters to each standby for example: standby1.wal_sender_timeout= 50s standby2.wal_sender_timeout= 40s The idea is to allow configuration of standby servers such that they have there own set of

Re: [HACKERS] Patch for fast gin cache performance improvement

2013-09-26 Thread Etsuro Fujita
Hi Ian, This patch contains a performance improvement for the fast gin cache. As you may know, the performance of the fast gin cache decreases with its size. Currently, the size of the fast gin cache is tied to work_mem. The size of work_mem can often be quite high. The large size of work_mem

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Bruce Momjian
On Thu, Sep 26, 2013 at 07:43:15AM -0400, Bruce Momjian wrote: On Wed, Sep 25, 2013 at 08:48:11PM -0700, Peter Geoghegan wrote: On Wed, Sep 25, 2013 at 8:19 PM, Bruce Momjian br...@momjian.us wrote: This thread had a lot of discussion about bloating. I wonder, does the code check to see

Re: [HACKERS] dynamic shared memory

2013-09-26 Thread Noah Misch
On Tue, Sep 24, 2013 at 12:19:51PM -0400, Robert Haas wrote: On Fri, Sep 20, 2013 at 7:44 AM, Andres Freund and...@2ndquadrant.com wrote: Actually, I was wondering if we ought to have a directory under pgdata whose explicit charter it was to contain files that shouldn't be copied as part of

Re: [HACKERS] pgbench filler columns

2013-09-26 Thread Noah Misch
On Thu, Sep 26, 2013 at 03:23:30PM +0530, Pavan Deolasee wrote: On Thu, Sep 26, 2013 at 2:05 PM, Pavan Deolasee pavan.deola...@gmail.comwrote: While looking at the compressibility of WAL files generated by pgbench, which is close to 90%, I first thought its because of the filler column

Re: [HACKERS] record identical operator

2013-09-26 Thread Robert Haas
On Tue, Sep 24, 2013 at 3:22 PM, Stephen Frost sfr...@snowman.net wrote: * Robert Haas (robertmh...@gmail.com) wrote: Now I admit that's an arguable point. We could certainly define an intermediate notion of equality that is more equal than whatever = does, but not as equal as exact binary

Re: [HACKERS] [PATCH] bitmap indexes

2013-09-26 Thread Jeff Janes
On Wed, Sep 25, 2013 at 3:20 PM, Alvaro Herrera alvhe...@2ndquadrant.comwrote: Hi, Here are some quick items while skimming this patch. I am looking at commit 6448de29d from your github repo, branch bmi. What's with the pg_bitmapindex stuff in pg_namespace.h? It doesn't seem to be used

Re: [HACKERS] FW: REVIEW: Allow formatting in log_line_prefix

2013-09-26 Thread Robert Haas
On Wed, Sep 25, 2013 at 4:46 AM, David Rowley dgrowle...@gmail.com wrote: Ok, I think I've managed to narrow the performance gap to just about nothing but noise, though to do this the code is now a bit bigger. I've added a series of tests to see if the padding is 0 and if it's not then I'm

Re: [HACKERS] Minmax indexes

2013-09-26 Thread Robert Haas
On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Here's an updated version of this patch, with fixes to all the bugs reported so far. Thanks to Thom Brown, Jaime Casanova, Erik Rijkers and Amit Kapila for the reports. I'm not very happy with the use of a

[HACKERS] Extra functionality to createuser

2013-09-26 Thread Christopher Browne
Sitting on my todo list for a while has been to consider the idea of adding a bit of additional functionality to createuser. One of the functions of CREATE ROLE is to associate the role with other roles, thus... create role my_new_user nosuperuser nocreatedb login IN ROLE

Re: [HACKERS] backup.sgml-cmd-v003.patch

2013-09-26 Thread Ivan Lezhnjov IV
On Sep 3, 2013, at 6:56 AM, Karl O. Pinc k...@meme.com wrote: On 07/31/2013 12:08:12 PM, Ivan Lezhnjov IV wrote: Patch filename: backup.sgml-cmd-v003.patch The third version of this patch takes into consideration feedback received after original submission (it can be read starting from

Re: [HACKERS] Minmax indexes

2013-09-26 Thread Alvaro Herrera
Robert Haas escribió: On Wed, Sep 25, 2013 at 4:34 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Here's an updated version of this patch, with fixes to all the bugs reported so far. Thanks to Thom Brown, Jaime Casanova, Erik Rijkers and Amit Kapila for the reports. I'm not very

Re: [HACKERS] Minmax indexes

2013-09-26 Thread Alvaro Herrera
Erik Rijkers wrote: I have the impression it's not quite working correctly. The attached program returns different results for different values of enable_bitmapscan (consistently). Clearly there's some bug somewhere. I'll investigate it more. ( Btw, I had to make the

Re: [HACKERS] dynamic shared memory

2013-09-26 Thread Jim Nasby
On 9/26/13 8:27 AM, Noah Misch wrote: On Tue, Sep 24, 2013 at 12:19:51PM -0400, Robert Haas wrote: On Fri, Sep 20, 2013 at 7:44 AM, Andres Freundand...@2ndquadrant.com wrote: Actually, I was wondering if we ought to have a directory under pgdata whose explicit charter it was to contain

Re: [HACKERS] logical changeset generation v6

2013-09-26 Thread Steve Singer
On 09/25/2013 01:20 PM, Steve Singer wrote: On 09/25/2013 11:08 AM, Andres Freund wrote: On 2013-09-25 11:01:44 -0400, Steve Singer wrote: On 09/17/2013 10:31 AM, Andres Freund wrote: This patch set now fails to apply because of the commit Rename various freeze multixact variables. And I am

Re: [HACKERS] pgbench progress report improvements - split 3 v2 - A

2013-09-26 Thread Fabien COELHO
Patch (4): Redefine latency as reported by pgbench and report lag more. Here is a first partial patch, which focusses on measuring latency and reporting the measure under --progress. ** Improve pgbench measurements progress report Measure transaction latency instead under --rate and

Re: [HACKERS] Minmax indexes

2013-09-26 Thread Jim Nasby
On 9/26/13 12:00 PM, Robert Haas wrote: More generally, I fear we really opened a bag of worms with this relation fork stuff. Every time I turn around I run into a problem that could be solved by adding another relation fork. I'm not terribly sure that it was a good idea to go that way to

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Peter Geoghegan
On Thu, Sep 26, 2013 at 4:43 AM, Bruce Momjian br...@momjian.us wrote: So, I guess my question is if we are only bloating on a contended operation, do we expect that to happen so much that bloat is a problem? Maybe I could have done a better job of explaining the nature of my concerns around

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Robert Haas
On Tue, Sep 24, 2013 at 10:15 PM, Peter Geoghegan p...@heroku.com wrote: Well, I think we can rule out value locks that are held for the duration of a transaction right away. That's just not going to fly. I think I agree with that. I don't think I remember hearing that proposed. If we're

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Robert Haas
On Thu, Sep 26, 2013 at 3:07 PM, Peter Geoghegan p...@heroku.com wrote: When you consider that the feature will frequently be used with the assumption that updating is a much more likely outcome, it becomes clear that we need to be careful about this sort of interplay. I think one thing that's

Re: [HACKERS] Minmax indexes

2013-09-26 Thread Robert Haas
On Thu, Sep 26, 2013 at 2:58 PM, Jim Nasby j...@nasby.net wrote: Why would we add additional code complexity when forks do the trick? That seems like a step backwards, not forward. Well, they sorta do the trick, but see e.g. commit ece01aae479227d9836294b287d872c5a6146a11. I doubt that's the

Re: [HACKERS] FW: REVIEW: Allow formatting in log_line_prefix

2013-09-26 Thread David Rowley
On Fri, Sep 27, 2013 at 4:44 AM, Robert Haas robertmh...@gmail.com wrote: On Wed, Sep 25, 2013 at 4:46 AM, David Rowley dgrowle...@gmail.com wrote: Ok, I think I've managed to narrow the performance gap to just about nothing but noise, though to do this the code is now a bit bigger. I've

Re: [HACKERS] dynamic shared memory

2013-09-26 Thread Robert Haas
On Thu, Sep 26, 2013 at 2:45 PM, Jim Nasby j...@nasby.net wrote: On 9/26/13 8:27 AM, Noah Misch wrote: On Tue, Sep 24, 2013 at 12:19:51PM -0400, Robert Haas wrote: On Fri, Sep 20, 2013 at 7:44 AM, Andres Freundand...@2ndquadrant.com wrote: Actually, I was wondering if we ought to have

Re: [HACKERS] Re: Request for Patch Feedback: Lag Lead Window Functions Can Ignore Nulls

2013-09-26 Thread Alvaro Herrera
I gave this a quick look. It took me a while to figure out how to apply it -- turns out you used the ignore whitespace option to diff, so the only way to apply it was with patch -p1 --ignore-whitespace. Please don't do that. I ran pgindent over the patched code and there were a number of

Re: [HACKERS] gaussian distribution pgbench

2013-09-26 Thread Peter Eisentraut
On 9/20/13 2:42 AM, KONDO Mitsumasa wrote: I create gaussinan distribution pgbench patch that can access records with gaussian frequency. And I submit this commit fest. This patch no longer applies. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Michael Paquier
On Thu, Sep 26, 2013 at 8:56 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-09-26 20:47:33 +0900, Michael Paquier wrote: On Thu, Sep 26, 2013 at 8:43 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-09-26 20:40:40 +0900, Michael Paquier wrote: On Thu, Sep 26, 2013 at 7:34

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-09-26 Thread Andres Freund
On 2013-09-27 05:41:26 +0900, Michael Paquier wrote: In this case, doing a call to WaitForOldSnapshots after the swap phase is enough. It was included in past versions of the patch but removed in the last 2 versions. I don't think it is. I really, really suggest following the protocol used by

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Bruce Momjian
On Thu, Sep 26, 2013 at 03:33:34PM -0400, Robert Haas wrote: On Thu, Sep 26, 2013 at 3:07 PM, Peter Geoghegan p...@heroku.com wrote: When you consider that the feature will frequently be used with the assumption that updating is a much more likely outcome, it becomes clear that we need to

Re: [HACKERS] FW: REVIEW: Allow formatting in log_line_prefix

2013-09-26 Thread Robert Haas
On Thu, Sep 26, 2013 at 3:55 PM, David Rowley dgrowle...@gmail.com wrote: I think I must have forgot to save it before I emailed it... Here's the correct version. Ah ha. Looks better. I'm working on getting this committed now. Aside from some stylistic things, I've found one serious

Re: [HACKERS] record identical operator - Review

2013-09-26 Thread Bruce Momjian
On Thu, Sep 19, 2013 at 06:31:38PM -0400, Steve Singer wrote: On 09/12/2013 06:27 PM, Kevin Grittner wrote: Attached is a patch for a bit of infrastructure I believe to be necessary for correct behavior of REFRESH MATERIALIZED VIEW CONCURRENTLY as well as incremental maintenance of matviews.

Re: [HACKERS] [PATCH] pg_upgrade: Split off pg_fatal() from pg_log()

2013-09-26 Thread Bruce Momjian
On Thu, Sep 12, 2013 at 10:50:42PM -0400, Peter Eisentraut wrote: The experiences with elog() and ereport() have shown that having one function that can return or not depending on some log level parameter isn't a good idea when you want to communicate well with the compiler. In pg_upgrade,

Re: [HACKERS] GIN improvements part 1: additional information

2013-09-26 Thread Alexander Korotkov
On Wed, Sep 25, 2013 at 5:22 PM, Peter Eisentraut pete...@gmx.net wrote: On 9/23/13 5:36 PM, Alexander Korotkov wrote: In the attached version of patch double finding of ItemPointer during insert is avoided. Overhead becomes lower as expected. Fails cpluspluscheck:

Re: [HACKERS] FW: REVIEW: Allow formatting in log_line_prefix

2013-09-26 Thread Robert Haas
On Thu, Sep 26, 2013 at 5:18 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Sep 26, 2013 at 3:55 PM, David Rowley dgrowle...@gmail.com wrote: I think I must have forgot to save it before I emailed it... Here's the correct version. Ah ha. Looks better. I'm working on getting this

[HACKERS] Wait free LW_SHARED acquisition

2013-09-26 Thread Andres Freund
Hello, We have had several customers running postgres on bigger machines report problems on busy systems. Most recently one where a fully cached workload completely stalled in s_lock()s due to the *shared* lwlock acquisition in BufferAlloc() around the buffer partition lock. Increasing the

Re: [HACKERS] Wait free LW_SHARED acquisition

2013-09-26 Thread Peter Geoghegan
On Thu, Sep 26, 2013 at 3:55 PM, Andres Freund and...@2ndquadrant.com wrote: We have had several customers running postgres on bigger machines report problems on busy systems. Most recently one where a fully cached workload completely stalled in s_lock()s due to the *shared* lwlock acquisition

Re: [HACKERS] Wait free LW_SHARED acquisition

2013-09-26 Thread Andres Freund
On 2013-09-26 16:56:30 -0700, Peter Geoghegan wrote: On Thu, Sep 26, 2013 at 3:55 PM, Andres Freund and...@2ndquadrant.com wrote: We have had several customers running postgres on bigger machines report problems on busy systems. Most recently one where a fully cached workload completely

Re: [HACKERS] record identical operator - Review

2013-09-26 Thread Bruce Momjian
On Thu, Sep 26, 2013 at 05:50:08PM -0400, Bruce Momjian wrote: I think we need to see a patch from Kevin that shows all the row comparisons documented and we can see the impact of the user-visibile part. One interesting approach would be to only allow the operator to be called from SPI

Re: [HACKERS] [PATCH] bitmap indexes

2013-09-26 Thread Abhijit Menon-Sen
At 2013-09-26 08:39:05 -0700, jeff.ja...@gmail.com wrote: I don't know about grottiness, but it certainly seems like it would deadlock like crazy. Hi Jeff. I don't understand the deadlock scenario you're thinking of. Could you explain, please? -- Abhijit -- Sent via pgsql-hackers mailing

Re: [HACKERS] backup.sgml-cmd-v003.patch

2013-09-26 Thread Karl O. Pinc
On 09/26/2013 12:15:25 PM, Ivan Lezhnjov IV wrote: On Sep 3, 2013, at 6:56 AM, Karl O. Pinc k...@meme.com wrote: On 07/31/2013 12:08:12 PM, Ivan Lezhnjov IV wrote: Patch filename: backup.sgml-cmd-v003.patch The third version of this patch takes into consideration feedback

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Peter Geoghegan
On Thu, Sep 26, 2013 at 12:15 PM, Robert Haas robertmh...@gmail.com wrote: Well, I think we can rule out value locks that are held for the duration of a transaction right away. That's just not going to fly. I think I agree with that. I don't think I remember hearing that proposed. I think I

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-09-26 Thread Peter Geoghegan
On Thu, Sep 26, 2013 at 12:33 PM, Robert Haas robertmh...@gmail.com wrote: I think one thing that's pretty clear at this point is that almost any version of this feature could be optimized for either the insert case or the update case. For example, my proposal could be modified to search for