Re: [HACKERS] Turning off HOT/Cleanup sometimes

2014-09-12 Thread Michael Paquier
On Mon, Feb 3, 2014 at 3:42 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Jan 15, 2014 at 2:43 AM, Simon Riggs si...@2ndquadrant.com wrote: On 8 January 2014 08:33, Simon Riggs si...@2ndquadrant.com wrote: Patch attached, implemented to reduce writes by SELECTs only. This patch is

Re: [HACKERS] proposal (9.5) : psql unicode border line styles

2014-09-12 Thread Pavel Stehule
2014-09-12 5:14 GMT+02:00 Stephen Frost sfr...@snowman.net: Pavel, * Pavel Stehule (pavel.steh...@gmail.com) wrote: Any idea, tip how to it? Attached is what I had been thinking. Thoughts? yes, it is better. I didn't understand before. I though about it because it allows to design

Re: [HACKERS] Scaling shared buffer eviction

2014-09-12 Thread Amit Kapila
On Thu, Sep 11, 2014 at 4:31 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-09-10 12:17:34 +0530, Amit Kapila wrote: +++ b/src/backend/postmaster/bgreclaimer.c A fair number of comments in that file refer to bgwriter... Will fix. @@ -0,0 +1,302 @@

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2014-09-12 Thread Michael Paquier
On Fri, Sep 12, 2014 at 3:19 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Mon, Feb 3, 2014 at 3:42 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Jan 15, 2014 at 2:43 AM, Simon Riggs si...@2ndquadrant.com wrote: On 8 January 2014 08:33, Simon Riggs si...@2ndquadrant.com

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2014-09-12 Thread Albe Laurenz
Tom Lane wrote: Stephen Frost sfr...@snowman.net writes: I have to admit that, while I applaud the effort made to have this change only be to postgres_fdw, I'm not sure that having the update/delete happening during the Scan phase and then essentially no-op'ing the

Re: [HACKERS] inherit support for foreign tables

2014-09-12 Thread Etsuro Fujita
(2014/09/11 20:51), Heikki Linnakangas wrote: On 09/11/2014 02:30 PM, Etsuro Fujita wrote: So, should I split the patch into the two? Yeah, please do. OK, Will do. Thanks, Best regards, Etsuro Fujita -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] Incorrect initialization of sentPtr in walsender.c

2014-09-12 Thread Heikki Linnakangas
On 09/12/2014 03:17 AM, Michael Paquier wrote: On Fri, Sep 12, 2014 at 9:08 AM, Michael Paquier michael.paqu...@gmail.com wrote: In walsender.c, sentPtr is initialized as follows: static XLogRecPtr sentPtr = 0; Isn't that incorrect and shouldn't we use InvalidXLogRecPtr instead? Actually by

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2014-09-12 Thread Etsuro Fujita
(2014/09/12 16:03), Albe Laurenz wrote: Tom Lane wrote: Stephen Frost sfr...@snowman.net writes: I have to admit that, while I applaud the effort made to have this change only be to postgres_fdw, I'm not sure that having the update/delete happening during the Scan phase and then essentially

Re: [HACKERS] pg_basebackup vs. Windows and tablespaces

2014-09-12 Thread Dilip kumar
On 11 September 2014 10:21, Amit kapila Wrote, I don't think currently such a limitation is mentioned in docs, however I think we can update the docs at below locations: 1. In description of pg_start_backup in below page:

Re: [HACKERS] Suspicious check (src/backend/access/gin/gindatapage.c)

2014-09-12 Thread Heikki Linnakangas
On 09/12/2014 03:48 AM, Michael Paquier wrote: On Fri, Sep 12, 2014 at 7:35 AM, Gaetano Mendola mend...@gmail.com wrote: At line 650 I can read: if ((leaf-lsize - segsize) - (leaf-lsize - segsize) BLCKSZ / 4) break; I believe one of the two should be leaf-rsize Yes this

Re: [HACKERS] Suspicious check (src/backend/access/gin/gindatapage.c)

2014-09-12 Thread Heikki Linnakangas
On 09/12/2014 11:38 AM, Heikki Linnakangas wrote: Now that the logic is fixed, I hope we won't get complaints that the indexes are bigger, if you fill a table by appending to the end. I wonder if we should aim at an even more uneven split; the default fillfactor for B-trees is 90%, for example.

Re: [HACKERS] pg_basebackup vs. Windows and tablespaces

2014-09-12 Thread Amit Kapila
On Fri, Sep 12, 2014 at 1:50 PM, Dilip kumar dilip.ku...@huawei.com wrote: On 11 September 2014 10:21, Amit kapila Wrote, I don't think currently such a limitation is mentioned in docs, however I think we can update the docs at below locations: 1. In description of pg_start_backup in below

Re: [HACKERS] Stating the significance of Lehman Yao in the nbtree README

2014-09-12 Thread Heikki Linnakangas
On 09/11/2014 11:47 PM, Peter Geoghegan wrote: On Tue, Sep 9, 2014 at 12:01 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: + Although it could be argued that Lehman and Yao isn't followed to the + letter because single pages are read locked as the tree is descended, + this is at least

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-12 Thread David Rowley
On Fri, Sep 12, 2014 at 3:35 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Sep 11, 2014 at 7:14 AM, David Rowley dgrowle...@gmail.com wrote: 1. I don't think that I'm currently handling removing eclass members properly. So far the code just removes the Vars that belong to the

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-09-12 Thread Heikki Linnakangas
On 09/12/2014 12:46 AM, Peter Geoghegan wrote: On Thu, Sep 11, 2014 at 1:50 PM, Robert Haas robertmh...@gmail.com wrote: I think I said pretty clearly that it was. I agree that you did, but it wasn't clear exactly what factors you were asking me to simulate. All factors. Do you want me to

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-12 Thread David Rowley
On Fri, Sep 12, 2014 at 3:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Thu, Sep 11, 2014 at 7:14 AM, David Rowley dgrowle...@gmail.com wrote: 5. I've added a flag to pg_class called relhasfkey. Currently this gets set to true when a foreign key

Re: [HACKERS] Scaling shared buffer eviction

2014-09-12 Thread Ants Aasma
On Thu, Sep 11, 2014 at 4:22 PM, Andres Freund and...@2ndquadrant.com wrote: Hm. Perhaps we should do a bufHdr-refcount != zero check without locking here? The atomic op will transfer the cacheline exclusively to the reclaimer's CPU. Even though it very shortly afterwards will be touched

Re: [HACKERS] Scaling shared buffer eviction

2014-09-12 Thread Andres Freund
On 2014-09-12 12:38:48 +0300, Ants Aasma wrote: On Thu, Sep 11, 2014 at 4:22 PM, Andres Freund and...@2ndquadrant.com wrote: Hm. Perhaps we should do a bufHdr-refcount != zero check without locking here? The atomic op will transfer the cacheline exclusively to the reclaimer's CPU. Even

Re: [HACKERS] implement subject alternative names support for SSL connections

2014-09-12 Thread Heikki Linnakangas
On 09/11/2014 08:46 PM, Alexey Klyukin wrote: On Mon, Sep 8, 2014 at 8:04 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: 2. I still wonder if we should follow the RFC 6125 and not check the Common Name at all, if SANs are present. I don't really understand the point of that rule, and it

[HACKERS] END_OF_RECOVERY shutdowns and ResetUnloggedRelations()

2014-09-12 Thread Andres Freund
Hi, Abhijit and I investigated a customer problem which has showed that crash recovery + unlogged relations don't always work well together: A condensed version of how crash recovery works is: StartupXLOG() { ... if (ControlFile-state != DB_SHUTDOWNED) InRecovery = true; if

Re: [HACKERS] END_OF_RECOVERY shutdowns and ResetUnloggedRelations()

2014-09-12 Thread Andres Freund
Hi, On 2014-09-12 13:22:46 +0200, Andres Freund wrote: Because at least the former is something that obviously we can't (and don't want) to change in the back branches I think the solution for this particular problem is to simply move the ResetUnloggedRelations() call a couple lines up to

[HACKERS] expanded mode is still broken

2014-09-12 Thread Pavel Stehule
Hi I checked HEAD and border formatting is broken postgres=# \pset border 3 Border style (border) is 3. postgres=# \l List of databases [ RECORD 1 ]-+-- | Name | postgres | | Owner | postgres | | Encoding | UTF8

Re: [HACKERS] proposal (9.5) : psql unicode border line styles

2014-09-12 Thread Pavel Stehule
2014-09-12 8:19 GMT+02:00 Pavel Stehule pavel.steh...@gmail.com: 2014-09-12 5:14 GMT+02:00 Stephen Frost sfr...@snowman.net: Pavel, * Pavel Stehule (pavel.steh...@gmail.com) wrote: Any idea, tip how to it? Attached is what I had been thinking. Thoughts? yes, it is better. I didn't

Re: [HACKERS] pg_basebackup vs. Windows and tablespaces

2014-09-12 Thread Dilip kumar
On 12 September 2014 14:34, Amit Kapila Wrote Please find updated patch to include those documentation changes. Looks fine, Moved to Ready for committer. With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.comhttp://www.enterprisedb.com/

Re: [HACKERS] Incorrect initialization of sentPtr in walsender.c

2014-09-12 Thread Michael Paquier
On Fri, Sep 12, 2014 at 4:55 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I haven't looked at those places closely, but it seems possible that at least some of those variables are supposed to be initialized to a value smaller than any valid WAL position, rather than just Invalid. In

Re: [HACKERS] bad estimation together with large work_mem generates terrible slow hash joins

2014-09-12 Thread Robert Haas
On Thu, Sep 11, 2014 at 5:57 PM, Tomas Vondra t...@fuzzy.cz wrote: Attached is the patch split as suggested: (a) hashjoin-nbuckets-v14a-size.patch * NTUP_PER_BUCKET=1 * counting buckets towards work_mem * changes in ExecChooseHashTableSize (due to the other changes) OK, I'm

Re: [HACKERS] pg_basebackup vs. Windows and tablespaces

2014-09-12 Thread Amit Kapila
On Fri, Sep 12, 2014 at 5:07 PM, Dilip kumar dilip.ku...@huawei.com wrote: On 12 September 2014 14:34, Amit Kapila Wrote Please find updated patch to include those documentation changes. Looks fine, Moved to Ready for committer. Thanks a lot for the review. With Regards, Amit Kapila.

Re: [HACKERS] proposal (9.5) : psql unicode border line styles

2014-09-12 Thread Stephen Frost
* Pavel Stehule (pavel.steh...@gmail.com) wrote: I though about it, and we cannot to drop it now. These symbols are necessary, because we don't support line between rows. I am thinking so 05 patch should be final Ok. I'm going to play with it a bit more but barring issues, should get it

Re: [HACKERS] Patch to support SEMI and ANTI join removal

2014-09-12 Thread Tom Lane
David Rowley dgrowle...@gmail.com writes: On Fri, Sep 12, 2014 at 3:35 AM, Robert Haas robertmh...@gmail.com wrote: I haven't read the patch, but I think the question is why this needs to be different than what we do for left join removal. I discovered over here -

[HACKERS] jsonb contains behaviour weirdness

2014-09-12 Thread Alexander Korotkov
Hi! Let's consider some examples. # select '[1,2]'::jsonb @ '[1,2,2]'::jsonb; ?column? -- f (1 row) One may think it's because second jsonb array contain two 2. So, contains takes care about count of equal elements. # select '[1,1,2]'::jsonb @ '[1,2,2]'::jsonb; ?column? --

Re: [HACKERS] expanded mode is still broken

2014-09-12 Thread Stephen Frost
Pavel, * Pavel Stehule (pavel.steh...@gmail.com) wrote: I checked HEAD and border formatting is broken I agree that 'border 3' and the ascii / unicode linestyles generates an odd looking output, however... [...] Probably not all Sergey's fixes was applied, when I tested it (9.4 with Sergey'

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2014-09-12 Thread Alvaro Herrera
Michael Paquier wrote: On Fri, Sep 12, 2014 at 3:19 PM, Michael Paquier michael.paqu...@gmail.com wrote: On Mon, Feb 3, 2014 at 3:42 PM, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Jan 15, 2014 at 2:43 AM, Simon Riggs si...@2ndquadrant.com wrote: On 8 January 2014 08:33, Simon Riggs

Re: [HACKERS] expanded mode is still broken

2014-09-12 Thread Stephen Frost
Pavel, * Stephen Frost (sfr...@snowman.net) wrote: That said, it looks like it worked in 9.3. Hmm, actually, no, it didn't. sfrost@tamriel:~ psql --version psql (PostgreSQL) 9.3.5 sfrost@tamriel:~ psql -d postgres psql (9.3.5) Type help for help. postgres=# \pset expanded Expanded display is

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2014-09-12 Thread Simon Riggs
On 12 September 2014 14:54, Alvaro Herrera alvhe...@2ndquadrant.com wrote: My idea is that we would have a new executor flag, say EXEC_FLAG_READ_ONLY; we would set it on nodes that are known to be read-only, and reset it on those that aren't, such as LockRows and ModifyTable (obviously we

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2014-09-12 Thread Alvaro Herrera
Simon Riggs wrote: On 12 September 2014 14:54, Alvaro Herrera alvhe...@2ndquadrant.com wrote: My idea is that we would have a new executor flag, say EXEC_FLAG_READ_ONLY; we would set it on nodes that are known to be read-only, and reset it on those that aren't, such as LockRows and

Re: [HACKERS] implement subject alternative names support for SSL connections

2014-09-12 Thread Heikki Linnakangas
On 09/12/2014 01:30 PM, Heikki Linnakangas wrote: On 09/11/2014 08:46 PM, Alexey Klyukin wrote: On Mon, Sep 8, 2014 at 8:04 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: 2. I still wonder if we should follow the RFC 6125 and not check the Common Name at all, if SANs are present. I

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2014-09-12 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 12 September 2014 14:54, Alvaro Herrera alvhe...@2ndquadrant.com wrote: My idea is that we would have a new executor flag, say EXEC_FLAG_READ_ONLY; we would set it on nodes that are known to be read-only, and reset it on those that aren't, such as

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2014-09-12 Thread Tom Lane
I wrote: I think there's another way to think about it: what about saying that the query's target relation(s) are subject to pruning, while others are not? Then you do not need an executor flag, you just need to look at the estate-es_result_relations array (or maybe even only at

Re: [HACKERS] expanded mode is still broken

2014-09-12 Thread Stephen Frost
Pavel, All, * Stephen Frost (sfr...@snowman.net) wrote: * Stephen Frost (sfr...@snowman.net) wrote: That said, it looks like it worked in 9.3. Hmm, actually, no, it didn't. [...] Alright, attached is a patch which fixes it. Barring objections, I'll go ahead and back-patch this also, since

Re: [HACKERS] pgbench throttling latency limit

2014-09-12 Thread Heikki Linnakangas
On 09/11/2014 03:36 PM, Fabien COELHO wrote: Hello Heikki, Now that I've finished the detour and committed and backpatched the changes to the way latency is calculated, we can get back to this patch. It needs to be rebased. Before rebasing, I think that there are a few small problems with

Re: [HACKERS] ALTER SYSTEM RESET?

2014-09-12 Thread Fujii Masao
On Wed, Sep 10, 2014 at 9:06 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Sep 3, 2014 at 12:08 AM, Christoph Berg c...@df7cb.de wrote: Re: Vik Fearing 2014-09-02 5405d2d9.9050...@dalibo.com Uhm, are we agreed on the decision on not to backpatch this? I would think this should have

Re: [HACKERS] pgbench throttling latency limit

2014-09-12 Thread Heikki Linnakangas
On 09/11/2014 05:16 PM, Fabien COELHO wrote: How should skipped transactions should be taken into account in the log file output, with and without aggregation? I assume we'll want to have some trace of skipped transactions in the logs. The problem with this point is that how to report

Re: [HACKERS] Optimization for updating foreign tables in Postgres FDW

2014-09-12 Thread Tom Lane
Albe Laurenz laurenz.a...@wien.gv.at writes: Tom Lane wrote: I'm not sure offhand what the new plan tree ought to look like. We could just generate a ForeignScan node, but that seems like rather a misnomer. Is it worth inventing a new ForeignUpdate node type? Or maybe it'd still be

Re: [HACKERS] pgcrypto: PGP signatures

2014-09-12 Thread Alvaro Herrera
Marko Tiikkaja wrote: On 9/8/14 7:30 PM, Jeff Janes wrote: If i understand the sequence here: The current git HEAD is that pgp_pub_decrypt would throw an error if given a signed and encrypted message, and earlier version of your patch changed that to decrypt the message and ignore the

Re: [HACKERS] proposal (9.5) : psql unicode border line styles

2014-09-12 Thread Stephen Frost
* Stephen Frost (sfr...@snowman.net) wrote: * Pavel Stehule (pavel.steh...@gmail.com) wrote: I though about it, and we cannot to drop it now. These symbols are necessary, because we don't support line between rows. I am thinking so 05 patch should be final Ok. I'm going to play with

Re: [HACKERS] proposal (9.5) : psql unicode border line styles

2014-09-12 Thread Pavel Stehule
2014-09-12 18:09 GMT+02:00 Stephen Frost sfr...@snowman.net: * Stephen Frost (sfr...@snowman.net) wrote: * Pavel Stehule (pavel.steh...@gmail.com) wrote: I though about it, and we cannot to drop it now. These symbols are necessary, because we don't support line between rows. I am

Re: [HACKERS] expanded mode is still broken

2014-09-12 Thread Pavel Stehule
It works perfectly now Thank you Pavel 2014-09-12 16:37 GMT+02:00 Stephen Frost sfr...@snowman.net: Pavel, All, * Stephen Frost (sfr...@snowman.net) wrote: * Stephen Frost (sfr...@snowman.net) wrote: That said, it looks like it worked in 9.3. Hmm, actually, no, it didn't. [...]

Re: [HACKERS] bad estimation together with large work_mem generates terrible slow hash joins

2014-09-12 Thread Robert Haas
On Fri, Sep 12, 2014 at 8:28 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Sep 11, 2014 at 5:57 PM, Tomas Vondra t...@fuzzy.cz wrote: Attached is the patch split as suggested: (a) hashjoin-nbuckets-v14a-size.patch * NTUP_PER_BUCKET=1 * counting buckets towards work_mem *

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-12 Thread Josh Berkus
On 09/11/2014 06:56 PM, Arthur Silva wrote: In my testings with the github archive data the savings - performance-penalty was fine, but I'm not confident in those results since there were only 8 top level keys. Well, we did want to see that the patch doesn't create a regression with data

Re: [HACKERS] Stating the significance of Lehman Yao in the nbtree README

2014-09-12 Thread Peter Geoghegan
On Fri, Sep 12, 2014 at 2:15 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Amit: did you notice that paragraph in the README? If not, and now that you have read it, does that paragraph make things clear? If that's not enough, what do you think is missing? Amit raised the fact that LY

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-12 Thread Robert Haas
On Thu, Sep 11, 2014 at 9:01 PM, Josh Berkus j...@agliodbs.com wrote: So, I finally got time to test Tom's latest patch on this. TLDR: we want to go with Tom's latest patch and release beta3. Figures: So I tested HEAD against the latest lengths patch. Per Arthur Silva, I checked

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-12 Thread Robert Haas
On Fri, Sep 12, 2014 at 1:00 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Sep 11, 2014 at 9:01 PM, Josh Berkus j...@agliodbs.com wrote: So, I finally got time to test Tom's latest patch on this. TLDR: we want to go with Tom's latest patch and release beta3. Figures: So I tested

Re: [HACKERS] Stating the significance of Lehman Yao in the nbtree README

2014-09-12 Thread Robert Haas
On Fri, Sep 12, 2014 at 12:57 PM, Peter Geoghegan p...@heroku.com wrote: On Fri, Sep 12, 2014 at 2:15 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Amit: did you notice that paragraph in the README? If not, and now that you have read it, does that paragraph make things clear? If that's

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-12 Thread Josh Berkus
On 09/12/2014 10:00 AM, Robert Haas wrote: On Fri, Sep 12, 2014 at 1:00 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Sep 11, 2014 at 9:01 PM, Josh Berkus j...@agliodbs.com wrote: So, I finally got time to test Tom's latest patch on this. TLDR: we want to go with Tom's latest patch and

Re: [HACKERS] pgbench throttling latency limit

2014-09-12 Thread Fabien COELHO
(1) ISTM that the + 0.5 which remains in the PoissonRand computation comes from the previous integer approach and is not needed here. If I'm not mistaken the formula should be plain: -log(uniform) * center No. The +0.5 is to round the result to the nearest integer, instead of

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2014-09-12 Thread Simon Riggs
On 12 September 2014 15:30, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: I think there's another way to think about it: what about saying that the query's target relation(s) are subject to pruning, while others are not? Then you do not need an executor flag, you just need to look at the

Re: [HACKERS] Stating the significance of Lehman Yao in the nbtree README

2014-09-12 Thread Peter Geoghegan
On Fri, Sep 12, 2014 at 10:10 AM, Robert Haas robertmh...@gmail.com wrote: Gosh, I think you're making this way more complicated than it needs to be. My interpretation of the above statement was that they knew individual page reads and writes would need to be made atomic - probably using some

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-12 Thread Robert Haas
On Fri, Sep 12, 2014 at 1:11 PM, Josh Berkus j...@agliodbs.com wrote: On 09/12/2014 10:00 AM, Robert Haas wrote: On Fri, Sep 12, 2014 at 1:00 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Sep 11, 2014 at 9:01 PM, Josh Berkus j...@agliodbs.com wrote: So, I finally got time to test Tom's

Re: [HACKERS] Scaling shared buffer eviction

2014-09-12 Thread Gregory Smith
On 9/11/14, 7:01 AM, Andres Freund wrote: I'm not convinced that these changes can be made without also changing the bgwriter logic. Have you measured whether there are differences in how effective the bgwriter is? Not that it's very effective right now :) The current background writer tuning

Re: [HACKERS] jsonb contains behaviour weirdness

2014-09-12 Thread Peter Geoghegan
On Fri, Sep 12, 2014 at 6:40 AM, Alexander Korotkov aekorot...@gmail.com wrote: Even more weird :) Agreed. The reason why jsonb contains behaves so is check in the beginning of jsonb_contains. It makes fast check of jsonb type and elements count before calling JsonbDeepContains. if

Re: [HACKERS] [v9.5] Custom Plan API

2014-09-12 Thread Robert Haas
On Thu, Sep 11, 2014 at 8:40 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: On Thu, Sep 11, 2014 at 11:24 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Don't the changes to src/backend/optimizer/plan/createplan.c belong in patch #2? The borderline between #1 and #2 is little bit bogus.

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

2014-09-12 Thread Fujii Masao
On Wed, Sep 10, 2014 at 10:37 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao wrote: On Wed, Sep 10, 2014 at 12:15 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: PENDING_LIST_CLEANUP_SIZE and work_mem, for this setting. Wouldn't it be easy-to-use to have only one

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-12 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Fri, Sep 12, 2014 at 1:11 PM, Josh Berkus j...@agliodbs.com wrote: It's certainly possible that there is a test case for which Heikki's approach is superior, but if so we haven't seen it. And since it's approach is also more complicated, sticking

Re: [HACKERS] replication commands and log_statements

2014-09-12 Thread Fujii Masao
On Wed, Sep 10, 2014 at 7:39 PM, Fujii Masao masao.fu...@gmail.com wrote: Thanks for reviewing the patch! On Wed, Sep 10, 2014 at 4:57 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 08/28/2014 11:38 AM, Fujii Masao wrote: On Thu, Jun 19, 2014 at 5:29 PM, Ian Barwick

Re: [HACKERS] pgbench throttling latency limit

2014-09-12 Thread Fabien COELHO
The output would look something like this (modified from the manual's example by hand, so the numbers don't add up): 0 199 2241 0 1175850568 995598 1020 0 200 2465 0 1175850568 998079 1010 0 201 skipped 1175850569 608 3011 0 202 skipped 1175850569 608 2400 0 203 skipped 1175850569 608 1000 0

Re: [HACKERS] jsonb contains behaviour weirdness

2014-09-12 Thread Tom Lane
Peter Geoghegan p...@heroku.com writes: I think this is due to commit 364ddc. That removed the extra step that had arrays sorted (and then de-duped) ahead of time, which made arrays behave like objects at the top level. I think that this sort + de-dup step was mischaracterized as purely a

Re: [HACKERS] jsonb contains behaviour weirdness

2014-09-12 Thread Peter Geoghegan
On Fri, Sep 12, 2014 at 11:09 AM, Tom Lane t...@sss.pgh.pa.us wrote: I'm confused. Are you proposing to return to sort + de-dup of JSON arrays? Surely that is completely broken. Arrays are ordered. Sorry, my earlier remarks were premature. In fact, that alteration only applied to existence,

Re: [HACKERS] jsonb contains behaviour weirdness

2014-09-12 Thread Josh Berkus
On 09/12/2014 06:40 AM, Alexander Korotkov wrote: Hi! Let's consider some examples. # select '[1,2]'::jsonb @ '[1,2,2]'::jsonb; ?column? -- f (1 row) One may think it's because second jsonb array contain two 2. So, contains takes care about count of equal elements. JSONB

Re: [HACKERS] pgcrypto: PGP signatures

2014-09-12 Thread Abhijit Menon-Sen
(I have't read the patch, or even earlier correspondence in this thread, so I apologise for just jumping in.) At 2014-09-12 12:50:45 -0300, alvhe...@2ndquadrant.com wrote: +1 for ignoring sigs. If somebody want to check sigs, that's a separate step. For what it's worth, although it seems

Re: [HACKERS] pgbench throttling latency limit

2014-09-12 Thread Gregory Smith
On 9/10/14, 10:57 AM, Fabien COELHO wrote: Indeed. I think that people do not like it to change. I remember that I suggested to change timestamps to .yy instead of the unreadable yyy, and be told not to, because some people have tool which process the output so the format MUST NOT

Re: [HACKERS] Support for N synchronous standby servers

2014-09-12 Thread Robert Haas
On Fri, Sep 12, 2014 at 1:13 AM, Michael Paquier michael.paqu...@gmail.com wrote: OK. I see your point. Now, what about the following assumptions (somewhat restrictions to facilitate the user experience for setting syncrep and the parametrization of this feature): - Nodes are defined within

Re: [HACKERS] jsonb contains behaviour weirdness

2014-09-12 Thread Tom Lane
Peter Geoghegan p...@heroku.com writes: On Fri, Sep 12, 2014 at 11:09 AM, Tom Lane t...@sss.pgh.pa.us wrote: I'm confused. Are you proposing to return to sort + de-dup of JSON arrays? Surely that is completely broken. Arrays are ordered. Sorry, my earlier remarks were premature. In fact,

Re: [HACKERS] jsonb contains behaviour weirdness

2014-09-12 Thread Peter Geoghegan
On Fri, Sep 12, 2014 at 11:21 AM, Josh Berkus j...@agliodbs.com wrote: # select '[1,1,2]'::jsonb @ '[1,2,2]'::jsonb; ?column? -- t (1 row) But, it's not. Jsonb contains takes care only about length of array. OK, now, that's messed up. To be clear: I don't think that this

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-09-12 Thread Robert Haas
On Fri, Sep 12, 2014 at 5:28 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 09/12/2014 12:46 AM, Peter Geoghegan wrote: On Thu, Sep 11, 2014 at 1:50 PM, Robert Haas robertmh...@gmail.com wrote: I think I said pretty clearly that it was. I agree that you did, but it wasn't clear

Re: [HACKERS] Support for N synchronous standby servers

2014-09-12 Thread Gregory Smith
On 9/12/14, 2:28 PM, Robert Haas wrote: I hate to be the guy always suggesting a mini-language (cf. recent discussion of an expression syntax for pgbench), but we could do much more powerful and flexible things here if we had one. For example, suppose we let each element of

Re: [HACKERS] END_OF_RECOVERY shutdowns and ResetUnloggedRelations()

2014-09-12 Thread Robert Haas
On Fri, Sep 12, 2014 at 7:22 AM, Andres Freund and...@2ndquadrant.com wrote: What I like even less is that pg_control is actually marked as DB_SHUTDOWNED due to END_OF_RECOVERY. That's just plain wrong. Obviously the database was *NOT* shutdown peacefully. I don't see active bugs due it

Re: [HACKERS] Support for N synchronous standby servers

2014-09-12 Thread Robert Haas
On Fri, Sep 12, 2014 at 2:44 PM, Gregory Smith gregsmithpg...@gmail.com wrote: On 9/12/14, 2:28 PM, Robert Haas wrote: I hate to be the guy always suggesting a mini-language (cf. recent discussion of an expression syntax for pgbench), but we could do much more powerful and flexible things here

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-09-12 Thread Peter Geoghegan
On Fri, Sep 12, 2014 at 11:38 AM, Robert Haas robertmh...@gmail.com wrote: Based on discussion thus far it seems that there's a possibility that the trade-off may be different for short strings vs. long strings. If the string is small enough to fit in the L1 CPU cache, then it may be that

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-09-12 Thread Robert Haas
On Fri, Sep 12, 2014 at 2:58 PM, Peter Geoghegan p...@heroku.com wrote: On Fri, Sep 12, 2014 at 11:38 AM, Robert Haas robertmh...@gmail.com wrote: Based on discussion thus far it seems that there's a possibility that the trade-off may be different for short strings vs. long strings. If the

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2014-09-12 Thread Peter Geoghegan
On Fri, Sep 12, 2014 at 12:02 PM, Robert Haas robertmh...@gmail.com wrote: I think I've said a few times now that I really want to get this additional data before forming an opinion. As a certain Mr. Doyle writes, It is a capital mistake to theorize before one has data. Insensibly one begins

Re: [HACKERS] documentation update for doc/src/sgml/func.sgml

2014-09-12 Thread Andreas 'ads' Scherbaum
On 08/21/2014 12:35 PM, Fabien COELHO wrote: I do not understand why the last sentence in the first paragraph about bitwise ops is put there with rounding issues, which seem unrelated. It seems to me that it belongs to the second paragraph which is about bitwise operators. That's the part

Re: [HACKERS] jsonb contains behaviour weirdness

2014-09-12 Thread Josh Berkus
On 09/12/2014 11:38 AM, Peter Geoghegan wrote: To be clear: I don't think that this example is messed up (in isolation). I think it's the correct behavior. What I find objectionable is the inconsistency. I believe that this is Alexander's concern too. Alexander's first example exhibits broken

Re: [HACKERS] pgcrypto: PGP signatures

2014-09-12 Thread Jeff Janes
On Fri, Sep 12, 2014 at 8:50 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Marko Tiikkaja wrote: On 9/8/14 7:30 PM, Jeff Janes wrote: If i understand the sequence here: The current git HEAD is that pgp_pub_decrypt would throw an error if given a signed and encrypted message, and

Re: [HACKERS] pgbench throttling latency limit

2014-09-12 Thread Robert Haas
On Fri, Sep 12, 2014 at 2:27 PM, Gregory Smith gregsmithpg...@gmail.com wrote: If there's a good case that the whole format needs to be changed anyway, like adding a new field, then we might as well switch to fractional epoch timestamps too now though. When I added timestamps to the latency

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-12 Thread Heikki Linnakangas
On 09/02/2014 09:52 AM, Fujii Masao wrote: [RESULT] Throughput in the benchmark. MultipleSingle off2162.62164.5 on891.8895.6 pglz

Re: [HACKERS] bad estimation together with large work_mem generates terrible slow hash joins

2014-09-12 Thread Tomas Vondra
On 12.9.2014 18:49, Robert Haas wrote: On Fri, Sep 12, 2014 at 8:28 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Sep 11, 2014 at 5:57 PM, Tomas Vondra t...@fuzzy.cz wrote: Attached is the patch split as suggested: (a) hashjoin-nbuckets-v14a-size.patch * NTUP_PER_BUCKET=1 *

Re: [HACKERS] Stating the significance of Lehman Yao in the nbtree README

2014-09-12 Thread Kevin Grittner
Peter Geoghegan p...@heroku.com wrote: The introductory paragraph of LY says the following: Our solution compares favorably with earlier solutions in that the locking scheme is simpler (no read-locks are used) and only a (small) constant number of nodes are locked by any update process at

Re: [HACKERS] pgbench throttling latency limit

2014-09-12 Thread Heikki Linnakangas
On 09/12/2014 08:59 PM, Fabien COELHO wrote: The output would look something like this (modified from the manual's example by hand, so the numbers don't add up): 0 199 2241 0 1175850568 995598 1020 0 200 2465 0 1175850568 998079 1010 0 201 skipped 1175850569 608 3011 0 202 skipped 1175850569

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-12 Thread Abhijit Menon-Sen
At 2014-09-12 22:38:01 +0300, hlinnakan...@vmware.com wrote: We probably should consider switching to a faster CRC algorithm again, regardless of what we do with compression. As it happens, I'm already working on resurrecting a patch that Andres posted in 2010 to switch to zlib's faster CRC

CRC algorithm (was Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes)

2014-09-12 Thread Heikki Linnakangas
On 09/12/2014 10:54 PM, Abhijit Menon-Sen wrote: At 2014-09-12 22:38:01 +0300, hlinnakan...@vmware.com wrote: We probably should consider switching to a faster CRC algorithm again, regardless of what we do with compression. As it happens, I'm already working on resurrecting a patch that

Re: [HACKERS] Stating the significance of Lehman Yao in the nbtree README

2014-09-12 Thread Peter Geoghegan
On Fri, Sep 12, 2014 at 12:39 PM, Kevin Grittner kgri...@ymail.com wrote: It's been a while since I read that paper, but my recollection is that they assumed that each process or thread looking at a buffer would have its own private copy of that buffer, which it could be sure nobody was

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-12 Thread Ants Aasma
On Fri, Sep 12, 2014 at 10:38 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I don't mean that we should abandon this patch - compression makes the WAL smaller which has all kinds of other benefits, even if it makes the raw TPS throughput of the system worse. But I'm just saying that

Re: CRC algorithm (was Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes)

2014-09-12 Thread Andres Freund
On 2014-09-12 23:03:00 +0300, Heikki Linnakangas wrote: On 09/12/2014 10:54 PM, Abhijit Menon-Sen wrote: At 2014-09-12 22:38:01 +0300, hlinnakan...@vmware.com wrote: We probably should consider switching to a faster CRC algorithm again, regardless of what we do with compression. As it

Re: [HACKERS] bad estimation together with large work_mem generates terrible slow hash joins

2014-09-12 Thread Robert Haas
On Fri, Sep 12, 2014 at 3:39 PM, Tomas Vondra t...@fuzzy.cz wrote: On 12.9.2014 18:49, Robert Haas wrote: On Fri, Sep 12, 2014 at 8:28 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Sep 11, 2014 at 5:57 PM, Tomas Vondra t...@fuzzy.cz wrote: Attached is the patch split as suggested: (a)

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-12 Thread Andres Freund
On 2014-09-12 23:17:12 +0300, Ants Aasma wrote: On Fri, Sep 12, 2014 at 10:38 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I don't mean that we should abandon this patch - compression makes the WAL smaller which has all kinds of other benefits, even if it makes the raw TPS

Re: [HACKERS] jsonb contains behaviour weirdness

2014-09-12 Thread Alexander Korotkov
On Fri, Sep 12, 2014 at 10:38 PM, Peter Geoghegan p...@heroku.com wrote: On Fri, Sep 12, 2014 at 11:21 AM, Josh Berkus j...@agliodbs.com wrote: # select '[1,1,2]'::jsonb @ '[1,2,2]'::jsonb; ?column? -- t (1 row) But, it's not. Jsonb contains takes care only about length

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-12 Thread Andres Freund
On 2014-09-12 22:38:01 +0300, Heikki Linnakangas wrote: It's worth noting that there are faster CRC implementations out there than what we use. The Slicing-by-4 algorithm was discussed years ago, but was not deemed worth it back then IIRC because we typically calculate CRC over very small

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-12 Thread Heikki Linnakangas
On 09/12/2014 08:52 PM, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Sep 12, 2014 at 1:11 PM, Josh Berkus j...@agliodbs.com wrote: It's certainly possible that there is a test case for which Heikki's approach is superior, but if so we haven't seen it. And since it's

Re: [HACKERS] jsonb contains behaviour weirdness

2014-09-12 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: However, this better become a FAQ item, because it's not necessarily the behavior that folks used to JSON but not Postgres will expect. No, it's a bug, not a documentation deficiency. regards, tom lane -- Sent via pgsql-hackers

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-09-12 Thread k...@rice.edu
On Fri, Sep 12, 2014 at 11:17:12PM +0300, Ants Aasma wrote: On Fri, Sep 12, 2014 at 10:38 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: I don't mean that we should abandon this patch - compression makes the WAL smaller which has all kinds of other benefits, even if it makes the raw

  1   2   >