Re: [HACKERS] recovery_target_time and standby_mode

2014-11-07 Thread Josh Berkus
on fails with "end of wal reached on timeline 1 320/478ff780; new timeline 2 forked timeline 1 before current recovery point 320/47e0". In order for this to work, the archive would need to stop before recovery_target_time. On 11/07/2014 12:07 PM, Robert Haas wrote:> On Fri

Re: [HACKERS] recovery_target_time and standby_mode

2014-11-07 Thread Josh Berkus
r our current behavior is correct or not. For my part, I would like to have a different interacton, but I think that's a future feature rather than a bug, as long as we do the stuff in the Yes column. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers ma

Re: [HACKERS] recovery_target_time and standby_mode

2014-11-07 Thread Josh Berkus
On 11/07/2014 08:12 AM, Robert Haas wrote: > On Wed, Nov 5, 2014 at 9:15 PM, Josh Berkus wrote: >> What I'm pointing out is that you can't actually do that. You think you >> can, but you can't. > > I do think that. You haven't explained why I'm wr

Re: [HACKERS] recovery_target_time and standby_mode

2014-11-05 Thread Josh Berkus
On 11/05/2014 05:41 PM, Michael Paquier wrote: > On Thu, Nov 6, 2014 at 10:00 AM, Greg Stark wrote: >> On Thu, Nov 6, 2014 at 12:32 AM, Josh Berkus wrote: >>> When the recovery_target_time is reached, switch to streaming >>> replication and stay a standby. >&

Re: [HACKERS] recovery_target_time and standby_mode

2014-11-05 Thread Josh Berkus
On 11/05/2014 05:00 PM, Greg Stark wrote: > On Thu, Nov 6, 2014 at 12:32 AM, Josh Berkus wrote: >> When the recovery_target_time is reached, switch to streaming >> replication and stay a standby. > > Then shouldn't he just not specify a recovert_target at all? That

[HACKERS] recovery_target_time and standby_mode

2014-11-05 Thread Josh Berkus
switch to streaming replication and stay a standby. Note that there is a workaround for what the user wants to do. I'm just trying to clarify what our desired behavior is. From there we can either work on patches or on doc fixes. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com

Re: [HACKERS] Amazon Redshift

2014-11-05 Thread Josh Berkus
On 11/05/2014 02:36 PM, philip taylor wrote: > Ok, this is a summary of what they have that we don't (of course, I could > have missed something): > I can't see any functions on that list I'd want. For example, DATEADD is there just to be compatible with MSSQL. It&#x

Re: [HACKERS] pg_multixact not getting truncated

2014-11-05 Thread Josh Berkus
On 11/05/2014 10:40 AM, Jim Nasby wrote: > On 11/3/14, 7:40 PM, Josh Berkus wrote: >> On 11/03/2014 05:24 PM, Josh Berkus wrote: >>> BTW, the reason I started poking into this was a report from a user that >>> they have a pg_multixact directory which is 21GB in size, an

Re: [HACKERS] to_char_at_timezone()?

2014-11-05 Thread Josh Berkus
it would make sense to do this for "timestamp", or > whether there's even a clear intuitive behaviour there. Why wouldn't we just add the timezone as an additional parameter? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Let's drop two obsolete features which are bear-traps for novices

2014-11-04 Thread Josh Berkus
create versions that don't have operator conflicts. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Let's drop two obsolete features which are bear-traps for novices

2014-11-03 Thread Josh Berkus
to xml2 again? FWIW, I'd be fine with moving ISN, intarray and intagg to PGXN. In fact, I think it would be better for them to be there. And they're not core types, so there shouldn't be an issue with that. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent v

Re: [HACKERS] pg_multixact not getting truncated

2014-11-03 Thread Josh Berkus
On 11/03/2014 05:24 PM, Josh Berkus wrote: > BTW, the reason I started poking into this was a report from a user that > they have a pg_multixact directory which is 21GB in size, and is 2X the > size of the database. > > Here's XID data: > > Latest checkpoint's

Re: [HACKERS] pg_multixact not getting truncated

2014-11-03 Thread Josh Berkus
On 11/03/2014 05:06 PM, Alvaro Herrera wrote: > Josh Berkus wrote: >> Hackers, >> >> I'm looking at a couple of high-transaction-rate and high-FK-conflict >> rate servers where pg_multixact has grown to be more than 1GB in size. >> One such server doesn&#x

[HACKERS] pg_multixact not getting truncated

2014-11-03 Thread Josh Berkus
FREEZEing the oldest databases did not cause the pg_multixact dir to get smaller --- it may have even caused it to get larger. Why would pg_multixact not be truncating? Does it never truncate files with aborted multixacts in them? Might we have another multixact bug? -- Josh Berkus PostgreS

Re: [HACKERS] Let's drop two obsolete features which are bear-traps for novices

2014-11-01 Thread Josh Berkus
All, While there's argument about hash indexes, it looks like nobody minds if the MONEY type goes bye-bye. So, time for a patch ... -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Let's drop two obsolete features which are bear-traps for novices

2014-11-01 Thread Josh Berkus
TABLE > regression=# create index on foo using hash (f1); > WARNING: hash indexes are not WAL-logged and their use is discouraged > CREATE INDEX Yes, and I'm arguing that is the wrong decision. If hash indexes are "discouraged", then they shouldn't be in core in

[HACKERS] Let's drop two obsolete features which are bear-traps for novices

2014-10-31 Thread Josh Berkus
ust about to fix WAL-logging on hash indexes, or add casts to the money type. But if that hasn't happened in the last 5 years, it's not going to happen. We'd be doing our users a huge favor by just removing them in 9.5. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com

Re: [HACKERS] Allow peer/ident to fall back to md5?

2014-10-29 Thread Josh Berkus
ith peer and ident. Maybe someday (protocol bump) we can have a way to make other methods continue, and then nobody will need to change their files to support the new way. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] TODO request: log_long_transaction

2014-10-27 Thread Josh Berkus
good GSOC project or first-time contribution. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-10-24 Thread Josh Berkus
se of us who haven't followed every post in this thread, is there somewhere I can see the proposed syntax? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgr

Re: [HACKERS] Autovacuum fails to keep visibility map up-to-date in mostly-insert-only-tables

2014-10-20 Thread Josh Berkus
#x27;s no reason we *have* to do anything other than set hint bits and > possibly freeze xmin. +1 -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Vitesse DB call for testing

2014-10-17 Thread Josh Berkus
CK, Before we go any further on this, how is Vitesse currently licensed? last time we talked it was still proprietary. If it's not being open-sourced, we likely need to take discussion off this list. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-ha

Re: [HACKERS] Maximum number of WAL files in the pg_xlog directory

2014-10-15 Thread Josh Berkus
27;t give our users any kind of reasonable monitoring threshold at all sucks though. Also, it makes it kind of hard to allocate a wal partition if it could be 10X the minimum size, you know? What happened to the work Heikki was doing on making transaction log disk usage sane? -- Josh Berkus Postg

Re: [HACKERS] Maximum number of WAL files in the pg_xlog directory

2014-10-15 Thread Josh Berkus
Ls partition. If we don't count the WAL files, though, that eliminates the best way to detecting when archiving is failing. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to y

Re: [HACKERS] schema-only -n option in pg_restore fails

2014-10-09 Thread Josh Berkus
All, Crossing this over to -hackers because it's stopped being a bug and is now a TODO item. See below. For those not on pgsql-bugs, I've quoted the full bug report below my proposal. On 10/09/2014 12:36 PM, Josh Berkus wrote: > Summary: pg_restore -n attempts to restore objects

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2014-10-07 Thread Josh Berkus
this (and it seems like a good idea), we really, really, really need to include some general system views which expose the options in a user-friendly format (like columns, JSON or an array). It's already very hard for users to get information about what reloptions have been set. -- Josh Ber

Re: [HACKERS] Fixed xloginsert_locks for 9.4

2014-10-03 Thread Josh Berkus
ll don't have any docs for this even if we decided to accept the GUC? On that basis, I'd say accept the DEFINE and punt the GUC to the next 9.5 Commitfest ... assuming someone wants to write docs. Heck, I might write them after some testing. But that testing won't happen in time

Re: [HACKERS] DDL Damage Assessment

2014-10-02 Thread Josh Berkus
to hack my way around and send patches to > have it (the common answer) available in the next PostgreSQL release. Great! -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] CREATE IF NOT EXISTS INDEX

2014-09-30 Thread Josh Berkus
On 09/30/2014 04:58 PM, Fabrízio de Royes Mello wrote: > On Tue, Sep 30, 2014 at 8:47 PM, Josh Berkus wrote: >> >> On 09/30/2014 04:16 PM, Andres Freund wrote: >>> On 2014-09-30 16:03:01 -0700, Josh Berkus wrote: >>>> On 09/30/2014 03:53 PM, Andres Freund w

Re: [HACKERS] CREATE IF NOT EXISTS INDEX

2014-09-30 Thread Josh Berkus
On 09/30/2014 04:16 PM, Andres Freund wrote: > On 2014-09-30 16:03:01 -0700, Josh Berkus wrote: >> On 09/30/2014 03:53 PM, Andres Freund wrote: >>> Good point. I think it's fair enough to only allow CINE on named >>> indexes. >> >> On the other han

Re: [HACKERS] CREATE IF NOT EXISTS INDEX

2014-09-30 Thread Josh Berkus
On 09/30/2014 03:53 PM, Andres Freund wrote: > On 2014-09-30 18:47:24 -0400, Tom Lane wrote: >> Josh Berkus writes: >>> On 09/30/2014 02:43 PM, Tom Lane wrote: >>>> =?UTF-8?Q?Fabr=C3=ADzio_de_Royes_Mello?= writes: >>>>> What's your thoughts

Re: [HACKERS] CREATE IF NOT EXISTS INDEX

2014-09-30 Thread Josh Berkus
do think it should be name-based. While an "IF NOT EXISTS" which checked for a duplicate column declartion would be nice, there's a raft of issues with implementing it that way. Users I know are generally just looking to avoid getting a transaction-halting error when they run the sa

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-09-30 Thread Josh Berkus
On 09/30/2014 02:51 PM, Kevin Grittner wrote: > Josh Berkus wrote: >> On 09/30/2014 02:39 PM, Kevin Grittner wrote: >>> Josh Berkus wrote: >>>> On 09/30/2014 07:15 AM, Kevin Grittner wrote: >>>> >>>>> At the risk of pushing people away

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-09-30 Thread Josh Berkus
On 09/30/2014 02:39 PM, Kevin Grittner wrote: > Josh Berkus wrote: >> On 09/30/2014 07:15 AM, Kevin Grittner wrote: > >>> At the risk of pushing people away from this POV, I'll point out >>> that this is somewhat similar to what we do for unlogged bulk loa

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-09-30 Thread Josh Berkus
een fast/slow bulk loads affects *only* the speed of loading, not the locking rules. Having a statement silently take a full table lock when we were expecting it to be concurrent (because, for example, the index got rebuilt and someone forgot the UNIQUE) violates POLA from my perspecti

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-09-30 Thread Josh Berkus
27;s blocking it then fine. But if we might change the concurrency approach, then what's the point in quibbling about docs? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT ... ON CONFLICT {UPDATE | IGNORE}

2014-09-30 Thread Josh Berkus
dly anything which should block the patch. It has documentation, more than we'd require for a lot of other patches, and it's not like the 9.5 release is next month. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hacker

Re: [HACKERS] open items for 9.4

2014-09-30 Thread Josh Berkus
gt; actually write the documentation and commit it that way, I'm happy with > that too. I'm also in favor of removing it. We really don't need more GUCs which nobody has any clear idea how to change or why. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via p

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-29 Thread Josh Berkus
On 09/29/2014 11:49 AM, Arthur Silva wrote: > What's the call on the stride length? Are we going to keep it hardcoded? Please, yes. The complications caused by a variable stride length would be horrible. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql

Re: [HACKERS] open items for 9.4

2014-09-29 Thread Josh Berkus
UCs" curmudgeon hat. 1. What does it do? 2. Are there reasons why users would want to change it from the default? 3. Can we explain those reasons in the form of documentation? IMHO, if the answers to 2 or 3 are "no", then we shouldn't have a GUC. -- Josh Berkus PostgreSQL E

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-28 Thread Josh Berkus
On 09/26/2014 06:20 PM, Josh Berkus wrote: > Overall, I'm satisfied with the performance of the length-and-offset > patch. Oh, also ... no bugs found. So, can we get Beta3 out now? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-26 Thread Josh Berkus
ngth-and-offset when uncompressed (EXTERNAL) was faster on Q1 than head! This was surprising enough that I retested it. Overall, I'm satisfied with the performance of the length-and-offset patch. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers ma

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-09-26 Thread Josh Berkus
"round to nearest", but the two changes can be considered > independently. I'm good with the error. We'll want to add stuff to both the docs and pg_settings to *show* the minimum value, and an informative error message would help, i.e.: "Invalid value for log_rotation_i

Re: [HACKERS] proposal: rounding up time value less than its unit.

2014-09-26 Thread Josh Berkus
n-zero, as it would then be rounded to zero. That would not be a back-portable fix. There are many 3rd-party tools out there (AWS RDS, for one) which do not use the units. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.o

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-25 Thread Josh Berkus
ving up a couple percent of space in comparison to the > all-lengths version, but this is probably an acceptable tradeoff > for not degrading on very large arrays. > > I've not done any speed testing. I'll do some tommorrow. I should have some different DBs to test on, too. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-25 Thread Josh Berkus
ll us anything we > don't expect, but we should do it anyway. OK. I'll spend some time trying to get Socorro with JSONB working so that I'll have a second test case. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hack

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-25 Thread Josh Berkus
On 09/25/2014 10:26 AM, Andres Freund wrote: > On 2014-09-25 10:25:24 -0700, Josh Berkus wrote: >> If Heikki says it's ready, I'll test. So far he's said that it wasn't >> done yet. > > http://www.postgresql.org/message-id/541c242e.3030...@vmware.com Ye

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-25 Thread Josh Berkus
On 09/25/2014 10:20 AM, Andres Freund wrote: > On 2014-09-25 10:18:24 -0700, Josh Berkus wrote: >> On 09/25/2014 10:14 AM, Bruce Momjian wrote: >>> On Thu, Sep 25, 2014 at 06:01:08PM +0200, Andres Freund wrote: >>>> But independent of which version is chosen

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-25 Thread Josh Berkus
e for releasing based on Tom's lengths-only patch, which is done, tested, and ready to go. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-25 Thread Josh Berkus
On 09/25/2014 09:01 AM, Andres Freund wrote: > But independent of which version is chosen, we *REALLY* need to make the > decision soon. This issue has held up the next beta (like jsonb has > blocked previous beta) for *weeks*. Yes, please! -- Josh Berkus PostgreSQL Experts

Re: [HACKERS] RLS feature has been committed

2014-09-23 Thread Josh Berkus
commits it. That's certainly what it looks like to me, and on that basis Stephen should have held back the patch until he got reviewer OK. Fortunately, since we use Git and not CVS, reverting patches isn't the trauma it once was. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com

Re: [HACKERS] RLS feature has been committed

2014-09-22 Thread Josh Berkus
ately reviewed (and if not, if it should be reverted), not whether it should have been in the CF or not. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Should we excise the remnants of borland cc support?

2014-09-22 Thread Josh Berkus
~15 > years since the last release. > Since there's both msvc and mingw support for windows builds - borlands > only platform - I see little point in continuing to support it. +1 -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-22 Thread Josh Berkus
are we on this? Do we have a patch ready for testing? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] RLS Design

2014-09-20 Thread Josh Berkus
t can do more to help us with the > testing and review process. We discussed this at the last developer meeting, without coming up with a written procedure. Your ideas can help ... -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Final Patch for GROUPING SETS

2014-09-19 Thread Josh Berkus
> - [] > > Would that be better? (It's not consistent with other YAML outputs like > sort/group keys, but it's equally legal as far as I can tell and seems > more readable.) That works for me. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sen

Re: [HACKERS] Anonymous code block with parameters

2014-09-18 Thread Josh Berkus
before reading this :) >> >> I guess it proves (a little) that WITH is the right place to do these >> kind of things ... > > I've been wanting this syntax for a few years now, so I certainly vote > for it. > Just to clarify: I want the WITH syntax for different

Re: [HACKERS] Anonymous code block with parameters

2014-09-18 Thread Josh Berkus
nc(x,y,z); Oh! Awesome! -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-18 Thread Josh Berkus
engths patch so that we can get 9.4 out the door. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Final Patch for GROUPING SETS

2014-09-17 Thread Josh Berkus
] .. in JSON? Seems to me that we need a better way to display the grand total grouping set. > > Or maybe: >Grouping Set: (onek.four, onek.ten, onek.hundred) >Grouping Set: (onek.four, onek.ten) >Grouping Set: (onek.four) >Grouping Set: () The lat

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-16 Thread Josh Berkus
ess well, and we've got an approach that fixes that > problem while preserving the advantages of fast lookup. We should > have a darn fine reason to say no to that approach, and "it didn't > benefit my particular use case" is not it. Do you feel that way *as a code maint

Re: [HACKERS] jsonb contains behaviour weirdness

2014-09-16 Thread Josh Berkus
f using PostgreSQL arrays; I had to test it to verify the current behavior. Not sure exactly where this note should go, mind you. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-16 Thread Josh Berkus
On 09/16/2014 09:54 AM, Robert Haas wrote: > On Tue, Sep 16, 2014 at 12:47 PM, Josh Berkus wrote: >> On 09/16/2014 06:31 AM, Robert Haas wrote: >>> On Mon, Sep 15, 2014 at 7:44 PM, Peter Geoghegan wrote: >>>> On Mon, Sep 15, 2014 at 4:05 PM, Josh Berkus wrote: &

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-16 Thread Josh Berkus
On 09/16/2014 06:31 AM, Robert Haas wrote: > On Mon, Sep 15, 2014 at 7:44 PM, Peter Geoghegan wrote: >> On Mon, Sep 15, 2014 at 4:05 PM, Josh Berkus wrote: >>> Actually, having the keys all at the same level *is* relevant for the >>> issue we're discussing. If t

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-15 Thread Josh Berkus
On 09/15/2014 02:16 PM, Robert Haas wrote: > On Mon, Sep 15, 2014 at 3:09 PM, Josh Berkus wrote: >> On 09/15/2014 10:23 AM, Claudio Freire wrote: >>> Now, large small keys could be 200 or 2000, or even 20k. I'd guess >>> several should be tested to find the shape o

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-15 Thread Josh Berkus
On 09/15/2014 12:25 PM, Claudio Freire wrote: > On Mon, Sep 15, 2014 at 4:17 PM, Josh Berkus wrote: >> On 09/15/2014 12:15 PM, Claudio Freire wrote: >>> So while you're right that it's perhaps above what would be a common >>> use case, the range "somew

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-15 Thread Josh Berkus
On 09/15/2014 12:15 PM, Claudio Freire wrote: > So while you're right that it's perhaps above what would be a common > use case, the range "somewhere between 200 and 100K" for the tipping > point seems overly imprecise to me. Well, then, you know how to solve that

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-15 Thread Josh Berkus
th testing further if we think that having more than 200 top-level keys in one JSONB value is going to be a use case for more than 0.1% of our users. I personally do not. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-15 Thread Josh Berkus
why your patch would be faster for the "last element" case than the original offsets version? If not, I think the corner case is so obscure as to be not worth optimizing for. I can't imagine that more than a tiny minority of our users are going to have thousands of keys per datum. -

Re: [HACKERS] jsonb contains behaviour weirdness

2014-09-15 Thread Josh Berkus
On 09/12/2014 01:33 PM, Tom Lane wrote: > Josh Berkus writes: >> However, this better become a FAQ item, because it's not necessarily the >> behavior that folks used to JSON but not Postgres will expect. > > No, it's a bug, not a documentation deficiency. Hmmm? A

Re: [HACKERS] jsonb contains behaviour weirdness

2014-09-12 Thread Josh Berkus
--- t That's consistent with our docs and past behavior. However, this better become a FAQ item, because it's not necessarily the behavior that folks used to JSON but not Postgres will expect. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hacker

Re: [HACKERS] jsonb contains behaviour weirdness

2014-09-12 Thread Josh Berkus
e weird :) > The reason why jsonb contains behaves so is check in the beginning > of jsonb_contains. It makes fast check of jsonb type and elements count > before calling JsonbDeepContains. > > if (JB_ROOT_COUNT(val) < JB_ROOT_COUNT(tmpl) || > JB_ROOT_IS_OBJECT(val) != JB_ROOT

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-12 Thread Josh Berkus
On 09/12/2014 10:00 AM, Robert Haas wrote: > On Fri, Sep 12, 2014 at 1:00 PM, Robert Haas wrote: >> On Thu, Sep 11, 2014 at 9:01 PM, Josh Berkus wrote: >>> So, I finally got time to test Tom's latest patch on this. >>> >>> TLDR: we want to go

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-12 Thread Josh Berkus
ire-hose data. Yah, if we have enough time for me to get the Mozilla Socorro test environment working, I can also test with Mozilla crash data. That has some deep nesting and very large values. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (p

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-09-11 Thread Josh Berkus
t JSONB columns to EXTERNAL, and the the same performance as the unpatched version. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PL/pgSQL 2

2014-09-04 Thread Josh Berkus
sors that they work *exactly* like PL/SQL's packages, which is incompatible with Postgres namespacing. Also, we'd want any "package" concept to be usable with external PLs as well as PL/pgSQL, which necessitates other Oracle-incompatible changes. -- Josh Berkus PostgreSQL Experts Inc.

Re: [HACKERS] PL/pgSQL 2

2014-09-02 Thread Josh Berkus
On 09/02/2014 03:19 PM, Josh Berkus wrote: > Well, if I were designing a new procedural SQL extension language, I > wouldn't base it on the bastard child of ADA and SQL89. I would come up Ada, that is. One is a programming language, the other is the bane of architects. --

Re: [HACKERS] PL/pgSQL 2

2014-09-02 Thread Josh Berkus
amount to the proverbial porcine makeover. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Why data of timestamptz does not store value of timezone passed to it?

2014-08-28 Thread Josh Berkus
tension. On the other hand, I take partial responsibility for the mess which is our data type naming. What we call timestamptz should just be "timestamp", and whether or not it converts to local timezone on retrieval should be a GUC setting. And the type we call "timestamp" sh

Re: [HACKERS] Similar to csvlog but not really, json logs?

2014-08-28 Thread Josh Berkus
; : { "parameters" : { "$1" : 90700 } } ... This would allow me, or Dalibo, to remove literally dozens of lines of error-prone regexing code. That fix would, IMHO, make it worth enabling JSON logging as a logging hook or something similar. If we're just going to convert the C

Re: [HACKERS] Need Multixact Freezing Docs

2014-08-28 Thread Josh Berkus
On 08/28/2014 09:09 AM, Alvaro Herrera wrote: > Josh Berkus wrote: >> On 04/16/2014 01:30 PM, Alvaro Herrera wrote: >>> Josh Berkus wrote: >>>> >>>>> You can see the current multixact value in pg_controldata output. Keep >>>>> tim

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Josh Berkus
storage. Testing STORAGE EXTENDED soon. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Josh Berkus
On 08/26/2014 11:40 AM, Tom Lane wrote: > Josh Berkus writes: >> Anyway, I called for feedback on by blog, and have gotten some: >> http://www.databasesoup.com/2014/08/the-great-jsonb-tradeoff.html > > I was hoping you'd get some useful data from that, but so far i

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-26 Thread Josh Berkus
it's an order-of-magnitude slower. Anyway, I called for feedback on by blog, and have gotten some: http://www.databasesoup.com/2014/08/the-great-jsonb-tradeoff.html -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@post

Re: [HACKERS] postgresql latency & bgwriter not doing its job

2014-08-25 Thread Josh Berkus
IO while it works to write out all of RAM, which makes me suspect you're using Ext3 or HFS+. Making the bgwriter more aggressive adds a significant risk of writing the same pages multiple times between checkpoints, so it's not a simple fix. -- Josh Berkus PostgreSQL Experts Inc. http://p

[HACKERS] Code bug or doc bug?

2014-08-24 Thread Josh Berkus
in tablespace "ssd" HINT: You must move them back to the database's default tablespace before using this command. Aside from being a stupid limitation (I need to copy the tables back to the old tablespace so that I can recopy them to the new one?), the above seems to be in direct con

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-21 Thread Josh Berkus
ned that my workload is unusual and don't want us to make this decision based entirely on it. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-20 Thread Josh Berkus
On 08/20/2014 08:29 AM, Tom Lane wrote: > Josh Berkus writes: >> On 08/15/2014 04:19 PM, Tom Lane wrote: >>> Personally I'd prefer to go to the all-lengths approach, but a large >>> part of that comes from a subjective assessment that the hybrid approach >

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-20 Thread Josh Berkus
N^2) penalties in findJsonbValueFromContainer and > JsonbIteratorNext OK, will test. This means we need a beta3, no? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http:

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-19 Thread Josh Berkus
ime: 0.095 ms Execution time: 5292.047 ms (7 rows) So, that extraction test is about 1% *slower* than the basic Tom Lane lengths-only patch, and still 80% slower than original JSONB. And it's the same size as the lengths-only version. Huh? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [patch] pg_copy - a command for reliable WAL archiving

2014-08-19 Thread Josh Berkus
ince it's in the examples, people will > probably use it, even if they don't need to or shouldn't. And not > recommending it for the restore_command is also confusing. I'm afraid that I agree with Peter here. pg_copy looks like a nice foundation for the eventual

Re: [Bad Attachment] Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-15 Thread Josh Berkus
e pattern of having between 50 and 200 keys, each of which has a short value. So we don't need to *optimize* for that case, but it also shouldn't be disastrously slow or 300% of the size of comparable TEXT. Mind you, I don't find +80% to be disastrously slow (especially not with a spac

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-15 Thread Josh Berkus
case, the *index* on the JSONB is only 60MB. Which shows just how terrific the improvement in GIN index size/performance is. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscript

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-14 Thread Josh Berkus
t;2003-06-30"}'::jsonb) Planning time: 0.098 ms Execution time: 5214.212 ms ... so, an 80% increase in lookup and extraction time for swapping offsets for lengths. That's actually all extraction time; I tried removing the extraction from the query, and without it both queries are close enough to be statstically insignificant. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Reporting the commit LSN at commit time

2014-08-14 Thread Josh Berkus
cases for which automated connection failover without a managed proxy is a Seriously Bad Idea. For one thing, you're setting up a strong risk of split-brain. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-14 Thread Josh Berkus
On 08/14/2014 04:47 PM, Josh Berkus wrote: > thetype |colsize_distribution > -+ > json| {1777,1803,1890,1940,4424} > jsonb | {5902,5926,5978,6002,6208} Just realized my query was counting the whole row size instead of just the column s

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-14 Thread Josh Berkus
json| {1777,1803,1890,1940,4424} jsonb | {5902,5926,5978,6002,6208} Shouldn't the lower end stuff be smaller? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-14 Thread Josh Berkus
On 08/14/2014 04:02 PM, Tom Lane wrote: > Josh Berkus writes: >> So, here's a destruction test case: >> 200,000 JSON values (plus 2 key columns) >> Average width 4K (+/- 1K) >> 183 keys per JSON value > > Is that 183 keys exactly each time, or is 183 the av

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-14 Thread Josh Berkus
e_pretty -------- 11 MB (1 row) Next up: Tom's patch and indexing! -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb format is pessimal for toast compression

2014-08-14 Thread Josh Berkus
ce the whitespace and syntax > sugar is gone in JSONB, I was unclear how much compression would help. I thought the destruction case was when we have enough top-level keys that the offsets are more than 1K total, though, yes? So we need to test that set ... -- Josh Berkus PostgreSQL Experts

<    1   2   3   4   5   6   7   8   9   10   >