Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-11-06 Thread Jim Nasby
(like we do when we fill vacrelstats-dead_tuples. But like I said above, I think this is already making a mountain out of a mole-hill, until we have evidence there's a real problem. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-11-10 Thread Jim Nasby
On 11/7/14, 8:21 PM, Robert Haas wrote: On Thu, Nov 6, 2014 at 8:03 PM, Jim Nasby jim.na...@bluetreble.com wrote: The problem right now is there's no way to actually obtain evidence that this is (or isn't) something to worry about, because we just silently skip pages. If we had any kind

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-11-10 Thread Jim Nasby
On 11/10/14, 11:28 AM, Alvaro Herrera wrote: Jim Nasby wrote: On 11/7/14, 8:21 PM, Robert Haas wrote: On Thu, Nov 6, 2014 at 8:03 PM, Jim Nasby jim.na...@bluetreble.com wrote: The problem right now is there's no way to actually obtain evidence that this is (or isn't) something to worry about

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-11-10 Thread Jim Nasby
. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-11-10 Thread Jim Nasby
On 11/10/14, 12:56 PM, Andres Freund wrote: On 2014-11-10 12:37:29 -0600, Jim Nasby wrote: On 11/10/14, 12:15 PM, Andres Freund wrote: If what we want is to quantify the extent of the issue, would it be more convenient to save counters to pgstat? Vacuum already sends pgstat messages, so

Re: [HACKERS] Add CREATE support to event triggers

2014-11-10 Thread Jim Nasby
cases, but now some folks are saying that everyone should just use streaming rep and be done with it. :P -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Jim Nasby
(which should be infrequent). I believe the low normal space usage is just an indication that most databases don't use many MultiXacts. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] tracking commit timestamps

2014-11-11 Thread Jim Nasby
On 11/11/14, 2:03 PM, Alvaro Herrera wrote: Jim Nasby wrote: On 11/10/14, 7:40 AM, Alvaro Herrera wrote: Ah, right. So AFAIK we don't need to keep anything older than RecentXmin or something like that -- which is not too old. If I recall correctly Josh Berkus was saying in a thread about

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-11-12 Thread Jim Nasby
On 11/11/14, 2:01 AM, Andres Freund wrote: On 2014-11-10 19:36:18 -0600, Jim Nasby wrote: On 11/10/14, 12:56 PM, Andres Freund wrote: On 2014-11-10 12:37:29 -0600, Jim Nasby wrote: On 11/10/14, 12:15 PM, Andres Freund wrote: If what we want is to quantify the extent of the issue, would

Re: [HACKERS] tracking commit timestamps

2014-11-12 Thread Jim Nasby
be more appropriate? -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Race condition between hot standby and restoring a FPW

2014-11-12 Thread Jim Nasby
into it? Presumably that means we'd have to keep the page locked until the image is written... -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] On partitioning

2014-11-12 Thread Jim Nasby
multi-key partitions with something like an anyarray. Perhaps that's OK as a first pass, but I expect it'll be one of the next things folks ask for. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] pg_multixact not getting truncated

2014-11-12 Thread Jim Nasby
is a lot more limited than for an XID, so it's probably pretty safe setting that to zero. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Teaching pg_dump to use NOT VALID constraints

2014-11-12 Thread Jim Nasby
then we shouldn't have to *any* re-validate when we load? (Though, we'd have to be careful of that with CHECK because that can call user code...) -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-13 Thread Jim Nasby
. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TODO request: log_long_transaction

2014-11-14 Thread Jim Nasby
of statement_timeout for transactions; the ability to abort a transaction if it runs for too long. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] proposal: plpgsql - Assert statement

2014-11-17 Thread Jim Nasby
for simple sanity checking because it was much nicer than typeing IF THEN RAISE ERROR END IF. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] proposal: plpgsql - Assert statement

2014-11-18 Thread Jim Nasby
use another PL or it's just syntax sugar improve things for our users. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] 9.5: Better memory accounting, towards memory-bounded HashAgg

2014-11-18 Thread Jim Nasby
haven't been touched since the last time you calculated then you're don't need to recalc. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] pg_multixact not getting truncated

2014-11-21 Thread Jim Nasby
we'd need to prevent anyone from copying a database while being vacuumed, as well as preventing anyone from changing datcanconnect while the vacuum is running. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers

Re: [HACKERS] Functions used in index definitions shouldn't be changed

2014-11-21 Thread Jim Nasby
impacts a lot of reporting scenarios. I fully agree that it's impractical to completely restrict this case, but something akin to FORCE seems reasonable. If nothing else, I'd think we should at least issue a warning if someone does something that might affect index viability. -- Jim Nasby, Data

[HACKERS] Determining typmod of *source* of a cast

2014-11-29 Thread Jim Nasby
Is there any way to determine the typemod of the source data for a cast? Perhaps a modification on get_call_expr_argtype(), though I'd hate to put that in an extension... BTW, it'd be nice if we better emphasized that the typmod passed to a cast function is for the destination... :/ -- Jim

Re: [HACKERS] How about a option to disable autovacuum cancellation on lock conflict?

2014-11-29 Thread Jim Nasby
doesn't get cancelled anymore. Opinions? What do you mean by never succeed? Is it skipping a large number of pages? Might re-trying the locks within the same vacuum help, or are the user locks too persistent? -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-11-30 Thread Jim Nasby
On 11/10/14, 7:52 PM, Tom Lane wrote: On the whole, I'm +1 for just logging the events and seeing what we learn that way. That seems like an appropriate amount of effort for finding out whether there is really an issue. Attached is a patch that does this. -- Jim Nasby, Data Architect, Blue

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-12-01 Thread Jim Nasby
On 12/1/14, 11:57 AM, Andres Freund wrote: On 2014-11-30 20:46:51 -0600, Jim Nasby wrote: On 11/10/14, 7:52 PM, Tom Lane wrote: On the whole, I'm +1 for just logging the events and seeing what we learn that way. That seems like an appropriate amount of effort for finding out whether

[HACKERS] No documentation on record *= record?

2014-12-01 Thread Jim Nasby
There doesn't seem to be documentation on *= (or search isn't finding it). Is this intentional? -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] How about a option to disable autovacuum cancellation on lock conflict?

2014-12-02 Thread Jim Nasby
point. Now that's not a bad idea. This would basically mean just saving a block number in pg_class after every intermediate index clean and then setting that back to zero when we're done with that relation, right? -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get

Re: [HACKERS] On partitioning

2014-12-03 Thread Jim Nasby
) is that you could leave holes in your partitioning map. Note that in the multi-key case we could still have a record of rangetypes. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Sequence Access Method WIP

2014-12-04 Thread Jim Nasby
for this purpose maybe? Why not? RAID arrays typically use stripe sizes in the 128-256k range, which means only 16 or 32 sequences per stripe. It still might make sense to allow controlling what tablespace a sequence is in, but IMHO the default should just be pg_default. -- Jim Nasby, Data Architect

Re: [HACKERS] duplicated partial-column assigns allowed by checkInsertTargets

2014-12-04 Thread Jim Nasby
: insert into t (col, col) values ((42, 43), (44, 43)); ^ Isn't this a bit odd? Yes, and sounds like a good way to create bugs... my vote would be to fix this. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com

Re: [HACKERS] Parallel Seq Scan

2014-12-05 Thread Jim Nasby
instead of at plan time? That would allow us to dynamically scale parallelism based on system load. If we don't even consider parallelism until we've pulled some number of tuples/pages from a relation, this would also eliminate all parallel overhead on small relations. -- Jim Nasby, Data

Re: [HACKERS] On partitioning

2014-12-05 Thread Jim Nasby
would need to be a dumb varlena that stores the composite HeapTupleHeader. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] On partitioning

2014-12-05 Thread Jim Nasby
On 12/5/14, 1:22 PM, Jim Nasby wrote: On 12/5/14, 3:42 AM, Amit Langote wrote: I think you are right. I think in this case we need something similar to column pg_index.indexprs which is of type pg_node_tree(which seems to be already suggested by Robert). So may be we can proceed

Re: [HACKERS] On partitioning

2014-12-05 Thread Jim Nasby
On 12/5/14, 2:02 PM, Robert Haas wrote: On Fri, Dec 5, 2014 at 2:52 PM, Jim Nasby jim.na...@bluetreble.com wrote: The other option would be to use some custom rowtype to store boundary values and have a method that can form a boundary tuple from a real one. Either way, I suspect this is better

Re: [HACKERS] Elusive segfault with 9.3.5 query cancel

2014-12-05 Thread Jim Nasby
that's huge because of shared buffers, but perhaps there's some way to avoid writing those? (That means the core won't help if the bug is due to something in a buffer, but that seems unlikely enough that the tradeoff is worth it...) -- Jim Nasby, Data Architect, Blue Treble Consulting Data

Re: [HACKERS] Elusive segfault with 9.3.5 query cancel

2014-12-05 Thread Jim Nasby
On 12/5/14, 5:49 PM, Josh Berkus wrote: On 12/05/2014 02:41 PM, Jim Nasby wrote: Perhaps we should also officially recommend production servers be setup to create core files. AFAIK the only downside is the time it would take to write a core that's huge because of shared buffers, but perhaps

Re: [HACKERS] [v9.5] Custom Plan API

2014-12-08 Thread Jim Nasby
interface yet. I've made some minor edits, with an emphasis on not changing original intent. Each section was saved as a separate edit, so if anyone objects to something just revert the relevant change. Once the code is available more editing can be done. -- Jim Nasby, Data Architect, Blue

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-12-08 Thread Jim Nasby
On 12/7/14, 6:16 PM, Simon Riggs wrote: On 20 October 2014 at 10:57, Jim Nasby jim.na...@bluetreble.com wrote: Currently, a non-freeze vacuum will punt on any page it can't get a cleanup lock on, with no retry. Presumably this should be a rare occurrence, but I think it's bad that we just

Re: [HACKERS] Casting issues with domains

2014-12-08 Thread Jim Nasby
without requiring C. C isn't an option on many (even most) environments in today's cloud world, aside from the intimidation factor. There are comments in the code that hypothesize about making cstring a full type; that might be all that's needed. -- Jim Nasby, Data Architect, Blue Treble Consulting

Re: [HACKERS] On partitioning

2014-12-08 Thread Jim Nasby
on the disk. If you think about it, partitioning is really a hack anyway. It clutters up your logical set implementation with a bunch of physical details. What most people really want when they implement partitioning is simply data locality. -- Jim Nasby, Data Architect, Blue Treble Consulting Data

Re: [HACKERS] On partitioning

2014-12-08 Thread Jim Nasby
for a catchall partition and supported normal inheritance/triggers on that partition then users could continue to do whatever they needed with data that didn't fit the normal partitioning pattern. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http

[HACKERS] operator does not exist: character varying[] character[]

2014-12-09 Thread Jim Nasby
Is there any particular reason we don't allow comparing char and varchar arrays? If not I'll submit a patch. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] On partitioning

2014-12-09 Thread Jim Nasby
On 12/8/14, 5:19 PM, Josh Berkus wrote: On 12/08/2014 02:12 PM, Jim Nasby wrote: On 12/8/14, 12:26 PM, Josh Berkus wrote: 4. Creation Locking Problem high probability of lock pile-ups whenever a new partition is created on demand due to multiple backends trying to create the partition

Re: [HACKERS] operator does not exist: character varying[] character[]

2014-12-09 Thread Jim Nasby
On 12/9/14, 4:19 PM, Jim Nasby wrote: Is there any particular reason we don't allow comparing char and varchar arrays? If not I'll submit a patch. We're also missing operators on text and varchar arrays. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble

Re: [HACKERS] logical column ordering

2014-12-09 Thread Jim Nasby
thinking that in DefineRelation we can randomize stmt-tableElts before merging in inheritance attributes. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] operator does not exist: character varying[] character[]

2014-12-09 Thread Jim Nasby
On 12/9/14, 4:30 PM, Tom Lane wrote: Jim Nasby jim.na...@bluetreble.com writes: On 12/9/14, 4:19 PM, Jim Nasby wrote: Is there any particular reason we don't allow comparing char and varchar arrays? If not I'll submit a patch. We're also missing operators on text and varchar arrays

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Jim Nasby
the holidays, but not before then. FWIW, I suspect a call for help on -general or IRC would find volunteers for any necessary data entry work... -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Jim Nasby
because I don't feel I have enough knowledge to fulfill that role. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] On partitioning

2014-12-12 Thread Jim Nasby
that calls a function as opposed to glomming a text string together and passing that to EXECUTE. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] operator does not exist: character varying[] character[]

2014-12-12 Thread Jim Nasby
On 12/9/14, 5:06 PM, Jim Nasby wrote: On 12/9/14, 4:30 PM, Tom Lane wrote: Jim Nasby jim.na...@bluetreble.com writes: On 12/9/14, 4:19 PM, Jim Nasby wrote: Is there any particular reason we don't allow comparing char and varchar arrays? If not I'll submit a patch. We're also missing

Re: [HACKERS] On partitioning

2014-12-13 Thread Jim Nasby
On 12/12/14, 3:48 PM, Robert Haas wrote: On Fri, Dec 12, 2014 at 4:28 PM, Jim Nasby jim.na...@bluetreble.com wrote: Sure. Mind you, I'm not proposing that the syntax I just mooted is actually for the best. What I'm saying is that we need to talk about it. Frankly, if we're going to require

Re: [HACKERS] operator does not exist: character varying[] character[]

2014-12-13 Thread Jim Nasby
On 12/12/14, 7:16 PM, Tom Lane wrote: Jim Nasby jim.na...@bluetreble.com writes: I'd say that array_eq (and probably _cmp) just needs to be taught to fall back to what oper() does, but this part of the commit message gives me pause: Change the operator search algorithms to look

Re: [HACKERS] pgbench -f and vacuum

2014-12-13 Thread Jim Nasby
in the server logfile, I think checking for the table is the right answer. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

[HACKERS] REVIEW: Track TRUNCATE via pgstat

2014-12-15 Thread Jim Nasby
something if it times out; otherwise you'll have a test failure and won't have any indication why. I've attached a patch that adds logging on timeout and contains a test case that demonstrates the rollback to savepoint bug. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble

Re: [HACKERS] Fractions in GUC variables

2014-12-15 Thread Jim Nasby
that for all GUCs that include units, so presumably there's no good way around it. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

[HACKERS] .gitignore config.cache and comment about regression.(out|diff)

2014-12-16 Thread Jim Nasby
a comment about that. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com From a681953a802230e73e5e4f91607eca9dd99c34f2 Mon Sep 17 00:00:00 2001 From: Jim Nasby jim.na...@bluetreble.com Date: Mon, 15 Dec 2014 18:35:50 -0600 Subject: [PATCH] Ignore

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-12-17 Thread Jim Nasby
it untranslatable? Doesn't the translation happen down-stream, so that at most we'd just need two translation messages? Or worst case we could have two separate elog calls, if we wanted to go that route. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-12-18 Thread Jim Nasby
of for the later is something like failed initial lock attempt. That's the only thing that will be true in all cases. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] NUMERIC private methods?

2014-12-18 Thread Jim Nasby
be corrected. If copying data/palloc is the root of numeric's performance problems then we need to address that, because it will provide benefit across the entire database. The pattern of (palloc; copy) is repeated throughout a large part of the codebase. -- Jim Nasby, Data Architect, Blue

Re: [HACKERS] Commitfest problems

2014-12-18 Thread Jim Nasby
it myself. :) -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

2014-12-18 Thread Jim Nasby
to decide what we're doing first. ;) -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: hash_create API changes (was Re: [HACKERS] speedup tidbitmap patch: hash BlockNumber)

2014-12-18 Thread Jim Nasby
information cache|4 2 db hash|4 52 json object hashtable|64 28958 smgr relation table|16 -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com diff --git a/src/backend/utils/hash/dynahash.c b/src/backend/utils/hash/dynahash.c index 5f6d6cb

Re: hash_create API changes (was Re: [HACKERS] speedup tidbitmap patch: hash BlockNumber)

2014-12-19 Thread Jim Nasby
On 12/18/14, 5:00 PM, Jim Nasby wrote: 2201582 20 -- Mostly LOCALLOCK and Shared Buffer Started looking into this; perhaps https://code.google.com/p/fast-hash/ would be worth looking at, though it requires uint64. It also occurs to me that we're needlessly shoving a lot of 0's into the hash

Re: hash_create API changes (was Re: [HACKERS] speedup tidbitmap patch: hash BlockNumber)

2014-12-19 Thread Jim Nasby
On 12/19/14, 5:13 PM, Tom Lane wrote: Jim Nasby jim.na...@bluetreble.com writes: On 12/18/14, 5:00 PM, Jim Nasby wrote: 2201582 20 -- Mostly LOCALLOCK and Shared Buffer Started looking into this; perhaps https://code.google.com/p/fast-hash/ would be worth looking at, though it requires

Re: [HACKERS] Commitfest problems

2014-12-19 Thread Jim Nasby
to revise a commit message; it just makes downstream pulls uglier if the commit was already pushed (see https://help.github.com/articles/changing-a-commit-message/). It might be possible to minimize or even eliminate that pain via git hooks. -- Jim Nasby, Data Architect, Blue Treble Consulting

Re: hash_create API changes (was Re: [HACKERS] speedup tidbitmap patch: hash BlockNumber)

2014-12-20 Thread Jim Nasby
On 12/20/14, 11:51 AM, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-12-19 22:03:55 -0600, Jim Nasby wrote: What I am thinking is not using all of those fields in their raw form to calculate the hash value. IE: something analogous to: hash_any(SharedBufHash, (rot

Re: [HACKERS] Parallel Seq Scan

2014-12-21 Thread Jim Nasby
. ... In the above tests, it seems to me that the maximum benefit due to 'a' is realized upto 4~8 workers I'd think a good first estimate here would be to just use effective_io_concurrency. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com

Re: [HACKERS] Proposal VACUUM SCHEMA

2014-12-21 Thread Jim Nasby
a command that allows you to apply a second command to every object in a schema. We would have to be careful about PreventTransactionChain commands. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] PATCH: decreasing memory needlessly consumed by array_agg

2014-12-21 Thread Jim Nasby
accumArrayResult. Currently, the code isn't using the rcontext for anything except for old API calls (in first call to accumArrayResult). Until we eliminate the API though, we should leave something in place that still uses the old one, to make certain we don't accidentally break it. -- Jim Nasby

Re: [HACKERS] Proposal VACUUM SCHEMA

2014-12-22 Thread Jim Nasby
-transactional commands dynamically, because VACUUM isn't the only one that suffers from this problem. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] Proposal VACUUM SCHEMA

2014-12-22 Thread Jim Nasby
. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal VACUUM SCHEMA

2014-12-23 Thread Jim Nasby
of solving this only for vacuum we create something generic? :) Possibly using Robert's background worker work? -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Proposal VACUUM SCHEMA

2014-12-23 Thread Jim Nasby
On 12/23/14, 7:44 AM, Robert Haas wrote: On Mon, Dec 22, 2014 at 5:00 PM, Jim Nasby jim.na...@bluetreble.com wrote: I would MUCH rather that we find a way to special-case executing non-transactional commands dynamically, because VACUUM isn't the only one that suffers from this problem

Re: hash_create API changes (was Re: [HACKERS] speedup tidbitmap patch: hash BlockNumber)

2014-12-23 Thread Jim Nasby
On 12/20/14, 2:13 PM, Jim Nasby wrote: On 12/20/14, 11:51 AM, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-12-19 22:03:55 -0600, Jim Nasby wrote: What I am thinking is not using all of those fields in their raw form to calculate the hash value. IE: something analogous

Re: hash_create API changes (was Re: [HACKERS] speedup tidbitmap patch: hash BlockNumber)

2014-12-23 Thread Jim Nasby
On 12/24/14, 12:27 AM, Jim Nasby wrote: There were several select-only runs on both to warm shared_buffers (set to 512MB for this test, and fsync is off). BTW, presumably this ~380M database isn't big enough to show any problems with hash collisions, and I'm guessing you'd need way more

Re: [HACKERS] Proposal VACUUM SCHEMA

2014-12-24 Thread Jim Nasby
On 12/23/14, 8:49 PM, Fabrízio de Royes Mello wrote: Em terça-feira, 23 de dezembro de 2014, Jim Nasby jim.na...@bluetreble.com mailto:jim.na...@bluetreble.com escreveu: On 12/23/14, 8:54 AM, Fabrízio de Royes Mello wrote: Right now a lot of people just work around

Re: hash_create API changes (was Re: [HACKERS] speedup tidbitmap patch: hash BlockNumber)

2014-12-24 Thread Jim Nasby
On 12/24/14, 10:58 AM, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-12-24 00:27:39 -0600, Jim Nasby wrote: pgbench -S -T10 -c 4 -j 4 master: tps = 9556.356145 (excluding connections establishing) tps = 9897.324917 (excluding connections establishing) tps = 9287.286907

Re: [HACKERS] nls and server log

2014-12-29 Thread Jim Nasby
that tells you the encoding for that entry? -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [HACKERS] 9.5: Better memory accounting, towards memory-bounded HashAgg

2014-12-29 Thread Jim Nasby
of a tapesort being faster than an internal sort. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] BUG #12330: ACID is broken for unique constraints

2014-12-29 Thread Jim Nasby
to enforce RI rather than defining FKs precisely so that they can get a serialization failure return code and do automatic retry if it is caused by a race condition. That's less practical to compensate for when it comes to unique indexes or constraints. Wow, that's horrible. :( -- Jim Nasby, Data

Re: [HACKERS] Additional role attributes superuser review

2014-12-29 Thread Jim Nasby
of the database. My how this has become a can of worms... -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] Detecting backend failures via libpq / DBD::Pg

2014-12-29 Thread Jim Nasby
SELECT 1, but it would prevent a bunch of pointless work on the backend side, and should greatly simplify DBD's ping(). Only thing I'm not sure of is if this could be made to be safe within a COPY... :( -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http

Re: [HACKERS] Additional role attributes superuser review

2014-12-29 Thread Jim Nasby
this, my inclination is DUMP with appropriate hints for pg_dumpall, and BINARY. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] Publish autovacuum informations

2015-01-02 Thread Jim Nasby
at fcinfo-flinfo-fn_expr, and I have no idea how likely that is to get broken. Since it's a parse node, my guess is likely, but I'm just guessing. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Using 128-bit integers for sum, avg and statistics aggregates

2015-01-02 Thread Jim Nasby
?), and changing docs as needed? Presumably that would be best as a separate patch... -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Final Patch for GROUPING SETS

2015-01-02 Thread Jim Nasby
simultaneously? That would allow things like GroupAggregate -- Sort(a) \ +-- Sort(a,b) -\ -- Hash(b) + \-- SeqScan That would allow the planner to trade off things like total memory consumption vs IO. -- Jim Nasby, Data Architect, Blue Treble

Re: [HACKERS] Using 128-bit integers for sum, avg and statistics aggregates

2015-01-02 Thread Jim Nasby
; that would come later. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Jim Nasby
things itself? Or perhaps some custom command file that could then be replayed by pg_upgrade on another server? Of course, that's assuming that replicas are compatible enough with masters for that to work... -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Jim Nasby
SUPERUSER NOINHERIT NOLOGIN; CREATE ROLE su_role IN ROLE su NOLOGIN; GRANT su_role TO bob; and have su_role:*:* Does bob get audited all the time then? Only when he does SET ROLE su? For that matter, how does SET ROLE affect auditing? -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Jim Nasby
there isn't really an issue here; I suspect ADAT is very rarely used. What I'm worried about though is that someone is using it regularly for some reason, with non-trivial databases, and this is going to completely hose them. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Jim Nasby
to get the design pulled together. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel Seq Scan

2015-01-23 Thread Jim Nasby
and 8 workers make no difference, then suddenly 16 cuts times by 1/2 to 1/3? Then 32 cuts time by another 1/2 to 1/3? -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Jim Nasby
, but allow the user to over-ride that. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] proposal: row_to_array function

2015-01-26 Thread Jim Nasby
, what about: foreach a,b in '{{{1,2},{3,4}},{{5,6},{7,8}}}'::int[] ? -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-26 Thread Jim Nasby
, it should be more (or differently) restrictive than SU, so that you can effectively audit your superusers with minimal worries about superusers tampering with auditing. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql

Re: [HACKERS] Parallel Seq Scan

2015-01-26 Thread Jim Nasby
worker reading the index and another worker taking index tuples and reading heap tuples... -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] proposal: searching in array function - array_position

2015-01-26 Thread Jim Nasby
the operator caching code? You should remove the array_length() from the last array_offsets test; I don't see that it buys anything. I think there should be tests for what happens when you feed these functions a multi-dimensional array. Other than that, looks good. -- Jim Nasby, Data Architect

Re: [HACKERS] pg_upgrade and rsync

2015-01-26 Thread Jim Nasby
that would be any different from an rsync --size-only in the end, presuming everything went according to plan. Yeah, if everything is shut down maybe we're OK. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers

Re: Partitioning: issues/ideas (Was: Re: [HACKERS] On partitioning)

2015-01-26 Thread Jim Nasby
). Or maybe I'm missing something... Wasn't the idea that the parent table in a partitioned table wouldn't actually have a heap of it's own? If there's no heap there can't be an index. That said, I think this is premature optimization that could be done later. -- Jim Nasby, Data Architect, Blue

Re: [HACKERS] PL/pgSQL, RAISE and error context

2015-01-26 Thread Jim Nasby
be better to just do this per-subtransaction. I do agree that this needs to work across the board, not just for plpgsql. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

<    4   5   6   7   8   9   10   11   12   13   >