Re: [HACKERS] Test code is worth the space

2015-08-12 Thread Peter Geoghegan
be. I think that once the facility is there, we can determine over time how expansive that second category gets to be. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] How to compile, link and use a C++ extension

2015-08-14 Thread Peter Geoghegan
RC they've made specific trade-offs that might be useful for Jacques' use case too. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] More WITH

2015-08-17 Thread Peter Geoghegan
the top-level. Having it work any other way would require significant refactoring. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] missing documentation for partial WAL files

2015-08-17 Thread Peter Geoghegan
. > We should probably add something. Uh, some documentation around .ready files would be nice too. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] jsonb array-style subscripting

2015-08-17 Thread Peter Geoghegan
On Mon, Aug 17, 2015 at 12:26 PM, Merlin Moncure wrote: > I'm not sure if this: > update test_jsonb_subscript set test_json['a']['a1']['a2'] = 42; > > ...is a good idea. This kind of "targetlist indirection" is already possible with arrays

Re: [HACKERS] jsonb array-style subscripting

2015-08-17 Thread Peter Geoghegan
k, but certain things do seem to suggest themselves. For example, maybe the parser can be made to create a query tree that uses an indexable operator based on special-case logic. Although maybe that's a kludge too far, since I can imagine it breaking other legitimate things. My sense is that this w

Re: [HACKERS] Our trial to TPC-DS but optimizer made unreasonable plan

2015-08-19 Thread Peter Geoghegan
ses for third-party benchmarks than I am about the problems for real users. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Using quicksort for every external sort run

2015-08-19 Thread Peter Geoghegan
ought to balance certain things. Assuming a one-pass merge step, then we should balance the time lost waiting on the first run and time quicksorting the last run with the gradual increase in the cost during the merge step. Maybe the non-use of abbreviated keys during the merge step should also be co

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Peter Geoghegan
che oblivious, and that's an enormous advantage. This was before abbreviated keys; these days, the difference must be larger. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Peter Geoghegan
On Thu, Aug 20, 2015 at 8:15 AM, Simon Riggs wrote: > On 20 August 2015 at 03:24, Peter Geoghegan wrote: >> >> >> The patch is ~3.25x faster than master > > > I've tried to read this post twice and both times my work_mem overflowed. > ;-) > > Can you

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Peter Geoghegan
nsure that, then you probably have a mostly I/O bound workload, which can be made faster by adding sequential I/O capacity (or, on the Postgres internals side, adding asynchronous I/O, or with memory prefetching). You cannot really buy a faster CPU to make a degenerate heapsort faster. -- Peter Geog

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Peter Geoghegan
factor in unrepresentative cases, as I mentioned, but even then it's not going to be a dramatic difference as it would have been in the past. By the way, please don't top-post. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Peter Geoghegan
On Thu, Aug 20, 2015 at 6:05 AM, Greg Stark wrote: > On Thu, Aug 20, 2015 at 3:24 AM, Peter Geoghegan wrote: >> I believe, in general, that we should consider a multi-pass sort to be >> a kind of inherently suspect thing these days, in the same way that >> checkpoints occu

Re: [HACKERS] 9.5 release notes

2015-08-20 Thread Peter Geoghegan
On Sat, Jun 13, 2015 at 3:53 PM, Peter Geoghegan wrote: > I think we should really address this. Attached patch adds a new > release note item for it. It also adds to the documentation that > explains why users should prefer varchar(n)/text to character(n); the > lack of abbreviated

Re: [HACKERS] Using quicksort for every external sort run

2015-08-20 Thread Peter Geoghegan
EC2 instance with 244GB memory, and lots of SSDs. It seems like the point of the 100TB minimum rule in the "Gray" contest category is that that's practically impossible to fit entirely in memory (to avoid merging). Eventually, linearithmic growth becomes extremely painful, not matter ho

Re: [HACKERS] Using quicksort for every external sort run

2015-08-21 Thread Peter Geoghegan
Besides, you need to partition the input anyway, and external sorting naturally needs to do that while not precluding runs not actually being dumped to disk. [1] http://queue.acm.org/detail.cfm?id=1814327 -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 9.5 release notes

2015-08-26 Thread Peter Geoghegan
On Wed, Aug 26, 2015 at 7:33 AM, Bruce Momjian wrote: > I have applied the attached patch to document this in the data type docs. Thanks, but shouldn't varchar/text also be mentioned in the release notes, rather than "character fields"? -- Peter Geoghegan -- Sent via pgs

Re: [HACKERS] 9.5 release notes

2015-08-26 Thread Peter Geoghegan
On Wed, Aug 26, 2015 at 11:58 AM, Bruce Momjian wrote: > I am sorry it so long for me to address this. Peter brought it up in > June, but I just wasn't around to address it cleanly before now. I am > glad he reminded me. Well, you got around to it eventually. Thanks. -- P

Re: [HACKERS] Our trial to TPC-DS but optimizer made unreasonable plan

2015-08-26 Thread Peter Geoghegan
equality operator (think HOT style checks). That would have the advantage of requiring no per-type/operator class support at all, since it's safe to assume that binary equality is a proxy for "equivalence" of sort order (or whatever we call the case where 5.00::numeric and 5.000::nu

Re: [HACKERS] pg_stat_statements query jumbling question

2015-08-31 Thread Peter Geoghegan
ts does not change the representative query string, renaming a table does not make it a substantively different table. There is, IIRC, one case where a string is jumbled directly (CTE name). It's usually not the right thing, IMV. -- Peter Geoghegan -- Sent via pgsql-hackers mailing lis

Re: [HACKERS] pg_stat_statements query jumbling question

2015-08-31 Thread Peter Geoghegan
e to be even a thing that pg_stat_statements theoretically has the right to write into. Rather, it just asks the core system to do the fingerprinting, and finds it within queryId. At the same time, other extensions may do the same, and don't need to care about each other. Does that work f

Re: [HACKERS] [PROPOSAL] Effective storage of duplicates in B-tree index.

2015-09-01 Thread Peter Geoghegan
7;t see it happening soon. It isn't worth compressing unless it compresses enough to cross an "alignment boundary", where we're not actually obliged to store as much data on disk. This point may be obvious, not sure. [1] http://www.postgresql.org/message-id/flat/cam3swzt-t9o_dchk8e4_yb

Re: [HACKERS] 9.4 broken on alpha

2015-09-01 Thread Peter Geoghegan
evere bugs (just like Postgres 9.4's Alpha support). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Horizontal scalability/sharding

2015-09-01 Thread Peter Geoghegan
tes >> * ability to manage large RAM and data caches > > I do agree that all of those things are problems, FWIW. These seem like our long term problems, to me. [1] http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf, page 213 -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Memory prefetching while sequentially fetching from SortTuple array, tuplestore

2015-09-02 Thread Peter Geoghegan
On Wed, Sep 2, 2015 at 7:12 AM, Andres Freund wrote: > On 2015-07-19 16:34:52 -0700, Peter Geoghegan wrote: > Hm. Is a compiler test actually test anything reliably here? Won't this > just throw a warning during compile time about an unknown function? I'll nee

Re: [HACKERS] ON CONFLICT DO UPDATE using EXCLUDED.column gives an error about mismatched types

2015-09-02 Thread Peter Geoghegan
On Wed, Sep 2, 2015 at 1:18 AM, Amit Langote wrote: > Did you get around to making a patch for this? I've worked on it inconsistently. I'll pick this up again soon. I may take the opportunity to talk this over with Andres in person when we meet at Postgres Open shortly. -- Pe

Re: [HACKERS] Memory prefetching while sequentially fetching from SortTuple array, tuplestore

2015-09-02 Thread Peter Geoghegan
daries in the cases addressed by the patch, it isn't hard to see why the compiler does not do this automatically (prefetch instructions added by the compiler are not common anyway, IIRC). The compiler has no way of knowing that gettuple_common() is ultimately called from an important inner loop,

Re: [HACKERS] Memory prefetching while sequentially fetching from SortTuple array, tuplestore

2015-09-02 Thread Peter Geoghegan
way. > I'd be interested to see a perf stat -ddd comparison to the patch > with/without prefetches. It'll be interesting to see how the number of > cache hits/misses and prefetches changes. > > Which microarchitecture did you test this on? My laptop has an Intel Core i7-3520M, w

Re: [HACKERS] Memory prefetching while sequentially fetching from SortTuple array, tuplestore

2015-09-02 Thread Peter Geoghegan
suppose I copied that pattern here. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Memory prefetching while sequentially fetching from SortTuple array, tuplestore

2015-09-02 Thread Peter Geoghegan
On Wed, Sep 2, 2015 at 12:24 PM, Peter Geoghegan wrote: > On Wed, Sep 2, 2015 at 7:12 AM, Andres Freund wrote: >> On 2015-07-19 16:34:52 -0700, Peter Geoghegan wrote: >> Hm. Is a compiler test actually test anything reliably here? Won't this >> just throw a warning du

Re: [HACKERS] Memory prefetching while sequentially fetching from SortTuple array, tuplestore

2015-09-02 Thread Peter Geoghegan
see an improvement on a variety of other cases, although the Intel vTune tool made the difference particularly obvious. The only thing that definitely won't be helped is pass-by-value datum sort cases. In case it matters, I used GCC 4.8. -- Peter Geoghegan -- Sent via pgsql-hackers maili

Re: [HACKERS] Using quicksort for every external sort run

2015-09-06 Thread Peter Geoghegan
robably more than a simpler case involving only one CREATE INDEX), because each core would be require fewer main memory accesses overall. Maybe you can test it and let us know how it goes. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

[HACKERS] Should TIDs be typbyval = FLOAT8PASSBYVAL to speed up CREATE INDEX CONCURRENTLY?

2015-09-07 Thread Peter Geoghegan
seems best. I'm more than a little busy at the moment, so I would be happy for someone else to write the patch. It seems worthwhile. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] Should TIDs be typbyval = FLOAT8PASSBYVAL to speed up CREATE INDEX CONCURRENTLY?

2015-09-07 Thread Peter Geoghegan
eally only needs to happen in one place. I hope someone picks this up soon. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-14 Thread Peter Geoghegan
On Mon, Jan 13, 2014 at 6:45 PM, Peter Geoghegan wrote: > + uint32 > + SpeculativeInsertionIsInProgress(TransactionId xid, RelFileNode rel, > ItemPointer tid) > + { For the purposes of preventing unprincipled deadlocking, commenting out the following (the only caller of the a

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-14 Thread Peter Geoghegan
red. Also, upserting with exclusion constraints is a non-starter. It's only applicable to the case where you're using exclusion constraints exactly as you would use unique constraints, which has to be very rare. It will cause much more confusion than anything else. INSERT IGNORE in

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-14 Thread Peter Geoghegan
is proposed here, because exclusion constraints only ever have to abort the xact if things don't work out. There is no value locking. That's far easier to pin down. You definitely don't have to do anything new with visibility. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-15 Thread Peter Geoghegan
onclusively a proper (non-promise) tuple. Those should always come last. And even without upsert, strictly inserting into unique indexes first seems like a useful thing relative to the cost. Unique violations are the cause of many aborted transactions, and there is no need to ever bloat non-unique ind

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-15 Thread Peter Geoghegan
make both patches livelock, for reasons that I believe are related: https://github.com/petergeoghegan/upsert/blob/master/torture.sh I have an idea of what I could do to fix this, but I don't have time to make sure that my hunch is correct. I'm travelling tomorrow to give

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-15 Thread Peter Geoghegan
On Wed, Jan 15, 2014 at 8:23 PM, Peter Geoghegan wrote: > I have an idea of what I could do to fix this, but I don't have time > to make sure that my hunch is correct. It might just be a matter of: @@ -186,6 +186,13 @@ ExecLockHeapTupleForUpdateSpec(EState *estate, s

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-16 Thread Peter Geoghegan
set abalance = pgbench_accounts.abalance + rej.abalance from rej where pgbench_accounts.aid = rej.aid; (This benchmark used an unlogged table, if only because to do otherwise would severely starve this particular server of I/O). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (

Re: [HACKERS] Backup throttling

2014-01-16 Thread Peter Geoghegan
every process should use the process latch, except for the archiver which uses a local latch because it pointedly does not touch shared memory. I think I wrote a comment that made it into the latch header comments encouraging this, but never saw to it that it was universally adhered to. --

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-16 Thread Peter Geoghegan
On Wed, Jan 15, 2014 at 11:02 PM, Peter Geoghegan wrote: > It might just be a matter of: > > @@ -186,6 +186,13 @@ ExecLockHeapTupleForUpdateSpec(EState *estate, > switch (test) > { > case

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-18 Thread Peter Geoghegan
implementation easier. I was surprised when I learned that things didn't already work this way. Attached patch, broken off from my patch has relcache sort indexes by (!indisunique, relindexid). -- Peter Geoghegan *** a/src/backend/utils/cache/relcache.c --- b/src/backend/utils/cache/relca

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-18 Thread Peter Geoghegan
On Thu, Jan 16, 2014 at 6:31 PM, Peter Geoghegan wrote: > I think we need to give this some more thought. I have not addressed > the implications for MVCC snapshots here. So I gave this some more thought, and this is what I came up with: + static bool + ExecLockHeapTupleForUpdateSpec(

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-18 Thread Peter Geoghegan
On Sat, Jan 18, 2014 at 6:17 PM, Peter Geoghegan wrote: > MySQL users are not notified that this happened, and are probably > blissfully unaware that there has been a limited form of data loss. So > it's The Right Thing to say to Postgres users: "if you inserted these > ro

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-18 Thread Peter Geoghegan
On Sat, Jan 18, 2014 at 7:49 PM, Peter Geoghegan wrote: > Personally, I favor just making "case HeapTupleSelfUpdated:" within > the patch's ExecLockHeapTupleForUpdateSpec() function complain when > "hufd.cmax == estate->es_output_cid)" (currently there is

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-20 Thread Peter Geoghegan
going to break in > future. What we need to do is embed the API version in the C name of the > pg_stat_statements function instead. I agree that it isn't scalable. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Performance Improvement by reducing WAL for Update Operation

2014-01-20 Thread Peter Geoghegan
O for free then let's get decreased I/O in > exchange for increased CPU usage". While I haven't been following the development of this patch, I will note that on the face of it the latter seem like a trade-off I'd be quite willing to make. -- Peter Geoghegan -- Sent via

Re: [HACKERS] Funny representation in pg_stat_statements.query.

2014-01-21 Thread Peter Geoghegan
that was addressed by this commit: http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=5d3fcc4c2e137417ef470d604fee5e452b22f6a7 -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-21 Thread Peter Geoghegan
third-party tools. I am very wary of enlarging the counters structure, because it is protected by a spinlock. There has been no attempt to quantify that cost, nor has anyone even theorized that it is not likely to be appreciable. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2014-01-21 Thread Peter Geoghegan
scheme, perhaps XLogInsert() could differentiate between a soft and hard failure. Hopefully the reserve function you mentioned, which is still called at the same place, just before each critical section thereby becomes cheap. Perhaps I'm just restating what you said, though. -- Peter Geoghegan

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2014-01-21 Thread Peter Geoghegan
nt probability of 1.0. People are not going to develop a lackadaisical attitude about running out of disk space on the pg_xlog partition if we do so. They still have plenty of incentive to make sure that that doesn't happen. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-h

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-21 Thread Peter Geoghegan
bed by the standard as flushing its stream. It is my understanding that calling fflush() on a stream immediately prior to calling fclose() on the same stream is always superfluous. > We could also eliminate some palloc/pfree cycles by using the string > directly in the buffer (at the small

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-21 Thread Peter Geoghegan
On Tue, Jan 21, 2014 at 8:01 PM, Peter Geoghegan wrote: > Not necessarily. Under Linux for example, POSIX_FADV_SEQUENTIAL "sets > the readahead window to the default size for the backing device" Excuse me; I meant to put "POSIX_FADV_SEQUENTIAL doubles this size [default

Re: [HACKERS] Funny representation in pg_stat_statements.query.

2014-01-21 Thread Peter Geoghegan
ete failure to normalize a query string (it is a limitation of pg_stat_statements that this can happen with very low probability). It's a bit of a kludge, but the query string is always just strictly speaking a convenience to help the user understand what the entry is. Although, I suppose, most p

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-22 Thread Peter Geoghegan
On Tue, Jan 21, 2014 at 8:01 PM, Peter Geoghegan wrote: >> Actually, I think the whole thing is rather badly designed, as warm >> cache or no you're still proposing to do thousands of kernel calls >> while holding a potentially contended LWLock. I suggest what we >>

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-22 Thread Peter Geoghegan
On Tue, Jan 21, 2014 at 8:01 PM, Peter Geoghegan wrote: >> BTW shouldn't there be an fflush() in qtext_store? > > I don't think so, no. Whenever qtext_store() callers don't fflush() > before releasing their exclusive lock, they FreeFile() before doin

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-22 Thread Peter Geoghegan
atements() and pg_stat_statements_reset(). The latter was sometimes rigged to do a direct garbage collection to stress test things. My expectation was that the sanity checking done everywhere would complain if there were any race conditions or other bugs. This was pretty effective as a smoke test d

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-22 Thread Peter Geoghegan
re only executed once or twice and yet there are enough of them. I think this is generally useful for testing. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-22 Thread Peter Geoghegan
what we do for current counter fields instead) and see how much difference it makes. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Failure while inserting parent tuple to B-tree is not fun

2014-01-23 Thread Peter Geoghegan
sure that I like the way that that and similar loops get "stuck on" page splits, although no better design occurs to me right now. That's all I have for now. I've written plenty of notes, and will work back through other points of possible concern. I don't suppose you have any testing infrastructure that you could publish? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe

2014-01-23 Thread Peter Geoghegan
[-Werror=declaration-after-statement]". -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-25 Thread Peter Geoghegan
ed_query() reruns the > core lexer on the given statement, and if you got a warning the first > time, you'll get another one. Oh, yes, I noticed that and reached the same conclusion. Sorry, I probably should have mentioned this pro-actively. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-25 Thread Peter Geoghegan
it could possibly matter, even though it seems like there might be one. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-25 Thread Peter Geoghegan
Why do you think it's better to release the shared lock while generating a normalized query text, only to acquire it once more? I'm not suggesting that it's the wrong thing to do. I'm curious about the reasoning around assessing the costs. -- Peter Geoghegan -- Sent via p

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-25 Thread Peter Geoghegan
wice for something like that struck me as unjustified. I though it was closer to the original coding to lex with the lock, because of course the original coding will only ever acquire the shared lock once. The original lex-with-no-lock coding is only justified by "well, might as well". -- Peter

Re: [HACKERS] Freezing without write I/O

2014-01-25 Thread Peter Geoghegan
Shouldn't this patch be in the January commitfest? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-26 Thread Peter Geoghegan
this was something I felt strongly about. I'm glad that the overhead has been shown to be quite low, and I think that lexing without the lock held will be fine. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscrip

Re: [HACKERS] Failure while inserting parent tuple to B-tree is not fun

2014-01-27 Thread Peter Geoghegan
u very probably really do >> make progress not limited to completing a page split, the occurrence >> of which the _bt_moveright() loop gets "stuck on"). You know, do what >> is done in the non-split-handling case. It may not always be the same >> buff

Re: [HACKERS] Failure while inserting parent tuple to B-tree is not fun

2014-01-27 Thread Peter Geoghegan
lding an accurate mental model of things is a very good thing. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe

2014-01-27 Thread Peter Geoghegan
On Mon, Jan 27, 2014 at 12:25 PM, Robert Haas wrote: > I haven't reviewed the patch, but -1 for adding a GUC. I'm pretty surprised that it's been suggested that some people might prefer AccessExclusiveLocks. Why would anyone prefer that? -- Peter Geoghegan -- Sent via pgs

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-27 Thread Peter Geoghegan
On Mon, Jan 27, 2014 at 12:39 PM, Tom Lane wrote: > OK. Committed after a couple of small further revisions. Great. Thank you. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailp

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-27 Thread Peter Geoghegan
without a second thought. That's hugely valuable. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-27 Thread Peter Geoghegan
ave. Oh, I see. Of course the overhead of calling the pg_stat_statements() function is much less important. Actually, I think that calling that function is going to add a lot of noise to any benchmark aimed at measuring added overhead as the counters struct is expanded. -- Peter Geoghegan -- Sen

Re: [HACKERS] Race condition in b-tree page deletion

2014-01-27 Thread Peter Geoghegan
_bt_doinsert() say at one point: * This routine is called by the public interface routines, btbuild * and btinsert. This is obsolete; since commit 89bda95d, only the insert public interface routine calls _bt_doinsert(). Perhaps fix this in passing. [1] L&S: ht

[HACKERS] Re: [COMMITTERS] pgsql: Keep pg_stat_statements' query texts in a file, not in shared me

2014-01-27 Thread Peter Geoghegan
Then I will post my response. On Mon, Jan 27, 2014 at 8:54 PM, Peter Geoghegan wrote: > On Mon, Jan 27, 2014 at 8:20 PM, KONDO Mitsumasa > wrote: >> At least, only postgres superuser can see pg_stat_statemnet view in old >> version. >> And you should change document a

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-28 Thread Peter Geoghegan
I noticed a minor omission in the patch as committed. Attached patch corrects this. -- Peter Geoghegan *** a/contrib/pg_stat_statements/pg_stat_statements.c --- b/contrib/pg_stat_statements/pg_stat_statements.c *** generate_normalized_query(pgssJumbleStat *** 2726,2732 if

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-01-28 Thread Peter Geoghegan
0002-Very-basic-atomic-ops-implementation.patch. Please rebase. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-28 Thread Peter Geoghegan
ruct should be required to very conclusively demonstrate that there is no appreciably increase in overhead. Having a standard deviation column would be nice, but it's still not that important. Maybe when we have portable atomic addition we can afford to be much more inclusive of that kind of thin

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-29 Thread Peter Geoghegan
s, since I raised this concern about 3 months ago. [1] http://www.postgresql.org/message-id/52e10e6a.5060...@lab.ntt.co.jp -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Failure while inserting parent tuple to B-tree is not fun

2014-01-29 Thread Peter Geoghegan
On Mon, Jan 27, 2014 at 10:54 AM, Peter Geoghegan wrote: > On Mon, Jan 27, 2014 at 10:27 AM, Heikki Linnakangas > wrote: >>> I think I see some bugs in _bt_moveright(). If you examine >>> _bt_finish_split() in detail, you'll see that it doesn't just drop the >

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-30 Thread Peter Geoghegan
ect on performance by relieving cache pressure and thereby causing less exclusive locking for an entry_dealloc(). Do you suppose that this very surprising result might just be because this isn't a very representative benchmark? Nothing ever has the benefit of *not* having to exclusive l

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-30 Thread Peter Geoghegan
't, they can double or triple their "max" setting, because the shared memory overhead is so low. No one has any additional overhead once their query is in the hash table already. In reality, actual applications could hardly be further from the perfectly uniform distribution of disti

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-30 Thread Peter Geoghegan
away at little cost (a lower setting is better, but a setting high enough to cache everything is best). But you're not going to have terribly much use for pg_stat_statements anywayif you really do experience churn at that rate with 5,000 possible entries, the module is ipso facto useless,

[HACKERS] Making strxfrm() blobs in indexes work

2014-01-30 Thread Peter Geoghegan
might not be that bad in larger indexes. My guess is that on the whole it would be well worth it. This is obviously pretty hand-wavy, but I thought it was still worth sharing while the basic idea was fresh in my mind. Thoughts? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Making strxfrm() blobs in indexes work

2014-01-30 Thread Peter Geoghegan
indexes too badly, and >> without breaking anything? We only use inner pages to find leaf pages. >> They're redundant copies of the data within the index. > > It's a cute idea though, and perhaps worth pursuing as long as you've > got the pitfalls in mind. I

Re: [HACKERS] Making strxfrm() blobs in indexes work

2014-01-30 Thread Peter Geoghegan
On Thu, Jan 30, 2014 at 4:45 PM, Peter Geoghegan wrote: > So we consider the > appropriateness of a regular strcoll() or a strxfrm() in all contexts > (in a generic and extensible manner, but that's essentially what we > do). I'm not too discouraged by this restriction, be

Re: [HACKERS] Making strxfrm() blobs in indexes work

2014-01-30 Thread Peter Geoghegan
t per column, no? I don't think so. It would be no big deal if it was all or nothing. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Making strxfrm() blobs in indexes work

2014-01-30 Thread Peter Geoghegan
On Thu, Jan 30, 2014 at 3:49 PM, Peter Geoghegan wrote: > So ISTM that we could come up with an infrastructure, possibly just > for insertion scanKeys (limiting the code footprint of all of this) in > order to inner-page-process datums at this juncture, and store a blob > instea

Re: [HACKERS] Making strxfrm() blobs in indexes work

2014-01-30 Thread Peter Geoghegan
On Thu, Jan 30, 2014 at 5:05 PM, Peter Geoghegan wrote: > On Thu, Jan 30, 2014 at 5:04 PM, Tom Lane wrote: >>> That's not hard to prevent. If that should happen, we don't go with >>> the strxfrm() datum. We have a spare IndexTuple bit we could use to >>>

Re: [HACKERS] Prefix compression of B-Tree keys

2014-01-30 Thread Peter Geoghegan
hat offhand I think that it'd be harder to come up with a very generic infrastructure to make it work across diverse types, and it isn't that useful where there isn't much redundancy in prefixes. The B-Tree code doesn't really know anything about the type indexed, and that's a use

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-31 Thread Peter Geoghegan
gn much weight to any test case where the large majority or all statistics are wrong afterwards, due to there being so much churn. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-01-31 Thread Peter Geoghegan
r client counts: http://postgres-benchmarks.s3-website-us-east-1.amazonaws.com/rwlock-contention/ This is exactly what you said would happen, but at least now we have a sense of that cost. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chang

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-01 Thread Peter Geoghegan
rovement in transaction throughput. It's not as large as the improvement you reported for your 2 socket workstation, but it's there, just about. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-01 Thread Peter Geoghegan
e. [1] http://www.postgresql.org/message-id/20131115194725.gg5...@awork2.anarazel.de -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Wait free LW_SHARED acquisition - v0.2

2014-02-01 Thread Peter Geoghegan
s-frontend" and "stalled-cycles-backend" Linux perf events were at huge variance depending on whether these Intel systems were patched or unpatched. They were about the same on the AMD system to begin with. [1] http://www.intel.com/content/dam/www/public/us/en/documents/white-paper

Re: [HACKERS] [COMMITTERS] pgsql: Include planning time in EXPLAIN ANALYZE output.

2014-02-02 Thread Peter Geoghegan
On Sun, Feb 2, 2014 at 8:13 AM, Tom Lane wrote: > Perhaps s/Total runtime/Execution time/ ? +1 -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Making strxfrm() blobs in indexes work

2014-02-02 Thread Peter Geoghegan
On Thu, Jan 30, 2014 at 8:51 PM, Peter Geoghegan wrote: > I've done some more digging. It turns out that the 1977 paper "An > Encoding Method for Multifield Sorting and Indexing" describes a > technique that involves concatenating multiple column values and > comparing

Re: [HACKERS] narwhal and PGDLLIMPORT

2014-02-02 Thread Peter Geoghegan
finally come around to implementing a few C99 features in Visual Studio 2013 precisely because they want there to be an open source ecosystem on Windows. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

<    5   6   7   8   9   10   11   12   13   14   >