Re: [HACKERS] Performance optimization of btree binary search

2013-12-04 Thread Peter Geoghegan
On Wed, Dec 4, 2013 at 5:28 PM, Peter Geoghegan p...@heroku.com wrote: I'm also curious about the impact on insertion into primary key indexes. Presently, we hold an exclusive buffer lock for the duration of a couple of operations when checkUnique != UNIQUE_CHECK_NO. _bt_binsrch() is one

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Peter Geoghegan
the fseek() scalability issue? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Peter Geoghegan
On Wed, Dec 4, 2013 at 5:28 PM, Peter Geoghegan p...@heroku.com wrote: I'm also curious about the impact on insertion into primary key indexes. Presently, we hold an exclusive buffer lock for the duration of a couple of operations when checkUnique != UNIQUE_CHECK_NO. _bt_binsrch() is one

Re: [HACKERS] ANALYZE sampling is too good

2013-12-05 Thread Peter Geoghegan
scan really is very expensive, would it be so unreasonable to attempt to amortize that cost? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_stat_statements: calls under-estimation propagation

2013-12-06 Thread Peter Geoghegan
. -- Peter Geoghegan diff --git a/contrib/pg_stat_statements/Makefile b/contrib/pg_stat_statements/Makefile new file mode 100644 index e8aed61..95a2767 *** a/contrib/pg_stat_statements/Makefile --- b/contrib/pg_stat_statements/Makefile *** MODULE_big = pg_stat_statements *** 4,11

Re: [HACKERS] Performance optimization of btree binary search

2013-12-06 Thread Peter Geoghegan
On Thu, Dec 5, 2013 at 2:19 AM, Peter Geoghegan p...@heroku.com wrote: I did a relatively short variant 'insert' pgbench-tools benchmark, with a serial primary index created on the pgbench_history table. Results: http://postgres-benchmarks.s3-website-us-east-1.amazonaws.com/insert/ I saw

Re: [HACKERS] pg_stat_statements: calls under-estimation propagation

2013-12-07 Thread Peter Geoghegan
On Sat, Dec 7, 2013 at 3:50 PM, Peter Eisentraut pete...@gmx.net wrote: 32-bit buildfarm members are having problems with this patch. This should fix that problem. Thanks. -- Peter Geoghegan diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements

Re: [HACKERS] Performance optimization of btree binary search

2013-12-09 Thread Peter Geoghegan
On Fri, Dec 6, 2013 at 4:53 PM, Peter Geoghegan p...@heroku.com wrote: I had considered that something like Intel Speedstep technology had a role here, but I'm pretty sure it steps up very aggressively when things are CPU bound - I tested that against a Core 2 Duo desktop a couple of years

Re: [HACKERS] ANALYZE sampling is too good

2013-12-09 Thread Peter Geoghegan
On Mon, Dec 9, 2013 at 1:18 PM, Jeff Janes jeff.ja...@gmail.com wrote: I don't recall ever tracing a bad plan down to a bad n_distinct. It does happen. I've seen it several times. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2013-12-09 Thread Peter Geoghegan
-0cpwzxvqh49reybchb95t95fcrgs...@mail.gmail.com -- Peter Geoghegan pg_stat_statements_ext_text.v5.2013_12_09.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2013-12-09 Thread Peter Geoghegan
On Mon, Dec 9, 2013 at 7:31 PM, Peter Geoghegan p...@heroku.com wrote: I go to some lengths to to avoid doing this with only a shared lock. I should have said: I go to great lengths to do this with only a shared lock, and not an exclusive (see gc_count stuff). -- Peter Geoghegan -- Sent

[HACKERS] pg_stat_statements fingerprinting logic and ArrayExpr

2013-12-10 Thread Peter Geoghegan
I'm pretty sure that tools that do regex normalization are already doing something analogous. Thoughts? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ANALYZE sampling is too good

2013-12-10 Thread Peter Geoghegan
of ameliorating the cost. Back in 2005/6, I advocated a block sampling method, as described by Chaudri et al (ref?) I don't think that anyone believes that not doing block sampling is tenable, fwiw. Clearly some type of block sampling would be preferable for most or all purposes. -- Peter Geoghegan

Re: [HACKERS] ANALYZE sampling is too good

2013-12-10 Thread Peter Geoghegan
. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_stat_statements fingerprinting logic and ArrayExpr

2013-12-10 Thread Peter Geoghegan
either. I do think I know who should determine what is the essential nature of a query for fingerprinting purposes: we should. We should pick the scheme that is most widely useful, while weighing the worst case. I'm not asserting that this is closer to that, but it might be. -- Peter Geoghegan

Re: [HACKERS] pg_stat_statements fingerprinting logic and ArrayExpr

2013-12-10 Thread Peter Geoghegan
that keen on doing a lot of work that will only be applicable to Postgres. Did you really find pg_stat_statements to be almost useless in such situations? That seems worse than I thought. I guess it in no small part comes down to what the long term future of the query finger-printer is. -- Peter

Re: [HACKERS] pg_stat_statements fingerprinting logic and ArrayExpr

2013-12-10 Thread Peter Geoghegan
we need to acquire an exclusive lock to do so. If there was a lot of churn, I'd worry that the performance overhead of pg_stat_statements. It could be quite a lot higher than necessary. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] pg_stat_statements fingerprinting logic and ArrayExpr

2013-12-10 Thread Peter Geoghegan
not. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_stat_statements fingerprinting logic and ArrayExpr

2013-12-10 Thread Peter Geoghegan
On Tue, Dec 10, 2013 at 2:55 PM, Peter Geoghegan p...@heroku.com wrote: You might get lucky and have this exact case, and be able to leverage the knowledge that the 2 constants in the ArrayExpr must be the latter and 1 must be the former, but experience suggests very probably not. When things

Re: [HACKERS] ANALYZE sampling is too good

2013-12-10 Thread Peter Geoghegan
point us in the right direction. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_stat_statements fingerprinting logic and ArrayExpr

2013-12-10 Thread Peter Geoghegan
, but that's hardly a scalable model. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ANALYZE sampling is too good

2013-12-10 Thread Peter Geoghegan
into mitigating and/or detecting the downsides of block-based sampling. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ANALYZE sampling is too good

2013-12-10 Thread Peter Geoghegan
that paper, because undoubtedly it's all patented. But before I figured that out, after finding it on Google randomly, I did read the first couple of paragraphs, which more or less said what follows - the entire paper - is an explanation as to why it's okay that we do block sampling. -- Peter

Re: [HACKERS] autovacuum_work_mem

2013-12-11 Thread Peter Geoghegan
the time and skills available. I think that that's a very optimistic assessment of the amount of work required. Even by the rose-tinted standards of software project time estimation. A ton of data is required to justify fundamental infrastructural changes like that. -- Peter Geoghegan -- Sent via

Re: [HACKERS] ANALYZE sampling is too good

2013-12-11 Thread Peter Geoghegan
On Tue, Dec 10, 2013 at 4:48 PM, Peter Geoghegan p...@heroku.com wrote: Why would I even mention that to a statistician? We want guidance. But yes, I bet I could give a statistician an explanation of statistics target that they'd understand without too much trouble. Actually, I think

Re: [HACKERS] autovacuum_work_mem

2013-12-11 Thread Peter Geoghegan
, and sometimes I forget to do so, particularly when under pressure to relieve a production performance issues on a random customer's database. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] Why standby.max_connections must be higher than primary.max_connections?

2013-12-11 Thread Peter Geoghegan
.2ndQuadrant I'd thought about revisiting this myself, but I think that the impetus to do so is lessened by recent work on logical replication. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-12 Thread Peter Geoghegan
dropped the ball on this. I'm doing a bit more testing of an approach to fixing the new bugs. I'll let you know how I get on tomorrow (later today for you). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] stuck spinlock

2013-12-12 Thread Peter Geoghegan
On Thu, Dec 12, 2013 at 3:33 PM, Andres Freund and...@2ndquadrant.com wrote: Any other changes but the upgrade? Maybe a different compiler version? Show pg_config output. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] stuck spinlock

2013-12-12 Thread Peter Geoghegan
this is in a production debugging situation. It seems like an interesting way of debugging these sorts of issues that should be explored and perhaps subsequently codified. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] stuck spinlock

2013-12-12 Thread Peter Geoghegan
be the app setting it locally? In fact, isn't that the recommended usage? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-14 Thread Peter Geoghegan
for the indexam API spec, since you're changing that significantly. I'll certainly do that in any future revision. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-18 Thread Peter Geoghegan
On Thu, Dec 12, 2013 at 4:18 PM, Peter Geoghegan p...@heroku.com wrote: Both of these revisions have identical ad-hoc test cases included as new files - see testcase.sh and upsert.sql. My patch doesn't have any unique constraint violations, and has pretty consistent performance, while yours

Re: [HACKERS] gaussian distribution pgbench

2013-12-19 Thread Peter Geoghegan
normal distributions abound in nature. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-20 Thread Peter Geoghegan
, but it is practical to do in a localized way as problems emerge. In contrast, if we allowed unprincipled deadlocking, the only advice we could give is stop doing so much upserting. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-21 Thread Peter Geoghegan
...@mail.gmail.com -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-22 Thread Peter Geoghegan
On Fri, Dec 20, 2013 at 11:59 PM, Peter Geoghegan p...@heroku.com wrote: I think that the way forward is to refine my design in order to upgrade locks from exclusive buffer locks to something else, managed by the lock manager but perhaps through an additional layer of indirection

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-23 Thread Peter Geoghegan
). I have been trying to focus attention of these aspects throughout this discussion. I'm not sure how successful I was here. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-24 Thread Peter Geoghegan
are also coarser than strictly necessary. [1] http://www.postgresql.org/message-id/cam3swzsodumg4899tjc09r2uortyhb0vl9aasc1fz7aw4gs...@mail.gmail.com -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-25 Thread Peter Geoghegan
to conventional row locking occurs in nodeModifyTable.c (or else we just IGNORE). If you think that there could be unpleasant interactions because of the lack of Locktuple() arbitration within _bt_lockinsert() or something like that, please provide a test-case. -- Peter Geoghegan -- Sent via pgsql

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-26 Thread Peter Geoghegan
no sense. Should you really lock a value in an earlier unique index for hours, pending conflicter xact finishing, because you just might happen to want to insert said value, but probably not? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-27 Thread Peter Geoghegan
. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-28 Thread Peter Geoghegan
buffer locks are released. While not a hard and fast rule, presently we avoid heavyweight page locking more than one page per unique index concurrently. Happy new year -- Peter Geoghegan btreelock_insert_on_dup.v5.2013_12_28.patch.gz Description: GNU Zip compressed data -- Sent via pgsql

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-29 Thread Peter Geoghegan
://www.postgresql.org/message-id/20131227075453.gb17...@alap2.anarazel.de [3] http://www.postgresql.org/message-id/20130914221524.gf4...@awork2.anarazel.de -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-30 Thread Peter Geoghegan
+hfdgucyx2prep0y3a7nccejeow...@mail.gmail.com -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-30 Thread Peter Geoghegan
that may or may not be well suited to implementing this the same way). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-30 Thread Peter Geoghegan
have the original problem again, ie. the updater unnecessarily waits for the inserting transaction to finish, even though it already killed the tuple it inserted. Right. Can you suggest a workaround to the above problems? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-31 Thread Peter Geoghegan
, of course. What if you have multiple promise tuples from a contended attempt to insert a single slot, or multiple broken promise tuples across multiple slots or even multiple commands in the same xact? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2013-12-31 Thread Peter Geoghegan
() call, and so it should not wait on XactLockTableWait()? Does whatever mechanism you have in mind not have race conditions? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] Patch: show relation and tuple infos of a lock to acquire

2013-12-31 Thread Peter Geoghegan
the tuple proper. Doing any less could be highly confusing. You should do an analysis of where this kind of exception applies. For better direction about where that new infrastructure ought to live, you might find some useful insight from commit 991f3e5ab3f8196d18d5b313c81a5f744f3baaea. -- Peter

Re: [HACKERS] Patch: show relation and tuple infos of a lock to acquire

2014-01-02 Thread Peter Geoghegan
gain by not logging the ctid. I suppose so, but the tuple probably isn't going to be visible anyway, at least when the message is initially logged. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Peter Geoghegan
be in a position to do it from SQL as well... I believe you can. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Peter Geoghegan
://www.postgresql.org/message-id/cam3swzshbe29kpod44cvc3vpzjgmder6k_6fghiszeozgmt...@mail.gmail.com -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Peter Geoghegan
, that still has reasonable performance characteristics today, including I believe better worst-case latency. Heavyweight locks on btree pages are very well precedented, if you look beyond Postgres. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Peter Geoghegan
resulting in insert, and then perform that work, potentially deleting the row inserted if and when you change your mind? Is there any real difference between what that does for you, and what any particular variety of promise tuple might do for you? -- Peter Geoghegan -- Sent via pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Peter Geoghegan
On Thu, Jan 2, 2014 at 11:58 AM, Peter Geoghegan p...@heroku.com wrote: My executive summary is that the exclusion patch performs about the same on lower client counts, presumably due to not having the additional window of btree lock contention. By 8 clients, the exclusion patch does

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-03 Thread Peter Geoghegan
deserve closer inspection, but perhaps I shouldn't be so hasty: No consensus on the way forward looks even close to emerging. How do people feel about my approach now? -- Peter Geoghegan btreelock_insert_on_dup.v6.2014_01_03.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-03 Thread Peter Geoghegan
RETURNING would project in the same context. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Disallow arrays with non-standard lower bounds

2014-01-09 Thread Peter Geoghegan
with it. I agree with this, but I think it's too late. I don't think the answer is any type of parameter. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Disallow arrays with non-standard lower bounds

2014-01-09 Thread Peter Geoghegan
like that, it should always be zero. Person 1: We can all be winners. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-09 Thread Peter Geoghegan
On Tue, Jan 7, 2014 at 8:46 PM, Peter Geoghegan p...@heroku.com wrote: I've worked on a simple set of tests, written quickly in bash, that I think exercise interesting cases: https://github.com/petergeoghegan/upsert Perhaps most notably, these tests make comparisons between the performance

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-10 Thread Peter Geoghegan
/message-id/cam3swzshbe29kpod44cvc3vpzjgmder6k_6fghiszeozgmt...@mail.gmail.com I'd focus on that test case. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-10 Thread Peter Geoghegan
(i.e. there were no unfilled promise tuples), and you happened to perform conflict handling first, it could still happen, albeit with lower probability, no? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-10 Thread Peter Geoghegan
imagination around how to make that work, but there are a lot of ifs and buts either way. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-10 Thread Peter Geoghegan
=fOZ2Wr4fIxj0eFQD0tCGPLTsfY0hl=ensa=Xei=PpTQUquoBMfFsATtw4CADAved=0CDIQ6AEwAQ#v=onepageq=predicate%20locking%20np%20completef=false -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-10 Thread Peter Geoghegan
] http://postgres-benchmarks.s3-website-us-east-1.amazonaws.com/upsert-cmp-2/ [2] http://www.postgresql.org/message-id/cam3swzqbhs0jrid6efew3motxy1ek-8wdr6fvffr0aycdgc...@mail.gmail.com [3] http://postgres-benchmarks.s3-website-us-east-1.amazonaws.com/vs-vanilla-insert/ -- Peter Geoghegan

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-10 Thread Peter Geoghegan
that approach is obviously broken. In general, making row locking work for exclusion constraints, so you can treat them in a way that allows you to merge on arbitrary operators seems to me like a tar pit. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-11 Thread Peter Geoghegan
On Fri, Jan 10, 2014 at 7:59 PM, Peter Geoghegan p...@heroku.com wrote: *shrug*. I'm not too concerned about performance during contention. But let's see how this fixed version performs. Could you repeat the tests you did with this? Why would you not be too concerned about the performance

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-11 Thread Peter Geoghegan
On Sat, Jan 11, 2014 at 2:39 AM, Peter Geoghegan p...@heroku.com wrote: So I re-ran the same old benchmark, where we're almost exclusively updating. Results for your latest revision were very similar to my patch: http://postgres-benchmarks.s3-website-us-east-1.amazonaws.com/exclusion

[HACKERS] Syntax of INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-11 Thread Peter Geoghegan
a serialization failure. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Race condition in b-tree page deletion

2014-01-11 Thread Peter Geoghegan
in the January commitfest. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Syntax of INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-12 Thread Peter Geoghegan
On Sun, Jan 12, 2014 at 8:12 AM, Andreas Karlsson andr...@proxel.se wrote: On 01/11/2014 11:42 PM, Peter Geoghegan wrote: I recently suggested that rather than RETURNING REJECTS, we could have a REJECTING clause, which would see a DML statement project strictly the complement of what RETURNING

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Peter Geoghegan
/contract for the benefit of any future amcanunique AM author. It's ON DUPLICATE KEY, not ON OVERLAPPING KEY, or any other syntax suggestive of exclusion constraints and their arbitrary commutative operators. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Peter Geoghegan
with unique indexes only. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Peter Geoghegan
replication use-cases. There may be some value in knowing it cannot have been earlier unique indexes (and so the existing values for those unique indexes in the locked row should stay the same - don't many conflict resolution policies work that way?). -- Peter Geoghegan -- Sent via pgsql

Re: [HACKERS] Syntax of INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Peter Geoghegan
tuples where a BEFORE trigger disallowed the insert, we probably want to expose that that's why those tuples were rejected (as opposed to them being rejected due to a duplicate key violation). The ctid *won't* indicate a specific rejecting row then, I guess, which will do it. -- Peter

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-13 Thread Peter Geoghegan
useful to you here. It checks latestCompletedXid before doing a linear search through the proc array too. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-14 Thread Peter Geoghegan
On Mon, Jan 13, 2014 at 6:45 PM, Peter Geoghegan p...@heroku.com wrote: + uint32 + SpeculativeInsertionIsInProgress(TransactionId xid, RelFileNode rel, ItemPointer tid) + { For the purposes of preventing unprincipled deadlocking, commenting out the following (the only caller of the above) has

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-14 Thread Peter Geoghegan
, unique constraints, and all other constraints. FWIW I think that it would be kind of arbitrary to make IGNORE work with exclusion constraints and not other types of constraints, whereas when it's specifically ON DUPLICATE KEY, that seems far less surprising. -- Peter Geoghegan -- Sent via

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-14 Thread Peter Geoghegan
far easier to pin down. You definitely don't have to do anything new with visibility. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-15 Thread Peter Geoghegan
come last. And even without upsert, strictly inserting into unique indexes first seems like a useful thing relative to the cost. Unique violations are the cause of many aborted transactions, and there is no need to ever bloat non-unique indexes of the same slot when that happens. -- Peter

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-15 Thread Peter Geoghegan
to make sure that my hunch is correct. I'm travelling tomorrow to give a talk at PDX pug, so I'll have limited access to e-mail. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-15 Thread Peter Geoghegan
On Wed, Jan 15, 2014 at 8:23 PM, Peter Geoghegan p...@heroku.com wrote: I have an idea of what I could do to fix this, but I don't have time to make sure that my hunch is correct. It might just be a matter of: @@ -186,6 +186,13 @@ ExecLockHeapTupleForUpdateSpec(EState *estate, switch

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-16 Thread Peter Geoghegan
pgbench_accounts set abalance = pgbench_accounts.abalance + rej.abalance from rej where pgbench_accounts.aid = rej.aid; (This benchmark used an unlogged table, if only because to do otherwise would severely starve this particular server of I/O). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Backup throttling

2014-01-16 Thread Peter Geoghegan
to. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-16 Thread Peter Geoghegan
On Wed, Jan 15, 2014 at 11:02 PM, Peter Geoghegan p...@heroku.com wrote: It might just be a matter of: @@ -186,6 +186,13 @@ ExecLockHeapTupleForUpdateSpec(EState *estate, switch (test) { case HeapTupleInvisible

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-18 Thread Peter Geoghegan
was surprised when I learned that things didn't already work this way. Attached patch, broken off from my patch has relcache sort indexes by (!indisunique, relindexid). -- Peter Geoghegan *** a/src/backend/utils/cache/relcache.c --- b/src/backend/utils/cache/relcache.c *** typedef struct

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-18 Thread Peter Geoghegan
On Thu, Jan 16, 2014 at 6:31 PM, Peter Geoghegan p...@heroku.com wrote: I think we need to give this some more thought. I have not addressed the implications for MVCC snapshots here. So I gave this some more thought, and this is what I came up with: + static bool

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-18 Thread Peter Geoghegan
On Sat, Jan 18, 2014 at 6:17 PM, Peter Geoghegan p...@heroku.com wrote: MySQL users are not notified that this happened, and are probably blissfully unaware that there has been a limited form of data loss. So it's The Right Thing to say to Postgres users: if you inserted these rows

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-18 Thread Peter Geoghegan
On Sat, Jan 18, 2014 at 7:49 PM, Peter Geoghegan p...@heroku.com wrote: Personally, I favor just making case HeapTupleSelfUpdated: within the patch's ExecLockHeapTupleForUpdateSpec() function complain when hufd.cmax == estate-es_output_cid) (currently there is a separate complaint, but only

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-20 Thread Peter Geoghegan
the API version in the C name of the pg_stat_statements function instead. I agree that it isn't scalable. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Performance Improvement by reducing WAL for Update Operation

2014-01-20 Thread Peter Geoghegan
I/O in exchange for increased CPU usage. While I haven't been following the development of this patch, I will note that on the face of it the latter seem like a trade-off I'd be quite willing to make. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] Funny representation in pg_stat_statements.query.

2014-01-21 Thread Peter Geoghegan
, that was addressed by this commit: http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=5d3fcc4c2e137417ef470d604fee5e452b22f6a7 -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-21 Thread Peter Geoghegan
work well with third-party tools. I am very wary of enlarging the counters structure, because it is protected by a spinlock. There has been no attempt to quantify that cost, nor has anyone even theorized that it is not likely to be appreciable. -- Peter Geoghegan -- Sent via pgsql-hackers mailing

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2014-01-21 Thread Peter Geoghegan
() could differentiate between a soft and hard failure. Hopefully the reserve function you mentioned, which is still called at the same place, just before each critical section thereby becomes cheap. Perhaps I'm just restating what you said, though. -- Peter Geoghegan -- Sent via pgsql-hackers

Re: [HACKERS] Hard limit on WAL space used (because PANIC sucks)

2014-01-21 Thread Peter Geoghegan
of 1.0. People are not going to develop a lackadaisical attitude about running out of disk space on the pg_xlog partition if we do so. They still have plenty of incentive to make sure that that doesn't happen. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-21 Thread Peter Geoghegan
cycles by using the string directly in the buffer (at the small cost of needing to include the strings' trailing nulls in the file). It might be useful to do that anyway, for sanity checking purposes. So +1 from me. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-21 Thread Peter Geoghegan
On Tue, Jan 21, 2014 at 8:01 PM, Peter Geoghegan p...@heroku.com wrote: Not necessarily. Under Linux for example, POSIX_FADV_SEQUENTIAL sets the readahead window to the default size for the backing device Excuse me; I meant to put POSIX_FADV_SEQUENTIAL doubles this size [default size

Re: [HACKERS] Funny representation in pg_stat_statements.query.

2014-01-21 Thread Peter Geoghegan
(it is a limitation of pg_stat_statements that this can happen with very low probability). It's a bit of a kludge, but the query string is always just strictly speaking a convenience to help the user understand what the entry is. Although, I suppose, most people don't realize that. -- Peter Geoghegan

Re: [HACKERS] Storing pg_stat_statements query texts externally, pg_stat_statements in core

2014-01-22 Thread Peter Geoghegan
On Tue, Jan 21, 2014 at 8:01 PM, Peter Geoghegan p...@heroku.com wrote: Actually, I think the whole thing is rather badly designed, as warm cache or no you're still proposing to do thousands of kernel calls while holding a potentially contended LWLock. I suggest what we do is (1) read

<    4   5   6   7   8   9   10   11   12   13   >