Re: [HACKERS] Re: How to reproduce serialization failure for a read only transaction.

2014-01-07 Thread Dan Ports
On Mon, Jan 06, 2014 at 05:14:12PM -0800, AK wrote: Also I cannot reproduce a scenario when applications must not depend on results read during a transaction that later aborted;. In this example the SELECT itself has failed. Can you show an example where a SELECT completes, but the COMMIT

Re: [HACKERS] SSI freezing bug

2013-10-07 Thread Dan Ports
On Mon, Oct 07, 2013 at 12:26:37PM +0300, Heikki Linnakangas wrote: When updating a tuple, CheckTargetForConflictsIn() only marks a conflict if the transaction holding the predicate lock overlapped with the updating transaction. Ah, this is the bit I was forgetting. (I really ought to have

Re: [HACKERS] SSI freezing bug

2013-10-03 Thread Dan Ports
On Thu, Oct 03, 2013 at 06:19:49AM -0700, Kevin Grittner wrote: Heikki Linnakangas hlinnakan...@vmware.com wrote: IMHO it would be better to remove xmin from the lock key, and vacuum away the old predicate locks when the corresponding tuple is vacuumed. The xmin field is only required to

Re: [HACKERS] lwlock contention with SSI

2013-04-10 Thread Dan Ports
On Tue, Apr 09, 2013 at 07:49:51PM -0400, Robert Haas wrote: These locks are all SSI-related and they're all really hot. Lock 28 is SerializableXactHashLock and lock 29 is SerializableFinishedListLock; both are acquired an order of magnitude more often than any non-SSI lock, and cause two

Re: [HACKERS] Question about SSI, subxacts, and aborted read-only xacts

2012-09-11 Thread Dan Ports
On Mon, Sep 10, 2012 at 10:44:57PM -0700, Jeff Davis wrote: For the archives, and for those not following the paper in detail, there is one situation in which SSI will abort a read-only transaction. When there are three transactions forming a dangerous pattern where T1 (read-only) has a

Re: [HACKERS] Question about SSI, subxacts, and aborted read-only xacts

2012-09-10 Thread Dan Ports
On Sat, Sep 08, 2012 at 11:34:56AM -0700, Jeff Davis wrote: If so, I think we need a documentation update. The serializable isolation level docs don't quite make it clear that serializability only applies to transactions that commit. It might not be obvious to a user that there's a difference

Re: [HACKERS] a slightly stale comment

2012-03-07 Thread Dan Ports
On Wed, Mar 07, 2012 at 07:46:32AM +, Simon Riggs wrote: There is much wisdom there and much wisdom in leaving ancient warnings as we find them. The comment is a wise and insightful statement -- about a totally different system than we have today. Are these the words you object to? we

[HACKERS] a slightly stale comment

2012-03-06 Thread Dan Ports
While mucking around in src/backend/utils/time/tqual.c today, I noticed the following comment attached to HeapTupleSatisfiesNow: * mao says 17 march 1993: the tests in this routine are correct; * if you think they're not, you're wrong, and you should think * about it again. i

Re: [HACKERS] SSI rw-conflicts and 2PC

2012-02-14 Thread Dan Ports
On Tue, Feb 14, 2012 at 10:04:15AM +0200, Heikki Linnakangas wrote: Perhaps it would be simpler to add the extra information to the commit records of the transactions that commit after the first transaction is prepared. In the commit record, you would include a list of prepared transactions

Re: [HACKERS] SSI rw-conflicts and 2PC

2012-02-14 Thread Dan Ports
On Tue, Feb 14, 2012 at 09:27:58AM -0600, Kevin Grittner wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 14.02.2012 04:57, Dan Ports wrote: The easiest answer would be to just treat every prepared transaction found during recovery as though it had a conflict

[HACKERS] SSI rw-conflicts and 2PC

2012-02-13 Thread Dan Ports
Looking over the SSI 2PC code recently, I noticed that I overlooked a case that could lead to non-serializable behavior after a crash. When we PREPARE a serializable transaction, we store part of the SERIALIZABLEXACT in the statefile (in addition to the list of SIREAD locks). One of the pieces of

Re: [HACKERS] RFC: Making TRUNCATE more MVCC-safe

2012-02-11 Thread Dan Ports
On Fri, Feb 10, 2012 at 01:59:18PM -0500, Robert Haas wrote: I guess I'm not particularly excited by the idea of trying to make TRUNCATE MVCC-safe. I notice that the example involves the REPEATABLE READ isolation level, which is already known to be busted in a variety of ways; that's why we

[HACKERS] patch: fix SSI finished list corruption

2012-01-06 Thread Dan Ports
There's a corner case in the SSI cleanup code that isn't handled correctly. It can arise when running workloads that are comprised mostly (but not 100%) of READ ONLY transactions, and can corrupt the finished SERIALIZABLEXACT list, potentially causing a segfault. The attached patch fixes it.

[HACKERS] autovacuum and default_transaction_isolation

2011-11-29 Thread Dan Ports
I was doing some tests recently with default_transaction_isolation set to 'serializable' in postgresql.conf when I noticed pg_locks filling up with SIReadLocks rather more quickly than I expected. After some investigation, I found that an autovacuum worker was starting a transaction at the

Re: [HACKERS] autovacuum and default_transaction_isolation

2011-11-29 Thread Dan Ports
On Tue, Nov 29, 2011 at 07:04:23PM -0500, Tom Lane wrote: Hmm. Shouldn't we make the autovac launcher use READ COMMITTED, too? Yeah, probably. That one doesn't seem so important because its transactions aren't long-running (IIRC, it only starts a transaction to scan pg_database). But it

Re: [HACKERS] FlexLocks

2011-11-15 Thread Dan Ports
On Tue, Nov 15, 2011 at 10:55:49AM -0600, Kevin Grittner wrote: And I would be surprised if some creative thinking didn't yield a far better FL scheme for SSI than we can manage with existing LW locks. One place I could see it being useful is for SerializableFinishedListLock, which protects

Re: [HACKERS] SSI implementation question

2011-10-20 Thread Dan Ports
On Thu, Oct 20, 2011 at 07:33:59AM -0500, Kevin Grittner wrote: Dan Ports wrote: The part that's harder is building the list of potential conflicts that's used to identify safe snapshots for r/o transactions. That (currently) has to happen atomically taking the snapshot. That's

Re: [HACKERS] SSI implementation question

2011-10-19 Thread Dan Ports
On Wed, Oct 19, 2011 at 12:56:58PM -0400, Tom Lane wrote: Is it really necessary for GetSerializableTransactionSnapshotInt to acquire an empty SERIALIZABLEXACT before it acquires a snapshot? If so, why? *That* isn't necessary, no. It is necessary, however, to acquire the snapshot while

Re: [HACKERS] SSI implementation question

2011-10-19 Thread Dan Ports
On Wed, Oct 19, 2011 at 04:36:41PM -0400, Tom Lane wrote: No, the intention of the synchronized-snapshots feature is just to be able to start multiple transactions using exactly the same snapshot. They're independent after that. The aspect of it that is worrying me is that if xact A starts,

Re: [HACKERS] SSI implementation question

2011-10-19 Thread Dan Ports
On Wed, Oct 19, 2011 at 05:04:52PM -0400, Tom Lane wrote: I wonder whether it would be prudent to set the synchronized-snapshots patch aside until you've finished that work (assuming you're actively working on it). It's evidently going to require some nontrivial changes in predicate.c, and I

Re: [HACKERS] SSI implementation question

2011-10-19 Thread Dan Ports
I think it would be fairly sensible to push some of this into ProcArray, actually. The commit sequence numbers just involve assigning/ incrementing a global counter when taking a snapshot and finishing a transaction -- that's not too much work, doesn't require any additional locking beyond

Re: [HACKERS] Update on documentation builds on OSX w/ macports

2011-10-19 Thread Dan Ports
On Thu, Oct 20, 2011 at 02:02:09AM +0200, Florian Pflug wrote: I've patched the ports for openjade, iso8879 and docbook-dsssl, and added a new port for docbook-sgml-4.2. These patches are sitting in the macports trac now, waiting to be applied. I'll try to take a look at them in the next

Re: [HACKERS] Overhead cost of Serializable Snapshot Isolation

2011-10-10 Thread Dan Ports
On Mon, Oct 10, 2011 at 02:25:59PM -0400, Greg Sabino Mullane wrote: I agree it is better versus SELECT FOR, but what about repeatable read versus the new serializable? How much overhead is there in the 'monitoring of read/write dependencies'? This is my only concern at the moment. Are we

Re: [HACKERS] Overhead cost of Serializable Snapshot Isolation

2011-10-10 Thread Dan Ports
On Mon, Oct 10, 2011 at 02:59:04PM -0500, Kevin Grittner wrote: I do have some concern about whether the performance improvements from reduced LW locking contention elsewhere in the code may (in whack-a-mole fashion) cause the percentages to go higher in SSI. The biggest performance issues in

Re: [HACKERS] Overhead cost of Serializable Snapshot Isolation

2011-10-10 Thread Dan Ports
On Mon, Oct 10, 2011 at 04:10:18PM -0500, Kevin Grittner wrote: Did you ever see much contention on SerializablePredicateLockListLock, or was it just SerializableXactHashLock? I think the former might be able to use the non-blocking techniques, but I fear the main issue is with the latter,

Re: [HACKERS] sinval synchronization considered harmful

2011-07-21 Thread Dan Ports
On Thu, Jul 21, 2011 at 02:31:15PM -0400, Robert Haas wrote: 1. Machines with strong memory ordering. On this category of machines (which include x86), the CPU basically does not perform loads or stores out of order. On some of these machines, it is apparently possible for there to be some

Re: [HACKERS] SSI atomic commit

2011-07-07 Thread Dan Ports
On Thu, Jul 07, 2011 at 04:48:55PM -0400, Tom Lane wrote: Seems to me there's a more fundamental reason not to do that, which is that once you've done PREPARE it is no longer legitimate to decide to roll back the transaction to get out of a dangerous structure --- ie, you have to target one of

Re: [HACKERS] SSI atomic commit

2011-07-07 Thread Dan Ports
We should also apply the attached patch, which corrects a minor issue with the conditions for flagging transactions that could potentially make a snapshot unsafe. There's a small window wherein a transaction is committed but not yet on the finished list, and we shouldn't flag it as a potential

Re: [HACKERS] SSI atomic commit

2011-07-05 Thread Dan Ports
On Tue, Jul 05, 2011 at 01:15:13PM -0500, Kevin Grittner wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Hmm, I think it would be simpler to decide that instead of SerializableXactHashLock, you must hold ProcArrayLock to access LastSxactCommitSeqNo, and move the

Re: [HACKERS] SSI 2PC coverage

2011-07-05 Thread Dan Ports
On Tue, Jul 05, 2011 at 09:14:30PM +0300, Heikki Linnakangas wrote: I think that needs some explanation, why only those SxactIsCommitted() tests need to be replaced with SxactIsPrepared()? Here is the specific problem this patch addresses: If there's a dangerous structure T0 --- T1 --- T2,

Re: [HACKERS] Repeated PredicateLockRelation calls during seqscan

2011-06-22 Thread Dan Ports
On Wed, Jun 22, 2011 at 12:07:04PM +0300, Heikki Linnakangas wrote: Hmm, I wonder if we should move this logic to heapam.c. The optimization to acquire a relation lock straight away should apply to all heap scans, not only those coming from ExecSeqScan. The distinction is academic at the

Re: [HACKERS] Coding style point: const in function parameter declarations

2011-06-21 Thread Dan Ports
On Tue, Jun 21, 2011 at 06:51:20PM -0400, Tom Lane wrote: I find this to be poor style, and would like to see if there's any support for getting rid of the const keywords. I'm in favor of removing them too. Dan -- Dan R. K. Ports MIT CSAILhttp://drkp.net/ --

[HACKERS] Repeated PredicateLockRelation calls during seqscan

2011-06-21 Thread Dan Ports
I was looking at ExecSeqScan today and noticed that it invokes PredicateLockRelation each time it's called, i.e. for each tuple returned. Any reason we shouldn't skip that call if rs_relpredicatelocked is already set, as in the attached patch? That would save us a bit of overhead, since checking

Re: [HACKERS] pika buildfarm member failure on isolationCheck tests

2011-06-21 Thread Dan Ports
On Tue, Jun 21, 2011 at 03:01:48PM +0300, Heikki Linnakangas wrote: Thanks, committed. Thanks. In the long term, I'm not sure this is the best way to handle this. It feels a bit silly to set the flag, release the SerializableXactHashLock, and reacquire it later to remove the transaction

Re: [HACKERS] pika buildfarm member failure on isolationCheck tests

2011-06-21 Thread Dan Ports
On Wed, Jun 22, 2011 at 01:31:11AM -0400, Dan Ports wrote: Yes, I suspect it can be done better. The reason it's tricky is a lock ordering issue; part of releasing a SerializableXact has to be done while holding SerializableXactHashLock and part has to be done without it (because it involves

Re: [HACKERS] pika buildfarm member failure on isolationCheck tests

2011-06-20 Thread Dan Ports
On Sun, Jun 19, 2011 at 09:10:02PM -0400, Robert Haas wrote: Is this an open item for 9.1beta3? Yes. I've put it on the list. The SxactGlobalXmin and its refcount were getting out of sync with the actual transaction state. This is timing-dependent but I can reproduce it fairly reliably under

Re: [HACKERS] pika buildfarm member failure on isolationCheck tests

2011-06-20 Thread Dan Ports
While testing the fix for this one, I found another bug. Patches for both are attached. The first patch addresses this bug by re-adding SXACT_FLAG_ROLLED_BACK, in a more limited form than its previous incarnation. We need to be able to distinguish transactions that have already called

Re: [HACKERS] patch: update README-SSI

2011-06-16 Thread Dan Ports
On Thu, Jun 16, 2011 at 04:39:09PM +0300, Heikki Linnakangas wrote: There's no mention on what T1 is. I believe it's supposed to be Tin, in the terminology used in the graph. Yes, I changed the naming after I originally wrote it, and missed a couple spots. T1 should be Tin. I don't see how

Re: [HACKERS] SSI work for 9.1

2011-06-16 Thread Dan Ports
On Thu, Jun 16, 2011 at 11:49:48PM -0400, Robert Haas wrote: Does this mean that the open item more SSI loose ends can now be marked resolved? I was just looking at it and contemplating moving it to the non-blockers list. Of the five items: - (1) and (4) are resolved - (2) isn't an issue --

Re: [HACKERS] SSI work for 9.1

2011-06-16 Thread Dan Ports
On Fri, Jun 17, 2011 at 12:32:46AM -0400, Robert Haas wrote: Perhaps it would be best to remove the general item and replace it with a list of more specific things that need doing - which might just mean #5. Done. -- Dan R. K. Ports MIT CSAILhttp://drkp.net/ --

[HACKERS] patch: update README-SSI

2011-06-15 Thread Dan Ports
The attached patch updates README-SSI. In addition to some minor edits, changes include: - add a section at the beginning that more clearly describes the SSI rule and defines dangerous structure with a diagram. It describes the optimizations we use about the relative commit times, and the

Re: [HACKERS] SSI patch renumbered existing 2PC resource managers??

2011-06-13 Thread Dan Ports
On Mon, Jun 13, 2011 at 10:22:19PM +0300, Heikki Linnakangas wrote: As far as I can tell it was for purely cosmetic reasons, to have lock and predicate lock lines together. Yes, that is the only reason. Dan -- Dan R. K. Ports MIT CSAILhttp://drkp.net/ -- Sent

Re: [HACKERS] SSI patch renumbered existing 2PC resource managers??

2011-06-13 Thread Dan Ports
On Mon, Jun 13, 2011 at 03:33:24PM -0400, Tom Lane wrote: We can either change that now, or undo the unnecessary change in existing RM IDs. I vote for the latter. Sounds good to me. I'd offer a patch, but it'd probably take you longer to apply than to make the change yourself. Dan -- Dan R.

Re: [HACKERS] Small SSI issues

2011-06-11 Thread Dan Ports
On Sat, Jun 11, 2011 at 01:38:31PM -0500, Kevin Grittner wrote: I'm not concerned about references covered by SerializableXactHashLock. I am more concerned about some of the tests for whether the (MySerializableXact == InvalidSerializableXact) checks and any other tests not covered by that

Re: [HACKERS] Small SSI issues

2011-06-10 Thread Dan Ports
On Fri, Jun 10, 2011 at 09:43:58PM +0300, Heikki Linnakangas wrote: Do checks such as that argue for keeping the volatile flag, or do you think we can drop it if we make those changes? (That would also allow dropping a number of casts which exist just to avoid warnings.) I believe we

Re: [HACKERS] SSI work for 9.1

2011-06-09 Thread Dan Ports
On Thu, Jun 09, 2011 at 07:06:18AM -0500, Kevin Grittner wrote: Sounds reasonable, but why did you pass the snapshot to the PredicateLockPage() call but not the PredicateLockRelation() call? Oversight? Yep, just an oversight; long day yesterday. I'll fix the patch shortly (unless you can get

Re: [HACKERS] SSI work for 9.1

2011-06-09 Thread Dan Ports
On Thu, Jun 09, 2011 at 01:30:27PM -0400, Dan Ports wrote: On Thu, Jun 09, 2011 at 07:06:18AM -0500, Kevin Grittner wrote: Sounds reasonable, but why did you pass the snapshot to the PredicateLockPage() call but not the PredicateLockRelation() call? Oversight? Yep, just an oversight

Re: [HACKERS] reindex creates predicate lock on index root

2011-06-08 Thread Dan Ports
On Tue, Jun 07, 2011 at 10:14:30PM -0400, Tom Lane wrote: Do you mean page zero, as in the metapage (for most index types), or do you mean the root page? If the former, how is that not an outright bug, since it corresponds to no data? If the latter, how is that not a serious performance

Re: [HACKERS] SSI heap_insert and page-level predicate locks

2011-06-08 Thread Dan Ports
On Wed, Jun 08, 2011 at 11:23:48AM +0300, Heikki Linnakangas wrote: AFAICS, the check for page lock is actually unnecessary. A page-level lock on a heap only occurs when tuple-level locks are promoted. It is just a coarser-grain representation of holding locks on all tuples on the page,

Re: [HACKERS] SSI work for 9.1

2011-06-08 Thread Dan Ports
On Wed, Jun 08, 2011 at 05:48:26PM -0500, Kevin Grittner wrote: (1) Pass snapshot in to some predicate.c functions. The particular functions have yet to be determined, but certainly any which acquire predicate locks, and probably all which are guarded by the SkipSerialization() macro. Skip

Re: [HACKERS] SSI work for 9.1

2011-06-08 Thread Dan Ports
On Wed, Jun 08, 2011 at 09:17:04PM -0500, Kevin Grittner wrote: A patch is attached which just covers the predicate lock acquisition, where a snapshot is available without too much pain. There are two functions which acquire predicate locks where a snapshot was not readily available:

Re: [HACKERS] reindex creates predicate lock on index root

2011-06-07 Thread Dan Ports
On Tue, Jun 07, 2011 at 07:45:43PM -0500, Kevin Grittner wrote: During testing of the SSI DDL changes I noticed that a REINDEX INDEX created a predicate lock on page 0 of the index. Really? That surprises me, and I couldn't reproduce it just now. Dan -- Dan R. K. Ports MIT CSAIL

Re: [HACKERS] SIREAD lock versus ACCESS EXCLUSIVE lock

2011-06-05 Thread Dan Ports
On Sun, Jun 05, 2011 at 12:45:41PM -0500, Kevin Grittner wrote: Is this possible? If a transaction gets its snapshot while OID of N is assigned to relation X, can that transaction wind up seeing an OID of N as a reference to relation Y? If not, there aren't any false positives possible.

Re: [HACKERS] SSI predicate locking on heap -- tuple or row?

2011-06-02 Thread Dan Ports
On Wed, Jun 01, 2011 at 05:09:09PM -0500, Kevin Grittner wrote: I won't be shocked if Dan can come up with a shorter proof, but I'm confident this one is solid. Well, so happens I wrote a proof on the airplane today, before I saw your mail. It's actually quite straightforward... (well, at least

Re: [HACKERS] SSI predicate locking on heap -- tuple or row?

2011-06-02 Thread Dan Ports
On Thu, Jun 02, 2011 at 01:01:05PM -0500, Kevin Grittner wrote: If we're going to put this into the README-SSI as the proof of the validity of this optimization, I'd like to have a footnote pointing to a paper describing the first commit in the cycle aspect of a dangerous structure. Got any

Re: [HACKERS] SSI predicate locking on heap -- tuple or row?

2011-05-24 Thread Dan Ports
On Tue, May 24, 2011 at 04:18:37AM -0500, Kevin Grittner wrote: These proofs show that there is no legitimate cycle which could cause an anomaly which the move from row-based to tuple-based logic will miss. They don't prove that the change will generate all the same serialization failures;

Re: [HACKERS] SSI predicate locking on heap -- tuple or row?

2011-05-23 Thread Dan Ports
On Sat, May 21, 2011 at 03:09:12PM -0500, Kevin Grittner wrote: I went back to the example which persuaded me and took another look. On review I see that this didn't prove the point because there was a dangerous structure with T1 as a pivot which should have caused SSI to break the cycle.

Re: [HACKERS] SSI predicate locking on heap -- tuple or row?

2011-05-23 Thread Dan Ports
I have thought about this quite a bit and am fairly certain we do not need to track this linkage between row versions. One strong hint to this is that all the work I've seen on multiversion serializability theory defines a rw-conflict to be one transaction reading an object and the other writing

Re: [HACKERS] SSI predicate locking on heap -- tuple or row?

2011-05-21 Thread Dan Ports
On Sat, May 21, 2011 at 04:45:15PM -0400, Pavan Deolasee wrote: As a first step, it would be great if you can upload the slides on the conference website. To expect that the attendees would have understood the nitty-gritties of SSI just listening to the presentation is so unhuman :-) I just

Re: [HACKERS] patch: fix race in SSI's CheckTargetForConflictsIn

2011-05-08 Thread Dan Ports
On Fri, May 06, 2011 at 10:49:22PM -0400, Dan Ports wrote: Will update the patch. Updated patch (in response to Robert's comments) attached. Dan -- Dan R. K. Ports MIT CSAILhttp://drkp.net/ diff --git a/src/backend/storage/lmgr/predicate.c b/src/backend/storage

Re: [HACKERS] patch: fix race in SSI's CheckTargetForConflictsIn

2011-05-06 Thread Dan Ports
On Fri, May 06, 2011 at 09:35:39PM -0400, Robert Haas wrote: Why does this HASH_FIND the applicable hash table entries and then HASH_REMOVE it as a separate step, instead of just HASH_REMOVE-ing it in one go? For PredicateLockHash, we need to find the lock entry first so that we can call

[HACKERS] Re: Why is RegisterPredicateLockingXid called while holding XidGenLock?

2011-05-05 Thread Dan Ports
On Thu, May 05, 2011 at 11:12:40PM -0400, Tom Lane wrote: Even if it's actually necessary to set up that data structure while holding XidGenLock, I would *really* like the call to not be exactly where it is. Good question. I don't believe it needs to be while XidGenLock is being held at all;

[HACKERS] patch: fix race in SSI's CheckTargetForConflictsIn

2011-05-04 Thread Dan Ports
While running some benchmarks to test SSI performance, I found a race condition that's capable of causing a segfault. A patch is attached. The bug is in CheckTargetForConflictsIn, which scans the list of SIREAD locks on a lock target when it's modified. There's an optimization in there where the

Re: [HACKERS] Predicate locking

2011-05-03 Thread Dan Ports
On Tue, May 03, 2011 at 01:36:36PM +0900, Vlad Arkhipov wrote: Then I commited the both and the second one raised an exception: ERROR: could not serialize access due to read/write dependencies among transactions SQL state: 40001 However the second transaction does not access the records

Re: [HACKERS] SSI non-serializable UPDATE performance

2011-04-29 Thread Dan Ports
On Thu, Apr 28, 2011 at 06:45:54PM +0200, Robert Haas wrote: Yeah, I think Dan's notes about memory ordering would be good to include. I left it out initially because I didn't want to make things more confusing. As far as memory ordering is concerned, this is the same story as anything else that

Re: [HACKERS] SSI non-serializable UPDATE performance

2011-04-28 Thread Dan Ports
On Thu, Apr 28, 2011 at 08:43:30AM +0100, Simon Riggs wrote: We added a quick return which didn't need to check any locks at the front of this routine which is taken if there are no active serializable transactions on the cluster at the moment of update. Surprised to hear nobody

Re: [HACKERS] SSI non-serializable UPDATE performance

2011-04-27 Thread Dan Ports
On Wed, Apr 27, 2011 at 06:26:52PM +0100, Simon Riggs wrote: Reading the code, IIUC, we check for RW conflicts after each write but only if the writer is running a serializable transaction. Am I correct in thinking that there is zero impact of SSI if nobody is running a serializable

Re: [HACKERS] SIREAD lock versus ACCESS EXCLUSIVE lock

2011-04-27 Thread Dan Ports
On Wed, Apr 27, 2011 at 02:59:19PM -0500, Kevin Grittner wrote: For correct serializable behavior in the face of concurrent DDL execution, I think that a request for a heavyweight ACCESS EXCLUSIVE lock might need to block until all SIREAD locks on the relation have been released. Picture,

Re: [HACKERS] SSI non-serializable UPDATE performance

2011-04-24 Thread Dan Ports
On Sat, Apr 23, 2011 at 08:54:31AM -0500, Kevin Grittner wrote: Even though this didn't show any difference in Dan's performance tests, it seems like reasonable insurance against creating a new bottleneck in very high concurrency situations. Dan, do you have a patch for this, or should I

[HACKERS] SSI non-serializalbe UPDATE performance (was: getting to beta)

2011-04-22 Thread Dan Ports
For background, the issue here is that there are three SSI calls that get invoked even on non-serializable transactions: - PredicateLockPageSplit/Combine, during B-tree page splits/combines - PredicateLockTupleRowVersionLink, from heap_update These have to update any matching SIREAD locks to

Re: [HACKERS] Formatting Curmudgeons WAS: MMAP Buffers

2011-04-17 Thread Dan Ports
On Sat, Apr 16, 2011 at 11:26:34PM -0500, Kevin Grittner wrote: I have to say, I've been rather mystified by the difficulty attributed to running pgindent. During work on the SSI patch, I ran it about once every two weeks on files involved in the patch Well, as a counterpoint: during work on

Re: [HACKERS] getting to beta

2011-04-06 Thread Dan Ports
On Wed, Apr 06, 2011 at 12:25:26PM -0500, Kevin Grittner wrote: By the way, the problem with SSI potentially running out of shared memory is rather parallel to how heavyweight locks can run out of shared memory. The SLRU prevents the number of transactions from being limited in that way, and

Re: [HACKERS] time table for beta1

2011-04-04 Thread Dan Ports
On Mon, Apr 04, 2011 at 10:33:22AM -0500, Kevin Grittner wrote: There are patches for all known issues except one. Dan Ports was able to replicate the latest issue uncovered by YAMAMOTO Takashi using a particular DBT-2 configuration, found the issue, and posted a patch: Well, it would

Re: [HACKERS] time table for beta1

2011-04-04 Thread Dan Ports
On Mon, Apr 04, 2011 at 07:04:59PM -0400, Robert Haas wrote: On Mon, Apr 4, 2011 at 6:41 PM, Stephen Frost sfr...@snowman.net wrote: What'd be horribly useful would be the pid and the *time* that the lock was taken.. ?Knowing just the pid blows, since the pid could technically end up reused

Re: [HACKERS] SSI bug?

2011-04-03 Thread Dan Ports
I think I see what is going on now. We are sometimes failing to set the commitSeqNo correctly on the lock. In particular, if a lock assigned to OldCommittedSxact is marked with InvalidSerCommitNo, it will never be cleared. The attached patch corrects this: TransferPredicateLocksToNewTarget

[HACKERS] trivial patch: show SIREAD pids in pg_locks

2011-04-01 Thread Dan Ports
While looking into a SSI bug, I noticed that we don't actually display the pid of the holding transaction, even though we have that information available. The attached patch fixes that. One note is that it will show the pid of the backend that executed the transaction, even if that transaction

Re: [HACKERS] trivial patch: show SIREAD pids in pg_locks

2011-04-01 Thread Dan Ports
On Fri, Apr 01, 2011 at 12:20:25PM -0500, Kevin Grittner wrote: I thought we already had that, but clearly I was mistaken. Yeah, so did I. Turns out we had the vxid but not the pid. IIRC, we weren't tracking a SERIALIZABLEXACT's pid yet, at the time we wrote the code for pg_locks. I guess the

Re: [HACKERS] SSI bug?

2011-03-31 Thread Dan Ports
On Thu, Mar 31, 2011 at 11:06:30AM -0500, Kevin Grittner wrote: The only thing I've been on the fence about is whether it makes more sense to allocate it all up front or to continue to allow incremental allocation but set a hard limit on the number of entries allocated for each shared memory

Re: [HACKERS] SSI bug?

2011-03-26 Thread Dan Ports
On Fri, Mar 25, 2011 at 04:06:30PM -0400, Tom Lane wrote: Up to now, I believe the lockmgr's lock table is the only shared hash table that is expected to grow past the declared size; that can happen anytime a session exceeds max_locks_per_transaction, which we consider to be only a soft limit.

Re: [HACKERS] SSI bug?

2011-03-18 Thread Dan Ports
It would probably also be worth monitoring the size of pg_locks to see how many predicate locks are being held. On Fri, Mar 18, 2011 at 12:50:16PM -0500, Kevin Grittner wrote: Even with the above information it may be far from clear where allocations are going past their maximum, since one

Re: [HACKERS] SSI bug?

2011-03-01 Thread Dan Ports
On Tue, Mar 01, 2011 at 07:07:42PM +0200, Heikki Linnakangas wrote: Was there test cases for any of the issues fixed by this patch that we should add to the suite? Some of these issues are tricky to test, e.g. some of the code about transferring predicate locks to a new target doesn't get

Re: [HACKERS] SSI bug?

2011-02-28 Thread Dan Ports
An updated patch to address this issue is attached. It fixes a couple issues related to use of the backend-local lock table hint: - CheckSingleTargetForConflictsIn now correctly handles the case where a lock that's being held is not reflected in the local lock table. This fixes the

Re: [HACKERS] SSI bug?

2011-02-22 Thread Dan Ports
On Tue, Feb 22, 2011 at 10:51:05AM -0600, Kevin Grittner wrote: Dan Ports d...@csail.mit.edu wrote: It looks like CheckTargetForConflictsIn is making the assumption that the backend-local lock table is accurate, which was probably even true at the time it was written. I remember we

Re: [HACKERS] SSI bug?

2011-02-22 Thread Dan Ports
On Tue, Feb 22, 2011 at 05:54:49PM -0600, Kevin Grittner wrote: I'm not sure it's safe to assume that the index page won't get reused before the local lock information is cleared. In the absence of a clear proof that it is safe, or some enforcement mechanism to ensure that it is, I don't

Re: [HACKERS] SSI bug?

2011-02-21 Thread Dan Ports
On Mon, Feb 21, 2011 at 11:42:36PM +, YAMAMOTO Takashi wrote: i tested ede45e90dd1992bfd3e1e61ce87bad494b81f54d + ssi-multi-update-1.patch with my application and got the following assertion failure. #4 0x0827977e in CheckTargetForConflictsIn (targettag=0xbfbfce78) at

Re: [HACKERS] SSI bug?

2011-02-17 Thread Dan Ports
On Wed, Feb 16, 2011 at 10:13:35PM +, YAMAMOTO Takashi wrote: i got the following SEGV when runnning vacuum on a table. (the line numbers in predicate.c is different as i have local modifications.) oldlocktag.myTarget was NULL. it seems that TransferPredicateLocksToNewTarget sometimes use

Re: [HACKERS] SSI patch version 14

2011-02-08 Thread Dan Ports
On Tue, Feb 08, 2011 at 11:25:34AM +0200, Heikki Linnakangas wrote: On 08.02.2011 10:43, Kevin Grittner wrote: I see that at least three BuildFarm critters don't have UINT64_MAX defined. I guess we'll have to just #define it ourselves. Or could we just pick another magic value, do we

Re: [HACKERS] SSI patch version 14

2011-02-08 Thread Dan Ports
On Tue, Feb 08, 2011 at 10:14:44AM -0600, Kevin Grittner wrote: I do have some concern that if this defaults to too low a number, those who try SSI without bumping it and restarting the postmaster will not like the performance under load very much. SSI performance would not be affected by a

Re: [HACKERS] SSI patch version 14

2011-02-08 Thread Dan Ports
One other nit re. the predicate lock table size GUCs: the out-of-memory case in RegisterPredicateLockingXid (predicate.c:1592 in my tree) gives the hint to increase max_predicate_locks_per_transaction. I don't think that's correct, since that GUC isn't used to size SerializableXidHash. In fact,

Re: [HACKERS] SSI patch version 14

2011-02-08 Thread Dan Ports
On Tue, Feb 08, 2011 at 04:04:39PM -0600, Kevin Grittner wrote: (2) The predicate lock and lock target initialization code was initially copied and modified from the code for heavyweight locks. The heavyweight lock code adds 10% to the calculated maximum size. So I wound up doing that for

Re: [HACKERS] SSI patch version 14

2011-02-08 Thread Dan Ports
On Tue, Feb 08, 2011 at 09:09:48PM -0500, Robert Haas wrote: No way to fail is a tall order. Well, no way to fail due to running out of shared memory in RegisterPredicateLock/RegisterPredicateLockingXid, but that doesn't have quite the same ring to it... If we don't allocate all the memory up

Re: [HACKERS] SSI patch version 14

2011-01-31 Thread Dan Ports
On Sun, Jan 30, 2011 at 04:01:56PM -0600, Kevin Grittner wrote: I'm wondering how this differs from what is discussed in Section 2.7 (Serialization Graph Testing) of Cahill's doctoral thesis. That discusses a technique for trying to avoid false positives by testing the full graph for cycles,

Re: [HACKERS] SSI patch version 14

2011-01-28 Thread Dan Ports
On Thu, Jan 27, 2011 at 09:18:23AM -0800, Jeff Davis wrote: On Tue, 2011-01-25 at 05:57 -0500, Dan Ports wrote: This summary is right on. I would add one additional detail or clarification to the last point, which is that rather than checking for a cycle, we're checking for a transaction

Re: [HACKERS] SSI patch version 14

2011-01-26 Thread Dan Ports
On Wed, Jan 26, 2011 at 10:01:28AM -0600, Kevin Grittner wrote: In looking at it just now, I noticed that after trying it in a couple different places what was left in the repository was not the optimal version for code coverage. I've put this back to the version which did a better job, for

Re: [HACKERS] SSI patch version 14

2011-01-26 Thread Dan Ports
On Wed, Jan 26, 2011 at 01:42:23PM -0600, Kevin Grittner wrote: Dan, do you still have access to that machine you were using for the DBT-2 runs? Could we get a coverage run with and without TEST_OLDSERXID defined? Sure, I'll give it a shot (once I figure out how to enable coverage...) Dan

Re: [HACKERS] SSI patch version 14

2011-01-26 Thread Dan Ports
On Wed, Jan 26, 2011 at 02:36:25PM -0600, Kevin Grittner wrote: Same benefit in terms of exercising more lines of code, but *without* exposing the uninitialized structure to other threads. Won't this cause a deadlock because locks are being acquired out of order? Dan -- Dan R. K. Ports

Re: [HACKERS] SSI patch version 14

2011-01-25 Thread Dan Ports
Thanks for working your way through this patch. I'm certainly well aware that that's not a trivial task! I'm suffering through a bout of insomnia, so I'll respond to some of your high-level comments in hopes that serializability will help put me to sleep (as it often does). I'll leave the more

Re: [HACKERS] SSI and Hot Standby

2011-01-20 Thread Dan Ports
What I'm still not clear on is why that HS is different. Whatever rules apply on the master must also apply on the standby, immutably. Why is it we need to pass explicit snapshot information from master to standby? We don't do that, except at startup for normal HS. Why do we need that? I

Re: [HACKERS] SSI and Hot Standby

2011-01-20 Thread Dan Ports
On Fri, Jan 21, 2011 at 08:44:59AM +0200, Heikki Linnakangas wrote: We have enough information in the standby to reconstruct all writes done in the master. I gather that's not enough, in order to roll back read-only transaction T3 on the standby which would see an anomaly, we'd also need to

Re: [HACKERS] SSI and Hot Standby

2011-01-19 Thread Dan Ports
Kevin's suggestion seems eminently reasonable to me and probably the best approach one can do for SSI and hot standby. Pulling it off in time for 9.1 would be a stretch; 9.2 seems quite doable. It's worth noting that one way or another, the semantics of SERIALIZABLE transactions on hot standby

  1   2   >