Re: [ZODB-Dev] RelStorage now in Subversion
Dieter Maurer wrote: For "read committed" this means: it garantees that I will only see committed transactions but not necessarily that I will see the effect of a transaction as soon as it is committed. Your conflict resolution requires that it sees a transaction as soon as it is commited. Looking into this more, I believe I found the semantic we need in the PostgreSQL reference for the LOCK statement [1]. It says this about obtaining a share lock in read committed mode: "once you obtain the lock, there are no uncommitted writes outstanding". My understanding of that statement and the rest of the paragraph suggests the following guarantee: in read committed mode, once a reader obtains a share lock on a table, it sees the effect of all previous transactions on that table. In RelStorage, all conflict detection and resolution already happens under the protection of an exclusive lock on the commit_lock table. However, the table we're using for conflict detection is current_object, not commit_lock, so we are not yet fulfilling the conditions of the above-mentioned guarantee. We could be relying on undocumented behavior. It's quite conceivable that Postgres might aggressively release locks at transaction commit, then allow the data updates to flow lazily to other sessions until a share lock is acquired. To correct this, it appears we only need to add "LOCK current_object IN SHARE MODE" before the conflict detection and resolution code. Do you agree that will plug the hole? Your diligence is much appreciated. Shane [1] http://www.postgresql.org/docs/8.1/interactive/sql-lock.html ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage now in Subversion
Dieter Maurer wrote: Your conflict resolution requires that it sees a transaction as soon as it is commited. The supported relational databases may have this property -- but I expect we do not have a written garantee that this will definitely be the case. I plan to make a test which tries to provoke a conflict resolution failure -- and gives me confidance that the "read committed" of Postgres really has the required property. You have a point. I would be interested in that test as well. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Roché Compaan wrote at 2008-2-3 09:15 +0200: > ... >I have tried different commit intervals. The published results are for a >commit interval of 100, iow 100 inserts per commit. > >> Your profile looks very surprising: >> >> I would expect that for a single insertion, typically >> one persistent object (the bucket where the insertion takes place) >> is changed. About every 15 inserts, 3 objects are changed (the bucket >> is split) about every 15*125 inserts, 5 objects are changed >> (split of bucket and its container). >> But the mean value of objects changed in a transaction is 20 >> in your profile. >> The changed objects typically have about 65 subobjects. This >> fits with "OOBucket"s. > >It was very surprising to me too since the insertion is so basic. I >simply assign a Persistent object with 1 string attribute that is 1K in >size to a key in a OOBTree. I mentioned this earlier on the list and I >thought that Jim's explanation was sufficient when he said that the >persistent_id method is called for all objects including simple types >like strings, ints, etc. I don't know if it explains all the calls that >add up to a mean value of 20 though. I guess the calls are being made by >the cPickle module, but I don't have the experience to investigate this. The number of "persitent_id" calls suggests that a written persistent object has a mean value of 65 subobjects -- which fits well will OOBuckets. However, when the profile is for commits with 100 insertions each, then the number of written persistent objects is far too small. In fact, we would expect about 200 persistent object writes per transaction: the 100 new persistent objects assigned plus about as many buckets changed by these insertions. > >The keys that I lookup are completely random so it is probably the case >that the lookup causes disk lookups all the time. If this is the case, >is 230ms not still to slow? Unreasonably slow in fact. A tree with size 10**7 does likely not have a depth larger than 4 (internal nodes should typically have at least 125 entries, leaves should have at least 15 -- a tree of depth 4 thus can have about 125**3*15 = 29.x * 10**6). Therefore, one would expect at most 4 disk accesses. On my (6 year old) computer, a disk access can take up to 30 ms. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage now in Subversion
Meanwhile I have carefully studied your implementation. There is only a single point I am not certain about: As I understand isolation levels, they garantee that some bad things will not happen but not that all not bad thing will happen. For "read committed" this means: it garantees that I will only see committed transactions but not necessarily that I will see the effect of a transaction as soon as it is committed. Your conflict resolution requires that it sees a transaction as soon as it is commited. The supported relational databases may have this property -- but I expect we do not have a written garantee that this will definitely be the case. I plan to make a test which tries to provoke a conflict resolution failure -- and gives me confidance that the "read committed" of Postgres really has the required property. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
