Re: [ZODB-Dev] fixing far-future timestamps (patch)
On Jun 10, 2009, at 6:52 PM, Ross J. Reedstrom wrote: On Tue, Jun 09, 2009 at 06:29:38PM -0400, Jim Fulton wrote: On Jun 9, 2009, at 4:19 PM, Ross J. Reedstrom wrote: Hmm, I seem to not be receiving email from this list, even though I signed up. Anyway, saw this via the archives. Would a doctest in FileStorage be o.k.? Yup. The change is in BaseStorage, but the only case I've experienced uses FileStorage (and I can leverage the tests in there for out-of-order tids ...) Base storage was a mistake. :/ I assume you mean for my patch? No, sorry! BaseStorage itself was a mistake. ... And what's the preferred patch-submission format? *I* prefer svn branches. :) Are you a contributor? Otherwise, a launchpad submission, as Tres suggested, is fine. *checks his zope.org account* 2000? Had it really been that long? Wow. I've got an account, but sadly never completed the contributor agreement (or at least, I don't recall if I have) Hmm, perhaps I should fill one out ... +1 Make sure you fill out the new Zope Foundation version. Tres gave you the link. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] fixing far-future timestamps (patch)
On Tue, Jun 09, 2009 at 06:29:38PM -0400, Jim Fulton wrote: On Jun 9, 2009, at 4:19 PM, Ross J. Reedstrom wrote: Hmm, I seem to not be receiving email from this list, even though I signed up. Anyway, saw this via the archives. Would a doctest in FileStorage be o.k.? Yup. The change is in BaseStorage, but the only case I've experienced uses FileStorage (and I can leverage the tests in there for out-of-order tids ...) Base storage was a mistake. :/ I assume you mean for my patch? I put it next to the other code that also corrects tids by the same means: setting it to None. Walking the code, I don't really see anywhere else to do this, especially keeping it out of the main path: the next stop is tcp_begin, that'd be right out. And what's the preferred patch-submission format? *I* prefer svn branches. :) Are you a contributor? Otherwise, a launchpad submission, as Tres suggested, is fine. *checks his zope.org account* 2000? Had it really been that long? Wow. I've got an account, but sadly never completed the contributor agreement (or at least, I don't recall if I have) Hmm, perhaps I should fill one out ... Ross -- Ross Reedstrom, Ph.D. reeds...@rice.edu Systems Engineer Admin, Research Scientistphone: 713-348-6166 The Connexions Project http://cnx.orgfax: 713-348-3665 Rice University MS-375, Houston, TX 77005 GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E F888 D3AE 810E 88F0 BEDE ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] fixing far-future timestamps (patch)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ross J. Reedstrom wrote: On Tue, Jun 09, 2009 at 06:29:38PM -0400, Jim Fulton wrote: On Jun 9, 2009, at 4:19 PM, Ross J. Reedstrom wrote: And what's the preferred patch-submission format? *I* prefer svn branches. :) Are you a contributor? Otherwise, a launchpad submission, as Tres suggested, is fine. *checks his zope.org account* 2000? Had it really been that long? Wow. I've got an account, but sadly never completed the contributor agreement (or at least, I don't recall if I have) Hmm, perhaps I should fill one out ... Form: http://foundation.zope.org/agreements/ZopeFoundation_Committer_Agreement Directions: http://docs.zope.org/developer/becoming-a-contributor.html Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKMGsa+gerLs4ltQ4RAjTmAJ9IcE2UsvnKSCKf5huTjB10qcN3NgCg3KSq knpo5ZL9qJOXmoEfLASGcmk= =v60H -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] fixing far-future timestamps (patch)
Hmm, I seem to not be receiving email from this list, even though I signed up. Anyway, saw this via the archives. Would a doctest in FileStorage be o.k.? The change is in BaseStorage, but the only case I've experienced uses FileStorage (and I can leverage the tests in there for out-of-order tids ...) And what's the preferred patch-submission format? Ross -- Ross Reedstrom, Ph.D. reeds...@rice.edu Systems Engineer Admin, Research Scientistphone: 713-348-6166 The Connexions Project http://cnx.orgfax: 713-348-3665 Rice University MS-375, Houston, TX 77005 GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E F888 D3AE 810E 88F0 BEDE On Thu, Jun 04, 2009 at 01:13:04PM -0400, Jim Fulton wrote: On Jun 4, 2009, at 12:04 PM, Ross J. Reedstrom wrote: I've successfully used this in a small python script to correct my problem, reading one FileStorage and writing a new one. I thought I should send this here, for comment. Is this something that should go into mainline ZODB? +1, with tests. :) Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] fixing far-future timestamps (patch)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ross J. Reedstrom wrote: ... And what's the preferred patch-submission format? Upload it as an attachment to a new issue in launchpad: https://bugs.launchpad.net/zodb using the format:: $ svn diff -u Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKLtaf+gerLs4ltQ4RAtgvAKCA7Son+upU5/JqJfpGEVImBgv1pACdEjYP HCO5JuHVToUsujsZJ1UJ4tk= =+FJ9 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] fixing far-future timestamps (patch)
On Jun 9, 2009, at 4:19 PM, Ross J. Reedstrom wrote: Hmm, I seem to not be receiving email from this list, even though I signed up. Anyway, saw this via the archives. Would a doctest in FileStorage be o.k.? Yup. The change is in BaseStorage, but the only case I've experienced uses FileStorage (and I can leverage the tests in there for out-of-order tids ...) Base storage was a mistake. :/ And what's the preferred patch-submission format? *I* prefer svn branches. :) Are you a contributor? Otherwise, a launchpad submission, as Tres suggested, is fine. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] fixing far-future timestamps (patch)
Hi all -
I'm one of the unfortunates who managed to break a Data.fs when
migrating a ZEO backend to new hardware. Unfortunately, I missed the
'CRITICAL' error logged by ZOE (aside: is there a fail_on_critical
option somewhere?) and ended up with transaction ids that parse as
timestamps in the year 4732 (or there abouts). This caused surprisingly
few issues for our Zope/Plone install. Of course ZODB itself, didn't
care, it just started using incremental tids. The biggest impact was
that packing had no effect any more. (Versions are Zope 2.9.10, ZODB
3.6.3)
Long story short, I dug into the code for FileStorage, and patched
copyTransactionsFrom to detect and correct future timestamps. This
approach was inspired by the existing code that detects and corrects
out-of-order transactions, as well as the FileStorage __init__ code
that detects future timestamps. I've attached a patch.
I've successfully used this in a small python script to correct my
problem, reading one FileStorage and writing a new one. I thought I
should send this here, for comment. Is this something that should go
into mainline ZODB?
Ross
--
Ross Reedstrom, Ph.D. reeds...@rice.edu
Systems Engineer Admin, Research Scientistphone: 713-348-6166
The Connexions Project http://cnx.orgfax: 713-348-3665
Rice University MS-375, Houston, TX 77005
GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E F888 D3AE 810E 88F0 BEDE
--- BaseStorage.py.orig 2009-05-28 12:17:31.0 -0500
+++ BaseStorage.py 2009-05-28 12:51:34.0 -0500
@@ -414,16 +414,24 @@
print ('Time stamps back in order %s' % (t))
ok=1
+t=TimeStamp(tid)
+t_now = time.time()
+t_now = TimeStamp(*time.gmtime(t_now)[:5] + (t_now % 60,))
+if t t_now:
+print ('Time stamp from the future, resetting: %s' % (t))
+tid = None
+
if verbose:
print _ts
self.tpc_begin(transaction, tid, transaction.status)
+tid=self._tid
for r in transaction:
oid=r.oid
if verbose:
print oid_repr(oid), r.version, len(r.data)
if restoring:
-self.restore(oid, r.tid, r.data, r.version,
+self.restore(oid, tid, r.data, r.version,
r.data_txn, transaction)
else:
pre=preget(oid, None)
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] fixing far-future timestamps (patch)
On Jun 4, 2009, at 12:04 PM, Ross J. Reedstrom wrote: Hi all - I'm one of the unfortunates who managed to break a Data.fs when migrating a ZEO backend to new hardware. Unfortunately, I missed the 'CRITICAL' error logged by ZOE (aside: is there a fail_on_critical option somewhere?) and ended up with transaction ids that parse as timestamps in the year 4732 (or there abouts). This caused surprisingly few issues for our Zope/Plone install. Of course ZODB itself, didn't care, it just started using incremental tids. The biggest impact was that packing had no effect any more. (Versions are Zope 2.9.10, ZODB 3.6.3) Long story short, I dug into the code for FileStorage, and patched copyTransactionsFrom to detect and correct future timestamps. This approach was inspired by the existing code that detects and corrects out-of-order transactions, as well as the FileStorage __init__ code that detects future timestamps. I've attached a patch. I've successfully used this in a small python script to correct my problem, reading one FileStorage and writing a new one. I thought I should send this here, for comment. Is this something that should go into mainline ZODB? +1, with tests. :) Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
