Re: [ZODB-Dev] Blobs revisited...
Chris McDonough wrote at 2005-10-8 10:35 -0400: >On Oct 8, 2005, at 6:08 AM, Dieter Maurer wrote: > >> Chris McDonough wrote at 2005-10-5 09:58 -0400: >> >>> ... savepoint difficulty ... >>> Sure. BlobStorage is a wrapper around an existing storage. The >>> wrapped storage's connections might support savepoints, which implies >>> that they support rollback. But supporting rollback for a >>> BlobDataManager is non-trivial because you'd need to make copies of >>> all writable blobs involved in the transaction and maintain the file >>> position state for all writeable and readable blobs in order to roll >>> back to exactly that state. >>> >> >> Why not make the additional wrapper transparent as >> soon the object is written (i.e. directly read (and write) >> from the underlying storage, once the object was modified). >> > >Sorry, I'm afraid I don't understand the suggestion. Could you say >it a different way? As I understood your explanation (I never looked at the implementation): * blob support is implemented on top of a normal storage * you are lazy to implement savepoints in this blob layer My suggestions for this case: * let work your blob layer as it does now until it observes both a write and a savepoint * at this point, it clears the data it may maintain and rereads it from the storage (which is savepoint capabable). You will loose your blob optimizations in the rare case that the blob was both written and savepointed. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs revisited...
On Oct 8, 2005, at 6:08 AM, Dieter Maurer wrote: Chris McDonough wrote at 2005-10-5 09:58 -0400: ... savepoint difficulty ... Sure. BlobStorage is a wrapper around an existing storage. The wrapped storage's connections might support savepoints, which implies that they support rollback. But supporting rollback for a BlobDataManager is non-trivial because you'd need to make copies of all writable blobs involved in the transaction and maintain the file position state for all writeable and readable blobs in order to roll back to exactly that state. Why not make the additional wrapper transparent as soon the object is written (i.e. directly read (and write) from the underlying storage, once the object was modified). Sorry, I'm afraid I don't understand the suggestion. Could you say it a different way? ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs revisited...
Chris McDonough wrote at 2005-10-5 09:58 -0400: > ... savepoint difficulty ... >Sure. BlobStorage is a wrapper around an existing storage. The >wrapped storage's connections might support savepoints, which implies >that they support rollback. But supporting rollback for a >BlobDataManager is non-trivial because you'd need to make copies of >all writable blobs involved in the transaction and maintain the file >position state for all writeable and readable blobs in order to roll >back to exactly that state. Why not make the additional wrapper transparent as soon the object is written (i.e. directly read (and write) from the underlying storage, once the object was modified). -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs revisited...
On Oct 5, 2005, at 6:11 AM, Jim Fulton wrote: Chris McDonough wrote: FYI, Last week, Christian Theune and I muscled the elderly ctheune- blobsupport-branch of ZODB into shape to work again against the ZODB trunk. Yay! Did you also resolve the code issues I pointed out to Christian? The original "workingness" was a bit suspect. :) The one issue I knew about was that _abort was never called on a BlobStorage when a transaction is aborted... it couldn't have been because there was basically a bit of nonsensical code in the method that would have raised an exception. I don't think we solved this. I think we did figure out why it wasn't called -- it seems that methods of a zope.proxy-wrapped object aren't rebound to the wrapper but instead to the wrapped object. I then tried every which way to have some cleanup code invoked on abort (overriding "tpc_abort" instead of "_abort", overriding "abort" instead of "_abort") but without success. I think I'm doing something stupid. Late in the day I think we figured that the abort cleanup code might need to go in the "BlobDataManager" (each blob has its own data manager) instead of the BlobStorage because it's unclear when BlobStorage's tpc_abort/_abort/abort methods will be called. In short, confusion reigns. ;-) We ran into some "interesting" issues with supporting savepoints (difficult to do efficiently with blobs, so we didn't try), I added some somewhat suspect code to support blobs using a TmpStore, and we as always need to write some more tests, but it works and the result is on the blob-merge-branch of ZODB in SVN. Could you eleborate or point at a document that does? Sure. BlobStorage is a wrapper around an existing storage. The wrapped storage's connections might support savepoints, which implies that they support rollback. But supporting rollback for a BlobDataManager is non-trivial because you'd need to make copies of all writable blobs involved in the transaction and maintain the file position state for all writeable and readable blobs in order to roll back to exactly that state. This was more than we could chew off during the sprint, although it'd be possible. But essentially right now, if you have a transaction that involves any blobs and you want to use a savepoint (other than an "optimistic" one), it will raise an error. I hacked TmpStore to have loadBlob and storeBlob methods (required because during a savepoint, TmpStore "becomes" the storage) but there is currently no code to clean up the temporary blob files created while TmpStore is in use. TmpStore is also currently very simple and does no mutex locking like other storages. I don't really understand why not, so this bugs me. We also created a Zope branch to handle the ZODB head. Namely, we used multidatabase support. This is in the zodb-blobs-branch of Zope in SVN. This basically consisted of ripping out old DBTab code and replacing it with calls into ZODB's multidatabase support. It also works. Yay. Does this actually have anything to do with blobs? Nope. We just needed to solve the mounting problem in order to test the ZODB blob-merge-branch within Zope. I would rather have seen these two projects remain separate. Is there a Bloc-specific ZODB branch now? Or is all of the work in a Zope branch? (I'm a bit confused here.) There is a ZODB branch named "blob-merge-branch". That's where all the work is and you can use this without worrying about Zope. There is a Zope branch named zodb-blobs-branch that uses the ZODB "blobs-merge-branch" in various SVN-externals as opposed to a tagged ZODB version. It also has the changes to support multidatabases. This is really just a convenience. I'm still working on this in my downtime. If anyone else is interested in helping out, let me know! Thanks for the work! I'll try to squeeze in some time soon to review the current blob code. Where is it? Cool! svn co svn+ssh://svn.zope.org/repos/main/ZODB/branches/blob-merge-branch - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs revisited...
Chris McDonough wrote: FYI, Last week, Christian Theune and I muscled the elderly ctheune- blobsupport-branch of ZODB into shape to work again against the ZODB trunk. Yay! Did you also resolve the code issues I pointed out to Christian? The original "workingness" was a bit suspect. :) We ran into some "interesting" issues with supporting savepoints (difficult to do efficiently with blobs, so we didn't try), I added some somewhat suspect code to support blobs using a TmpStore, and we as always need to write some more tests, but it works and the result is on the blob-merge-branch of ZODB in SVN. Could you eleborate or point at a document that does? We also created a Zope branch to handle the ZODB head. Namely, we used multidatabase support. This is in the zodb-blobs-branch of Zope in SVN. This basically consisted of ripping out old DBTab code and replacing it with calls into ZODB's multidatabase support. It also works. Yay. Does this actually have anything to do with blobs? I would rather have seen these two projects remain separate. Is there a Bloc-specific ZODB branch now? Or is all of the work in a Zope branch? (I'm a bit confused here.) I'm still working on this in my downtime. If anyone else is interested in helping out, let me know! Thanks for the work! I'll try to squeeze in some time soon to review the current blob code. Where is it? I'll -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
