Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
A few high-level observations. (I don't have time to get into the weeds.) 1. The parent pointers used in Zope 3 are, obviously, a problem for ZODB exports. You've already noticed this. A possible way to mitigate this would be to extend the export API to be able to omit objects from the export. Then, when copying a tree, you could tell it to omit the parent of the root. There would have to be some way to tell it what to use instead. 2. I doubt that blobs have been factored into ZODB exports. This is, obviously, an oversight. 3. I think that zope.fssync/zope.app.fssync might provide a better export/import technology, although they *may* require more work. On the up site, they are more pluggable. We are, *finally*, about to start using zope.fssync in a production application. Jim On Oct 24, 2008, at 2:09 AM, Jeff Shell wrote: For the past few years we've been using an ugly, but working, trick to export data out of a Zope 3 instance without exporting every bit of content. Given an object to export, we use 'locationCopy' (zope.location.pickling) to get a deep copy, set the __parent__ attribute of the copied object to None, put it in the ZODB root (root[EXPORT_KEY] = detached_copy), and then commit the transaction. This is so that we have the 'oid' and connection needed for ZODB's 'exportFile' to work. If there's a better way of setting `__parent__` to None (or just not following the __parent__ ref during Export, without affecting the saved state in the database), I'd love to know. But that's an aside. I exported some content tonight that had Blobs in it. Upon import, I noticed that the blob files in the destination were empty (zero- length). Looking at the ZEXP file, I could see the BLOBSTART markers, but could see no binary data. Tracing down into the export, and then just playing around with 'locationCopy' to make other detached copies, I noticed that the Blob contents were never copied into the new file. What can I do here? I'm in the midst of a botched deployment of some content updates for a customer. I thought that copying something 'containing' a Blob would cause its contents to be copied. If this is an exercise for application layer code, how can I get plugged in and know when a Blob is being copied inside of a deep copy? Thanks, -- Jeff Shell [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: 2. I doubt that blobs have been factored into ZODB exports. This is, obviously, an oversight. They were factored in and we have tests. However, the initial pickle will empty them: copying blobs this way including their content isn't currently supported. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] NotImplementedError when attempting to undo transactions
Jim Fulton wrote: File /opt/Zope-2.11/lib/python/ZODB/DB.py, line 809, in abort raise NotImplementedError NotImplementedError You can ignore this error. It has no consequence. What does it mean? (It prevented my undo, which was a bit of a bummer :-S There's a bunch of stuff going on in this code that really ought to get cleaned up. Among other things, that would eliminate this error. What is this code? cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
Christian Theune wrote: On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: 2. I doubt that blobs have been factored into ZODB exports. This is, obviously, an oversight. They were factored in and we have tests. However, the initial pickle will empty them: copying blobs this way including their content isn't currently supported. That doesn't sound like factored in to me, that sounds like bodged and ignored ;-) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
On Oct 24, 2008, at 10:00 AM, Christian Theune wrote: On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: 2. I doubt that blobs have been factored into ZODB exports. This is, obviously, an oversight. They were factored in and we have tests. However, the initial pickle will empty them: copying blobs this way including their content isn't currently supported. So were they factored in without tests? :) Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
On Fri, 2008-10-24 at 15:06 +0100, Chris Withers wrote: Christian Theune wrote: On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: 2. I doubt that blobs have been factored into ZODB exports. This is, obviously, an oversight. They were factored in and we have tests. However, the initial pickle will empty them: copying blobs this way including their content isn't currently supported. That doesn't sound like factored in to me, that sounds like bodged and ignored ;-) No. I was pointing out that Jeff first does a pickle and then the export. *His* pickle breaks it. The export works: it exports the (then empty) blob. -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
Christian Theune wrote: On Fri, 2008-10-24 at 15:06 +0100, Chris Withers wrote: Christian Theune wrote: On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: 2. I doubt that blobs have been factored into ZODB exports. This is, obviously, an oversight. They were factored in and we have tests. However, the initial pickle will empty them: copying blobs this way including their content isn't currently supported. That doesn't sound like factored in to me, that sounds like bodged and ignored ;-) No. I was pointing out that Jeff first does a pickle and then the export. *His* pickle breaks it. The export works: it exports the (then empty) blob. What should he have done to get the blob to export as expected? Chris (who's always been shy of using blobs for exactly these reasons..) -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] NotImplementedError when attempting to undo transactions
Carlos de la Guardia wrote: That's happened to me before. As Jim says, this error is not the culprit. If you check the error log on the ZMI you will find the *real* error. This *was* the error that showed up in the ZMI... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
On Oct 24, 2008, at 5:06 AM, Christophe Combelles wrote: What can I do here? I'm in the midst of a botched deployment of some content updates for a customer. I thought that copying something 'containing' a Blob would cause its contents to be copied. If this is an exercise for application layer code, how can I get plugged in and know when a Blob is being copied inside of a deep copy? I suppose it is related to this? https://bugs.launchpad.net/zope3/+bug/240381 It is exactly related to that. As Christian mentions elsewhere in this thread, it's the copy that kills the blob content due to something happening at Pickle time. Does the patch in that bug report (which patches locationCopy) work on all objects inside of a deep copy? Or does it only work on the root object being copied? Thanks, Jeff Shell [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)
On Oct 24, 2008, at 7:51 AM, Jim Fulton wrote: A few high-level observations. (I don't have time to get into the weeds.) 1. The parent pointers used in Zope 3 are, obviously, a problem for ZODB exports. You've already noticed this. A possible way to mitigate this would be to extend the export API to be able to omit objects from the export. Then, when copying a tree, you could tell it to omit the parent of the root. There would have to be some way to tell it what to use instead. Since Export appears to work by following OID references, I guess a stupid simple way of providing alternate values/lookups for certain OIDs might work. I'm not sure how zc.copy / zope.location.pickling.locationCopy actually do their work (the guts of pickling are a mystery to me), but I imagine that there could be a way of providing something similar for exporting - basically some way of finding out if a referenced object is outside the containment zone and providing an alternative value/ reference. 2. I doubt that blobs have been factored into ZODB exports. This is, obviously, an oversight. They're actually factored in quite well. The copy stunt which I use to kill the __parent__ link of the root exported object is the problem. This is the scenario I now worry about - how Blobs will work in applications like Zope which provide 'copy/paste' functionality. 3. I think that zope.fssync/zope.app.fssync might provide a better export/import technology, although they *may* require more work. On the up site, they are more pluggable. We are, *finally*, about to start using zope.fssync in a production application. I'll take a look at those. I think they're what we (Bottlerocket) need (at least at a conceptual level). Thanks, Jeff Shell [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Zope-dev] Blobs and modes
On Fri, 2008-10-24 at 15:50 -0400, Benji York wrote: On Fri, Oct 24, 2008 at 3:28 PM, Jim Fulton [EMAIL PROTECTED] wrote: On Oct 24, 2008, at 3:24 PM, Benji York wrote: Is there a good reason blobs don't support b or t in the mode strings passed to open? I'm refactoring some code that expects a file-like object to use blobs and it wants to pass wb as the mode (which is a sane thing to do). Blobs are implicitly binary. (Note the B in Blob.) I'll add support for b and t to the blob code if no one objects. I object. BTW, this questions should have been asked o zodb-dev. [yep, copied on this message] For perpetuity: In a private discussion Jim explained that the blob .open() method returns things intended to be file-like, but the method itself isn't intended to mimic Python's open function, and as-such doesn't support the additional mode variations. It's intended to be similar but not identical. We originally did support 'b' and 't' but decided, as Jim pointed out, hat we only care for binary data and thus only the binary modes. I can't remember the reason that drove us to actually rip it out. Maybe the archive of zodb-dev knows. This distinction is similar to why __init__ methods aren't described in (zope.interface) interfaces; how you construct an object isn't prescribed by the interface exposed by the object itself. I'm not sure what you're saying. It's not the __init__ problem, as .open() is described on the Blob class. Christian -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
