Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Jim Fulton

A few high-level observations. (I don't have time to get into the  
weeds.)

1. The parent pointers used in Zope 3 are, obviously, a problem for  
ZODB exports.  You've already noticed this.
  A possible way to mitigate this would be to extend the export  
API to be able to omit objects from the export.
 Then, when copying a tree, you could tell it to omit the parent  
of the root. There would have to be some way
 to tell it what to use instead.

2. I doubt that blobs have been factored into ZODB exports. This is,  
obviously, an oversight.

3. I think that zope.fssync/zope.app.fssync might provide a better  
export/import technology, although they *may* require more work.  On  
the up site, they are more pluggable. We are, *finally*, about to  
start using zope.fssync in a production application.

Jim

On Oct 24, 2008, at 2:09 AM, Jeff Shell wrote:

 For the past few years we've been using an ugly, but working, trick to
 export data out of a Zope 3 instance without exporting every bit of
 content. Given an object to export, we use
 'locationCopy' (zope.location.pickling) to get a deep copy, set the
 __parent__ attribute of the copied object to None, put it in the ZODB
 root (root[EXPORT_KEY] = detached_copy), and then commit the
 transaction. This is so that we have the 'oid' and connection needed
 for ZODB's 'exportFile' to work.

 If there's a better way of setting `__parent__` to None (or just not
 following the __parent__ ref during Export, without affecting the
 saved state in the database), I'd love to know. But that's an aside.

 I exported some content tonight that had Blobs in it. Upon import, I
 noticed that the blob files in the destination were empty (zero-
 length). Looking at the ZEXP file, I could see the BLOBSTART markers,
 but could see no binary data. Tracing down into the export, and then
 just playing around with 'locationCopy' to make other detached copies,
 I noticed that the Blob contents were never copied into the new file.

 What can I do here? I'm in the midst of a botched deployment of some
 content updates for a customer. I thought that copying something
 'containing' a Blob would cause its contents to be copied. If this is
 an exercise for application layer code, how can I get plugged in and
 know when a Blob is being copied inside of a deep copy?

 Thanks,

 --
 Jeff Shell
 [EMAIL PROTECTED]

 ___
 For more information about ZODB, see the ZODB Wiki:
 http://www.zope.org/Wikis/ZODB/

 ZODB-Dev mailing list  -  ZODB-Dev@zope.org
 http://mail.zope.org/mailman/listinfo/zodb-dev

--
Jim Fulton
Zope Corporation


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Christian Theune
On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote:
 2. I doubt that blobs have been factored into ZODB exports. This is,  
 obviously, an oversight.

They were factored in and we have tests. However, the initial pickle
will empty them: copying blobs this way including their content isn't
currently supported.

-- 
Christian Theune · [EMAIL PROTECTED]
gocept gmbh  co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1
Zope and Plone consulting and development


signature.asc
Description: This is a digitally signed message part
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Chris Withers
Christian Theune wrote:
 On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote:
 2. I doubt that blobs have been factored into ZODB exports. This is,  
 obviously, an oversight.
 
 They were factored in and we have tests. However, the initial pickle
 will empty them: copying blobs this way including their content isn't
 currently supported.

That doesn't sound like factored in to me, that sounds like bodged and 
ignored ;-)

Chris

-- 
Simplistix - Content Management, Zope  Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Jim Fulton

On Oct 24, 2008, at 10:00 AM, Christian Theune wrote:

 On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote:
 2. I doubt that blobs have been factored into ZODB exports. This is,
 obviously, an oversight.

 They were factored in and we have tests. However, the initial pickle
 will empty them: copying blobs this way including their content isn't
 currently supported.


So were they factored in without tests? :)

Jim

--
Jim Fulton
Zope Corporation


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Christian Theune
On Fri, 2008-10-24 at 15:06 +0100, Chris Withers wrote:
 Christian Theune wrote:
  On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote:
  2. I doubt that blobs have been factored into ZODB exports. This is,  
  obviously, an oversight.
  
  They were factored in and we have tests. However, the initial pickle
  will empty them: copying blobs this way including their content isn't
  currently supported.
 
 That doesn't sound like factored in to me, that sounds like bodged and 
 ignored ;-)

No. I was pointing out that Jeff first does a pickle and then the
export. *His* pickle breaks it. The export works: it exports the (then
empty) blob. 

-- 
Christian Theune · [EMAIL PROTECTED]
gocept gmbh  co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1
Zope and Plone consulting and development


signature.asc
Description: This is a digitally signed message part
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Chris Withers
Christian Theune wrote:
 On Fri, 2008-10-24 at 15:06 +0100, Chris Withers wrote:
 Christian Theune wrote:
 On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote:
 2. I doubt that blobs have been factored into ZODB exports. This is,  
 obviously, an oversight.
 They were factored in and we have tests. However, the initial pickle
 will empty them: copying blobs this way including their content isn't
 currently supported.
 That doesn't sound like factored in to me, that sounds like bodged and 
 ignored ;-)
 
 No. I was pointing out that Jeff first does a pickle and then the
 export. *His* pickle breaks it. The export works: it exports the (then
 empty) blob. 

What should he have done to get the blob to export as expected?

Chris (who's always been shy of using blobs for exactly these reasons..)

-- 
Simplistix - Content Management, Zope  Python Consulting
- http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Jeff Shell
On Oct 24, 2008, at 5:06 AM, Christophe Combelles wrote:
 What can I do here? I'm in the midst of a botched deployment of some
 content updates for a customer. I thought that copying something
 'containing' a Blob would cause its contents to be copied. If this is
 an exercise for application layer code, how can I get plugged in and
 know when a Blob is being copied inside of a deep copy?


 I suppose it is related to this?
 https://bugs.launchpad.net/zope3/+bug/240381


It is exactly related to that. As Christian mentions elsewhere in this  
thread, it's the copy that kills the blob content due to something  
happening at Pickle time.

Does the patch in that bug report (which patches locationCopy) work on  
all objects inside of a deep copy? Or does it only work on the root  
object being copied?



Thanks,
Jeff Shell
[EMAIL PROTECTED]



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Jeff Shell
On Oct 24, 2008, at 7:51 AM, Jim Fulton wrote:

 A few high-level observations. (I don't have time to get into the  
 weeds.)

 1. The parent pointers used in Zope 3 are, obviously, a problem for  
 ZODB exports.  You've already noticed this.
 A possible way to mitigate this would be to extend the export  
 API to be able to omit objects from the export.
Then, when copying a tree, you could tell it to omit the parent  
 of the root. There would have to be some way
to tell it what to use instead.

Since Export appears to work by following OID references, I guess a  
stupid simple way of providing alternate values/lookups for certain  
OIDs might work.

I'm not sure how zc.copy / zope.location.pickling.locationCopy  
actually do their work (the guts of pickling are a mystery to me), but  
I imagine that there could be a way of providing something similar for  
exporting - basically some way of finding out if a referenced object  
is outside the containment zone and providing an alternative value/ 
reference.

 2. I doubt that blobs have been factored into ZODB exports. This is,  
 obviously, an oversight.

They're actually factored in quite well. The copy stunt which I use to  
kill the __parent__ link of the root exported object is the problem.

This is the scenario I now worry about - how Blobs will work in  
applications like Zope which provide 'copy/paste' functionality.

 3. I think that zope.fssync/zope.app.fssync might provide a better  
 export/import technology, although they *may* require more work.  On  
 the up site, they are more pluggable. We are, *finally*, about to  
 start using zope.fssync in a production application.


I'll take a look at those. I think they're what we (Bottlerocket) need  
(at least at a conceptual level).

Thanks,
Jeff Shell
[EMAIL PROTECTED]

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev