Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-03-01 Thread Maurits van Rees
Op 01-03-11 09:17, Shane Hathaway schreef: > On 02/28/2011 07:37 PM, Shane Hathaway wrote: >> On 02/28/2011 08:19 AM, Maurits van Rees wrote: >>> I wonder if there is some code that mistakenly throws away the wrong >>> blob; it should throw away the oldest one I'd think, and not the one >>> that it

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-03-01 Thread Shane Hathaway
On 02/28/2011 07:37 PM, Shane Hathaway wrote: > On 02/28/2011 08:19 AM, Maurits van Rees wrote: >> I wonder if there is some code that mistakenly throws away the wrong >> blob; it should throw away the oldest one I'd think, and not the one >> that it just loaded. >> >> The workaround is probably ju

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Shane Hathaway
On 02/28/2011 08:19 AM, Maurits van Rees wrote: > I wonder if there is some code that mistakenly throws away the wrong > blob; it should throw away the oldest one I'd think, and not the one > that it just loaded. > > The workaround is probably just to not set the blob-cache-size this low. >But

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Martijn Pieters
On Mon, Feb 28, 2011 at 17:09, Maurits van Rees wrote: > That works: the blobs stay.  But then it looks like the blob cache size > is only checked once when starting up the zope instance.  For example > with a blob cache size of 100,000 bytes and two images of about 65K and > one of about 1MB, I d

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Hanno Schlichting
On Mon, Feb 28, 2011 at 5:09 PM, Maurits van Rees wrote: > Op 28-02-11 16:34, Martijn Pieters schreef: >> On Mon, Feb 28, 2011 at 16:22, Hanno Schlichting  wrote: >>> Blobs are considered experimental in ZODB 3.8. Especially the entire >>> blob cache changed completely for ZODB 3.9. >>> >>> I thin

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Maurits van Rees
Op 28-02-11 16:34, Martijn Pieters schreef: > On Mon, Feb 28, 2011 at 16:22, Hanno Schlichting wrote: >> Blobs are considered experimental in ZODB 3.8. Especially the entire >> blob cache changed completely for ZODB 3.9. >> >> I think you might want to upgrade to Plone 4 and ZODB 3.9 to get a >> s

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Martijn Pieters
On Mon, Feb 28, 2011 at 16:22, Hanno Schlichting wrote: > Blobs are considered experimental in ZODB 3.8. Especially the entire > blob cache changed completely for ZODB 3.9. > > I think you might want to upgrade to Plone 4 and ZODB 3.9 to get a > stable environment. Or you'll likely run into more p

Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Hanno Schlichting
On Mon, Feb 28, 2011 at 4:19 PM, Maurits van Rees wrote: > This is with RelStorage 1.5.0-b1, with blob-dir enabled, shared-blob-dir > false and a low blob-cache-size, say 10 bytes.  Using Plone 3.3.5, > plone.app.blob 1.3, plone.app.imaging 1.0.1, but the same is probably > true for non-Plone

[ZODB-Dev] blobs missing with relstorage and small blob cache dir

2011-02-28 Thread Maurits van Rees
Hi, This is with RelStorage 1.5.0-b1, with blob-dir enabled, shared-blob-dir false and a low blob-cache-size, say 10 bytes. Using Plone 3.3.5, plone.app.blob 1.3, plone.app.imaging 1.0.1, but the same is probably true for non-Plone setups. Create two images that are stored as blobs. Each

[ZODB-Dev] Blobs and packing

2010-10-15 Thread McCuaig, Scott (MGS)
I'm the lead developer for a very large Plone/Zope service being used by the Ontario government. We provide our Collaboration Tool Service to all of the 9 clusters and have grown to ~250 sites with ~1 users. We're currently using Plone 3.2.3 with Zope 2.10 and RelStorage 1.4.0 (love it!) with O

Re: [ZODB-Dev] [Blobs] Rsyncing Data.fs/Blob files updated/replace upon object changes?

2010-05-07 Thread Shane Hathaway
On 05/06/2010 10:26 PM, Andreas Jung wrote: > So: assuming that a persistent object is managed as a blob...what will > happen on the filesystem level with the blob file upon a change to this > object? Will the ZODB create a new blob file with the changed content or > will it override/update the exi

[ZODB-Dev] [Blobs] Rsyncing Data.fs/Blob files updated/replace upon object changes?

2010-05-06 Thread Andreas Jung
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The current backup strategy for backing up a Data.fs with blob storage seems to be using repozo for the Data.fs and rsync for the blob storage. Rsyncing the blob storage makes only sense when it would be possible to restore a particular blob file for

Re: [ZODB-Dev] blobs

2009-11-20 Thread Chris Withers
Jeff Shell wrote: > On Nov 20, 2009, at 11:52 AM, Jim Fulton wrote: >>> If configured, sure. >> I would not trust repozo + blob backups. > > I don't trust Blobs. A year ago, I reverted all of the code I had for our CMS > that supported Blobs as it was impossible to copy their data in a basic Zope

Re: [ZODB-Dev] blobs for noobs

2009-11-16 Thread Christian Theune
On 11/17/2009 04:46 AM, Shane Hathaway wrote: > James Bergstra wrote: >> I'm surprised by the complete silence... was this post sent to the >> wrong list? What would be a better one? > > This is the right list. I don't know why you didn't get an answer earlier. > >>> Where can I find informatio

Re: [ZODB-Dev] blobs for noobs

2009-11-16 Thread Shane Hathaway
James Bergstra wrote: > I'm surprised by the complete silence... was this post sent to the > wrong list? What would be a better one? This is the right list. I don't know why you didn't get an answer earlier. >> Where can I find information on using BLOBs outside of plone? I use >> ZODB directl

Re: [ZODB-Dev] blobs for noobs

2009-11-16 Thread James Bergstra
I'm surprised by the complete silence... was this post sent to the wrong list? What would be a better one? James On Fri, Nov 13, 2009 at 11:29 AM, James Bergstra wrote: > Where can I find information on using BLOBs outside of plone?  I use > ZODB directly from my python programs and I would lik

[ZODB-Dev] blobs for noobs

2009-11-13 Thread James Bergstra
Where can I find information on using BLOBs outside of plone?  I use ZODB directly from my python programs and I would like to store large files across ZEO.  I'm using a recent ZODB (3.9.3) - I use runzeo.py to start my server, how do I tell it where on the filesystem to put blob data? - when usi

Re: [ZODB-Dev] Blobs without shared-blob directory

2009-05-15 Thread Hanno Schlichting
Jim Fulton wrote: > On May 15, 2009, at 6:22 AM, Hanno Schlichting wrote: > >> Am I correct to assume that sharing the blob-cache between the four >> processes on the same physical machine is not supported, as these >> don't >> have any mechanism to synchronize read/write activity to the blob-

Re: [ZODB-Dev] Blobs without shared-blob directory

2009-05-15 Thread Jim Fulton
On May 15, 2009, at 6:22 AM, Hanno Schlichting wrote: > Hi. > > I have a configuration with one ZEO database server (ZODB 3.8.1) and > two > physical servers housing four single-threaded ZEO clients each. > There's > some 15gb of blob data in the (Plone) site. > > Setting up and maintaining N

[ZODB-Dev] Blobs without shared-blob directory

2009-05-15 Thread Hanno Schlichting
Hi. I have a configuration with one ZEO database server (ZODB 3.8.1) and two physical servers housing four single-threaded ZEO clients each. There's some 15gb of blob data in the (Plone) site. Setting up and maintaining NFS to share direct access to the blob data is beyond what I can support in p

Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Christian Theune
On Fri, 2008-10-24 at 11:45 -0600, Jeff Shell wrote: > On Oct 24, 2008, at 8:36 AM, Christian Theune wrote: > > Again: blobs export as expected. > > > > However, Jeff pickles/unpickles the subtree to get rid of the > > '__parent__' attribute. This and blobs' inability to get copied around > > in th

Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Jeff Shell
On Oct 24, 2008, at 8:36 AM, Christian Theune wrote: > Again: blobs export as expected. > > However, Jeff pickles/unpickles the subtree to get rid of the > '__parent__' attribute. This and blobs' inability to get copied around > in this specific manner make him end up with empty blobs. This is th

Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Jeff Shell
On Oct 24, 2008, at 7:51 AM, Jim Fulton wrote: > A few high-level observations. (I don't have time to get into the > weeds.) > > 1. The parent pointers used in Zope 3 are, obviously, a problem for > ZODB exports. You've already noticed this. > A possible way to mitigate this would be to e

Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Jeff Shell
On Oct 24, 2008, at 5:06 AM, Christophe Combelles wrote: >> What can I do here? I'm in the midst of a botched deployment of some >> content updates for a customer. I thought that copying something >> 'containing' a Blob would cause its contents to be copied. If this is >> an exercise for applicatio

Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Jim Fulton
On Oct 24, 2008, at 10:36 AM, Christian Theune wrote: > On Fri, 2008-10-24 at 15:22 +0100, Chris Withers wrote: >> Christian Theune wrote: >>> On Fri, 2008-10-24 at 15:06 +0100, Chris Withers wrote: Christian Theune wrote: > On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: >> 2.

Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Christian Theune
On Fri, 2008-10-24 at 15:22 +0100, Chris Withers wrote: > Christian Theune wrote: > > On Fri, 2008-10-24 at 15:06 +0100, Chris Withers wrote: > >> Christian Theune wrote: > >>> On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: > 2. I doubt that blobs have been factored into ZODB exports. Th

Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Chris Withers
Christian Theune wrote: > On Fri, 2008-10-24 at 15:06 +0100, Chris Withers wrote: >> Christian Theune wrote: >>> On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: 2. I doubt that blobs have been factored into ZODB exports. This is, obviously, an oversight. >>> They were factored in a

Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Christian Theune
On Fri, 2008-10-24 at 15:06 +0100, Chris Withers wrote: > Christian Theune wrote: > > On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: > >> 2. I doubt that blobs have been factored into ZODB exports. This is, > >> obviously, an oversight. > > > > They were factored in and we have tests. Howe

Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Jim Fulton
On Oct 24, 2008, at 10:00 AM, Christian Theune wrote: > On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: >> 2. I doubt that blobs have been factored into ZODB exports. This is, >> obviously, an oversight. > > They were factored in and we have tests. However, the initial pickle > will empty th

Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Chris Withers
Christian Theune wrote: > On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: >> 2. I doubt that blobs have been factored into ZODB exports. This is, >> obviously, an oversight. > > They were factored in and we have tests. However, the initial pickle > will empty them: copying blobs this way in

Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Christian Theune
On Fri, 2008-10-24 at 09:51 -0400, Jim Fulton wrote: > 2. I doubt that blobs have been factored into ZODB exports. This is, > obviously, an oversight. They were factored in and we have tests. However, the initial pickle will empty them: copying blobs this way including their content isn't curren

Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Jim Fulton
A few high-level observations. (I don't have time to get into the weeds.) 1. The parent pointers used in Zope 3 are, obviously, a problem for ZODB exports. You've already noticed this. A possible way to mitigate this would be to extend the export API to be able to omit objects from t

Re: [ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-24 Thread Christophe Combelles
Jeff Shell a écrit : > For the past few years we've been using an ugly, but working, trick to > export data out of a Zope 3 instance without exporting every bit of > content. Given an object to export, we use > 'locationCopy' (zope.location.pickling) to get a deep copy, set the > __parent__

[ZODB-Dev] Blobs, Copies, and Exports (Zope 3)

2008-10-23 Thread Jeff Shell
For the past few years we've been using an ugly, but working, trick to export data out of a Zope 3 instance without exporting every bit of content. Given an object to export, we use 'locationCopy' (zope.location.pickling) to get a deep copy, set the __parent__ attribute of the copied object

Re: [ZODB-Dev] Blobs revisited...

2005-10-16 Thread Dieter Maurer
Chris McDonough wrote at 2005-10-8 10:35 -0400: >On Oct 8, 2005, at 6:08 AM, Dieter Maurer wrote: > >> Chris McDonough wrote at 2005-10-5 09:58 -0400: >> >>> ... savepoint difficulty ... >>> Sure. BlobStorage is a wrapper around an existing storage. The >>> wrapped storage's connections might su

Re: [ZODB-Dev] Blobs revisited...

2005-10-08 Thread Chris McDonough
On Oct 8, 2005, at 6:08 AM, Dieter Maurer wrote: Chris McDonough wrote at 2005-10-5 09:58 -0400: ... savepoint difficulty ... Sure. BlobStorage is a wrapper around an existing storage. The wrapped storage's connections might support savepoints, which implies that they support rollback. But

Re: [ZODB-Dev] Blobs revisited...

2005-10-08 Thread Dieter Maurer
Chris McDonough wrote at 2005-10-5 09:58 -0400: > ... savepoint difficulty ... >Sure. BlobStorage is a wrapper around an existing storage. The >wrapped storage's connections might support savepoints, which implies >that they support rollback. But supporting rollback for a >BlobDataManager

Re: [ZODB-Dev] Blobs revisited...

2005-10-05 Thread Chris McDonough
On Oct 5, 2005, at 6:11 AM, Jim Fulton wrote: Chris McDonough wrote: FYI, Last week, Christian Theune and I muscled the elderly ctheune- blobsupport-branch of ZODB into shape to work again against the ZODB trunk. Yay! Did you also resolve the code issues I pointed out to Christian?

Re: [ZODB-Dev] Blobs revisited...

2005-10-05 Thread Jim Fulton
Chris McDonough wrote: FYI, Last week, Christian Theune and I muscled the elderly ctheune- blobsupport-branch of ZODB into shape to work again against the ZODB trunk. Yay! Did you also resolve the code issues I pointed out to Christian? The original "workingness" was a bit suspect. :) We

[ZODB-Dev] Blobs revisited...

2005-10-04 Thread Chris McDonough
FYI, Last week, Christian Theune and I muscled the elderly ctheune- blobsupport-branch of ZODB into shape to work again against the ZODB trunk. We ran into some "interesting" issues with supporting savepoints (difficult to do efficiently with blobs, so we didn't try), I added some somewh