This may be more of a zodb / relstorage question - I hope it's ok to ask on the Zope list.

I'm seeing behavior using relstorage and blobs that I didn't expect:
If I upload a large file, say 2 gigs, I am noticing that our SQL database also grows by 2 Gigs, along with the blob storage. After a pack, the space is reclaimed on the SQL side, and everyone is happy.
  FWIW - it's videos that are doing this.

I am pretty sure it's the undo log that's growing, based on the fact that a pack reclaims the space.

Can this behavior be turned off for a specific field or content type? So undo logs are preserved for everything BUT this monster of a content type?

Seems strange to do this tho.

Are there other alternatives, like calling .pack() directly on the field's storage after it's set?

Our problem is that our sql database grows to a huge size between our weekly packs, and backups of the sql dumps are becoming unmanageable. Our blob backups are ready to deal with this kind of size, but not the sql backups.

Going deeper down the rabbit hole, although I don't think it's relevant, is the fact that I hacked and replaced the storage class for the field. Instead of using AnnotationStorage - which I found used as default for ImageField - I intercept the data during storage.set(), ship it out to a separate storage facility, and replace the data with a happy message "This is not where your data is" which is then written to the blobs. It works just great - keeping our blob storage growth from going crazy. If you try to 'download' the file from Plone, you'll get the text file with the happy message.

Now that I've been shown that the Blob Storage is functioning just fine, but the SQL storage size is going off the charts, I hope I'm not back at square one.

The goal is to allow users to think they are uploading 4Gb videos into Plone, when under the covers, we're actually shipping the video files off to some fancy off-site storage. (Akamai) So we don't have to store them and back them up on-site, and our blob directories remain manageable in size.

The storage hack can be seen here:

I'm not proud of it, but it works.

Mike McFadden
Radio Free Asia
Technical Operations Division
2025 M Street NW
Washington DC 20036 USA

This e-mail message is intended only for the use of the addressee and may 
contain information that is privileged and confidential.  Any unauthorized 
dissemination, distribution or copying is strictly prohibited.  If you receive 
this transmission in error, please contact

Zope maillist  -
**   No cross posts or HTML encoding!  **
(Related lists - )

Reply via email to