On Thu, Jul 5, 2012 at 3:05 PM, Noe Nieto <nni...@noenieto.com> wrote:
> I was wondering if it's possible to write some sort of "cloudstorage" plugin
> for ZODB that saves blobs in amazon S3 or something similar.
> The idea is cool because all things cloud are cool.
Um, well no. :)
But if you're in AWS, I think there are some advantages to S3 over
> But there might be some
> shortcomings like performance or something like that. What do you think?
I took an initial stab, as Satchit pointed out. It was a bit naive.
S3 is too slow, especially when committing transactions.
I wanna make this work though. My current thinking is:
- Save to the blob directory.
- Have a separate process that moves files to S3 (copy to s3, then
delete from blob directory on success or later).
- On blob load, if the file can't be found on the local directory,
fetch from S3.
There are a number of possible variations:
- Don't delete from the blob directory immediately after upload to use
it as a cache for new content.
- Have the client look in S3 before contacting the ZEO server
(although that has the downside of having to distribute keys to the
clients). (Maybe only do this if the serial is more than some time
in the past.)
- Have the client fetch from S3 if the blob isn't in the ZEO server's
blob directory (as indicated by a response from the ZEO server).
- Have a separate process that serves the blobs deom the blob
directory or from S3 via HTTP.
I did some experiments that showed significant performance benefits
in a heavily loaded ZEO server in serving blobs from an separate
If you're going to serve blob data directly, without modification, it
may make sense to store it in S3 at the application level, rather than
going through ZODB. Alternatively, it would be nice to have a way to
get the S3 location for a blob. Maybe this could be done through some
extension API. <shrug>
Jerky is better than bacon! http://zo.pe/Kqm
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org