On Thu, Jul 5, 2012 at 3:05 PM, Noe Nieto <nni...@noenieto.com> wrote: > I was wondering if it's possible to write some sort of "cloudstorage" plugin > for ZODB that saves blobs in amazon S3 or something similar. > > The idea is cool because all things cloud are cool.
Um, well no. :) But if you're in AWS, I think there are some advantages to S3 over EBS. > But there might be some > shortcomings like performance or something like that. What do you think? I took an initial stab, as Satchit pointed out. It was a bit naive. S3 is too slow, especially when committing transactions. I wanna make this work though. My current thinking is: - Save to the blob directory. - Have a separate process that moves files to S3 (copy to s3, then delete from blob directory on success or later). - On blob load, if the file can't be found on the local directory, fetch from S3. There are a number of possible variations: - Don't delete from the blob directory immediately after upload to use it as a cache for new content. - Have the client look in S3 before contacting the ZEO server (although that has the downside of having to distribute keys to the clients). (Maybe only do this if the serial is more than some time in the past.) - Have the client fetch from S3 if the blob isn't in the ZEO server's blob directory (as indicated by a response from the ZEO server). - Have a separate process that serves the blobs deom the blob directory or from S3 via HTTP. I did some experiments that showed significant performance benefits in a heavily loaded ZEO server in serving blobs from an separate HTTP server. If you're going to serve blob data directly, without modification, it may make sense to store it in S3 at the application level, rather than going through ZODB. Alternatively, it would be nice to have a way to get the S3 location for a blob. Maybe this could be done through some extension API. <shrug> Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm _______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
