[ZODB-Dev] Storing blob data out of zodb into amazon s3?
I was wondering if it's possible to write some sort of cloudstorage plugin for ZODB that saves blobs in amazon S3 or something similar. The idea is cool because all things cloud are cool. But there might be some shortcomings like performance or something like that. What do you think? -- --- Noe Nieto NNieto Consulting Services M: nni...@noenieto.com W: http://noenieto.com T: @tzicatl https://twitter.com/#%21/tzicatl Li: Perfil en LinkedIn http://www.linkedin.com/profile/view?id=84300665 ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storing blob data out of zodb into amazon s3?
On Jul 5, 2012, at 3:05 PM, Noe Nieto wrote: I was wondering if it's possible to write some sort of cloudstorage plugin for ZODB that saves blobs in amazon S3 or something similar. The idea is cool because all things cloud are cool. But there might be some shortcomings like performance or something like that. What do you think? You might want to check out: http://svn.zope.org/ZODB/branches/jim-s3-zeo-blobs/ Satchit -- --- Noe Nieto NNieto Consulting Services M: nni...@noenieto.com W: http://noenieto.com T: @tzicatl https://twitter.com/#%21/tzicatl Li: Perfil en LinkedIn http://www.linkedin.com/profile/view?id=84300665 ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev -- Satchidanand Haridas Zope Corporation Phone# (540) 361-1738 ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storing blob data out of zodb into amazon s3?
On Thu, Jul 5, 2012 at 3:05 PM, Noe Nieto nni...@noenieto.com wrote: I was wondering if it's possible to write some sort of cloudstorage plugin for ZODB that saves blobs in amazon S3 or something similar. The idea is cool because all things cloud are cool. Um, well no. :) But if you're in AWS, I think there are some advantages to S3 over EBS. But there might be some shortcomings like performance or something like that. What do you think? I took an initial stab, as Satchit pointed out. It was a bit naive. S3 is too slow, especially when committing transactions. I wanna make this work though. My current thinking is: - Save to the blob directory. - Have a separate process that moves files to S3 (copy to s3, then delete from blob directory on success or later). - On blob load, if the file can't be found on the local directory, fetch from S3. There are a number of possible variations: - Don't delete from the blob directory immediately after upload to use it as a cache for new content. - Have the client look in S3 before contacting the ZEO server (although that has the downside of having to distribute keys to the clients). (Maybe only do this if the serial is more than some time in the past.) - Have the client fetch from S3 if the blob isn't in the ZEO server's blob directory (as indicated by a response from the ZEO server). - Have a separate process that serves the blobs deom the blob directory or from S3 via HTTP. I did some experiments that showed significant performance benefits in a heavily loaded ZEO server in serving blobs from an separate HTTP server. If you're going to serve blob data directly, without modification, it may make sense to store it in S3 at the application level, rather than going through ZODB. Alternatively, it would be nice to have a way to get the S3 location for a blob. Maybe this could be done through some extension API. shrug Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storing blob data out of zodb into amazon s3?
On Thu, Jul 5, 2012 at 1:05 PM, Noe Nieto nni...@noenieto.com wrote: I was wondering if it's possible to write some sort of cloudstorage plugin for ZODB that saves blobs in amazon S3 or something similar. The idea is cool because all things cloud are cool. But there might be some shortcomings like performance or something like that. What do you think? A bit of a tangent: I think this conversation is more interesting in the general case -- not specific to S3/EBS/AWS. I have a different idea bubbling based on eventual need: * I want to partition blobs to different named storage destinations (local, near, far/cloud) based on policy (e.g. byte-length of data, mime detected from magic number / file signature of data, or some kind of MRU/LRU ruleset). * I would rather leave the actual location and implementation details up to filesystem... e.g. local FS, NFS, FUSE. * My primary use case is shuffling big (video) files to NFS instead of storing on my VPS/VM local storage, but I may want to cache MRU files and certainly keep all small blobs on local storage. But the idea here is just as applicable to any IaaS case where vendor(s) (in my case, my University's internal/campus cloud services) provide multiple storage venues (VPS/VM local, EBS, S3, etc). Sean ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
