Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir
On 02/28/2011 07:37 PM, Shane Hathaway wrote: On 02/28/2011 08:19 AM, Maurits van Rees wrote: I wonder if there is some code that mistakenly throws away the wrong blob; it should throw away the oldest one I'd think, and not the one that it just loaded. The workaround is probably just to not set the blob-cache-size this low. But I tried this low setting to work around a different problem, which I will report in a different post. This reveals how naive the blob cache management is in both RelStorage and ZEO. It is smart enough to use the atime of files (thus you should not disable atime on a filesystem that hosts a blob cache), but the cache management does not guarantee downloaded files are kept long enough to actually read them. Oops, I take it back after reviewing the blob code tonight. ZODB 3.9 and above is not naive about this issue. While ZODB 3.8 allows blob files to disappear at the wrong times, ZODB 3.9 has internal retry mechanisms that ensure blob files will be opened even when the blob cache size is too small. RelStorage can not fix this issue. If you want stable blob support, as Hanno said, you need ZODB 3.9 or above. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir
Op 01-03-11 09:17, Shane Hathaway schreef: On 02/28/2011 07:37 PM, Shane Hathaway wrote: On 02/28/2011 08:19 AM, Maurits van Rees wrote: I wonder if there is some code that mistakenly throws away the wrong blob; it should throw away the oldest one I'd think, and not the one that it just loaded. The workaround is probably just to not set the blob-cache-size this low. But I tried this low setting to work around a different problem, which I will report in a different post. This reveals how naive the blob cache management is in both RelStorage and ZEO. It is smart enough to use the atime of files (thus you should not disable atime on a filesystem that hosts a blob cache), but the cache management does not guarantee downloaded files are kept long enough to actually read them. Oops, I take it back after reviewing the blob code tonight. ZODB 3.9 and above is not naive about this issue. While ZODB 3.8 allows blob files to disappear at the wrong times, ZODB 3.9 has internal retry mechanisms that ensure blob files will be opened even when the blob cache size is too small. RelStorage can not fix this issue. If you want stable blob support, as Hanno said, you need ZODB 3.9 or above. Not an option for the short term, as we are still on Plone 3, but good to know, thanks. -- Maurits van Rees Web App Programmer at Zest Software: http://zestsoftware.nl Personal website: http://maurits.vanrees.org/ ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir
On Mon, Feb 28, 2011 at 4:19 PM, Maurits van Rees m.van.r...@zestsoftware.nl wrote: This is with RelStorage 1.5.0-b1, with blob-dir enabled, shared-blob-dir false and a low blob-cache-size, say 10 bytes. Using Plone 3.3.5, plone.app.blob 1.3, plone.app.imaging 1.0.1, but the same is probably true for non-Plone setups. Blobs are considered experimental in ZODB 3.8. Especially the entire blob cache changed completely for ZODB 3.9. I think you might want to upgrade to Plone 4 and ZODB 3.9 to get a stable environment. Or you'll likely run into more problems. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir
On Mon, Feb 28, 2011 at 16:22, Hanno Schlichting ha...@hannosch.eu wrote: Blobs are considered experimental in ZODB 3.8. Especially the entire blob cache changed completely for ZODB 3.9. I think you might want to upgrade to Plone 4 and ZODB 3.9 to get a stable environment. Or you'll likely run into more problems. Actually, RelStorage has it's own cache cleanup code for the database-stored blobs scenario. And by the looks of it, it does trigger a blob cache cleanup (in a separate thread) during blob load. Maurits, see blobhelper.py, BlobHelper.download_blob calls BlobCacheChecker.loaded with check=True (the default). Try calling it with check=False instead. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir
Op 28-02-11 16:34, Martijn Pieters schreef: On Mon, Feb 28, 2011 at 16:22, Hanno Schlichtingha...@hannosch.eu wrote: Blobs are considered experimental in ZODB 3.8. Especially the entire blob cache changed completely for ZODB 3.9. I think you might want to upgrade to Plone 4 and ZODB 3.9 to get a stable environment. Or you'll likely run into more problems. Thanks for the advice, Hanno, also on the plone-dev list. This is the first time I have read such a clear warning against using blobs in Plone 3 though (same for Zope 2.10.x I guess). http://plone.org/products/plone.app.blob also gives no clear warning. What do others think? I concur that the combination with relstorage could be less stable, with some more bugs that need to be ironed out, at least in the blob cache. Looks like most of that part of the RelStorage code was directly copied from a more ZODB3 version though, at least according to a comment in blobhelper.py. Actually, RelStorage has it's own cache cleanup code for the database-stored blobs scenario. And by the looks of it, it does trigger a blob cache cleanup (in a separate thread) during blob load. Maurits, see blobhelper.py, BlobHelper.download_blob calls BlobCacheChecker.loaded with check=True (the default). Try calling it with check=False instead. That works: the blobs stay. But then it looks like the blob cache size is only checked once when starting up the zope instance. For example with a blob cache size of 100,000 bytes and two images of about 65K and one of about 1MB, I don't see any blobs getting removed from the cache while visiting those images or pages that show them. -- Maurits van Rees Web App Programmer at Zest Software: http://zestsoftware.nl Personal website: http://maurits.vanrees.org/ ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir
On Mon, Feb 28, 2011 at 5:09 PM, Maurits van Rees m.van.r...@zestsoftware.nl wrote: Op 28-02-11 16:34, Martijn Pieters schreef: On Mon, Feb 28, 2011 at 16:22, Hanno Schlichtingha...@hannosch.eu wrote: Blobs are considered experimental in ZODB 3.8. Especially the entire blob cache changed completely for ZODB 3.9. I think you might want to upgrade to Plone 4 and ZODB 3.9 to get a stable environment. Or you'll likely run into more problems. Thanks for the advice, Hanno, also on the plone-dev list. This is the first time I have read such a clear warning against using blobs in Plone 3 though (same for Zope 2.10.x I guess). http://plone.org/products/plone.app.blob also gives no clear warning. What do others think? To quote Jim: I consider blob support in 3.8 to be somewhat experimental. [1] You can use blobs in Plone 3 / ZODB 3.8. That's how we developed it and then ironed out all the problems. But since we have more stable versions of the software today, it doesn't make a lot of sense to go back to the early development version. ZODB 3.9 has much more stable blob support and direct support for RelStorage (without patches). If you want to use either of those, you do yourself a favor by using the latest stable versions. Hanno [1] https://mail.zope.org/pipermail/zodb-dev/2009-November/012837.html ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir
On Mon, Feb 28, 2011 at 17:09, Maurits van Rees m.van.r...@zestsoftware.nl wrote: That works: the blobs stay. But then it looks like the blob cache size is only checked once when starting up the zope instance. For example with a blob cache size of 100,000 bytes and two images of about 65K and one of about 1MB, I don't see any blobs getting removed from the cache while visiting those images or pages that show them. There is a cache cleanup run during transaction vote as well, but it looks like that only is applied when there are blob changes to commit. But we've now at least established that the cache cleanup in RelStorage is to blame here, so we can focus on _check_blob_cache_size, specifically on line 490 and further. -- Martijn Pieters ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] blobs missing with relstorage and small blob cache dir
On 02/28/2011 08:19 AM, Maurits van Rees wrote: I wonder if there is some code that mistakenly throws away the wrong blob; it should throw away the oldest one I'd think, and not the one that it just loaded. The workaround is probably just to not set the blob-cache-size this low. But I tried this low setting to work around a different problem, which I will report in a different post. This reveals how naive the blob cache management is in both RelStorage and ZEO. It is smart enough to use the atime of files (thus you should not disable atime on a filesystem that hosts a blob cache), but the cache management does not guarantee downloaded files are kept long enough to actually read them. The usual Unixy solution is to pass around an open file descriptor instead of a filename, so that even if the blob file is deleted, the thread that asked for a file is sure to be able to read it. However, the ZODB blob APIs demand a filename, not an open file, so that solution is not feasible. We could make this kind of exception retryable, or we could make the blob cache management refuse to delete files within some time limit, but those solutions would be workarounds too. In the end I think it's best to just tell everyone that the minimum reasonable blob cache size is some multiple of your largest blob. I wouldn't set it lower than 1 GB. I would set it much larger for hosting videos or CD/DVD images. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
