[ZODB-Dev] Wrong blob file being returned (similar to https://mail.zope.org/pipermail/zodb-dev/2011-February/014067.html )
Hi, I have a setup where 4 ZEO clients running on separate machines connect to a single DB server which runs on a different system by itself. The ZEO clients and the DB server all are at version ZODB3-3.10.2. Now, since the last few weeks some of our users have been reporting that they occasionally see incorrect images being returned. On googling I came across the thread below and was wondering whether I am seeing the same thing as this: https://mail.zope.org/pipermail/zodb-dev/2011-February/014067.htm ...although the setup and version is different (ie: ZODB-3.8 and RelStorage). Unfortunately, sorry but I do not know enough about ZODB internals to be able to say for sure. Is there a way I can test whether the problem is indeed with the wrong blob file being returned from the blobcache ? FWIW, we haven't figured out a way to consistently reproduce this error ourselves. Other things that may/may-not be relevant: a. our site is image heavy (36293 blob files) and the servers are behind a load balancer so in a single request to the web-app (a repoze.bfg site) we might even load collectively 20+ blobs from any of the 4 servers. b. zeo connection string on the clients zodb_uri = zeo://xxx..xxx.xxx:8886/?blob_dir=%(here)s/../var/blobsshared_blob_dir=falseconnection_pool_size=50cache_size=1024MBdrop_cache_rather_verify=true c. $ cat var/blobs/.layout zeocache Any comments/suggestion on how to isolate and fix this problem would be appreciated. cheers, - steve -- random spiel: http://lonetwin.net/ what i'm stumbling into: http://lonetwin.stumbleupon.com/ ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Wrong blob file being returned (similar to https://mail.zope.org/pipermail/zodb-dev/2011-February/014067.html )
On Tue, Jul 12, 2011 at 6:33 AM, steve st...@lonetwin.net wrote: Hi, I have a setup where 4 ZEO clients running on separate machines connect to a single DB server which runs on a different system by itself. The ZEO clients and the DB server all are at version ZODB3-3.10.2. Now, since the last few weeks some of our users have been reporting that they occasionally see incorrect images being returned. On googling I came across the thread below and was wondering whether I am seeing the same thing as this: https://mail.zope.org/pipermail/zodb-dev/2011-February/014067.htm ...although the setup and version is different (ie: ZODB-3.8 and RelStorage). Yeah, quite a bit different. Unfortunately, sorry but I do not know enough about ZODB internals to be able to say for sure. Is there a way I can test whether the problem is indeed with the wrong blob file being returned from the blobcache ? FWIW, we haven't figured out a way to consistently reproduce this error ourselves. Dang. Other things that may/may-not be relevant: a. our site is image heavy (36293 blob files) and the servers are behind a load balancer so in a single request to the web-app (a repoze.bfg site) we might even load collectively 20+ blobs from any of the 4 servers. b. zeo connection string on the clients zodb_uri = zeo://xxx..xxx.xxx:8886/?blob_dir=%(here)s/../var/blobsshared_blob_dir=falseconnection_pool_size=50cache_size=1024MBdrop_cache_rather_verify=true c. $ cat var/blobs/.layout zeocache Any comments/suggestion on how to isolate and fix this problem would be appreciated. We have a number of large apps with multiple terabytes of blobs and a vaguely similar configuration. We haven't seen this sort of problem. One difference is that we set the blob cache size. I don't suppose you're running of disk space? The only suggestion I have is to keep an eye on it and try to reporoduce the problem. I would think that if a request returns an incorrect Blob, it would continue to. If someone reports a bad blob, get the URL and see if you can reproduce by making the same request to each of the app servers, bypassing the load balencer. If one server is being bad, you can remove it from the LB pool to debug it. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Wrong blob file being returned (similar to https://mail.zope.org/pipermail/zodb-dev/2011-February/014067.html )
On Tuesday 12 July 2011, steve wrote: Hi, I have a setup where 4 ZEO clients running on separate machines connect to a single DB server which runs on a different system by itself. The ZEO clients and the DB server all are at version ZODB3-3.10.2. Now, since the last few weeks some of our users have been reporting that they occasionally see incorrect images being returned. One thing you may want to look at is the load balancer. Apache has a bug that keeps being opened and closed again for swapping data between requests under load. Because it happens at the apache level and not the zope level you will never see this problem in any of the zope logs. Just make sure you don't have a similar situation or you could end up debugging the wrong thing to a huge waste of time. In my case I spent a lot of time debugging zope and when I finally discovered it was apache that was screwing up I ended up just dumping apache for nginx. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
