On Fri, Jan 18, 2013 at 2:38 PM, Claudiu Saftoiu <csaft...@gmail.com> wrote: > >> I wonder if disk latency is the problem?. As a test you could put the >> index.fs file into a tmpfs and see if that improves things, or cat >> index.fs > /dev/null to try and force it into the fs cache. > > > Hmm, it would seem not... the cat happens instantly: > > (env)tsa@sp2772c:~/sports$ time cat Data_IndexDB.fs > /dev/null > > real 0m0.065s > user 0m0.000s > sys 0m0.064s > > The database isn't even very big: > > rw-r--r-- 1 tsa tsa 233M Jan 18 14:34 Data_IndexDB.fs > > Which makes me wonder why it takes so long to load it into memory.... it's > just a bit frustrating that the server has 7gb of RAM and it's proving to be > so difficult to get ZODB to keep ~300 megs of it up in there. Or, indeed, if > linux already has the whole .fs file in a memory cache, where are these > delays coming from? There's something I don't quite understand about this > whole situation...
Some high-level comments: - ZODB doesn't simply load your database into memory. It loads objects when you try to access their state. If you're using ZEO (or relstorage, or neo), each load requires a round-trip to the server. That's typically a millisecond or two, depending on your network setup. (Your database is small, so disk access shouldn't be an issue as it is, presumably in your disk cache. - You say it often takes you a couple of minutes to handle requests. This is obviously very long. It sounds like there's an issue with the way you're using the catalog. It's not that hard get this wrong. I suggest either hiring someone with experience in this area to help you or consider using another tool, like solr. (You could put more details of your application here, but I doubt people will be willing to put in the time to really analyze it and tell you how to fix it. I know I can't.) - solr is so fast it almost makes me want to cry. At ZC, we're increasingly using solr instead of the catalog. As the original author of the catalog, this makes me sad, but we just don't have the time to put in the effort to equal solr/lucene. - A common mistake when using ZODB is to use it like a relational database, puting most data in catalog-like data structures and querying to get most of your data. The strength of a OODB is that you don't have to query to get data from a well-designed object model. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm _______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
