Re: [ZODB-Dev] zeopack requirements?
On Dec 17, 2007, at 12:20 PM, Chris Bainbridge wrote: On 17/12/2007, Jim Fulton <[EMAIL PROTECTED]> wrote: I doubt that taking the database offline would make much difference. How much memory do you have on the machine? How much memory did the server process use before packing? Thanks for your answers (Alan too). The ZEO server is supposed to have 4GB of memory, and a dual-core Pentium 3GHz CPU. I don't have access to actually login and check this, though I have no reason to doubt it. You do now. :) 4GB should be plenty of memory for a 9GB database. Of course, that really depends on the number of objects. I've been experimenting with a 22GB database that packs to about 11GB. It uses about 500MB of memory to pack. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack requirements?
On 17/12/2007, Jim Fulton <[EMAIL PROTECTED]> wrote: > I doubt that taking the database offline would make much difference. > How much memory do you have on the machine? > > How much memory did the server process use before packing? Thanks for your answers (Alan too). The ZEO server is supposed to have 4GB of memory, and a dual-core Pentium 3GHz CPU. I don't have access to actually login and check this, though I have no reason to doubt it. The ZEO server should be the only process running on it, zeopack and the clients run on other PCs. I don't really need a single server though, and it is becoming a bottleneck, so I'm going abandon this approach and instead have each process use its own ZODB file. Thanks again for your help, Chris ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack requirements?
On Dec 17, 2007, at 9:39 AM, Alan Runyan wrote: Makes sense. Is zc.FileStorage an alternative packing strategy? Yes. I expect to roll this work into 3.9. It is a separate package now so I can deploy it with 3.8. It is very much a work in progress. Does it use less memory? No, it uses quite a bit more. :) Memory is pretty cheap. The current packing approach beats the [EMAIL PROTECTED] out of the disk. This new approach tries to be much kinder by storing cross-reference data in memory rather than massively seeking to read cross-reference data from disk. Also, on linus systems, it uses posix_fadvise to avoid trashing the disk cache. It 2-3 times as fast as the old pack algorithm and I think it will be much kinder to the server and other processes on the system. It does most of the work in a sub-process so it can take better advantage of multiple CPUs and, although it uses more memory, it will to a much better job of returning the memory to the OS. I'm not aware that reducing memory is a goal. If it is a goal, we can look at having the packing code actually use a database for the data it needs. This would slow down packing quite a bit, but could limit the amount of memory used. Another thing I looked at was adding an option to skip GC. This seems to speed packing and reduce memory usage quite a bit. I plan to add this as an database configuration option. I'd like to be able to decide whether to do GC at pack time. The pack API doesn't support this. I think I'll eventually resort to using the extension mechanism to add additional pack methods. :) Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack requirements?
Makes sense. Is zc.FileStorage an alternative packing strategy? Does it use less memory? http://svn.zope.org/zc.FileStorage/dev/src/zc/FileStorage/ cheers alan On Dec 17, 2007 8:29 AM, Jim Fulton <[EMAIL PROTECTED]> wrote: > > On Dec 17, 2007, at 9:14 AM, Alan Runyan wrote: > ... > > not believe zeopack will bring the entire database into memory at > > once. It may/ > > probably brings the index into memory. > > > > Suggestion: run zeopack and zeoserver on a separate machine or > > allocate more > > memory to your zeoserver process. > > > zeopack doesn't consume any reasources to speak of. It just tells the > *server* to pack. The resources are consumed by the server. > > > Jim > > -- > Jim Fulton > Zope Corporation > > > -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack requirements?
On Dec 17, 2007, at 9:14 AM, Alan Runyan wrote: ... not believe zeopack will bring the entire database into memory at once. It may/ probably brings the index into memory. Suggestion: run zeopack and zeoserver on a separate machine or allocate more memory to your zeoserver process. zeopack doesn't consume any reasources to speak of. It just tells the *server* to pack. The resources are consumed by the server. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack requirements?
On Dec 17, 2007, at 7:36 AM, Chris Bainbridge wrote: Would bringing the database offline and then packing it work (ie. we run out of memory because we have many active clients), I assume that the clients are running on a different machine. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack requirements?
On Dec 17, 2007, at 7:36 AM, Chris Bainbridge wrote: I have a ~9GB live ZEO database. Last night I tried to pack it, but got this error: 2007-12-17T01:27:08 (4179) Error raised in delayed method Traceback (most recent call last): File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZEO/ StorageServer.py", line 1016, in run result = self._method(*self._args) File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZEO/ StorageServer.py", line 351, in _pack_impl self.storage.pack(time, referencesf) File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/ FileStorage/FileStorage.py", line 1352, in pack opos = p.pack() File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/ FileStorage/fspack.py", line 482, in pack self.gc.findReachable() File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/ FileStorage/fspack.py", line 228, in findReachable self.findReachableAtPacktime([z64]) File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/ FileStorage/fspack.py", line 315, in findReachableAtPacktime self.reachable[oid] = pos File "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/ fsIndex.py", line 84, in __setitem__ tree[key[6:]] = value MemoryError 2007-12-17T01:27:16 (7412) opening storage '1' using FileStorage 2007-12-17T01:34:25 datastore.fs truncated, possibly due to damaged records at 8580513351 2007-12-17T01:34:25 Writing truncated data from datastore.fs to datastore.fs.tr0 Are there any limits on the pack operation? No, but it *does* use a fair bit of memory. I assume that it doesn't have to hold the entire DB in memory at once? No, but it does need to hold some large data structures in memory. Would bringing the database offline and then packing it work (ie. we run out of memory because we have many active clients), or is there some fundamental constraint in the underlying code? I doubt that taking the database offline would make much difference. How much memory do you have on the machine? How much memory did the server process use before packing? Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack requirements?
You are running zeopack on the production machine with "many clients"? Are you running the zeoserver on the same machine, "with many clients"? It seems it is the machine that ZEO server is on that is throwing the MemoryError. I do not believe zeopack will bring the entire database into memory at once. It may/ probably brings the index into memory. Suggestion: run zeopack and zeoserver on a separate machine or allocate more memory to your zeoserver process. Zope NOTE: Running zeo clients with zserver-threads 2 is common and rarely does this impact performance. you will consume much less memory per zeo client than the 'out of the box' zope.conf which specifies zserver-threads 4 (I am -1 on this default) cheers alan On Dec 17, 2007 6:36 AM, Chris Bainbridge <[EMAIL PROTECTED]> wrote: > I have a ~9GB live ZEO database. Last night I tried to pack it, but > got this error: > > 2007-12-17T01:27:08 (4179) Error raised in delayed method > Traceback (most recent call last): > File > "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZEO/StorageServer.py", > line 1016, in run > result = self._method(*self._args) > File > "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZEO/StorageServer.py", > line 351, in _pack_impl > self.storage.pack(time, referencesf) > File > "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/FileStorage/FileStorage.py", > line 1352, in pack > opos = p.pack() > File > "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/FileStorage/fspack.py", > line 482, in pack > self.gc.findReachable() > File > "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/FileStorage/fspack.py", > line 228, in findReachable > self.findReachableAtPacktime([z64]) > File > "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/FileStorage/fspack.py", > line 315, in findReachableAtPacktime > self.reachable[oid] = pos > File > "/home/user/lib/python/ZODB3-3.7.2-py2.4-linux-i686.egg/ZODB/fsIndex.py", > line 84, in __setitem__ > tree[key[6:]] = value > MemoryError > 2007-12-17T01:27:16 (7412) opening storage '1' using FileStorage > 2007-12-17T01:34:25 datastore.fs truncated, possibly due to damaged > records at 8580513351 > 2007-12-17T01:34:25 Writing truncated data from datastore.fs to > datastore.fs.tr0 > > Are there any limits on the pack operation? I assume that it doesn't > have to hold the entire DB in memory at once? Would bringing the > database offline and then packing it work (ie. we run out of memory > because we have many active clients), or is there some fundamental > constraint in the underlying code? > > Thanks for your help, > Chris > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev > -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
