[Zope-dev] Non-undoable storage
Hia guys, during testing of a mail product I've discovered that the Data.fs file may bloat considerably after storing 50 messages. Packing the database will reduce the Data.fs file to 20 MB (from 40 MB). Another thing is that storing 50 messages takes a *long time* on a 600Mhz 256 MB RAM system. The thing that struck me was that transactions may cause a lot of overhead, both in database size and for execution speed, so, the thing I'm wondering about is if there exists a non-undoable storage, or something similar that doesn't support transactions.. Another thing I noticed BTW, is that retrieving 500 messages and packing the database on every 50th message (all messages retrieved during one request) would lead to off the charts memory usage, the process ended up using 700 MB of RAM (had to add swap files as the footprint grew!) Cheers, Morten ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non-undoable storage
Morten W. Petersen wrote: during testing of a mail product I've discovered that the Data.fs file may bloat considerably after storing 50 messages. Packing the database will reduce the Data.fs file to 20 MB (from 40 MB). Another thing is that storing 50 messages takes a *long time* on a 600Mhz 256 MB RAM system. Did you catalog each message? What version of Zope? The thing that struck me was that transactions may cause a lot of overhead, both in database size and for execution speed, so, the thing I'm wondering about is if there exists a non-undoable storage, or something similar that doesn't support transactions.. Yep, see BerkeleyStorage. Another thing I noticed BTW, is that retrieving 500 messages and packing the database on every 50th message (all messages retrieved during one request) would lead to off the charts memory usage, the process ended up using 700 MB of RAM (had to add swap files as the footprint grew!) You could either: 1) Import messages in a single transaction with subtransactions. This would take less time and incur less memory consumption and database bloat. We at DC have experimented with bulk-loading mail into Zope and this is what works the best. 2) Change the database cache parameters to keep the RAM cache as low as possible (set the timeout to 3 seconds and the target size to 10). 3) Manually zap the caches periodically, which is a capability of Zope 2.4.x. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non-undoable storage
Shane Hathaway wrote: Morten W. Petersen wrote: during testing of a mail product I've discovered that the Data.fs file may bloat considerably after storing 50 messages. Packing the database will reduce the Data.fs file to 20 MB (from 40 MB). Another thing is that storing 50 messages takes a *long time* on a 600Mhz 256 MB RAM system. Did you catalog each message? What version of Zope? Hold your tongue, Shane! ;-) The thing that struck me was that transactions may cause a lot of overhead, both in database size and for execution speed, so, the thing I'm wondering about is if there exists a non-undoable storage, or something similar that doesn't support transactions.. Yep, see BerkeleyStorage. Another thing I noticed BTW, is that retrieving 500 messages and packing the database on every 50th message (all messages retrieved during one request) would lead to off the charts memory usage, the process ended up using 700 MB of RAM (had to add swap files as the footprint grew!) You could either: 1) Import messages in a single transaction with subtransactions. This would take less time and incur less memory consumption and database bloat. We at DC have experimented with bulk-loading mail into Zope and this is what works the best. See ZCatalog's ZCatalog.py/catalog_object method for an example of using subtransactions. Note that the cacheFullSweep call in there is necessary to see any RAM savings. 2) Change the database cache parameters to keep the RAM cache as low as possible (set the timeout to 3 seconds and the target size to 10). 3) Manually zap the caches periodically, which is a capability of Zope 2.4.x. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non-undoable storage
On Tue, 12 Jun 2001, Shane Hathaway wrote: Did you catalog each message? What version of Zope? Yes, every message was cataloged. Zope version 2.3.2 3) Manually zap the caches periodically, which is a capability of Zope 2.4.x. Okay, this is interesting. Any examples on how to implement cache zapping? Cheers, Morten ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non-undoable storage
Chris McDonough wrote: Shane Hathaway wrote: Morten W. Petersen wrote: during testing of a mail product I've discovered that the Data.fs file may bloat considerably after storing 50 messages. Packing the database will reduce the Data.fs file to 20 MB (from 40 MB). Another thing is that storing 50 messages takes a *long time* on a 600Mhz 256 MB RAM system. Did you catalog each message? What version of Zope? Hold your tongue, Shane! ;-) Eh? confusedly... Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non-undoable storage
Morten W. Petersen wrote: On Tue, 12 Jun 2001, Shane Hathaway wrote: Did you catalog each message? What version of Zope? Yes, every message was cataloged. Zope version 2.3.2 Were subtransactions in the Catalog turned on (see the Advanced page)? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non-undoable storage
On Tue, 12 Jun 2001, Chris McDonough wrote: Morten W. Petersen wrote: Yes, every message was cataloged. Zope version 2.3.2 Were subtransactions in the Catalog turned on (see the Advanced page)? Yes, and the threshold was at 1. -Morten ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non-undoable storage
On Tue, 12 Jun 2001, Morten W. Petersen wrote: On Tue, 12 Jun 2001, Shane Hathaway wrote: Did you catalog each message? What version of Zope? Yes, every message was cataloged. Zope version 2.3.2 3) Manually zap the caches periodically, which is a capability of Zope 2.4.x. Okay, this is interesting. Any examples on how to implement cache zapping? self._p_jar._resetCache() is how you do it. It doesn't break circular references within the database, but normally there shouldn't be any circular references. Be careful with _resetCache(). It should only be called when you know there aren't any objects waiting to be written (within the thread), such as just after a call to get_transaction().commit(1). Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
