Re: [Zope-dev] Non-undoable storage
On Tue, 12 Jun 2001, Morten W. Petersen wrote: > On Tue, 12 Jun 2001, Shane Hathaway wrote: > > > Did you catalog each message? What version of Zope? > > Yes, every message was cataloged. Zope version 2.3.2 > > > 3) Manually zap the caches periodically, which is a capability of Zope > > 2.4.x. > > Okay, this is interesting. Any examples on how to implement cache > zapping? self._p_jar._resetCache() is how you do it. It doesn't break circular references within the database, but normally there shouldn't be any circular references. Be careful with _resetCache(). It should only be called when you know there aren't any objects waiting to be written (within the thread), such as just after a call to get_transaction().commit(1). Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non-undoable storage
On Tue, 12 Jun 2001, Chris McDonough wrote: > "Morten W. Petersen" wrote: > > > Yes, every message was cataloged. Zope version 2.3.2 > > Were subtransactions in the Catalog turned on (see the Advanced page)? Yes, and the threshold was at 1. -Morten ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non-undoable storage
"Morten W. Petersen" wrote: > > On Tue, 12 Jun 2001, Shane Hathaway wrote: > > > Did you catalog each message? What version of Zope? > > Yes, every message was cataloged. Zope version 2.3.2 Were subtransactions in the Catalog turned on (see the Advanced page)? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non-undoable storage
Chris McDonough wrote: > > Shane Hathaway wrote: > > > > "Morten W. Petersen" wrote: > > > during testing of a mail product I've discovered that the Data.fs file may > > > bloat considerably after storing 50 messages. Packing the database will > > > reduce the Data.fs file to 20 MB (from 40 MB). Another thing is that > > > storing 50 messages takes a *long time* on a 600Mhz 256 MB RAM system. > > > > Did you catalog each message? What version of Zope? > > Hold your tongue, Shane! ;-) Eh? confusedly... Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non-undoable storage
On Tue, 12 Jun 2001, Shane Hathaway wrote: > Did you catalog each message? What version of Zope? Yes, every message was cataloged. Zope version 2.3.2 > 3) Manually zap the caches periodically, which is a capability of Zope > 2.4.x. Okay, this is interesting. Any examples on how to implement cache zapping? Cheers, Morten ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non-undoable storage
Shane Hathaway wrote: > > "Morten W. Petersen" wrote: > > during testing of a mail product I've discovered that the Data.fs file may > > bloat considerably after storing 50 messages. Packing the database will > > reduce the Data.fs file to 20 MB (from 40 MB). Another thing is that > > storing 50 messages takes a *long time* on a 600Mhz 256 MB RAM system. > > Did you catalog each message? What version of Zope? Hold your tongue, Shane! ;-) > > The thing that struck me was that transactions may cause a lot of > > overhead, both in database size and for execution speed, so, the thing I'm > > wondering about is if there exists a non-undoable storage, or something > > similar that doesn't support transactions.. > > Yep, see BerkeleyStorage. > > > Another thing I noticed BTW, is that retrieving 500 messages and packing > > the database on every 50th message (all messages retrieved during one > > request) would lead to off the charts memory usage, the process ended up > > using 700 MB of RAM (had to add swap files as the footprint grew!) > > You could either: > > 1) Import messages in a single transaction with subtransactions. This > would take less time and incur less memory consumption and database > bloat. We at DC have experimented with bulk-loading mail into Zope and > this is what works the best. See ZCatalog's ZCatalog.py/catalog_object method for an example of using subtransactions. Note that the cacheFullSweep call in there is necessary to see any RAM savings. > > 2) Change the database cache parameters to keep the RAM cache as low as > possible (set the timeout to 3 seconds and the target size to 10). > > 3) Manually zap the caches periodically, which is a capability of Zope > 2.4.x. > > Shane > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Non-undoable storage
"Morten W. Petersen" wrote: > during testing of a mail product I've discovered that the Data.fs file may > bloat considerably after storing 50 messages. Packing the database will > reduce the Data.fs file to 20 MB (from 40 MB). Another thing is that > storing 50 messages takes a *long time* on a 600Mhz 256 MB RAM system. Did you catalog each message? What version of Zope? > The thing that struck me was that transactions may cause a lot of > overhead, both in database size and for execution speed, so, the thing I'm > wondering about is if there exists a non-undoable storage, or something > similar that doesn't support transactions.. Yep, see BerkeleyStorage. > Another thing I noticed BTW, is that retrieving 500 messages and packing > the database on every 50th message (all messages retrieved during one > request) would lead to off the charts memory usage, the process ended up > using 700 MB of RAM (had to add swap files as the footprint grew!) You could either: 1) Import messages in a single transaction with subtransactions. This would take less time and incur less memory consumption and database bloat. We at DC have experimented with bulk-loading mail into Zope and this is what works the best. 2) Change the database cache parameters to keep the RAM cache as low as possible (set the timeout to 3 seconds and the target size to 10). 3) Manually zap the caches periodically, which is a capability of Zope 2.4.x. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
