Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-31 Thread Christian Theune
Am Montag, den 30.05.2005, 23:59 -0400 schrieb Paul Winkler: > On Mon, May 30, 2005 at 11:11:52AM +0200, Christian Theune wrote: > > * Blocking during large streams is solved by a protocol extension for > > streaming blobs. Additionally you can make the ZEO-Clients share the > > Blob data via a net

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-30 Thread Paul Winkler
On Mon, May 30, 2005 at 11:11:52AM +0200, Christian Theune wrote: > * Blocking during large streams is solved by a protocol extension for > streaming blobs. Additionally you can make the ZEO-Clients share the > Blob data via a networked file system. > * The pData cache problem is solved as well Co

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-30 Thread Dieter Maurer
Jeremy Hylton wrote at 2005-5-29 21:04 -0400: > ... >What's >twisted's RPC layer?) Or at least something less difficult to use >than asyncore. "medusa" (on top of "asyncore") obviously supports multi-threaded request execution. But, probably, it is more difficult than "asyncore"... -- Dieter __

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-30 Thread Christian Theune
Am Sonntag, den 29.05.2005, 10:37 +0200 schrieb Tino Wildenhain: > > > Actually Pdata has some drawbacks. When the blobsupport branch gets > > > declared stable (I think it's not gonna happen in 3.4, but nobody told > > > me otherwise) we'll have really good blob support without this black > > > ma

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-29 Thread Jeremy Hylton
On 5/29/05, Shane Hathaway <[EMAIL PROTECTED]> wrote: > > Would a multi thread ZEO server improve anything here? Especially > > with concurrent access? > > It's possible. Although ZEO talks over the network using async sockets, > it reads files synchronously, so I suspect it will frequently sit a

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-29 Thread Shane Hathaway
Tino Wildenhain wrote: > Am Sonntag, den 29.05.2005, 09:51 +0200 schrieb Andreas Jung: >>The Pdata approach in general is not bad. I have implemented a CVS-like file >>repository lately where we store binary content using a pdata like >>structure. >>Our largest files are around (100MB) and the per

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-29 Thread Tino Wildenhain
Am Sonntag, den 29.05.2005, 09:51 +0200 schrieb Andreas Jung: > > --On 29. Mai 2005 11:29:06 +0200 Christian Theune <[EMAIL PROTECTED]> wrote: > > > Am Samstag, den 21.05.2005, 17:38 +0200 schrieb Christian Heimes: > >> Grab the Zope2 sources and read lib/python/OFS/Image.py. Zope's > >> OFS.Imag

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-29 Thread Andreas Jung
--On 29. Mai 2005 11:29:06 +0200 Christian Theune <[EMAIL PROTECTED]> wrote: Am Samstag, den 21.05.2005, 17:38 +0200 schrieb Christian Heimes: Grab the Zope2 sources and read lib/python/OFS/Image.py. Zope's OFS.Image.Image class (and also Zope3's implementation) is using a so called possible

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-29 Thread Christian Theune
Am Samstag, den 21.05.2005, 17:38 +0200 schrieb Christian Heimes: > Grab the Zope2 sources and read lib/python/OFS/Image.py. Zope's > OFS.Image.Image class (and also Zope3's implementation) is using a so > called possible large data class (Pdata) that is a subclass of Persistent. > > Pdata is us

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-22 Thread Jeremy Hylton
On 5/21/05, DJTB <[EMAIL PROTECTED]> wrote: > [posted to comp.lang.python, mailed to [EMAIL PROTECTED] [Following up to both places.] > I'm having problems storing large amounts of objects in a ZODB. > After committing changes to the database, elements are not cleared from > memory. Since the num

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-21 Thread Dieter Maurer
DJTB wrote at 2005-5-21 13:00 +0200: > ... >I'm having problems storing large amounts of objects in a ZODB. >After committing changes to the database, elements are not cleared from >memory. You can control how many objects the ZODB cache may contain. Note, however, that the objects are usually fl

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-21 Thread Christian Heimes
DJTB wrote: What should I do to make sure RAM is no longer a limiting factor? (in other words: The program should work with any (large) value of self.__range and self.__et_count Because in my case, self.__et_count = 5000 is only a toy example...) I'm now working on a PC with 2.5 GB RAM and even t