RE: [ZODB-Dev] ZODB memory problems (was: processing a Very Largefile)

2005-06-01 Thread Christian Theune
Am Mittwoch, den 01.06.2005, 12:59 -0400 schrieb Chris McDonough: > How's Sat June 11? I'll just mark that on my calendar as a day to write > some blob tests and I'll catch you (and whoever else might want to > contribute) on irc.freenode.net between then and now to talk about what > needs to be t

RE: [ZODB-Dev] ZODB memory problems (was: processing a Very Largefile)

2005-06-01 Thread Chris McDonough
On Wed, 2005-06-01 at 12:04 +0200, Christian Theune wrote: > Am Dienstag, den 31.05.2005, 16:02 -0400 schrieb Chris McDonough: > > Christian, > > > > Can we pick a day next week to have a sprint via IRC? Would you be > > willing to help write some tests during that sprint? > > Does "next week" i

RE: [ZODB-Dev] ZODB memory problems (was: processing a Very Largefile)

2005-06-01 Thread Christian Theune
Am Dienstag, den 31.05.2005, 16:02 -0400 schrieb Chris McDonough: > Christian, > > Can we pick a day next week to have a sprint via IRC? Would you be > willing to help write some tests during that sprint? Does "next week" include saturday/sunday? We just acquired a new client and next week is ve

Re: [ZODB-Dev] ZODB memory problems

2005-05-31 Thread Shane Hathaway
Tim Peters wrote: > [Jeremy Hylton] > >>>It's really too bad that ZEO only allows a single outstanding request. >>> Restructuring the protocol to allow multiple simulatenous requests >>>was on the task list years ago, but the protocol implementation is so >>>complex I doubt it will get done :-(.

RE: [ZODB-Dev] ZODB memory problems (was: processing a Very Largefile)

2005-05-31 Thread Chris McDonough
Christian, Can we pick a day next week to have a sprint via IRC? Would you be willing to help write some tests during that sprint? On Tue, 2005-05-31 at 11:21 -0400, Tim Peters wrote: > [Paul Winkler] > >> Cool! This is on ctheune-blobsupport branch, right? Did that ever get > >> merged to trun

RE: [ZODB-Dev] ZODB memory problems

2005-05-31 Thread Tim Peters
... [Toby Dickenson] > pb; perspective broker. I thought that suggestion was crazy when Jeremey > first presented it in his blog, but increasing exposure to other twisted > tools makes it seem more reasonable. Just thought I'd guess at the probably-intended references: http://www.python.org/

RE: [ZODB-Dev] ZODB memory problems

2005-05-31 Thread Tim Peters
[Jeremy Hylton] >> It's really too bad that ZEO only allows a single outstanding request. >> Restructuring the protocol to allow multiple simulatenous requests >> was on the task list years ago, but the protocol implementation is so >> complex I doubt it will get done :-(. I can't help but think

Re: [ZODB-Dev] ZODB memory problems

2005-05-31 Thread Toby Dickenson
On Tuesday 31 May 2005 18:43, Shane Hathaway wrote: > Jeremy Hylton wrote: > > It's really too bad that ZEO only allows a single outstanding request. > > Restructuring the protocol to allow multiple simulatenous requests > > was on the task list years ago, but the protocol implementation is so > >

Re: [ZODB-Dev] ZODB memory problems

2005-05-31 Thread Shane Hathaway
Jeremy Hylton wrote: > It's really too bad that ZEO only allows a single outstanding request. > Restructuring the protocol to allow multiple simulatenous requests > was on the task list years ago, but the protocol implementation is so > complex I doubt it will get done :-(. I can't help but think

RE: [ZODB-Dev] ZODB memory problems (was: processing a Very Largefile)

2005-05-31 Thread Christian Theune
Am Dienstag, den 31.05.2005, 11:21 -0400 schrieb Tim Peters: > I hope there's action before September -- that would be half a year after > the branch was created, and merging gets harder over time. Maybe we could > do a virtual sprint via zodb-dev <0.9 wink>. Ok. There is not much todo except of

RE: [ZODB-Dev] ZODB memory problems (was: processing a Very Largefile)

2005-05-31 Thread Tim Peters
[Paul Winkler] >> Cool! This is on ctheune-blobsupport branch, right? Did that ever get >> merged to trunk? [Christian Theune] > AFAIK not. It was a "maybe" for the 3.4 release but really didn't have > enough public exposure to make it in there. That's all correct. 3.4 is in beta now, so blob s

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-31 Thread Christian Theune
Am Montag, den 30.05.2005, 23:59 -0400 schrieb Paul Winkler: > On Mon, May 30, 2005 at 11:11:52AM +0200, Christian Theune wrote: > > * Blocking during large streams is solved by a protocol extension for > > streaming blobs. Additionally you can make the ZEO-Clients share the > > Blob data via a net

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-30 Thread Paul Winkler
On Mon, May 30, 2005 at 11:11:52AM +0200, Christian Theune wrote: > * Blocking during large streams is solved by a protocol extension for > streaming blobs. Additionally you can make the ZEO-Clients share the > Blob data via a networked file system. > * The pData cache problem is solved as well Co

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-30 Thread Dieter Maurer
Jeremy Hylton wrote at 2005-5-29 21:04 -0400: > ... >What's >twisted's RPC layer?) Or at least something less difficult to use >than asyncore. "medusa" (on top of "asyncore") obviously supports multi-threaded request execution. But, probably, it is more difficult than "asyncore"... -- Dieter __

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-30 Thread Christian Theune
Am Sonntag, den 29.05.2005, 10:37 +0200 schrieb Tino Wildenhain: > > > Actually Pdata has some drawbacks. When the blobsupport branch gets > > > declared stable (I think it's not gonna happen in 3.4, but nobody told > > > me otherwise) we'll have really good blob support without this black > > > ma

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-29 Thread Jeremy Hylton
On 5/29/05, Shane Hathaway <[EMAIL PROTECTED]> wrote: > > Would a multi thread ZEO server improve anything here? Especially > > with concurrent access? > > It's possible. Although ZEO talks over the network using async sockets, > it reads files synchronously, so I suspect it will frequently sit a

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-29 Thread Shane Hathaway
Tino Wildenhain wrote: > Am Sonntag, den 29.05.2005, 09:51 +0200 schrieb Andreas Jung: >>The Pdata approach in general is not bad. I have implemented a CVS-like file >>repository lately where we store binary content using a pdata like >>structure. >>Our largest files are around (100MB) and the per

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-29 Thread Tino Wildenhain
Am Sonntag, den 29.05.2005, 09:51 +0200 schrieb Andreas Jung: > > --On 29. Mai 2005 11:29:06 +0200 Christian Theune <[EMAIL PROTECTED]> wrote: > > > Am Samstag, den 21.05.2005, 17:38 +0200 schrieb Christian Heimes: > >> Grab the Zope2 sources and read lib/python/OFS/Image.py. Zope's > >> OFS.Imag

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-29 Thread Andreas Jung
--On 29. Mai 2005 11:29:06 +0200 Christian Theune <[EMAIL PROTECTED]> wrote: Am Samstag, den 21.05.2005, 17:38 +0200 schrieb Christian Heimes: Grab the Zope2 sources and read lib/python/OFS/Image.py. Zope's OFS.Image.Image class (and also Zope3's implementation) is using a so called possible

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-29 Thread Christian Theune
Am Samstag, den 21.05.2005, 17:38 +0200 schrieb Christian Heimes: > Grab the Zope2 sources and read lib/python/OFS/Image.py. Zope's > OFS.Image.Image class (and also Zope3's implementation) is using a so > called possible large data class (Pdata) that is a subclass of Persistent. > > Pdata is us

RE: [ZODB-Dev] ZODB memory problems (was: processing a Very Largefile)

2005-05-24 Thread Tim Peters
[Jeremy Hylton] > ... > It looks like your application has a single persistent instance -- the > root ExtendedTupleTable -- so there's no way for ZODB to manage the > memory. That object and everything reachable from it must be in memory > at all times. Indeed, I tried running this program under

RE: [ZODB-Dev] ZODB memory problems (was: processing a Very Largefile)

2005-05-22 Thread Tim Peters
[Jeremy Hylton] > ... > The ObjectInterning instance is another source of problem, because it's > a dictionary that has an entry for every object you touch. Some vital context was missing in this post. Originally, on c.l.py, DJTB wasn't using ZODB at all. In effect, he had about 5000 lists each

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-22 Thread Jeremy Hylton
On 5/21/05, DJTB <[EMAIL PROTECTED]> wrote: > [posted to comp.lang.python, mailed to [EMAIL PROTECTED] [Following up to both places.] > I'm having problems storing large amounts of objects in a ZODB. > After committing changes to the database, elements are not cleared from > memory. Since the num

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-21 Thread Dieter Maurer
DJTB wrote at 2005-5-21 13:00 +0200: > ... >I'm having problems storing large amounts of objects in a ZODB. >After committing changes to the database, elements are not cleared from >memory. You can control how many objects the ZODB cache may contain. Note, however, that the objects are usually fl

Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

2005-05-21 Thread Christian Heimes
DJTB wrote: What should I do to make sure RAM is no longer a limiting factor? (in other words: The program should work with any (large) value of self.__range and self.__et_count Because in my case, self.__et_count = 5000 is only a toy example...) I'm now working on a PC with 2.5 GB RAM and even t