Re: [Zope] Large BTreeFolder2 batching/pagination

2009-11-08 Thread Andreas Jung
Am 08.11.09 13:17, schrieb Ken Ara:
> I take your suggestion to find a Catalog-based solution. I have set the 
> subtransactions very low (100).
>
> But I have got the following error:
>
> Traceback (innermost last):
>   Module ZPublisher.Publish, line 119, in publish
>   Module ZPublisher.mapply, line 88, in mapply
>   Module ZPublisher.Publish, line 42, in call_object
>   Module Products.ZCatalog.ZCatalog, line 348, in manage_catalogFoundItems
>   Module Products.ZCatalog.ZCatalog, line 800, in ZopeFindAndApply
>   Module Products.ZCatalog.ZCatalog, line 559, in catalog_object
>   Module transaction._manager, line 110, in savepoint
>   Module transaction._transaction, line 312, in savepoint
>   Module transaction._transaction, line 309, in savepoint
>   Module transaction._transaction, line 737, in __init__
>   Module ZODB.Connection, line 1034, in savepoint
>   Module ZODB.Connection, line 543, in _commit
>   Module ZODB.Connection, line 571, in _store_objects
>   Module ZODB.Connection, line 1168, in store
> IOError: [Errno 28] No space left on device
>
> I am guessing that this process is using the /tmp directory which 
> unfortunately was incorrectly sized at only 1.9Gb. Can someone confirm this? 
>
> Then either I must find a way to change the /tmp size, or is there a solution 
> through Zope?
export TMPDIR= ...

-aj
<>___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Large BTreeFolder2 batching/pagination

2009-11-08 Thread Ken Ara
I take your suggestion to find a Catalog-based solution. I have set the 
subtransactions very low (100).

But I have got the following error:

Traceback (innermost last):
  Module ZPublisher.Publish, line 119, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 42, in call_object
  Module Products.ZCatalog.ZCatalog, line 348, in manage_catalogFoundItems
  Module Products.ZCatalog.ZCatalog, line 800, in ZopeFindAndApply
  Module Products.ZCatalog.ZCatalog, line 559, in catalog_object
  Module transaction._manager, line 110, in savepoint
  Module transaction._transaction, line 312, in savepoint
  Module transaction._transaction, line 309, in savepoint
  Module transaction._transaction, line 737, in __init__
  Module ZODB.Connection, line 1034, in savepoint
  Module ZODB.Connection, line 543, in _commit
  Module ZODB.Connection, line 571, in _store_objects
  Module ZODB.Connection, line 1168, in store
IOError: [Errno 28] No space left on device

I am guessing that this process is using the /tmp directory which unfortunately 
was incorrectly sized at only 1.9Gb. Can someone confirm this? 

Then either I must find a way to change the /tmp size, or is there a solution 
through Zope?

Thank you.

Ken


--- On Mon, 11/2/09, Andreas Jung  wrote:
>
> We have not been successful to create a Catalog of this many objects.
> The process seemed to time out after many hours, probably
> hardware-bound. Also, the size of the resulting ZODB is of concern,
> but we may try again with the Catalog on a mounted database.
>
we have ZCatalogs with millions of objects..likely you are indexing
everything within one big transaction
without using savepoints or subtransactions..


>
-aj



  
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Large BTreeFolder2 batching/pagination

2009-11-02 Thread Ken Ara
I should also mention that one of these large collections contains objects with 
sequential, but not consecutive, numeric ids (timestrings) like some blogs. 
Users of this database can benefit from the ability to access items added 
around the same time.

With or without a catalog, how would one get the next or previous id in such a 
case?

--- On Mon, 11/2/09, Andreas Jung  wrote:

From: Andreas Jung 
Subject: Re: [Zope] Large BTreeFolder2 batching/pagination
To: "Ken Ara" 
Cc: zope@zope.org
Date: Monday, November 2, 2009, 11:36 AM

Am 02.11.09 11:28, schrieb Ken Ara:
> I agree it is hard to imagine, but I am just the web guy...
>
> If I dare to guess, I would say that what they want to do is improve
> access to the information for the users. The two million objects are
> strongly interlinked, so this is the navigation system used up to now.
> But why not open it up to browsing?
>

Because no human can deal in a reasonable way with 500k object..build
your *own*
custom and working navigation throughout the data records...presenting
the ZMI view for BTreefolders to a human is just sick.
>
> For the content team, it might also be nice to use the ZMI or similar
> when working with this content.
>
> We have not been successful to create a Catalog of this many objects.
> The process seemed to time out after many hours, probably
> hardware-bound. Also, the size of the resulting ZODB is of concern,
> but we may try again with the Catalog on a mounted database.
>
we have ZCatalogs with millions of objects..likely you are indexing
everything within one big transaction
without using savepoints or subtransactions..


>
-aj



  ___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Large BTreeFolder2 batching/pagination

2009-11-02 Thread Andreas Jung
Am 02.11.09 11:28, schrieb Ken Ara:
> I agree it is hard to imagine, but I am just the web guy...
>
> If I dare to guess, I would say that what they want to do is improve
> access to the information for the users. The two million objects are
> strongly interlinked, so this is the navigation system used up to now.
> But why not open it up to browsing?
>

Because no human can deal in a reasonable way with 500k object..build
your *own*
custom and working navigation throughout the data records...presenting
the ZMI view for BTreefolders to a human is just sick.
>
> For the content team, it might also be nice to use the ZMI or similar
> when working with this content.
>
> We have not been successful to create a Catalog of this many objects.
> The process seemed to time out after many hours, probably
> hardware-bound. Also, the size of the resulting ZODB is of concern,
> but we may try again with the Catalog on a mounted database.
>
we have ZCatalogs with millions of objects..likely you are indexing
everything within one big transaction
without using savepoints or subtransactions..


>
-aj
<>___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Large BTreeFolder2 batching/pagination

2009-11-02 Thread Ken Ara
I agree it is hard to imagine, but I am just the web guy... 

If I dare to guess, I would say that what they want to do is improve access to 
the information for the users. The two million objects are strongly 
interlinked, so this is the navigation system used up to now. But why not open 
it up to browsing?

For the content team, it might also be nice to use the ZMI or similar when 
working with this content.

We have not been successful to create a Catalog of this many objects. The 
process seemed to time out after many hours, probably hardware-bound. Also, the 
size of the resulting ZODB is of concern, but we may try again with the Catalog 
on a mounted database.

My general question remains: Is there a way to address the objects contained in 
a BTreeFolder2 using something like an 'offset' or other identifier? Has anyone 
found a strategy that scales better than getBatchObjectListing?

Thanks

Ken

--- On Mon, 11/2/09, Andreas Jung  wrote:

From: Andreas Jung 
Subject: Re: [Zope] Large BTreeFolder2 batching/pagination
To: "Ken Ara" 
Cc: zope@zope.org
Date: Monday, November 2, 2009, 6:23 AM

What kind of practical sense does it make to batch 400k objects? I can no 
imagine single usecase where one
would be interested in walking through such an amount of objects manually. 
Better organize your data in a more

handy way or implement some search logic for bringing the batch size to a 
number of relevant object.s

-aj

On Mon, Nov 2, 2009 at 00:04, Ken Ara  wrote:




Under our setup, beyond 300-400,000 contained objects, the batching provided by 
getBatchObjectListing becomes unusable. I've seen this problem mentioned 
somewhere but never any hint of a solution.








  ___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Large BTreeFolder2 batching/pagination

2009-11-01 Thread Andreas Jung
What kind of practical sense does it make to batch 400k objects? I can no
imagine single usecase where one
would be interested in walking through such an amount of objects manually.
Better organize your data in a more
handy way or implement some search logic for bringing the batch size to a
number of relevant object.s

-aj

On Mon, Nov 2, 2009 at 00:04, Ken Ara  wrote:

>
>
> Under our setup, beyond 300-400,000 contained objects, the batching
> provided by getBatchObjectListing becomes unusable. I've seen this problem
> mentioned somewhere but never any hint of a solution.
>
>
>
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )