Re: [Zope-dev] limit to number of operations in a transaction?

2002-09-17 Thread Karl Anderson

Christopher N.Deckard [EMAIL PROTECTED] writes:

 I have roughly 1,500 people in the database.  I wrote a script to
 migrate the people.  Basically it does getProperty for each property
 on each person folder, then creates the new person object which uses
 XML.  Zope apparently cannot handle this number of operations in one
 transaction.  Zope gets slower and slower and eventually becomes on
 responsive.  It looks like the script has completed, but nothing is
 ever committed to the ZODB, and since Zope is unresponsive it must
 be restarted.  This, as expected, kills that entire transaction
 which was never committed.
 
 Is it known that large numbers of operations, such as above, in a
 single transaction can cause problems?  A transaction, of course,
 being a request, and an operation being something like
 manage_addProduct.  

In practice I've only seen problems when I'm dealing with lots of data
and need to be thinking about this anyway :)

 I've solved the problem by using xmlrpc and for person in people
 calling my migrate_person script for only one person at a time. 
 This is SO MUCH FASTER.  I previously ran the script that migrates
 all of the people, and after 8 hours it still had not completed.

As Jens replied, this is because the transactions are getting
committed for each person, and you can do this without xmlrpc by
committing in your script.

The possible drawback of committing after each person is that you're
committing a new version of the object being modified.  Depending on
how you're storing the stuff this can grow your ZODB - if every person
is a node in a single ParsedXML document, you'd be storing 1500
versions, each one person bigger than the last.

You can use transactions and subtransactions to juggle database size,
memory usage, and temp file usage.

-- 
Karl Anderson  [EMAIL PROTECTED]   http://www.monkey.org/~kra/

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



[Zope-dev] limit to number of operations in a transaction?

2002-08-25 Thread Christopher N . Deckard

Hello,
So I've run into a problem.  I had the need to migrate our a
database of people from an old storage format to a new one.  The
old format uses Folders and Properties to store information about
a person.  Name, phone, email, address, etc.  The new format uses a
Product that I wrote which stores said data in a ParsedXML document.

I have roughly 1,500 people in the database.  I wrote a script to
migrate the people.  Basically it does getProperty for each property
on each person folder, then creates the new person object which uses
XML.  Zope apparently cannot handle this number of operations in one
transaction.  Zope gets slower and slower and eventually becomes on
responsive.  It looks like the script has completed, but nothing is
ever committed to the ZODB, and since Zope is unresponsive it must
be restarted.  This, as expected, kills that entire transaction
which was never committed.

Is it known that large numbers of operations, such as above, in a
single transaction can cause problems?  A transaction, of course,
being a request, and an operation being something like
manage_addProduct.  

I've solved the problem by using xmlrpc and for person in people
calling my migrate_person script for only one person at a time. 
This is SO MUCH FASTER.  I previously ran the script that migrates
all of the people, and after 8 hours it still had not completed.

Thoughts?

-Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] limit to number of operations in a transaction?

2002-08-25 Thread Jens Vagelpohl

in your script you must commit a transaction once in a while with 
get_transaction().commit().

when you use xml-rpc then every call to zope will end up being its own 
transaction and you will not have this problem.

jens

On Sunday, Aug 25, 2002, at 10:12 US/Eastern, Christopher N. Deckard 
wrote:

 Hello,
 So I've run into a problem.  I had the need to migrate our a
 database of people from an old storage format to a new one.  The
 old format uses Folders and Properties to store information about
 a person.  Name, phone, email, address, etc.  The new format uses a
 Product that I wrote which stores said data in a ParsedXML document.

 I have roughly 1,500 people in the database.  I wrote a script to
 migrate the people.  Basically it does getProperty for each property
 on each person folder, then creates the new person object which uses
 XML.  Zope apparently cannot handle this number of operations in one
 transaction.  Zope gets slower and slower and eventually becomes on
 responsive.  It looks like the script has completed, but nothing is
 ever committed to the ZODB, and since Zope is unresponsive it must
 be restarted.  This, as expected, kills that entire transaction
 which was never committed.

 Is it known that large numbers of operations, such as above, in a
 single transaction can cause problems?  A transaction, of course,
 being a request, and an operation being something like
 manage_addProduct.

 I've solved the problem by using xmlrpc and for person in people
 calling my migrate_person script for only one person at a time.
 This is SO MUCH FASTER.  I previously ran the script that migrates
 all of the people, and after 8 hours it still had not completed.

 Thoughts?

 -Chris



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )