[Zope] Running multiple transactions during single request
Hi,
I have an app under Plone (2.5.1, Zope 2.9.6, Five 1.4.3), with a
browser view that gets fed csv data, and creates a bunch of objects
(with some further processing) (essentially like a data import problem).
For various reasons, while this view is doing its work (ie during course
of the single request), it seems attractive to commit the added objects
regularly - e.g. to commit after every 5 newly created objects, say.
(as opposed to using savepoints).
I would like to confirm if a view like so:
from Products.Five.formlib import formbase
import transaction
class UploadForm( formbase.PageForm ):
...
...
@form.action(Process)
def process(self, action, data):
...
while someLongLoop:
...
# Create some persistent objects,
# set some attributes on them, etc.
...
if everyFifthPass:
transaction.commit()
transaction.begin()
# end of while loop
really does have the desired effect, where also ConflictError handling
etc continue to work as expected, without nasty side effects etc.
I've looked at ZPublisher/Publish.py that seems to do essentially this
in the publish() function, except that this also does recordMetaData() {
which method in turn carries the comment Is this code needed? -- but
I'm guessing this may be related to the Undo ZMI functionality?
Anyway my view is obviously not calling this repeatedly, is this cause
for concern? }
This is the deepest I've yet dug into the transaction machinery, so I
thought it would be prudent to ask the question -- google doesn't
have much to say on this one,
Thanks in advance for any insights,
Charl
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Running multiple transactions during single request
Charl Coetzee wrote: if everyFifthPass: transaction.commit() transaction.begin() # end of while loop really does have the desired effect, where also ConflictError handling etc continue to work as expected, without nasty side effects etc. No, this does not handle conflict errors. The conflict error will be caught in the publisher and the whole request will be retried, resulting in whatever commits you've manually done up until that point being re-done... Why are you using commits rather than savepoints here? cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Running multiple transactions during single request
Hmmn, I should have realized this. This is one of those cases where functionality was needed and got added quickly, now needs to be kept afloat to buy time in order to implement it more sensibly. What was originally going to be small imports, now has the potential to have each single import request run for several hours just creating the data (e.g. saw a 2h one this morning). We are already using savepoints, but during the course of the import the objects do logically come in units (sort of akin to a human creating them individually by filling out forms, which would of course be separate transactions); besides concerns for ConflictErrors, there is value to the objects incrementally appearing to other users even while the import is still running. (So e.g. QueueCatalog helps some aspects of this, but not all -- there is value to this being dealt with as multiple transactions). Candidates for tackling this in the longer run, include: - Having an external process split the file in small batches, and sending them as individual web requests. (But it seems there must be a better way). - Having a separate app connecting to ZEO, using Zope2.app() and manipulating everything programmatically outside of the publisher. (Though this is new terrain for me, and I haven't had much time to check this out and all the various ramifications -- e.g the import needs to use things like Plone's _createObjectByType(), and potentially may also want to piggy-back on existing formlib validation). - Some sort of queuing system, perhaps lovely.remotetask (again new terrain) ( BTW its only a handful of back-office administrative users who need this type of function, so their user experience of the import is not really important.) For the short term, I have a feeling that breaking the CSV into smaller chunks externally and running separate requests, will be a faster interim solution, than trying to run multiple transactions inside the request properly (sounds like that would need monkeypatching). Thanks, Charl Chris Withers wrote: Charl Coetzee wrote: if everyFifthPass: transaction.commit() transaction.begin() # end of while loop really does have the desired effect, where also ConflictError handling etc continue to work as expected, without nasty side effects etc. No, this does not handle conflict errors. The conflict error will be caught in the publisher and the whole request will be retried, resulting in whatever commits you've manually done up until that point being re-done... Why are you using commits rather than savepoints here? cheers, Chris ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Running multiple transactions during single request
Charl Coetzee wrote: What was originally going to be small imports, now has the potential to have each single import request run for several hours just creating the data (e.g. saw a 2h one this morning). Sounds like you want Stepper ;-) http://www.simplistix.co.uk/software/zope/stepper (I assume you're running ZEO already?) Stepper has support for retrying conflicts. cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
