Hmmn, I should have realized this.
This is one of those cases where functionality was needed and got added
quickly, now needs to be "kept afloat" to buy time in order to implement
it more sensibly.
What was originally going to be small imports, now has the potential to
have each single import request run for several hours just creating the
data (e.g. saw a 2h one this morning). We are already using savepoints,
but during the course of the import the objects do logically come in
"units" (sort of akin to a human creating them individually by filling
out forms, which would of course be separate transactions); besides
concerns for ConflictErrors, there is value to the objects incrementally
appearing to other users even while the import is still running. (So
e.g. QueueCatalog helps some aspects of this, but not all -- there is
value to this being dealt with as multiple transactions).
Candidates for tackling this in the longer run, include:
- Having an external process split the file in small batches, and
sending them as individual web requests. (But it seems there must be a
- Having a separate app connecting to ZEO, using Zope2.app() and
manipulating everything programmatically outside of the publisher.
(Though this is new terrain for me, and I haven't had much time to check
this out and all the various ramifications -- e.g the import needs to
use things like Plone's _createObjectByType(), and potentially may also
want to piggy-back on existing formlib validation).
- Some sort of queuing system, perhaps lovely.remotetask (again new terrain)
( BTW its only a handful of back-office administrative users who need
this type of function, so their "user experience" of the import is not
For the short term, I have a feeling that breaking the CSV into smaller
chunks externally and running separate requests, will be a faster
interim solution, than trying to run multiple transactions inside the
request properly (sounds like that would need monkeypatching).
Chris Withers wrote:
Charl Coetzee wrote:
# end of while loop
really does have the desired effect, where also ConflictError
handling etc continue to work as expected, without nasty side effects
No, this does not handle conflict errors.
The conflict error will be caught in the publisher and the whole
request will be retried, resulting in whatever commits you've manually
done up until that point being re-done...
Why are you using commits rather than savepoints here?
Zope maillist - Zope@zope.org
** No cross posts or HTML encoding! **
(Related lists -