Hmmn, I should have realized this.

This is one of those cases where functionality was needed and got added quickly, now needs to be "kept afloat" to buy time in order to implement it more sensibly.

What was originally going to be small imports, now has the potential to have each single import request run for several hours just creating the data (e.g. saw a 2h one this morning). We are already using savepoints, but during the course of the import the objects do logically come in "units" (sort of akin to a human creating them individually by filling out forms, which would of course be separate transactions); besides concerns for ConflictErrors, there is value to the objects incrementally appearing to other users even while the import is still running. (So e.g. QueueCatalog helps some aspects of this, but not all -- there is value to this being dealt with as multiple transactions).

Candidates for tackling this in the longer run, include:
- Having an external process split the file in small batches, and sending them as individual web requests. (But it seems there must be a better way). - Having a separate app connecting to ZEO, using and manipulating everything programmatically outside of the publisher. (Though this is new terrain for me, and I haven't had much time to check this out and all the various ramifications -- e.g the import needs to use things like Plone's _createObjectByType(), and potentially may also want to piggy-back on existing formlib validation).
- Some sort of queuing system, perhaps lovely.remotetask (again new terrain)

( BTW its only a handful of back-office administrative users who need this type of function, so their "user experience" of the import is not really important.)

For the short term, I have a feeling that breaking the CSV into smaller chunks externally and running separate requests, will be a faster interim solution, than trying to run multiple transactions inside the request properly (sounds like that would need monkeypatching).



Chris Withers wrote:
Charl Coetzee wrote:
       if everyFifthPass:
       # end of while loop

really does have the desired effect, where also ConflictError handling etc continue to work as expected, without nasty side effects etc.

No, this does not handle conflict errors.
The conflict error will be caught in the publisher and the whole request will be retried, resulting in whatever commits you've manually done up until that point being re-done...

Why are you using commits rather than savepoints here?



Zope maillist  -
**   No cross posts or HTML encoding!  **
(Related lists - )

Reply via email to