Hi, I am trying to implement a fork/join like mechanism using the Taskqueue API. I am running into some problems with transactions.
I have WorkGroup and a WorkItem model. Task A creates a WorkGroup record and then adds Task B to the task queue, giving it the key of the WorkGroup record it created. Task B then grabs the work item, performs an HTTP query (to some remote API) and fetches the first result page. Then, Task B creates k WorkItem records, ancestor set to the original WorkGroup and spawns tasks C1...Ck where k is the number of pages. Each task gets the key of the WorkItem it is to process. Tasks Ci then fetch another page from the 3rd party remote API. It stores the page in its WorkItem record. Then, it checks whether the WorkGroup of the beginning has any unprocessed WorkItems (each WorkItem has a state attribute that is set to DONE when the received page is stored in the record). If there are no more WorkItems for this WorkGroup, the state of the WorkGroup is set to DONE and a new task D is added to the task queue to process all retrieved pages. In order for all of this to work, I have to use atomicity at least in the last step. The part where multiple tasks race for updating the WorkGroup's state to DONE and then possibly add a task has to be placed into a transaction. **However, I am getting a lot of errors with this in the deployed app since the transaction fails too often for tasks.** In pseudocode, this looks like this: 1 Create WorkGroup g. 2 Fetch WorkGroup g and create n Work items i_1..i_n. 3 (k times in parallel) Process work item i_k 3.1 Fetch page via HTTP for work item i_k 3.2 BEGIN TRANSACTION 3.3 Update i_k (state and response) 3.4 If number of undone items of g is == 1 (i_k is the last one!) then update its state 3.5 COMMIT TRANSACTION Is there a better approach to do what I want? I want to use task queues since it seems the easiest way to enforce rate metering on the remote API. Bests, Manuel -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
