Responses inline.
On Thu, Sep 16, 2010 at 17:32, Ben <[email protected]> wrote: > I have a question about Brett Slatkin's talk at I/O 2010 on data > pipelines. The question is about slide #67 of his pdf, corresponding > to minute 51:30 of his talk > http://code.google.com/events/io/2010/sessions/high-throughput-data-pipelines-appengine.html > > I am wondering what is supposed to happen in the transactional task > (bullet point 2c). Would these updates to the materialized view cause > you to write too frequently to the entity group containing the > materialized view? I think there are really two different approaches you can use to insert your work models. 1) The work models get added to the original entity's group. So, inside of the original transaction you do not write to the entity group containing the materialized view -- so no contention on it. Commit the transaction and proceed to step 3. 2) You kick off a transactional task to insert the work model, or fan-out more tasks to create work models :). Then you proceed to step 3. You can use method 1 if you have only a few aggregates. If you have more aggregates use the second method. I have a "library" I am almost ready to open source that makes method 2 really easy, so you can have lots of aggregates. I'll post to this group when I release it. > > And a related question, what happens if there is a failure just after > the transaction in bullet #2, but right before the named task gets > inserted in bullet #3. In my current implementation I just left out > the transactional task (bullet point 2c) but I think that causes me to > lose the eventual consistency. Failure between steps 2 and 3 just means _that_ particular update will not try to kick-off, ie insert, the fan-in (aggregation) task. But it might have already been inserted by the previous update, or the next update. However, if nothing else kicks of the fan-in task you will need some periodic "cleanup" method to catch the update and kick of the fan-in task. Depending on exactly how you implemented step 2 you may not need a transactional task. Robert > > Thanks! > -Ben > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
