On Thu, Jun 18, 2009 at 12:00 PM, Dennis <[email protected]> wrote:
> > PS: what does google do internally (wrt coding to avoid datastore > timeouts)? The same as we recommend to everyone else. :) > > On Jun 18, 6:58 pm, Dennis <[email protected]> wrote: > > > I presume by 'data structure', you mean multiple entities? > > > > Yes, multiple entity structures need to be integrity checked each time > > they are used. > > Thanks for the confirmation of how we should code. > > > > > only partially written. In general, though, the 'roll forward' approach > > > works quite well. > > > > What is the 'roll forward' approach? > The approach documented in the talk you referenced: First, record your intent. Then, execute it, and if your transaction fails, a later read can detect this and pick up where it left off ('rolling forward' the transaction). -Nick Johnson > > > > On Jun 18, 6:49 pm, "Nick Johnson (Google)" <[email protected]> > > wrote: > > > > > > > > > On Thu, Jun 18, 2009 at 11:45 AM, Dennis <[email protected]> wrote: > > > > > > I asked about this (datastore timeout exceptions) in yesterday's app > > > > engine chat and just wanted to follow up. > > > > > > I'm trying to figure out an architecture to deal with the timeout > > > > exceptions. > > > > The "solution" of pushing the exception back to the end user is > > > > actually ok for my app. > > > > This works for read timeouts. > > > > > > For write timeouts, my code can be in the middle of storing a data > > > > structure when the timeout happens. > > > > Thus, I could have a corrupt data structure instance in my datastore. > > > > > I presume by 'data structure', you mean multiple entities? Because > > > individual entities (and entity groups inside transactions) will always > be > > > written atomically - all or nothing - and never 'corrupted' or > partially > > > written. > > > > > > So, for fault-tolerant production code, any (later) access to any > > > > datastore data structure should check > > > > the integrity data structure before using it. > > > > This is what Ken Ashcraft's google i/o 2008 talk recommended (without > > > > referring specifically to timeout issues): > > > > > > > http://sites.google.com/site/io/best-practices---building-a-productio... > > > > > > A bit troublesome but at least we know what to do: > > > > every datastore read needs to do data structure integrity checking. > > > > > > I just want to get feedback on whether this line of thinking and the > > > > resulting coding style is what the appengine team recommends. > > > > Also, it would be very helpful to know what google engineers do to > > > > avoid this issue (do they use the same style or do they have internal > > > > tools or libraries that hide this issue from them). > > > > > More or less. If your write order is predictable, some of your queries > will > > > probably not care about incomplete data, or that they may see data > that's > > > only partially written. In general, though, the 'roll forward' approach > > > works quite well. > > > > > -Nick Johnson > > > > > > Dennis > > > > > > search term: DatastoreTimeoutException > > > > > > On Jun 18, 6:01 pm, Iap <[email protected]> wrote: > > > > > 2009/6/18 Nick Johnson (Google) [email protected] > > > > > > > > For datastore puts and deletes, we currently automatically retry > when > > > > > > timeouts occur, and for datastore transactions, your user code is > rerun > > > > if a > > > > > > timeout occurs. Gets and queries are currently not retried. > > > > > > > It's exciting that the GAE inherites the "I'm feeling lucky" > tradition of > > > > > Google. >_* > > > > > > > Why not retry the Get and Queires ? > > > > > If the retry will eventually succeed, > > > > > How about provide a decrator for convenience to do the things > right. > > > > > > > Iap > > > > > -- > > > Nick Johnson, App Engine Developer Programs Engineer > > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration > Number: > > > 368047- Hide quoted text - > > > > > - Show quoted text -- Hide quoted text - > > > > > - Show quoted text -- Hide quoted text - > > > > - Show quoted text - > > > -- Nick Johnson, App Engine Developer Programs Engineer Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
