Hi, The problem isn't that the key isn't retrieved; get result *IS succeeding. *If it failed we wouldn't queue the task.The issue is that the _post_put_hook is running outside of the transaction and is queuing up a task before it's actually been committed (so the task that was queued up will retry and then eventually succeeds [because it ends up being committed].
*The part I'm trying to understand is why the post_put_hook code is running outside of the transaction.* Thanks Rob On Wednesday, September 4, 2019 at 5:04:45 PM UTC+2, Diogo Almeida wrote: > > If an exception is detected by Datastore, it would be raised when the code > calls get_result() > <https://cloud.google.com/appengine/docs/standard/python/ndb/async#using>, > so the key would not return. However, note that “all post- hooks have a > Future argument at the end of the call signature. This Future object holds > the result of the action. You can call get_result() on this Future to > retrieve the result; you can be sure that get_result() won't block, since > the Future is complete by the time the hook is called.” [1] > <https://cloud.google.com/appengine/docs/standard/python/ndb/creating-entity-models#using_model_hooks> > . > > That said, in case you don’t have an exception, the future already has the > result and get_result function is not blocking, occasionally failing to > retrieve the key. Take a look at this > <https://stackoverflow.com/a/15744244> Stack Overflow post with a > suggestion to resolve an issue similar to your case. > > On Tuesday, September 3, 2019 at 2:58:47 PM UTC-4, Rob Curtis wrote: >> >> Thanks Diogo, >> But that’s why we check get_result for the key. If the get_result failed >> for the the future, the key wouldn’t be returned would it? >> >> >> On Tue, 03 Sep 2019 at 20:28, 'Diogo Almeida' via Google App Engine < >> [email protected] <javascript:>> wrote: >> >>> Note that post hooks do not check whether the RPC was successful. The >>> hook runs regardless of failure that might have occurred due to issues, >>> more specifically the contention >>> <https://cloud.google.com/appengine/docs/standard/python/ndb/creating-entity-models> >>> >>> which is when you attempt to write to a single entity group too quickly. >>> Also note that it is normal that a small number of datastore operations >>> will result in timeout in normal operation. Read more here >>> <https://cloud.google.com/appengine/articles/handling_datastore_errors> >>> about the most common datastore issues and here >>> <https://cloud.google.com/appengine/articles/scaling/contention> how to >>> avoid the contention. >>> >>> In case you need any coding assistance, I suggest you post your >>> inquiries on Stack Overflow where the community of developers are better >>> prepared to assist you in that matter. Google Groups is oriented more >>> towards general opinions, trends, and issues of general nature regarding >>> Google Cloud Platform. >>> >>> On Monday, September 2, 2019 at 11:17:01 AM UTC-4, Rob Curtis wrote: >>>> >>>> Hi, >>>> >>>> I have some code that queues up a task inside *_post_put_hook*. >>>> The task retrieves the key and fetches the entity. However sometimes >>>> the worker fails because the object for that key hasn't been created yet, >>>> but will succeed when it next runs. >>>> Note that we're retrieving the object by key, so I expect the data to >>>> be consistent. >>>> >>>> I want to understand WHY the object isn't available. I'm only calling >>>> the enqueue on commit, but it seems this isn't always working. >>>> Below is the example code which does the committing of the entity. >>>> >>>> >>>> @ndb.synctasklet >>>> def log_usage(self): >>>> >>>> @ndb.transactional_tasklet(xg=True) >>>> >>>> def _txn(): >>>> >>>> # some other stuff happens here, but trying to make this example as >>>> simple as possible. >>>> yield Log.insert_document_log_async() >>>> >>>> >>>> >>>> yield _txn() >>>> >>>> >>>> >>>> class Log(ndb.Expando): >>>> >>>> @classmethod >>>> >>>> @ndb.tasklet >>>> >>>> def insert_document_log_async(cls): >>>> >>>> log = cls() >>>> >>>> logging.debug("insert document log in transaction: >>>> {}".format(ndb.in_transaction())) >>>> >>>> yield log.put_async() >>>> >>>> >>>> >>>> @ndb.synctasklet >>>> >>>> def _post_put_hook(self, future): >>>> >>>> >>>> @ndb.synctasklet >>>> >>>> def _callback_on_commit(): >>>> >>>> key = future.get_result() >>>> >>>> yield SqlTaskHelper.enqueue_syncronise_sql_model_async(key) >>>> >>>> >>>> logging.debug("_post_put_hook In transaction: >>>> {}".format(ndb.in_transaction())) >>>> >>>> ndb.get_context().call_on_commit(lambda: _callback_on_commit()) >>>> >>>> >>>> >>>> >>>> >>>> 1. *log_usage *is called. >>>> 2. When calling *insert_document_log_async* logging indicates that >>>> we're in a transaction (insert document log in transaction: *True*). >>>> >>>> 3. But the post put hook* _post_put_hook* In transaction: *False *(so >>>> call_on_commit is executed immediately). The task runs shortly after an >>>> occasional fails. >>>> >>>> >>>> Can someone please shed some light on where the problem is? >>>> >>>> Thanks >>>> Rob >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Google App Engine" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/google-appengine/Zx5LN3AVjUA/unsubscribe >>> . >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected] <javascript:>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/google-appengine/2a78bb0e-79a4-4e4e-97c4-1dd76a2fb6e3%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/google-appengine/2a78bb0e-79a4-4e4e-97c4-1dd76a2fb6e3%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/a5ca6ad8-377c-491b-80b4-867416c2a39f%40googlegroups.com.
