Each entity is in an entity group by itself.
I'm not using transactions and It would be too hard to isolate the
code, so I'm not providing an example.
I sidestepped the issue by doing only the network operations in
parallel threads, then use remote_api sequentially. The problem could
have been just that you can't safely use remote_api from multiple
threads...
Along the way I implemented a parallel map function in Python using
threading. Here is it in case someone's interested (remember GIL, it
is only useful for doing tasks heavy on I/O in parallel, it won't take
advantage of multiple cores):
def pmap(function, *iterables):
import threading
class pmap_thread(threading.Thread):
def __init__(self, *args):
super(pmap_thread, self).__init__()
self.result = None
self.args = args
def run(self):
self.result = function(*args)
threads = []
iterables = map(iter, iterables)
try:
while True:
args = [it.next() for it in iterables]
threads.append(pmap_thread(*args))
except StopIteration:
pass
for t in threads:
t.start()
def gen():
for t in threads:
t.join()
yield t.result
return gen() # start all threads immediately and return without
blocking
On 19 Фев, 12:17, Nick Johnson <[email protected]> wrote:
> It's really hard to say based on your description. Are you using
> transactions? Can you show us your code?
>
> -Nick Johnson
>
> On Feb 18, 3:41 pm, Nikola <[email protected]> wrote:
>
>
>
> > I am trying to go through all entities of a kind, using the __key__
> > property. I take 10 at a time, and run threads in parallel to do
> > something and then update these entities (I am doing this viaremote_api).
> > What I often get is:
>
> > TransactionFailedError: too much contention on these datastore
> > entities. please try again.
>
> > How is this possible? I thought the datastore uses optimistic
> > concurrency based on timestamps of individual records, not pages of
> > adjacent entities?
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~----------~----~----~----~------~----~------~--~---
