A few more updates on this,
It looks like Vinny was correct - writes of a smaller size do not seem to
cause a problem. Specifically, writes from backend instances to datastore,
blobstore and cloud storage of 32KBs all display the hanging behaviour I
have been talking about. Writes to each of these of 30KBs all seem to work
without problem (the actual size at which there is a problem seems to vary
slightly depending on the service, presumably because of varying overhead -
I have had full success with writes of 31.5KBs to datastore).
My guess therefore is that there is some kind of block size used somewhere
in the underlying urlfetch/RPC implementation and whenever multiple blocks
are required for the request, there is some probability of the request
hanging. Why this issue affects backend instances only, I have no idea.
Using the deprecated files API for blobstore, it is possible to write large
files simply by writing in chunks of 30KBs:
def write_in_chunks(self,data):
chunk_size = 1024 * 30
file_name = files.blobstore.create(mime_type="text/plain")
num_chunks = int(math.ceil(len(data)/chunk_size))
with files.open(file_name, 'a') as f:
for i in range(0, num_chunks):
f.write(data[i*chunk_size:(i+1)*chunk_size])
files.finalize(file_name)
Using the cloudstorage library, this isn't possible. It looks like this is
because the library only flushes data once 256K characters have been
written; this can be seen in the write method in storage_api.py (the
blocksize is set to 256K on line 489):
def write(self, data):
"""Write some bytes."""
self._check_open()
assert isinstance(data, str)
if not data:
return
self._buffer.append(data)
self._buffered += len(data)
self._offset += len(data)
if self._buffered >= self._blocksize:
self._flush()
I tried fiddling this parameter, but this seems to cause too many requests
to GCS as I started receiving 503 responses, which corresponds to a request
rate that is too high (
https://developers.google.com/storage/docs/reference-status#standardcodes).
Hopefully this is helpful for others and for identifying and fixing the
underlying issue.
Ben
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.
