Colin Phipps created BEAM-2439:
----------------------------------
Summary: Datastore writer can fail to progress if Datastore is slow
Key: BEAM-2439
URL: https://issues.apache.org/jira/browse/BEAM-2439
Project: Beam
Issue Type: Bug
Components: sdk-java-gcp
Reporter: Colin Phipps
Assignee: Stephen Sisk
Priority: Minor
When writing to Datastore, Beam groups writes into large batches (usually 500
entities per write, the maximum permitted by the API). If these writes are slow
to commit on the serving side, the request may time out before all of the
entities are written.
When this happens, it loses any progress that has been made on those entities
(the connector uses non-transactional writes, so some entities might have been
written, but partial results are not returned to the connector so it has to
assume that all entities need rewriting). It will retry the write with the same
set of entities, which may time out in the same way repeatedly. This can be
influenced by factors on the Datastore serving side, some of which are
transient (hotspots) but some of which are not.
We (Datastore) are developing a fix for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
