That source is not available to you as it is part of the Dataflow service. On Mon, Oct 16, 2017 at 10:25 AM, Derek Hao Hu <[email protected]> wrote:
> Thanks Lukasz! > > "For an unbounded (streaming) pipeline, Dataflow will retry the bundle > forever until the pipeline is cancelled by the user." > > Can you help point out where this behavior is implemented? I'd like to > take a look and see if it is possible to modify it a bit (e.g. write this > to an external source if it fails after a certain number of retries.) > > Derek > > On Mon, Oct 16, 2017 at 9:40 AM, Lukasz Cwik <[email protected]> wrote: > >> It depends on the runner but that exception that is thrown is per bundle >> processed and it is up to the runner to choose what to do with bundles that >> fail. >> >> For a bounded (batch) pipeline, Dataflow will fail the pipeline after a >> fixed number of retries of each bundle. >> For an unbounded (streaming) pipeline, Dataflow will retry the bundle >> forever until the pipeline is cancelled by the user. >> >> On Sun, Oct 15, 2017 at 11:33 PM, Derek Hao Hu <[email protected]> >> wrote: >> >>> Hi, >>> >>> I'm using DatastoreV1 API to write data into Datastore. I've briefly >>> gone through the implementation and it seems the Write transform will throw >>> a DatastoreException (https://github.com/apache/bea >>> m/blob/1bd17d1b95a6b27331626fa9bdbaa723969b710d/sdks/java/io >>> /google-cloud-platform/src/main/java/org/apache/beam/sdk/io/ >>> gcp/datastore/DatastoreV1.java#L1346) when it fails to commit. >>> >>> In a streaming pipeline, I think it might be possible that some commits >>> might occasionally fail even after five retries, what is the expected >>> behavior here? Is there a way to catch these failed mutations and then save >>> them somewhere? I'm not sure what is the recommended approach since `Write` >>> itself is already a transform which means there seems to be no easy way to >>> catch which mutations / commits actually failed. >>> >>> Could someone help explain what the best approach here is? Right now I'm >>> thinking of writing my own DoFn which just writes each entity to Datastore >>> without even batching - by doing this it seems I'll be easy to catch or >>> write the failed commits to logs. It doesn't seem to be the right approach >>> though - considering there is already a significant amount of effort for >>> Beam to provide it's own DatastoreIO. >>> >>> Thanks, >>> -- >>> Derek Hao Hu >>> >>> Software Engineer | Snapchat >>> Snap Inc. >>> >> >> > > > -- > Derek Hao Hu > > Software Engineer | Snapchat > Snap Inc. >
