Re: Kafka Offset handling for Restart/failure scenarios.

2017-03-24 Thread Jins George
Currently /org.apache.beam.runners.flink.FlinkPipelineOptions/ does not have a way to configure externalized checkpoints. Is that something in the road map for FlinkRunner? Thanks, Jins George On 03/23/2017 10:27 AM, Aljoscha Krettek wrote: For this you would use externalised checkpoints

Re: Kafka Offset handling for Restart/failure scenarios.

2017-03-21 Thread Jins George
no state was passed to UnboundedSource.createReader() on starting the job again. I am currently using flink 1.1.3 and beam 0.3. I will upgrade to beam 0.6.0 and flink 1.2 and perform the tests again. Thanks, Jins George On 03/21/2017 09:39 AM, Dan Halperin wrote: hey, The native Beam

Re: AvroCoder + KafkaIO + Flink problem

2017-04-28 Thread Jins George
a new yarn session and submit. Thanks, Jins George On 04/28/2017 09:34 AM, Frances Perry wrote: I have the same problem and am working around it with SerializableCoder. +1 to a real solution. On Fri, Apr 28, 2017 at 8:46 AM, Aljoscha Krettek <aljos...@apache.org <mailto:aljos...@apac

Re: Controlling parallelism of a ParDo Transform while writing to DB

2018-05-17 Thread Jins George
I am a user running beam+flink. Flink runner currently exposes only the job level parallelism, not at an operator level. This is a really nice feature if can be supported. Flink's Datastream api provide that option though. Thanks, Jins George On 05/16/2018 10:24 PM, Chamikara Jayalath

Re: Flink 1.6 Support

2018-10-31 Thread Jins George
to release both versions: > > https://issues.apache.org/jira/browse/BEAM-5419 > > Thanks, > Max > > On 31.10.18 06:28, Jins George wrote: >> Thank you Thomas.  Idea of providing different build targets  for >> runners is great, as it enables users to pick from a li

Flink 1.6 Support

2018-10-30 Thread Jins George
Hi Community, Noticed that the Beam 2.8 release comes with flink  1.5.x dependency. Are there any plans to upgrade flink to  1.6.x  in next beam release. ( I am looking for the better k8s  support in Flink 1.6) Thanks, Jins George

Re: Flink 1.6 Support

2018-10-30 Thread Jins George
Thank you Thomas. Idea of providing different build targets for runners is great, as it enables users to pick from a list of runner versions. Thanks Jins George On 10/30/18 12:36 PM, Thomas Weise wrote: There has not been any decision to move to 1.6.x for the next release yet. There has

Accessing PipelineOptions in DoFn @Setup

2019-02-28 Thread Jins George
as it does not support dynamic topics. Thanks, Jins George