Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

Do you set ENABLE_AUTO_COMMIT_CONFIG? On Sun, Jan 7, 2018 at 10:57 PM, Sushil Ks wrote: > HI Aljoscha, >The issue is let's say I consumed 100 elements in 5 > mins Fixed Window with *GroupByKey* and later I applied *ParDO* for all > those elements. If

Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

HI Aljoscha, The issue is let's say I consumed 100 elements in 5 mins Fixed Window with *GroupByKey* and later I applied *ParDO* for all those elements. If there is an issue while processing element 70 in *ParDo *and the pipeline restarts with *UserCodeException *it's skipping

Re: Switching to Java 8

Yes, that's the plan: build first, example "merge" after. Regards JB On 01/08/2018 07:43 AM, Eugene Kirpichov wrote: Sounds great, thanks! Probably best done as 2 separate steps, because after updating the build scripts, everything else can begin in parallel? On Sun, Jan 7, 2018 at 10:38 PM

Re: Switching to Java 8

Sounds great, thanks! Probably best done as 2 separate steps, because after updating the build scripts, everything else can begin in parallel? On Sun, Jan 7, 2018 at 10:38 PM Jean-Baptiste Onofré wrote: > Hi Eugene, > > I'm taking the build update: Maven/Gradle with enforcer

Re: Switching to Java 8

Hi Eugene, I'm taking the build update: Maven/Gradle with enforcer + merge of the examples all together. Regards JB On 01/08/2018 07:34 AM, Eugene Kirpichov wrote: The vote on user@ about switching to Java 8 has concluded, affirmatively. What needs to be done to complete the switch? I can

Switching to Java 8

The vote on user@ about switching to Java 8 has concluded, affirmatively. What needs to be done to complete the switch? I can see at least the following: - Change maven and gradle scripts to use 1.8 source and target version - Fix resulting compilation/test errors (Java8 has slightly different

Re: jackson to parse options?

Le 7 janv. 2018 12:53, "Jean-Baptiste Onofré" a écrit : Hi Romain, I guess you are assuming that pipeline options are flat command line like argument, right ? Actually, theoretically, pipeline options can be represented as json, that's why we use jackson. The pipeline

Re: jackson to parse options?

Hi Romain, I guess you are assuming that pipeline options are flat command line like argument, right ? Actually, theoretically, pipeline options can be represented as json, that's why we use jackson. The pipeline options can be serialized/deserialized as json. The purpose is to remove the

jackson to parse options?

Hi guys, not sure i fully get why jackson is used to parse pipeline options in the testing integration why not using a token parser like [1] which allows to map 1-1 with the user interface (command line) the options? [1]

Jenkins build is still unstable: beam_Release_NightlySnapshot #644

See