Re: jackson to parse options?

Lukasz, the use case is to znsure the config used can still map the CLI and that options dont start to abuse json so it is exactly the opposite of "we can be fancy" and is closer to "we can be robust". Also the default should be easy and not json (i just want to set the runner, --runner=xxx is

Re: jackson to parse options?

Now that I think about this more. I looked at some of the examples in the pom.xml and they don't seem to be tricky to write in JSON. I also looked at the Jenkins job configurations (specifically the performance tests) and they pass around maps which they convert to the required format without

Re: jackson to parse options?

Good point for \t and ,. Any objection to use jackson as a fallback for that purpose - for backwqrd compat only - and make it optional then? Will create the ticket if not. Le 8 janv. 2018 20:32, "Robert Bradshaw" a écrit : > Part of the motivation to use JSON for more

Re: Dataflow runner examples build fail

+1 On Mon, Jan 8, 2018 at 10:07 AM, Kenneth Knowles wrote: > +1 > > On Mon, Jan 8, 2018 at 9:33 AM, Henning Rohde wrote: > >> +1 >> >> On Mon, Jan 8, 2018 at 1:32 AM, Ted Yu wrote: >> >>> +1 >>> >>> Original message

Re: [DISCUSS] Towards Beam 2.3.0

+1! I like the predictability a schedule would bring. And I think it helps feature users to budget their time a little better -- there's always the next scheduled train, so no need to stress out to ship in the current one. On Mon, Jan 8, 2018 at 10:37 AM Kenneth Knowles

Re: jackson to parse options?

Part of the motivation to use JSON for more complex options was that it avoids having to define (and document, test, have users learn, ...) yet another format for expressing lists, maps, etc. On Mon, Jan 8, 2018 at 11:19 AM, Lukasz Cwik wrote: > Ken, this is specifically about

Re: jackson to parse options?

On Mon, Jan 8, 2018 at 11:19 AM, Lukasz Cwik wrote: > Ken, this is specifically about running integration tests and not a users > main(). > Yea, I was just addressing the bit about advanced users having a headache. Hopefully no end users are really writing options as JSON

Re: jackson to parse options?

Ken, this is specifically about running integration tests and not a users main(). Note, that PipelineOptions JSON format was used because it was a convenient serialized form that is easy to explain to people what is required. Using a different string tokenizer and calling

Re: [DISCUSS] Towards Beam 2.3.0

Fully agree of course ! ;) Regards JB On 01/08/2018 07:32 PM, Reuven Lax wrote: +1 - this is definitely one of the (multiple) things that delayed 2.2.0. In my opinion releases should be held up for critical bug fixes, but not for features. Any feature work can always go into 2.4.0, and with

Re: [DISCUSS] Towards Beam 2.3.0

+1 to not holding except for critical bugs and regressions. Using 2.3.0 to improve automation is a great idea. Features can make the next release, and backwards incompatible refinements should have quiesced long before a feature comes out of @Experimental status. On Mon, Jan 8, 2018 at 10:32 AM,

Re: jackson to parse options?

We do have a plain command line syntax, and whoever writes the main(String[]) function is responsible for invoking the parser. It isn't quite as nice as standard arg parse libraries, but it isn't too bad. It would be great to improve, though. Jackson is for machine-to-machine communication or

Re: [DISCUSS] Towards Beam 2.3.0

+1 - this is definitely one of the (multiple) things that delayed 2.2.0. In my opinion releases should be held up for critical bug fixes, but not for features. Any feature work can always go into 2.4.0, and with any luck we can get 2.3.0 out much faster than 2.2.0. Reuven On Mon, Jan 8, 2018 at

Re: [DISCUSS] Towards Beam 2.3.0

+1 on having a release started in January and not holding on feature requests. I'd like to learn more about the release process in order to automate parts of it. I'm not volunteering to do the release, but would like to pair with whomever is doing the release to discuss pain points and help in

Re: Dataflow runner examples build fail

+1 On Mon, Jan 8, 2018 at 9:33 AM, Henning Rohde wrote: > +1 > > On Mon, Jan 8, 2018 at 1:32 AM, Ted Yu wrote: > >> +1 >> >> Original message >> From: Jean-Baptiste Onofré >> Date: 1/8/18 1:26 AM (GMT-08:00) >> To:

Re: [DISCUSS] Towards Beam 2.3.0

+1 for starting the 2.3 ball rolling. In general, I'd like to avoid holding up releases for specific features/PRs. The is (one of the things) that holds up releases which then is a vicious cycle for more people wanting to make their feature a condition of the next release, etc. (Bugs,

Re: Dataflow runner examples build fail

+1 On Mon, Jan 8, 2018 at 1:32 AM, Ted Yu wrote: > +1 > > Original message > From: Jean-Baptiste Onofré > Date: 1/8/18 1:26 AM (GMT-08:00) > To: dev@beam.apache.org > Subject: Dataflow runner examples build fail > > Hi guys, > > The PRs

Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

Yea have tried ENABLE_AUTO_COMMIT_CONFIG , its helping me before GroupByKey but not after that. In other words once the window is materialized and if any exception occurs while processing the elements in it, its dropping the remaining elements of that window when restarted. On Jan 8, 2018 12:45

Re: [DISCUSS] Towards Beam 2.3.0

Hi Romain, no problem: let's try a best effort and define the target version in the Jira. Regards JB On 01/08/2018 03:51 PM, Romain Manni-Bucau wrote: Hi JB, I'd like https://github.com/apache/beam/pull/4235 to be integrated if possible Also the JUnit 5 PR brings some light changes which

Re: [DISCUSS] Towards Beam 2.3.0

Hi JB, I'd like https://github.com/apache/beam/pull/4235 to be integrated if possible Also the JUnit 5 PR brings some light changes which can be worth the "3" digit upgrade so if anyone has some time to review it can be a good candidate too. Thanks for driving it Romain Manni-Bucau

[DISCUSS] Towards Beam 2.3.0

Hi guys, In a previous discussion thread, we agreed that we should have a regular pace in term of releases. We released Beam 2.2.0 on the 16th of November '17, but the release takes a pretty long time. I think it's reasonable to think about Beam 2.3.0 in the coming weeks. I would like to

Re: Switching to Java 8

Hi Romain, Good point. However, AFAIR, AutoValue supports Java 8 (not Java 8 "style" but it compiles without problem using Java 8). Regards JB On 01/08/2018 11:36 AM, Romain Manni-Bucau wrote: +1000 also requires to upgrade @Auto* processor which was not supporting j8 in current (beam

Re: Switching to Java 8

I created https://issues.apache.org/jira/browse/BEAM-3426 as umbrella Jira and created the sub-tasks related to build and examples. Feel free to add the relevant sub-tasks there. Regards JB On 01/08/2018 11:33 AM, Ismaël Mejía wrote: Excellent news ! Probably a good idea to fill JIRAs to all

Re: Switching to Java 8

Good idea for the Jira. I will create the Jira for build and examples as I gonna work on those ones. Regards JB On 01/08/2018 11:33 AM, Ismaël Mejía wrote: Excellent news ! Probably a good idea to fill JIRAs to all of those. I would add: - Remove the references in the website to Java 7 -

Re: Switching to Java 8

Good point Romain, this is also another (same) point: - Update dependencies pinned into older version because of the lack of java 8 support (guava, auto, etc). On Mon, Jan 8, 2018 at 11:36 AM, Romain Manni-Bucau wrote: > +1000 > > also requires to upgrade @Auto* processor

Re: Switching to Java 8

+1000 also requires to upgrade @Auto* processor which was not supporting j8 in current (beam dependency) version. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github

Re: JUnit 5 TestPipeline

Decided to go with https://github.com/apache/beam/pull/4360 to start. It still relies on hamcrest - which should be dropped at some point for JUnit 5 - but it is not a blocker to use JUnit 5 and the asserts can be done with assertTrue(BooleanSupplier) or any other JUnit 5 utility. Romain

Re: Switching to Java 8

Excellent news ! Probably a good idea to fill JIRAs to all of those. I would add: - Remove the references in the website to Java 7 - Remove Java 7 and any related task from the CI - Update the docker dev build images (I will take this one since reproducible build is my pet project) - Upgrade the

Re: Dataflow runner examples build fail

+1 Original message From: Jean-Baptiste Onofré Date: 1/8/18 1:26 AM (GMT-08:00) To: dev@beam.apache.org Subject: Dataflow runner examples build fail Hi guys, The PRs and nightly builds are failing due to an issue with the dataflow platform: it seems we

Dataflow runner examples build fail

Hi guys, The PRs and nightly builds are failing due to an issue with the dataflow platform: it seems we have a disk quota exceeded on the us-central1 region. I would like to do a clean out and increase the quota a bit. Thoughts ? Thanks Regards JB -- Jean-Baptiste Onofré jbono...@apache.org

Jenkins build is still unstable: beam_Release_NightlySnapshot #645

See