Re: Flink 1.6 Support

2018-10-30 Thread Jins George
Thank you Thomas. Idea of providing different build targets for runners is great, as it enables users to pick from a list of runner versions. Thanks Jins George On 10/30/18 12:36 PM, Thomas Weise wrote: There has not been any decision to move to 1.6.x for the next release yet. There has

Re: Follow up ideas, to simplify creating MonitoringInfos.

2018-10-30 Thread Alex Amato
I am not sure on the correct syntax to populate the instances of my MonitoringInfoSpec messages message MonitoringInfoSpec { string urn = 1; string type_urn = 2; repeated string required_labels = 3; * map annotations = 4;* } Notice how the annotations field is not used anywhere. I was

Re: error with DirectRunner

2018-10-30 Thread Udi Meiri
+Robert Bradshaw I would be happy to debug and fix this, but I'd need more guidance on where to look. On Tue, Oct 30, 2018 at 4:07 PM Udi Meiri wrote: > Created https://issues.apache.org/jira/browse/BEAM-5927 > > On Tue, Oct 30, 2018 at 1:13 PM Lukasz Cwik wrote: > >> Udi, do you know if we

Re: error with DirectRunner

2018-10-30 Thread Udi Meiri
Created https://issues.apache.org/jira/browse/BEAM-5927 On Tue, Oct 30, 2018 at 1:13 PM Lukasz Cwik wrote: > Udi, do you know if we have a bug tracking this issue? > > If not, can you file one referencing this e-mail thread? > > On Tue, Oct 30, 2018 at 6:33 AM Allie Chen wrote: > >> Thanks

Re: [PROPOSAL] ParquetIO support for Python SDK

2018-10-30 Thread Heejong Lee
Thanks all for the valuable feedback on the document. Here's the summary of planned features for ParquetIO Python SDK: - Can read from Parquet file on any storage system supported by Beam - Can write to Parquet file on any storage system supported by Beam - Can configure the

Re: [FYI] Jenkins is restarting, please do not merge PRs without validation

2018-10-30 Thread Rui Wang
Thanks for this heads up! -Rui On Tue, Oct 30, 2018 at 1:50 PM Scott Wegner wrote: > > https://lists.apache.org/thread.html/d8704933befb71b5753dae8eaab7372f00a72307d66af86c120d79b8@%3Cbuilds.apache.org%3E > > INFRA is currently restarting Jenkins. While it is down, precommit checks > will not

Re: [PROPOSAL] Additional design for the Beam Python State and Timers API

2018-10-30 Thread Kenneth Knowles
Yea, I would expect A. B would be ill-defined for processing time timers, and trouble for event time timers once we decouple firing time and effective timestamp. C could easily be very confusing; generally automatic window assignment outside the Window transform is weird. The timestamp has to be <

Re: Follow up ideas, to simplify creating MonitoringInfos.

2018-10-30 Thread Lukasz Cwik
I'm not sure what you mean by "Using a map in an option." For your second issue, the google docs around this show[1]: Note that if you want to use a custom option in a package other than the one in which it was defined, you must prefix the option name with the package name, just as you would for

Re: [PROPOSAL] Additional design for the Beam Python State and Timers API

2018-10-30 Thread Lukasz Cwik
My concerns are around item 4 (left the same comments in the doc). What window should timers be using when looking up a side input? A) The window corresponding to the element that set the original timer. B) The window that would have been assigned based upon when the timer is scheduled to fire.

Re: error with DirectRunner

2018-10-30 Thread Lukasz Cwik
Udi, do you know if we have a bug tracking this issue? If not, can you file one referencing this e-mail thread? On Tue, Oct 30, 2018 at 6:33 AM Allie Chen wrote: > Thanks Udi. I agree, since it works fine removing either the side input or > the last flatten and combine operation. > > On Mon,

Re: Flink 1.6 Support

2018-10-30 Thread Thomas Weise
There has not been any decision to move to 1.6.x for the next release yet. There has been related general discussion about upgrading runners recently [1] Overall we need to consider the support for newer Flink versions that users find (the Flink version in distributions and what users typically

Re: Flink 1.6 Support

2018-10-30 Thread Lukasz Cwik
+dev On Tue, Oct 30, 2018 at 10:30 AM Jins George wrote: > Hi Community, > > Noticed that the Beam 2.8 release comes with flink 1.5.x dependency. > Are there any plans to upgrade flink to 1.6.x in next beam release. ( > I am looking for the better k8s support in Flink 1.6) > > Thanks, > >

Re: error with DirectRunner

2018-10-30 Thread Allie Chen
Thanks Udi. I agree, since it works fine removing either the side input or the last flatten and combine operation. On Mon, Oct 29, 2018 at 9:02 PM Udi Meiri wrote: > This looks like a FnApiRunner bug. > When I override use_fnapi_runner = False in direct_runner.py the pipeline > works. > > It