Re: [PROPOSAL] Custom JVM initialization for Beam workers

2019-04-10 Thread Kenneth Knowles
On Wed, Apr 10, 2019 at 8:18 PM Ahmet Altay wrote: > > > On Wed, Apr 10, 2019 at 7:59 PM Kenneth Knowles wrote: > >> TL;DR I like the simple approach better than the ServiceLoader solution >> when a particular DoFn depends on the result. The ServiceLoader solution >> fits when it is somewhat

Re: [PROPOSAL] Custom JVM initialization for Beam workers

2019-04-10 Thread Ahmet Altay
On Wed, Apr 10, 2019 at 7:59 PM Kenneth Knowles wrote: > TL;DR I like the simple approach better than the ServiceLoader solution > when a particular DoFn depends on the result. The ServiceLoader solution > fits when it is somewhat independent of a particular DoFn (I'm not sure the > use

Re: [PROPOSAL] Custom JVM initialization for Beam workers

2019-04-10 Thread Kenneth Knowles
TL;DR I like the simple approach better than the ServiceLoader solution when a particular DoFn depends on the result. The ServiceLoader solution fits when it is somewhat independent of a particular DoFn (I'm not sure the use case(s)). On Wed, Apr 10, 2019 at 4:10 PM Brian Hulette wrote: > -

Re: Java DirectRunner BagState Problem

2019-04-10 Thread Steve Niemitz
This sounds a lot like what I had reported in https://issues.apache.org/jira/plugins/servlet/mobile#issue/BEAM-6813 (sorry for the mobile link). On Wed, Apr 10, 2019 at 6:59 PM Anton Kedin wrote: > Hi dev@, > > I am debugging a flaky test and observing an interesting problem with a > BagState.

[PROPOSAL] Custom JVM initialization for Beam workers

2019-04-10 Thread Brian Hulette
Hi everyone, In Dataflow we've had some Java SDK user issues that could be easily solved if users were able to define some one-time initialization code (e.g. set a system property) that their workers execute before they begin to process data. One solution we've found that works with currently

Re: [Forked] BEAM-4046 (was [PROPOSAL] Introduce beam-sdks-java gradle project)

2019-04-10 Thread Michael Luckey
To my understanding, that's it, yes. Of course, there might be other places/plugins which use plugin.group. But maven coordinates are definitely those which need to be consistent. On Thu, Apr 11, 2019 at 12:57 AM Lukasz Cwik wrote: > We may be saying the same thing but wanted to be clear that

Re: [Forked] BEAM-4046 (was [PROPOSAL] Introduce beam-sdks-java gradle project)

2019-04-10 Thread Lukasz Cwik
We may be saying the same thing but wanted to be clear that we only need to override the default that publishing plugin uses to always be "org.apache.beam" instead of defaulting to project.group On Wed, Apr 10, 2019 at 3:22 PM Kenneth Knowles wrote: > So, if we set the "group" on projects only

Re: [DISCUSS] change the encoding scheme of Python StrUtf8Coder

2019-04-10 Thread Kenneth Knowles
On Mon, Apr 8, 2019 at 4:03 PM Robert Bradshaw wrote: > This email is already very long, but in summary I think the right > answer is to just get rid of Outer altogether (except possibly for > IOs, which we'd only preserve for legacy reasons until 3.0). > > - Robert > I had forgotten that

Re: [Forked] BEAM-4046 (was [PROPOSAL] Introduce beam-sdks-java gradle project)

2019-04-10 Thread Kenneth Knowles
So, if we set the "group" on projects only as part of publishing then everything starts to work? That sounds ideal. Kenn On Tue, Apr 9, 2019 at 3:49 PM Lukasz Cwik wrote: > It would be good if we did as much as possible to make our project as much > as a conventional Gradle project. It means

Re: Projects Can Apply Individually for Google Season of Docs

2019-04-10 Thread Pablo Estrada
I'd be happy to be a mentor for this to help add getting started documentation for Python on Flink. I'd want to focus on the reviews and less on the administration - so I'm willing to be a secondary administrator if that's necessary to move forward, but I'd love it if someone would help

Re: [ANNOUNCE] New committer announcement: Boyuan Zhang

2019-04-10 Thread Chamikara Jayalath
Congrats Boyuan! On Wed, Apr 10, 2019 at 11:14 AM Yifan Zou wrote: > Congratulations Boyuan! > > On Wed, Apr 10, 2019 at 10:49 AM Daniel Oliveira > wrote: > >> Congrats Boyuan! >> >> On Wed, Apr 10, 2019 at 10:20 AM Rui Wang wrote: >> >>> So well deserved! >>> >>> -Rui >>> >>> On Wed, Apr 10,

Re: ParDo Execution Time stat is always 0

2019-04-10 Thread Pablo Estrada
This sounds like a bug then? +Alex Amato On Wed, Apr 10, 2019 at 3:59 AM Maximilian Michels wrote: > Hi @all, > > From a quick debugging session, I conclude that the wiring is in place > for the Flink Runner. There is a ProgressReporter that reports > MonitoringInfos to Flink, in a similar

Re: [ANNOUNCE] New committer announcement: Boyuan Zhang

2019-04-10 Thread Daniel Oliveira
Congrats Boyuan! On Wed, Apr 10, 2019 at 10:20 AM Rui Wang wrote: > So well deserved! > > -Rui > > On Wed, Apr 10, 2019 at 10:12 AM Pablo Estrada wrote: > >> Well deserved : ) congrats Boyuan! >> >> On Wed, Apr 10, 2019 at 10:08 AM Aizhamal Nurmamat kyzy < >> aizha...@google.com> wrote: >> >>>

Re: [ANNOUNCE] New committer announcement: Boyuan Zhang

2019-04-10 Thread Rui Wang
So well deserved! -Rui On Wed, Apr 10, 2019 at 10:12 AM Pablo Estrada wrote: > Well deserved : ) congrats Boyuan! > > On Wed, Apr 10, 2019 at 10:08 AM Aizhamal Nurmamat kyzy < > aizha...@google.com> wrote: > >> Congratulations Boyuan! >> >> On Wed, Apr 10, 2019 at 9:52 AM Ruoyun Huang wrote:

Re: [ANNOUNCE] New committer announcement: Boyuan Zhang

2019-04-10 Thread Pablo Estrada
Well deserved : ) congrats Boyuan! On Wed, Apr 10, 2019 at 10:08 AM Aizhamal Nurmamat kyzy wrote: > Congratulations Boyuan! > > On Wed, Apr 10, 2019 at 9:52 AM Ruoyun Huang wrote: > >> Thanks for your contributions and congratulations Boyuan! >> >> On Wed, Apr 10, 2019 at 9:00 AM Kenneth

Re: [ANNOUNCE] New committer announcement: Boyuan Zhang

2019-04-10 Thread Aizhamal Nurmamat kyzy
Congratulations Boyuan! On Wed, Apr 10, 2019 at 9:52 AM Ruoyun Huang wrote: > Thanks for your contributions and congratulations Boyuan! > > On Wed, Apr 10, 2019 at 9:00 AM Kenneth Knowles wrote: > >> Hi all, >> >> Please join me and the rest of the Beam PMC in welcoming a new committer: >>

Re: [ANNOUNCE] New committer announcement: Boyuan Zhang

2019-04-10 Thread Ruoyun Huang
Thanks for your contributions and congratulations Boyuan! On Wed, Apr 10, 2019 at 9:00 AM Kenneth Knowles wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > Boyuan Zhang. > > Boyuan has been contributing to Beam since early 2018. She has proposed >

Re: [ANNOUNCE] New committer announcement: Boyuan Zhang

2019-04-10 Thread Robin Qiu
Congratulations, Boyuan! On Wed, Apr 10, 2019 at 9:28 AM Connell O'Callaghan wrote: > Well done and congratulations Boyuan!!! > > On Wed, Apr 10, 2019, 5:04 PM Lukasz Cwik wrote: > >> Congratulations and well deserved. >> >> On Wed, Apr 10, 2019 at 9:00 AM Kenneth Knowles wrote: >> >>> Hi

Re: [ANNOUNCE] New committer announcement: Boyuan Zhang

2019-04-10 Thread Connell O'Callaghan
Well done and congratulations Boyuan!!! On Wed, Apr 10, 2019, 5:04 PM Lukasz Cwik wrote: > Congratulations and well deserved. > > On Wed, Apr 10, 2019 at 9:00 AM Kenneth Knowles wrote: > >> Hi all, >> >> Please join me and the rest of the Beam PMC in welcoming a new committer: >> Boyuan Zhang.

Re: [ANNOUNCE] New committer announcement: Boyuan Zhang

2019-04-10 Thread Lukasz Cwik
Congratulations and well deserved. On Wed, Apr 10, 2019 at 9:00 AM Kenneth Knowles wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > Boyuan Zhang. > > Boyuan has been contributing to Beam since early 2018. She has proposed > 100+ pull requests

[ANNOUNCE] New committer announcement: Boyuan Zhang

2019-04-10 Thread Kenneth Knowles
Hi all, Please join me and the rest of the Beam PMC in welcoming a new committer: Boyuan Zhang. Boyuan has been contributing to Beam since early 2018. She has proposed 100+ pull requests across a wide range of topics: bug fixes, to integration tests, build improvements, metrics features, release

Re: Spotless check on build.gradle files

2019-04-10 Thread Łukasz Gajowy
Yes - the way it works now is that the groovy spotless configuration is not used until the "applyGroovyNature()" is explicitly invoked in a build.gradle file. IMO the spotless config should be "on" for all groovy files in the project by default (build.gradle's included), even if

Re: JDK11 support?

2019-04-10 Thread Michał Walenia
Yes, if you look at the DataflowRunner code [1], if you set a system property 'java.specification.version' to 11, the runner will automatically pull java11 enabled worker harness. [1]

Re: JDK11 support?

2019-04-10 Thread Gleb Kanterov
Is there a way to try JDK11 harness for Dataflow without building own docker image? On Wed, Apr 10, 2019 at 2:10 AM Yi Pan wrote: > Hi, Pablo, > > Thanks for the clarification. Does that mean that there needs to be a > separate effort to ensure KafkaIO to be Java 8 source compat and Java 11 >

Re: Updates on Beam Jenkins

2019-04-10 Thread Connell O'Callaghan
Yifan thank you for this update and progressing this replacement!!! On Tue, Apr 9, 2019, 11:14 PM Yifan Zou wrote: > Thanks, Pablo. The new workers use custom image in the boot disk and it's > easy to reboot without re-imaging. Means we will no longer need assistance > from Infra to reconnect