Re: Unexpected TestStream behavior when testing stateful DoFn

Thanks, will do! -Amar On Tue, Apr 9, 2019 at 5:46 PM Pablo Estrada wrote: > I've added your snippet as a unit test, and fixed that test[1] : ) - > thanks for reporting the issue. Please let me know if you see any other > issues like this. > Best > -P. > > [1]

Re: Unexpected TestStream behavior when testing stateful DoFn

I've added your snippet as a unit test, and fixed that test[1] : ) - thanks for reporting the issue. Please let me know if you see any other issues like this. Best -P. [1] https://github.com/apache/beam/pull/8252 On Thu, Apr 4, 2019 at 2:46 PM Pablo Estrada wrote: > I saw similar issues. I'll

Re: JDK11 support?

Hi, Pablo, Thanks for the clarification. Does that mean that there needs to be a separate effort to ensure KafkaIO to be Java 8 source compat and Java 11 runtime compat? -Yi On Tue, Apr 9, 2019 at 5:03 PM Pablo Estrada wrote: > Our work (mostly done by Michal), consisted on testing that Beam

Re: JDK11 support?

Hi, all, Thanks a lot for the pointers. Very useful and glad to see it in motion! One thing I would like to consider is: except for some core modules that all Beam code uses, can we start marking individual SDKs as JDK11 compatible? For example, we are using DataflowRunner and Kafka, is it

Re: JDK11 support?

Our work (mostly done by Michal), consisted on testing that Beam artifacts built with Java 8 can run in a Java 11 environment in the DirectRunner and Dataflow. We have not planned other work in this area. On Tue, Apr 9, 2019 at 3:00 PM Ahmet Altay wrote: > Most recently +Pablo Estrada and

Re: JDK11 support?

To clarify: This included adding tests to run continuously on Jenkins (by my previous phrasing, it may have sounded like we only tested this manually :P). On Tue, Apr 9, 2019 at 4:57 PM Pablo Estrada wrote: > Our work (mostly done by Michal), consisted on testing that Beam artifacts > built

Re: Spotless check on build.gradle files

Thanks Lukasz. It seems that there's a spotless task added to applyGroovyNature: https://github.com/apache/beam/blob/0868e7544fd1e96db67ff5b9e70a67802c0f0c8e/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1346-L1355 Perhaps it's missing some more conditions? -P. On

Re: [QUESTION] Should DoFns be able to get the watermark?

Yes, those are the ones I was referring to. On Tue, Apr 9, 2019 at 4:08 PM Pablo Estrada wrote: > Yup! : ) - I think > > On Tue, Apr 9, 2019 at 3:52 PM Brian Hulette wrote: > >> Are these the blog posts? >> >> https://beam.apache.org/blog/2017/02/13/stateful-processing.html >>

Re: [QUESTION] Should DoFns be able to get the watermark?

Yup! : ) - I think On Tue, Apr 9, 2019 at 3:52 PM Brian Hulette wrote: > Are these the blog posts? > > https://beam.apache.org/blog/2017/02/13/stateful-processing.html > https://beam.apache.org/blog/2017/08/28/timely-processing.html > > On Tue, Apr 9, 2019 at 3:41 PM Pablo Estrada wrote: > >>

[VOTE] Release 2.12.0, release candidate #3

Hi everyone, Hopefully this is the last one. Please review and vote on the release candidate #3 for the version 2.12.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which

Re: [QUESTION] Should DoFns be able to get the watermark?

Are these the blog posts? https://beam.apache.org/blog/2017/02/13/stateful-processing.html https://beam.apache.org/blog/2017/08/28/timely-processing.html On Tue, Apr 9, 2019 at 3:41 PM Pablo Estrada wrote: > sunds good. Thanks guys <3 > > On Tue, Apr 9, 2019 at 3:19 PM Lukasz Cwik wrote:

Re: [Forked] BEAM-4046 (was [PROPOSAL] Introduce beam-sdks-java gradle project)

It would be good if we did as much as possible to make our project as much as a conventional Gradle project. It means that more people will be familiar with the setup, our setup will likely require less maintenance with version bumps in gradle and also that examples/solutions online will relate

Re: [QUESTION] Should DoFns be able to get the watermark?

sunds good. Thanks guys <3 On Tue, Apr 9, 2019 at 3:19 PM Lukasz Cwik wrote: > UnboundedSources and SplittableDoFns report watermarks which the runner > uses to compute how much the watermark could advance if it processed some > outstanding work. But it is always upto the runner to choose

Re: [QUESTION] Should DoFns be able to get the watermark?

UnboundedSources and SplittableDoFns report watermarks which the runner uses to compute how much the watermark could advance if it processed some outstanding work. But it is always upto the runner to choose when the watermark advances. The runner could process each work item in watermark priority

Re: Updates on Beam Jenkins

Thanks, Pablo. The new workers use custom image in the boot disk and it's easy to reboot without re-imaging. Means we will no longer need assistance from Infra to reconnect an offline node. Dockerizing the environment would be helpful to make changes on the environment, such as installing/updating

Re: Updates on Beam Jenkins

This is great stuff, my last couple of PRs before this had several > 2hr waits before the tests started up. On Tue, Apr 9, 2019 at 2:39 PM Pablo Estrada wrote: > Thanks for the updates Yifan. I am sure this process has been difficult, > and I appreciate the good communication, and that this

Re: Updates on Beam Jenkins

Yes, thanks Yifan! This is critical infrastructure that was in real trouble without your work. Kenn On Tue, Apr 9, 2019 at 2:39 PM Pablo Estrada wrote: > Thanks for the updates Yifan. I am sure this process has been difficult, > and I appreciate the good communication, and that this didn't

Re: JDK11 support?

Most recently +Pablo Estrada and +Michał Walenia were working on Java 11 related JIRAs. They may have more context. On Tue, Apr 9, 2019 at 1:22 PM Maximilian Michels wrote: > Hi Yi, > > That's a great question. Beam is still on Java 8 at the moment. There is > a JIRA issue for making Beam

Re: Updates on Beam Jenkins

Thanks for the updates Yifan. I am sure this process has been difficult, and I appreciate the good communication, and that this didn't really affect the workflow of anyone to validate the new setup for nodes. I imagine that once we move to dockerizing the testing environment, it will be much

Re: [QUESTION] Should DoFns be able to get the watermark?

In state & timers and new DoFn in the past It was an explicit decision to not allow direct observation of the watermark, but only to set a timer in event time. Is there a design doc I can read to catch up? Kenn On Tue, Apr 9, 2019 at 1:44 PM Lukasz Cwik wrote: > WatermarkReporterParam is about

Updates on Beam Jenkins

Hello, I have some good news about our Jenkins nodes. We're now having 7 new nodes online named as "apache-beam-jenkins-[1, 2, 4, 7, 8, 9, 12]", which substitute the old broken agents "beam[1, 2, 4, 7, 8, 9, 12]". This helps to reduce the job waiting queue

Re: [QUESTION] Should DoFns be able to get the watermark?

hi Luke, thanks for the prompt reply: ) That makes sense. I think I'll go back to my cave to read a bunch about streaming. : ) I was looking for this to try to write a sequence generator for Python in streaming, and I was trying to debug what was going on. I was trying to allow the DoFn to

Re: [QUESTION] Should DoFns be able to get the watermark?

WatermarkReporterParam is about reporting the watermark. The main usecase is for SplittableDoFns to be able to report the data watermark. The watermark is per input and output of a DoFn. Also each bundle being processed has its local watermarks while the runner computes the global watermark. The

[QUESTION] Should DoFns be able to get the watermark?

I am experimenting with state / timers in Python. As I look at the DoFnProcessParams[1], I see that it's possible for a DoFn to receive several arguments (e.g. Timers, Side Inputs, etc). Also the Watermark via WatermarkReporterParam. I see that this parameter is not handled by runners when

Re: JDK11 support?

Hi Yi, That's a great question. Beam is still on Java 8 at the moment. There is a JIRA issue for making Beam compatible with Java 11: https://issues.apache.org/jira/browse/BEAM-2530 As you can read in the issue, the upcoming Beam 2.12.0 has experimental support for Java 11. That said, there

JDK11 support?

Hi, everyone, I checked out the latest Beam code and seems that the document still says "JDK8". Is there any plan to officially support JDK11? Thanks a lot! -Yi

Re: PostCommit tests currently failing

+Lukasz Cwik +Mikhail Gryzykhin (Adding authors for #8203 and #8204 explicitly.) On Tue, Apr 9, 2019 at 5:05 AM Michael Luckey wrote: > Hi, > > looks as if > - beam_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow > - beam_PostCommit_Java_Nexmark_Dataflow > -

Re: PostCommit tests currently failing

I disabled the tests for now until Dataflow is ready to resume testing this again. The prior implementation didn't work as intended when using the portability APIs. On Tue, Apr 9, 2019 at 9:54 AM Andrew Pilloud wrote: > The nexmark issues are https://issues.apache.org/jira/browse/BEAM-7002 > >

Re: PostCommit tests currently failing

The nexmark issues are https://issues.apache.org/jira/browse/BEAM-7002 On Tue, Apr 9, 2019 at 9:50 AM Lukasz Cwik wrote: > The SplittableDoFn failures are because of the changes associated with > BEAM-6978. > > On Tue, Apr 9, 2019 at 5:05 AM Michael Luckey wrote: > >> Hi, >> >> looks as if >>

Re: PostCommit tests currently failing

The SplittableDoFn failures are because of the changes associated with BEAM-6978. On Tue, Apr 9, 2019 at 5:05 AM Michael Luckey wrote: > Hi, > > looks as if > - beam_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow > - beam_PostCommit_Java_Nexmark_Dataflow > -

Re: [VOTE] Release 2.12.0, release candidate #2

Thanks for finding this Ahmet. Please consider the vote for RC2 canceled. Andrew On Tue, Apr 9, 2019 at 9:25 AM Ahmet Altay wrote: > -1 unfortunately. > > Reason is https://issues.apache.org/jira/browse/BEAM-7038. 2.12 includes > changes to coders that are not compatible with previous versions

Re: [VOTE] Release 2.12.0, release candidate #2

-1 unfortunately. Reason is https://issues.apache.org/jira/browse/BEAM-7038. 2.12 includes changes to coders that are not compatible with previous versions breaking some Dataflow use cases. On Mon, Apr 8, 2019 at 3:54 PM Andrew Pilloud wrote: > Hi everyone, > > Please review and vote on the

PostCommit tests currently failing

Hi, looks as if - beam_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow - beam_PostCommit_Java_Nexmark_Dataflow - beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow are consistently failing since a few days [1]. Those validate runner tests seem to fail on SplittableDoFnTests

Spotless check on build.gradle files

Hi, it seems that spotless doesn't check build.gradle files if applyGroovyNature() is not used in them. We noticed this during code review - spotless succeeded despite improper indentation in a build.gradle file. IMO, such files should be checked by default, so I created an issue for this: