Re: Bug of the MqttIO.java

Hi, I've created PR for this bug, https://github.com/apache/beam/pull/6701, please check it. The PR page said the "java precommit" failed, but I'm not sure where is the failure and all tests of MqttIO are passed. (

Python Datastore client upgrade plan

Hi, Sadly upgrading googledatastore -> google-cloud-datastore is non-trivial ( https://issues.apache.org/jira/browse/BEAM-4543). I wrote a doc to summarize the plan: https://docs.google.com/document/d/1sL9p7NE5Z0p-5SB5uwpxWrddj_UCESKSrsvDTWNKqb4/edit?usp=sharing Contents pasted below: Beam Python

Re: [PROPOSAL] allow the users to anticipate the support of features in the targeted runner.

Sounds like a good idea. Sounds like while coding, user gets a list to show if a feature is supported on different runners. User can check the list for the answer. Is my understanding correct? Will this approach become slow as number of runner grows? (it's just a question as I am not familiar the

Re: Java pre-commit failures

Seems to point to slow/stuck dataflow tests: org.apache.beam.examples.WindowedWordCountIT > testWindowedWordCountInBatchStaticSharding STANDARD_ERROR Oct 16, 2018 5:11:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process SEVERE: 2018-10-16T17:11:41.351Z:

Java pre-commit failures

https://builds.apache.org/job/beam_PreCommit_Java_Commit/ The jobs run very slow and are eventually cancelled. Thanks, Thomas

Re: Beam Summit community feedback

Hey Max, Great stuff, thank you for sharing this. In case anyone has feedback on the summit as a whole, please feel free to fill out the survey as well. Thank you! Best regards, Matthias On Tue, 9 Oct 2018 at 10:48 Maximilian Michels wrote: > Thanks

Re: [BEAM-5442] Store duplicate unknown (runner) options in a list argument

On Tue, Oct 16, 2018 at 7:03 PM Lukasz Cwik wrote: > > For all unknown options, the SDK can require that all flag values be specified explicitly as a valid JSON type. > starts with { -> object > starts with [ -> list > starts with " -> string > is null / true / false -> null / true / false >

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

Thanks for bringing this to a conclusion. On Mon, Oct 15, 2018 at 6:18 PM Thomas Weise wrote: > > Here is my attempt to summarize the discussion, please see the TBDs. > > I would work on a PR with respective contributor and committer guideline > updates next. > > Thanks, > Thomas > > > Goals: >

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

+1 in general. However, I'd suggest to use "rebase and merge" more often. Otherwise reading history is really inconvenient. Like in attached screenshot. You can never easily understand what is included in any of previous commits of remotes/upstream/master, unless you do explicit git log on those

Re: Does anyone have a strong intelliJ setup?

Thanks for this info and work! A couple relevant notes: - There is a #beam-intellij slack channel where I tried to collect some info a few weeks ago when I was debugging IntelliJ issues - I tried to figure out where IntelliJ stores the info about the vendored JARs we manually add to

Re: python tests run as part of 'javaPreCommit'

Can we do a separate targets for integration, component and unittests? Ideally with different locations of test code. Bundling things together makes it hard to navigate. It also makes it cleaner when we separate relevant code. --Mikhail Have feedback ? On Tue, Oct

Re: python tests run as part of 'javaPreCommit'

Good idea on having a flag to make the test run without GCP credentials. I logged this as https://issues.apache.org/jira/browse/BEAM-5771 and can look into this, unless someone else prefers. On Tue, Oct 16, 2018 at 9:18 AM Scott Wegner wrote: > There was some discussion recently about ensuring

Re: [BEAM-5442] Store duplicate unknown (runner) options in a list argument

For all unknown options, the SDK can require that all flag values be specified explicitly as a valid JSON type. starts with { -> object starts with [ -> list starts with " -> string is null / true / false -> null / true / false otherwise is number. This isn't great for strings but works well for

Re: Does anyone have a strong intelliJ setup?

FYI, I've opened BEAM-5762 to track the work to document and improve IntelliJ integration. It's broken down into sub-tasks for documenting individual scenarios. I've grabbed a couple; if you're feeling motivated feel free to grab one or two to help out!

Re: [ANNOUNCE] New committers & PMC members, Summer 2018 edition

Congrats all! And thanks Kenn and the PMC for recognizing these contributions. On Mon, Oct 15, 2018 at 9:45 AM Kenneth Knowles wrote: > Hi all, > > Since our last announcement in May, we have added many more committers and > a new PMC member. Some of these may have been in the monthly

Re: python tests run as part of 'javaPreCommit'

There was some discussion recently about ensuring anyone can easily run and reproduce precommit test results locally. The precommits run Dataflow jobs, which will fail if you don't have access to an Google Cloud project. One idea would be to add a flag to disable Google Cloud tests, i.e. ./gradlew

Re: [Proposal] Euphoria DSL - looking for reviewers

Congrats to the Euphoria team! On Tue, Oct 16, 2018 at 8:51 AM Kenneth Knowles wrote: > Merged. Welcome to the repo :-) > > Kenn > > On Thu, Oct 11, 2018 at 10:06 AM Kenneth Knowles wrote: > >> I've filed the IP Clearance. I'll report back here. >> >> Kenn >> >> On Wed, Oct 10, 2018 at 3:33 PM

Re: [Call for items] October Beam Newsletter

Since the newsletter is still pending, we can probably also mention the just merged Euphoria Java 8 DSL? https://issues.apache.org/jira/browse/BEAM-3900 On Tue, Oct 16, 2018 at 6:21 AM Maximilian Michels wrote: > Hi Rose, > > A bit late but since the newsletter does not seem to be out yet, I

Re: python tests run as part of 'javaPreCommit'

Yes, it is exactly that. The :javaPreCommit is a deliberate attempt to make a single task that runs all the tests that Jenkins runs, so it includes some lightweight smoke tests on runners, including Google Cloud Dataflow. With maven it was impossible to have a single mvn invocation that would

Re: [Proposal] Euphoria DSL - looking for reviewers

Merged. Welcome to the repo :-) Kenn On Thu, Oct 11, 2018 at 10:06 AM Kenneth Knowles wrote: > I've filed the IP Clearance. I'll report back here. > > Kenn > > On Wed, Oct 10, 2018 at 3:33 PM David Morávek > wrote: > >> >> >> Anton: >> All of the points are be correct, with one minor

Re: python tests run as part of 'javaPreCommit'

Thanks Kenn, rookie mistake on my part :-) A further question if I may - "./gradlew :javaPreCommit" is failing for me with: org.apache.beam.examples.WindowedWordCountIT > testWindowedWordCountInBatchDynamicSharding FAILED org.apache.beam.sdk.Pipeline$PipelineExecutionException at

Re: [DISCUSS] Separate Jenkins notifications to a new mailing list

Hi all, I moved pretty quickly and without adequate announcement of the establishment issues@ alias. Sorry if that caught anyone by surprise. This time I want to properly announce, but the support is pretty overwhelming so I'll just get the list set up as there's some latency on that. I'll let 72

Re: Rethinking Timers as PCollections

On Tue, Oct 16, 2018 at 1:26 AM Robert Bradshaw wrote: > I've stated before I think it's simpler for both users and runners to > model that loop as an internal property of ParDo. > Strong agree. OTOH I think it is helpful to be explicit about "the current watermark" of the timer channel, even

Re: python tests run as part of 'javaPreCommit'

One thing to clarify is that `:javaPreCommit` is a task and `build` is another task. There's so verb-object relationship in your commandline. So as written, you've asked for a whole-project `build`, which weirdly in Gradle means "build and test". Since it is one commandline, all the necessary

Re: [BEAM-5442] Store duplicate unknown (runner) options in a list argument

Discovering options from the job server seems preferable over replicating runner options in SDKs. Runners evolve on their own, and with portability the SDK does not need to know anything about the runner. Regarding --runner-option. It is true that this looks less user friendly. On the other hand

Re: [Call for items] October Beam Newsletter

Hi Rose, A bit late but since the newsletter does not seem to be out yet, I added some items for the Portable Flink Runner. Cheers, Max On 08.10.18 18:59, Rose Nguyen wrote: Hi Beamers: So much has been going on that it's time to sync up again in the October Beam Newsletter [1]! :) *Add

python tests run as part of 'javaPreCommit'

Hi all, Just a quick question - I was wondering why the python tests/build run as part of the 'javaPreCommit' task? i.e. executing "./gradlew build :javaPreCommit" leads to python tests being run as well, which is not something you might expect from the name of the task. Colm. -- Colm O

Re: [DISCUSS] Separate Jenkins notifications to a new mailing list

+1 Regards JB Le 16 oct. 2018 à 12:46, à 12:46, Alexey Romanenko a écrit: >+1 > >> On 16 Oct 2018, at 00:02, Chamikara Jayalath >wrote: >> >> +1 for new lists. >> >> Thanks, >> Cham >> >> On Mon, Oct 15, 2018 at 12:09 PM Ismaël Mejía > wrote: >> +1 >> On Mon, Oct 15,

Re: [DISCUSS] Separate Jenkins notifications to a new mailing list

+1 I can switch of all my filters then, and people new here will be less overwhelmed by email. On 16.10.18 12:46, Alexey Romanenko wrote: +1 On 16 Oct 2018, at 00:02, Chamikara Jayalath > wrote: +1 for new lists. Thanks, Cham On Mon, Oct 15, 2018 at 12:09 PM

Re: [BEAM-5442] Store duplicate unknown (runner) options in a list argument

Fetching options directly from the Runner's JobServer seems like the ideal solution. I agree with Robert that it creates additional complexity for SDK authors, so the `--runner-option` flag would be an easy and explicit way to specify additional Runner options. The format I prefer would be:

Re: [DISCUSS] Separate Jenkins notifications to a new mailing list

+1 > On 16 Oct 2018, at 00:02, Chamikara Jayalath wrote: > > +1 for new lists. > > Thanks, > Cham > > On Mon, Oct 15, 2018 at 12:09 PM Ismaël Mejía > wrote: > +1 > On Mon, Oct 15, 2018 at 8:14 PM Mikhail Gryzykhin > wrote: > > > > +1 also

Re: [BEAM-5442] Store duplicate unknown (runner) options in a list argument

Yes, we don't know how to parse and/or validate it. On Tue, Oct 16, 2018 at 1:14 AM Lukasz Cwik wrote: > I see, is the issue that we currently are using a JSON representation for > options when being serialized and when we get some unknown option, we don't > know how to convert it into its JSON

Re: Rethinking Timers as PCollections

On Mon, Oct 15, 2018 at 11:59 PM Lukasz Cwik wrote: > > As Kenn mentioned, the timer is tracking the watermark of the main input PCollection and not the input watermark of the ParDo which would allow it to fire and thus it couldn't block itself. Yes, and having to treat the watermarks of the

Re: [Proposal] Euphoria DSL - looking for reviewers

Ideally one (or all) of you can become committers [1], which I think should be the goal. While for the time being this would involve the overhead of getting existing committers to sign off on PRs (which can be reviewed by others as well), this can actually be beneficial as it will be a forcing

Re: Why not adding all coders into ModelCoderRegistrar?

Any coders added to the ModelCoderRegistrar requires support from *all* SDKs, which is why that set is chosen sparingly. Could you clarify exactly what you're trying to achieve. It sounds like there's some case where you know the SDK will submit a KV with a Void and/or VarIntCoder in the key, and