Re: Implementing @OnWindowExpiration in StatefulParDo [BEAM-1589]

As echauchot@ mentioned, it will make it easier and error-free. On Mon, Mar 19, 2018 at 11:59 PM Romain Manni-Bucau wrote: > Hi Huygaa, > > Cant it be predefined timers? > > Romain > > Le 20 mars 2018 00:52, "Huygaa Batsaikhan" a écrit : > > Hi

From Beam Summit - On SDKs and Contributor Experience

Hello everyone, at the Beam Summit in San Francisco, a number of folks had a breakout session where we considered questions of the experience for new contributors in Beam. First of all, I'd like to thank everyone for participating with your insightful comments and discussion. Along with Rafael, we

Re: Common model for runners

Go currently prints out the model pipeline (as well as the Dataflow representation) if you use the Dataflow runner. Pass --dry_run=true to not actually submit a job, but just print out the representations. The graphx package can also be used to generate a model pipeline manually. On Tue, Mar 20,

Re: Common model for runners

The proto representation isn't (yet) part of the public API, and is still under active development. However, if you're curious you can see it via calling pipeline.to_runner_api() in Python or manually invoking classes under

Common model for runners

Hi,  When I build a data flow using the Beam SDK, can someone point me to the code that represents the underlying representation of the beam model itself?  Is there an API that lets me retrieve the underlying protobuf-based graph for the data flow? Perhaps some pointers to what code in the

Re: Help with Dynamic writing

On Tue, Mar 20, 2018 at 12:54 PM OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > Hi Cham, > > Please see inline. If possible, code / pseudo code will help a lot. > Thanks, > Eila > > On Tue, Mar 20, 2018 at 1:15 PM, Chamikara Jayalath > wrote: > >> Hi

Re: Pubsub API feedback

Thank you Udi. Left some high level comments on the PR. On Mon, Mar 19, 2018 at 5:13 PM, Udi Meiri wrote: > Hi, > I wanted to get feedback about the upcoming Python Pubsub API. It is > currently experimental and only supports reading and writing UTF-8 strings. > My current

[RESULT] [VOTE] Release 2.4.0, release candidate #3

I'm happy to announce that we have unanimously approved this release. There are 9 approving votes, 5 of which are binding: * Lukasz Cwik * Ahmet Altay * Robert Bradshaw * Jean-Baptiste Onofré * Ismaël Mejía The strong desire to get the teardown fixes in ASAP was also noted. Thanks everyone!

Re: [VOTE] Release 2.4.0, release candidate #3

+1 (binding) On Tue, Mar 20, 2018 at 10:46 AM Robert Bradshaw wrote: > On Tue, Mar 20, 2018 at 4:08 AM Ismaël Mejía wrote: > >> +1 (binding) >> >> - Validated hashs >> - mvn clean verify -Prelease OK >> - Run nexmark on direct/flink/spark (it works save

Re: [VOTE] Release 2.4.0, release candidate #3

On Tue, Mar 20, 2018 at 4:08 AM Ismaël Mejía wrote: > +1 (binding) > > - Validated hashs > - mvn clean verify -Prelease OK > - Run nexmark on direct/flink/spark (it works save the regression > already tracked on RC2). > > Thanks Robert for being managing the release. > > ps.

Re: PipelineOptions strict mode broken?

2018-03-20 17:53 GMT+01:00 Lukasz Cwik : > The only current validator is the @Required validator, there were some > ideas to integrate another system to perform validation on options like >=0 > for numbers. I'm not sure how much use this has gotten from users, I would > be for

Re: Help with Dynamic writing

Hi Eila, Please find my comments inline. On Tue, Mar 20, 2018 at 8:02 AM OrielResearch Eila Arich-Landkof < e...@orielresearch.org> wrote: > Hello all, > > It was nice to meet you last week!!! > > It was nice to meet you as well :) > I am writing genomic pCollection that is created from

Re: PipelineOptions strict mode broken?

The only current validator is the @Required validator, there were some ideas to integrate another system to perform validation on options like >=0 for numbers. I'm not sure how much use this has gotten from users, I would be for leaving it as is (if users get value out of it) or removing it and

PipelineOptions strict mode broken?

Hi guys, PipelineOptionsFactory has a nice strict mode validating the options you pass. Concretely if you pass --sudoMakeItWork you will ikely see: java.lang.IllegalArgumentException: Class interface org.apache.beam.sdk.options.PipelineOptions missing a property named ' sudoMakeItWork'. This

Help with Dynamic writing

Hello all, It was nice to meet you last week!!! I am writing genomic pCollection that is created from bigQuery to a folder. Following is the code with output so you can run it with any small BQ table and let me know what your thoughts are: rows = [{u'index': u'GSM2313641', u'SNRPCP14':

Re: Implementing @OnWindowExpiration in StatefulParDo [BEAM-1589]

+1 It sounds good to me. Regards JB Le 20 mars 2018 à 00:52, à 00:52, Huygaa Batsaikhan a écrit: >Hi everyone, I am working on BEAM-1589 >. In short, currently, >there is no default way of saving/flushing state before a window

Build failed in Jenkins: beam_Release_NightlySnapshot #719

See Changes: [sidhom] Add ExecutableStagePayload to make aid runner stage reconstruction [sidhom] Fix typo [tgroh] Use InstructionRequestHandler in RemoteEnvironment [axelmagn] Add a generic

Re: Implementing @OnWindowExpiration in StatefulParDo [BEAM-1589]

Hi Huygaa, Cant it be predefined timers? Romain Le 20 mars 2018 00:52, "Huygaa Batsaikhan" a écrit : Hi everyone, I am working on BEAM-1589 . In short, currently, there is no default way of saving/flushing state before a

Re: [VOTE] Release 2.4.0, release candidate #3

I also tried to run python streaming mobile gaming examples (hourly_team_score, leader_board) on direct runner with little success: https://issues.apache.org/jira/browse/BEAM-3889. I think they escaped our attention on previous release validations. I just tried them with 2.3.0 and didn't have

Re: [VOTE] Release 2.4.0, release candidate #3

+1 (binding) Verified on beam samples (Java) with spark, direct and flink runner. No major change regarding execution time with nexmark. Thanks Regards JB Le 19 mars 2018 à 21:49, à 21:49, Robert Bradshaw a écrit: >Thanks! > >BTW, in case anyone's wondering where the md5

Re: [VOTE] Release 2.4.0, release candidate #3

+1 (binding) I did some more manual validation myself, the burn-down list is empty, and I the artifacts should be correct (though it feels odd to validate that myself, as I'm the one who created them). On Mon, Mar 19, 2018 at 7:22 PM Valentyn Tymofieiev wrote: > +1. > Ran