Re: [PROPOSAL] Prepare Beam 2.7.0 release

Thanks! I talked to Boyuan who indicated that this may be an issue specific to the JVM version I am running, since she saw a similar issue while performing the 2.5.0 release. Next step on my end is to set up another machine with a different JVM version to run these steps. On Wed, Sep 5, 2018 at

Uber Jar: DataflowRunner + GCP I/O + Pipeline

I have had a few users ask me directly how do I create an uber jar containing the DataflowRunner, GCP I/O and their pipeline. I wanted to shared this example Maven project[1] that created an executable uber jar containing those components and the Apache Beam examples. The instructions in README.md

Re: [Proposal] Creating a reproducible environment for Beam Jenkins Tests

Thank you all for making comments on this and I apologize for the late reply. To clarify the concerns of testing locally, it is still able to run tests without Docker. One of the purposes of this is to create an identical environment as we are running in Jenkins that would be helpful to reproduce

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

+1 on the jira "fix version". The release frequency of dependencies are various, so that using new information such as versions from the Jira closing date to reopen the issues might not be very proper. We could check the fix versions first, and if specified, then reopen the issue in that version's

Re: builds.apache.org refused connections since last night

Update: I looked into mergebot and it was down, so I requested a restart (thank you Jason) and robot merging on beam-site is now working again. On Tue, Sep 4, 2018 at 5:15 AM Jean-Baptiste Onofré wrote: > Let me check. > > Regards > JB > > On 04/09/2018 14:10, Ismaël Mejía wrote: > > The issu

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

On Wed, Sep 5, 2018 at 12:50 PM Tim Robertson wrote: > Thank you Cham, and everyone for contributing > > Sorry for slow reply to a thread I started, but I've been swamped on non > Beam projects. > > KafkaIO's policy of 'let the user decide exact version at runtime' has >> been quite useful so far

Re: SplittableDoFn

Thanks to all those who have provided interest in this topic by the questions they have asked on the doc already and for those interested in having this discussion. I have setup this doodle to allow people to provide their availability: https://doodle.com/poll/nrw7w84255xnfwqy I'll send out the ch

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

Thank you Cham, and everyone for contributing Sorry for slow reply to a thread I started, but I've been swamped on non Beam projects. KafkaIO's policy of 'let the user decide exact version at runtime' has been > quite useful so far. How feasible is that for other connectors? I presume shimming

Re: [PROPOSAL] Test performance of basic Apache Beam operations

neat! left a comment or two On Mon, Sep 3, 2018 at 3:53 AM Łukasz Gajowy wrote: > Hi all! > > I'm bumping this (in case you missed it). Any feedback and questions are > welcome! > > Best regards, > Łukasz > > pon., 13 sie 2018 o 13:51 Jean-Baptiste Onofré > napisał(a): > >> Hi Lukasz, >> >> Tha

Re: Test set fails: beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle

The first issue to resolve is actually not individual tests but the build: https://issues.apache.org/jira/browse/BEAM-5283 I will file a separate JIRA for the tests that I see failing locally. On Wed, Sep 5, 2018 at 7:26 PM Connell O'Callaghan wrote: > Thank you both for addressing failing te

Re: INFO:root:Executing Error when executing a pipeline on dataflow

Ella, I believe you are using version 2.0.0. Use of --download flag is fixed at head. (I do not recall the exact version of the fix, could be 2.2 or 2.3). If possible please try to use a newer version of Beam. Also, as Luke suggested, we would welcome any contributions to the documentation. On Fr

Re: Test set fails: beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle

Thank you both for addressing failing tests. On Wed, Sep 5, 2018, 10:05 AM Thomas Weise wrote: > I will take a look. I know that still 2 tests fail in streaming mode that > need to be investigated and fixed. Will create a ticket with more context. > > -- > sent from mobile > > On Wed, Sep 5, 201

[Call for items] September Beam Newsletter

Hi Beamers: Here's [1] the template for the September Beam Newsletter! *Add the highlights from August to now (or planned events and talks) that you want to share with the community by 9/8 11:59 p.m

Re: Test set fails: beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle

I will take a look. I know that still 2 tests fail in streaming mode that need to be investigated and fixed. Will create a ticket with more context. -- sent from mobile On Wed, Sep 5, 2018, 6:59 PM Mikhail Gryzykhin wrote: > Hi everyone, > > We have added beam_PostCommit_Python_PortableValidate

Re: Python 3: final step

This is awesome! Kudos to Robbe and Matthias who have been pushing this forward! On Wed, Sep 5, 2018 at 9:45 AM Charles Chen wrote: > This is great! Feel free to add me as a reviewer. > > On Wed, Sep 5, 2018 at 9:38 AM Andrew Pilloud wrote: > >> Cool! I know very little about Python 3, but hap

Test set fails: beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle

Hi everyone, We have added beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle job to Jenkins several days ago. However all of its runs have failed so far. Are we working on the fix, or can we dis

Re: Python 3: final step

This is great! Feel free to add me as a reviewer. On Wed, Sep 5, 2018 at 9:38 AM Andrew Pilloud wrote: > Cool! I know very little about Python 3, but happy to help review. > > Andrew > > On Wed, Sep 5, 2018 at 9:21 AM Ahmet Altay wrote: > >> Thank you Robbe, this is great news! >> >> On Wed, S

Re: Python 3: final step

Cool! I know very little about Python 3, but happy to help review. Andrew On Wed, Sep 5, 2018 at 9:21 AM Ahmet Altay wrote: > Thank you Robbe, this is great news! > > On Wed, Sep 5, 2018 at 9:11 AM, Robbe Sneyders > wrote: > >> Hi everyone, >> >> With the merging of [1], we now have Python 3 t

Re: Python 3: final step

Thank you Robbe, this is great news! On Wed, Sep 5, 2018 at 9:11 AM, Robbe Sneyders wrote: > Hi everyone, > > With the merging of [1], we now have Python 3 tests running on Jenkins, > which allows us to move forward with the last step of the Python 3 porting. > > You can follow the progress on t

Python 3: final step

Hi everyone, With the merging of [1], we now have Python 3 tests running on Jenkins, which allows us to move forward with the last step of the Python 3 porting. You can follow the progress on the Jira Kanban Board [2]. If you're interested in helping by porting a module, you can assign one of the

Re: [Proposal] Creating a reproducible environment for Beam Jenkins Tests

BIG +1, the previous work on having docker build images [1] had a similar goal (to have a reproducible build environment). But this is even better because we will guarantee the exact same environment in Jenkins as well as any further improvements. It is important to document the setup process as pa

Re: [BEAM-960] Backoff in the DirectRunner if no work is available

Hello Vojtech, Thanks a lot for your work on this fix and for pinging us for review. Improving all runners is important and CPU improvements in direct runner definitely are welcome because we all benefit of this (for testing). Review is in process so you should have news soon. On Thu, Aug 30, 20

Re: [PROPOSAL] Prepare Beam 2.7.0 release

Charles, Does it happen when you run this task separately or as a part of the whole project build? I run it independently and don’t see any issues on my side. Also, I don’t remember that it failed on Jenkins recently. So, probably this segfault is not related to exactly this task. > On 5 Sep 20

Re: Should we mention TF Transform in Beam site?

+1 Perhaps, it would be also helpful to add a small “get started” example to attract user attention. > On 5 Sep 2018, at 02:16, Ahmet Altay wrote: > > +1 > > It would be nice to showcase other tools built on top of Beam. > > On Mon, Aug 20, 2018 at 1:02 PM, Ankur Goenka