New Post Commit Task fails in SetupVirtualEnv when running on Jenkins

Hi, I added a new Post Commit task "beam_PostCommit_Python_PortableValidatesRunner_Flink_Gradle" to Jenkins in https://github.com/apache/beam/pull/6322 but the jenkins job is failing with logs below. However the gradle task works fine when i execute it on my local machine using ./gradlew

Re: Failing: beam_PerformanceTests_Python

I created a PR-6344 that synced to a point earlier and expected to see Jenkins error 2 days ago, but not. Probably the current issue is relevant to Jenkins build environment. We can also think about moving performance tests to Gradle which seems

Re: JIRA Permissions

A PMC member had reached out to me saying that I should have not granted access. I have revoked JIRA administration access for the three individuals. Because of the private nature of the conversation, I will allow the PMC member to choose to share any details. On Fri, Aug 31, 2018 at 12:42 PM

Re: [PROPOSAL] Prepare Beam 2.7.0 release

Making steady progress, but there is an issue compiling with Protobuf for Python which I am investigating right now. On Wed, Sep 5, 2018 at 11:13 PM Charles Chen wrote: > Thanks! I talked to Boyuan who indicated that this may be an issue > specific to the JVM version I am running, since she

Re: [NEW CONTRIBUTOR] ElasticsearchIO now supports Elasticsearch v6.x

Welcome Dan! Thank you for the contribution. On Thu, Sep 6, 2018 at 3:13 PM Ismaël Mejía wrote: > Dal Tran this is really nice, this is a much awaited contribution, welcome! > On Thu, Sep 6, 2018 at 9:18 PM Ahmet Altay wrote: > > > > Welcome! Thank you! > > > > On Thu, Sep 6, 2018 at 11:40 AM,

Re: [NEW CONTRIBUTOR] ElasticsearchIO now supports Elasticsearch v6.x

Dal Tran this is really nice, this is a much awaited contribution, welcome! On Thu, Sep 6, 2018 at 9:18 PM Ahmet Altay wrote: > > Welcome! Thank you! > > On Thu, Sep 6, 2018 at 11:40 AM, Chamikara Jayalath > wrote: >> >> Welcome Dat Tran and thanks for the contribution. >> >> On Thu, Sep 6,

Re: [NEW CONTRIBUTOR] ElasticsearchIO now supports Elasticsearch v6.x

Thank you. Etienne Chauchot and Tim Robertson helped me a lot to get familiar with Beam code. On Fri, Sep 7, 2018 at 2:59 AM Thomas Weise wrote: > Support for Elastic 6.x is really good to have. Thanks Dat and welcome to > the Beam community! > > Thomas > > On Thu, Sep 6, 2018 at 9:18 PM Ahmet

Re: Failing: beam_PerformanceTests_Python

Thank you for looking into this. --Mikhail Have feedback ? On Thu, Sep 6, 2018 at 11:20 AM Mark Liu wrote: > BEAM-5296 is tracking > the previous error. Some debugging efforts are made and summarized in > PR-6321

Re: [NEW CONTRIBUTOR] ElasticsearchIO now supports Elasticsearch v6.x

Support for Elastic 6.x is really good to have. Thanks Dat and welcome to the Beam community! Thomas On Thu, Sep 6, 2018 at 9:18 PM Ahmet Altay wrote: > Welcome! Thank you! > > On Thu, Sep 6, 2018 at 11:40 AM, Chamikara Jayalath > wrote: > >> Welcome Dat Tran and thanks for the contribution.

Re: [NEW CONTRIBUTOR] ElasticsearchIO now supports Elasticsearch v6.x

Welcome! Thank you! On Thu, Sep 6, 2018 at 11:40 AM, Chamikara Jayalath wrote: > Welcome Dat Tran and thanks for the contribution. > > On Thu, Sep 6, 2018 at 1:17 AM Alexey Romanenko > wrote: > >> Welcome to Beam community, Dat Tran! Thank you for your work! >> >> On 6 Sep 2018, at 10:10,

Re: [DISCUSS] Unification of Hadoop related IO modules

I think 4 is best for users since when a user comes from the Hadoop ecosystem, it is likely they are using many parts of Hadoop and would likely get value from having everything together. My concern with 4 is whether a single Hadoop package would be overwhelming from a dependencies point of view.

Re: [NEW CONTRIBUTOR] ElasticsearchIO now supports Elasticsearch v6.x

Welcome Dat Tran and thanks for the contribution. On Thu, Sep 6, 2018 at 1:17 AM Alexey Romanenko wrote: > Welcome to Beam community, Dat Tran! Thank you for your work! > > On 6 Sep 2018, at 10:10, Etienne Chauchot wrote: > > Hi all , > > Just to let you know that Elasticsearch IO now supports

Re: Uber Jar: DataflowRunner + GCP I/O + Pipeline

Thanks Luke. Should this information be added to wiki/Website ? - Cham On Wed, Sep 5, 2018 at 5:20 PM Lukasz Cwik wrote: > I have had a few users ask me directly how do I create an uber jar > containing the DataflowRunner, GCP I/O and their pipeline. I wanted to > shared this example Maven

Re: Failing: beam_PerformanceTests_Python

BEAM-5296 is tracking the previous error. Some debugging efforts are made and summarized in PR-6321 . But I didn't realize test failed in different error now. Mark On Thu, Sep 6,

Re: Should we mention TF Transform in Beam site?

There are several examples on using TFT on the cloudml-samples repo . Particularly, I've been working on an end-to-end sample using Beam+TFT to do the

Failing: beam_PerformanceTests_Python

Hi everyone, I see that beam_PerformanceTests_Python job is failing during last several days. Corresponding errors: 2 days ago it was failure in beam_integration_benchmark(1/1)

Re: [DISCUSS] Unification of Hadoop related IO modules

I'd vote for (1). For most of the IO modules, it makes sense to develop and keep read and write parts together given that they usually connect to the same datastore. But hadoop-input-format and hadoop-output-format are simply a level of indirection to connect to various data stores supported by

Re: [Discussion] Clarify the support story for released Beam versions

Sorry for the late reply, I was out of office for a while. On Thu, Aug 23, 2018 at 3:21 PM, Andrew Pilloud wrote: > It would be good to gather data on who the users will be and what they > expect out of it. Beam users appear to be tech savvy early adopters, and > Enterprise users of that class

[DISCUSS] Unification of Hadoop related IO modules

Hello everyone, I’d like to discuss the following topic (see below) with community since the optimal solution is not clear for me. There is Java IO module, called “hadoop-input-format”, which allows to use MapReduce InputFormat implementations to read data from different sources (for example,

Re: "Apache Jenkins Server" emails missing?

Hi, sorry, I forgot to take a look yesterday. I will take a look later today. Regards JB On 06/09/2018 15:10, Łukasz Gajowy wrote: > Hi JB, > > those emails were a good indicator of bad things going on. What can we > do to bring them back?  > > Best regards,  > Łukasz > > > wt., 4 wrz 2018

Re: "Apache Jenkins Server" emails missing?

Hi JB, those emails were a good indicator of bad things going on. What can we do to bring them back? Best regards, Łukasz wt., 4 wrz 2018 o 14:09 Jean-Baptiste Onofré napisał(a): > Hi, > > we had a major Jenkins issue last week. But, it should be back now (I > can see it working on Karaf). >

Re: Beam Schemas: current status

It's probably worth publishing this update as a blog post. On Fri, Aug 31, 2018 at 9:58 PM Reuven Lax wrote: > In addition, JdbcIO is another source that could integrate with schemas. > > Another point of integration could be with shared schema registries (such > as Kafka Schema Registry.). Any

Re: [NEW CONTRIBUTOR] ElasticsearchIO now supports Elasticsearch v6.x

Welcome to Beam community, Dat Tran! Thank you for your work! > On 6 Sep 2018, at 10:10, Etienne Chauchot wrote: > > Hi all , > > Just to let you know that Elasticsearch IO now supports version 6 in addition > to version 2 and 5 now that this PR is merged: >

[NEW CONTRIBUTOR] ElasticsearchIO now supports Elasticsearch v6.x

Hi all , Just to let you know that Elasticsearch IO now supports version 6 in addition to version 2 and 5 now that this PR is merged: https://github.com/apache/beam/pull/6211#pullrequestreview-152477892. At this occasion we gained a new contributor: Dat Tran Best Etienne

Re: [PROPOSAL] Prepare Beam 2.7.0 release

Thanks! I talked to Boyuan who indicated that this may be an issue specific to the JVM version I am running, since she saw a similar issue while performing the 2.5.0 release. Next step on my end is to set up another machine with a different JVM version to run these steps. On Wed, Sep 5, 2018 at