RFC: switching to pylint 2.4

2019-10-10 Thread Chad Dombrova
Hi all, I made a PR to update to the latest version of pylint here: https://github.com/apache/beam/pull/9725 I think it's ready to go, but would love to get a few eyes on it. -chad

Re: Beam Python fails to run on macOS 10.15?

2019-10-10 Thread Ahmet Altay
Kamil, would it be possible for you to test with 0.15? I suggest moving the discussion to BEAM-8368 to have a single place for the discussion. On Thu, Oct 10, 2019 at 12:35 PM Brian Hulette wrote: > As noted in BEAM-8368 it seems like pinning to 0.13.0 is a workaround for > now. It would be

Re: [spark structured streaming runner] merge to master?

2019-10-10 Thread Kenneth Knowles
On Thu, Oct 10, 2019 at 2:44 PM Ismaël Mejía wrote: > +1 > > The earlier we get to master the better to encourage not only code > contributions but as important to have early user feedback. > > > Question is: do we keep the "old" spark runner for a while or not (or > just keep on previous

Re: [spark structured streaming runner] merge to master?

2019-10-10 Thread Ismaël Mejía
+1 The earlier we get to master the better to encourage not only code contributions but as important to have early user feedback. > Question is: do we keep the "old" spark runner for a while or not (or just > keep on previous version/tag on git) ? It is still too early to even start discussing

Re: Contributor Permissions

2019-10-10 Thread Luke Cwik
Welcome, I have added you as a contributor. On Thu, Oct 10, 2019 at 2:02 PM Igor Durovic wrote: > Hi! > > I'm Igor Durovic, an intern at LinkedIn. I'm working on Samza runner and > interactive beam. My JIRA username is idurovic. > > Thanks, > Igor Durovic >

Re: Introduction to the mailing list

2019-10-10 Thread Ahmet Altay
Manuela, how is your initial review going? Do you need any help? On Mon, Oct 7, 2019 at 6:51 AM Ismaël Mejía wrote: > Welcome Manuela! > > The documentation on the Beam website [1] can help you too, in > particular to get some familiarity with the runtime part of Nexmark > (and the Java version

Contributor Permissions

2019-10-10 Thread Igor Durovic
Hi! I'm Igor Durovic, an intern at LinkedIn. I'm working on Samza runner and interactive beam. My JIRA username is idurovic. Thanks, Igor Durovic

Re: Beam Python fails to run on macOS 10.15?

2019-10-10 Thread Brian Hulette
As noted in BEAM-8368 it seems like pinning to 0.13.0 is a workaround for now. It would be useful to know what pyarrow versions have this problem, is it just 0.14, or 0.15 as well? Could someone with macOS 10.15 try pip installing pyarrow==0.15.0 and see if they can import it? We currently have a

Re: [DISCUSS] JIRA open/close emails?

2019-10-10 Thread Kenneth Knowles
Sounds like pretty good consensus to keep our current configuration. Kenn On Thu, Oct 10, 2019 at 12:05 PM Robert Bradshaw wrote: > I would also propose that we keep this traffic off of dev@--I don't > think the average dev@ subscriber would want to see these and for > those that do there are

Re: Beam Python fails to run on macOS 10.15?

2019-10-10 Thread Kyle Weaver
Apparently this is an issue with our arrow import. There is a JIRA for this now: https://issues.apache.org/jira/browse/BEAM-8368 On Thu, Oct 10, 2019 at 12:11 PM Robert Bradshaw wrote: > Looks like an issue with the protobuf library. Do you know what > version of protobuf you're using? (E.g. by

Re: Beam Python fails to run on macOS 10.15?

2019-10-10 Thread Robert Bradshaw
Looks like an issue with the protobuf library. Do you know what version of protobuf you're using? (E.g. by running pip freeze.) I don't have Catalina to test this on, but it'd be useful if you could winnow this down to the import that fails. On Thu, Oct 10, 2019 at 8:15 AM Kamil Wasilewski

Re: [spark structured streaming runner] merge to master?

2019-10-10 Thread Kenneth Knowles
+1 I think our experiences with things that go to master early have been very good. So I am in favor ASAP. We can exclude it from releases easily until it is ready for end users. I have the same question as Robert - how much is modifications and how much is new? I notice it is in a subdirectory

Re: [spark structured streaming runner] merge to master?

2019-10-10 Thread Robert Bradshaw
On Thu, Oct 10, 2019 at 12:39 AM Etienne Chauchot wrote: > > Hi guys, > > You probably know that there has been for several months an work > developing a new Spark runner based on Spark Structured Streaming > framework. This work is located in a feature branch here: >

Re: [spark structured streaming runner] merge to master?

2019-10-10 Thread Xinyu Liu
+1 for merging to master. It's going to help a lot for us to try it out, and also contribute back for the missing features. Thanks, Xinyu On Thu, Oct 10, 2019 at 6:40 AM Alexey Romanenko wrote: > +1 for merging this new runner too (even if it’s not 100% ready for the > moment) in case if it

Re: Do we know why gradle scans are not working?

2019-10-10 Thread Luke Cwik
The Gradle forums[1] suggested that we will need to downgrade the build scan plugin to 2.3 to get the build scans working again. Tested it locally and it worked. Filed BEAM-8378 and opened pr/9762 with the downgrade. 1: https://discuss.gradle.org/t/your-build-scan

Re: [DISCUSS] JIRA open/close emails?

2019-10-10 Thread Kyle Weaver
Customized email notifications from Jira +1. Jira notifications will be useful to some, but not all, so they should be opt-in rather than filter-out. I spend enough time filtering internal spam as is :/ On Thu, Oct 10, 2019 at 8:12 AM Chad Dombrova wrote: > > > On Thu, Oct 10, 2019 at 10:56 AM

Beam Python fails to run on macOS 10.15?

2019-10-10 Thread Kamil Wasilewski
Hi all, I've recently updated my macOS to 10.15 Catalina. Since then, I have the following error when I try to import apache_beam package (both in python 2.7 and 3.x): >>> import apache_beam [libprotobuf ERROR google/protobuf/descriptor_database.cc:58] File already exists in database:

Re: [DISCUSS] JIRA open/close emails?

2019-10-10 Thread Chad Dombrova
On Thu, Oct 10, 2019 at 10:56 AM Maximilian Michels wrote: > Our mailing list already has decent traffic. The current solution is > better for its readability. People would simply adapt to more emails by > creating a filter or ignoring them. +1. I find the dev generally informative, but as a

Re: [spark structured streaming runner] merge to master?

2019-10-10 Thread Alexey Romanenko
+1 for merging this new runner too (even if it’s not 100% ready for the moment) in case if it doesn’t break/fail/affect all other tests and Jenkins jobs. I mean, it should be transparent for other Beam components. Also, since it won’t be officially “released” right after merging, we need to

Re: Apply Confluence permission for fix `Python Tips` wiki command issue

2019-10-10 Thread Thomas Weise
Thanks for spotting this. Access granted (for user "jincheng"). On Thu, Oct 10, 2019 at 12:01 AM jincheng sun wrote: > Hi all, > > The docker config has been removed with the latest python3 docker related > commit [1], So the command: > > ./gradlew :sdks:python:container:docker > in `Python

Re: [spark structured streaming runner] merge to master?

2019-10-10 Thread Ryan Skraba
Merging to master sounds like a really good idea, even if it is not feature-complete yet. It's already a pretty big accomplishment getting it to the current state (great job all!). Merging it into master would give it a pretty good boost for visibility and encouraging some discussion about where

Re: [spark structured streaming runner] merge to master?

2019-10-10 Thread Jean-Baptiste Onofré
+1 As the runner seems almost "equivalent" to the one we have, it makes sense. Question is: do we keep the "old" spark runner for a while or not (or just keep on previous version/tag on git) ? Regards JB On 10/10/2019 09:39, Etienne Chauchot wrote: > Hi guys, > > You probably know that there

[spark structured streaming runner] merge to master?

2019-10-10 Thread Etienne Chauchot
Hi guys, You probably know that there has been for several months an work developing a new Spark runner based on Spark Structured Streaming framework. This work is located in a feature branch here: https://github.com/apache/beam/tree/spark-runner_structured-streaming To attract more

Apply Confluence permission for fix `Python Tips` wiki command issue

2019-10-10 Thread jincheng sun
Hi all, The docker config has been removed with the latest python3 docker related commit [1], So the command: > ./gradlew :sdks:python:container:docker in `Python Tips`[2] can not work well, we should correct it, something like: > ./gradlew :sdks:python:container:py35:docker The flink 1.5 and

Re: Beam Summit Videos in youtube

2019-10-10 Thread Matthias Baetens
Hi Chad, Apologies for the delay here. We are post-processing the videos into something that is more watchable, but happy to share the raw videos if that is something you are interested in. Which sessions specifically can we help with? :) Best regards, Matthias On Mon, 7 Oct 2019 at 10:00,