Re: python precommit error - google-auth depenedency?

Hi, google-auth has been released (with the wider pin on rsa). On Wed, Jun 10, 2020 at 6:07 PM Ahmet Altay wrote: > > > On Wed, Jun 10, 2020 at 4:07 PM Kyle Weaver wrote: > >>

Re: python precommit error - google-auth depenedency?

On Wed, Jun 10, 2020 at 7:11 PM Bu Sun Kim wrote: > Hi, > > google-auth has been released (with the wider pin > > on > rsa). > Thank you! Much appreciated! > > On Wed, Jun

Re: contributor permission for Beam Jira tickets

Done. Welcome! On Wed, Jun 10, 2020 at 12:37 PM Stuart Perks wrote: > Hi, > > Can I be added to the JIRA contributor so I can assign a Jira to myself > please? > > User Name: Perks > > Thanks, > > Stuart >

Re: [External] Re: Ensuring messages are processed and emitted in-order

Thank y’all for the input! About the RequiresTimeSortedInput, we were thinking of the following 2 potential approaches: 1. Assign kafka offset as the timestamp while doing a GroupByKey on partition_id in a GlobalWindow 2. Rely on the fact that Flink consumes from kafka

Re: Beam Summit Status Report - 6/10

Thank you Brittany and all others working on this. Progress looks good. :) On Wed, Jun 10, 2020 at 4:56 PM Brittany Hermann wrote: > Hi folks, > > I wanted to provide you with the Beam Summit Status report from today's > meeting. If you would like to join the next public meeting on Wednesday, >

Re: python precommit error - google-auth depenedency?

On Wed, Jun 10, 2020 at 4:07 PM Kyle Weaver wrote: > The fix to google-auth has been merged. Is the plan just to wait until a > new version of google-auth is released and ignore the failing tests until > then? (btw I filed a JIRA for this before I realized it was already being > discussed here:

Beam Summit Status Report - 6/10

Hi folks, I wanted to provide you with the Beam Summit Status report from today's meeting. If you would like to join the next public meeting on Wednesday, June 24th at 11:30 AM PST please let me know and I will send a calendar invite over to you! Also don't forget to submit your CFP

Re: python precommit error - google-auth depenedency?

The fix to google-auth has been merged. Is the plan just to wait until a new version of google-auth is released and ignore the failing tests until then? (btw I filed a JIRA for this before I realized it was already being discussed here: https://issues.apache.org/jira/browse/BEAM-10232) On Wed,

Re: python precommit error - google-auth depenedency?

Yes you're right, Py2 envs are still using 4.0. On Wed, Jun 10, 2020 at 3:03 PM Ahmet Altay wrote: > > > On Wed, Jun 10, 2020 at 2:25 PM Udi Meiri wrote: > >> 4.1 drops Python 2 support, so I'm not sure if we're ready for that yet. >> > > Wouldn't that work by default? In python 2

Re: python precommit error - google-auth depenedency?

On Wed, Jun 10, 2020 at 2:25 PM Udi Meiri wrote: > 4.1 drops Python 2 support, so I'm not sure if we're ready for that yet. > Wouldn't that work by default? In python 2 oauth2client's rsa>3.14 requirement will resolve to latest python2 supporting version of rsa (4.0?) > > On Wed, Jun 10, 2020

Re: python precommit error - google-auth depenedency?

4.1 drops Python 2 support, so I'm not sure if we're ready for that yet. On Wed, Jun 10, 2020 at 2:20 PM Ahmet Altay wrote: > Looks like there is an attempt to fix this: > https://github.com/googleapis/google-auth-library-python/pull/524 > > On Wed, Jun 10, 2020 at 2:07 PM Udi Meiri wrote: >

Re: python precommit error - google-auth depenedency?

Looks like there is an attempt to fix this: https://github.com/googleapis/google-auth-library-python/pull/524 On Wed, Jun 10, 2020 at 2:07 PM Udi Meiri wrote: > > > On Wed, Jun 10, 2020 at 1:59 PM Ahmet Altay wrote: > >> >> >> On Wed, Jun 10, 2020 at 1:29 PM Kenneth Knowles wrote: >> >>> You

Re: python precommit error - google-auth depenedency?

On Wed, Jun 10, 2020 at 1:59 PM Ahmet Altay wrote: > > > On Wed, Jun 10, 2020 at 1:29 PM Kenneth Knowles wrote: > >> You may be interested in following https://github.com/pypa/pip/issues/988 if >> you are not already. >> >> Kenn >> >> On Wed, Jun 10, 2020 at 12:17 PM Udi Meiri wrote: >> >>>

Re: python precommit error - google-auth depenedency?

On Wed, Jun 10, 2020 at 1:29 PM Kenneth Knowles wrote: > You may be interested in following https://github.com/pypa/pip/issues/988 if > you are not already. > > Kenn > > On Wed, Jun 10, 2020 at 12:17 PM Udi Meiri wrote: > >> Seems like manually installing rsa==4.0 satisfies deps, but pip

Re: Question on NEXMark

It sounds like it could be something worth addressing. I don't really know the cost of this behavior. The pipeline is pretty easy to read. The pipeline itself does not explicitly manage any state, so it would be in the Flink execution of the GroupByKey primitive transform. The relevant code is

Re: python precommit error - google-auth depenedency?

You may be interested in following https://github.com/pypa/pip/issues/988 if you are not already. Kenn On Wed, Jun 10, 2020 at 12:17 PM Udi Meiri wrote: > Seems like manually installing rsa==4.0 satisfies deps, but pip doesn't do > transitive deps well. > > Would it be right to put a direct

Re: Ensuring messages are processed and emitted in-order

I don't know how well RequiresTimeSortedInput will work for any late data. I think you will want to include the Kafka offset in your records (unless the records have their own sequence number) and then use state to buffer and sort. There is a proposal (and work in progress) for a sorted state

Re: Ensuring messages are processed and emitted in-order

For runners that support @RequiresTimeSortedInput, all your input will come time sorted (as long as your element's timestamp tracks the order that you want). For runners that don't support this, you need to build a StatefulDoFn that buffers out of order events and reorders them to the order that

Re: DRAFT - Beam board report June 2020

Added some updates related to IO connectors support. Thanks. On Tue, Jun 9, 2020 at 9:04 PM Jean-Baptiste Onofre wrote: > Hi, > > It looks good with the latest proposed changes. > > Regards > JB > > Le 9 juin 2020 à 20:36, Kenneth Knowles a écrit : > > Ping! It is now June, and time to submit

[ANNOUNCE] Beam 2.22.0 Released

The Apache Beam team is pleased to announce the release of version 2.22.0. Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream (continuous) processing. See https://beam.apache.org You can download the release

Re: python precommit error - google-auth depenedency?

Seems like manually installing rsa==4.0 satisfies deps, but pip doesn't do transitive deps well. Would it be right to put a direct dependency on rsa<4.1,>=3.1.4 in setup.py? On Wed, Jun 10, 2020 at 11:48 AM Udi Meiri wrote: > Thanks, that helped in an unexpected way. :) > I should have used

Re: Remove EOL'd Runners

Sounds good, thanks. I removed Gearpump first and will move on to Apex later today. When that PR is merged we can clean up the Jenkins jobs in one swoop for both removed runners. On Wed, Jun 10, 2020, 11:18 AM Luke Cwik wrote: > The jobs won't be deleted but will be disabled. I can help delete

Re: python precommit error - google-auth depenedency?

Thanks, that helped in an unexpected way. :) I should have used the "gcp" extra instead of "cloud" in my pip install command above. On Wed, Jun 10, 2020 at 11:37 AM Valentyn Tymofieiev wrote: > > Any ideas on how to debug where this requirement is coming from? > You could try installing and

Re: python precommit error - google-auth depenedency?

> Any ideas on how to debug where this requirement is coming from? You could try installing and calling pipdeptree [1] from a Jenkins job, and see if it helps. [1] https://pypi.org/project/pipdeptree/ On Wed, Jun 10, 2020 at 11:00 AM Udi Meiri wrote: > Hi, > I'm trying to understand these "pip

Re: Remove EOL'd Runners

The jobs won't be deleted but will be disabled. I can help delete the jobs from Jenkins once the jenkins configurations are removed either ping me directly or update this thread when that should be done. On Wed, Jun 10, 2020 at 10:38 AM Kenneth Knowles wrote: > +1 > > All Jenkins configs are in

python precommit error - google-auth depenedency?

Hi, I'm trying to understand these "pip check" failures: ERROR: google-auth 1.16.1 has requirement rsa<4.1,>=3.1.4, but you'll have rsa 4.1 which is incompatible https://builds.apache.org/job/beam_PreCommit_Python_Cron/2860/console However, when I do pip install

Re: Question on NEXMark

I think the author of this test is long gone, but the code originated inside google. This query is not part of the original Nexmark suite but was designed to exercise corner cases caused by out of order events, so that is what you are probably seeing. Here are relevant bits from the original

Season of Docs: Interested in working with Apache Beam

Greetings, I am hoping this finds you well. I am keenly interested in contributing to Apache Beam during the Season of Docs. I am interested in growing my technical writing skills and see the Season of Docs as an excellent way to achieve that goal. As an entry-level technical writer, I would

contributor permission for Beam Jira tickets

Hi, Can I be added to the JIRA contributor so I can assign a Jira to myself please? User Name: Perks Thanks, Stuart

Re: Remove EOL'd Runners

+1 All Jenkins configs are in the repo. There's a lag between merge and run of the "seed job" that syncs our configs. We can do a manual run of it, or just not worry about the temporary redness in the jobs that will be deleted anyhow. On Wed, Jun 10, 2020 at 8:57 AM Jan Lukavský wrote: > +1 >

Question on NEXMark

Hi, We are working on a Flink project and enhancing some state backend functionality. We are using NEXMark benchmark to compare different state backends performance of Flink. While running NEXMark queries using Flink runner of Beam we have noticed that there is quite a lot of non-existent read

Re: Running apache_beam python sdk without c/c++ libs

I'm not sure. It depends on whether the Spark -> Beam Python integration will interfere with the magic built into AWS Glue. On Wed, Jun 10, 2020 at 8:57 AM Noah Goodrich wrote: > I was hoping to use the Spark runner since Glue is just Spark with some > magic on top. And in our specific use

Re: Running apache_beam python sdk without c/c++ libs

I was hoping to use the Spark runner since Glue is just Spark with some magic on top. And in our specific use case, we'd be looking at working with S3, Kinesis, and MySQL RDS. Sounds like this is a non-starter? On Wed, Jun 10, 2020 at 9:33 AM Luke Cwik wrote: > Most runners are written in Java

Re: Remove EOL'd Runners

+1 On 6/10/20 5:51 PM, David Morávek wrote: +1 On Tue, Jun 9, 2020 at 7:43 PM Ahmet Altay > wrote: Thank you Tyson! On Tue, Jun 9, 2020 at 10:20 AM Thomas Weise mailto:t...@apache.org>> wrote: +1 On Tue, Jun 9, 2020 at 9:41 AM Robert

Re: Remove EOL'd Runners

+1 On Tue, Jun 9, 2020 at 7:43 PM Ahmet Altay wrote: > Thank you Tyson! > > On Tue, Jun 9, 2020 at 10:20 AM Thomas Weise wrote: > >> +1 >> >> >> On Tue, Jun 9, 2020 at 9:41 AM Robert Bradshaw >> wrote: >> >>> Makes sense to me. >>> >>> On Tue, Jun 9, 2020 at 8:45 AM Maximilian Michels >>>

Re: Running apache_beam python sdk without c/c++ libs

Most runners are written in Java while others are cloud offerings which wouldn't work for your use case which limits you to use the direct runner (not meant for production/high performance applications). Beam Python SDK uses cython for performance reasons but I don't believe it strictly requires

Running apache_beam python sdk without c/c++ libs

I am looking at using the Beam Python SDK in AWS Glue but it doesn't support non-native python libraries (anything that is c/c++ based). Is the Beam Python SDK / runners able to be used without any c/c++ library dependencies?

beam_PreCommit_Java_Phrase is hanging

Hello, Seems like “beam_PreCommit_Java_Phrase” is hanging. Last job run was triggered on 8th June [1] and new jobs can’t be started. Could someone with a Jenkins "master power" take a look on this? Thanks! https://builds.apache.org/job/beam_PreCommit_Java_Phrase/2303/