Re: What quick command to catch common issues before pushing a python PR?

2019-03-19 Thread Pablo Estrada
Correction - the command is now: tox -e py35-gcp,py35-lint And it ran on my machine in 5min 40s. Not blazing fast, but at least significantly faster than waiting for Jenkins : ) Best -P. On Tue, Mar 19, 2019 at 5:22 PM Pablo Estrada wrote: > I use a selection of tox tasks. Here are the tox

Re: Added a Jira beginner's guide to the wiki.

2019-03-19 Thread Pablo Estrada
Hi Dan! Very nice intro. An idea is that some of the descriptions could have links to the sections they're talking about (e.g. when talking about different projects in JIRA, provide a link to issues fro the BEAM project). Other than that, it's a nice introduction. : ) Udi:

Re: Selectively running tests?

2019-03-19 Thread Pablo Estrada
Oops. False alarm. Thanks guys : ) Best -P. On Tue, Mar 19, 2019 at 10:53 AM Udi Meiri wrote: > At least for python, any changes under runners/ trigger a precommit: > https://github.com/apache/beam/blob/34b071c0b0f08fceb451ce9c4a4bcaab6160abcd/.test-infra/jenkins/job_PreCommit_Python.groovy#L27

Re: What quick command to catch common issues before pushing a python PR?

2019-03-19 Thread Pablo Estrada
I use a selection of tox tasks. Here are the tox tasks that I use the most: - py27-gcp - py35-gcp - py27-cython - py35-cython - py35-lint - py27-lint Most recently, I'll run `tox -e py3-gcp,py3-lint`, which run fairly quickly. You can choose which subset works for you. My insight is: Lints are

Re: Python Datastore client upgrade plan

2019-03-19 Thread Kenneth Knowles
Thanks for taking the time to communicate about the updated plan. Really appreciate it. Kenn On Tue, Mar 19, 2019 at 10:25 AM Udi Meiri wrote: > Update: I'm back to working on this. > To allow a smoother migration, I'm planning on having apache-beam depend > on both googledatastore and

Re: Selectively running tests?

2019-03-19 Thread Udi Meiri
At least for python, any changes under runners/ trigger a precommit: https://github.com/apache/beam/blob/34b071c0b0f08fceb451ce9c4a4bcaab6160abcd/.test-infra/jenkins/job_PreCommit_Python.groovy#L27 On Tue, Mar 19, 2019 at 10:38 AM Pablo Estrada wrote: > Perhaps it's just that the rules are more

Re: [Announcement] New Website for Beam Summits

2019-03-19 Thread Matthias Baetens
Awesome Aizhamal! Great work and thanks for your continued efforts on this :) Looking forward to the summit. On Mon, 18 Mar 2019 at 23:17, Aizhamal Nurmamat kyzy wrote: > Hello everybody! > > We are thrilled to announce the launch of beamsummit.org dedicated to > Beam Summits! > > The current

Re: [Announcement] New Website for Beam Summits

2019-03-19 Thread Maximilian Michels
Great stuff! Looking forward to seeing many Beam folks in Berlin. In case you want to speak at Beam Summit Europe, the Call for Papers is open until April 1: https://sessionize.com/beam-summit-europe-2019 -Max On 19.03.19 09:49, Matthias Baetens wrote: Awesome Aizhamal! Great work and thanks

Beam meetup London at Revolut 03/04/2019

2019-03-19 Thread Matthias Baetens
Hi everyone, On Wednesday 03/04, we will have the next Beam meetup in London at the Revolut offices. Robert Bradshaw will do a deep dive on portability, while Revolut will present its use-case on using Beam for fraud detection. More info and RSVP on the meetup page

Re: Connected streams with Beam

2019-03-19 Thread Suneel Marthi
Thanks, CoGroupByKey is what i need. On Tue, Mar 19, 2019 at 12:24 PM Maximilian Michels wrote: > Hi Suneel, > > Depending on what exactly you want to to, please have a look at Flatten > or CoGroupByKey: > > https://beam.apache.org/documentation/programming-guide/#flatten >

Re: [Announcement] New Website for Beam Summits

2019-03-19 Thread Łukasz Gajowy
Looks great! Thanks for doing this! :) Łukasz wt., 19 mar 2019 o 12:30 Maximilian Michels napisał(a): > Great stuff! Looking forward to seeing many Beam folks in Berlin. > > In case you want to speak at Beam Summit Europe, the Call for Papers is > open until April 1:

Re: Connected streams with Beam

2019-03-19 Thread Maximilian Michels
Hi Suneel, Depending on what exactly you want to to, please have a look at Flatten or CoGroupByKey: https://beam.apache.org/documentation/programming-guide/#flatten https://beam.apache.org/documentation/programming-guide/#cogroupbykey Both are available in Python/Java. -Max On 18.03.19

Re: Python Datastore client upgrade plan

2019-03-19 Thread Udi Meiri
Update: I'm back to working on this. To allow a smoother migration, I'm planning on having apache-beam depend on both googledatastore and google-cloud-datastore and having 2 Beam modules. The newer client is a bit more limited in expressing queries (only ANDs for composite filtering). OTOH it

Re: Selectively running tests?

2019-03-19 Thread Udi Meiri
Do you have an example PR Pablo? On Mon, Mar 18, 2019, 18:23 Alan Myrvold wrote: > The includedRegions was set up as part of > https://issues.apache.org/jira/browse/BEAM-4445 and there are additional > paths added from >

Re: Selectively running tests?

2019-03-19 Thread Pablo Estrada
Perhaps it's just that the rules are more extensive for Java worker changes. I noticed that my change runs Go, Python and Java precommits: https://github.com/apache/beam/pull/8080 On Tue, Mar 19, 2019 at 10:27 AM Udi Meiri wrote: > Do you have an example PR Pablo? > > > On Mon, Mar 18, 2019,