Examples of poolArgs in WorkerPool

There is field called poolArgs in the definition of a WorkerPool. How was this intended to be used? Are there known

Re: NOTICE: New Python PreCommit jobs

I haven’t used nose’s parallel execution plugin, but I have used pytest with xdist with success. If your tests are designed to run in any order and are properly sandboxed to prevent crosstalk between concurrent runs, which they *should* be, then in my experience it works very well. On Fri, Sep

Re: Dockerhub push denied for py3.6 and py3.7 image

When she set up the repo, Hannah requested PMC members to ask for privileges, so I did. The set of admins currently is just Hannah and myself - and I don't think this is available in a public page. We could either have a PMC-managed account, or allow more PMC members to have admin privileges -

Re: Multiple iterations after GroupByKey with SparkRunner

On Mon, Sep 30, 2019 at 2:02 AM Jan Lukavský wrote: > > The fact that the annotation on the ParDo "changes" the GroupByKey > implementation is very specific to the Spark runner implementation. > > I don't quite agree. It is not very specific to Spark, it is specific to > generally all runners,

Re: Dockerhub push denied for py3.6 and py3.7 image

Who are the admins on dockerhub currently? Is there a page that shows a list? The next person doing the release will probably run into similar issues. For example, pypi page for beam [1] shows lists of maintainers. [1] https://pypi.org/project/apache-beam/ Thank you, Ahmet On Tue, Oct 1, 2019

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 On Tue, Oct 1, 2019 at 4:27 PM Ruoyun Huang wrote: > +1 > > On Tue, Oct 1, 2019 at 3:52 PM Rui Wang wrote: > >> +1 >> >> I needed to use https://python3statement.org to access the website BTW >> (https, not http). >> >> >> -Rui >> >> On Tue, Oct 1, 2019 at 3:29 PM Cam Mach wrote: >> >>> +1

Re: Multiple iterations after GroupByKey with SparkRunner

For this specific usecase, I would suggest this be done via PTranform URNs. E.g. one could have a GroupByKeyOneShot whose implementation is input .apply(GroupByKey.of() .apply(kv -> KV.of(kv.key(), kv.iterator()) A runner would be free to recognize and optimize this in the graph (based

Re: Reading from BigQuery on portable runners in Python SDK

Yes this is something we wanted to do for sometime but could not prioritize due to other high priority work. JIRA is https://issues.apache.org/jira/browse/BEAM-1440. Note that BigQuery sources have many moving parts and Java BigQuery source [1] is one of the most complicated sources we have. So I

Re: New JIRA Component Request

Thanks, Pablo! On Tue, Oct 1, 2019 at 3:21 PM Pablo Estrada wrote: > I've created a runner-py-interactive component: > https://jira.apache.org/jira/issues/?jql=project+%3D+BEAM+AND+component+%3D+runner-py-interactive > > Hope that helps! > -P. > > On Tue, Oct 1, 2019 at 3:16 PM Ning Kang

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 On Tue, Oct 1, 2019 at 9:44 AM Udi Meiri wrote: > +1 > > On Tue, Oct 1, 2019 at 3:22 AM Łukasz Gajowy > wrote: > >> +1 >> >> wt., 1 paź 2019 o 11:29 Maximilian Michels napisał(a): >> >>> +1 >>> >>> On 30.09.19 23:03, Reza Rokni wrote: >>> > +1 >>> > >>> > On Tue, 1 Oct 2019 at 13:54,

Re: New JIRA Component Request

Thanks for the quick response Pablo! On Tue, Oct 1, 2019 at 3:21 PM Pablo Estrada wrote: > I've created a runner-py-interactive component: > https://jira.apache.org/jira/issues/?jql=project+%3D+BEAM+AND+component+%3D+runner-py-interactive > > Hope that helps! > -P. > > On Tue, Oct 1, 2019 at

Re: New JIRA Component Request

I've created a runner-py-interactive component: https://jira.apache.org/jira/issues/?jql=project+%3D+BEAM+AND+component+%3D+runner-py-interactive Hope that helps! -P. On Tue, Oct 1, 2019 at 3:16 PM Ning Kang wrote: > +1 > FYI, I'm temporarily using examples-python component. > > On Tue, Oct 1,

Re: New JIRA Component Request

+1 FYI, I'm temporarily using examples-python component. On Tue, Oct 1, 2019 at 3:04 PM Sam Rohde wrote: > Hi All, > > I am working improvements to the InteractiveRunner along side with +David > Yan , +Ning Kang , and Alexey > Strokach. I am requesting on behalf of this working group to add a

New JIRA Component Request

Hi All, I am working improvements to the InteractiveRunner along side with +David Yan , +Ning Kang , and Alexey Strokach. I am requesting on behalf of this working group to add a new Jira component "runner-interactive" as the current list of components is insufficient. Regards, Sam

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 On Tue, Oct 1, 2019, 10:40 PM Lukasz Cwik wrote: > +1 > > On Tue, Oct 1, 2019 at 10:39 AM Ning Kang wrote: > >> +1 >> >> On Tue, Oct 1, 2019 at 10:17 AM Pablo Estrada wrote: >> >>> +1 >>> >>> I guess it was http://python3statement.org : ) >>> >>> On Tue, Oct 1, 2019 at 10:14 AM Mark Liu

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 On Tue, Oct 1, 2019 at 10:39 AM Ning Kang wrote: > +1 > > On Tue, Oct 1, 2019 at 10:17 AM Pablo Estrada wrote: > >> +1 >> >> I guess it was http://python3statement.org : ) >> >> On Tue, Oct 1, 2019 at 10:14 AM Mark Liu wrote: >> >>> +1 >>> >>> btw, the link (http://python3stament.org) you

Re: Beam KinesisIO Migration V1 to V2

+dev On Tue, Oct 1, 2019 at 8:35 PM Ismaël Mejía wrote: > > Thanks a lot Cam for bringing this document to the mailing list (I let > some comments there). There was a recent proposal doc about supporting > async on Beam so you can be interested on taking a look at the > evolution of that [1]. It

Re: Dockerhub push denied for py3.6 and py3.7 image

I can push them now. Thank you Pablo! On Tue, Oct 1, 2019 at 11:05 AM Pablo Estrada wrote: > You were right that the push permissions for repository maintainers were > missing. I've just added the permissions, and you should be able to push to > them now. > Thanks Mark! > > On Tue, Oct 1, 2019

Re: Dockerhub push denied for py3.6 and py3.7 image

I'll check for you. One second. On Tue, Oct 1, 2019 at 10:32 AM Mark Liu wrote: > Hello Dockerhub Admins, > > I was able to push Java, Go, py2.7 and py3.5 images to > hub.docker.com/u/apachebeam for 2.16 release, but failed for py3.6 and > py3.7 due to "denied: requested access to the resource

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 I guess it was http://python3statement.org : ) On Tue, Oct 1, 2019 at 10:14 AM Mark Liu wrote: > +1 > > btw, the link (http://python3stament.org) you provided is broken. > > On Tue, Oct 1, 2019 at 9:44 AM Udi Meiri wrote: > >> +1 >> >> On Tue, Oct 1, 2019 at 3:22 AM Łukasz Gajowy >>

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

The correct link is https://python3statement.org/ On Tue, Oct 1, 2019 at 10:14 AM Mark Liu wrote: > > +1 > > btw, the link (http://python3stament.org) you provided is broken. > > On Tue, Oct 1, 2019 at 9:44 AM Udi Meiri wrote: >> >> +1 >> >> On Tue, Oct 1, 2019 at 3:22 AM Łukasz Gajowy wrote:

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

Thanks, sorry, there is a typo. The link is https://python3statement.org. On Tue, Oct 1, 2019 at 10:14 AM Mark Liu wrote: > +1 > > btw, the link (http://python3stament.org) you provided is broken. > > On Tue, Oct 1, 2019 at 9:44 AM Udi Meiri wrote: > >> +1 >> >> On Tue, Oct 1, 2019 at 3:22

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 btw, the link (http://python3stament.org) you provided is broken. On Tue, Oct 1, 2019 at 9:44 AM Udi Meiri wrote: > +1 > > On Tue, Oct 1, 2019 at 3:22 AM Łukasz Gajowy > wrote: > >> +1 >> >> wt., 1 paź 2019 o 11:29 Maximilian Michels napisał(a): >> >>> +1 >>> >>> On 30.09.19 23:03, Reza

Re: Introduction + Support in Comms for Beam!

Welcome Maria! : ) On Tue, Oct 1, 2019 at 8:32 AM Ahmet Altay wrote: > Welcome! > > On Tue, Oct 1, 2019 at 3:26 AM Jesse Anderson > wrote: > >> Excellent and welcome! >> >> [image: Big Data Institute] Jesse Anderson >> Managing Director >> Big Data Institute >> (775) 393 9122 |

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 On Tue, Oct 1, 2019 at 3:22 AM Łukasz Gajowy wrote: > +1 > > wt., 1 paź 2019 o 11:29 Maximilian Michels napisał(a): > >> +1 >> >> On 30.09.19 23:03, Reza Rokni wrote: >> > +1 >> > >> > On Tue, 1 Oct 2019 at 13:54, Tanay Tummalapalli > > > wrote: >> > >> > +1

Re: Reading from BigQuery on portable runners in Python SDK

+Chamikara Jayalath and +Pablo Estrada might have ideas related to this. On Tue, Oct 1, 2019 at 2:39 AM Kamil Wasilewski < kamil.wasilew...@polidea.com> wrote: > If anyone is interested, here is a link to my code: > https://github.com/kamilwu/beam/tree/bounded-source-for-bq > > On Tue, Oct 1,

Re: Introduction + Support in Comms for Beam!

Welcome! On Tue, Oct 1, 2019 at 3:26 AM Jesse Anderson wrote: > Excellent and welcome! > > [image: Big Data Institute] Jesse Anderson > Managing Director > Big Data Institute > (775) 393 9122 | je...@bigdatainstitute.io > bigdatainstitute.io > > > On Tue, Oct

Re: [ANNOUNCE] New committer: Alan Myrvold

Congratulations! Well deserved! > On 30 Sep 2019, at 19:12, Udi Meiri wrote: > > Congrats Alan! > > On Mon, Sep 30, 2019 at 11:12 AM Alan Myrvold > wrote: > Thanks!! Looking forward to making more impact to Apache Beam > > On Mon, Sep 30, 2019 at 10:56 AM Mikhail

Re: Introduction + Support in Comms for Beam!

Excellent and welcome! [image: Big Data Institute] Jesse Anderson Managing Director Big Data Institute (775) 393 9122 | je...@bigdatainstitute.io bigdatainstitute.io On Tue, Oct 1, 2019 at 10:46 AM Łukasz Gajowy wrote: > Welcome! :) > > wt., 1 paź 2019 o

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 wt., 1 paź 2019 o 11:29 Maximilian Michels napisał(a): > +1 > > On 30.09.19 23:03, Reza Rokni wrote: > > +1 > > > > On Tue, 1 Oct 2019 at 13:54, Tanay Tummalapalli > > wrote: > > > > +1 > > > > On Tue, Oct 1, 2019 at 8:19 AM Suneel Marthi >

Re: Introduction + Support in Comms for Beam!

Welcome! :) wt., 1 paź 2019 o 11:30 Maximilian Michels napisał(a): > Welcome Maria! Looking forward to your proposal. > > Cheers, > Max > > On 01.10.19 00:33, Reza Rokni wrote: > > Welcome! > > > > On Tue, 1 Oct 2019 at 11:18, Lukasz Cwik > > wrote: > > > > Welcome

Re: Reading from BigQuery on portable runners in Python SDK

If anyone is interested, here is a link to my code: https://github.com/kamilwu/beam/tree/bounded-source-for-bq On Tue, Oct 1, 2019 at 11:17 AM Kamil Wasilewski < kamil.wasilew...@polidea.com> wrote: > Hi all, > > At the moment, we have a BigQuery native source for Python SDK, which can > be used

Re: Introduction + Support in Comms for Beam!

Welcome Maria! Looking forward to your proposal. Cheers, Max On 01.10.19 00:33, Reza Rokni wrote: Welcome! On Tue, 1 Oct 2019 at 11:18, Lukasz Cwik > wrote: Welcome to the community. On Mon, Sep 30, 2019 at 3:15 PM María Cruz mailto:macruz...@gmail.com>>

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 On 30.09.19 23:03, Reza Rokni wrote: +1 On Tue, 1 Oct 2019 at 13:54, Tanay Tummalapalli > wrote: +1 On Tue, Oct 1, 2019 at 8:19 AM Suneel Marthi mailto:smar...@apache.org>> wrote: +1 On Mon, Sep 30, 2019 at 10:33 PM Manu Zhang

Reading from BigQuery on portable runners in Python SDK

Hi all, At the moment, we have a BigQuery native source for Python SDK, which can be used only by Dataflow runner. Consequently, it doesn't work on portable runners, such as Flink. Recently I have written a prototypical source which implements iobase.BoundedSource, so that other runners can read

Re: Multiple iterations after GroupByKey with SparkRunner

The car analogy was meant to say, that in real world you have to make decision before you take any action. There is no retroactivity possible. Reuven pointed out, that it is possible (although it seems a little weird to me, but that is the only thing I can tell against it :-)), that the way a

Re: Introduction + Support in Comms for Beam!

Welcome! On Tue, 1 Oct 2019 at 11:18, Lukasz Cwik wrote: > Welcome to the community. > > On Mon, Sep 30, 2019 at 3:15 PM María Cruz wrote: > >> Hi everyone, >> my name is María Cruz, I am from Buenos Aires but I live in the Bay Area. >> I recently became acquainted with Apache Beam project,

Re: [VOTE] Sign a pledge to discontinue support of Python 2 in 2020.

+1 On Tue, 1 Oct 2019 at 13:54, Tanay Tummalapalli wrote: > +1 > > On Tue, Oct 1, 2019 at 8:19 AM Suneel Marthi wrote: > >> +1 >> >> On Mon, Sep 30, 2019 at 10:33 PM Manu Zhang >> wrote: >> >>> +1 >>> >>> On Tue, Oct 1, 2019 at 9:44 AM Austin Bennett < >>> whatwouldausti...@gmail.com> wrote: