Re: [RESULT] [VOTE] Release 2.33.0, release candidate #1

2021-09-29 Thread Claire McGinty
Thanks Udi! I just tagged you in a cherrypick PR. - Claire On Wed, Sep 29, 2021 at 3:39 PM Udi Meiri wrote: > Yes, we can include this fix in RC2. I haven't started preparing it yet. > Could someone prepare a cherrypick? > > On Wed, Sep 29, 2021 at 10:10 AM Ryan Skraba wrote: > >> Just to

Flaky test issue report (33)

2021-09-29 Thread Beam Jira Bot
This is your daily summary of Beam's current flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake) These are P1 issues because they have a major negative impact on the community and make it hard to

Re: Will Apache Beam adopt a Pandas-like syntax to program in Python?

2021-09-29 Thread Brian Hulette
Hi David, Yes! Apache Beam now has a DataFrame API [1], which provides similar functionality. It exited experimental in Beam 2.32.0 [2]. You can see some example pipelines that use it here [3]. Brian [1] https://beam.apache.org/documentation/dsls/dataframes/overview/ [2]

Re: SBE Beam Schema

2021-09-29 Thread Kenneth Knowles
Thanks for the super clear summary. Makes sense! On Tue, Sep 28, 2021 at 9:57 AM Zachary Houfek wrote: > Hi, everyone, > > This is a follow-up to my original proposal for adding SBE message support > in Beam. A couple things came out in that review: > > 1. A coder-focused design for working

Will Apache Beam adopt a Pandas-like syntax to program in Python?

2021-09-29 Thread David Ciudad Gomez
Hi, Apache Spark is adopting a new Pandas-like syntax ( https://github.com/databricks/koalas) for programming in Python. Will Apache Beam adopt a similar syntax in the future? Thanks and best regards. David Ciudad

Contributor permission for Beam Jira tickets

2021-09-29 Thread David Prieto Rivera
Hi, My name is David Prieto. I work at Europcar Mobility Group as a Data Engineer. We are currently using Beam for both batch and streaming pipelines and I would like to be added as a contributor to the project to be able to assign issues to me. I created this issue a few days ago:

Re: Multi Environment Support

2021-09-29 Thread Ke Wu
Thanks for the advice. Here are some more background: We are building a feature called “split deployment” such that, we can isolate framework/platform core from user code/dependencies to address couple of operational challenges such as dependency conflict, alert/exception triaging. With

P1 issues report (46)

2021-09-29 Thread Beam Jira Bot
This is your daily summary of Beam's current P1 issues, not including flaky tests (https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake). See

P0 (outage) report

2021-09-29 Thread Beam Jira Bot
This is your daily summary of Beam's current outages. See https://beam.apache.org/contribute/jira-priorities/#p0-outage for the meaning and expectations around P0 issues. BEAM-12959: Dataflow error in CombinePerKey operation (https://issues.apache.org/jira/browse/BEAM-12959)

Re: [RESULT] [VOTE] Release 2.33.0, release candidate #1

2021-09-29 Thread Ryan Skraba
Just to follow up -- the cherry-pick of https://github.com/apache/beam/pull/15616 changes the default value of a configuration option that appears for the first time in 2.33.0. I think it's a strong argument for making the change now, before unwary developers start using the wrong default. I

Re: [RESULT] [VOTE] Release 2.33.0, release candidate #1

2021-09-29 Thread Alexey Romanenko
Is it still possible to cherry-pick this fix [1][2] since it’s a recent regression that touches pipelines in production? [1] https://github.com/apache/beam/pull/15616 [2] https://issues.apache.org/jira/browse/BEAM-12628 > On 28 Sep 2021, at 03:31, Udi Meiri wrote: > > I spoke too soon. We

Re: [BULK] Re: [EXTERNAL] [BULK] Re: Jira Permission Request

2021-09-29 Thread Alexander Zhuravlev
Thank you! Alexander Zhuravlev Flutter Developer, Akvelon Skype: hello030998 Telegram: @ftredmist Phone: +79128760565 От: Alexey Romanenko Отправлено: 29 сентября 2021 г. 16:29:11 Кому: dev@beam.apache.org Тема: Re: [BULK] Re: [EXTERNAL] [BULK] Re: Jira

Re: [BULK] Re: [EXTERNAL] [BULK] Re: Jira Permission Request

2021-09-29 Thread Alexey Romanenko
Thanks, it’s ok now. I added you as a contributor. Welcome! — Alexey > On 29 Sep 2021, at 14:19, Alexander Zhuravlev > wrote: > > I'm sorry, signed up 5 mins ago, can you try again? > > Alexander Zhuravlev > Flutter Developer, Akvelon > Skype: hello030998 > Telegram: @ftredmist > Phone:

Re: [BULK] Re: [EXTERNAL] [BULK] Re: Jira Permission Request

2021-09-29 Thread Alexander Zhuravlev
I'm sorry, signed up 5 mins ago, can you try again? Alexander Zhuravlev Flutter Developer, Akvelon Skype: hello030998 Telegram: @ftredmist Phone: +79128760565 От: Alexey Romanenko Отправлено: 29 сентября 2021 г. 16:08:22 Кому: dev@beam.apache.org Тема: [BULK]

Re: [EXTERNAL] [BULK] Re: Jira Permission Request

2021-09-29 Thread Alexey Romanenko
Hmm, I can’t find it. Did you already signed up at Apache Jira before? > On 29 Sep 2021, at 13:25, Alexander Zhuravlev > wrote: > > Jira name - Alexander Zhuravlev > > Alexander Zhuravlev > Flutter Developer, Akvelon > Skype: hello030998 > Telegram: @ftredmist > Phone: +79128760565 > От:

Re: [EXTERNAL] [BULK] Re: Jira Permission Request

2021-09-29 Thread Alexander Zhuravlev
Jira name - Alexander Zhuravlev Alexander Zhuravlev Flutter Developer, Akvelon Skype: hello030998 Telegram: @ftredmist Phone: +79128760565 От: Alexey Romanenko Отправлено: 29 сентября 2021 г. 14:59:58 Кому: dev@beam.apache.org Тема: [EXTERNAL] [BULK] Re: Jira

Re: Jira Permission Request

2021-09-29 Thread Alexey Romanenko
Hi Alexander, What is your Jira ID/name? — Alexey > On 29 Sep 2021, at 12:28, Alexander Zhuravlev > wrote: > > Hello, > My name is Alexander. I work as an engineer in Akvelon. I'll work at Beam > Playground project and contribute to Apache Beam. Can someone add me as a > contributor to

Jira Permission Request

2021-09-29 Thread Alexander Zhuravlev
Hello, My name is Alexander. I work as an engineer in Akvelon. I'll work at Beam Playground project and contribute to Apache Beam. Can someone add me as a contributor to Beam’s Jira issue tracker? I would like to create/assign tickets for my work. Alexander Zhuravlev Flutter Developer, Akvelon

Re: Documenting per-key delivery semantics for various runners

2021-09-29 Thread Jan Lukavský
Hi Pablo, thanks for the motivating examples. I understand the motivation now, but one question comes to mind - we do not mind the case, when in-between the emitting PTransform and the comsuming PTransform is another (grouping) PTransform which _changes the key_? Via the change of key, two