Re: [PROPOSAL] Preparing for 2.57.0 Release

[Ahmet Altay via dev]

Thank you Kenn!

On Wed, May 15, 2024 at 8:00 AM Kenneth Knowles  wrote:

> Hi everyone,
>
> The next release (2.57.0) branch cut is scheduled on May 29th, 2024,
> according to the release calendar [1].
>
> I volunteer to perform this release. My plan is to cut the branch on that
> date, and cherrypick release-blocking fixes afterwards, if any.
>
> Please help me make sure the release goes smoothly by:
>
>  - Making sure that any unresolved release blocking issues for 2.55.0
> should
> have their "Milestone" marked as "2.57.0 Release" as soon as possible.
>  - Reviewing the current release blockers [2] and remove the Milestone if
> they don't meet the criteria at [3].
>
> Let me know if you have any comments/objections/questions.
>
> Thanks,
>
> Kenn
>
> [1]
> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
> [2] https://github.com/apache/beam/milestone/21
> [3] https://beam.apache.org/contribute/release-blocking/
>

Re: Beam + Google Summer of Code 2024

[Ahmet Altay via dev]

Welcome! It is great to have new and returning contributors working on
exciting projects!

On Thu, May 2, 2024 at 9:35 AM Svetak Sundhar via dev 
wrote:

> Welcome and welcome (back)!
>
>
> Svetak Sundhar
>
>   Data Engineer
> s vetaksund...@google.com
>
>
>
> On Thu, May 2, 2024 at 2:06 AM Reeba Qureshi  wrote:
>
>> Hello everyone!
>>
>> I'm really excited to be working with Apache Beam again! Looking forward
>> to it!
>>
>> Thanks,
>> Reeba
>>
>> On Thu, 2 May, 2024, 10:04 Ayush Pandey,  wrote:
>>
>>> Hi Danny,
>>>
>>> Thank you for the kind introduction. I really look forward to
>>> collaborating with and learning from this amazing community.
>>>
>>>
>>> Best Regards,
>>> Ayush
>>>
>>>
>>> On Wed, 1 May 2024 at 14:40, XQ Hu  wrote:
>>>
 Welcome to Beam!

 On Wed, May 1, 2024 at 4:13 PM Danny McCormick via dev <
 dev@beam.apache.org> wrote:

> Hey everyone,
>
> It's my pleasure to announce 2 contributors have been accepted as GSoC
> students for Beam this year!
>
> Ayush Pandey will be working on a project to implement RAG example
> pipelines using Beam [1]. This will be a really valuable addition to 
> Beam's
> ML offering, showing how users can leverage things like MLTransform and
> Enrichment for interacting with LLMs. @Jack McCluskey
>  and I will be mentoring Ayush for this
> project.
>
> Reeba Qureshi will be working on adding new features to Beam Yaml,
> including onboarding new IOs and ML transforms [2]. This will help more
> fully round out our growing Yaml offering and should make low code
> pipelines even more attainable. Reeba also was a GSoC contributor last 
> year
> [3] and we're really excited to have her back! @Jeff Kinard
>  and I will be mentoring Reeba for this project.
>
> Welcome to the community Ayush, and welcome back Reeba!
>
> Thanks,
> Danny
>
> [1]
> https://docs.google.com/document/d/1M_8fvqKVBi68hQo_x1AMQ8iEkzeXTcSl0CwTH00cr80/edit#heading=h.mp9iumh7r8v
> [2]
> https://docs.google.com/document/d/1vXj1qhy0Asiosn3gFDgYVKYQs3Lsyj972klSv5_hfG8/edit
> [3] https://lists.apache.org/thread/5yb0jr41xg1xonlxr97p0o06mnk3ktbb
>

Re: Structured Logging in python

[Ahmet Altay via dev]

Is there an open github issue for this? Perhaps it would be a good project
to implement in Python, using the java version as a
reference implementation?

On Thu, Apr 11, 2024 at 12:04 PM Udi Meiri  wrote:

> Hi,
>
> I believe this wasn't implemented for Python (only Java). You can try
> adding structured data (extra keyword) under the key "custom_data" and that
> might work.
>
> On 2024/04/11 17:49:43 Valentyn Tymofieiev wrote:
> > Thanks for reaching out. There was a proposal a while back:
> > https://s.apache.org/beam-structured-logging
> >
> > /cc: @u...@apache.org - do you know the current status?
> >
> > Thanks a lot!
> >
> > On Thu, Apr 11, 2024 at 8:29 AM Geddy Schellevis <
> geddyschelle...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I would like to know if it is possible to have structured logging in
> > > Dataflow.
> > > In the attached file, you can find the code that I am trying to do.
> > >
> > > I see the logs are appearing in gcp log explorer, but I cannot see the
> > > extra fields.
> > >
> > > Best regards,
> > >
> >
>

Re: 2.56.0 release

[Ahmet Altay via dev]

Thank you Danny!

On Fri, Apr 5, 2024 at 3:20 AM Maciej Szwaja via dev 
wrote:

> Hi,
>
> a nice round version number :)
>
> Cheers,
> Maciej
>
> On Thu, Apr 4, 2024 at 3:56 PM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> Hey everyone,
>>
>> On top of running the current 2.55.1 patch release, I'd like to volunteer
>> as release manager for the 2.56.0 release, which will be cut on April 17.
>> Please add any release blockers to the 2.56.0 milestone [1], I will be
>> trying to push on those as the cut date approaches.
>>
>> Thanks,
>> Danny
>>
>> [1] https://github.com/apache/beam/milestone/20
>>
>

Re: [Important] GSoC 2024 Project Ideas

[Ahmet Altay via dev]

Friendly reminder: Feb 3rd is the latest day to record GSoC 2024 project
ideas. There are no Beam projects in the list at the moment.

If you have an idea and you need help, please feel free to ask your
questions here.

On Fri, Jan 26, 2024 at 12:12 PM Ahmet Altay  wrote:

> Hi all!
>
> GSoC is a good opportunity to get new community members and a
> good opportunity for you to mentor junior folks. If you have any ideas
> please record them (https://s.apache.org/gsoc2024ideas) *by 3rd Feb*.
>
> As for some ideas, these are good opportunities for (i) some low hanging
> but low priority projects: usability, infra or website improvements, some
> new IOs or IO related features (ii) good for trying prototyping out stuff
> that we have not spend much time yet: ray on beam (iii) long awaiting
> projects that are well known but not prioritized: fixing known runner
> issues, running some benchmarks.
>
> Feel free to share this information with other relevant folks.
>
> Ahmet
>
> -- Forwarded message -
> From: Priya Sharma 
> Date: Thu, Jan 25, 2024 at 4:21 AM
> Subject: [Important] GSoC 2024 Project Ideas
> To: , Swapnil M Mane ,
> Sanyam Goel , Maxim Solodovnik 
>
>
> Hello PMCs,
>
> Google Summer of Code is the ideal opportunity for you to attract new
> contributors to your projects and GSoC 2024 is here.
>
> The ASF will be applying as a participating organization for GSoC 2024.
> As a part of the application we need you all to *mandatorily* start
> recording your ideas now [1] latest by 3rd Feb.
>
> There is slight change in the rules this year, just reiterating here:
> - For the 2024 program, there will be three options for project scope:
> medium at ~175 hours, large at ~350 hours and a new size: small at ~90
> hours.
>   Please add "*full-time*" label to the JIRA for 350 hour project ,
> "*part-time*" label for 175 hours project and “*small*” for a 90 hour
> project.
>
> Note: They are looking to bring more open source projects in the AI/ML
> field into GSoC 2024, so we encourage more projects from this domain
> to participate.
>
> If you are a new mentor or your project is participating for the first
> time, please read [2][3].
>
> On behalf of the GSoC 2024 admins,
> Please feel free to reach out to us in case of queries or concerns.
>
> [1] https://s.apache.org/gsoc2024ideas
> [2] https://community.apache.org/gsoc.html
> [3] https://community.apache.org/guide-to-being-a-mentor.html
>

Fwd: [Important] GSoC 2024 Project Ideas

[Ahmet Altay via dev]

Hi all!

GSoC is a good opportunity to get new community members and a
good opportunity for you to mentor junior folks. If you have any ideas
please record them (https://s.apache.org/gsoc2024ideas) *by 3rd Feb*.

As for some ideas, these are good opportunities for (i) some low hanging
but low priority projects: usability, infra or website improvements, some
new IOs or IO related features (ii) good for trying prototyping out stuff
that we have not spend much time yet: ray on beam (iii) long awaiting
projects that are well known but not prioritized: fixing known runner
issues, running some benchmarks.

Feel free to share this information with other relevant folks.

Ahmet

-- Forwarded message -
From: Priya Sharma 
Date: Thu, Jan 25, 2024 at 4:21 AM
Subject: [Important] GSoC 2024 Project Ideas
To: , Swapnil M Mane ,
Sanyam Goel , Maxim Solodovnik 


Hello PMCs,

Google Summer of Code is the ideal opportunity for you to attract new
contributors to your projects and GSoC 2024 is here.

The ASF will be applying as a participating organization for GSoC 2024.
As a part of the application we need you all to *mandatorily* start
recording your ideas now [1] latest by 3rd Feb.

There is slight change in the rules this year, just reiterating here:
- For the 2024 program, there will be three options for project scope:
medium at ~175 hours, large at ~350 hours and a new size: small at ~90
hours.
  Please add "*full-time*" label to the JIRA for 350 hour project ,
"*part-time*" label for 175 hours project and “*small*” for a 90 hour
project.

Note: They are looking to bring more open source projects in the AI/ML
field into GSoC 2024, so we encourage more projects from this domain
to participate.

If you are a new mentor or your project is participating for the first
time, please read [2][3].

On behalf of the GSoC 2024 admins,
Please feel free to reach out to us in case of queries or concerns.

[1] https://s.apache.org/gsoc2024ideas
[2] https://community.apache.org/gsoc.html
[3] https://community.apache.org/guide-to-being-a-mentor.html

Re: [ANNOUNCE] New Committer: Sam Whittle

[Ahmet Altay via dev]

Congratulations Sam!

On Mon, Oct 16, 2023 at 10:42 AM Byron Ellis via dev 
wrote:

> Congrats Sam!
>
> On Mon, Oct 16, 2023 at 10:32 AM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Sam!
>>
>> On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Sam Whittle (scwhit...@apache.org).
>>>
>>> Sam has been contributing to Beam since 2016! In particular, he
>>> specializes in streaming and the Dataflow Java worker but his contributions
>>> expand naturally from there to the Java SDK, IOs, and even a bit of Python
>>> :-). Sam has contributed a ton of code over the years and is generous in
>>> code review and sharing his expertise.
>>>
>>> Considering his contributions to the project over this timeframe, the
>>> Beam PMC trusts Sam with the responsibilities of a Beam committer. [1]
>>>
>>> Thank you Sam! And we are looking to see more of your contributions!
>>>
>>> Kenn, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>>
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>

Re: [ANNOUNCE] New Committer: Byron Ellis

[Ahmet Altay via dev]

Congratulations Byron!

On Mon, Oct 16, 2023 at 10:35 AM Tomo Suzuki via dev 
wrote:

> Congratulations!
>
>
> On Mon, Oct 16, 2023 at 1:33 PM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Byron!
>>
>> On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Byron Ellis (b...@apache.org).
>>>
>>> Byron has been with Beam for over a year now. You may all know him as
>>> the guy who just decided to write a Swift SDK :-). In addition to that big
>>> contribution Byron has also fixed plenty of bugs, prototyped DBT-tyle
>>> pipeline authoring, and participated in our collective decision-making
>>> process.
>>>
>>> Considering his contributions to the project over this timeframe, the
>>> Beam PMC trusts Byron with the responsibilities of a Beam committer. [1]
>>>
>>> Thank you Byron! And we are looking to see more of your contributions!
>>>
>>> Kenn, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>>
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>
>
> --
> Regards,
> Tomo
>

Re: [VOTE] Release 2.51.0, release candidate #1

[Ahmet Altay via dev]

Thank you for the information.

I agree with Kenn in that case. This could wait for the next release.
Unless there is another reason to do the RC2.

On Tue, Oct 10, 2023 at 12:30 PM Yi Hu  wrote:

>
> Would it impact all python users including breaking the new user, quick
>> start experience? Or would it impact users of a specific IO or
>> configuration?
>>
>
> It is the latter. It will impact users of Specific IO (BigQueryIO read)
> specific configuration (Direct_Read). Note that the default configuration
> for BigQueryIO read is EXPORT. So this won't affect "quick-start" examples
> having default settings.
>
> It also won't affect users using SDK docker containers (e.g. Dataflow
> users and Flink/Spark users running on a remote cluster). It will affect
> users running in direct runner, and local portable runners (e.g. Flink
> local cluster) with LOOPBACK configuration, which is exactly what our
> Python PostComit is doing.
>
>

Re: [VOTE] Release 2.51.0, release candidate #1

[Ahmet Altay via dev]

Would it impact all python users including breaking the new user, quick
start experience? Or would it impact users of a specific IO or
configuration? If it is the former, I think it would be worth fixing it
just to have a working new user experience. With new user experience I am
thinking about someone doing a pip install beam and trying out a simple
pipeline and not thinking about typical production deployment options.

On Tue, Oct 10, 2023 at 11:03 AM Kenneth Knowles  wrote:

> After thinking this through a bit more, I am inclined to release RC1 with
> this noted as a known issue, unless there are other more compelling reasons
> to issues a second RC.
>
> Why?
>
>  - It is more-or-less by design that end users of Beam Python have
> dependencies shift under them; breakage and recovery (via pinning to known
> good versions) must be part of that design.
>  - In many contexts, users will already know this and will have pinned
> dependencies which therefore won't be impacted.
>
> So I am still working through the other failures on
> https://github.com/apache/beam/pull/28663 to confirm if they are all
> benign before closing the vote. If someone wants to actually -1 the RC they
> can do that, but I won't (yet).
>
> Kenn
>
> On Mon, Oct 9, 2023 at 4:22 PM Kenneth Knowles  wrote:
>
>> OK I can cherrypick it so they have an upgrade fix. But also we should
>> instruct users to pin their fastavro version to a good version. That is
>> probably safer and easier than upgrading Beam.
>>
>> Our containers that we build have the version pinned, right? So will this
>> also cause all the prior containers to have slow start up?
>>
>> Kenn
>>
>> On Mon, Oct 9, 2023 at 4:13 PM Yi Hu via dev  wrote:
>>
>>> Yes, and moreover, this specific issue will break the user the same way
>>> for *all* Beam versions (2.50.0, 2.49.0, etc) after Oct 3. That said
>>> the issue is not limited to Beam 2.50.0 though.
>>>
>>> On Mon, Oct 9, 2023 at 4:08 PM Kenneth Knowles  wrote:
>>>
 If we had closed the release today, this would still have broken all
 our users, correct?

 Kenn

 On Mon, Oct 9, 2023 at 3:37 PM Anand Inguva via dev <
 dev@beam.apache.org> wrote:

> There was a regression[1] on fastavro latest release 1.8.4. Fix was
> merged at https://github.com/apache/beam/pull/28896. The RC1 includes
> that version in the range for fastavro[2]. I think we need to CP
> https://github.com/apache/beam/pull/28896 to solve the fastavro
> regression.
>
> [1] https://github.com/apache/beam/issues/28811
> [2]
> https://github.com/apache/beam/blob/cd653e33b342bd09c76c2bbaca12597fec5b4a2c/sdks/python/setup.py#L245
>
>
> On Mon, Oct 9, 2023 at 3:15 PM Kenneth Knowles 
> wrote:
>
>> Ran a couple of Java pipelines "as a newb user" to make sure our
>> instructions weren't out of date. There are some errors in the 
>> instructions
>> but they don't have to do with this release.
>>
>> Re-ran mass_comment.py on https://github.com/apache/beam/pull/28663.
>> There are enough red signals there that some triage is needed. Any help
>> triaging would be appreciated.
>>
>> I'll close the vote once everything is run and examined.
>>
>> Kenn
>>
>> On Sat, Oct 7, 2023 at 9:58 AM Yi Hu via dev 
>> wrote:
>>
>>> +1 (non-binding) Tested on Java IO load tests (
>>> https://github.com/bvolpato/DataflowTemplates/tree/56d18a31c1c95e58543d7a1656bd83d7e859b482/it)
>>> BigQueryIO, TextIO, BigtableIO, SpannerIO on Dataflow legacy runner and
>>> runner v2
>>>
>>>
>>> On Fri, Oct 6, 2023 at 3:23 PM Kenneth Knowles 
>>> wrote:
>>>
 Additionally we need
 https://github.com/apache/beam/pull/28665/files in order to run
 GHA tests.

 On Fri, Oct 6, 2023 at 3:19 PM Kenneth Knowles 
 wrote:

> That PR was prior to many cherry-picks so it is not the signal we
> need. I have updated it to the tip of the release-2.51.0 branch.
>
> There were some post-commit tests involving JPMS that I believe
> need https://github.com/apache/beam/pull/28726 to pass.
>
> Kenn
>
> On Fri, Oct 6, 2023 at 2:53 PM Valentyn Tymofieiev via dev <
> dev@beam.apache.org> wrote:
>
>> > PR to run tests against release branch [12].
>>
>>  https://github.com/apache/beam/pull/28663 is closed and test
>> signal is no longer available. did all the tests pass?
>>
>> On Fri, Oct 6, 2023 at 5:32 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> +1 (binding)
>>>
>>> —
>>> Alexey
>>>
>>> > On 5 Oct 2023, at 18:38, Jean-Baptiste Onofré 
>>> wrote:
>>> >
>>> > +1 (binding)
>>> >
>>> > Thanks !
>>> > Regards
>>> > JB

Re: [VOTE] Release 2.51.0, release candidate #1

[Ahmet Altay via dev]

+1 (binding) - reviewed doc updates.

On Thu, Oct 5, 2023 at 12:28 AM Jan Lukavský  wrote:

> +1 (binding)
>
> Tested Java SDK with Flink Runner on own test-cases.
>
>  Jan
> On 10/4/23 21:10, Bruno Volpato via dev wrote:
>
> +1 (non-binding).
>
> Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates
> (Java SDK 11, Dataflow Runner using both legacy and v2).
>
> Thanks Kenn!
>
> On Wed, Oct 4, 2023 at 3:03 PM Robert Bradshaw via dev <
> dev@beam.apache.org> wrote:
>
>> +1 (binding)
>>
>> Verified artifacts and signatures and tested a simple python pipeline in
>> a fresh environment with a wheel.
>>
>> On Wed, Oct 4, 2023 at 8:05 AM Ritesh Ghorse via dev 
>> wrote:
>>
>>> +1 (non-binding) validated Go SDK quickstart and Python Streaming
>>> quickstart on Dataflow runner.
>>>
>>> Thanks!
>>>
>>> On Tue, Oct 3, 2023 at 5:40 PM XQ Hu via dev 
>>> wrote:
>>>
 +1 (non-binding). Tested the simple dataflow ML starter job with
 https://github.com/google/dataflow-ml-starter/actions/runs/6397130175/job/17364408813
 .

 On Tue, Oct 3, 2023 at 2:29 PM Danny McCormick via dev <
 dev@beam.apache.org> wrote:

> All Beam Python versions 2.50 and greater run exclusively on Dataflow
> runner v2, so we don't need to test v1 anymore. I'll delete those rows 
> from
> the spreadsheet
>
> On Tue, Oct 3, 2023 at 2:25 PM Svetak Sundhar <
> svetaksund...@google.com> wrote:
>
>> +1 Non Binding
>>
>> Tested Python Direct Runner and Dataflow Runner as well.
>>
>> On the spreadsheet, I came across "Dataflow v1 (until 2.49.0,
>> inclusive)", and do not fully understand what this means.
>>
>> Does this mean
>> (1) we shouldn't be testing on Dataflow runner v1 for releases after
>> 2.49 or
>> (2) make sure we test on runner v1 for this release?
>>
>> Thanks in advance for the clarification,
>>
>>
>>
>> Svetak Sundhar
>>
>>   Data Engineer
>> s vetaksund...@google.com
>>
>>
>>
>> On Tue, Oct 3, 2023 at 2:14 PM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> +1 (non-binding)
>>>
>>> Tested python/ML execution with
>>> https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/run_inference_huggingface.ipynb
>>> (interactive runner) and
>>> https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/automatic_model_refresh.ipynb
>>> (Dataflow runner).
>>>
>>> Thanks,
>>> Danny
>>>
>>> On Tue, Oct 3, 2023 at 1:58 PM Kenneth Knowles 
>>> wrote:
>>>
 Hi everyone,

 Please review and vote on the release candidate #1 for the version
 2.51.0, as follows:

 [ ] +1, Approve the release
 [ ] -1, Do not approve the release (please provide specific
 comments)

 Reviewers are encouraged to test their own use cases with the
 release candidate, and vote +1 if no issues are found. Only PMC member
 votes will count towards the final vote, but votes from all community
 members is encouraged and helpful for finding regressions; you can 
 either
 test your own use cases or use cases from the validation sheet [10].

 The complete staging area is available for your review, which
 includes:

- GitHub Release notes [1],
- the official Apache source release to be deployed to
dist.apache.org [2], which is signed with the key with
fingerprint  [3],
- all artifacts to be deployed to the Maven Central Repository
[4],
- source code tag "v1.2.3-RC3" [5],
- website pull request listing the release [6], the blog post
[6], and publishing the API reference manual [7].
- Java artifacts were built with Gradle GRADLE_VERSION and
OpenJDK/Oracle JDK JDK_VERSION.
- Python artifacts are deployed along with the source release
to the dist.apache.org [2] and PyPI[8].
- Go artifacts and documentation are available at pkg.go.dev [9]
- Validation sheet with a tab for 1.2.3 release to help with
validation [10].
- Docker images published to Docker Hub [11].
- PR to run tests against release branch [12].

 The vote will be open for at least 72 hours. It is adopted by
 majority approval, with at least 3 PMC affirmative votes.

 For guidelines on how to try the release in your projects, check
 out our blog post at
 https://beam.apache.org/blog/validate-beam-release/.

 Thanks,
 Kenn

 [1] https://github.com/apache/beam/milestone/15
 [2] https://dist.apache.org/repos/dist/dev/beam/2.51.0
 [3] https://dist.apache.org/repos/dist/release/beam/KEYS

Re: [ANNOUNCE] New PMC Member: Alex Van Boxel

[Ahmet Altay via dev]

Congratulations Alex! Well deserved!

On Tue, Oct 3, 2023 at 11:54 AM Ritesh Ghorse via dev 
wrote:

> Congratulations Alex!
>
> On Tue, Oct 3, 2023 at 2:54 PM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Alex, this is well deserved!
>>
>> On Tue, Oct 3, 2023 at 2:50 PM Jack McCluskey via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congrats, Alex!
>>>
>>> On Tue, Oct 3, 2023 at 2:49 PM XQ Hu via dev 
>>> wrote:
>>>
 Configurations, Alex!

 On Tue, Oct 3, 2023 at 2:40 PM Kenneth Knowles  wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming Alex Van
> Boxel  as our newest PMC member.
>
> Alex has been with Beam since 2016, very early in the life of the
> project. Alex has contributed code, design ideas, and perhaps most
> importantly been a huge part of organizing Beam Summits, and of course
> presenting at them as well. Alex really brings the ASF community spirit to
> Beam.
>
> Congratulations Alex and thanks for being a part of Apache Beam!
>
> Kenn, on behalf of the Beam PMC (which now includes Alex)
>

Re: [ANNOUNCE] New PMC Member: Valentyn Tymofieiev

[Ahmet Altay via dev]

Congratulations Valentyn! Well deserved!

On Tue, Oct 3, 2023 at 11:54 AM Ritesh Ghorse via dev 
wrote:

> Congratulations Valentyn!
>
> On Tue, Oct 3, 2023 at 2:53 PM Jack McCluskey via dev 
> wrote:
>
>> Congrats Valentyn!
>>
>> On Tue, Oct 3, 2023 at 2:40 PM Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming Valentyn
>>> Tymofieiev  as our newest PMC member.
>>>
>>> Valentyn has been contributing to Beam since 2017. Notable highlights
>>> include his work on the Python SDK and also in our container management.
>>> Valentyn also is involved in many discussions around Beam's infrastructure
>>> and community processes. If you look through Valentyn's history, you will
>>> see an abundance of the most critical maintenance work that is the beating
>>> heart of any project.
>>>
>>> Congratulations Valentyn and thanks for being a part of Apache Beam!
>>>
>>> Kenn, on behalf of the Beam PMC (which now includes Valentyn)
>>>
>>

Re: [ANNOUNCE] New PMC Member: Robert Burke

[Ahmet Altay via dev]

Congratulations Robert! Well deserved!

On Tue, Oct 3, 2023 at 11:54 AM Ritesh Ghorse via dev 
wrote:

> Congratulations Robert!
>
> On Tue, Oct 3, 2023 at 2:52 PM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Robert, this is very well deserved!
>>
>> On Tue, Oct 3, 2023 at 2:50 PM Anand Inguva via dev 
>> wrote:
>>
>>> Congratulations!!
>>>
>>> On Tue, Oct 3, 2023 at 2:49 PM XQ Hu via dev 
>>> wrote:
>>>
 Congratulations, Robert!

 On Tue, Oct 3, 2023 at 2:40 PM Kenneth Knowles  wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming Robert Burke <
> lostl...@apache.org> as our newest PMC member.
>
> Robert has been a part of the Beam community since 2017. He is our
> resident Gopher, producing the Go SDK and most recently the local,
> portable, Prism runner. Robert has presented on Beam many times, having
> written not just core Beam code but quite interesting pipelines too :-)
>
> Congratulations Robert and thanks for being a part of Apache Beam!
>
> Kenn, on behalf of the Beam PMC (which now includes Robert)
>

Re: Beam High Priority Issue Report (43)

[Ahmet Altay via dev]

Do people find these emails useful? It does not look like we are acting on
these? Could we consider turning them off or finding a way to make those
more useful (e.g. just list issues created/changed in th last week, or
auto-downgrade to P2 after N month etc.)

And this section "P1 Issues with no update in the last week:" seems to be
wrong. Most of the listed issues did not get any updates in the last week,
not just one.


On Mon, Oct 2, 2023 at 3:03 AM  wrote:

> This is your daily summary of Beam's current high priority issues that may
> need attention.
>
> See https://beam.apache.org/contribute/issue-priorities for the
> meaning and expectations around issue priorities.
>
> Unassigned P1 Issues:
>
> https://github.com/apache/beam/issues/28760 [Bug]: EFO Kinesis IO reader
> provided by apache beam does not pick the event time for watermarking
> https://github.com/apache/beam/issues/28703 [Failing Test]: Building a
> wheel for integration tests sometimes times out
> https://github.com/apache/beam/issues/28383 [Failing Test]:
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest.testMaxThreadMetric
> https://github.com/apache/beam/issues/28339 Fix failing
> "beam_PostCommit_XVR_GoUsingJava_Dataflow" job
> https://github.com/apache/beam/issues/28326 Bug:
> apache_beam.io.gcp.pubsublite.ReadFromPubSubLite not working
> https://github.com/apache/beam/issues/28142 [Bug]: [Go SDK] Memory seems
> to be leaking on 2.49.0 with Dataflow
> https://github.com/apache/beam/issues/27892 [Bug]: ignoreUnknownValues
> not working when using CreateDisposition.CREATE_IF_NEEDED
> https://github.com/apache/beam/issues/27648 [Bug]: Python SDFs (e.g.
> PeriodicImpulse) running in Flink and polling using tracker.defer_remainder
> have checkpoint size growing indefinitely
> https://github.com/apache/beam/issues/27616 [Bug]: Unable to use
> applyRowMutations() in bigquery IO apache beam java
> https://github.com/apache/beam/issues/27486 [Bug]: Read from datastore
> with inequality filters
> https://github.com/apache/beam/issues/27314 [Failing Test]:
> bigquery.StorageApiSinkCreateIfNeededIT.testCreateManyTables[1]
> https://github.com/apache/beam/issues/27238 [Bug]: Window trigger has lag
> when using Kafka and GroupByKey on Dataflow Runner
> https://github.com/apache/beam/issues/26981 [Bug]: Getting an error
> related to SchemaCoder after upgrading to 2.48
> https://github.com/apache/beam/issues/26969 [Failing Test]: Python
> PostCommit is failing due to exceeded rate limits
> https://github.com/apache/beam/issues/26911 [Bug]: UNNEST ARRAY with a
> nested ROW (described below)
> https://github.com/apache/beam/issues/26354 [Bug]: BigQueryIO direct read
> not reading all rows when set --setEnableBundling=true
> https://github.com/apache/beam/issues/26343 [Bug]:
> apache_beam.io.gcp.bigquery_read_it_test.ReadAllBQTests.test_read_queries
> is flaky
> https://github.com/apache/beam/issues/26329 [Bug]: BigQuerySourceBase
> does not propagate a Coder to AvroSource
> https://github.com/apache/beam/issues/26041 [Bug]: Unable to create
> exactly-once Flink pipeline with stream source and file sink
> https://github.com/apache/beam/issues/25975 [Bug]: Reducing parallelism
> in FlinkRunner leads to a data loss
> https://github.com/apache/beam/issues/24776 [Bug]: Race condition in
> Python SDK Harness ProcessBundleProgress
> https://github.com/apache/beam/issues/24389 [Failing Test]:
> HadoopFormatIOElasticTest.classMethod ExceptionInInitializerError
> ContainerFetchException
> https://github.com/apache/beam/issues/24313 [Flaky]:
> apache_beam/runners/portability/portable_runner_test.py::PortableRunnerTestWithSubprocesses::test_pardo_state_with_custom_key_coder
> https://github.com/apache/beam/issues/23944  beam_PreCommit_Python_Cron
> regularily failing - test_pardo_large_input flaky
> https://github.com/apache/beam/issues/23709 [Flake]: Spark batch flakes
> in ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and
> ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
> https://github.com/apache/beam/issues/23525 [Bug]: Default PubsubMessage
> coder will drop message id and orderingKey
> https://github.com/apache/beam/issues/22913 [Bug]:
> beam_PostCommit_Java_ValidatesRunner_Flink is flakes in
> org.apache.beam.sdk.transforms.GroupByKeyTest$BasicTests.testAfterProcessingTimeContinuationTriggerUsingState
> https://github.com/apache/beam/issues/22605 [Bug]: Beam Python failure
> for
> dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest.test_metrics_it
> https://github.com/apache/beam/issues/21714
> PulsarIOTest.testReadFromSimpleTopic is very flaky
> https://github.com/apache/beam/issues/21708
> beam_PostCommit_Java_DataflowV2, testBigQueryStorageWrite30MProto failing
> consistently
> https://github.com/apache/beam/issues/21706 Flaky timeout in github
> Python unit test action
> StatefulDoFnOnDirectRunnerTest.test_dynamic_timer_clear_then_set_timer
> 

Re: Beam ML Use Cases - Google Summer of Code 2023

[Ahmet Altay via dev]

Thank you for your hard work and writing this Reeba!

@Danielle Syse  - could we please share this on Beam's
social channels?

On Wed, Sep 13, 2023 at 10:54 AM Danny McCormick via dev <
dev@beam.apache.org> wrote:

> Thanks for all your hard work this summer Reeba! I've really enjoyed
> getting to work closely with you, and I know that Beam and its users are
> better off because of your contributions.
>
> Thanks,
> Danny
>
> On Wed, Sep 13, 2023 at 1:01 PM XQ Hu via dev  wrote:
>
>> The blog looks great! Thanks for doing this and I hope you have learned a
>> lot! Thanks a lot to Danny for your support!
>>
>> On Wed, Sep 13, 2023 at 12:58 PM Reeba Qureshi 
>> wrote:
>>
>>> Hi everyone
>>>
>>> I have completed Google Summer of Code 2023 with Apache Beam, where I
>>> worked on developing real-world ML use cases using Beam. Thank you Danny
>>> for your constant support! I wrote a blog summarizing my journey, available
>>> here
>>> 
>>> .
>>>
>>> Here are the use cases I built during the summer:
>>> 1. Batch Image Processing | GitHub
>>> 
>>> 2. Streaming Sentiment Analysis | GitHub
>>> 
>>> 3. Batch Speech Emotion Recognition | GitHub
>>> 
>>>
>>> I had a great experience and look forward to contributing more.
>>>
>>> Thanks,
>>> Reeba
>>>
>>

Re: Disabling Jenkins Jobs

[Ahmet Altay via dev]

Thank you for doing this.

Is there a list of jobs that will be disabled? I am particularly curious
about: website publishing job (which I need to use manually sometimes) and
the job that publishes daily staging builds (which we share with users
sometimes.)

Thank you.
Ahmet

On Tue, Sep 12, 2023 at 11:14 AM Danny McCormick via dev <
dev@beam.apache.org> wrote:

> Hey everyone, I wanted to let you know that as part of the migration from
> Jenkins to GitHub Actions we are going to start disabling Jenkins jobs if
> they have a corresponding GitHub Actions job that has been running
> successfully for a while. We are starting with Yi's PR here -
> https://github.com/apache/beam/pull/28316. This is the next step in the
> process we kicked off last year [1] now that self-hosted runners have been
> in place and working for a while [2].
>
> We will not migrate jobs until we've confirmed we have parity with the
> existing Jenkins implementations (for example, some jobs are still missing
> test publishing and we won't remove the Jenkins version until they have
> it). In the meantime, migrating some load off should help reduce the
> overall load on Jenkins so that it experiences fewer issues.
>
> If you have any objections with this approach, please respond here. If you
> run into any problems, please file an issue and tag me (@damccorm), Yi
> (@abacn), Andrey (@andreydevyatkin), or Vlado (@volatilemolotov) - or just
> tag all of us :).
>
> Thanks,
> Danny
>
> [1] https://lists.apache.org/thread/0brbkmbd522d1ow43gx5b13dmywt2dgn
> [2] - https://lists.apache.org/thread/3k1owt5k16byv39b9lszd3l7qv7od4r8
>

Re: [VOTE] Release 2.50.0, release candidate #2

[Ahmet Altay via dev]

+1 (binding).

I validated python quick starts on direct and dataflow runners. Thank you
for working on the release!

On Mon, Aug 28, 2023 at 8:48 AM Robert Burke  wrote:

> Good morning!
>
> RC2 validation and vote is still open!
>
> On Sun, Aug 27, 2023, 1:28 PM XQ Hu via dev  wrote:
>
>> +1
>> Ran the simple Dataflow ML GPU batch job using
>> https://github.com/google/dataflow-ml-starter with Python 2.50.0rc2 to
>> validate the RC works well.
>>
>> On Sat, Aug 26, 2023 at 12:16 AM Valentyn Tymofieiev via dev <
>> dev@beam.apache.org> wrote:
>>
>>> +1
>>>
>>> Verified that the issue detected in RC0 has been resolved. Successfully
>>> ran a Python pipeline on ARM Dataflow workers.
>>>
>>> Noted that Dataflow runner logs became less verbose as the result of
>>> https://github.com/apache/beam/pull/27788. One line that I often pay
>>> attention to no longer appears at the default  INFO log level:
>>>
>>> ```
>>> INFO:apache_beam.runners.dataflow.dataflow_runner:2023-08-26T03:45:35.126Z:
>>> JOB_MESSAGE_DETAILED: All workers have finished the startup processes and
>>> began to receive work requests.
>>> ```
>>>
>>> Dataflow service can be adjusted to compensate for this (internal
>>> change: http://cl/560265419 ).
>>>
>>> On Fri, Aug 25, 2023 at 3:05 PM Bruno Volpato via dev <
>>> dev@beam.apache.org> wrote:
>>>
 +1 (non-binding).

 Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates
 (Java SDK 11, Dataflow runner).

 Thanks Robert!

 On Thu, Aug 24, 2023 at 7:12 PM Robert Burke 
 wrote:

> Two minor erata from the previous email:
>
> The validation spreadsheet link should be:
>
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1014811464
>
> And the source code tag is: "v2.50.0-RC2"
>
> On 2023/08/24 23:09:23 Robert Burke wrote:
> > Hi everyone,
> > Please review and vote on the release candidate #2 for the version
> 2.50.0,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > Reviewers are encouraged to test their own use cases with the release
> > candidate, and vote +1 if
> > no issues are found. Only PMC member votes will count towards the
> final
> > vote, but votes from all
> > community members is encouraged and helpful for finding regressions;
> you
> > can either test your own
> > use cases or use cases from the validation sheet [10].
> >
> > Issues noted in RC1 vote proposal [13] have now been resolved.
> >
> > The staging area is available for your review, which includes:
> > * GitHub Release notes [1],
> > * the official Apache source release to be deployed to
> dist.apache.org [2],
> > which is signed with the key with fingerprint 02677FF4371A3756 (
> > lostl...@apache.org) or D20316F712213422
> > (GitHub Action automated) [[3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "v2.50.0-RC2" [5],
> > * website pull request listing the release [6], the blog post [6],
> and
> > publishing the API reference manual [7].
> > * Java artifacts were built with Gradle 7.5.1 and OpenJDK
> (Temurin)(build
> > 1.8.0_382-b05).
> > * Python artifacts are deployed along with the source release to the
> > dist.apache.org [2] and PyPI[8].
> > * Go artifacts and documentation are available at pkg.go.dev [9]
> > * Validation sheet with a tab for 2.50.0 release to help with
> validation
> > [10].
> > * Docker images published to Docker Hub [11].
> > * PR to run tests against release branch [12].
> >
> > The vote will be open for at least 72 hours. It is adopted by
> majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > For guidelines on how to try the release in your projects, check out
> our
> > blog post at https://beam.apache.org/blog/validate-beam-release/.
> >
> > Thanks,
> > Robert Burke
> > Apache Beam 2.50.0 Release Manager
> >
> > [1] https://github.com/apache/beam/milestone/14
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.50.0/
> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1355/
> > [5] https://github.com/apache/beam/tree/v2.50.0-RC2
> > [6] https://github.com/apache/beam/pull/28055
> > [7] https://github.com/apache/beam-site/pull/648
> > [8] https://pypi.org/project/apache-beam/2.50.0rc2/
> > [9]
> >
> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.50.0-RC2/go/pkg/beam
> > [10]
> >
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1014811464
> > [11] https://hub.docker.com/search?q=apache%2Fbeam=image

Re: [Request for Feedback] Mechanism for "Beam Website Feedback"

[Ahmet Altay via dev]

Thank you XQ!

For reference, responses are going to this spreadsheet [1] for now. We can
set up an email trigger for that if people here think it would be useful.

[1]
https://docs.google.com/spreadsheets/d/1bvPFLgMamtYXiUULjx3JY-b1SB-voUqaj6o1UMqTgX4/edit?usp=sharing
-- not shared by default, you need to ask for access.

On Thu, Aug 24, 2023 at 6:38 AM XQ Hu via dev  wrote:

> Hi all,
>
> Following what Ahmet proposed here (
> https://lists.apache.org/thread/9l18gs7vgqz6g1cfc2m66rbkrcnr9qmj), we
> changed the "SEED FEEDBACK" button to open the Google form (
> https://docs.google.com/forms/d/e/1FAIpQLSfID7abne3GE6k6RdJIyZhPz2Gef7UkpggUEhTIDjjplHuxSA/viewform?usp=sharing=0-Qq8aUWTrw1trYbivzEKZ0Q).
> We would like to get any feedback to improve this form.
>
> Thanks a lot!
>
> Best,
> XQ
>

[ANNOUNCE] New committer: Ahmed Abualsaud

[Ahmet Altay via dev]

Hi all,

Please join me and the rest of the Beam PMC in welcoming a new
committer: Ahmed Abualsaud (ahmedabuals...@apache.org).

Ahmed has been part of the Beam community since January 2022, working
mostly on IO connectors, made a large amount of contributions to make Beam
IOs more usable, performant, and reliable. And at the same time Ahmed was
active in the user list and at the Beam summit helping users by sharing his
knowledge.

Considering their contributions to the project over this timeframe, the
Beam PMC trusts Ahmed with the responsibilities of a Beam committer. [1]

Thank you Ahmed! And we are looking to see more of your contributions!

Ahmet, on behalf of the Apache Beam PMC

[1]
https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

Re: Beam Website Feedback

[Ahmet Altay via dev]

Explicitly adding Jonas in case he is not subscribed to the dev list.

On Thu, Aug 24, 2023 at 5:50 AM Kerry Donny-Clark via dev <
dev@beam.apache.org> wrote:

> Thanks Jonas. Can you please submit a quick PR to update the text? I'm
> happy to review.
> Kerry
>
> On Wed, Aug 23, 2023, 10:03 PM Jonas Eyob  wrote:
>
>> Hi on this page:
>> https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api
>>
>>
>>
>> Under ”Using Storage Read API” there is a paragraph and example showing
>> how Beam SDK for Python can use the Bigquery Storage API.
>>
>> But in code snippet box just below it says *“The SDK for Python does not
>> support the BigQuery Storage API”*
>>
>>
>>
>> As it seems supported perhaps this needs updating.
>>
>>
>>
>> Cheers,
>>
>> Jonas
>> --
>> ATTENTION: This e-mail may contain confidential information that is
>> intended solely for the addressee. If you are not the intended recipient,
>> you should delete this message and are hereby notified that any disclosure,
>> copying, or distribution of this message, or the taking of any action based
>> on it, is strictly prohibited.
>> --
>> PRIVACY NOTICE: Your privacy is important for us at ICA Gruppen AB and
>> its subsidiaries (ICA). We are transparent with how we collect and process
>> any personal data that you share with us. More detailed information on how
>> we process your personal data can be found at www.ica.se/dataskydd.
>>
>

Re: Mechanism for "Beam Website Feedback"

[Ahmet Altay via dev]

On Mon, Aug 14, 2023 at 1:48 PM Svetak Sundhar 
wrote:

> Hi Ahmet,
>
> I'm +1 on the idea-- one clarification question:
>
> Do you propose that when feedback is sent, it gets forwarded to the dev
> list? If not, we will need to ensure that the backend (eg a Google sheet)
> is monitored.
>

I imagine we can convert it to an email. IIRC this was possible as a native
feature of Google sheets, but if not we can use some other form -> email
product.


>
>
>
> Svetak Sundhar
>
>   Data Engineer
> s vetaksund...@google.com
>
>
>
> On Mon, Aug 14, 2023 at 4:29 PM Ahmet Altay via dev 
> wrote:
>
>> Hi all,
>>
>> We regularly get emails with "Subject: Beam Website Feedback", they are
>> filtered out before they reach this mailing list. I believe the reason for
>> that is people have some feedback to share, but clicking the feedback will
>> open the default email application and they will be surprised and close it.
>> Sometimes they close it by sending us an empty email.
>>
>> My proposal -  We can change the feedback button with a simple embedded
>> form (a text box + submit button). We can use something like google forms
>> to implement this without making a more complex backend change.
>>
>> For reference, this is a partial and it is implemented here [1].
>>
>> What do you think?
>>
>> Ahmet
>>
>> [1]
>> https://github.com/apache/beam/blob/bbaa7ebd3eec614832d76cfc577858638a96a11d/website/www/site/layouts/partials/feedback.html#L21
>>
>

Mechanism for "Beam Website Feedback"

[Ahmet Altay via dev]

Hi all,

We regularly get emails with "Subject: Beam Website Feedback", they are
filtered out before they reach this mailing list. I believe the reason for
that is people have some feedback to share, but clicking the feedback will
open the default email application and they will be surprised and close it.
Sometimes they close it by sending us an empty email.

My proposal -  We can change the feedback button with a simple embedded
form (a text box + submit button). We can use something like google forms
to implement this without making a more complex backend change.

For reference, this is a partial and it is implemented here [1].

What do you think?

Ahmet

[1]
https://github.com/apache/beam/blob/bbaa7ebd3eec614832d76cfc577858638a96a11d/website/www/site/layouts/partials/feedback.html#L21

Re: [ANNOUNCE] Transform Service

[Ahmet Altay via dev]

Congratulations! This is a great usability improvement, lowering the bar
for using multi language features.

On Thu, Aug 10, 2023 at 3:48 PM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> Hi All,
>
> We recently added a Docker Compose based service named Transform Service
> to Beam.
>
> Transform service includes a number of transforms released with Beam and
> provides a single endpoint for accessing them via the Beam's multi-language
> pipelines framework.
>
> I've updated Beam Java/Python SDKs to automatically use this service to
> expand cross-language transforms used by multi-lang pipelines
> when possible. This means that Beam pipelines can use cross-language
> transforms without installing other language runtimes if they have Docker
> (and Docker Compose which comes with Docker) available locally at job
> submission. Go SDK updates are in development.
>
> Users also have the option to manually startup a Transform Service with
> utilities provided with Beam SDKs if needed.
>
> For mode details regarding the Transform Service please see the
> documentation here
> 
> .
>
> A list of transforms currently included with the Transform Service is
> available here
> .
>
> Please see here
>  for a
> previous discussion on this and please let me know if you have any
> questions.
>
> Thanks,
> Cham
>
>

Re: [VOTE] Vendored Dependency guava 32.1.2-jre Release

[Ahmet Altay via dev]

+1 - I checked the tests on the linked testing PR. Thank you!

On Thu, Aug 3, 2023 at 12:02 PM Yi Hu via dev  wrote:

> Hi everyone,
>
>
> Please review the release of the following artifacts that we vendor:
>
>
> * beam-vendor-guava-32_1_2-jre
>
>
> Please review and vote on the release candidate 1 for the version 0.1, as
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
> * the official Apache source release to be deployed to dist.apache.org
> [1], which is signed with the key with fingerprint 170405CB [2],
> * all artifacts to be deployed to the Maven Central Repository [3],
> * commit hash "ef2ca7a" [4],
>
> * Testing PR on the vendored dependency [5]
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Release Manager
>
> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
> [2] https://dist.apache.org/repos/dist/release/beam/KEYS
> [3] https://repository.apache.org/content/repositories/orgapachebeam-1350/
> [4]
> https://github.com/apache/beam/commit/ef2ca7aa49ec75c3ab4e3a94b7ad8162e1c81c1e
> [5] https://github.com/apache/beam/pull/27825
>
> Regards,
> Yi
>
> --
>
> Yi Hu, (he/him/his)
>
> Software Engineer
>
>
>

Re: [DISCUSS] Upgrade vended guava version

[Ahmet Altay via dev]

Thanks Yi!

On Thu, Aug 3, 2023 at 7:44 AM Yi Hu  wrote:

> Hi Hong,
>
> Thanks for bringing this up. Sure, I would like to volunteer to work as
> release manager [1] to vendor guava 32.1.2-jre. Created GitHub Issue for
> tracking [2].
>
> Regards,
> Yi
>
> [1] https://s.apache.org/beam-release-vendored-artifacts
> [2] https://github.com/apache/beam/issues/27801
>
>
> On Mon, Jul 31, 2023 at 1:08 PM Ahmet Altay via dev 
> wrote:
>
>> Hi Hong,
>>
>> Thank you for reaching out and thank you for offering to help. If you can
>> start the PR and do the testing, one of the committers could help with the
>> process.
>>
>> Thank you!
>> Ahmet
>>
>> On Mon, Jul 31, 2023 at 9:13 AM Hong Teoh  wrote:
>>
>>> Hi all,
>>>
>>> The current version of guava that is vended in Beam is
>>> com.google.guava:guava:26.0-jre.
>>>
>>> This version is really old, and has active vulnerabilities [1] [2]
>>> [1] https://mvnrepository.com/artifact/com.google.guava/guava/26.0-jre
>>> [2] CVE-2023-2976
>>> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2976
>>> [3] CVE-2020-8908
>>> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908
>>>
>>> Is there anyone else keen on upgrading the vended guava version to match
>>> the guava version of 32.1.1-jre ? [4]
>>> [4]
>>> https://github.com/apache/beam/blame/df6964aac62a521081481b21c96ecd506ea3c503/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L542
>>>
>>> I am happy to contribute the PR to upgrade the guava dependencies in the
>>> Beam repository, but I would need a committer to drive the release of the
>>> vended version first! [5]
>>> [5]
>>> https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit#heading=h.vhcuqlttpnog
>>>
>>>
>>> Side question: Does anyone know why we have libraries that use the
>>> non-vended guava version? [6]
>>> [6]
>>> https://github.com/search?q=repo%3Aapache%2Fbeam%20library.java.guava=code
>>>
>>
>> @Kenneth Knowles  - might know.
>>
>>
>>>
>>>
>>>
>>> Regards,
>>> Hong
>>>
>>

Re: [DISCUSS] Upgrade vended guava version

[Ahmet Altay via dev]

Hi Hong,

Thank you for reaching out and thank you for offering to help. If you can
start the PR and do the testing, one of the committers could help with the
process.

Thank you!
Ahmet

On Mon, Jul 31, 2023 at 9:13 AM Hong Teoh  wrote:

> Hi all,
>
> The current version of guava that is vended in Beam is
> com.google.guava:guava:26.0-jre.
>
> This version is really old, and has active vulnerabilities [1] [2]
> [1] https://mvnrepository.com/artifact/com.google.guava/guava/26.0-jre
> [2] CVE-2023-2976
> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2976
> [3] CVE-2020-8908
> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908
>
> Is there anyone else keen on upgrading the vended guava version to match
> the guava version of 32.1.1-jre ? [4]
> [4]
> https://github.com/apache/beam/blame/df6964aac62a521081481b21c96ecd506ea3c503/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L542
>
> I am happy to contribute the PR to upgrade the guava dependencies in the
> Beam repository, but I would need a committer to drive the release of the
> vended version first! [5]
> [5]
> https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit#heading=h.vhcuqlttpnog
>
>
> Side question: Does anyone know why we have libraries that use the
> non-vended guava version? [6]
> [6]
> https://github.com/search?q=repo%3Aapache%2Fbeam%20library.java.guava=code
>

@Kenneth Knowles  - might know.


>
>
>
> Regards,
> Hong
>

Re: [PROPOSAL] Preparing for 2.50.0 Release

[Ahmet Altay via dev]

Thank you!

On Wed, Jul 26, 2023 at 9:57 AM Kenneth Knowles  wrote:

> Hooray!
>
> Thanks for volunteering
>
> Kenn
>
> On Wed, Jul 26, 2023 at 11:49 AM Robert Burke  wrote:
>
>> Hey Beam community,
>>
>> The next release (2.50.0) branch cut is scheduled on August 9th, 2023,
>> according to
>> the release calendar [1].
>>
>> I volunteer to perform this release. My plan is to cut the branch on that
>> date, and cherrypick release-blocking fixes afterwards, if any.
>>
>> Please help me make sure the release goes smoothly by:
>> - Making sure that any unresolved release blocking issues for 2.50.0
>> should
>> have their "Milestone" marked as "2.50.0 Release" as soon as possible.
>> - Reviewing the current release blockers [2] and remove the Milestone if
>> they don't meet the criteria at [3].
>>
>> Let me know if you have any comments/objections/questions.
>>
>> Thanks,
>>
>> Robert Burke (he/him)
>> Beam Go Busybody
>>
>> [1]
>>
>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>> [2] https://github.com/apache/beam/milestone/14
>> [3] https://beam.apache.org/contribute/release-blocking/
>>
>

Re: Why there are no certifications for Apache Beam

[Ahmet Altay via dev]

Hi Abhishek,

We recently launched a new Beam certificate. You can find more information
here:
https://beam.apache.org/get-started/resources/learning-resources/#getting-started-with-apache-beam-quest.
As part of the launch we also have a code for completing the program for
free (more info here: https://beam.apache.org/blog/beamquest/ -- the code
should be valid until Aug 18).

Hope you find this useful.

Ahmet

On Thu, Jul 27, 2023 at 12:08 PM Abhishek Patre 
wrote:

> Hello Team,
>
> I hope this email finds you well. I was wondering if you could provide
> some clarity on the availability of certification for Apache Beam. I
> apologize if this isn't the appropriate mailing list for this inquiry.
> Having a certification option for Apache Beam would be incredibly
> beneficial in terms of showcasing our skills and expertise.
>
> Thank you for your time and assistance.
>
> Regards
> Abhishek Patre
>

Re: fastavro Cython fixes

[Ahmet Altay via dev]

It looks like the fastavro issue marked as fixed now. (/cc @Valentyn
Tymofieiev  - I saw him on that issue thread.) Work on
https://github.com/apache/beam/issues/27526 is still actively ongoing.

Thank you for reaching out.


On Mon, Jul 17, 2023 at 4:19 PM Eddie Zhou  wrote:

> Hi folks,
>
> I see some active work on https://github.com/apache/beam/issues/27526 for
> Cython 3.0 fixes, but wondering if anyone knows the fastavro maintainers
> and can prioritize the fix proposed in
> https://github.com/fastavro/fastavro/issues/701
>
> Thanks!
>
> Best,
> Eddie
>

Re: [ANNOUNCE] Beam 2.49.0 Released

[Ahmet Altay via dev]

Congratulations! Thanks to the release manager and all contributors!

On Mon, Jul 17, 2023 at 8:54 AM Yi Hu via user  wrote:

> The Apache Beam Team is pleased to announce the release of version 2.49.0.
>
> You can download the release here:
>
> https://beam.apache.org/get-started/downloads/ (website daily update
> pending)
>
> This release includes bug fixes, features, and improvements detailed on the
> Beam Blog: https://beam.apache.org/blog/beam-2.49.0/ (website daily
> update pending)
> and the Github release page
> https://github.com/apache/beam/releases/tag/v2.49.0
>
> Thanks to everyone who contributed to this release, and we hope you enjoy
> using Beam 2.49.0.
>
> -- Yi, on behalf of the Apache Beam Team.
>
>
> --
>
> Yi Hu, (he/him/his)
>
> Software Engineer
>
>
>

Re: [VOTE] Release 2.49.0, release candidate #2

[Ahmet Altay via dev]

+1 (binding)

Validated python quick start on direct runner. Thank you!

On Thu, Jul 13, 2023 at 12:54 PM Svetak Sundhar via dev 
wrote:

> +1 (Non-Binding)
>
> Python quickstart Dataflow runner.
>
>
> Svetak Sundhar
>
>   Data Engineer
> s vetaksund...@google.com
>
>
>
> On Thu, Jul 13, 2023 at 5:03 AM Jan Lukavský  wrote:
>
>> +1 (binding)
>>
>> Tested Java SDK with FlinkRunner.
>>
>>  Jan
>> On 7/13/23 02:30, Bruno Volpato via dev wrote:
>>
>> +1 (non-binding).
>>
>> Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates
>> (Java SDK 11, Dataflow runner).
>>
>> Thanks Yi!
>>
>> On Tue, Jul 11, 2023 at 4:23 PM Yi Hu via dev 
>> wrote:
>>
>>> Hi everyone,
>>> Please review and vote on the release candidate #2 for the version
>>> 2.49.0, as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>>
>>> Reviewers are encouraged to test their own use cases with the release
>>> candidate, and vote +1 if
>>> no issues are found. Only PMC member votes will count towards the final
>>> vote, but votes from all
>>> community members is encouraged and helpful for finding regressions; you
>>> can either test your own
>>> use cases or use cases from the validation sheet [10].
>>>
>>> The complete staging area is available for your review, which includes:
>>> * GitHub Release notes [1],
>>> * the official Apache source release to be deployed to dist.apache.org
>>> [2], which is signed with the key with
>>> fingerprint either CB6974C8170405CB (y...@apache.org) or
>>> D20316F712213422 (GitHub Action automated) [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "v2.49.0-RC2" [5],
>>> * website pull request listing the release [6], the blog post [6], and
>>> publishing the API reference manual [7].
>>> * Java artifacts were built with Gradle GRADLE_VERSION and
>>> OpenJDK/Oracle JDK JDK_VERSION.
>>> * Python artifacts are deployed along with the source release to the
>>> dist.apache.org [2] and PyPI [8].
>>> * Go artifacts and documentation are available at pkg.go.dev [9]
>>> * Validation sheet with a tab for 2.49.0 release to help with validation
>>> [10].
>>> * Docker images published to Docker Hub [11].
>>> * PR to run tests against release branch [12].
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> For guidelines on how to try the release in your projects, check out our
>>> blog post at /blog/validate-beam-release/.
>>>
>>> Thanks,
>>> Release Manager
>>>
>>> [1] https://github.com/apache/beam/milestone/13
>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.49.0/
>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [4]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1349/
>>> [5] https://github.com/apache/beam/tree/v2.49.0-RC2
>>> [6] https://github.com/apache/beam/pull/27374 (unchanged since RC1)
>>> [7] https://github.com/apache/beam-site/pull/646  (unchanged since RC1)
>>> [8] https://pypi.org/project/apache-beam/2.49.0rc2/
>>> [9]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.49.0-RC2/go/pkg/beam
>>> [10]
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=934901728
>>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>>> [12] https://github.com/apache/beam/pull/27307
>>>
>>> --
>>>
>>> Yi Hu, (he/him/his)
>>>
>>> Software Engineer
>>>
>>>
>>>

Re: Beam Infrastructure - GitHub ARC

[Ahmet Altay via dev]

Thank you Andrey!

On Fri, Jul 14, 2023 at 8:02 AM Kenneth Knowles  wrote:

> Nice!
>
> On Thu, Jul 13, 2023 at 6:23 PM Andrey Devyatkin <
> andrey.devyat...@akvelon.com> wrote:
>
>> Hi Beam community!
>>
>>
>>
>> We are glad to announce that we have implemented GitHub Actions Runner
>> Controller. Beam GitHub previously used dynamically provisioned Self Hosted
>> runners running on GKE and they had significant stability issues. With
>> GitHub ARC there are following benefits that made it more stable:
>>
>>- Authentication is done automatically using a GitHub App credentials
>>- The HorizontalRunnerAutoscaler allows us to define the number of
>>runners and the scaling mechanism. This is implemented as a webhook
>>service, which means that when new jobs are queued, ARC will schedule new
>>runners for deployment
>>- The K8S scaling mechanism allows us to scale up and down both the
>>number of pods and nodes. This also gives us the ability to deploy 
>> multiple
>>node pools with different specs and assign runners based on labels for
>>different jobs that require special nodes
>>- Terraform scripts to easily add/remove runner types
>>
>>
>>
>> For more information about GitHub ARC:
>> https://github.com/actions/actions-runner-controller
>>
>>
>>
>> Thanks,
>>
>> Andrey, on behalf of Beam Infrastructure team
>>
>

Re: DRAFT - Apache Beam Board Report - June 2023

[Ahmet Altay via dev]

Thank you Kenn! Added a few items.

On Fri, Jun 9, 2023 at 2:28 PM Kenneth Knowles  wrote:

> Hi all,
>
> The next Beam board report is due next Wednesday, June 14. Please help me
> to draft it at https://s.apache.org/beam-draft-report-2023-06.
>
> Ideas:
>
>  - highlights from CHANGES.md
>  - interesting technical discussions
>  - integrations with other projects
>  - community events
>  - major user facing addition/deprecation
>  - stuff that will be presented at Beam Summit next week :-)
>
> Past reports are at https://whimsy.apache.org/board/minutes/Beam.html for
> examples.
>
> I will edit the final version from everyone's suggestions.
>
> Thanks,
>
> Kenn
>
>
>

Re: [Launch Announcement] Beam Quest

[Ahmet Altay via dev]

(dropping the user list)

Please also help us with spreading the word by sharing on your social media:
- Twitter: https://twitter.com/ApacheBeam/status/1666915798823084034
- Linkedin post:
https://www.linkedin.com/feed/update/urn:li:activity:7072675442012377088

On Fri, Jun 9, 2023 at 10:11 AM Ahmet Altay  wrote:

> Thank you Svetak! I would encourage everyone to try out, and get the
> badges :)
>
> On Fri, Jun 9, 2023 at 7:03 AM Svetak Sundhar via user <
> u...@beam.apache.org> wrote:
>
>> Hi Beam Community,
>>
>> We're excited to launch the "Getting Started with Apache Beam" Quest
>> <https://www.cloudskillsboost.google/quests/310>. This quest provides a
>> completion badge that can be shared on social media (such as Linkedin and
>> Twitter) upon completion of four qwiklabs.
>>
>> These labs venture into various concepts of Beam in the Java and Python
>> SDK (that many of you have developed), and should take less than 7 hours to
>> obtain. I've written about it in our Beam Blog
>> <https://beam.apache.org/blog/beamquest/>; we are offering this free of
>> charge till July 8.
>>
>> Please share the information with whomever you think may be interested,
>> and please share on social media once you obtain your badge. Additionally,
>> if you have any feedback on the labs, please contact me directly at
>> svetaksund...@google.com-- we plan to have these labs evolve over time!
>>
>> I look forward to discussing this more at Beam Summit next week.
>>
>> As this was one of GCP's first OSS quests, there were many people
>> instrumental in making this possible.
>>
>> Thanks to:
>> -Danielle Syse
>> -Ajay Hemnani
>> -Joellen Saunders
>> -Grzegorz Wierzchows
>> -Ahmet Altay
>> -XQ Hu
>> -Jenny Palomino
>> -Svetak Sundhar
>> -Shunping Huang
>>
>> Thanks,
>>
>>
>>
>> Svetak Sundhar
>>
>>   Data Engineer
>> s vetaksund...@google.com
>>
>>

Re: [Launch Announcement] Beam Quest

[Ahmet Altay via dev]

Thank you Svetak! I would encourage everyone to try out, and get the badges
:)

On Fri, Jun 9, 2023 at 7:03 AM Svetak Sundhar via user 
wrote:

> Hi Beam Community,
>
> We're excited to launch the "Getting Started with Apache Beam" Quest
> <https://www.cloudskillsboost.google/quests/310>. This quest provides a
> completion badge that can be shared on social media (such as Linkedin and
> Twitter) upon completion of four qwiklabs.
>
> These labs venture into various concepts of Beam in the Java and Python
> SDK (that many of you have developed), and should take less than 7 hours to
> obtain. I've written about it in our Beam Blog
> <https://beam.apache.org/blog/beamquest/>; we are offering this free of
> charge till July 8.
>
> Please share the information with whomever you think may be interested,
> and please share on social media once you obtain your badge. Additionally,
> if you have any feedback on the labs, please contact me directly at
> svetaksund...@google.com-- we plan to have these labs evolve over time!
>
> I look forward to discussing this more at Beam Summit next week.
>
> As this was one of GCP's first OSS quests, there were many people
> instrumental in making this possible.
>
> Thanks to:
> -Danielle Syse
> -Ajay Hemnani
> -Joellen Saunders
> -Grzegorz Wierzchows
> -Ahmet Altay
> -XQ Hu
> -Jenny Palomino
> -Svetak Sundhar
> -Shunping Huang
>
> Thanks,
>
>
>
> Svetak Sundhar
>
>   Data Engineer
> s vetaksund...@google.com
>
>

Re: Client-Side Throttling in Apache Beam

[Ahmet Altay via dev]

Thank you. +1 to adding this to wiki.

FYI - @Damon Douglas  shared a related doc earlier
for alternative ideas for Beam to prevent overloading external services. (
https://docs.google.com/document/d/1VZ9YphDO7kewBSz5oMXVPHWaib3S03Z6aZ66BhciB3E/edit?usp=sharing=0-ItxMSG72EzfSwVedSz-Zeg
)

On Tue, May 30, 2023 at 3:52 PM Robert Burke  wrote:

> Great article!
>
> Though it's depressing to see we have a pair of magic counter names to
> help modulate scaling behavior.
>
> On Tue, May 30, 2023, 11:42 AM Jack McCluskey via dev 
> wrote:
>
>> Hey everyone,
>>
>> While working on some remote model handler code I hit a point where I
>> needed to understand how Beam IOs interpret and action on being throttled
>> by an external service. This turned into a few discussions and then a small
>> write-up doc (
>> https://docs.google.com/document/d/1ePorJGZnLbNCmLD9mR7iFYOdPsyDA1rDnTpYnbdrzSU/edit?usp=sharing)
>> to encapsulate the basics of what I learned. If you're familiar with this
>> topic feel free to make suggestions on the doc, I'm intending to add this
>> to the wiki so there's a resource for how this works in the future!
>>
>> Thanks,
>>
>> Jack McCluskey
>>
>> --
>>
>>
>> Jack McCluskey
>> SWE - DataPLS PLAT/ Dataflow ML
>> RDU
>> jrmcclus...@google.com
>>
>>
>>

Re: [VOTE] Release 2.48.0 release candidate #2

[Ahmet Altay via dev]

+1 (binding)

On Tue, May 30, 2023 at 2:01 PM Ritesh Ghorse via dev 
wrote:

> Thanks Danny and Jack! Dataflow containers are up!
>
> Only PMC votes count but feel free to test your use cases and vote on this
> thread!
>
> On Tue, May 30, 2023 at 11:26 AM Alexey Romanenko <
> aromanenko@gmail.com> wrote:
>
>> +1 (binding)
>>
>> Tested with  https://github.com/Talend/beam-samples/
>> (Java SDK v8/v11/v17, Spark 3.x runner).
>>
>> On 27 May 2023, at 19:38, Bruno Volpato via dev 
>> wrote:
>>
>> I was able to check that containers are all there and complete
>> my validation.
>>
>> +1 (non-binding).
>>
>> Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates (Java
>> SDK 11, Dataflow runner).
>>
>>
>> Thanks Ritesh and Danny!
>>
>> On Fri, May 26, 2023 at 10:09 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> It looks like some Dataflow containers didn't get published, so some
>>> jobs using the legacy runner (runner v2 disabled) will fail. I kicked off
>>> the container release, so that should hopefully be available later today.
>>>
>>> Thanks,
>>> Danny
>>>
>>> On Thu, May 25, 2023 at 11:19 PM Ritesh Ghorse via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Hi everyone,
 Please review and vote on the release candidate #2 for the version
 2.48.0, as follows:
 [ ] +1, Approve the release
 [ ] -1, Do not approve the release (please provide specific comments)


 Reviewers are encouraged to test their own use cases with the release
 candidate, and vote +1 if no issues are found. Only PMC member votes will
 count towards the final vote, but votes from all community members are
 encouraged and helpful for finding regressions; you can either test your
 own use cases or use cases from the validation sheet [10].

 The complete staging area is available for your review, which includes:
 * GitHub Release notes [1],
 * the official Apache source release to be deployed to dist.apache.org [2],
 which is signed with the key with fingerprint
 E4C74BEC861570F5A3E44E46280A0AC32DBAE62B [3],
 * all artifacts to be deployed to the Maven Central Repository [4],
 * source code tag "v2.48.0-RC2" [5],
 * website pull request listing the release [6], the blog post [6], and
 publishing the API reference manual [7] (to be generated).
 * Java artifacts were built with Gradle 7.5.1 and OpenJDK/Oracle JDK
 8.0.322.
 * Python artifacts are deployed along with the source release to the
 dist.apache.org [2] and PyPI[8].
 * Go artifacts and documentation are available at pkg.go.dev [9]
 * Validation sheet with a tab for 2.48.0 release to help with
 validation [10].
 * Docker images published to Docker Hub [11].
 * PR to run tests against release branch [12].

 The vote will be open for at least 72 hours. It is adopted by majority
 approval, with at least 3 PMC affirmative votes.

 For guidelines on how to try the release in your projects, check out
 our blog post at /blog/validate-beam-release/.

 *NOTE: Dataflow containers for Python are not finalized yet (likely to
 happen on tuesday). I will follow up on this thread once that is done. Feel
 free to test it on other runners until then. *

 Thanks,
 Ritesh Ghorse

 [1] https://github.com/apache/beam/milestone/12
 [2] https://dist.apache.org/repos/dist/dev/beam/2.48.0/
 [3] https://dist.apache.org/repos/dist/release/beam/KEYS
 [4]
 https://repository.apache.org/content/repositories/orgapachebeam-1346/
 [5] https://github.com/apache/beam/tree/v2.48.0-RC2
 [6] https://github.com/apache/beam/pull/26903
 [7] https://github.com/apache/beam-site/pull/645
 [8] https://pypi.org/project/apache-beam/2.48.0rc2/
 [9]
 https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.48.0-RC2/go/pkg/beam
 
 [10]
 https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=458120434
 [11] https://hub.docker.com/search?q=apache%2Fbeam=image
 [12] https://github.com/apache/beam/pull/26811


>>

Re: Introducing beam.MLTransform

[Ahmet Altay via dev]

Thank you for writing this, Anand! I think this is a good way to leverage
existing libraries and also a good usability improvement.

On Wed, May 10, 2023 at 11:04 AM XQ Hu via dev  wrote:

> Agree with Danny. Thanks for writing this!
>
> On Wed, May 10, 2023 at 10:35 AM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> Thanks Anand! I left a few comments, but overall I think this is a
>> great/well constructed proposal - this is a good way to quickly bring a
>> suite of preprocessing operations to Beam.
>>
>> Thanks,
>> Danny
>>
>> On Tue, May 9, 2023 at 12:52 PM Anand Inguva via dev 
>> wrote:
>>
>>> Hi all,
>>>
>>> In Apache Beam, we plan to introduce a *beam.MLTransform* for carrying
>>> out common ML centric processing tasks.
>>>
>>> Using the tensorflow_transform as the backend, we will introduce several
>>> data processing transforms in Beam. These can be easily utilized by simply
>>> wrapping them with the beam.MLTransform. This approach not only simplifies
>>> the process but also creates a pathway for implementing a comprehensive set
>>> of ML-centric data processing transforms in Apache Beam.
>>>
>>> Please go through the design doc
>>> [1]
>>> for details and share your feedback.
>>>
>>> Thanks,
>>> Anand
>>>
>>> [1]
>>> https://docs.google.com/document/d/1rQkSm_8tseLqDQaLohtlCGqt5pvMaP0XIpPi5UD0LCQ/edit#
>>>
>>

Re: [VOTE] Release 2.47.0, release candidate #3

[Ahmet Altay via dev]

+1 (binding) - thank you for completing most of the validations on the
validation sheet!

On Fri, May 5, 2023 at 5:51 PM Anand Inguva via dev 
wrote:

> +1 (non-binding)
>
> Tested python quick start guide on Dataflow runner with Python 3.11.
>
> Thanks,
> Anand
>
> On Thu, May 4, 2023 at 10:53 PM Jack McCluskey via dev <
> dev@beam.apache.org> wrote:
>
>> Hi everyone,
>>
>> Please review and vote on the release candidate #3 for the version
>> 2.47.0, as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>> Reviewers are encouraged to test their own use cases with the release
>> candidate, and vote +1 if no issues are found. *Non-PMC members are
>> allowed and encouraged to vote. Please help validate the release for your
>> use case!*
>>
>> The complete staging area is available for your review, which includes:
>> * GitHub Release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org [2],
>> which is signed with the key with fingerprint DF3CBA4F3F4199F4 [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.47.0-RC3" [5],
>> * website pull request listing the release [6], the blog post [6], and
>> publishing the API reference manual [7].
>> * Java artifacts were built with Gradle 7.5.1 and OpenJDK/Oracle JDK
>> 8.0.322.
>> * Python artifacts are deployed along with the source release to the
>> dist.apache.org [2] and PyPI[8].
>> * Go artifacts and documentation are available at pkg.go.dev [9]
>> * Validation sheet with a tab for 2.47.0 release to help with validation
>> [10].
>> * Docker images published to Docker Hub [11].
>> * PR to run tests against release branch [12].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> The GCR copies of the FnAPI containers are rolling out now, they should
>> be out within the next 8 hours or so.
>>
>> For guidelines on how to try the release in your projects, check out our
>> blog post at /blog/validate-beam-release/.
>>
>> Thanks,
>>
>> Jack McCluskey
>>
>> [1] https://github.com/apache/beam/milestone/10
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.47.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1322/
>> [5] https://github.com/apache/beam/tree/v2.47.0-RC3
>> [6] https://github.com/apache/beam/pull/26439
>> [7] https://github.com/apache/beam-site/pull/644
>> [8] https://pypi.org/project/apache-beam/2.47.0rc3/
>> [9]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.47.0-RC3/go/pkg/beam
>> [10]
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=.
>> ..
>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>> [12] https://github.com/apache/beam/pull/26152
>>
>> --
>>
>>
>> Jack McCluskey
>> SWE - DataPLS PLAT/ Dataflow ML
>> RDU
>> jrmcclus...@google.com
>>
>>
>>

Re: [VOTE] Release 2.47.0, release candidate #2

[Ahmet Altay via dev]

https://github.com/apache/beam/issues/26521 was a severe issue. Should we
cherry pick the fix (https://github.com/apache/beam/pull/26503) and build
an RC3? (/cc @Reuven Lax  @John Casey
)

On Wed, May 3, 2023 at 2:01 PM Jack McCluskey via dev 
wrote:

> I have just now realized I didn't replace the release version or RC number
> from the template's first line in either vote email. Oops. To be clear this
> is 2.47.0 RC2
>
> On Wed, May 3, 2023 at 1:58 PM Jack McCluskey 
> wrote:
>
>> Hi everyone,
>>
>> Please review and vote on the release candidate #3 for the version 1.2.3,
>> as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>> Reviewers are encouraged to test their own use cases with the release
>> candidate, and vote +1 if no issues are found.
>>
>> The complete staging area is available for your review, which includes:
>> * GitHub Release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org [2],
>> which is signed with the key with fingerprint DF3CBA4F3F4199F4 [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.47.0-RC2" [5],
>> * website pull request listing the release [6], the blog post [6], and
>> publishing the API reference manual [7].
>> * Java artifacts were built with Gradle 7.5.1 and OpenJDK/Oracle JDK
>> 8.0.322.
>> * Python artifacts are deployed along with the source release to the
>> dist.apache.org [2] and PyPI[8].
>> * Go artifacts and documentation are available at pkg.go.dev [9]
>> * Validation sheet with a tab for 2.47.0 release to help with validation
>> [10].
>> * Docker images published to Docker Hub [11].
>> * PR to run tests against release branch [12].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> For guidelines on how to try the release in your projects, check out our
>> blog post at /blog/validate-beam-release/.
>>
>> Thanks,
>>
>> Jack McCluskey
>>
>> [1] https://github.com/apache/beam/milestone/10
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.47.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1315/
>> [5] https://github.com/apache/beam/tree/v2.47.0-RC2
>> [6] https://github.com/apache/beam/pull/26439
>> [7] https://github.com/apache/beam-site/pull/644
>> [8] https://pypi.org/project/apache-beam/2.47.0rc2/
>> [9]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.47.0-RC2/go/pkg/beam
>> [10]
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=.
>> ..
>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>> [12] https://github.com/apache/beam/pull/26152
>>
>>
>> --
>>
>>
>> Jack McCluskey
>> SWE - DataPLS PLAT/ Dataflow ML
>> RDU
>> jrmcclus...@google.com
>>
>>
>>

Re: [PROPOSAL] Preparing for 2.48.0 Release

[Ahmet Altay via dev]

It is every 6 weeks. There is also a published calendar for release branch
cut dates:
https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com=America/Los_Angeles

On Wed, May 3, 2023 at 10:13 PM Jeff Zhang  wrote:

> I just saw another thread about the vote of 2.47.0 release, just curious
> to know what is beam's release cadence, is it monthly?
>
>
> On Thu, May 4, 2023 at 1:58 AM Kenneth Knowles  wrote:
>
>> Excellent, thank you!
>>
>> On Wed, May 3, 2023 at 7:21 AM Ahmet Altay via dev 
>> wrote:
>>
>>> Thank you Ritesh!
>>>
>>> On Wed, May 3, 2023 at 10:00 AM Ritesh Ghorse via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> Hey everyone,
>>>>
>>>> The next release branch 2.48.0 cut is scheduled for May 17th, according
>>>> to
>>>> the release calendar [1].
>>>>
>>>> I would like to volunteer myself to do this release. I'll cut the
>>>> branch on the scheduled date, and cherrypick release-blocking fixes later.
>>>>
>>>> Please help me make sure the release goes smoothly by:
>>>> - Making sure that any unresolved release blocking issues for 2.48.0
>>>> have their "Milestone" marked as "2.48.0 Release".
>>>> - Reviewing the current release blockers [2] and remove the Milestone
>>>> if they don't meet the criteria at [3].
>>>>
>>>> [1]
>>>> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com
>>>> [2] https://github.com/apache/beam/milestone/12
>>>> [3] https://beam.apache.org/contribute/release-blocking/
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> Regards,
>>>> Ritesh Ghorse
>>>>
>>>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: [PROPOSAL] Preparing for 2.48.0 Release

[Ahmet Altay via dev]

Thank you Ritesh!

On Wed, May 3, 2023 at 10:00 AM Ritesh Ghorse via dev 
wrote:

> Hey everyone,
>
> The next release branch 2.48.0 cut is scheduled for May 17th, according to
> the release calendar [1].
>
> I would like to volunteer myself to do this release. I'll cut the branch
> on the scheduled date, and cherrypick release-blocking fixes later.
>
> Please help me make sure the release goes smoothly by:
> - Making sure that any unresolved release blocking issues for 2.48.0 have
> their "Milestone" marked as "2.48.0 Release".
> - Reviewing the current release blockers [2] and remove the Milestone if
> they don't meet the criteria at [3].
>
> [1]
> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com
> [2] https://github.com/apache/beam/milestone/12
> [3] https://beam.apache.org/contribute/release-blocking/
>
> Thanks!
>
> --
> Regards,
> Ritesh Ghorse
>

Re: [LAZY CONSENSUS] Drop @Experimental annotations and concept from Beam

[Ahmet Altay via dev]

Thank you very much for doing this Kenn!

On Mon, May 1, 2023 at 3:49 PM Kenneth Knowles  wrote:

> https://github.com/apache/beam/pull/26490
>
> On Mon, May 1, 2023 at 10:42 AM Austin Bennett  wrote:
>
>> great, thanks, @* Kenneth Knowles  !
>>
>> On Mon, May 1, 2023 at 10:12 AM Kenneth Knowles  wrote:
>>
>>> We are well past the lazy consensus. I will remove the @Experimental
>>> annotations and concept from Beam.
>>>
>>> Kenn
>>>
>>> On Tue, Apr 25, 2023 at 3:40 PM Kenneth Knowles  wrote:
>>>
 Hello!

 I propose to drop @Experimental annotations and concept from Beam.
 Discussion occurred at
 https://lists.apache.org/thread/tvvdckdom8jtv2xr9mzg0ltjjpbmydrv.

 Once approved, I will make the code changes to eliminate the annotation.

 If no one has an objection or further discussion needed in 72 hours, it
 can be considered approved. See
 https://community.apache.org/committers/lazyConsensus.html

 Kenn

>>>

Re: [ANNOUNCE] New committer: Damon Douglas

[Ahmet Altay via dev]

Congratulations Damon!

On Mon, Apr 24, 2023 at 1:00 PM Robert Burke  wrote:

> Congratulations Damon!!!
>
> On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new committer:
>> Damon Douglas (damondoug...@apache.org)
>>
>> Damon has contributed widely: Beam Katas, playground, infrastructure, and
>> many IO connectors. Damon does lots of code review in addition to code.
>> (yes, you can review code as a non-committer!)
>>
>> Considering their contributions to the project over this timeframe, the
>> Beam PMC trusts Damon with the responsibilities of a Beam committer. [1]
>>
>> Thank you Damon! And we are looking to see more of your contributions!
>>
>> Kenn, on behalf of the Apache Beam PMC
>>
>> [1]
>>
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>

Re: [ANNOUNCE] New committer: Anand Inguva

[Ahmet Altay via dev]

Congratulations Anand!

On Mon, Apr 24, 2023 at 8:05 AM Kerry Donny-Clark via dev <
dev@beam.apache.org> wrote:

> Great work Anand, this is well deserved.
>
>
> On Mon, Apr 24, 2023 at 10:35 AM Yi Hu via dev 
> wrote:
>
>> Congrats Anand!
>>
>> On Fri, Apr 21, 2023 at 3:54 PM Danielle Syse via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congratulations!
>>>
>>> On Fri, Apr 21, 2023 at 3:53 PM Damon Douglas via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congratulations Anand!

 On Fri, Apr 21, 2023 at 12:28 PM Ritesh Ghorse via dev <
 dev@beam.apache.org> wrote:

> Congratulations Anand!
>
> On Fri, Apr 21, 2023 at 3:24 PM Ahmed Abualsaud via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Anand!
>>
>> On Fri, Apr 21, 2023 at 3:18 PM Anand Inguva via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Thanks everyone. Really excited to be a part of Beam Committers.
>>>
>>> On Fri, Apr 21, 2023 at 3:07 PM XQ Hu via dev 
>>> wrote:
>>>
 Congratulations, Anand!!!

 On Fri, Apr 21, 2023 at 2:31 PM Jack McCluskey via dev <
 dev@beam.apache.org> wrote:

> Congratulations, Anand!
>
> On Fri, Apr 21, 2023 at 2:28 PM Valentyn Tymofieiev via dev <
> dev@beam.apache.org> wrote:
>
>> Congratulations!
>>
>> On Fri, Apr 21, 2023 at 8:19 PM Jan Lukavský 
>> wrote:
>>
>>> Congrats Anand!
>>> On 4/21/23 20:05, Robert Burke wrote:
>>>
>>> Congratulations Anand!
>>>
>>> On Fri, Apr 21, 2023, 10:55 AM Danny McCormick via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Woohoo, congrats Anand! This is very well deserved!

 On Fri, Apr 21, 2023 at 1:54 PM Chamikara Jayalath <
 chamik...@apache.org> wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Anand Inguva (ananding...@apache.org)
>
> Anand has been contributing to Apache Beam for more than a
> year and  authored and reviewed more than 100 PRs. Anand has been 
> a core
> contributor to Beam Python SDK and drove the efforts to support 
> Python 3.10
> and Python 3.11.
>
> Considering their contributions to the project over this
> timeframe, the Beam PMC trusts Anand with the responsibilities of 
> a Beam
> committer. [1]
>
> Thank you Anand! And we are looking to see more of your
> contributions!
>
> Cham, on behalf of the Apache Beam PMC
>
> [1]
> https://beam.apache.org/contribute/become-a-committer
> /#an-apache-beam-committer
>

Re: Beam Release DockerHub Group

[Ahmet Altay via dev]

Thank you for doing this!

I try to stay engaged with the releases and would be happy to be kept on
this list. At the same time I agree with Robert, release managers should
take precedence over PMC members if we cannot get the number to 6.

On Mon, Apr 17, 2023 at 10:32 AM Robert Bradshaw via dev <
dev@beam.apache.org> wrote:

> Well, I don't know that PMC should take precedence over release managers
> if it comes to that.
>
> On Mon, Apr 17, 2023 at 10:11 AM Danny McCormick <
> dannymccorm...@google.com> wrote:
>
>> I can ask if we can keep 6 seats instead of 5 (and keep Ahmet in that
>> seat). If not, my vote would be to stick with the 5 that I suggested, but
>> if that's not acceptable then I'll drop one of the people (probably myself
>> since I just finished a release and will probably be last up out of the
>> group).
>>
>> Thanks,
>> Danny
>>
>> On Mon, Apr 17, 2023 at 12:34 PM Robert Bradshaw 
>> wrote:
>>
>>> I think it'd be good if the intersection between this list and the PMC
>>> had cardinality greater than 1. Ahmet might be a good person to keep there.
>>>
>>> On Mon, Apr 17, 2023 at 9:25 AM Danny McCormick via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Yeah, that is part of the proposal. To be clear, our end state would be
 a single group with

 - damccorm
 - jrmccluskey
 - kennknowles
 - lostluck
 - abacn

 Thanks,
 Danny

 On Mon, Apr 17, 2023 at 12:24 PM Kerry Donny-Clark 
 wrote:

> +1, should there also be an update to remove folks who are not active
> on the project?
> Kerry
>
> On Mon, Apr 17, 2023 at 11:40 AM Jack McCluskey via dev <
> dev@beam.apache.org> wrote:
>
>> +1 to simplifying the infra side, especially with an aim towards
>> automating the processes we can. The more we can streamline and simplify
>> the better.
>>
>> On Mon, Apr 17, 2023 at 11:18 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hey everyone, in an effort to reduce the burden of running a Beam
>>> release, a few committers (self included) have volunteered to try to 
>>> take a
>>> larger role in releases (including both running them and contributing to
>>> making them better going forward). To aid in that process, I would like 
>>> to
>>> request that they all be added to our Beam DockerHub group. Those
>>> committers are:
>>>
>>> - damccorm
>>> - jrmccluskey
>>> - kennknowles
>>> - lostluck
>>> - abacn
>>>
>>> At the same time, Infra would like us to reduce the number of people
>>> with DockerHub seats to 5 because they have a limited number of seats 
>>> for
>>> all of Apache. Currently, we have 2 groups taking up 10 seats: Beam 
>>> admin
>>> and Beam maintainers.
>>>
>>> Beam admin has admin privileges over most (though not quite all) of
>>> our DockerHub repos and includes:
>>>
>>> - aaltay
>>> - hannahjiang
>>> - kileysok
>>> - pabloem
>>> - robertwb
>>>
>>> Beam maintainers has write privileges and some additional admin
>>> privileges and includes:
>>>
>>> - aaltay
>>> - chamikaramj
>>> - kennknowles
>>> - kileysok
>>> - robertwb
>>>
>>> To get down to 5 seats, I propose we consolidate to a single group
>>> with admin privileges and add just the committers I mentioned since they
>>> will likely be the most actively involved in the release process in the
>>> short term. A future goal of mine is to automate the DockerHub release
>>> steps so that we just need 2 dockerhub seats: 1 for the automation and 1
>>> for an easy manual fallback (probably for the PMC chair).
>>>
>>> If you have any concerns (or would like to help with this effort),
>>> please respond here. Otherwise I will follow up with infra to make this
>>> change in a day.
>>>
>>> Thanks,
>>> Danny
>>>
>>

Re: [PROPOSAL] Preparing for 2.47.0 Release

[Ahmet Altay via dev]

Thank you for the update Jack!

On Tue, Apr 18, 2023 at 11:23 AM Jack McCluskey 
wrote:

> Quick update for everyone, the initial release blockers on the 2.47.0
> milestone have been resolved and the RC1 commit has been tagged. I'll be
> working on getting RC1 artifacts built now.
>
> On Thu, Apr 13, 2023 at 12:22 PM Ahmet Altay  wrote:
>
>> Sounds good. Thank you. And if you need help please reach out.
>>
>> On Thu, Apr 13, 2023 at 6:29 AM Jack McCluskey 
>> wrote:
>>
>>> We're making good progress on finding and fixing bugs. Not quite to
>>> building an RC candidate yet, but so far nothing that seems to be a
>>> difficult fix.
>>>
>>> On Wed, Apr 12, 2023 at 8:10 PM Ahmet Altay  wrote:
>>>
>>>> Jack, how is the release coming along?
>>>>
>>>> On Tue, Apr 4, 2023 at 12:23 PM Jack McCluskey via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> Hey everyone,
>>>>>
>>>>> I need a PMC member's help adding my pubkey to
>>>>> https://dist.apache.org/repos/dist/release/beam/KEYS as well as
>>>>> adding PyPI user jrmccluskey to the maintainers of the Apache Beam 
>>>>> package.
>>>>> These are the last steps I have to do to complete prep for the release.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jack McCluskey
>>>>>
>>>>> On Wed, Mar 22, 2023 at 11:38 AM Jack McCluskey <
>>>>> jrmcclus...@google.com> wrote:
>>>>>
>>>>>> Hey all,
>>>>>>
>>>>>> The next (2.47.0) release branch cut is scheduled for April 5th,
>>>>>> 2023, according to
>>>>>> the release calendar [1].
>>>>>>
>>>>>> I will be performing this release. My plan is to cut the branch on
>>>>>> that date, and cherrypick release-blocking fixes afterwards, if any.
>>>>>>
>>>>>> Please help me make sure the release goes smoothly by:
>>>>>> - Making sure that any unresolved release blocking issues
>>>>>> for 2.47.0 should have their "Milestone" marked as "2.47.0 Release"
>>>>>> as soon as possible.
>>>>>> - Reviewing the current release blockers [2] and remove the
>>>>>> Milestone if they don't meet the criteria at [3].
>>>>>>
>>>>>> Let me know if you have any comments/objections/questions.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jack McCluskey
>>>>>>
>>>>>> [1]
>>>>>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>>>>>> [2] https://github.com/apache/beam/milestone/10
>>>>>> [3] https://beam.apache.org/contribute/release-blocking/
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>> Jack McCluskey
>>>>>> SWE - DataPLS PLAT/ Dataflow ML
>>>>>> RDU
>>>>>> jrmcclus...@google.com
>>>>>>
>>>>>>
>>>>>>

Re: [DISCUSS] @Experimental, @Internal, @Stable, etc annotations

[Ahmet Altay via dev]

It sounds like there is agreement in eliminating the
experimental annotation. Should we stop using them in new code? Or should
we do a pass to remove those annotations?

On Mon, Apr 17, 2023 at 11:24 AM Kenneth Knowles  wrote:

>
>
> On Mon, Apr 17, 2023 at 9:34 AM Kerry Donny-Clark via dev <
> dev@beam.apache.org> wrote:
>
>> +1 to eliminating @Experimental as a Beam level annotation.
>> I think the main point is that if no one pays attention to such
>> annotations, then they are only noise and deliver negative value.
>>
>
> Yes. Consider these two scenarios
>
> 1. We change an "experimental" API that is widely used. This causes a pain
> for many users. We would probably not do it, and we would catch it in code
> review.
> 2. We change a non-"experimental" API that is fairly new. This applies to
> many APIs, since we rarely remember to annotate new APIs. This causes just
> minor pain for just a few users. TBH I would be OK with this. Rigidity in
> rejecting such changes just means your first draft is your final draft. Try
> that in any other endeavor and see how it works for you :-)
>
> And it is worse than noise - there are some users who do pay attention to
> the annotations and are not using things even though they are super safe.
> That was the main reason I started this thread. The rest of my proposal was
> just to try to recover some flexibility, but it seems too hard and no
> immediate consensus on how/if we could manage it.
>
> Kenn
>
> PS I do agree with Kerry's PS and would love to have that discussion.
> Perhaps separately, since it will start from square one either way. Every
> time someone says "Beam 3.0" we should really be thinking "how can we
> iterate". One big breaking version change doesn't work.
>

+1 - Thinking about "How can we iterate" would allow us to build something
users' want in shorter timelines.


>
>
>
> Kerry
>>
>> PS- Kenn says " the point about the culture of stagnation came from my
>> recent experiences as code reviewer where there was some idea that we
>> couldn't change things even when they were plainly wrong and the change was
>> plainly a fix." This seems like a major point that deserves a more focused
>> discussion.
>>
>> On Fri, Apr 14, 2023 at 5:47 PM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>>> I think we've been using the Java Experimental tags in two ways.
>>>
>>> * New APIs
>>> * Any APIs that use specific features identified by pre-defined
>>> experimental Kind types defined in [1] (for example, I/O connectors APIs
>>> that use Beam Schemas).
>>>
>>> Removing the experimental tag has the effect of finalizing a number of
>>> APIs we've been reluctant to call stable (for example, Beam Schemas,
>>> portability, metrics related APIs). These APIs have been around for a long
>>> time and I don't see them changing so probably this is the right thing to
>>> do. But I just wanted to call it out.
>>>
>>> Thanks,
>>> Cham
>>>
>>> [1]
>>> https://github.com/apache/beam/blob/b9f27f9da2e63b564feecaeb593d7b12783192b0/sdks/java/core/src/main/java/org/apache/beam/sdk/annotations/Experimental.java#L48
>>>
>>> On Fri, Apr 14, 2023 at 1:26 PM Ahmet Altay via dev 
>>> wrote:
>>>
>>>>
>>>>
>>>> On Fri, Apr 14, 2023 at 1:15 PM Kenneth Knowles 
>>>> wrote:
>>>>
>>>>>
>>>>> Thanks for the discussion. Many good points. Probably just removing
>>>>> all the annotations is a noop to users, and will solve the "afraid to use
>>>>> experimental features" problem.
>>>>>
>>>>> Regarding stability, the capabilities of Java (and Python is much much
>>>>> worse) make it infeasible to produce quality software with the rule "once
>>>>> it is public it is frozen forever". But on the other hand, there isn't 
>>>>> much
>>>>> of a practical alternative. Most projects just make breaking changes at
>>>>> minor releases quite often, in my experience. I don't want to follow that
>>>>> pattern, for sure.
>>>>>
>>>>> Regarding Danny's comment of not seeing this culture - check out any
>>>>> of our more mature IOs, which all have very high cyclomatic complexity due
>>>>> to never being significantly refactored. Adhering to in-place state
>>>>> compatibility for update instead of focusing on blue/green depl

Re: [DISCUSS] @Experimental, @Internal, @Stable, etc annotations

[Ahmet Altay via dev]

On Fri, Apr 14, 2023 at 1:15 PM Kenneth Knowles  wrote:

>
> Thanks for the discussion. Many good points. Probably just removing all
> the annotations is a noop to users, and will solve the "afraid to use
> experimental features" problem.
>
> Regarding stability, the capabilities of Java (and Python is much much
> worse) make it infeasible to produce quality software with the rule "once
> it is public it is frozen forever". But on the other hand, there isn't much
> of a practical alternative. Most projects just make breaking changes at
> minor releases quite often, in my experience. I don't want to follow that
> pattern, for sure.
>
> Regarding Danny's comment of not seeing this culture - check out any of
> our more mature IOs, which all have very high cyclomatic complexity due to
> never being significantly refactored. Adhering to in-place state
> compatibility for update instead of focusing on blue/green deployment is
> also a culprit here. I don't have examples to mind, but the point about the
> culture of stagnation came from my recent experiences as code
> reviewer where there was some idea that we couldn't change things even when
> they were plainly wrong and the change was plainly a fix.
>
> Often, it comes from corners like triggered side inputs where we simply
> never had a clear concept and so bringing things into alignment with a spec
> will break someone, by necessity. To be clear: I have not received pushback
> on that one (yet). Some other examples are
> https://s.apache.org/finishing-triggers-drop-data (breaking change
> necessary to eliminate data loss risk)
> https://github.com/apache/beam/issues/20528 (fix was too slow because we
> were hesitant to commit a breaking fix)
> https://github.com/apache/beam/pull/8134#pullrequestreview-218592801
> (left unsafe API in place, applied doc-only fix).
>
> But indeed, of all the issues I raised, the customer concern with
> `@Experimental` was the most important. We have had a few threads about it
> in the past, too, and it hasn't gotten better.
>
>  1. It does not have the intended effect (making users OK with evolving
> APIs and behavior to allow us to reach a high level of quality)
>  2. It has an unintended effect (making users afraid to use things which
> they should be happy to use)
>  3. We don't use it consistently (many less-safe things are not
> experimental, many totally stable things are experimental)
>
> Because of 3, if we don't have a feasible way to move to
> "evolving/unstable by default" in a way that users know and are OK with,
> then 1 is impossible. And so the only way to fix 2 is to just eliminate the
> annotation approach entirely and go with language conventions.
>

+1 to eliminating @Experimental as a Beam level annotation. That is the
simplest approach that will get us to a consistent state, and it will align
the goals and intentions of us with users'.


>
> Kenn
>
> On Wed, Apr 12, 2023 at 5:10 PM Ahmet Altay via dev 
> wrote:
>
>> I agree with Alexey and Byron.
>> 1. We do not have any concrete evidence of our users paying attention to
>> any of those annotations. Experimental API that were in that state for a
>> long while are good examples. A possible exception is a deprecated
>> annotation. My preference would be to simplify annotations to nothing
>> (stable enough for use and will evolve backward compatibility), and maybe
>> deprecated annotations.
>> 2. If you all think that Experimental annotation is needed, Byron's
>> suggestion (more or less what we do today) but with some concrete life
>> cycle definitions of those annotations would be useful to our users. (An
>> example could be: experimental APIs either need to graduate or be removed
>> in X releases.)
>>
>>
>>
>> On Tue, Apr 4, 2023 at 9:01 AM Alexey Romanenko 
>> wrote:
>>
>>> Great and long-to-wait topic to discuss.
>>>
>>> My personal opinion based on what I saw on different open-source
>>> projects is that all such annotations, like @Experimental or @Stable, are
>>> not usefull along the time and even rather useless and misleading. What
>>> actually play roles is artifacts publishing and public API despite how it
>>> was annotated. Once a class/method was published and available for users to
>>> use, it should be considered as “stable" (even if it’s not yet stable from
>>> its developers point of view) and can’t be easily removed/changed in the
>>> next releases.
>>>
>>> At Beam, we have a “good" example with @Experimental that was used to
>>> annotate many parts of code in the beginning of its creation but then
>>> perhap

Re: Python 3.11 support in Apache Beam

[Ahmet Altay via dev]

I forgot to add the link, [1] was meant to be :
https://docs.python.org/3/whatsnew/3.11.html#faster-cpython

On Thu, Apr 13, 2023 at 10:17 AM Anand Inguva 
wrote:

> Yes Ahmet. That would be great.
>
> There are some load tests defined in the
> https://github.com/apache/beam/blob/master/.test-infra which could be
> useful for performance testing of Beam between 3.10 and 3.11. Do you
> suggest any other tests?
>

I have not looked at the full list. I do not think we will see much in IO
bound pipelines, or pipelines that do most of their work with a C extension
library already. Maybe some of the load tests like pardo load tests?

If feasible, we could convert benchmarks to run on 3.11 and see which ones
will see a larger improvement.

Also apparently there is a potential regression of using up to 20% more
memory (
https://docs.python.org/3/whatsnew/3.11.html#will-cpython-3-11-use-more-memory).
I wonder if that will negatively impact us. If feasible, it would be useful
to understand that as well.


>
> On Wed, Apr 12, 2023 at 8:04 PM Ahmet Altay  wrote:
>
>> Thank you, this is great!
>>
>> Python 3.11 announcement had a claim about performance [1]:
>>
>> "CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
>> with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
>> Linux. Depending on your workload, the overall speedup could be 10-60%."
>>
>> Have we measured this in Beam? Are we seeing any benefits? If not, why?
>> If yes, this would be a cool blog post as well.
>>
>> Ahmet
>>
>>
>> On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev 
>> wrote:
>>
>>> Python 3.11 support has been merged at
>>> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0
>>> release.
>>>
>>> Please let me know if you have any questions.
>>>
>>> Thanks,
>>> Anand
>>>
>>> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev 
>>> wrote:
>>>
>>>> Thanks a lot Anand. I'll take a look at the PRs.
>>>>
>>>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva 
>>>> wrote:
>>>>
>>>>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>>>>> that updates the build dependencies of Apache Beam.
>>>>>
>>>>> Several GCP dependencies needed to be updated as well. I covered them
>>>>> in the PR: https://github.com/apache/beam/pull/24599
>>>>>
>>>>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva 
>>>>> wrote:
>>>>>
>>>>>> Yes, we may need to update all of them
>>>>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>.
>>>>>> I can add more information once I dig into the issue(most likely next
>>>>>> week). I will comment on my findings on the issue:
>>>>>> https://github.com/apache/beam/issues/24569 and will periodically
>>>>>> update this thread.
>>>>>>
>>>>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <
>>>>>> valen...@google.com> wrote:
>>>>>>
>>>>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, it is related to protobuf only. But I think the update of
>>>>>>>> these dependencies are required for Python 3.11 since the newer 
>>>>>>>> versions
>>>>>>>> have support for Python 3.11 wheels.
>>>>>>>>
>>>>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10
>>>>>>> for protobuf==3.x.x and that can cause friction.
>>>>>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>>>>>
>>>>>>> I would probably narrow the problem further to demonstrate which
>>>>>>> stubs are not being generated, and if reason not obvious we can also ask
>>>>>>> for feedback from protobuf maintainers. Also - do we by chance need to
>>>>>>> update some other deps from
>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>>>>>> for this to work?
>>>>>>>
>>>>>>> Also: tracking issue for protobuf4 support in Beam:
>>>>>>> https://github.com/apache/beam/issues/24569.
>>>>&

Re: [PROPOSAL] Preparing for 2.47.0 Release

[Ahmet Altay via dev]

Sounds good. Thank you. And if you need help please reach out.

On Thu, Apr 13, 2023 at 6:29 AM Jack McCluskey 
wrote:

> We're making good progress on finding and fixing bugs. Not quite to
> building an RC candidate yet, but so far nothing that seems to be a
> difficult fix.
>
> On Wed, Apr 12, 2023 at 8:10 PM Ahmet Altay  wrote:
>
>> Jack, how is the release coming along?
>>
>> On Tue, Apr 4, 2023 at 12:23 PM Jack McCluskey via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hey everyone,
>>>
>>> I need a PMC member's help adding my pubkey to
>>> https://dist.apache.org/repos/dist/release/beam/KEYS as well as adding
>>> PyPI user jrmccluskey to the maintainers of the Apache Beam package. These
>>> are the last steps I have to do to complete prep for the release.
>>>
>>> Thanks,
>>>
>>> Jack McCluskey
>>>
>>> On Wed, Mar 22, 2023 at 11:38 AM Jack McCluskey 
>>> wrote:
>>>
>>>> Hey all,
>>>>
>>>> The next (2.47.0) release branch cut is scheduled for April 5th, 2023,
>>>> according to
>>>> the release calendar [1].
>>>>
>>>> I will be performing this release. My plan is to cut the branch on that
>>>> date, and cherrypick release-blocking fixes afterwards, if any.
>>>>
>>>> Please help me make sure the release goes smoothly by:
>>>> - Making sure that any unresolved release blocking issues
>>>> for 2.47.0 should have their "Milestone" marked as "2.47.0 Release" as
>>>> soon as possible.
>>>> - Reviewing the current release blockers [2] and remove the Milestone
>>>> if they don't meet the criteria at [3].
>>>>
>>>> Let me know if you have any comments/objections/questions.
>>>>
>>>> Thanks,
>>>>
>>>> Jack McCluskey
>>>>
>>>> [1]
>>>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>>>> [2] https://github.com/apache/beam/milestone/10
>>>> [3] https://beam.apache.org/contribute/release-blocking/
>>>>
>>>> --
>>>>
>>>>
>>>> Jack McCluskey
>>>> SWE - DataPLS PLAT/ Dataflow ML
>>>> RDU
>>>> jrmcclus...@google.com
>>>>
>>>>
>>>>

Re: [PROPOSAL] Preparing for 2.47.0 Release

[Ahmet Altay via dev]

Jack, how is the release coming along?

On Tue, Apr 4, 2023 at 12:23 PM Jack McCluskey via dev 
wrote:

> Hey everyone,
>
> I need a PMC member's help adding my pubkey to
> https://dist.apache.org/repos/dist/release/beam/KEYS as well as adding
> PyPI user jrmccluskey to the maintainers of the Apache Beam package. These
> are the last steps I have to do to complete prep for the release.
>
> Thanks,
>
> Jack McCluskey
>
> On Wed, Mar 22, 2023 at 11:38 AM Jack McCluskey 
> wrote:
>
>> Hey all,
>>
>> The next (2.47.0) release branch cut is scheduled for April 5th, 2023,
>> according to
>> the release calendar [1].
>>
>> I will be performing this release. My plan is to cut the branch on that
>> date, and cherrypick release-blocking fixes afterwards, if any.
>>
>> Please help me make sure the release goes smoothly by:
>> - Making sure that any unresolved release blocking issues
>> for 2.47.0 should have their "Milestone" marked as "2.47.0 Release" as
>> soon as possible.
>> - Reviewing the current release blockers [2] and remove the Milestone if
>> they don't meet the criteria at [3].
>>
>> Let me know if you have any comments/objections/questions.
>>
>> Thanks,
>>
>> Jack McCluskey
>>
>> [1]
>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>> [2] https://github.com/apache/beam/milestone/10
>> [3] https://beam.apache.org/contribute/release-blocking/
>>
>> --
>>
>>
>> Jack McCluskey
>> SWE - DataPLS PLAT/ Dataflow ML
>> RDU
>> jrmcclus...@google.com
>>
>>
>>

Re: [DISCUSS] @Experimental, @Internal, @Stable, etc annotations

[Ahmet Altay via dev]

I agree with Alexey and Byron.
1. We do not have any concrete evidence of our users paying attention to
any of those annotations. Experimental API that were in that state for a
long while are good examples. A possible exception is a deprecated
annotation. My preference would be to simplify annotations to nothing
(stable enough for use and will evolve backward compatibility), and maybe
deprecated annotations.
2. If you all think that Experimental annotation is needed, Byron's
suggestion (more or less what we do today) but with some concrete life
cycle definitions of those annotations would be useful to our users. (An
example could be: experimental APIs either need to graduate or be removed
in X releases.)



On Tue, Apr 4, 2023 at 9:01 AM Alexey Romanenko 
wrote:

> Great and long-to-wait topic to discuss.
>
> My personal opinion based on what I saw on different open-source projects
> is that all such annotations, like @Experimental or @Stable, are not
> usefull along the time and even rather useless and misleading. What
> actually play roles is artifacts publishing and public API despite how it
> was annotated. Once a class/method was published and available for users to
> use, it should be considered as “stable" (even if it’s not yet stable from
> its developers point of view) and can’t be easily removed/changed in the
> next releases.
>
> At Beam, we have a “good" example with @Experimental that was used to
> annotate many parts of code in the beginning of its creation but then
> perhaps forgotten to be removed whenever this code is already used by many
> users and API can’t be just changed despite of this annotation.
>
> So, I’m pro to dismiss such annotations and consider all public and
> user-available API as “stable”. If it’s needed to change/remove a public
> API then we should follow the procedure of API deprecation and final
> removing, at least, after 3 major (x.y) Beam releases. It should help to
> have the clear rules for API changes and avoiding breaking changes for
> users.
>
> —
> Alexey
>
>
> On 3 Apr 2023, at 17:04, Byron Ellis via dev  wrote:
>
> Honestly, I think APIs could be pretty simply defined if you think of it
> in terms of the user:
>
> @Deprecated = this was either stable or evolve but the
> functionality/interface will go away at a future date
>
> @Stable = the user of this API opting out of changes to functionality and
> interface. For example, default options don't change for a transform
> annotated this way.
>
> Evolving (No Annotation) = the user is opting in to changes to
> functionality but not to interface. We should generally try to write
> backwards compatible code, but on the other hand the release model does not
> force users into an upgrade
>
> @Experimental = this functionality / interface might be a bad idea and
> could go away at any time
>
>
> On Mon, Apr 3, 2023 at 7:22 AM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> *;tldr - I'd like "evolving" to be further defined, specifically around
>> how we will make decisions about breaking behavior and API changes*
>>
>> I don't particularly care what tags we use as long as they're well
>> documented. With that said, I think the following framing needs to be
>> documented with more definition to flesh out the underlying philosophy:
>>
>> *>  - new code is changeable/evolving by default (so we don't have to
>> always remember to annotate it) but users have confidence they can use it
>> in production (because we have good software engineering practices)*
>>
>> * > - Experimental would be reserved for more risky things*
>> * > - after we are confident an API is stable, because it has been the
>> same across a couple releases, we mark it*
>>
>> Here, we have 3 classes of APIs - "experimental", "stable", and
>> "evolving" (or alternately "undefined").
>>
>> "Experimental" seems clear - we can make any changes we want. "Stable" is
>> reasonably straightforward as well - we will only make non-breaking changes
>> except in exceptional cases (e.g. security hole, total failure of
>> functionality, etc...)
>>
>> With "evolving" is the idea that we can still make any changes we want,
>> but we think it's less likely we'll need to? Are silent behavior changes
>> acceptable here (my vote would be no)? What about breaking API changes (my
>> vote would be rarely)?
>>
>> I think being able to change our APIs is an ok goal, but outside of a
>> true experimental context we should still be weighing the cost of API
>> changes against the benefit; we have a problem of people not updating to
>> newer SDKs, and introducing more breaking changes will just exacerbate that
>> problem. Maybe my concerns are just a consequence of me not really seeing
>> the same things that you're seeing, specifically: "*I'm seeing a culture
>> of being afraid to change things, even when it would be good for users,
>> because our API surface area is far too large and not explicitly chosen.*"
>> Mostly what I've seen is a healthy concern about 

Re: Python 3.11 support in Apache Beam

[Ahmet Altay via dev]

Thank you, this is great!

Python 3.11 announcement had a claim about performance [1]:

"CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
Linux. Depending on your workload, the overall speedup could be 10-60%."

Have we measured this in Beam? Are we seeing any benefits? If not, why? If
yes, this would be a cool blog post as well.

Ahmet


On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev 
wrote:

> Python 3.11 support has been merged at
> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0 release.
>
> Please let me know if you have any questions.
>
> Thanks,
> Anand
>
> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev 
> wrote:
>
>> Thanks a lot Anand. I'll take a look at the PRs.
>>
>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva 
>> wrote:
>>
>>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>>> that updates the build dependencies of Apache Beam.
>>>
>>> Several GCP dependencies needed to be updated as well. I covered them in
>>> the PR: https://github.com/apache/beam/pull/24599
>>>
>>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva 
>>> wrote:
>>>
 Yes, we may need to update all of them
 .
 I can add more information once I dig into the issue(most likely next
 week). I will comment on my findings on the issue:
 https://github.com/apache/beam/issues/24569 and will periodically
 update this thread.

 On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev 
 wrote:

> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva 
> wrote:
>
>> Yes, it is related to protobuf only. But I think the update of these
>> dependencies are required for Python 3.11 since the newer versions have
>> support for Python 3.11 wheels.
>>
> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
> protobuf==3.x.x and that can cause friction.
> https://pypi.org/project/protobuf/3.20.3/#files
>
> I would probably narrow the problem further to demonstrate which stubs
> are not being generated, and if reason not obvious we can also ask for
> feedback from protobuf maintainers. Also - do we by chance need to
> update some other deps from
> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
> for this to work?
>
> Also: tracking issue for protobuf4 support in Beam:
> https://github.com/apache/beam/issues/24569.
>
> If we use older versions of these packages, then we have to depend on
>> installing those packages on Python 3.11 from source distributions which 
>> is
>> not desired.
>>
>> I am working parallely on that issue in a different PR
>> https://github.com/apache/beam/pull/24599 but I think this issue
>> should be a blocker for Python 3.11 update.
>>
>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> Hi Anand,
>>>
>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Hi all,

 We are planning to work on adding support for Python 3.11[1] to
 Apache Beam Python SDK.

 As part of this effort, we are going to update the python build
 dependencies defined at [2].

 Right now, there is an error with the newer version of
 protobuf(4.21.11). It is not generating _urn files.

 It can be reproduced by

>>>
 1. python setup.py sdist
 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
 3. switch to python interpreter and run import apache_beam as beam

>>> I think the error you are describing is related to protobuf 4, so
>>> the repro should focus on the portion where generation of stubs is
>>> happening. Presumably some stubs are not generated on protobuf 4 + 
>>> Python
>>> 3.11?
>>>
>>>

 will lead to *ImportError: cannot import name
 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'.  
 *Running
 `python gen_protos.py` to forcefully generate files didn't help either.

 If you have encountered this error and found a resolution, please
 let me know(that would be super helpful).

 I am going to work on this soon. Please let me know if you want to
 collaborate.

 Thanks,
 Anand Inguva

 *[1] *https://github.com/apache/beam/pull/24721
 [2]
 https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt

>>>

Re: [PROPOSAL] Preparing for 2.47.0 Release

[Ahmet Altay via dev]

Thank you Jack!

On Wed, Mar 22, 2023 at 8:39 AM Jack McCluskey via dev 
wrote:

> Hey all,
>
> The next (2.47.0) release branch cut is scheduled for April 5th, 2023,
> according to
> the release calendar [1].
>
> I will be performing this release. My plan is to cut the branch on that
> date, and cherrypick release-blocking fixes afterwards, if any.
>
> Please help me make sure the release goes smoothly by:
> - Making sure that any unresolved release blocking issues
> for 2.47.0 should have their "Milestone" marked as "2.47.0 Release" as
> soon as possible.
> - Reviewing the current release blockers [2] and remove the Milestone if
> they don't meet the criteria at [3].
>
> Let me know if you have any comments/objections/questions.
>
> Thanks,
>
> Jack McCluskey
>
> [1]
> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
> [2] https://github.com/apache/beam/milestone/10
> [3] https://beam.apache.org/contribute/release-blocking/
>
> --
>
>
> Jack McCluskey
> SWE - DataPLS PLAT/ Dataflow ML
> RDU
> jrmcclus...@google.com
>
>
>

Re: [RESULT] [VOTE] Release 2.46.0, release candidate #1

[Ahmet Altay via dev]

Thank you very much for doing the release!

On Fri, Mar 10, 2023 at 7:30 PM Danny McCormick via dev 
wrote:

> The release has been finalized, thanks everyone!
>
> Thanks,
> Danny
>
> On Fri, Mar 10, 2023 at 2:38 PM Danny McCormick 
> wrote:
>
>> We determined that the same issue exists in the 2.45 release, so we are
>> going to continue finalizing the release candidate. Thank you for your
>> patience.
>>
>> Thanks,
>> Danny
>>
>> On Wed, Mar 8, 2023 at 2:30 PM Danny McCormick 
>> wrote:
>>
>>> @Reuven Lax  found a new potential regression in
>>> BigQuery I/O, so I have paused the release rollout. I had already pushed
>>> the Python artifacts and Go tags, but not the Java ones. We have since
>>> temporarily yanked <https://pypi.org/help/#yanked> the Python release
>>> and deleted the Go tags, they were live for around an hour. The possible
>>> regression is in Java, so neither of those releases should be affected, but
>>> x-lang may not work properly because it depends on versioning. I will
>>> update this thread with next steps when we know more.
>>>
>>> Thanks,
>>> Danny
>>>
>>> On Wed, Mar 8, 2023 at 1:02 PM Anand Inguva 
>>> wrote:
>>>
>>>> Thanks Danny!!
>>>>
>>>> On Wed, Mar 8, 2023 at 12:14 PM Danny McCormick via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> I'm happy to announce that we have unanimously approved release 2.46.0
>>>>> There are 8 approving votes, 5 of which are binding: * Robert Bradshaw
>>>>> (binding) * Chamikara Jayalath (binding) * Ahmet Altay (binding) * Alexey
>>>>> Romanenko (binding)
>>>>> * Jan Lukavský (binding)
>>>>>
>>>>> * Bruno Volpato (non-binding)
>>>>> * Anand Inguva (non-binding)
>>>>> * Ritesh Ghorse (non-binding)
>>>>> There are no disapproving votes. I will now begin the process of
>>>>> finalizing the release. Thanks everyone!
>>>>>
>>>>> - Danny McCormick
>>>>>
>>>>

Re: [VOTE] Release 2.46.0, release candidate #1

[Ahmet Altay via dev]

Thank you!

Is there a tracking issue for this known issue? And would the known issues
section of the release notes link to that?


On Fri, Mar 10, 2023 at 11:38 AM Danny McCormick via dev <
dev@beam.apache.org> wrote:

> We determined that the same issue exists in the 2.45 release, so we are
> going to continue finalizing the release candidate. Thank you for your
> patience.
>
> Thanks,
> Danny
>
> On Wed, Mar 8, 2023 at 3:15 PM Reuven Lax  wrote:
>
>> We are trying to reproduce and debug the issue we saw to validate whether
>> it was a real regression or not. Will update when we know more.
>>
>> On Wed, Mar 8, 2023 at 11:31 AM Danny McCormick <
>> dannymccorm...@google.com> wrote:
>>
>>>
>>> @Reuven Lax  found a new potential regression in
>>> BigQuery I/O, so I have paused the release rollout. I had already pushed
>>> the Python artifacts and Go tags, but not the Java ones. We have since
>>> temporarily yanked <https://pypi.org/help/#yanked> the Python release
>>> and deleted the Go tags, they were live for around an hour. The possible
>>> regression is in Java, so neither of those releases should be affected, but
>>> x-lang may not work properly because it depends on versioning. I will
>>> update this thread with next steps when we know more.
>>>
>>> Thanks,
>>> Danny
>>> On Wed, Mar 8, 2023 at 5:59 AM Jan Lukavský  wrote:
>>>
>>>> +1 (binding)
>>>>
>>>> Tested Java SDK with Flink and Spark 3 runner.
>>>>
>>>> Thanks,
>>>>  Jan
>>>>
>>>> On 3/8/23 01:53, Valentyn Tymofieiev via dev wrote:
>>>>
>>>> +1. Verified the composition of Python containers and ran Python
>>>> pipelines on Dataflow runner v1 and runner v2.
>>>>
>>>> On Tue, Mar 7, 2023 at 4:11 PM Ritesh Ghorse via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>> Validated Go SDK quickstart on direct and dataflow runner
>>>>>
>>>>> On Tue, Mar 7, 2023 at 10:54 AM Alexey Romanenko <
>>>>> aromanenko@gmail.com> wrote:
>>>>>
>>>>>> +1 (binding)
>>>>>>
>>>>>> Tested with  https://github.com/Talend/beam-samples/
>>>>>> (Java SDK v8/v11/v17, Spark 3.x runner).
>>>>>>
>>>>>> ---
>>>>>> Alexey
>>>>>>
>>>>>> On 7 Mar 2023, at 07:38, Ahmet Altay via dev 
>>>>>> wrote:
>>>>>>
>>>>>> +1 (binding) - I validated python quickstarts on direct & dataflow
>>>>>> runners.
>>>>>>
>>>>>> Thank you for doing the release!
>>>>>>
>>>>>> On Sat, Mar 4, 2023 at 8:01 AM Chamikara Jayalath via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>> Validated multi-language Java and Python pipelines.
>>>>>>>
>>>>>>> On Fri, Mar 3, 2023 at 1:59 PM Danny McCormick via dev <
>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>
>>>>>>>> > I have encountered a failure in a Python pipeline running with
>>>>>>>> Runner v1:
>>>>>>>>
>>>>>>>> > RuntimeError: Beam SDK base version 2.46.0 does not match
>>>>>>>> Dataflow Python worker version 2.45.0. Please check Dataflow worker 
>>>>>>>> startup
>>>>>>>> logs and make sure that correct version of Beam SDK is installed.
>>>>>>>>
>>>>>>>> > We should understand why Python ValidatesRunner tests (which have
>>>>>>>> passed)  didn't catch this error.
>>>>>>>>
>>>>>>>> > This can be remediated in Dataflow containers without  changes to
>>>>>>>> the release candidate.
>>>>>>>>
>>>>>>>> Good catch! I've kicked off a release to fix this, it should be
>>>>>>>> done later this evening - I won't be available when it completes, but I
>>>>>>>> would expect it to be around 5:00 PST.
>>>>>>>>
>>>>>>>> On Fri, Mar 3, 2023 at 3:49 PM Danny McCormick <
>>>

Re: GSOC

[Ahmet Altay via dev]

I do not have much time to be a mentor unfortunately. There are lots of
people on this mailing list who might be interested.  If you can share what
projects you would be interested in working on, someone here might be
interested te become a GSOC mentor.

On Wed, Mar 8, 2023 at 8:49 AM Saifuddin Adenwala 
wrote:

> I just need more favour from you sir that i wanted to participate in GSOC
> by completing a project by Apache beam so sir can you guide me in that ?
> And can become my mentor.
>
> On Wed, Mar 8, 2023, 10:09 PM Ahmet Altay  wrote:
>
>> Hi Saifuddin,
>>
>> It is here: https://github.com/apache/beam/tree/master/website - It is
>> built using Hugo, and README.md has more information for building the
>> website.
>>
>> Thank you for the bug report and the contribution. Feel free to reach out
>> with questions.
>>
>> Ahmet
>>
>> On Wed, Mar 8, 2023 at 8:37 AM Saifuddin Adenwala 
>> wrote:
>>
>>> Greetings !
>>> I just wanted to ask that where's the html file of the apache beam
>>> website so I can fix this bug and make a pull request .
>>>
>>>

Re: [Bug]: The item "Contact Us" displayed is in the Resources block of the footer #22695

[Ahmet Altay via dev]

Hi Saifuddin,

It is here: https://github.com/apache/beam/tree/master/website - It is
built using Hugo, and README.md has more information for building the
website.

Thank you for the bug report and the contribution. Feel free to reach out
with questions.

Ahmet

On Wed, Mar 8, 2023 at 8:37 AM Saifuddin Adenwala 
wrote:

> Greetings !
> I just wanted to ask that where's the html file of the apache beam website
> so I can fix this bug and make a pull request .
>
>

Re: [VOTE] Release 2.46.0, release candidate #1

[Ahmet Altay via dev]

+1 (binding) - I validated python quickstarts on direct & dataflow runners.

Thank you for doing the release!

On Sat, Mar 4, 2023 at 8:01 AM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> +1 (binding)
>
> Validated multi-language Java and Python pipelines.
>
> On Fri, Mar 3, 2023 at 1:59 PM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> > I have encountered a failure in a Python pipeline running with Runner
>> v1:
>>
>> > RuntimeError: Beam SDK base version 2.46.0 does not match Dataflow
>> Python worker version 2.45.0. Please check Dataflow worker startup logs and
>> make sure that correct version of Beam SDK is installed.
>>
>> > We should understand why Python ValidatesRunner tests (which have
>> passed)  didn't catch this error.
>>
>> > This can be remediated in Dataflow containers without  changes to the
>> release candidate.
>>
>> Good catch! I've kicked off a release to fix this, it should be done
>> later this evening - I won't be available when it completes, but I would
>> expect it to be around 5:00 PST.
>>
>> On Fri, Mar 3, 2023 at 3:49 PM Danny McCormick 
>> wrote:
>>
>>> Hey Reuven, could you provide some more context on the bug/why it is
>>> important? Does it meet the standard in
>>> https://beam.apache.org/contribute/release-guide/#7-triage-release-blocking-issues-in-github
>>> ?
>>>
>>> The release branch was cut last Wednesday, so that is why it is not
>>> included.
>>>
>>
> Seems like this was a revert of a previous commit that was also not
> included in the 2.46.0 release branch (
> https://github.com/apache/beam/pull/25627) ?
>
> If so we might not need a new RC but good to confirm.
>
> Thanks,
> Cham
>
>
>>> On Fri, Mar 3, 2023 at 3:24 PM Reuven Lax  wrote:
>>>
 If possible, I would like to see if we could include
 https://github.com/apache/beam/pull/25642 as we believe this bug has
 been impacting multiple users. This was merged 4 days ago, but this RC cut
 does not seem to include it.

 On Fri, Mar 3, 2023 at 12:18 PM Valentyn Tymofieiev via dev <
 dev@beam.apache.org> wrote:

> I have encountered a failure in a Python pipeline running with Runner
> v1:
>
> RuntimeError: Beam SDK base version 2.46.0 does not match Dataflow
> Python worker version 2.45.0. Please check Dataflow worker startup logs 
> and
> make sure that correct version of Beam SDK is installed.
>
> We should understand why Python ValidatesRunner tests (which have
> passed)  didn't catch this error.
>
> This can be remediated in Dataflow containers without  changes to the
> release candidate.
>
> On Fri, Mar 3, 2023 at 11:22 AM Robert Bradshaw via dev <
> dev@beam.apache.org> wrote:
>
>> +1 (binding).
>>
>> I verified that the artifacts and signatures all look good, all the
>> containers are pushed, and tested some pipelines with a fresh install
>> from one of the Python wheels.
>>
>> On Fri, Mar 3, 2023 at 11:13 AM Danny McCormick
>>  wrote:
>> >
>> > > The released artifacts seem to be missing the last commit at
>> > >
>> https://github.com/apache/beam/commit/c528eab18b32342daed53b750fe330d30c7e5224
>> > > . Is this essential to the release, or just useful for validating
>> it?
>> >
>> > It's strictly a test infrastructure change, it has no functional
>> impact. For context, the changes included were from
>> https://github.com/apache/beam/pull/25661 and
>> https://github.com/apache/beam/pull/25654, both were keeping
>> integration tests from running correctly.
>>
>> Thanks.
>>
>> > On Fri, Mar 3, 2023 at 2:09 PM Robert Bradshaw 
>> wrote:
>> >>
>> >> The released artifacts seem to be missing the last commit at
>> >>
>> https://github.com/apache/beam/commit/c528eab18b32342daed53b750fe330d30c7e5224
>> >> . Is this essential to the release, or just useful for validating
>> it?
>> >>
>> >> On Fri, Mar 3, 2023 at 11:02 AM Danny McCormick
>> >>  wrote:
>> >> >
>> >> > Thanks for calling that out, and thanks for helping me fix it!
>> We should be all set now
>> >> >
>> >> > On Fri, Mar 3, 2023 at 1:38 PM Robert Bradshaw <
>> rober...@google.com> wrote:
>> >> >>
>> >> >> It appears your public key is not published in
>> >> >> https://dist.apache.org/repos/dist/release/beam/KEYS .
>> >> >>
>> >> >> On Fri, Mar 3, 2023 at 8:33 AM Anand Inguva via dev <
>> dev@beam.apache.org> wrote:
>> >> >> >
>> >> >> > +1 (non-binding)
>> >> >> > Tested python wordcount quick start
>> https://beam.apache.org/get-started/quickstart-py/ on Direct Runner
>> and Dataflow Runner.
>> >> >> >
>> >> >> > Thanks!
>> >> >> >
>> >> >> > On Fri, Mar 3, 2023 at 11:21 AM Bruno Volpato via dev <
>> dev@beam.apache.org> wrote:
>> >> >> >>
>> >> >> >> +1 (non-binding)
>> >> >> >>
>> >> >> >> 

Re: [ANNOUNCE] New PMC Member: Jan Lukavský

[Ahmet Altay via dev]

Congratulations Jan!

On Fri, Feb 17, 2023 at 4:52 AM Jan Lukavský  wrote:

> Thanks everyone!
>
> This is great honor, I'm grateful for the support of the Apache Beam
> community.
>
> Best,
>
>  Jan
> On 2/17/23 11:15, Shivam Singhal wrote:
>
> Congratulations Jan!
>
> On Fri, 17 Feb 2023 at 14:26, Moritz Mack  wrote:
>
>> Congrats, Jan!
>>
>>
>>
>> On 16.02.23, 23:28, "Luke Cwik via dev"  wrote:
>>
>>
>>
>> Congrats, well deserved. On Thu, Feb 16, 2023 at 10: 32 AM Anand Inguva
>> via dev 
>>  wrote:
>> Congratulations!! On Thu, Feb 16, 2023 at 12: 42 PM Chamikara Jayalath via
>> dev 
>>  wrote: Congrats Jan!On
>>
>> Congrats, well deserved.
>>
>>
>>
>> On Thu, Feb 16, 2023 at 10:32 AM Anand Inguva via dev <
>> dev@beam.apache.org> wrote:
>>
>> Congratulations!!
>>
>>
>>
>> On Thu, Feb 16, 2023 at 12:42 PM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>> Congrats Jan!
>>
>>
>>
>> On Thu, Feb 16, 2023 at 8:35 AM John Casey via dev 
>> wrote:
>>
>> Thanks Jan!
>>
>>
>>
>> On Thu, Feb 16, 2023 at 11:11 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>> Congratulations!
>>
>>
>>
>> On Thu, Feb 16, 2023 at 11:09 AM Reza Rokni via dev 
>> wrote:
>>
>> Congratulations!
>>
>>
>>
>> On Thu, Feb 16, 2023 at 7:47 AM Robert Burke  wrote:
>>
>> Congratulations!
>>
>>
>>
>> On Thu, Feb 16, 2023, 7:44 AM Danielle Syse via dev 
>> wrote:
>>
>> Congrats, Jan! That's awesome news. Thank you for your continued
>> contributions!
>>
>>
>>
>> On Thu, Feb 16, 2023 at 10:42 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming Jan Lukavský <
>> j...@apache.org> as our newest PMC member.
>>
>> Jan has been a part of Beam community and a long time contributor since
>> 2018 in many significant ways, including code contributions in different
>> areas, participating in technical discussions, advocating for users, giving
>> a talk at Beam Summit and even writing one of the few Beam books!
>>
>> Congratulations Jan and thanks for being a part of Apache Beam!
>>
>> ---
>> Alexey
>>
>> *As a recipient of an email from the Talend Group, your personal data
>> will be processed by our systems. Please see our Privacy Notice
>> *for more information about our
>> collection and use of your personal information, our security practices,
>> and your data protection rights, including any rights you may have to
>> object to automated-decision making or profiling we use to analyze support
>> or marketing related communications. To manage or discontinue promotional
>> communications, use the communication preferences portal
>> . To exercise your data
>> protection rights, use the privacy request form
>> .
>> Contact us here or by mail to either of
>> our co-headquarters: Talend, Inc.: 400 South El Camino Real, Ste 1400, San
>> Mateo, CA 94402; Talend SAS: 5/7 rue Salomon De Rothschild, 92150 Suresnes,
>> France
>>
>

Re: [VOTE] Release 2.45.0, Release Candidate #1

[Ahmet Altay via dev]

+1 (binding) - I validated python quick starts on direct runner and python
streaming quickstart on dataflow.

Thank you!

On Mon, Feb 13, 2023 at 5:17 AM Bruno Volpato via dev 
wrote:

> +1 (non-binding)
>
> Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates (Java
> SDK 11, Dataflow runner).
>
>
> Thanks!
>
> On Mon, Feb 13, 2023 at 1:13 AM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
>
>> +1 (binding)
>>
>> Tried several Java and Python multi-language pipelines.
>>
>> Thanks,
>> Cham
>>
>> On Fri, Feb 10, 2023 at 1:52 PM Luke Cwik via dev 
>> wrote:
>>
>>> +1
>>>
>>> Validated release artifact signatures and verified the Java Flink and
>>> Spark quickstarts.
>>>
>>> On Fri, Feb 10, 2023 at 9:27 AM John Casey via dev 
>>> wrote:
>>>
 Addendum to above email.

 Java artifacts were built with Gradle 7.5.1 and OpenJDK 1.8.0_362

 On Fri, Feb 10, 2023 at 11:14 AM John Casey 
 wrote:

> Hi everyone,
> Please review and vote on the release candidate #3 for the version
> 2.45.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> Reviewers are encouraged to test their own use cases with the release
> candidate, and vote +1 if no issues are found.
>
> The complete staging area is available for your review, which includes:
> * GitHub Release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2], which is signed with the key with fingerprint 921F35F5EC5F5DDE [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.45.0-RC1" [5],
> * website pull request listing the release [6], the blog post [6], and
> publishing the API reference manual [7].
> * Java artifacts were built with Gradle GRADLE_VERSION and
> OpenJDK/Oracle JDK JDK_VERSION.
> * Python artifacts are deployed along with the source release to the
> dist.apache.org [2] and PyPI[8].
> * Go artifacts and documentation are available at pkg.go.dev [9]
> * Validation sheet with a tab for 2.45.0release to help with
> validation [10].
> * Docker images published to Docker Hub [11].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> For guidelines on how to try the release in your projects, check out
> our blog post at /blog/validate-beam-release/.
>
> Thanks,
> John Casey
>
> [1] https://github.com/apache/beam/milestone/8
> [2] https://dist.apache.org/repos/dist/dev/beam/2.45.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1293/
> [5] https://github.com/apache/beam/tree/v2.45.0-RC1
> [6] https://github.com/apache/beam/pull/25407
> [7] https://github.com/apache/beam-site/pull/640
> [8] https://pypi.org/project/apache-beam/2.45.0rc1/
> [9]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.45.0-RC1/go/pkg/beam
> [10]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=2030665842
> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>

Re: Community event cooperation

[Ahmet Altay via dev]

Great.

Beam summit submissions are open - https://sessionize.com/beam-summit -- I
am sure we would love to meet and hear about how two projects could be used
together, or two communities collaborate, or learn from your experience in
building a community.

Also, it would be great to see you folks there if more than one person
could attend.

Ahmet

On Wed, Feb 8, 2023 at 11:26 PM 曾辉  wrote:

> Thank you for the information!
>
> Great suggestion, I think, maybe we can submit an issue? If registration
> is still open? Or we have people from the community go to this summit,
> which is doable.
>
> Regards,
>
> Hui Zeng | Community Manager
> M: +86 18819063834 <+86%20188%201906%203834>
>
> Apache DolphinScheduler Committer
> zeng...@apache.org
> https://twitter.com/Niko_Zeng
>
>
> Ahmet Altay  于2023年2月9日周四 11:53写道：
>
>> Hi!
>>
>> Thank you for the follow up.
>>
>> Brittany is still part of this community but as you pointed she would
>> probably have less time for Beam after the changes at Google. I assume the
>> best contact will be the mailing list (dev@), and Danielle (who you
>> added) here is also still actively working on community engagement.
>>
>> And great to hear that you have a person working in the United States
>> too.
>>
>> If I remember correctly, we were not able to clearly identify use cases
>> where DolphinScheduler & Beam are used together and after that it was hard
>> to identify joint activities. Some concrete things we could do are a person
>> from DolphinScheduler could participate in the Beam summit (June 13 - 15:
>> https://beamsummit.org/). We could coordinate a meetup. Perhaps the
>> DolphinScheduler person US could help with that. If they organize a meetup
>> in a place local to one of the Beam community members they could
>> participate. If you have any other concrete ideas please share.
>>
>> Ahmet
>>
>>
>> On Wed, Feb 8, 2023 at 5:19 PM 曾辉  wrote:
>>
>>> Hi Ahmet，
>>>
>>> It's a pity that Google's recent personnel changes, including Brittany's
>>> post on LinkedIn, hope everyone is well, I don't know what happened, I
>>> haven't received feedback from the Beam community, and we have always hoped
>>> to cooperate with the Beam community Get cooperation so that users from
>>> both parties have a gathering place. Obviously, I am a little unclear about
>>> who would be better to contact now. If you feel it is unsuitable, you can
>>> also email me privately, and I will contact you in the future. Another good
>>> news is that we have found an evangelist in the United States. If there are
>>> offline activities, we can also support them!
>>>
>>> Regards,
>>>
>>> Hui Zeng | Community Manager
>>> M: +86 18819063834 <+86%20188%201906%203834>
>>>
>>> Apache DolphinScheduler Committer
>>> zeng...@apache.org
>>> https://twitter.com/Niko_Zeng
>>>
>>>
>>> Ahmet Altay via dev  于2022年8月9日周二 07:03写道：
>>>
>>>> Hi Niko,
>>>>
>>>> Thank you for reaching out. We do have contributors who might be
>>>> interested in participating but they might have limited time for
>>>> participating in an event. If you can clarify what are you looking for
>>>> (e.g. speaker? help with coordination? etc.) people might be able to give
>>>> you a better answer.
>>>>
>>>> Another note, we have been organizing meetups for a while (with @Danielle
>>>> Syse  and @Brittany Hermann  doing
>>>> the recent organizations), we could also see if someone from
>>>> DolphinScheduler could participate as a speaker in one of those.
>>>>
>>>> A question for my learning, I am not familiar with DolphinScheduler.
>>>> Are there users or potential use cases where DolphinScheduler & Beam are
>>>> used together?
>>>>
>>>> Thank you!
>>>> Ahmet
>>>>
>>>> On Sun, Aug 7, 2022 at 6:13 PM 曾辉  wrote:
>>>>
>>>>> Anyone interested?
>>>>>
>>>>> 曾辉  于2022年8月4日周四 16:23写道：
>>>>>
>>>>>> Hey， Developers in the Apache Beam community, How's your day?
>>>>>>
>>>>>>  I'm Apache DolphinScheduler Community Manager, you can call me Niko,
>>>>>> nice to meet you all
>>>>>>
>>>>>> Apache DolphinScheduler is a worldly renowned data orchestration tool
>>>>>> that has largely taken t

Re: Community event cooperation

[Ahmet Altay via dev]

Hi!

Thank you for the follow up.

Brittany is still part of this community but as you pointed she would
probably have less time for Beam after the changes at Google. I assume the
best contact will be the mailing list (dev@), and Danielle (who you added)
here is also still actively working on community engagement.

And great to hear that you have a person working in the United States too.

If I remember correctly, we were not able to clearly identify use cases
where DolphinScheduler & Beam are used together and after that it was hard
to identify joint activities. Some concrete things we could do are a person
from DolphinScheduler could participate in the Beam summit (June 13 - 15:
https://beamsummit.org/). We could coordinate a meetup. Perhaps the
DolphinScheduler person US could help with that. If they organize a meetup
in a place local to one of the Beam community members they could
participate. If you have any other concrete ideas please share.

Ahmet


On Wed, Feb 8, 2023 at 5:19 PM 曾辉  wrote:

> Hi Ahmet，
>
> It's a pity that Google's recent personnel changes, including Brittany's
> post on LinkedIn, hope everyone is well, I don't know what happened, I
> haven't received feedback from the Beam community, and we have always hoped
> to cooperate with the Beam community Get cooperation so that users from
> both parties have a gathering place. Obviously, I am a little unclear about
> who would be better to contact now. If you feel it is unsuitable, you can
> also email me privately, and I will contact you in the future. Another good
> news is that we have found an evangelist in the United States. If there are
> offline activities, we can also support them!
>
> Regards,
>
> Hui Zeng | Community Manager
> M: +86 18819063834 <+86%20188%201906%203834>
>
> Apache DolphinScheduler Committer
> zeng...@apache.org
> https://twitter.com/Niko_Zeng
>
>
> Ahmet Altay via dev  于2022年8月9日周二 07:03写道：
>
>> Hi Niko,
>>
>> Thank you for reaching out. We do have contributors who might be
>> interested in participating but they might have limited time for
>> participating in an event. If you can clarify what are you looking for
>> (e.g. speaker? help with coordination? etc.) people might be able to give
>> you a better answer.
>>
>> Another note, we have been organizing meetups for a while (with @Danielle
>> Syse  and @Brittany Hermann  doing
>> the recent organizations), we could also see if someone from
>> DolphinScheduler could participate as a speaker in one of those.
>>
>> A question for my learning, I am not familiar with DolphinScheduler. Are
>> there users or potential use cases where DolphinScheduler & Beam are used
>> together?
>>
>> Thank you!
>> Ahmet
>>
>> On Sun, Aug 7, 2022 at 6:13 PM 曾辉  wrote:
>>
>>> Anyone interested?
>>>
>>> 曾辉  于2022年8月4日周四 16:23写道：
>>>
>>>> Hey， Developers in the Apache Beam community, How's your day?
>>>>
>>>>  I'm Apache DolphinScheduler Community Manager, you can call me Niko,
>>>> nice to meet you all
>>>>
>>>> Apache DolphinScheduler is a worldly renowned data orchestration tool
>>>> that has largely taken the scheduler market in China. Over 1000 companies,
>>>> including IBM, Tencent, iFlytek, Meituan, 360, China Unicom, Shein, and SF
>>>> Express, are relying on its decentralized infrastructure and no-code DAG
>>>> interface. Apache DolphinScheduler also owns the largest developer
>>>> community in China and each meetup gathers over 3K attendees.
>>>>
>>>> We would love to find partners like Apache Beam to co-host events in
>>>> the Bay Area, to share our resources with fellow Apache teams.
>>>>
>>>> click the link
>>>> <https://docs.google.com/document/d/1ZqVbBAdqZPt_6D0gzfWUkr0KCVUzlxIToMqT76yCBL8/edit#>
>>>> is an introduction to our community programs. If you are interested in
>>>> becoming our partner and holding a Meetup together, please contact me in
>>>> the mail, or schedule a zoom call to discuss the details sometime next
>>>> week.   looking forward to your reply.
>>>>
>>>> I believe that combining the influence of our two sides can let more
>>>> people know about the Apache Beam open source project.
>>>>
>>>> Best，
>>>> Niko
>>>>
>>>

Re: Incorrect python bigquery storage read API docs

[Ahmet Altay via dev]

Adding @Pablo Estrada  who reviewed that issue,
hopefully he could help.

On Sun, Feb 5, 2023 at 4:37 PM Parth Doshi  wrote:

> Hi,
>
> I was reading through the documentation for the python bigquery read API
> 
>  and
> it says
> """
> The Beam SDK for Python does not support the BigQuery Storage API. See
> Issue 20687).
> """
>
> However, that issue has since been closed
>  and it seems it is now
> supported.
>
> Best,
> Parth
>

Re: Beam Website Feedback

[Ahmet Altay via dev]

Adding @John Casey  @Bruno Volpato
 - who might be able to point to relevant docs.

On Sat, Feb 4, 2023 at 11:59 AM Shlomi Elbaz  wrote:

> Hello All,
>
>
>
> We developed a service with Apache Beam where we read an Avro file that
> locate in GCP bucket,
>
> We had a load and benchmark tests, during the pipeline we got a bottleneck
> and *out-of-memory* issues in the stage where the service accesses the
> Avro’s by AvroIO.*parseFilesGenericRecords*
>
>
>
> The issue happened in highlight part:
>
> .apply("Match file names", FileIO.*matchAll*())
> .apply("Read Avro files", FileIO.*readMatches*())
> *.apply**(**"Parse Avro files into GenericRecord"**, **AvroIO**.*
> *parseFilesGenericRecords**(**new **CustomerTransformFn**()*
> *) **.withCoder**(**KvCoder**.**of**(**Customer**.**keyCoder**()**,
> **Customer**.**valueCoder**()**)**)**)*
> .apply("Chunk customer", GroupIntoBatches.*ofSize*(size)
> .withMaxBufferingDuration(Duration.*standardSeconds*(duration)))
>
>
>
> Issues we saw a tutorial regarding resource-hints in Apache Beam website,
> but there is no examples/information how to use with *AvroIO**.*
> *parseFilesGenericRecords*.
>
> https://beam.apache.org/documentation/runtime/resource-hints/
>
>
>
> is there more information or examples where we can read about ResourceHints
> and Accelerator’s?
>
>
>
> Also, would you please recommend us for optimal settings of using
> ResourceHints?
>
>
>
> The additional tutorials that we rely on:
>
> https://www.youtube.com/watch?v=9fc2MNQHQ2s
>
> https://cloud.google.com/dataflow/docs/guides/right-fitting
>
>
> https://cloud.google.com/blog/products/data-analytics/introducing-vertical-autoscaling-in-dataflow-prime
>
>
>
> Thanks,
>
>
>
> Shlomi Elbaz,
>
>
>
>
>
> ---
> Optimove Named a Leader in the 2022 IDC MarketScape for Retail CDP -
> 
> Download
> report here
> 
>
> Say Hello to Optitext - Optimove Adds Native SMS Capabilities-
> 
> 
> read
> about it here
> 
>
> ---
>
> *Shlomi Elbaz*
> Fullstack Developer
>
>
> 
> CRM Journeys, Mapped by AI
>
> Connect with us on LinkedIn  |
> Twitter  | Facebook
>  | Youtube
> 
> Read our thoughts on the Optimove Blog
> 
>
>
>

Re: [Peer Review] Pull Request Eliminate Finalize in Read

[Ahmet Altay via dev]

Thank you for reaching out. Adding @Yi Hu  for his input.
It looks like Yi was reviewing the PR back in December.

On Fri, Jan 27, 2023 at 1:52 AM Dejan Spasic 
wrote:

> Hello team,
>
> I posted a pull request[0] some time ago where I (we) need your support.
> In general, it is about the elimination of the finalize methods or the
> replacement of the finalize methods[1]. I am looking forward to your
> advice.
>
> Have a nice one!
>
> [0] https://github.com/apache/beam/pull/24841
> [1] https://github.com/apache/beam/issues/24181
>

Re: [VOTE] Release 2.44.0, release candidate #1

[Ahmet Altay via dev]

+1 (binding) - my concerns are addressed.

On Wed, Jan 11, 2023 at 3:44 PM Valentyn Tymofieiev 
wrote:

> +1. I validated that Dataflow and Beam Python containers include necessary
> dependencies of Apache Beam and did additional validation (see inline).
>
> On Wed, Jan 11, 2023 at 12:48 AM Ahmet Altay  wrote:
>
>> I validated python quick starts (direct, dataflow) X (batch, streaming).
>> I ran into an issue with the dataflow batch case, running the wordcount
>> with the standard:
>>
>> python -m apache_beam.examples.wordcount \
>> --output  \
>> --staging_location  \
>> --temp_location \
>> --runner DataflowRunner \
>> --job_name wordcount-$USER \
>> --project  \
>> --num_workers 1 \
>> --region us-central1 \
>> --sdk_location apache-beam-2.44.0.zip
>>
>> results in:
>>
>> "/usr/local/lib/python3.10/site-packages/dataflow_worker/shuffle.py",
>> line 589, in __enter__ raise
>> RuntimeError(_PYTHON_310_SHUFFLE_ERROR_MESSAGE) RuntimeError: This pipeline
>> requires Dataflow Runner v2 in order to run with currently used version of
>> Apache Beam on Python 3.10+. Please verify that the Dataflow Runner v2 is
>> not disabled in the pipeline options or enable it explicitly via:
>> --dataflow_service_option=use_runner_v2. Alternatively, downgrade to Python
>> 3.9 to use Dataflow Runner v1.
>>
>> Questions:
>> - I am not explicitly opting out of runner v2, and this is a standard
>> wordcount example, I expected it to just work.
>>
> You are most likely using a google-internal project for which Runner v2 is
> explicitly disabled, to enable  Runner v1 test coverage within Google. I
> can repro this error as well (maybe on the same project as you), but don't
> repro it on other projects, such as apache-beam-testing. Runner v1 is not
> supported on Python 3.10 (this is documented).
>
> Such behavior is WAI as far as Beam is concerned and the difference is due
> to configuration details in Dataflow.
>

Yes, I was using a Google owned project for testing. That makes sense. And
it is fine as long as it is not a user visible behavior. Thank you for the
clarification.

>
>
>
>>
>> Then I tried to add --dataflow_service_option=use_runner_v2 to the above
>> wordcount command, which results in the following error:
>>
>> "message": "Dataflow Runner v2 requires a valid FnApi job, Please
>> resubmit your job with a valid configuration. Note that if using Templates,
>> you may need to regenerate your template with the '--use_runner_v2'."
>>
>> Maybe I am doing something wrong and it is an error on my end. It would
>> be good for someone else with python experience to check this.
>>
>> /cc @Valentyn Tymofieiev 
>>
>> Ahmet
>>
>>
>>
>>
>> On Tue, Jan 10, 2023 at 10:54 AM Kenneth Knowles  wrote:
>>
>>> I have published a new maven staging repository:
>>> https://repository.apache.org/content/repositories/orgapachebeam-1290/
>>>
>>> It looks like it has everything, though I did not automate a check. At
>>> least there were no errors during publish which I ran with --no-parallel
>>> overnight, and some specific things that were missing from
>>> orgapachebeam-1289 are present.
>>>
>>> I will restart the 72 hour waiting period, since the RC is only now
>>> usable.
>>>
>>> Kenn
>>>
>>> On Mon, Jan 9, 2023 at 6:51 PM Kenneth Knowles  wrote:
>>>
>>>> I have discovered that many pom files are missing from the nexus
>>>> repository. I should be able to re-publish a new one. It will take some
>>>> time as this is one of the longest-running processes.
>>>>
>>>> On Mon, Jan 9, 2023 at 1:42 PM Kenneth Knowles  wrote:
>>>>
>>>>> Correction: this is release candidate #1.
>>>>>
>>>>> On Mon, Jan 9, 2023 at 1:25 PM Kenneth Knowles 
>>>>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> Please review and vote on the release candidate #3 for the version
>>>>>> 2.44.0, as follows:
>>>>>> [ ] +1, Approve the release
>>>>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>>>>
>>>>>> Reviewers are encouraged to test their own use cases with the release
>>>>>> candidate, and vote +1 if
>>>>>> no issues are found.
>>>>>>
>>>>>> The complete staging area is availab

Re: [VOTE] Release 2.44.0, release candidate #1

[Ahmet Altay via dev]

I validated python quick starts (direct, dataflow) X (batch, streaming). I
ran into an issue with the dataflow batch case, running the wordcount with
the standard:

python -m apache_beam.examples.wordcount \
--output  \
--staging_location  \
--temp_location \
--runner DataflowRunner \
--job_name wordcount-$USER \
--project  \
--num_workers 1 \
--region us-central1 \
--sdk_location apache-beam-2.44.0.zip

results in:

"/usr/local/lib/python3.10/site-packages/dataflow_worker/shuffle.py", line
589, in __enter__ raise RuntimeError(_PYTHON_310_SHUFFLE_ERROR_MESSAGE)
RuntimeError: This pipeline requires Dataflow Runner v2 in order to run
with currently used version of Apache Beam on Python 3.10+. Please verify
that the Dataflow Runner v2 is not disabled in the pipeline options or
enable it explicitly via: --dataflow_service_option=use_runner_v2.
Alternatively, downgrade to Python 3.9 to use Dataflow Runner v1.

Questions:
- I am not explicitly opting out of runner v2, and this is a standard
wordcount example, I expected it to just work.

Then I tried to add --dataflow_service_option=use_runner_v2 to the above
wordcount command, which results in the following error:

"message": "Dataflow Runner v2 requires a valid FnApi job, Please
resubmit your job with a valid configuration. Note that if using Templates,
you may need to regenerate your template with the '--use_runner_v2'."

Maybe I am doing something wrong and it is an error on my end. It would be
good for someone else with python experience to check this.

/cc @Valentyn Tymofieiev 

Ahmet




On Tue, Jan 10, 2023 at 10:54 AM Kenneth Knowles  wrote:

> I have published a new maven staging repository:
> https://repository.apache.org/content/repositories/orgapachebeam-1290/
>
> It looks like it has everything, though I did not automate a check. At
> least there were no errors during publish which I ran with --no-parallel
> overnight, and some specific things that were missing from
> orgapachebeam-1289 are present.
>
> I will restart the 72 hour waiting period, since the RC is only now usable.
>
> Kenn
>
> On Mon, Jan 9, 2023 at 6:51 PM Kenneth Knowles  wrote:
>
>> I have discovered that many pom files are missing from the nexus
>> repository. I should be able to re-publish a new one. It will take some
>> time as this is one of the longest-running processes.
>>
>> On Mon, Jan 9, 2023 at 1:42 PM Kenneth Knowles  wrote:
>>
>>> Correction: this is release candidate #1.
>>>
>>> On Mon, Jan 9, 2023 at 1:25 PM Kenneth Knowles  wrote:
>>>
 Hi everyone,

 Please review and vote on the release candidate #3 for the version
 2.44.0, as follows:
 [ ] +1, Approve the release
 [ ] -1, Do not approve the release (please provide specific comments)

 Reviewers are encouraged to test their own use cases with the release
 candidate, and vote +1 if
 no issues are found.

 The complete staging area is available for your review, which includes:
 * GitHub Release notes [1],
 * the official Apache source release to be deployed to dist.apache.org
 [2], which is signed with the key with fingerprint 6ED551A8AE02461C [3],
 * all artifacts to be deployed to the Maven Central Repository [4],
 * source code tag "v2.44.0-RC1" [5],
 * website pull request listing the release [6], the blog post [6], and
 publishing the API reference manual [7].
 * Java artifacts were built with Gradle 7.5.1 and OpenJDK 1.8.0_232.
 * Python artifacts are deployed along with the source release to the
 dist.apache.org [2] and PyPI [8].
 * Go artifacts and documentation are available at pkg.go.dev [9]
 (waiting on these to appear)
 * Validation sheet with a tab for 2.44.0 release to help with
 validation [10].
 * Docker images published to Docker Hub [11].

 The vote will be open for at least 72 hours. It is adopted by majority
 approval, with at least 3 PMC affirmative votes.

 For guidelines on how to try the release in your projects, check out
 our blog post at /blog/validate-beam-release/.

 Thanks,
 Kenn

 [1] https://github.com/apache/beam/milestone/7
 [2] https://dist.apache.org/repos/dist/dev/beam/2.44.0/
 [3] https://dist.apache.org/repos/dist/release/beam/KEYS
 [4]
 https://repository.apache.org/content/repositories/orgapachebeam-1289/
 [5] https://github.com/apache/beam/tree/v2.44.0-RC1
 [6] https://github.com/apache/beam/pull/24951
 [7] https://github.com/apache/beam-site/pull/638
 [8] https://pypi.org/project/apache-beam/2.44.0rc1/
 [9]
 https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.44.0-RC1/go/pkg/beam
 [10]
 https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=622267910
 [11] https://hub.docker.com/search?q=apache%2Fbeam=image

>>>

Re: Beam Website Feedback

[Ahmet Altay via dev]

Thank you: Alexey Romanenko, David Huntsperger, Pablo Estrada, Alya Boiko!
:)

On Fri, Jan 6, 2023 at 12:54 PM Alex Kosolapov 
wrote:

> Thank you, Ahmet! Happy to help! Both changes [1] and [2] have been
> reviewed and merged by Alexey Romanenko.
>
>
>
> We wanted to thank Alexey Romanenko, David Huntsperger, Pablo Estrada,
> Alya Boiko for reviewing and helping to contribute 52 enhancements, fixes
> and case study related additions for the Beam website in the last 6 months
> since July’22! [3]
>
>
>
> [1] https://github.com/apache/beam/pull/1
>
> [2] https://github.com/apache/beam/pull/24747
>
> [3]
> https://github.com/apache/beam/pulls?page=1=is%3Apr+author%3Abullet03+is%3Aclosed+merged%3A%3E%3D2022-07-01
>
>
>
> *From: *Ahmet Altay 
> *Date: *Tuesday, January 3, 2023 at 2:22 PM
> *To: *Alex Kosolapov 
> *Cc: *"dev@beam.apache.org" , Rebecca Szper <
> rsz...@google.com>, Bulat Safiullin , Alexey
> Romanenko , Rajkumar Gupta <
> rajkumargu...@google.com>
> *Subject: *[EXTERNAL] Re: Beam Website Feedback
>
>
>
> Thank you Alex and Bulat for improving this. We all very much appreciate
> it.
>
>
>
> On Thu, Dec 22, 2022 at 9:21 AM Alex Kosolapov 
> wrote:
>
> Hi all,
>
>
>
> We were preparing some improvements for check-links.sh script that is used
> for testing Apache Beam website links during the website build with Bulat (
> @bullet03 <https://github.com/bullet03>).
>
>
>
> We saw several categories of link checks and error statuses:
>
>- 404 - actual incorrect links - fixed in [1] and [2]
>- Valid links that appear to the script as incorrect, e..g., 9xx
>status code for LinkedIn requiring authentication in LinkedIn, some GitHub
>documentation links, example links, some Meetup links, etc.
>
>
>
> We propose to add a “verified_list” to check_links.sh so that manually
> verified links can be skipped in testing. Current verified list includes 15
> links based on review of most recent test review. Inconvenience of this
> approach is that a verified link may become outdated, and would require an
> update of the “verified_list” in check_links.sh. This approach implemented
> in [3].
>
>
>
> [3] also contains check-links.sh improvements:
>
>- Added a function that checks and reports Apache Beam staging website
>links to prevent the production website from having links to staging
>- Added script checks and reports Apache Beam website absolute links
>(links of the form https://beam.apache.org/path) - relative links in
>the sources are preferred to properly build and review website staging
>- Added sorting any invalid links by their error code - this may be
>more convenient for reviewing output
>
>
>
> [4] - optionally, update absolute links to relative links so that a
> staging website more closely resembles the production website
>
>
>
> We submitted [3] and [4] for PR review and tagged Alexey Romanenko to
> kindly help with reviewing these PRs. Please share your comments about
> proposed approach in the PRs or list.
>
>
>
> [1] https://github.com/apache/beam/pull/24635
>
> [2] https://github.com/apache/beam/pull/24744
>
> [3] https://github.com/apache/beam/pull/1
>
> [4] https://github.com/apache/beam/pull/24747
>
>
>
> Thank you,
>
> Alex
>
>
>
> *From: *Rebecca Szper via dev 
> *Reply-To: *"dev@beam.apache.org" , Rebecca Szper <
> rsz...@google.com>
> *Date: *Wednesday, December 21, 2022 at 10:15 AM
> *To: *Ahmet Altay 
> *Cc: *Alexey Romanenko , dev <
> dev@beam.apache.org>, Rajkumar Gupta 
> *Subject: *[EXTERNAL] Re: Beam Website Feedback
>
>
>
> Our team doesn't maintain the Beam website infrastructure, but last time
> something like this came up, David said that there are consultants that
> work on this type of thing. He pinged @bullet03
> <https://github.com/bullet03> on the Beam ticket, who was able to help.
>
>
>
> On Tue, Dec 20, 2022 at 5:06 PM Ahmet Altay  wrote:
>
>
>
>
>
> On Tue, Dec 20, 2022 at 1:12 PM Ahmet Altay  wrote:
>
>
>
>
>
> On Tue, Dec 20, 2022 at 9:14 AM Alexey Romanenko 
> wrote:
>
> Thanks Ahmet! I’d prefer to fix the links as you did and add the redirect
> from old one - perhaps, there are other similar links that have been
> changed in the same way.
>
>
>
> Thank you for the review. I fixed it, and added a redirect too.
>
>
>
>
>
> Btw, I’m not sure that we still check the broken links as it was before,
> iirc, but probably it would be a good idea to add such check before
> publishing a website.
>
>
>

Re: Beam Website Feedback

[Ahmet Altay via dev]

Thank you Alex and Bulat for improving this. We all very much appreciate it.

On Thu, Dec 22, 2022 at 9:21 AM Alex Kosolapov 
wrote:

> Hi all,
>
>
>
> We were preparing some improvements for check-links.sh script that is used
> for testing Apache Beam website links during the website build with Bulat (
> @bullet03 <https://github.com/bullet03>).
>
>
>
> We saw several categories of link checks and error statuses:
>
>- 404 - actual incorrect links - fixed in [1] and [2]
>- Valid links that appear to the script as incorrect, e..g., 9xx
>status code for LinkedIn requiring authentication in LinkedIn, some GitHub
>documentation links, example links, some Meetup links, etc.
>
>
>
> We propose to add a “verified_list” to check_links.sh so that manually
> verified links can be skipped in testing. Current verified list includes 15
> links based on review of most recent test review. Inconvenience of this
> approach is that a verified link may become outdated, and would require an
> update of the “verified_list” in check_links.sh. This approach implemented
> in [3].
>
>
>
> [3] also contains check-links.sh improvements:
>
>- Added a function that checks and reports Apache Beam staging website
>links to prevent the production website from having links to staging
>- Added script checks and reports Apache Beam website absolute links
>(links of the form https://beam.apache.org/path) - relative links in
>the sources are preferred to properly build and review website staging
>- Added sorting any invalid links by their error code - this may be
>more convenient for reviewing output
>
>
>
> [4] - optionally, update absolute links to relative links so that a
> staging website more closely resembles the production website
>
>
>
> We submitted [3] and [4] for PR review and tagged Alexey Romanenko to
> kindly help with reviewing these PRs. Please share your comments about
> proposed approach in the PRs or list.
>
>
>
> [1] https://github.com/apache/beam/pull/24635
>
> [2] https://github.com/apache/beam/pull/24744
>
> [3] https://github.com/apache/beam/pull/1
>
> [4] https://github.com/apache/beam/pull/24747
>
>
>
> Thank you,
>
> Alex
>
>
>
> *From: *Rebecca Szper via dev 
> *Reply-To: *"dev@beam.apache.org" , Rebecca Szper <
> rsz...@google.com>
> *Date: *Wednesday, December 21, 2022 at 10:15 AM
> *To: *Ahmet Altay 
> *Cc: *Alexey Romanenko , dev <
> dev@beam.apache.org>, Rajkumar Gupta 
> *Subject: *[EXTERNAL] Re: Beam Website Feedback
>
>
>
> Our team doesn't maintain the Beam website infrastructure, but last time
> something like this came up, David said that there are consultants that
> work on this type of thing. He pinged @bullet03
> <https://github.com/bullet03> on the Beam ticket, who was able to help.
>
>
>
> On Tue, Dec 20, 2022 at 5:06 PM Ahmet Altay  wrote:
>
>
>
>
>
> On Tue, Dec 20, 2022 at 1:12 PM Ahmet Altay  wrote:
>
>
>
>
>
> On Tue, Dec 20, 2022 at 9:14 AM Alexey Romanenko 
> wrote:
>
> Thanks Ahmet! I’d prefer to fix the links as you did and add the redirect
> from old one - perhaps, there are other similar links that have been
> changed in the same way.
>
>
>
> Thank you for the review. I fixed it, and added a redirect too.
>
>
>
>
>
> Btw, I’m not sure that we still check the broken links as it was before,
> iirc, but probably it would be a good idea to add such check before
> publishing a website.
>
>
>
> I agree. I also do not know about the state of this. It would be good to
> add that links checker again.
>
>
>
> Adding @Rebecca Szper  - in case this is something she
> can fix or would know who could fix it.
>
>
>
>
>
>
>
> —
>
> Alexey
>
>
>
>
>
>
>
> On 20 Dec 2022, at 18:04, Ahmet Altay via dev  wrote:
>
>
>
> I did a search and found a few places with the broken link. Correct links
> should be:
> https://beam.apache.org/get-started/resources/videos-and-podcasts/
>
>
>
> I created a PR to update the website (
> https://github.com/apache/beam/pull/24733). I do not know if that is the
> best solution. As an alternative we could consider setting up a redirect
> for the old link. We do not know who else would be still linking to the old
> one.
>
>
>
> Ahmet
>
>
>
> On Tue, Dec 20, 2022 at 8:52 AM Alexey Romanenko 
> wrote:
>
> Hi Rajkumar,
>
>
>
> Could you specify where (which page) this link was found?
>
> Thanks!
>
>
>
> —
>
> Alexey
>
>
>
> On 20 Dec 2022, at 10:08, Rajkumar Gupta via dev 
> wrote:
>
>
>
> Hi Team,
>
>
>
> Just a minor point, while browsing the site I noticed that the link below
> is not working. Can you please check?
>
> https://beam.apache.org/documentation/resources/videos-and-podcasts
>
>
>
> Regards,
>
> Raj
>
>
>
> --
>
> Rajkumar Gupta | Technical Solutions Engineer - Google Cloud
> Platform | rajkumargu...@google.com  | +91-9223541460
> <+91%2092235%2041460>
>
>
>
>
>
>

Fwd: Beam Java SDK - ReadableState.read() shouldn't it be Nullable?

[Ahmet Altay via dev]

Forwarding, because this message got lost in the list moderation.

-- Forwarded message --
From: Jeeno Lentin 
To: dev@beam.apache.org
Cc:
Bcc:
Date: Fri, 23 Dec 2022 00:36:55 -0500
Subject: Beam Java SDK - ReadableState.read() shouldn't it be Nullable?
Hi,

We use the Beam Java SDK and are trying to upgrade version of Beam from
version 2.31.0 to 2.43.0

While upgrading, we noticed that @Nullable annotation has been removed from
org.apache.beam.sdk.state.ReadableState.read()

I traced it back to this PR: https://github.com/apache/beam/pull/16721

We have the following concerns
- If ReadableState.read() is really not nullable, shouldn’t there be a way
to specify a default value when creating a state? Such a feature doesn’t
seem to exist.
- And what would it return initially when nothing is written to the state
yet initially?

Thank you,
Jeeno

Fwd: MODERATE for dev@beam.apache.org

[Ahmet Altay via dev]

Forwarding, because this message got lost in the list moderation.

-- Forwarded message --
From: bharataamer...@icloud.com
To: dev@beam.apache.org
Cc:
Bcc:
Date: Sat, 24 Dec 2022 13:36:09 +0530
Subject: Wanna contribute to this project guide where to start
Hi,

I am new to this project and wanna contribute to this project
Can you guide me from where to start.

Re: Beam Website Feedback

[Ahmet Altay via dev]

On Wed, Dec 21, 2022 at 10:04 AM Rebecca Szper  wrote:

> Our team doesn't maintain the Beam website infrastructure, but last time
> something like this came up, David said that there are consultants that
> work on this type of thing. He pinged @bullet03
> <https://github.com/bullet03> on the Beam ticket, who was able to help.
>

Makes sense. Thank you Rebecca.


>
> On Tue, Dec 20, 2022 at 5:06 PM Ahmet Altay  wrote:
>
>>
>>
>> On Tue, Dec 20, 2022 at 1:12 PM Ahmet Altay  wrote:
>>
>>>
>>>
>>> On Tue, Dec 20, 2022 at 9:14 AM Alexey Romanenko <
>>> aromanenko@gmail.com> wrote:
>>>
>>>> Thanks Ahmet! I’d prefer to fix the links as you did and add the
>>>> redirect from old one - perhaps, there are other similar links that have
>>>> been changed in the same way.
>>>>
>>>
>>> Thank you for the review. I fixed it, and added a redirect too.
>>>
>>>
>>>>
>>>> Btw, I’m not sure that we still check the broken links as it was
>>>> before, iirc, but probably it would be a good idea to add such check before
>>>> publishing a website.
>>>>
>>>
>>> I agree. I also do not know about the state of this. It would be good to
>>> add that links checker again.
>>>
>>
>> Adding @Rebecca Szper  - in case this is something
>> she can fix or would know who could fix it.
>>
>>
>>>
>>>
>>>>
>>>> —
>>>> Alexey
>>>>
>>>>
>>>>
>>>> On 20 Dec 2022, at 18:04, Ahmet Altay via dev 
>>>> wrote:
>>>>
>>>> I did a search and found a few places with the broken link. Correct
>>>> links should be:
>>>> https://beam.apache.org/get-started/resources/videos-and-podcasts/
>>>>
>>>> I created a PR to update the website (
>>>> https://github.com/apache/beam/pull/24733). I do not know if that is
>>>> the best solution. As an alternative we could consider setting up a
>>>> redirect for the old link. We do not know who else would be still linking
>>>> to the old one.
>>>>
>>>> Ahmet
>>>>
>>>> On Tue, Dec 20, 2022 at 8:52 AM Alexey Romanenko <
>>>> aromanenko@gmail.com> wrote:
>>>>
>>>>> Hi Rajkumar,
>>>>>
>>>>> Could you specify where (which page) this link was found?
>>>>> Thanks!
>>>>>
>>>>> —
>>>>> Alexey
>>>>>
>>>>> On 20 Dec 2022, at 10:08, Rajkumar Gupta via dev 
>>>>> wrote:
>>>>>
>>>>> Hi Team,
>>>>>
>>>>> Just a minor point, while browsing the site I noticed that the link
>>>>> below is not working. Can you please check?
>>>>> https://beam.apache.org/documentation/resources/videos-and-podcasts
>>>>>
>>>>> Regards,
>>>>> Raj
>>>>>
>>>>> --
>>>>> Rajkumar Gupta | Technical Solutions Engineer - Google Cloud
>>>>> Platform | rajkumargu...@google.com  |
>>>>> +91-9223541460 <+91%2092235%2041460>
>>>>>
>>>>>
>>>>>
>>>>

Re: Beam Website Feedback

[Ahmet Altay via dev]

On Tue, Dec 20, 2022 at 1:12 PM Ahmet Altay  wrote:

>
>
> On Tue, Dec 20, 2022 at 9:14 AM Alexey Romanenko 
> wrote:
>
>> Thanks Ahmet! I’d prefer to fix the links as you did and add the redirect
>> from old one - perhaps, there are other similar links that have been
>> changed in the same way.
>>
>
> Thank you for the review. I fixed it, and added a redirect too.
>
>
>>
>> Btw, I’m not sure that we still check the broken links as it was before,
>> iirc, but probably it would be a good idea to add such check before
>> publishing a website.
>>
>
> I agree. I also do not know about the state of this. It would be good to
> add that links checker again.
>

Adding @Rebecca Szper  - in case this is something she
can fix or would know who could fix it.


>
>
>>
>> —
>> Alexey
>>
>>
>>
>> On 20 Dec 2022, at 18:04, Ahmet Altay via dev 
>> wrote:
>>
>> I did a search and found a few places with the broken link. Correct links
>> should be:
>> https://beam.apache.org/get-started/resources/videos-and-podcasts/
>>
>> I created a PR to update the website (
>> https://github.com/apache/beam/pull/24733). I do not know if that is the
>> best solution. As an alternative we could consider setting up a redirect
>> for the old link. We do not know who else would be still linking to the old
>> one.
>>
>> Ahmet
>>
>> On Tue, Dec 20, 2022 at 8:52 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> Hi Rajkumar,
>>>
>>> Could you specify where (which page) this link was found?
>>> Thanks!
>>>
>>> —
>>> Alexey
>>>
>>> On 20 Dec 2022, at 10:08, Rajkumar Gupta via dev 
>>> wrote:
>>>
>>> Hi Team,
>>>
>>> Just a minor point, while browsing the site I noticed that the link
>>> below is not working. Can you please check?
>>> https://beam.apache.org/documentation/resources/videos-and-podcasts
>>>
>>> Regards,
>>> Raj
>>>
>>> --
>>> Rajkumar Gupta | Technical Solutions Engineer - Google Cloud Platform |
>>> rajkumargu...@google.com  | +91-9223541460
>>> <+91%2092235%2041460>
>>>
>>>
>>>
>>

Re: Beam Website Feedback

[Ahmet Altay via dev]

On Tue, Dec 20, 2022 at 9:14 AM Alexey Romanenko 
wrote:

> Thanks Ahmet! I’d prefer to fix the links as you did and add the redirect
> from old one - perhaps, there are other similar links that have been
> changed in the same way.
>

Thank you for the review. I fixed it, and added a redirect too.


>
> Btw, I’m not sure that we still check the broken links as it was before,
> iirc, but probably it would be a good idea to add such check before
> publishing a website.
>

I agree. I also do not know about the state of this. It would be good to
add that links checker again.


>
> —
> Alexey
>
>
>
> On 20 Dec 2022, at 18:04, Ahmet Altay via dev  wrote:
>
> I did a search and found a few places with the broken link. Correct links
> should be:
> https://beam.apache.org/get-started/resources/videos-and-podcasts/
>
> I created a PR to update the website (
> https://github.com/apache/beam/pull/24733). I do not know if that is the
> best solution. As an alternative we could consider setting up a redirect
> for the old link. We do not know who else would be still linking to the old
> one.
>
> Ahmet
>
> On Tue, Dec 20, 2022 at 8:52 AM Alexey Romanenko 
> wrote:
>
>> Hi Rajkumar,
>>
>> Could you specify where (which page) this link was found?
>> Thanks!
>>
>> —
>> Alexey
>>
>> On 20 Dec 2022, at 10:08, Rajkumar Gupta via dev 
>> wrote:
>>
>> Hi Team,
>>
>> Just a minor point, while browsing the site I noticed that the link below
>> is not working. Can you please check?
>> https://beam.apache.org/documentation/resources/videos-and-podcasts
>>
>> Regards,
>> Raj
>>
>> --
>> Rajkumar Gupta | Technical Solutions Engineer - Google Cloud Platform |
>> rajkumargu...@google.com  | +91-9223541460
>> <+91%2092235%2041460>
>>
>>
>>
>

Re: Beam Website Feedback

[Ahmet Altay via dev]

I did a search and found a few places with the broken link. Correct links
should be:
https://beam.apache.org/get-started/resources/videos-and-podcasts/

I created a PR to update the website (
https://github.com/apache/beam/pull/24733). I do not know if that is the
best solution. As an alternative we could consider setting up a redirect
for the old link. We do not know who else would be still linking to the old
one.

Ahmet

On Tue, Dec 20, 2022 at 8:52 AM Alexey Romanenko 
wrote:

> Hi Rajkumar,
>
> Could you specify where (which page) this link was found?
> Thanks!
>
> —
> Alexey
>
> On 20 Dec 2022, at 10:08, Rajkumar Gupta via dev 
> wrote:
>
> Hi Team,
>
> Just a minor point, while browsing the site I noticed that the link below
> is not working. Can you please check?
> https://beam.apache.org/documentation/resources/videos-and-podcasts
>
> Regards,
> Raj
>
> --
> Rajkumar Gupta | Technical Solutions Engineer - Google Cloud Platform |
> rajkumargu...@google.com  | +91-9223541460
> <+91%2092235%2041460>
>
>
>

Re: [PROPOSAL] Preparing for Apache Beam 2.44.0 Release

[Ahmet Altay via dev]

Take care Kenn, hope you will feel better soon.

How about you continue after the new year? Hopefully you will be feeling
better. Handing off would be hard, and I imagine not a lot of people
would be around to validate in the next 2 weeks anyway.

On Mon, Dec 19, 2022 at 10:17 AM Kenneth Knowles  wrote:

> I managed to acquire covid for my last work week before vacation, so I
> don't expect to make a lot of progress. I'm not sure the best way to hand
> off release processes.
>
> Kenn
>
> On Fri, Dec 16, 2022 at 5:30 PM Ahmet Altay via dev 
> wrote:
>
>> Hello! How is the RC coming along? Do you need help?
>>
>> On Wed, Dec 14, 2022 at 2:33 PM Kenneth Knowles  wrote:
>>
>>> I've edited the subject for this update. There are no more open bugs
>>> targeting the release milestone. I will prepare RC1 shortly.
>>>
>>> Kenn
>>>
>>> On Thu, Dec 1, 2022 at 12:55 PM Kenneth Knowles  wrote:
>>>
>>>> Just an update that the branch is cut.
>>>>
>>>> There are 8 issues targeted to the release milestone:
>>>> https://github.com/apache/beam/milestone/7 (thanks Cham for the
>>>> correct link!)
>>>>
>>>> Please help to close these out or triage them off the milestone. I will
>>>> be looking at them now.
>>>>
>>>> Kenn
>>>>
>>>> On Thu, Nov 17, 2022 at 2:27 PM Chamikara Jayalath via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>>
>>>>> Thanks Kenn.
>>>>> BTW the correct milestone for the 2.44.0 release should be this one:
>>>>> https://github.com/apache/beam/milestone/7
>>>>>
>>>>> - Cham
>>>>>
>>>>>
>>>>> On Thu, Nov 17, 2022 at 9:12 AM Ahmet Altay via dev <
>>>>> dev@beam.apache.org> wrote:
>>>>>
>>>>>> Thank you Kenn! :)
>>>>>>
>>>>>> On Wed, Nov 16, 2022 at 12:45 PM Kenneth Knowles 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> The 2.44.0 release cut is scheduled for Nov 30th [1]. I'd like to
>>>>>>> volunteer to do this release.
>>>>>>>
>>>>>>> As usual, my plan would be to cut right on that date and cherry
>>>>>>> pick critical fixes.
>>>>>>>
>>>>>>> Help me and the release by:
>>>>>>> - Making sure that any unresolved release blocking issues for 2.44.0
>>>>>>> have their "Milestone" marked as "2.44.0 Release" [2].
>>>>>>> - Reviewing the current release blockers [2] and removing the
>>>>>>> Milestone if they don't meet the criteria at [3].
>>>>>>>
>>>>>>> Kenn
>>>>>>>
>>>>>>> [1]
>>>>>>> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com
>>>>>>> [2] https://github.com/apache/beam/milestone/5
>>>>>>> [3] https://beam.apache.org/contribute/release-blocking/
>>>>>>>
>>>>>>> Kenn
>>>>>>>
>>>>>>

Re: [PROPOSAL] Preparing for Apache Beam 2.44.0 Release

[Ahmet Altay via dev]

Hello! How is the RC coming along? Do you need help?

On Wed, Dec 14, 2022 at 2:33 PM Kenneth Knowles  wrote:

> I've edited the subject for this update. There are no more open bugs
> targeting the release milestone. I will prepare RC1 shortly.
>
> Kenn
>
> On Thu, Dec 1, 2022 at 12:55 PM Kenneth Knowles  wrote:
>
>> Just an update that the branch is cut.
>>
>> There are 8 issues targeted to the release milestone:
>> https://github.com/apache/beam/milestone/7 (thanks Cham for the correct
>> link!)
>>
>> Please help to close these out or triage them off the milestone. I will
>> be looking at them now.
>>
>> Kenn
>>
>> On Thu, Nov 17, 2022 at 2:27 PM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>>>
>>> Thanks Kenn.
>>> BTW the correct milestone for the 2.44.0 release should be this one:
>>> https://github.com/apache/beam/milestone/7
>>>
>>> - Cham
>>>
>>>
>>> On Thu, Nov 17, 2022 at 9:12 AM Ahmet Altay via dev 
>>> wrote:
>>>
>>>> Thank you Kenn! :)
>>>>
>>>> On Wed, Nov 16, 2022 at 12:45 PM Kenneth Knowles 
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> The 2.44.0 release cut is scheduled for Nov 30th [1]. I'd like to
>>>>> volunteer to do this release.
>>>>>
>>>>> As usual, my plan would be to cut right on that date and cherry
>>>>> pick critical fixes.
>>>>>
>>>>> Help me and the release by:
>>>>> - Making sure that any unresolved release blocking issues for 2.44.0
>>>>> have their "Milestone" marked as "2.44.0 Release" [2].
>>>>> - Reviewing the current release blockers [2] and removing the
>>>>> Milestone if they don't meet the criteria at [3].
>>>>>
>>>>> Kenn
>>>>>
>>>>> [1]
>>>>> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com
>>>>> [2] https://github.com/apache/beam/milestone/5
>>>>> [3] https://beam.apache.org/contribute/release-blocking/
>>>>>
>>>>> Kenn
>>>>>
>>>>

Re: A Declarative API for Apache Beam

[Ahmet Altay via dev]

+1 to both of these proposals. In the past 12 months I have heard of at
least 3 YAML implementations built on top of Beam in large production
systems. Unfortunately, none of those were open sourced. Having these out
of the box would be great, and it will clearly have used demand. Thank
you all!

On Thu, Dec 15, 2022 at 10:59 AM Robert Bradshaw via dev <
dev@beam.apache.org> wrote:

> On Thu, Dec 15, 2022 at 3:37 AM Steven van Rossum
>  wrote:
> >
> > This is great! I developed a similar template a year or two ago as a
> reference for a customer to speed up their development process and
> unsurprisingly it did speed up their development.
> > Here's an example of the config layout I came up with at the time:
> >
> > options:
> >   runner: DirectRunner
> >
> > pipeline:
> > # - 
> > #   label: PubSub XML source
> > #   transform:
> > # !PTransform:apache_beam.io.ReadFromPubSub
> > # subscription: projects/PROJECT/subscriptions/SUBSCRIPTION
> > - _source_1
> >   label: XML source 1
> >   transform:
> > !PTransform:apache_beam.Create
> > values:
> > - /path/to/file.xml
> > - _source_2
> >   label: XML source 2
> >   transform:
> > !PTransform:apache_beam.Create
> > values:
> > - /path/to/another/file.xml
> > - _xml
> >   label: XMLs
> >   inputs:
> >   - step: *message_source_1
> >   - step: *message_source_2
> >   transform:
> > !PTransform:utils.transforms.ParseXmlDocument {}
> > - _messages
> >   label: Validate XMLs
> >   inputs:
> >   - step: *message_xml
> > tag: success
> >   transform:
> > !PTransform:utils.transforms.ValidateXmlDocumentWithXmlSchema
> > schema: /path/to/file.xsd
> > - _messages
> >   label: Convert XMLs
> >   inputs:
> >   - step: *validated_messages
> >   transform:
> > !PTransform:utils.transforms.ConvertXmlDocumentToDictionary
> > schema: /path/to/file.xsd
> > - label: Print XMLs
> >   inputs:
> >   - step: *converted_messages
> >   transform:
> > !PTransform:utils.transforms.Print {}
> >
> > Highlights:
> > Pipeline options are supplied under an options property.
>
> Yep, I was thinking exactly the same:
>
> https://github.com/apache/beam/blob/c5518014d47a42651df94419e3ccbc79eaf96cb3/sdks/python/apache_beam/yaml/main.py#L51
>
> > A pipeline is a flat set of all transforms in the pipeline.
>
> One can certainly enumerate the transforms as a flat set, but I do
> think being able to define a composite structure is nice. In addition,
> the "chain" composite allows one to automatically infer the
> input-output relation rather than having to spell it out (much as one
> can chain multiple transforms in the various SDKs rather than have to
> assign each result to a intermediate).
>
> > Transforms are defined using a YAML tag and named properties and can be
> used by constructing a YAML reference.
>
> That's an interesting idea. Can it be done inline as well?
>
> > DAG construction is done using a simple topological sort of transforms
> and their dependencies.
>
> Same.
>
> > Named side outputs can be referenced using a tag field.
>
> I didn't put this in any of the examples, but I do the same. If a
> transform Foo produces multiple outputs, one can (in fact must)
> reference the various outputs by Foo.output1, Foo.output2, etc.
>
> > Multiple inputs are merged with a Flatten transform.
>
> PTransfoms can have named inputs as well (they're not always
> symmetric), so I let inputs be a map if they care to distinguish them.
>
> > Not sure if there's any inspiration left to take from this, but I
> figured I'd throw it up here to share.
>
> Thanks. It's neat to see others coming up with the same idea, with
> very similar conventions, and validates that it'd be both natural and
> useful.
>
>
> > On Thu, Dec 15, 2022 at 12:48 AM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
> >>
> >> +1 for these proposals and agree that these will simplify and demystify
> Beam for many new users. I think when combined with the x-lang/Schema-Aware
> transform binding, these might end up being adequate solutions for many
> production use-cases as well (unless users need to define custom
> composites, I/O connectors, etc.).
> >>
> >> Also, thanks for providing prototype implementations with examples.
> >>
> >> - Cham
> >>
> >>
> >> On Wed, Dec 14, 2022 at 3:01 PM Sachin Agarwal via dev <
> dev@beam.apache.org> wrote:
> >>>
> >>> To build on Kenn's point, if we leverage existing stuff like dbt we
> get access to a ready made community which can help drive both adoption and
> incremental innovation by bringing more folks to Beam
> >>>
> >>> On Wed, Dec 14, 2022 at 2:57 PM Kenneth Knowles 
> wrote:
> 
>  1. I love the idea. Back in the early days people talked about an
> "XML SDK" or "JSON SDK" or "YAML SDK" and it didn't really make sense at
> the time. Portability and specifically cross-language schema transforms
> gives the right infrastructure so this is the perfect time: unique names
> (URNs) for transforms and explicit lists of 

Re: [Proposal] | Move FileIO and TextIO from :sdks:java:core to :sdks:java:io:file

[Ahmet Altay via dev]

I agree with Sachin. Keeping components that users will have to bring
together anyway leads to a better user experience. Counter example to that
is GCP libraries in my opinion. It was a frequent struggle for users to
find a working set of libraries until there was a BOM. And even after the
BOM it is still somewhat of a struggle for users and the developers of
those various libraries need to take on some of the toil of testing those
various libraries together anyway.

re: Talk it with a grain of salt since I'm not even a committer - All
inputs are welcome here. I do not think my comments should carry more
weight just because I am a committer.

On Wed, Dec 14, 2022 at 9:36 AM Sachin Agarwal via dev 
wrote:

> I strongly believe that we should continue to have Beam optimize for the
> user - and while having separate components would allow those of us who are
> contributors and committers move faster, the downsides of not having
> everything "in one box" for a new user where the components are all
> relatively guaranteed to work together at that version level are very high.
>
> Beam having everything included is absolutely a competitive advantage for
> Beam and I would not want to lose that.
>
> On Wed, Dec 14, 2022 at 9:31 AM Byron Ellis via dev 
> wrote:
>
>> Talk it with a grain of salt since I'm not even a committer, but is
>> perhaps the reorganization of Beam into smaller components the real work of
>> a 3.0 effort? Splitting of Beam into smaller more independently managed
>> components would be a pretty huge breaking change from a dependency
>> management perspective which would potentially be largely separate from any
>> code changes.
>>
>> Best,
>> B
>>
>> On Wed, Dec 14, 2022 at 9:23 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> On 12 Dec 2022, at 22:23, Robert Bradshaw via dev 
>>> wrote:
>>>
>>>
>>> Saving up all the breaking changes until a major release definitely
>>> has its downsides (look at Python 3). The migration path is often as
>>> important (if not more so) than the final destination.
>>>
>>>
>>> Actually, it proves that the major releases *should not* be delayed for
>>> a long period of time and *should* be issued more often to reduce the
>>> number of breaking changes (that, of course, likely may happen). That will
>>> help users to do much more smooth and less risky upgrades, and developers
>>> to not keep burden forever. Beam 2.0.0 was released back in may 2017 and
>>> we've almost never talked about Beam 3.0 and what are the criteria for it.
>>> I understand that it’s a completely different discussion but seems that
>>> this time has come =)
>>>
>>> As for this particular change, I would question how the benefit (it's
>>> unclear what the exact benefit is--better internal organization?)
>>> exceeds the pain of making every user refactor their code. I think a
>>> stronger case can be made for things like the Avro dependency that
>>> cause real pain.
>>>
>>>
>>> Agree. I think that if it doesn’t bring any pain with additional
>>> external dependecies and this code is used in almost every other SDK
>>> module, then there are no reasons for such breaking changes. On the other
>>> hand, Avro case, that you mentioned above, is a good example why sometimes
>>> it would be better to keep such code outside of “core”.
>>>
>>> As for the pipeline update feature, we've long discussed having
>>> "pick-your-implementation" transforms that specify alternative,
>>> equivalent implementations. Upgrades can choose the old one whereas
>>> new pipelines can get the latest and greatest. It won't solve all
>>> issues, and requires keeping old codepaths around, but could be an
>>> important step forward.
>>>
>>> On Mon, Dec 12, 2022 at 10:20 AM Kenneth Knowles 
>>> wrote:
>>>
>>>
>>> I agree with Mortiz. To answer a few specifics in my own words:
>>>
>>> - It is a perfectly sensible refactor, but as a counterpoint without
>>> file-based IO the SDK isn't functional so it is also a reasonable design
>>> point to have this included. There are other things in the core SDK that
>>> are far less "core" and could be moved out with greater benefit. The main
>>> goal for any separation of modules would be lighter weight transitive
>>> dependencies, IMO.
>>>
>>> - No, Beam has not made any deliberate breaking changes of this nature.
>>> Hence we are still on major version 2. We have made some bugfixes for data
>>> loss risks that could be called "breaking changes" but since the feature
>>> was unsafe to use in the first place we did not bump the major version.
>>>
>>> - It is sometimes possible to do such a refactor and have the deprecated
>>> location proxy to the new location. In this case that seems hard to achieve.
>>>
>>> - It is not actually necessary to maintain both locations, as we can
>>> declare the old location will be unmaintained (but left alone) and all new
>>> development goes to the new location. That isn't a great choice for users
>>> who may simply upgrade their SDK version 

Re: [PROPOSAL] Preparing for Apache Beam 2.43.0 Release

[Ahmet Altay via dev]

Thank you Kenn! :)

On Wed, Nov 16, 2022 at 12:45 PM Kenneth Knowles  wrote:

> Hi all,
>
> The 2.44.0 release cut is scheduled for Nov 30th [1]. I'd like to
> volunteer to do this release.
>
> As usual, my plan would be to cut right on that date and cherry
> pick critical fixes.
>
> Help me and the release by:
> - Making sure that any unresolved release blocking issues for 2.44.0 have
> their "Milestone" marked as "2.44.0 Release" [2].
> - Reviewing the current release blockers [2] and removing the Milestone if
> they don't meet the criteria at [3].
>
> Kenn
>
> [1]
> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com
> [2] https://github.com/apache/beam/milestone/5
> [3] https://beam.apache.org/contribute/release-blocking/
>
> Kenn
>

Re: [VOTE] Release 2.43.0, release candidate #2

[Ahmet Altay via dev]

+1 (binding). - I validated the python quick starts on direct runner.

Thank you!

On Tue, Nov 15, 2022 at 9:51 AM Jean-Baptiste Onofré 
wrote:

> +1 (binding)
>
> Regards
> JB
>
> On Sun, Nov 13, 2022 at 3:52 PM Chamikara Jayalath via dev
>  wrote:
> >
> > Hi everyone,
> > Please review and vote on the release candidate #2 for the version
> 2.43.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > Reviewers are encouraged to test their own use cases with the release
> candidate, and vote +1 if
> > no issues are found.
> >
> > The complete staging area is available for your review, which includes:
> > * GitHub Release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> [2], which is signed with the key with fingerprint
> 40C61FBE1761E5DB652A1A780CCD5EB2A718A56E [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "v2.43.0-RC2" [5],
> > * website pull request listing the release [6], the blog post [6], and
> publishing the API reference manual [7].
> > * Java artifacts were built with Gradle 7.5.1 and openjdk version
> 1.8.0_181-google-v7.
> > * Python artifacts are deployed along with the source release to the
> dist.apache.org [2] and PyPI[8].
> > * Go artifacts and documentation are available at pkg.go.dev [9]
> > * Validation sheet with a tab for 2.43.0 release to help with validation
> [10].
> > * Docker images published to Docker Hub [11].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> >
> > For guidelines on how to try the release in your projects, check out our
> blog post at https://beam.apache.org/blog/validate-beam-release/.
> >
> > Thanks,
> > Cham
> >
> > [1] https://github.com/apache/beam/milestone/5
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.43.0/
> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1288/
> > [5] https://github.com/apache/beam/tree/v2.43.0-RC2
> > [6] https://github.com/apache/beam/pull/24044
> > [7] https://github.com/apache/beam-site/pull/636
> > [8] https://pypi.org/project/apache-beam/2.43.0rc2/
> > [9]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.43.0-RC2/go/pkg/beam
> > [10]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1310009119
> > [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>

Re: [Announce] Release tasks available through GitHub Actions workflows

[Ahmet Altay via dev]

Thank you, that is excellent.

Adding the last few release manager, they would have the most applicable
feedback: @Kiley Sok  @Robert Burke
 @Chamikara
Jayalath 

On Thu, Nov 10, 2022 at 1:17 PM Fer Morales Martinez <
fernando.mora...@wizeline.com> wrote:

> Hi everyone,
> We would like to share with you that the team @ Wizeline will start
> working on the migration of some of the tasks related to the release
> process, detailed in [1].
>
> For example, in the following PR [2] a workflow was created to execute,
> within GitHub Actions (GHA), step 6 [3] of the release tasks.
>
> A similar effort is being worked on in this PR [4] to migrate step 8 [5]
> to GHA.
>
> The initial approach will be to migrate the tasks that do NOT require the
> use of keys to sign artifacts. The goal is to eventually replace the
> locally executed tasks for tasks executed in GHA.
>
> The tasks which require keys and signing will still be executed locally
> for the time being.
>
> We would love to receive any feedback or suggestion if any.
>
> Thanks!
>
> [1] https://beam.apache.org/contribute/release-guide
> [2] https://github.com/apache/beam/pull/23390
> [3]
> https://beam.apache.org/contribute/release-guide/#6-verify-release-branch
> [4] https://github.com/apache/beam/pull/24075
> [5]
> https://beam.apache.org/contribute/release-guide/#tag-a-chosen-commit-for-the-rc
>
> --
>
> Fernando Morales (He/Him) |  | wizeline.com
> 
>
> Software Engineer
>
> fernando.mora...@wizeline.com
>
> Remote Office, Mexico
>
>
>
>
>
>
>
>
> *This email and its contents (including any attachments) are being sent
> toyou on the condition of confidentiality and may be protected by
> legalprivilege. Access to this email by anyone other than the intended
> recipientis unauthorized. If you are not the intended recipient, please
> immediatelynotify the sender by replying to this message and delete the
> materialimmediately from your system. Any further use, dissemination,
> distributionor reproduction of this email is strictly prohibited. Further,
> norepresentation is made with respect to any content contained in this
> email.*

Re: bhulette stepping back (for now)

[Ahmet Altay via dev]

Thank you for everything Brian!

On Fri, Nov 11, 2022 at 11:27 AM Austin Bennett  wrote:

> Thanks for everything you've done, @bhule...@apache.org!
>
> On Fri, Nov 11, 2022 at 11:01 AM Pablo Estrada via dev <
> dev@beam.apache.org> wrote:
>
>> I promised I wouldn't cry so I won't. Cya!
>>
>> On Fri, Nov 11, 2022 at 10:46 AM Robin Qiu via dev 
>> wrote:
>>
>>> Thanks for your contribution Brian! Hope you enjoy your new team!
>>>
>>> Best,
>>> Robin
>>>
>>> On Fri, Nov 11, 2022 at 10:27 AM Kenneth Knowles 
>>> wrote:
>>>
 Your contributions have been huge. You will be missed! But have a
 fabulous time with BigQuery. And thank you so much for letting us know [1]

 Kenn

 [1] See "stepping down considerately" from
 https://www.apache.org/foundation/policies/conduct.html

 On Thu, Nov 10, 2022 at 4:00 PM Brian Hulette 
 wrote:

> Hi dev@beam,
>
> I just wanted to let the community know that I will be stepping back
> from Beam development for now. I'm switching to a different team within
> Google next week - I will be working on BigQuery.
>
> I'm removing myself from automated code review assignments [1], and
> won't actively monitor the beam lists anymore. That being said, I'm happy
> to contribute to discussions or code reviews when it would be particularly
> helpful, e.g. for anything relating to DataFrames/Schemas/SQL. I can 
> always
> be reached at bhule...@apache.org, and @TheNeuralBit [2] on GitHub.
>
> Brian
>
> [1] https://github.com/apache/beam/pull/24108
> [2] https://github.com/TheNeuralBit
>

Re: [ANNOUNCE] New committer: Yi Hu

[Ahmet Altay via dev]

Congratulations Yi!

On Fri, Nov 11, 2022 at 1:57 PM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> Contrats Yi!
>
> On Thu, Nov 10, 2022 at 10:48 AM Kerry Donny-Clark via dev <
> dev@beam.apache.org> wrote:
>
>> Great job Yi! I am happy to see your contributions recognized.
>>
>> On Thu, Nov 10, 2022 at 11:52 AM Yi Hu via dev 
>> wrote:
>>
>>> Thank you for the help of you all over the time, and I am glad to
>>> contribute and help with the community.
>>>
>>> Best,
>>> Yi
>>>
>>> On Thu, Nov 10, 2022 at 11:29 AM Alexey Romanenko <
>>> aromanenko@gmail.com> wrote:
>>>
 Congratulations! Well deserved!

 —
 Alexey

 On 9 Nov 2022, at 21:01, Tomo Suzuki via dev 
 wrote:

 Congratulations!

 On Wed, Nov 9, 2022 at 3:00 PM John Casey via dev 
 wrote:

> Congrats! this is well deserved YI
>
> On Wed, Nov 9, 2022 at 2:58 PM Austin Bennett <
> whatwouldausti...@gmail.com> wrote:
>
>> Congrats, and Thanks, Yi!
>>
>> On Wed, Nov 9, 2022 at 11:24 AM Valentyn Tymofieiev via dev <
>> dev@beam.apache.org> wrote:
>>
>>> I am with the Beam PMC on this, congratulations and very well
>>> deserved, Yi!
>>>
>>> On Wed, Nov 9, 2022 at 11:08 AM Byron Ellis via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congratulations!

 On Wed, Nov 9, 2022 at 11:00 AM Pablo Estrada via dev <
 dev@beam.apache.org> wrote:

> +1 thanks Yi : D
>
> On Wed, Nov 9, 2022 at 10:47 AM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Yi! I've really appreciated the ways you've consistently
>> taken responsibility for improving our team's infra and working 
>> through
>> sharp edges in the codebase that others have ignored. This is 
>> definitely
>> well deserved!
>>
>> Thanks,
>> Danny
>>
>> On Wed, Nov 9, 2022 at 1:37 PM Anand Inguva via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congratulations Yi!
>>>
>>> On Wed, Nov 9, 2022 at 1:35 PM Ritesh Ghorse via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congratulations Yi!

 On Wed, Nov 9, 2022 at 1:34 PM Ahmed Abualsaud via dev <
 dev@beam.apache.org> wrote:

> Congrats Yi!
>
> On Wed, Nov 9, 2022 at 1:33 PM Sachin Agarwal via dev <
> dev@beam.apache.org> wrote:
>
>> Congratulations Yi!
>>
>> On Wed, Nov 9, 2022 at 10:32 AM Kenneth Knowles <
>> k...@apache.org> wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a
>>> new committer: Yi Hu (y...@apache.org)
>>>
>>> Yi started contributing to Beam in early 2022. Yi's
>>> contributions are very diverse! I/Os, performance tests, 
>>> Jenkins, support
>>> for Schema logical types. Not only code but a very large amount 
>>> of code
>>> review. Yi is also noted for picking up smaller issues that 
>>> normally would
>>> be left on the backburner and filing issues that he finds 
>>> rather than
>>> ignoring them.
>>>
>>> Considering their contributions to the project over this
>>> timeframe, the Beam PMC trusts Yi with the responsibilities of 
>>> a Beam
>>> committer. [1]
>>>
>>> Thank you Yi! And we are looking to see more of your
>>> contributions!
>>>
>>> Kenn, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>>
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>

 --
 Regards,
 Tomo

Re: [PROPOSAL] Preparing for Apache Beam 2.43.0 Release

[Ahmet Altay via dev]

Thank you!

On Tue, Nov 8, 2022 at 1:39 PM Chamikara Jayalath 
wrote:

> External artifacts have been built. I'm working on getting Dataflow
> containers and documentation setup so should be out for review/vote soon :)
>
> On Tue, Nov 8, 2022 at 12:22 PM Ahmet Altay  wrote:
>
>> Any progress on the RC? Any blockers we can help with?
>>
>> On Fri, Nov 4, 2022 at 9:05 PM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Update:
>>>
>>> RC creation is still ongoing. I hope to get it out for review early next
>>> week
>>>
>>> Thanks,
>>> Cham
>>>
>>>
>>>
>>> On Fri, Nov 4, 2022 at 9:14 AM Ahmet Altay  wrote:
>>>
>>>> Thank you Cham!
>>>>
>>>> On Thu, Nov 3, 2022 at 10:54 PM Chamikara Jayalath <
>>>> chamik...@google.com> wrote:
>>>>
>>>>> Fix for the blocking issue was cherry-picked today. So I hope to build
>>>>> the RC1 tomorrow.
>>>>>
>>>>> Thanks,
>>>>> Cham
>>>>>
>>>>> On Thu, Nov 3, 2022 at 8:19 PM Ahmet Altay  wrote:
>>>>>
>>>>>> How is the release coming along? Do you need any help?
>>>>>>
>>>>>> On Mon, Oct 31, 2022 at 1:58 PM Chamikara Jayalath via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>>> Update:
>>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I've been validating the release branch by running all Jenkins test
>>>>>>> suites on it (as required by the release guide). This revealed two new
>>>>>>> potential issues. I added these to the release milestone [1]. Please
>>>>>>> comment on these issues if you are familiar with the errors (for 
>>>>>>> example,
>>>>>>> if they are known issues from a previous release). We can continue the
>>>>>>> release once these are resolved or moved out of the 2.43.0 release
>>>>>>> milestone.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Cham
>>>>>>>
>>>>>>> [1] https://github.com/apache/beam/milestone/5
>>>>>>>
>>>>>>> On Wed, Oct 26, 2022 at 12:42 PM Chamikara Jayalath <
>>>>>>> chamik...@google.com> wrote:
>>>>>>>
>>>>>>>> Update:
>>>>>>>>
>>>>>>>> All blocking issues have either been addressed or pushed to the
>>>>>>>> next release. I'll go ahead and create the first RC.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Cham
>>>>>>>>
>>>>>>>> On Thu, Oct 20, 2022 at 9:41 AM Chamikara Jayalath <
>>>>>>>> chamik...@google.com> wrote:
>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> The release branch was cut:
>>>>>>>>> https://github.com/apache/beam/tree/release-2.43.0
>>>>>>>>>
>>>>>>>>> We currently have three open blockers in the release milestone:
>>>>>>>>> https://github.com/apache/beam/milestone/5
>>>>>>>>>
>>>>>>>>> I'll look into cherry-picking fixes for these and hopefully
>>>>>>>>> creating a RC early next week.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Cham
>>>>>>>>>
>>>>>>>>> On Wed, Oct 5, 2022 at 3:25 PM Ahmet Altay 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> +1 - Thank you Cham!
>>>>>>>>>>
>>>>>>>>>> On Wed, Oct 5, 2022 at 1:38 PM Chamikara Jayalath via dev <
>>>>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> The next (2.43.0) release branch cut is scheduled for October
>>>>>>>>>>> 19th, according to the release calendar [1].
>>>>>>>>>>>
>>>>>>>>>>> I would like to volunteer myself to do this release. My plan is
>>>>>>>>>>> to cut the branch on that date, and cherrypick release-blocking 
>>>>>>>>>>> fixes
>>>>>>>>>>> afterwards, if any.
>>>>>>>>>>>
>>>>>>>>>>> Please help me make sure the release goes smoothly by:
>>>>>>>>>>> - Making sure that any unresolved release blocking issues for
>>>>>>>>>>> 2.43.0 should have their "Milestone" marked as "2.43.0 Release"
>>>>>>>>>>> [2] as soon as possible.
>>>>>>>>>>> - Reviewing the current release blockers [2] and removing the
>>>>>>>>>>> Milestone if they don't meet the criteria at [3].
>>>>>>>>>>>
>>>>>>>>>>> Let me know if you have any comments/objections/questions.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Cham
>>>>>>>>>>>
>>>>>>>>>>> [1]
>>>>>>>>>>> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com
>>>>>>>>>>> [2] https://github.com/apache/beam/milestone/5
>>>>>>>>>>> [3] https://beam.apache.org/contribute/release-blocking/
>>>>>>>>>>>
>>>>>>>>>>

Re: [PROPOSAL] Preparing for Apache Beam 2.43.0 Release

[Ahmet Altay via dev]

Any progress on the RC? Any blockers we can help with?

On Fri, Nov 4, 2022 at 9:05 PM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> Update:
>
> RC creation is still ongoing. I hope to get it out for review early next
> week
>
> Thanks,
> Cham
>
>
>
> On Fri, Nov 4, 2022 at 9:14 AM Ahmet Altay  wrote:
>
>> Thank you Cham!
>>
>> On Thu, Nov 3, 2022 at 10:54 PM Chamikara Jayalath 
>> wrote:
>>
>>> Fix for the blocking issue was cherry-picked today. So I hope to build
>>> the RC1 tomorrow.
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Thu, Nov 3, 2022 at 8:19 PM Ahmet Altay  wrote:
>>>
>>>> How is the release coming along? Do you need any help?
>>>>
>>>> On Mon, Oct 31, 2022 at 1:58 PM Chamikara Jayalath via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> Update:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I've been validating the release branch by running all Jenkins test
>>>>> suites on it (as required by the release guide). This revealed two new
>>>>> potential issues. I added these to the release milestone [1]. Please
>>>>> comment on these issues if you are familiar with the errors (for example,
>>>>> if they are known issues from a previous release). We can continue the
>>>>> release once these are resolved or moved out of the 2.43.0 release
>>>>> milestone.
>>>>>
>>>>> Thanks,
>>>>> Cham
>>>>>
>>>>> [1] https://github.com/apache/beam/milestone/5
>>>>>
>>>>> On Wed, Oct 26, 2022 at 12:42 PM Chamikara Jayalath <
>>>>> chamik...@google.com> wrote:
>>>>>
>>>>>> Update:
>>>>>>
>>>>>> All blocking issues have either been addressed or pushed to the next
>>>>>> release. I'll go ahead and create the first RC.
>>>>>>
>>>>>> Thanks,
>>>>>> Cham
>>>>>>
>>>>>> On Thu, Oct 20, 2022 at 9:41 AM Chamikara Jayalath <
>>>>>> chamik...@google.com> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> The release branch was cut:
>>>>>>> https://github.com/apache/beam/tree/release-2.43.0
>>>>>>>
>>>>>>> We currently have three open blockers in the release milestone:
>>>>>>> https://github.com/apache/beam/milestone/5
>>>>>>>
>>>>>>> I'll look into cherry-picking fixes for these and hopefully creating
>>>>>>> a RC early next week.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Cham
>>>>>>>
>>>>>>> On Wed, Oct 5, 2022 at 3:25 PM Ahmet Altay  wrote:
>>>>>>>
>>>>>>>> +1 - Thank you Cham!
>>>>>>>>
>>>>>>>> On Wed, Oct 5, 2022 at 1:38 PM Chamikara Jayalath via dev <
>>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> The next (2.43.0) release branch cut is scheduled for October
>>>>>>>>> 19th, according to the release calendar [1].
>>>>>>>>>
>>>>>>>>> I would like to volunteer myself to do this release. My plan is to
>>>>>>>>> cut the branch on that date, and cherrypick release-blocking fixes
>>>>>>>>> afterwards, if any.
>>>>>>>>>
>>>>>>>>> Please help me make sure the release goes smoothly by:
>>>>>>>>> - Making sure that any unresolved release blocking issues for 2.43
>>>>>>>>> .0 should have their "Milestone" marked as "2.43.0 Release" [2]
>>>>>>>>> as soon as possible.
>>>>>>>>> - Reviewing the current release blockers [2] and removing the
>>>>>>>>> Milestone if they don't meet the criteria at [3].
>>>>>>>>>
>>>>>>>>> Let me know if you have any comments/objections/questions.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Cham
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com
>>>>>>>>> [2] https://github.com/apache/beam/milestone/5
>>>>>>>>> [3] https://beam.apache.org/contribute/release-blocking/
>>>>>>>>>
>>>>>>>>

Re: github reviewer help / tips

[Ahmet Altay via dev]

On a related note: It is quite easy to miss GitHub notifications. If you
are waiting on a review feel free to ping (github, email, slack etc.) the
reviewer after a few days.

On Tue, Nov 8, 2022 at 11:08 AM Pablo Estrada via dev 
wrote:

> Nice pointer Valentyn.
>
> If you are willing to open the Github page sometimes, it's worth just
> using the PR tab: https://github.com/pulls - which should show all PRs
> that mention you/have review requests for you from all repos.
> -P.
>
> On Tue, Nov 8, 2022 at 10:56 AM Valentyn Tymofieiev via dev <
> dev@beam.apache.org> wrote:
>
>> I use Notifier for Github
>> 
>>  Chrome
>> extension.
>>
>
+1 for this. This is also my primary way of seeing GH notifications.


>
>> On Tue, Nov 8, 2022 at 10:29 AM Sachin Agarwal via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hey folks,
>>>
>>> I've found myself repeatedly being very untimely in providing reviews on
>>> PRs where I've been added as a reviewer.  (Mea culpa and thank you for your
>>> understanding to those who have tagged me and emailed me to nudge me along.)
>>>
>>> Does anyone have any great tips about how to be super on top of things
>>> in the Beam repos?  Any Github experts who can get my SLA from three weeks
>>> to a day or so would be great.
>>>
>>> Many thanks in advance -
>>>
>>> Cheers,
>>> Sachin
>>>
>>

Re: [ANNOUNCE] New committer: Ritesh Ghorse

[Ahmet Altay via dev]

Congratulations Ritesh!

On Fri, Nov 4, 2022 at 12:18 PM Ritesh Ghorse via dev 
wrote:

> Thanks everyone! I'm glad to be a part of this community and I look
> forward to making more contributions in whatever ways I can.
>
> On Fri, Nov 4, 2022 at 12:36 PM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats, Ritesh!
>>
>> On Fri, Nov 4, 2022 at 9:34 AM John Casey via dev 
>> wrote:
>>
>>> Congrats!
>>>
>>> On Fri, Nov 4, 2022 at 10:36 AM Ahmed Abualsaud via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congrats Ritesh!

 On Fri, Nov 4, 2022 at 10:29 AM Andy Ye via dev 
 wrote:

> Congrats Ritesh!
>
> On Fri, Nov 4, 2022 at 9:26 AM Kerry Donny-Clark via dev <
> dev@beam.apache.org> wrote:
>
>> Congratulations Ritesh, I'm happy to see your hard work and community
>> spirit recognized!
>>
>> On Fri, Nov 4, 2022 at 10:16 AM Jack McCluskey via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congrats Ritesh!
>>>
>>> On Thu, Nov 3, 2022 at 10:12 PM Danny McCormick via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congrats Ritesh! This is definitely well deserved!

 On Thu, Nov 3, 2022 at 8:08 PM Robert Burke 
 wrote:

> Woohoo! Well done Ritesh! :D
>
> On Thu, Nov 3, 2022, 5:04 PM Anand Inguva via dev <
> dev@beam.apache.org> wrote:
>
>> Congratulations Ritesh.
>>
>> On Thu, Nov 3, 2022 at 7:51 PM Yi Hu via dev 
>> wrote:
>>
>>> Congratulations Ritesh!
>>>
>>> On Thu, Nov 3, 2022 at 7:23 PM Byron Ellis via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congratulations!

 On Thu, Nov 3, 2022 at 4:21 PM Austin Bennett <
 whatwouldausti...@gmail.com> wrote:

> Congratulations, and Thanks @riteshgho...@apache.org!
>
> On Thu, Nov 3, 2022 at 4:17 PM Sachin Agarwal via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Ritesh!
>>
>> On Thu, Nov 3, 2022 at 4:16 PM Kenneth Knowles <
>> k...@apache.org> wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a
>>> new committer: Ritesh Ghorse (riteshgho...@apache.org)
>>>
>>> Ritesh started contributing to Beam in mid-2021 and has
>>> contributed immensely to bringin the Go SDK to fruition, in 
>>> addition to
>>> contributions to Java and Python and release validation.
>>>
>>> Considering their contributions to the project over this
>>> timeframe, the Beam PMC trusts Ritesh with the responsibilities 
>>> of a Beam
>>> committer. [1]
>>>
>>> Thank you Ritesh! And we are looking to see more of your
>>> contributions!
>>>
>>> Kenn, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>>
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>

Re: [PROPOSAL] Preparing for Apache Beam 2.43.0 Release

[Ahmet Altay via dev]

Thank you Cham!

On Thu, Nov 3, 2022 at 10:54 PM Chamikara Jayalath 
wrote:

> Fix for the blocking issue was cherry-picked today. So I hope to build the
> RC1 tomorrow.
>
> Thanks,
> Cham
>
> On Thu, Nov 3, 2022 at 8:19 PM Ahmet Altay  wrote:
>
>> How is the release coming along? Do you need any help?
>>
>> On Mon, Oct 31, 2022 at 1:58 PM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Update:
>>>
>>> Hi All,
>>>
>>> I've been validating the release branch by running all Jenkins test
>>> suites on it (as required by the release guide). This revealed two new
>>> potential issues. I added these to the release milestone [1]. Please
>>> comment on these issues if you are familiar with the errors (for example,
>>> if they are known issues from a previous release). We can continue the
>>> release once these are resolved or moved out of the 2.43.0 release
>>> milestone.
>>>
>>> Thanks,
>>> Cham
>>>
>>> [1] https://github.com/apache/beam/milestone/5
>>>
>>> On Wed, Oct 26, 2022 at 12:42 PM Chamikara Jayalath <
>>> chamik...@google.com> wrote:
>>>
>>>> Update:
>>>>
>>>> All blocking issues have either been addressed or pushed to the next
>>>> release. I'll go ahead and create the first RC.
>>>>
>>>> Thanks,
>>>> Cham
>>>>
>>>> On Thu, Oct 20, 2022 at 9:41 AM Chamikara Jayalath <
>>>> chamik...@google.com> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> The release branch was cut:
>>>>> https://github.com/apache/beam/tree/release-2.43.0
>>>>>
>>>>> We currently have three open blockers in the release milestone:
>>>>> https://github.com/apache/beam/milestone/5
>>>>>
>>>>> I'll look into cherry-picking fixes for these and hopefully creating a
>>>>> RC early next week.
>>>>>
>>>>> Thanks,
>>>>> Cham
>>>>>
>>>>> On Wed, Oct 5, 2022 at 3:25 PM Ahmet Altay  wrote:
>>>>>
>>>>>> +1 - Thank you Cham!
>>>>>>
>>>>>> On Wed, Oct 5, 2022 at 1:38 PM Chamikara Jayalath via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> The next (2.43.0) release branch cut is scheduled for October 19th,
>>>>>>> according to the release calendar [1].
>>>>>>>
>>>>>>> I would like to volunteer myself to do this release. My plan is to
>>>>>>> cut the branch on that date, and cherrypick release-blocking fixes
>>>>>>> afterwards, if any.
>>>>>>>
>>>>>>> Please help me make sure the release goes smoothly by:
>>>>>>> - Making sure that any unresolved release blocking issues for 2.43.0 
>>>>>>> should
>>>>>>> have their "Milestone" marked as "2.43.0 Release" [2] as soon as
>>>>>>> possible.
>>>>>>> - Reviewing the current release blockers [2] and removing the
>>>>>>> Milestone if they don't meet the criteria at [3].
>>>>>>>
>>>>>>> Let me know if you have any comments/objections/questions.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Cham
>>>>>>>
>>>>>>> [1]
>>>>>>> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com
>>>>>>> [2] https://github.com/apache/beam/milestone/5
>>>>>>> [3] https://beam.apache.org/contribute/release-blocking/
>>>>>>>
>>>>>>

Re: [PROPOSAL] Preparing for Apache Beam 2.43.0 Release

[Ahmet Altay via dev]

How is the release coming along? Do you need any help?

On Mon, Oct 31, 2022 at 1:58 PM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> Update:
>
> Hi All,
>
> I've been validating the release branch by running all Jenkins test suites
> on it (as required by the release guide). This revealed two new
> potential issues. I added these to the release milestone [1]. Please
> comment on these issues if you are familiar with the errors (for example,
> if they are known issues from a previous release). We can continue the
> release once these are resolved or moved out of the 2.43.0 release
> milestone.
>
> Thanks,
> Cham
>
> [1] https://github.com/apache/beam/milestone/5
>
> On Wed, Oct 26, 2022 at 12:42 PM Chamikara Jayalath 
> wrote:
>
>> Update:
>>
>> All blocking issues have either been addressed or pushed to the next
>> release. I'll go ahead and create the first RC.
>>
>> Thanks,
>> Cham
>>
>> On Thu, Oct 20, 2022 at 9:41 AM Chamikara Jayalath 
>> wrote:
>>
>>> Hi All,
>>>
>>> The release branch was cut:
>>> https://github.com/apache/beam/tree/release-2.43.0
>>>
>>> We currently have three open blockers in the release milestone:
>>> https://github.com/apache/beam/milestone/5
>>>
>>> I'll look into cherry-picking fixes for these and hopefully creating a
>>> RC early next week.
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Wed, Oct 5, 2022 at 3:25 PM Ahmet Altay  wrote:
>>>
>>>> +1 - Thank you Cham!
>>>>
>>>> On Wed, Oct 5, 2022 at 1:38 PM Chamikara Jayalath via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> The next (2.43.0) release branch cut is scheduled for October 19th,
>>>>> according to the release calendar [1].
>>>>>
>>>>> I would like to volunteer myself to do this release. My plan is to cut
>>>>> the branch on that date, and cherrypick release-blocking fixes afterwards,
>>>>> if any.
>>>>>
>>>>> Please help me make sure the release goes smoothly by:
>>>>> - Making sure that any unresolved release blocking issues for 2.43.0 
>>>>> should
>>>>> have their "Milestone" marked as "2.43.0 Release" [2] as soon as
>>>>> possible.
>>>>> - Reviewing the current release blockers [2] and removing the
>>>>> Milestone if they don't meet the criteria at [3].
>>>>>
>>>>> Let me know if you have any comments/objections/questions.
>>>>>
>>>>> Thanks,
>>>>> Cham
>>>>>
>>>>> [1]
>>>>> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com
>>>>> [2] https://github.com/apache/beam/milestone/5
>>>>> [3] https://beam.apache.org/contribute/release-blocking/
>>>>>
>>>>

Re: SSL issue: Kafka Avro write with Schema Registry (GCP)

[Ahmet Altay via dev]

(moving this to the user list, dev list to bcc.)

Adding relevant people: @John Casey .

(Keshav, for Dataflow issues you could also reach out to Dataflow support:
https://cloud.google.com/dataflow/docs/support)

On Mon, Oct 31, 2022 at 1:23 PM Chennakeshavlu Maddela <
chennakeshavlu.madd...@davita.com> wrote:

> Hi Team,
>
>
>
> We are setting up avro write to a kafka topic with confluent schema
> registry on SSL, its throwing (below) error.
>
>
>
> We are using SASL_SSL with PEM certificate for connecting Kafka broker,
> which is working fine with non-avro kafka topics. Can you please help us
> with configuring SSL for schema registry? (we are using dataflow runner)
>
>
>
> Please let me know if you need more details.
>
>
>
> Thank you,
>
> Keshav
>
>
>
> *Exception:*
>
> Failed to send HTTP request to endpoint:
> https://confluent-schemaregistry-.com/subjects/topic-value?deleted=false
>
>
>
> javax.net.ssl.SSLHandshakeException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target
>
> at
> java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
>
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:350)
>
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:293)
>
> at
> java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:288)
>
> at
> java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1356)
>
> at
> java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1231)
>
> at
> java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1174)
>
> at
> java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
>
> at
> java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
>
> at
> java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
>
> at
> java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
>
> at
> java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
>
> at
> java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1408)
>
> at
> java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1314)
>
> at
> java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
>
> at
> java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
>
> at
> java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
>
> at
> java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
>
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367)
>
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342)
>
> at
> java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246)
>
> at
> io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:199)
>
> at
> io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:256)
>
> at
> io.confluent.kafka.schemaregistry.client.rest.RestService.lookUpSubjectVersion(RestService.java:323)
>
> at
> io.confluent.kafka.schemaregistry.client.rest.RestService.lookUpSubjectVersion(RestService.java:311)
>
> at
> io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getIdFromRegistry(CachedSchemaRegistryClient.java:191)
>
> at
> io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getId(CachedSchemaRegistryClient.java:323)
>
> at
> io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:73)
>
> at
> io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:53)
>
> at
> org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
>
> at
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:952)
>
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:912)
>
> at
> com.davita.cwow.pmt.transformations.PmtAvroKafkaWriter$KafkaWriteEvaluationFn.processElement(PmtAvroKafkaWriter.java:86)
>
> at
> 

Re: [Proposal] Beam MultimapState API

[Ahmet Altay via dev]

Thank you for the message Buqian. Adding @Reuven Lax  @Lukasz
Cwik  explicitly (who are mentioned on the doc).

On Mon, Oct 31, 2022 at 12:17 PM 郑卜千  wrote:

> Gentle ping. Thanks!
>
> On Thu, Oct 27, 2022 at 2:55 PM 郑卜千  wrote:
>
>> Hi all,
>>
>> I've been working on adding MultimapState support to the Dataflow Runner,
>> and the state interface is currently missing from the Beam State API.
>>
>> I have an one pager proposing its API interface in
>> https://docs.google.com/document/d/1zm16QCxWEWNy4qW1KKoA3DmQBOLrTSPimXiqQkTxokc/edit#.
>> Please share suggestions/comments!
>>
>> Thanks!
>> Buqian Zheng
>>
>>

Re: Beam Website Feedback: Apache Beam WordCount Examples

[Ahmet Altay via dev]

Thank you for reporting William.

Adding @Valentyn Tymofieiev . Perhaps we could start
with creating an issue?

On Mon, Oct 24, 2022 at 9:25 AM William Pietri 
wrote:

> Hi! I have inherited code using Apache Beam and I'm trying to figure out
> the right way to use it.
>
> This seemed like it would be a good page for me:
> https://beam.apache.org/get-started/wordcount-example/
>
> I jumped in with the MinimalWordCount example
> ,
> which says it's explaining this python code
> .
> However, the explanation doesn't match the code.
>
> The code from the text goes like this:
>
>
> pipeline = beam.Pipeline(options=beam_options)
>
> pipeline| beam.io.ReadFromText(input_file)
>
> | 'ExtractWords' >> beam.FlatMap(lambda x: re.findall(r'[A-Za-z\']+', x))
>
> | beam.combiners.Count.PerElement()
>
> | beam.MapTuple(lambda word, count: '%s: %s' % (word, count))
>
> | beam.io.WriteToText(output_path)
>
> Fair enough, I'd say. However, the code in the example file is quite
> different:
>
>   with beam.Pipeline(options=pipeline_options) as p:
> # Read the text file[pattern] into a PCollection.
> lines = p | ReadFromText(known_args.input)
>
> # Count the occurrences of each word.
> counts = (
> lines
> | 'Split' >> (
> beam.FlatMap(
> lambda x: re.findall(r'[A-Za-z\']+', 
> x)).with_output_types(str))
> | 'PairWithOne' >> beam.Map(lambda x: (x, 1))
> | 'GroupAndSum' >> beam.CombinePerKey(sum))
>
> # Format the counts into a PCollection of strings.
> def format_result(word_count):
>   (word, count) = word_count
>   return '%s: %s' % (word, count)
>
> output = counts | 'Format' >> beam.Map(format_result)
>
> # Write the output using a "Write" transform that has side effects.
> # pylint: disable=expression-not-assigned
> output | WriteToText(known_args.output)
>
>
> I assume this is also probably good? But there are a number of differences
> here in both structure and content. This confusion is exactly what I don't
> need in intro documentation, so I'd love it if somebody made this
> consistent.
>
> Thanks,
>
> William
>

Re: [VOTE] Release 2.42.0, release candidate #2

[Ahmet Altay via dev]

+1 (binding)

Tested python quickstart examples on the direct runner. Thank you!

On Thu, Oct 13, 2022 at 5:35 PM Robert Bradshaw via dev 
wrote:

> +1 (binding)
>
> Validated release artifacts and signatures. Tested a Python pipeline
> on a clean install.
>
> On Thu, Oct 13, 2022 at 1:22 PM Ritesh Ghorse via dev
>  wrote:
> >
> > +1 (non-binding)
> > Validated Go SDK Quickstart on Direct and Dataflow runner.
> >
> > Thanks,
> > Ritesh Ghorse
> >
> > On Thu, Oct 13, 2022 at 4:01 PM Pablo Estrada via dev <
> dev@beam.apache.org> wrote:
> >>
> >> +1 (binding)
> >>
> >> I've validated local/unit tests for existing dataflow templates. They
> look good!
> >> Best
> >> -P.
> >>
> >> On Thu, Oct 13, 2022 at 10:41 AM Ning Kang via dev 
> wrote:
> >>>
> >>> +1 Thank you, Robert!
> >>>
> >>> On Thu, Oct 13, 2022 at 12:47 AM Robert Burke 
> wrote:
> 
>  Hi everyone,
>  Please review and vote on the release candidate #2 for the version
> 2.42.0, as follows:
>  [ ] +1, Approve the release
>  [ ] -1, Do not approve the release (please provide specific comments)
> 
>  Reviewers are encouraged to test their own use cases with the release
> candidate, and vote +1 if no issues are found.
> 
>  The complete staging area is available for your review, which
> includes:
>  * GitHub Release notes [1],
>  * the official Apache source release to be deployed to
> dist.apache.org [2], which is signed with the key with fingerprint
> A52F5C83BAE26160120EC25F3D56ACFBFB2975E1 [3],
>  * all artifacts to be deployed to the Maven Central Repository [4],
>  * source code tag "v2.42.0-RC2" [5],
>  * website pull request listing the release [6], the blog post [6],
> and publishing the API reference manual [7].
>  * Java artifacts were built with Gradle 7.5.1 and AdoptOpen JDK
> 1.8.0_292.
>  * Python artifacts are deployed along with the source release to the
> dist.apache.org [2] and PyPI [8]
>  * Go Package information and SDK RC [9]
>  * Validation sheet with a tab for 2.42.0 release to help with
> validation [10].
>  * Docker images published to Docker Hub [11]. (Soon)
> 
>  The vote will be open for at least 72 hours. It is adopted by
> majority approval, with at least 3 PMC affirmative votes.
> 
>  Updates from RC1 include a fix to SpannerIO backlog estimation [12]
> and a fix to the BigQueryIO interpretation of coders on an internal flatten
> [13]. Otherwise, previous validation should be unaffected.
> 
>  For guidelines on how to try the release in your projects, check out
> our blog post at https://beam.apache.org/blog/validate-beam-release/.
> 
>  Thanks,
>  Robert Burke
>  2.42.0 Release Manager
> 
>  [1] https://github.com/apache/beam/milestone/4
>  [2] https://dist.apache.org/repos/dist/dev/beam/2.42.0/
>  [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>  [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1286/
>  [5] https://github.com/apache/beam/tree/v2.42.0-RC2
>  [6] https://github.com/apache/beam/pull/23406
>  [7] https://github.com/apache/beam-site/pull/634
>  [8] https://pypi.org/project/apache-beam/2.42.0rc2/
>  [9]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.42.0-RC2/go/pkg/beam
>  [10]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=265602293
>  [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>  [12] https://github.com/apache/beam/issues/23494
>  [13] https://github.com/apache/beam/issues/23561
> 
>

Re: [PROPOSAL] Preparing for Apache Beam 2.43.0 Release

[Ahmet Altay via dev]

+1 - Thank you Cham!

On Wed, Oct 5, 2022 at 1:38 PM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> Hi all,
>
> The next (2.43.0) release branch cut is scheduled for October 19th,
> according to the release calendar [1].
>
> I would like to volunteer myself to do this release. My plan is to cut the
> branch on that date, and cherrypick release-blocking fixes afterwards, if
> any.
>
> Please help me make sure the release goes smoothly by:
> - Making sure that any unresolved release blocking issues for 2.43.0 should
> have their "Milestone" marked as "2.43.0 Release" [2] as soon as possible.
> - Reviewing the current release blockers [2] and removing the Milestone if
> they don't meet the criteria at [3].
>
> Let me know if you have any comments/objections/questions.
>
> Thanks,
> Cham
>
> [1]
> https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com
> [2] https://github.com/apache/beam/milestone/5
> [3] https://beam.apache.org/contribute/release-blocking/
>

Re: [VOTE] Release 2.42.0, release candidate #1

[Ahmet Altay via dev]

+1 (binding) - I validated python quick starts on direct runner.

Thank you for working on the release!

Ahmet

On Mon, Oct 3, 2022 at 9:06 AM Valentyn Tymofieiev via dev <
dev@beam.apache.org> wrote:

> I validated that Dataflow and Beam Python containers have dependencies
> that match Beam requirements.
>
> I came across https://github.com/apache/beam/pull/23200 - there are
> failed tests and I don't see test results for Python PostCommit suites. Do
> you know what's the status of both?
>
> Minor nits: missing substitution in  * Java artifacts were built with
> Gradle GRADLE_VERSION and OpenJDK/Oracle JDK JDK_VERSION.
>
> Thanks!
>
>
>
> On Mon, Oct 3, 2022 at 7:21 AM Ritesh Ghorse via dev 
> wrote:
>
>> +1 (non-binding)
>> Validated Go SDK Quickstart on Direct and Dataflow runner
>>
>>
>> On Mon, Oct 3, 2022 at 9:38 AM Alexey Romanenko 
>> wrote:
>>
>>> +1 (binding)
>>>
>>> Tested with  https://github.com/Talend/beam-samples/
>>> (Java SDK v8 & v11, Spark 3 runner).
>>>
>>> ---
>>> Alexey
>>>
>>> On 3 Oct 2022, at 14:32, Chamikara Jayalath via dev 
>>> wrote:
>>>
>>> +1 (binding)
>>>
>>> Verified checksums and signatures of artifacts.
>>> Validated some multi-language pipelines.
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Thu, Sep 29, 2022 at 6:12 PM Robert Burke via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Hi everyone,
 Please review and vote on the release candidate #1 for the version
 2.42.0, as follows:
 [ ] +1, Approve the release
 [ ] -1, Do not approve the release (please provide specific comments)

 Reviewers are encouraged to test their own use cases with the release
 candidate, and vote +1 if no issues are found.

 The complete staging area is available for your review, which includes:
 * GitHub Release notes [1],
 * the official Apache source release to be deployed to dist.apache.org [2],
 which is signed with the key with fingerprint
 A52F5C83BAE26160120EC25F3D56ACFBFB2975E1 [3],
 * all artifacts to be deployed to the Maven Central Repository [4],
 * source code tag "v2.42.0-RC1" [5],
 * website pull request listing the release [6], the blog post [6], and
 publishing the API reference manual [7].
 * Java artifacts were built with Gradle GRADLE_VERSION and
 OpenJDK/Oracle JDK JDK_VERSION.
 * Python artifacts are deployed along with the source release to the
 dist.apache.org [2] and PyPI [8]
 * Go Package information and SDK RC  [9]
 * Validation sheet with a tab for 2.42.0 release to help with
 validation [10].
 * Docker images published to Docker Hub [11].

 The vote will be open for at least 72 hours. It is adopted by majority
 approval, with at least 3 PMC affirmative votes.

 For guidelines on how to try the release in your projects, check out
 our blog post at https://beam.apache.org/blog/validate-beam-release/.

 Thanks,
 Robert Burke
 2.42.0 Release Manager

 [1] https://github.com/apache/beam/milestone/4
 [2] https://dist.apache.org/repos/dist/dev/beam/2.42.0/
 [3] https://dist.apache.org/repos/dist/release/beam/KEYS
 [4]
 https://repository.apache.org/content/repositories/orgapachebeam-1285/
 [5] https://github.com/apache/beam/tree/v2.42.0-RC1
 [6] https://github.com/apache/beam/pull/23406
 [7] https://github.com/apache/beam-site/pull/634
 [8] https://pypi.org/project/apache-beam/2.42.0rc1/
 [9]
 https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.42.0-RC1/go/pkg/beam

 [10]
 https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=265602293
 [11] https://hub.docker.com/search?q=apache%2Fbeam=image


>>>

Re: Rest connector

[Ahmet Altay via dev]

/cc @John Casey 

On Mon, Sep 12, 2022 at 12:03 PM Reddy annapureddy, Vijayas (V.) <
vredd...@ford.com> wrote:

> Hi Team,
>
> We are planning to use Apache beam for a new microservice project which is
> in the discovery phase.  Apache beam pipeline will read data from GCS and
> send data to other microservice via REST API.
>
> Wanted to check if there is any connector already in place for this use
> case.
>
> Thanks,
> Vijay
>