Re: [VOTE] Release 2.56.0, release candidate #2

[Yi Hu via dev]

+1 (non-binding)

Validated Dataflow Template [1] (except Yaml and Python xlang templates,
which will be validated separately), and GCPIO load tests on Dataflow
runner [2]

[1] https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/1487

[2]
https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/7154d4f4062b54a60dd43851d49d985cd4770e76/it/google-cloud-platform

Thanks,
Yi

On Tue, Apr 30, 2024 at 9:50 PM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> +1 (binding)
>
> Validated multi-lang Java/Python and the transform upgrade feature.
>
> Thanks,
> Cham
>
> On Mon, Apr 29, 2024 at 12:57 AM Jan Lukavský  wrote:
>
>> +1 (binding).
>>
>> Tested Java SDK with Flink runner.
>>
>>  Jan
>> On 4/28/24 15:32, XQ Hu via dev wrote:
>>
>> +1 (non-binding). Tested it using the dataflow ML pipeline:
>> https://github.com/google/dataflow-ml-starter/actions/runs/8862170843/job/24334816481
>>
>> On Sat, Apr 27, 2024 at 7:42 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hi everyone,
>>> Please review and vote on the release candidate #2 for the version
>>> 2.56.0, as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>> Reviewers are encouraged to test their own use cases with the release
>>> candidate, and vote +1 if no issues are found. Only PMC member votes will
>>> count towards the final vote, but votes from all community members is
>>> encouraged and helpful for finding regressions; you can either test your
>>> own use cases [13] or use cases from the validation sheet [10].
>>>
>>> The complete staging area is available for your review, which includes:
>>> * GitHub Release notes [1],
>>> * the official Apache source release to be deployed to dist.apache.org
>>> [2], which is signed with the key with fingerprint D20316F712213422 [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "v2.56.0-RC2" [5],
>>> * website pull request listing the release [6], the blog post [6], and
>>> publishing the API reference manual [7].
>>> * Python artifacts are deployed along with the source release to the
>>> dist.apache.org [2] and PyPI[8].
>>> * Go artifacts and documentation are available at pkg.go.dev [9]
>>> * Validation sheet with a tab for 2.56.0 release to help with validation
>>> [10].
>>> * Docker images published to Docker Hub [11].
>>> * PR to run tests against release branch [12].
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> For guidelines on how to try the release in your projects, check out our
>>> RC testing guide [13].
>>>
>>> Thanks,
>>> Danny
>>>
>>> [1] https://github.com/apache/beam/milestone/20
>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.56.0/
>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [4]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1377/
>>> [5] https://github.com/apache/beam/tree/v2.56.0-RC2
>>> [6] https://github.com/apache/beam/pull/31094
>>> [7] https://github.com/apache/beam-site/pull/665
>>> [8] https://pypi.org/project/apache-beam/2.56.0rc2/
>>> [9]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.56.0-RC2/go/pkg/beam
>>> [10]
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1992402651
>>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>>> [12] https://github.com/apache/beam/pull/31038
>>> [13]
>>> https://github.com/apache/beam/blob/master/contributor-docs/rc-testing-guide.md
>>>
>>

Re: Patch release proposal

[Yi Hu via dev]

> Just releasing Python can break multi-lang by default (unless expansion
service is overridden manually) since we match versions across languages
when picking the default expansion service.

Yes, that's why I proposed "the source code of release candidate (e.g.
apache_beam/version.py) still reads 2.55.0. " Anyways it seems doing a full
release is preferred as it reduces the risk of breakages.

On Thu, Mar 28, 2024 at 11:38 AM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

>
>
> On Thu, Mar 28, 2024 at 8:36 AM Chamikara Jayalath 
> wrote:
>
>> Just releasing Python can break multi-lang by default (unless expansion
>> service is overridden manually) since we match versions across languages
>> when picking the default expansion service.
>>
>>
>> https://github.com/apache/beam/blob/2f8854a3e34f31c1cc034f95ad36f317abc906ff/sdks/python/apache_beam/utils/subprocess_server.py#L42
>>
>
> Correct link:
> https://github.com/apache/beam/blob/2f8854a3e34f31c1cc034f95ad36f317abc906ff/sdks/python/apache_beam/utils/subprocess_server.py#L352
>
>
>>
>>
>> Thanks,
>> Cham
>>
>> On Thu, Mar 28, 2024 at 8:26 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> > The patch itself [1] is trivial, however, the release process is not
>>> trivial. There is little documentation nor practice for a patch release
>>> process. I could imagine two options
>>>
>>> I think there's not a ton of documentation because we haven't done it,
>>> but all the release workflows were authored in such a way that they should
>>> "just work", outside of cutting the release branch itself. So the workflow
>>> should be almost identical to the existing one, but with several steps
>>> skipped (cherry picks, beam website, most validation). Notably, this
>>> shouldn't be any easier/harder if we're doing it for one language or all 3.
>>>
>>> I can take that on if needed.
>>>
>>> > Besides, there should be a Beam YAML validation workflow and added in
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1368030253
>>>
>>> > If we do a patch release for Python SDK, let's also patch another
>>> known issue for which fix is available:
>>> https://github.com/apache/beam/blob/master/CHANGES.md#known-issues-1
>>>
>>> +1 to both of these
>>>
>>> On Thu, Mar 28, 2024 at 11:25 AM Yi Hu via dev 
>>> wrote:
>>>
>>>> Thanks Valentyn for raising this. In this case, Python containers will
>>>> also be included. Different from PyPI wheels, docker tag can override so it
>>>> can stay with 2.55.0
>>>>
>>>> On Thu, Mar 28, 2024 at 11:15 AM Valentyn Tymofieiev <
>>>> valen...@google.com> wrote:
>>>>
>>>>> If we do a patch release for Python SDK, let's also patch another
>>>>> known issue for which fix is available:
>>>>> https://github.com/apache/beam/blob/master/CHANGES.md#known-issues-1
>>>>>
>>>>> On Thu, Mar 28, 2024 at 8:01 AM Yi Hu via dev 
>>>>> wrote:
>>>>>
>>>>>> 2.55.0 release manager here
>>>>>>
>>>>>> The patch itself [1] is trivial, however, the release process is not
>>>>>> trivial. There is little documentation nor practice for a patch release
>>>>>> process. I could imagine two options
>>>>>>
>>>>>> 1. Do a full "2.55.1" release
>>>>>>
>>>>>> 2. Do a patch release only for Python SDK, that is
>>>>>>   a. cherry-pick [1] into release-2.55.0 branch
>>>>>>   b. tag a 2.55.1rc1 release candidate - note that the source code of
>>>>>> release candidate (e.g. apache_beam/version.py) still reads 2.55.0. This
>>>>>> ensures Python SDK picks up the Java expansion service / job server of
>>>>>> existing version (2.55.0). We did it once for Go SDK (
>>>>>> https://github.com/apache/beam/tree/sdks/v2.48.2)
>>>>>>   c. Build the release candidate for Python wheels (also Python
>>>>>> containers? Not sure if it is needed)
>>>>>>   d. send out the RC for validation
>>>>>>   e. finalize the release
>>>>>>
>>>>>> If we decided to do a patch release I would prefer option 2. I can
>>>>>> take on that if decided to do. However, if we decide do 

Re: Patch release proposal

[Yi Hu via dev]

Thanks Valentyn for raising this. In this case, Python containers will also
be included. Different from PyPI wheels, docker tag can override so it can
stay with 2.55.0

On Thu, Mar 28, 2024 at 11:15 AM Valentyn Tymofieiev 
wrote:

> If we do a patch release for Python SDK, let's also patch another known
> issue for which fix is available:
> https://github.com/apache/beam/blob/master/CHANGES.md#known-issues-1
>
> On Thu, Mar 28, 2024 at 8:01 AM Yi Hu via dev  wrote:
>
>> 2.55.0 release manager here
>>
>> The patch itself [1] is trivial, however, the release process is not
>> trivial. There is little documentation nor practice for a patch release
>> process. I could imagine two options
>>
>> 1. Do a full "2.55.1" release
>>
>> 2. Do a patch release only for Python SDK, that is
>>   a. cherry-pick [1] into release-2.55.0 branch
>>   b. tag a 2.55.1rc1 release candidate - note that the source code of
>> release candidate (e.g. apache_beam/version.py) still reads 2.55.0. This
>> ensures Python SDK picks up the Java expansion service / job server of
>> existing version (2.55.0). We did it once for Go SDK (
>> https://github.com/apache/beam/tree/sdks/v2.48.2)
>>   c. Build the release candidate for Python wheels (also Python
>> containers? Not sure if it is needed)
>>   d. send out the RC for validation
>>   e. finalize the release
>>
>> If we decided to do a patch release I would prefer option 2. I can take
>> on that if decided to do. However, if we decide do a full release (or both
>> Java and Python) I would suggest defer to next release cycle, as the
>> release process itself could take ~10 days minimum if there is single RC.
>>
>> Besides, there should be a Beam YAML validation workflow and added in
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1368030253
>>
>>
>> [1] https://github.com/apache/beam/pull/30780
>>
>> On Thu, Mar 28, 2024 at 10:22 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> +1 on a patch release - we've done a fair amount of work to make
>>> releasing easier, and one of my hopes is that it will enable quick patches
>>> like this. I'd vote we try to fix the underlying Java piece as well,
>>> though, doing a patch release for one language shouldn't be significantly
>>> cheaper than doing it for multiple languages.
>>>
>>> Thanks,
>>> Danny
>>>
>>> On Wed, Mar 27, 2024 at 7:19 PM Robert Burke  wrote:
>>>
>>>> +1 to a targeted patch release.
>>>>
>>>> We did the same for the Go SDK a little while back. It would be good to
>>>> see what's different for a different SDK.
>>>>
>>>> On Wed, Mar 27, 2024, 4:01 PM Robert Bradshaw via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> Given the severity of the breakage, and the simplicity of the
>>>>> workaround, I'm in favor of a patch release. I think we could do
>>>>> Python-only, which would make the process even more lightweight.
>>>>>
>>>>> On Wed, Mar 27, 2024 at 3:48 PM Jeff Kinard 
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> Beam 2.55 was released with a bug that causes WriteToJson on Beam
>>>>>> YAML to fail when using the Java variant. This also affects any user
>>>>>> attempting to use the Xlang JsonWriteTransformProvider -
>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/json/src/main/java/org/apache/beam/sdk/io/json/providers/JsonWriteTransformProvider.java
>>>>>>
>>>>>> This is due to a change to
>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/json/build.gradle
>>>>>> that removed
>>>>>> a dependency on everit which also removed it from being packaged into
>>>>>> the expansion service JAR:
>>>>>> beam-sdks-java-extensions-sql-expansion-service-2.55.0.jar
>>>>>>
>>>>>> There is a temporary fix to disable the provider in Beam YAML:
>>>>>> https://github.com/apache/beam/pull/30777
>>>>>>
>>>>>> I think with the total loss of function, and a trivial fix, it is
>>>>>> worth creating a patch release of Beam 2.55 to include this fix.
>>>>>>
>>>>>> - Jeff
>>>>>>
>>>>>>

Re: Patch release proposal

[Yi Hu via dev]

2.55.0 release manager here

The patch itself [1] is trivial, however, the release process is not
trivial. There is little documentation nor practice for a patch release
process. I could imagine two options

1. Do a full "2.55.1" release

2. Do a patch release only for Python SDK, that is
  a. cherry-pick [1] into release-2.55.0 branch
  b. tag a 2.55.1rc1 release candidate - note that the source code of
release candidate (e.g. apache_beam/version.py) still reads 2.55.0. This
ensures Python SDK picks up the Java expansion service / job server of
existing version (2.55.0). We did it once for Go SDK (
https://github.com/apache/beam/tree/sdks/v2.48.2)
  c. Build the release candidate for Python wheels (also Python containers?
Not sure if it is needed)
  d. send out the RC for validation
  e. finalize the release

If we decided to do a patch release I would prefer option 2. I can take on
that if decided to do. However, if we decide do a full release (or both
Java and Python) I would suggest defer to next release cycle, as the
release process itself could take ~10 days minimum if there is single RC.

Besides, there should be a Beam YAML validation workflow and added in
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1368030253


[1] https://github.com/apache/beam/pull/30780

On Thu, Mar 28, 2024 at 10:22 AM Danny McCormick via dev <
dev@beam.apache.org> wrote:

> +1 on a patch release - we've done a fair amount of work to make releasing
> easier, and one of my hopes is that it will enable quick patches like this.
> I'd vote we try to fix the underlying Java piece as well, though, doing a
> patch release for one language shouldn't be significantly cheaper than
> doing it for multiple languages.
>
> Thanks,
> Danny
>
> On Wed, Mar 27, 2024 at 7:19 PM Robert Burke  wrote:
>
>> +1 to a targeted patch release.
>>
>> We did the same for the Go SDK a little while back. It would be good to
>> see what's different for a different SDK.
>>
>> On Wed, Mar 27, 2024, 4:01 PM Robert Bradshaw via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Given the severity of the breakage, and the simplicity of the
>>> workaround, I'm in favor of a patch release. I think we could do
>>> Python-only, which would make the process even more lightweight.
>>>
>>> On Wed, Mar 27, 2024 at 3:48 PM Jeff Kinard  wrote:
>>>
 Hi all,

 Beam 2.55 was released with a bug that causes WriteToJson on Beam YAML
 to fail when using the Java variant. This also affects any user attempting
 to use the Xlang JsonWriteTransformProvider -
 https://github.com/apache/beam/blob/master/sdks/java/io/json/src/main/java/org/apache/beam/sdk/io/json/providers/JsonWriteTransformProvider.java

 This is due to a change to
 https://github.com/apache/beam/blob/master/sdks/java/io/json/build.gradle
 that removed
 a dependency on everit which also removed it from being packaged into
 the expansion service JAR:
 beam-sdks-java-extensions-sql-expansion-service-2.55.0.jar

 There is a temporary fix to disable the provider in Beam YAML:
 https://github.com/apache/beam/pull/30777

 I think with the total loss of function, and a trivial fix, it is worth
 creating a patch release of Beam 2.55 to include this fix.

 - Jeff

[Announce] Beam 2.55.0 Release

[Yi Hu via dev]

We are happy to present the new 2.55.0 release of Beam.
This release includes both improvements and new functionality.
See https://beam.apache.org/get-started/downloads/ for this release.

For more information on changes in 2.55.0, check out the detailed release
notes at https://github.com/apache/beam/milestone/19 .

- Highlights

* The Python SDK will now include automatically generated wrappers for
external Java transforms! (https://github.com/apache/beam/pull/29834)

- I/Os

* Added support for handling bad records to BigQueryIO (
https://github.com/apache/beam/pull/30081).
  * Full Support for Storage Read and Write APIs
  * Partial Support for File Loads (Failures writing to files supported,
failures loading files to BQ unsupported)
  * No Support for Extract or Streaming Inserts
* Added support for handling bad records to PubSubIO (
https://github.com/apache/beam/pull/30372).
  * Support is not available for handling schema mismatches, and enabling
error handling for writing to Pub/Sub topics with schemas is not recommended
* `--enableBundling` pipeline option for BigQueryIO DIRECT_READ is replaced
by `--enableStorageReadApiV2`. Both were considered experimental and
subject to change (Java) (https://github.com/apache/beam/issues/26354).

- New Features / Improvements

* Allow writing clustered and not time-partitioned BigQuery tables (Java) (
https://github.com/apache/beam/pull/30094).
* Redis cache support added to RequestResponseIO and Enrichment transform
(Python) (https://github.com/apache/beam/pull/30307).
* Merged `sdks/java/fn-execution` and `runners/core-construction-java` into
the main SDK. These artifacts were never meant for users, but noting
  that they no longer exist. These are steps to bring portability into the
core SDK alongside all other core functionality.
* Added Vertex AI Feature Store handler for Enrichment transform (Python) (
https://github.com/apache/beam/pull/30388).

- Breaking Changes

* Arrow version was bumped to 15.0.0 from 5.0.0 (
https://github.com/apache/beam/pull/30181).
* Go SDK users who build custom worker containers may run into issues with
the move to distroless containers as a base (see Security Fixes).
  * The issue stems from distroless containers lacking additional tools,
which current custom container processes may rely on.
  * See
https://beam.apache.org/documentation/runtime/environments/#from-scratch-go
for instructions on building and using a custom container.
* Python SDK has changed the default value for the
`--max_cache_memory_usage_mb` pipeline option from 100 to 0. This option
was first introduced in the 2.52.0 SDK version. This change restores the
behavior of the 2.51.0 SDK, which does not use the state cache. If your
pipeline uses iterable side inputs views, consider increasing the cache
size by setting the option manually. (
https://github.com/apache/beam/issues/30360).

- Deprecations

* N/A

- Bug fixes

* Fixed `SpannerIO.readChangeStream` to support propagating credentials
from pipeline options
  to the `getDialect` calls for authenticating with Spanner (Java) (
https://github.com/apache/beam/pull/30361).
* Reduced the number of HTTP requests in GCSIO function calls (Python) (
https://github.com/apache/beam/pull/30205).

- Security Fixes

* Go SDK base container image moved to distroless/base-nossl-debian12,
reducing vulnerable container surface to kernel and glibc (
https://github.com/apache/beam/pull/30011).

- Known Issues

* In Python pipelines, when shutting down inactive bundle processors,
shutdown logic can overaggressively hold the lock, blocking acceptance of
new work. Symptoms of this issue include slowness or stuckness in
long-running jobs. Fixed in 2.56.0 (
https://github.com/apache/beam/pull/30679).

- List of Contributors

According to git shortlog, the following people contributed to the
{$RELEASE_VERSION} release. Thank you to all contributors!

Ahmed Abualsaud

Anand Inguva

Andrew Crites

Andrey Devyatkin

Arun Pandian

Arvind Ram

Chamikara Jayalath

Chris Gray

Claire McGinty

Damon Douglas

Dan Ellis

Danny McCormick

Daria Bezkorovaina

Dima I

Edward Cui

Ferran Fernández Garrido

GStravinsky

Jan Lukavský

Jason Mitchell

JayajP

Jeff Kinard

Jeffrey Kinard

Kenneth Knowles

Mattie Fu

Michel Davit

Oleh Borysevych

Ritesh Ghorse

Ritesh Tarway

Robert Bradshaw

Robert Burke

Sam Whittle

Scott Strong

Shunping Huang

Steven van Rossum

Svetak Sundhar

Talat UYARER

Ukjae Jeong (Jay)

Vitaly Terentyev

Vlado Djerek

Yi Hu

akashorabek

case-k

clmccart

dengwe1

dhruvdua

hardshah

johnjcasey

liferoad

martin trieu

tvalentyn

-


Release Manager

-- 

Yi Hu, (he/him/his)

Software Engineer

Re: [VOTE] Release 2.55.0, release candidate #3

[Yi Hu via dev]

Hi,

Could a PMC member please help me finalize the source release? That is the
steps documented here:
https://github.com/apache/beam/blob/master/contributor-docs/release-guide.md#pmc-only-finalization


Thanks!


On Mon, Mar 25, 2024 at 9:37 AM Yi Hu  wrote:

> I'm happy to announce that we have unanimously approved this release.
>
> There are 7 approving votes, 3 of which are binding:
>
> * Ritesh Ghorse
> * XQ Hu
> * Chamikara Jayalath (binding)
> * Jan Lukavský (binding)
> * Danny McCormick
> * Yi Hu
> * Valentyn Tymofieiev
>
> There are no disapproving votes.
>
> Thanks everyone!
>
> Release Manager
>
> On Fri, Mar 22, 2024 at 6:01 PM Valentyn Tymofieiev 
> wrote:
>
>> +1 (binding). Checked some of the released artifacts, release blog, and
>> ran a couple Python pipelines on Dataflow.
>>
>>
>> > * GitHub Release notes [1]
>>
>> Is the link correct? It points to the milestone.
>>
>> On Fri, Mar 22, 2024 at 1:10 PM Yi Hu via dev 
>> wrote:
>>
>>> +1 (non-binding)
>>>
>>> 1. Checked published Java artifacts
>>>
>>> 2. Tested with GCP IO performance benchmark
>>>
>>> 3. Tested with Java PostRelease workflow (including
>>> QuickstartJavaDirect, QuickstartJavaDataflow, QuickstartJavaSpark,
>>> QuickstartJavaTwister2, QuickstartJavaFlinkLocal, MobileGamingJavaDirect,
>>> GamingJavaDataflow, MobileGamingJavaDataflowBom) [2]
>>>
>>> [1]
>>> https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/main/it/google-cloud-platform
>>> [2] https://github.com/apache/beam/pull/30721
>>>
>>> On Thu, Mar 21, 2024 at 10:59 AM Danny McCormick via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> +1 - validated some ML examples with the interactive runner
>>>>
>>>> Thanks,
>>>> Danny
>>>>
>>>> On Thu, Mar 21, 2024 at 9:21 AM Jan Lukavský  wrote:
>>>>
>>>>> +1 (binding)
>>>>>
>>>>> Tested Java SDK with FlinkRunner.
>>>>>
>>>>>  Jan
>>>>> On 3/20/24 22:40, Chamikara Jayalath via dev wrote:
>>>>>
>>>>> +1 (binding)
>>>>>
>>>>> Tested multi-lang Java/Python pipelines and upgrading BQ/Kafka
>>>>> transforms from 2.53.0 to 2.55.0 using the Transform Service.
>>>>>
>>>>> Thanks,
>>>>> Cham
>>>>>
>>>>> On Tue, Mar 19, 2024 at 2:10 PM XQ Hu via dev 
>>>>> wrote:
>>>>>
>>>>>> +1 (non-binding). Ran the simple ML pipeline without any issue:
>>>>>> https://github.com/google/dataflow-ml-starter/actions/runs/8349158153
>>>>>>
>>>>>> On Tue, Mar 19, 2024 at 11:55 AM Ritesh Ghorse via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>>> +1 (non-binding) - Ran a few python batch examples on Direct and
>>>>>>> Dataflow runner.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> On Tue, Mar 19, 2024 at 10:56 AM Yi Hu via dev 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>> Please review and vote on the release candidate #3 for the version
>>>>>>>> 2.55.0, as follows:
>>>>>>>>
>>>>>>>> [ ] +1, Approve the release
>>>>>>>> [ ] -1, Do not approve the release (please provide specific
>>>>>>>> comments)
>>>>>>>>
>>>>>>>>
>>>>>>>> Reviewers are encouraged to test their own use cases with the
>>>>>>>> release candidate, and vote +1 if
>>>>>>>> no issues are found. Only PMC member votes will count towards the
>>>>>>>> final vote, but votes from all
>>>>>>>> community members is encouraged and helpful for finding
>>>>>>>> regressions; you can either test your own
>>>>>>>> use cases [13] or use cases from the validation sheet [10].
>>>>>>>>
>>>>>>>> The complete staging area is available for your review, which
>>>>>>>> includes:
>>>>>>>> * GitHub Release notes [1],
>>>>>>>> * the official Apache source release to be deployed to
>&g

Re: [VOTE] Release 2.55.0, release candidate #3

[Yi Hu via dev]

I'm happy to announce that we have unanimously approved this release.

There are 7 approving votes, 3 of which are binding:

* Ritesh Ghorse
* XQ Hu
* Chamikara Jayalath (binding)
* Jan Lukavský (binding)
* Danny McCormick
* Yi Hu
* Valentyn Tymofieiev

There are no disapproving votes.

Thanks everyone!

Release Manager

On Fri, Mar 22, 2024 at 6:01 PM Valentyn Tymofieiev 
wrote:

> +1 (binding). Checked some of the released artifacts, release blog, and
> ran a couple Python pipelines on Dataflow.
>
>
> > * GitHub Release notes [1]
>
> Is the link correct? It points to the milestone.
>
> On Fri, Mar 22, 2024 at 1:10 PM Yi Hu via dev  wrote:
>
>> +1 (non-binding)
>>
>> 1. Checked published Java artifacts
>>
>> 2. Tested with GCP IO performance benchmark
>>
>> 3. Tested with Java PostRelease workflow (including QuickstartJavaDirect,
>> QuickstartJavaDataflow, QuickstartJavaSpark, QuickstartJavaTwister2,
>> QuickstartJavaFlinkLocal, MobileGamingJavaDirect, GamingJavaDataflow,
>> MobileGamingJavaDataflowBom) [2]
>>
>> [1]
>> https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/main/it/google-cloud-platform
>> [2] https://github.com/apache/beam/pull/30721
>>
>> On Thu, Mar 21, 2024 at 10:59 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> +1 - validated some ML examples with the interactive runner
>>>
>>> Thanks,
>>> Danny
>>>
>>> On Thu, Mar 21, 2024 at 9:21 AM Jan Lukavský  wrote:
>>>
>>>> +1 (binding)
>>>>
>>>> Tested Java SDK with FlinkRunner.
>>>>
>>>>  Jan
>>>> On 3/20/24 22:40, Chamikara Jayalath via dev wrote:
>>>>
>>>> +1 (binding)
>>>>
>>>> Tested multi-lang Java/Python pipelines and upgrading BQ/Kafka
>>>> transforms from 2.53.0 to 2.55.0 using the Transform Service.
>>>>
>>>> Thanks,
>>>> Cham
>>>>
>>>> On Tue, Mar 19, 2024 at 2:10 PM XQ Hu via dev 
>>>> wrote:
>>>>
>>>>> +1 (non-binding). Ran the simple ML pipeline without any issue:
>>>>> https://github.com/google/dataflow-ml-starter/actions/runs/8349158153
>>>>>
>>>>> On Tue, Mar 19, 2024 at 11:55 AM Ritesh Ghorse via dev <
>>>>> dev@beam.apache.org> wrote:
>>>>>
>>>>>> +1 (non-binding) - Ran a few python batch examples on Direct and
>>>>>> Dataflow runner.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> On Tue, Mar 19, 2024 at 10:56 AM Yi Hu via dev 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>> Please review and vote on the release candidate #3 for the version
>>>>>>> 2.55.0, as follows:
>>>>>>>
>>>>>>> [ ] +1, Approve the release
>>>>>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>>>>>
>>>>>>>
>>>>>>> Reviewers are encouraged to test their own use cases with the
>>>>>>> release candidate, and vote +1 if
>>>>>>> no issues are found. Only PMC member votes will count towards the
>>>>>>> final vote, but votes from all
>>>>>>> community members is encouraged and helpful for finding regressions;
>>>>>>> you can either test your own
>>>>>>> use cases [13] or use cases from the validation sheet [10].
>>>>>>>
>>>>>>> The complete staging area is available for your review, which
>>>>>>> includes:
>>>>>>> * GitHub Release notes [1],
>>>>>>> * the official Apache source release to be deployed to
>>>>>>> dist.apache.org [2], which is signed with the key with fingerprint
>>>>>>> D20316F712213422 [3],
>>>>>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>>>>>> * source code tag "v2.55.0-RC3" [5],
>>>>>>> * website pull request listing the release [6], the blog post [6],
>>>>>>> and publishing the API reference manual [7].
>>>>>>> * Python artifacts are deployed along with the source release to the
>>>>>>> dist.apache.org [2] and PyPI [8].
>>>>>>> * Go artifacts and documentation are available at pkg.go.dev [9]
>>>>

Re: [VOTE] Release 2.55.0, release candidate #3

[Yi Hu via dev]

+1 (non-binding)

1. Checked published Java artifacts

2. Tested with GCP IO performance benchmark

3. Tested with Java PostRelease workflow (including QuickstartJavaDirect,
QuickstartJavaDataflow, QuickstartJavaSpark, QuickstartJavaTwister2,
QuickstartJavaFlinkLocal, MobileGamingJavaDirect, GamingJavaDataflow,
MobileGamingJavaDataflowBom) [2]

[1]
https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/main/it/google-cloud-platform
[2] https://github.com/apache/beam/pull/30721

On Thu, Mar 21, 2024 at 10:59 AM Danny McCormick via dev <
dev@beam.apache.org> wrote:

> +1 - validated some ML examples with the interactive runner
>
> Thanks,
> Danny
>
> On Thu, Mar 21, 2024 at 9:21 AM Jan Lukavský  wrote:
>
>> +1 (binding)
>>
>> Tested Java SDK with FlinkRunner.
>>
>>  Jan
>> On 3/20/24 22:40, Chamikara Jayalath via dev wrote:
>>
>> +1 (binding)
>>
>> Tested multi-lang Java/Python pipelines and upgrading BQ/Kafka transforms
>> from 2.53.0 to 2.55.0 using the Transform Service.
>>
>> Thanks,
>> Cham
>>
>> On Tue, Mar 19, 2024 at 2:10 PM XQ Hu via dev 
>> wrote:
>>
>>> +1 (non-binding). Ran the simple ML pipeline without any issue:
>>> https://github.com/google/dataflow-ml-starter/actions/runs/8349158153
>>>
>>> On Tue, Mar 19, 2024 at 11:55 AM Ritesh Ghorse via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> +1 (non-binding) - Ran a few python batch examples on Direct and
>>>> Dataflow runner.
>>>>
>>>> Thanks!
>>>>
>>>> On Tue, Mar 19, 2024 at 10:56 AM Yi Hu via dev 
>>>> wrote:
>>>>
>>>>> Hi everyone,
>>>>> Please review and vote on the release candidate #3 for the version
>>>>> 2.55.0, as follows:
>>>>>
>>>>> [ ] +1, Approve the release
>>>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>>>
>>>>>
>>>>> Reviewers are encouraged to test their own use cases with the release
>>>>> candidate, and vote +1 if
>>>>> no issues are found. Only PMC member votes will count towards the
>>>>> final vote, but votes from all
>>>>> community members is encouraged and helpful for finding regressions;
>>>>> you can either test your own
>>>>> use cases [13] or use cases from the validation sheet [10].
>>>>>
>>>>> The complete staging area is available for your review, which includes:
>>>>> * GitHub Release notes [1],
>>>>> * the official Apache source release to be deployed to dist.apache.org
>>>>> [2], which is signed with the key with fingerprint D20316F712213422 [3],
>>>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>>>> * source code tag "v2.55.0-RC3" [5],
>>>>> * website pull request listing the release [6], the blog post [6], and
>>>>> publishing the API reference manual [7].
>>>>> * Python artifacts are deployed along with the source release to the
>>>>> dist.apache.org [2] and PyPI [8].
>>>>> * Go artifacts and documentation are available at pkg.go.dev [9]
>>>>> * Validation sheet with a tab for 2.55.0 release to help with
>>>>> validation [10].
>>>>> * Docker images published to Docker Hub [11].
>>>>> * PR to run tests against release branch [12].
>>>>>
>>>>> The vote will be open for at least 72 hours. It is adopted by majority
>>>>> approval, with at least 3 PMC affirmative votes.
>>>>>
>>>>> For guidelines on how to try the release in your projects, check out
>>>>> our RC testing guide [13].
>>>>>
>>>>> Thanks,
>>>>> Release Manager
>>>>>
>>>>> [1] https://github.com/apache/beam/milestone/19
>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.55.0/
>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>>> [4]
>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1373/
>>>>> [5] https://github.com/apache/beam/tree/v2.55.0-RC3
>>>>> [6] https://github.com/apache/beam/pull/30607
>>>>> [7] https://github.com/apache/beam-site/pull/661
>>>>> [8] https://pypi.org/project/apache-beam/2.55.0rc3/
>>>>> [9]
>>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.55.0-RC3/go/pkg/beam
>>>>> [10]
>>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1368030253
>>>>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>>>>> [12] https://github.com/apache/beam/pull/30569
>>>>> [13]
>>>>> https://github.com/apache/beam/blob/master/contributor-docs/rc-testing-guide.md
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Yi Hu, (he/him/his)
>>>>>
>>>>> Software Engineer
>>>>>
>>>>>
>>>>>

[VOTE] Release 2.55.0, release candidate #3

[Yi Hu via dev]

Hi everyone,
Please review and vote on the release candidate #3 for the version 2.55.0,
as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


Reviewers are encouraged to test their own use cases with the release
candidate, and vote +1 if
no issues are found. Only PMC member votes will count towards the final
vote, but votes from all
community members is encouraged and helpful for finding regressions; you
can either test your own
use cases [13] or use cases from the validation sheet [10].

The complete staging area is available for your review, which includes:
* GitHub Release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with fingerprint D20316F712213422 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.55.0-RC3" [5],
* website pull request listing the release [6], the blog post [6], and
publishing the API reference manual [7].
* Python artifacts are deployed along with the source release to the
dist.apache.org [2] and PyPI [8].
* Go artifacts and documentation are available at pkg.go.dev [9]
* Validation sheet with a tab for 2.55.0 release to help with validation
[10].
* Docker images published to Docker Hub [11].
* PR to run tests against release branch [12].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

For guidelines on how to try the release in your projects, check out our RC
testing guide [13].

Thanks,
Release Manager

[1] https://github.com/apache/beam/milestone/19
[2] https://dist.apache.org/repos/dist/dev/beam/2.55.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1373/
[5] https://github.com/apache/beam/tree/v2.55.0-RC3
[6] https://github.com/apache/beam/pull/30607
[7] https://github.com/apache/beam-site/pull/661
[8] https://pypi.org/project/apache-beam/2.55.0rc3/
[9]
https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.55.0-RC3/go/pkg/beam
[10]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1368030253
[11] https://hub.docker.com/search?q=apache%2Fbeam=image
[12] https://github.com/apache/beam/pull/30569
[13]
https://github.com/apache/beam/blob/master/contributor-docs/rc-testing-guide.md


--

Yi Hu, (he/him/his)

Software Engineer

Re: [VOTE] Release 2.55.0, release candidate #2

[Yi Hu via dev]

Hi everyone,

Thanks for contributing to the RC validation. As of actions relevant fixes
are made into the release branch. RC3 is tagged and will be built shortly.


On Mon, Mar 18, 2024 at 2:07 PM Arvind Ram Anantharam 
wrote:

> We had an issue at #29882 which resulted in excessive logging and causing
> a throttling to log in dataflow streaming jobs causing unpleasant
> experiences to our users. The revert(#30648) is already submitted in the
> master and would like to cherrypick that into the release-2.55.0.
>
> On Fri, Mar 15, 2024 at 2:43 PM Ahmed Abualsaud 
> wrote:
>
>>
>>
>> ------ Forwarded message -
>> From: Yi Hu via dev 
>> Date: Fri, Mar 15, 2024 at 10:54 AM
>> Subject: Re: [VOTE] Release 2.55.0, release candidate #2
>> To: dev 
>> CC: Yi Hu 
>>
>>
>> +1 (non-binding).
>>
>> Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates
>> (Java SDK 11, Dataflow runner, details:
>> https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/1366).
>>
>> On Fri, Mar 15, 2024 at 10:37 AM Yi Hu  wrote:
>>
>>> Hi everyone,
>>> Please review and vote on the release candidate #1 for the version
>>> 2.55.0, as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>>
>>> Reviewers are encouraged to test their own use cases with the release
>>> candidate, and vote +1 if
>>> no issues are found. Only PMC member votes will count towards the final
>>> vote, but votes from all
>>> community members is encouraged and helpful for finding regressions; you
>>> can either test your own
>>> use cases [13] or use cases from the validation sheet [10].
>>>
>>> The complete staging area is available for your review, which includes:
>>> * GitHub Release notes [1],
>>> * the official Apache source release to be deployed to dist.apache.org [2],
>>> which is signed with the key with fingerprint D20316F712213422 [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "v2.55.0-RC1" [5],
>>> * website pull request listing the release [6], the blog post [6], and
>>> publishing the API reference manual [7].
>>> * Python artifacts are deployed along with the source release to the
>>> dist.apache.org [2] and PyPI [8].
>>> * Go artifacts and documentation are available at pkg.go.dev [9]
>>> * Validation sheet with a tab for 2.55.0 release to help with validation
>>> [10].
>>> * Docker images published to Docker Hub [11].
>>> * PR to run tests against release branch [12].
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> For guidelines on how to try the release in your projects, check out our
>>> RC testing guide [13].
>>>
>>> Note: as there is only one cherry-pick compared to RC1 (#30637)
>>> regarding a dependency change, not all artifacts are rebuilt compared to
>>> RC1. The updated artifacts are
>>> 1. the Java artifacts [4]
>>> 2. the source release [2].
>>>
>>> Thanks,
>>> Release Manager
>>>
>>> [1] https://github.com/apache/beam/milestone/19
>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.55.0/
>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [4]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1372/
>>> [5] https://github.com/apache/beam/tree/v2.55.0-RC2
>>> [6] https://github.com/apache/beam/pull/30607
>>> [7] https://github.com/apache/beam-site/pull/660
>>> [8] https://pypi.org/project/apache-beam/2.55.0rc1/
>>> [9]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.55.0-RC1/go/pkg/beam
>>> [10]
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1368030253
>>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>>> [12] https://github.com/apache/beam/pull/30569
>>> [13]
>>> https://github.com/apache/beam/blob/master/contributor-docs/rc-testing-guide.md
>>>
>>> [14] https://github.com/apache/beam/compare/v2.55.0-RC1...v2.55.0-RC2
>>>
>>> --
>>>
>>> Yi Hu, (he/him/his)
>>>
>>> Software Engineer
>>>
>>>
>>>

Re: Update confluent dependencies version in kafka io

[Yi Hu via dev]

+1 to move forward with avro. Users can still stick with old avro version
if chosen to do so, and we have Avro compatibility tests also passed in
https://github.com/apache/beam/pull/30638

Thanks for doing this!

On Fri, Mar 15, 2024 at 3:30 PM Maciej Szwaja via dev 
wrote:

> Hi team, thanks for the answers and an update from me - I've created the
> PR https://github.com/apache/beam/pull/30638 which seems to pass the
> tests (sometimes they're flaky though, so I didn't manage to get all green
> in the checks yet), and also I was able to run a pipeline that reads avro
> records from kafka topic and then writes them down to bigquery.
>
> In the meantime though, Yi Hu pointed out that updating the avro version
> (and I need to do that if I want to update the confluent libs to the most
> recent version) is not a trivial issue, and there's been discussions around
> it in the past. I must admit I'd probably have to study the beam codebase a
> bit more to understand the risks around it - so I wanted to ask for advice
> on how to proceed. I could try updating the confluent version more
> gradually - but I think even the 5.4.0, which is the lowest one that
> includes the functionality that I wanted to have, would still probably
> require upgrading to avro 1.9.2. On the other hand - maybe the reservations
> that were relevant earlier (for example - the leaking of avro lib to the
> users' code as a result of it being part of the java sdk core - I think
> it's been migrated out to its own extension since then if I'm not mistaken)
> are no longer that worrisome, and we could try to finally move forward with
> avro?
>

Re: [VOTE] Release 2.55.0, release candidate #2

[Yi Hu via dev]

 Yes, there is only one cherry-pick compared to RC1 (#30637) regarding a
dependency change, not all artifacts are rebuilt compared to RC1. The
updated artifacts are

1. the Java artifacts [4]
2. the source release [2]

For Python release the artifacts to validate is RC1

[2] https://dist.apache.org/repos/dist/dev/beam/2.55.0/
[4] https://repository.apache.org/content/repositories/orgapachebeam-1372/
[8] https://pypi.org/project/apache-beam/2.55.0rc1/


On Sat, Mar 16, 2024 at 11:03 AM XQ Hu  wrote:

> I do not see RC2 for Python SDK here:
> https://pypi.org/project/apache-beam/#history
>
> On Fri, Mar 15, 2024 at 10:54 AM Yi Hu via dev 
> wrote:
>
>> +1 (non-binding).
>>
>> Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates
>> (Java SDK 11, Dataflow runner, details:
>> https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/1366).
>>
>> On Fri, Mar 15, 2024 at 10:37 AM Yi Hu  wrote:
>>
>>> Hi everyone,
>>> Please review and vote on the release candidate #1 for the version
>>> 2.55.0, as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>>
>>> Reviewers are encouraged to test their own use cases with the release
>>> candidate, and vote +1 if
>>> no issues are found. Only PMC member votes will count towards the final
>>> vote, but votes from all
>>> community members is encouraged and helpful for finding regressions; you
>>> can either test your own
>>> use cases [13] or use cases from the validation sheet [10].
>>>
>>> The complete staging area is available for your review, which includes:
>>> * GitHub Release notes [1],
>>> * the official Apache source release to be deployed to dist.apache.org [2],
>>> which is signed with the key with fingerprint D20316F712213422 [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "v2.55.0-RC1" [5],
>>> * website pull request listing the release [6], the blog post [6], and
>>> publishing the API reference manual [7].
>>> * Python artifacts are deployed along with the source release to the
>>> dist.apache.org [2] and PyPI [8].
>>> * Go artifacts and documentation are available at pkg.go.dev [9]
>>> * Validation sheet with a tab for 2.55.0 release to help with validation
>>> [10].
>>> * Docker images published to Docker Hub [11].
>>> * PR to run tests against release branch [12].
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> For guidelines on how to try the release in your projects, check out our
>>> RC testing guide [13].
>>>
>>> Note: as there is only one cherry-pick compared to RC1 (#30637)
>>> regarding a dependency change, not all artifacts are rebuilt compared to
>>> RC1. The updated artifacts are
>>> 1. the Java artifacts [4]
>>> 2. the source release [2].
>>>
>>> Thanks,
>>> Release Manager
>>>
>>> [1] https://github.com/apache/beam/milestone/19
>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.55.0/
>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [4]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1372/
>>> [5] https://github.com/apache/beam/tree/v2.55.0-RC2
>>> [6] https://github.com/apache/beam/pull/30607
>>> [7] https://github.com/apache/beam-site/pull/660
>>> [8] https://pypi.org/project/apache-beam/2.55.0rc1/
>>> [9]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.55.0-RC1/go/pkg/beam
>>> [10]
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1368030253
>>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>>> [12] https://github.com/apache/beam/pull/30569
>>> [13]
>>> https://github.com/apache/beam/blob/master/contributor-docs/rc-testing-guide.md
>>>
>>> [14] https://github.com/apache/beam/compare/v2.55.0-RC1...v2.55.0-RC2
>>>
>>> --
>>>
>>> Yi Hu, (he/him/his)
>>>
>>> Software Engineer
>>>
>>>
>>>

Re: [VOTE] Release 2.55.0, release candidate #2

[Yi Hu via dev]

+1 (non-binding).

Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates (Java
SDK 11, Dataflow runner, details:
https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/1366).

On Fri, Mar 15, 2024 at 10:37 AM Yi Hu  wrote:

> Hi everyone,
> Please review and vote on the release candidate #1 for the version 2.55.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> Reviewers are encouraged to test their own use cases with the release
> candidate, and vote +1 if
> no issues are found. Only PMC member votes will count towards the final
> vote, but votes from all
> community members is encouraged and helpful for finding regressions; you
> can either test your own
> use cases [13] or use cases from the validation sheet [10].
>
> The complete staging area is available for your review, which includes:
> * GitHub Release notes [1],
> * the official Apache source release to be deployed to dist.apache.org [2],
> which is signed with the key with fingerprint D20316F712213422 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.55.0-RC1" [5],
> * website pull request listing the release [6], the blog post [6], and
> publishing the API reference manual [7].
> * Python artifacts are deployed along with the source release to the
> dist.apache.org [2] and PyPI [8].
> * Go artifacts and documentation are available at pkg.go.dev [9]
> * Validation sheet with a tab for 2.55.0 release to help with validation
> [10].
> * Docker images published to Docker Hub [11].
> * PR to run tests against release branch [12].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> For guidelines on how to try the release in your projects, check out our
> RC testing guide [13].
>
> Note: as there is only one cherry-pick compared to RC1 (#30637) regarding
> a dependency change, not all artifacts are rebuilt compared to RC1. The
> updated artifacts are
> 1. the Java artifacts [4]
> 2. the source release [2].
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/beam/milestone/19
> [2] https://dist.apache.org/repos/dist/dev/beam/2.55.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1372/
> [5] https://github.com/apache/beam/tree/v2.55.0-RC2
> [6] https://github.com/apache/beam/pull/30607
> [7] https://github.com/apache/beam-site/pull/660
> [8] https://pypi.org/project/apache-beam/2.55.0rc1/
> [9]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.55.0-RC1/go/pkg/beam
> [10]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1368030253
> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
> [12] https://github.com/apache/beam/pull/30569
> [13]
> https://github.com/apache/beam/blob/master/contributor-docs/rc-testing-guide.md
>
> [14] https://github.com/apache/beam/compare/v2.55.0-RC1...v2.55.0-RC2
>
> --
>
> Yi Hu, (he/him/his)
>
> Software Engineer
>
>
>

[VOTE] Release 2.55.0, release candidate #2

[Yi Hu via dev]

Hi everyone,
Please review and vote on the release candidate #1 for the version 2.55.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


Reviewers are encouraged to test their own use cases with the release
candidate, and vote +1 if
no issues are found. Only PMC member votes will count towards the final
vote, but votes from all
community members is encouraged and helpful for finding regressions; you
can either test your own
use cases [13] or use cases from the validation sheet [10].

The complete staging area is available for your review, which includes:
* GitHub Release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with fingerprint D20316F712213422 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.55.0-RC1" [5],
* website pull request listing the release [6], the blog post [6], and
publishing the API reference manual [7].
* Python artifacts are deployed along with the source release to the
dist.apache.org [2] and PyPI [8].
* Go artifacts and documentation are available at pkg.go.dev [9]
* Validation sheet with a tab for 2.55.0 release to help with validation
[10].
* Docker images published to Docker Hub [11].
* PR to run tests against release branch [12].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

For guidelines on how to try the release in your projects, check out our RC
testing guide [13].

Note: as there is only one cherry-pick compared to RC1 (#30637) regarding a
dependency change, not all artifacts are rebuilt compared to RC1. The
updated artifacts are
1. the Java artifacts [4]
2. the source release [2].

Thanks,
Release Manager

[1] https://github.com/apache/beam/milestone/19
[2] https://dist.apache.org/repos/dist/dev/beam/2.55.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1372/
[5] https://github.com/apache/beam/tree/v2.55.0-RC2
[6] https://github.com/apache/beam/pull/30607
[7] https://github.com/apache/beam-site/pull/660
[8] https://pypi.org/project/apache-beam/2.55.0rc1/
[9]
https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.55.0-RC1/go/pkg/beam
[10]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1368030253
[11] https://hub.docker.com/search?q=apache%2Fbeam=image
[12] https://github.com/apache/beam/pull/30569
[13]
https://github.com/apache/beam/blob/master/contributor-docs/rc-testing-guide.md

[14] https://github.com/apache/beam/compare/v2.55.0-RC1...v2.55.0-RC2

-- 

Yi Hu, (he/him/his)

Software Engineer

Re: [VOTE] Release 2.55.0, release candidate #1

[Yi Hu via dev]

Dataflow template validation failed due to a dependency issue

Root cause:

1. org.hamcrest classes was leaked in Beam 2.54.0 and below and get removed
in Beam 2.55.0
2. However, it is actually required by io-gcp artifact in runtime static
context, but io-gcp declared org.hamcrest as provided dependency

The change 1 exposed an existing bug 2 and causes failure if io-gcp is in
runtime class path but org.hamcrest does not.

So I am -1 to this, and will prepare RC2



On Tue, Mar 12, 2024 at 3:38 PM Ritesh Ghorse 
wrote:

> +1 (non-binding)
>
> Ran a few batch and streaming examples with Python 3.9.
>
> Thanks!
>
> On Tue, Mar 12, 2024 at 10:21 AM Yi Hu via dev 
> wrote:
>
>> Hi everyone,
>> Please review and vote on the release candidate #1 for the version
>> 2.55.0, as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> Reviewers are encouraged to test their own use cases with the release
>> candidate, and vote +1 if
>> no issues are found. Only PMC member votes will count towards the final
>> vote, but votes from all
>> community members is encouraged and helpful for finding regressions; you
>> can either test your own
>> use cases [13] or use cases from the validation sheet [10].
>>
>> The complete staging area is available for your review, which includes:
>> * GitHub Release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org
>> [2], which is signed with the key with fingerprint D20316F712213422 [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.55.0-RC1" [5],
>> * website pull request listing the release [6], the blog post [6], and
>> publishing the API reference manual [7].
>> * Python artifacts are deployed along with the source release to the
>> dist.apache.org [2] and PyPI [8].
>> * Go artifacts and documentation are available at pkg.go.dev [9]
>> * Validation sheet with a tab for 2.55.0 release to help with validation
>> [10].
>> * Docker images published to Docker Hub [11].
>> * PR to run tests against release branch [12].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> For guidelines on how to try the release in your projects, check out our
>> RC testing guide [13].
>>
>> Thanks,
>> Release Manager
>>
>> [1] https://github.com/apache/beam/milestone/19
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.55.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1371/
>> [5] https://github.com/apache/beam/tree/v2.55.0-RC1
>> [6] https://github.com/apache/beam/pull/30607
>> [7] https://github.com/apache/beam-site/pull/660
>> [8] https://pypi.org/project/apache-beam/2.55.0rc1/
>> [9]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.55.0-RC1/go/pkg/beam
>> [10]
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1368030253
>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>> [12] https://github.com/apache/beam/pull/30569
>> [13]
>> https://github.com/apache/beam/blob/master/contributor-docs/rc-testing-guide.md
>>
>> --
>>
>> Yi Hu, (he/him/his)
>>
>> Software Engineer
>>
>>
>>

[VOTE] Release 2.55.0, release candidate #1

[Yi Hu via dev]

Hi everyone,
Please review and vote on the release candidate #1 for the version 2.55.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


Reviewers are encouraged to test their own use cases with the release
candidate, and vote +1 if
no issues are found. Only PMC member votes will count towards the final
vote, but votes from all
community members is encouraged and helpful for finding regressions; you
can either test your own
use cases [13] or use cases from the validation sheet [10].

The complete staging area is available for your review, which includes:
* GitHub Release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with fingerprint D20316F712213422 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.55.0-RC1" [5],
* website pull request listing the release [6], the blog post [6], and
publishing the API reference manual [7].
* Python artifacts are deployed along with the source release to the
dist.apache.org [2] and PyPI [8].
* Go artifacts and documentation are available at pkg.go.dev [9]
* Validation sheet with a tab for 2.55.0 release to help with validation
[10].
* Docker images published to Docker Hub [11].
* PR to run tests against release branch [12].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

For guidelines on how to try the release in your projects, check out our RC
testing guide [13].

Thanks,
Release Manager

[1] https://github.com/apache/beam/milestone/19
[2] https://dist.apache.org/repos/dist/dev/beam/2.55.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1371/
[5] https://github.com/apache/beam/tree/v2.55.0-RC1
[6] https://github.com/apache/beam/pull/30607
[7] https://github.com/apache/beam-site/pull/660
[8] https://pypi.org/project/apache-beam/2.55.0rc1/
[9]
https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.55.0-RC1/go/pkg/beam
[10]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1368030253
[11] https://hub.docker.com/search?q=apache%2Fbeam=image
[12] https://github.com/apache/beam/pull/30569
[13]
https://github.com/apache/beam/blob/master/contributor-docs/rc-testing-guide.md

-- 

Yi Hu, (he/him/his)

Software Engineer

[Release 2.55.0] Release Branch has been Cut

[Yi Hu via dev]

Hi Beam Developers!

The 2.55.0 release branch has been cut [1].

There are 2 outstanding issues to be triaged for 2.55.0 in the release
milestones [2]

- https://github.com/apache/beam/issues/30459

- https://github.com/apache/beam/issues/30534

Both issues are triaged and waiting for PR and cherry picks. Once cleared,
release candidate are planned to be built.

Thank you very much for your cooperation and support.

Release Manager

[1] https://github.com/apache/beam/tree/release-2.55.0
[2] https://github.com/apache/beam/milestone/19

-- 

Yi Hu, (he/him/his)

Software Engineer

[PROPOSAL] Preparing for 2.55.0 Release

[Yi Hu via dev]

Hey Beam community,

The next release (2.55.0) branch cut is scheduled on Mar 6th, 2024,
according to
the release calendar [1].

I volunteer to perform this release. My plan is to cut the branch on that
date, and cherrypick release-blocking fixes afterwards, if any.

Please help me make sure the release goes smoothly by:
- Making sure that any unresolved release blocking issues for 2.55.0 should
have their "Milestone" marked as "2.55.0 Release" as soon as possible.
- Reviewing the current release blockers [2] and remove the Milestone if
they don't meet the criteria at [3].

Let me know if you have any comments/objections/questions.

Thanks,

Yi

[1]
https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
[2] https://github.com/apache/beam/milestone/19
[3] https://beam.apache.org/contribute/release-blocking/

-- 

Yi Hu, (he/him/his)

Software Engineer

Re: [VOTE] Vendored Dependencies Release

[Yi Hu via dev]

+1 (non-binding)

checked artifact packages not leaking namespace (or under
org.apache.beam.vendor.grpc.v1p60p1) and the tests in
https://github.com/apache/beam/pull/30212




On Tue, Feb 13, 2024 at 4:29 AM Sam Whittle  wrote:

> Hi,
> Sorry I missed that close step. Done!
> Sam
>
> On Mon, Feb 12, 2024 at 8:32 PM Yi Hu via dev  wrote:
>
>> Hi,
>>
>> I am trying to open "
>> https://repository.apache.org/content/repositories/orgapachebeam-1369/;
>> but get "[id=orgapachebeam-1369] exists but is not exposed." It seems the
>> staging repository needs to be closed to have it available to public: [1]
>>
>> [1]
>> https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit?disco=vHX80XE
>>
>> On Mon, Feb 12, 2024 at 1:44 PM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>>> +1 (binding)
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Fri, Feb 9, 2024 at 5:25 AM Sam Whittle  wrote:
>>>
>>>> Please review the release of the following artifacts that we vendor,
>>>> following the process [5]:
>>>>
>>>>  * beam-vendor-grpc-1-60-1:0.2
>>>>
>>>> Hi everyone,
>>>>
>>>> Please review and vote on the release candidate #1 for the version
>>>> beam-vendor-grpc-1-60-1:0.2 as follows:
>>>>
>>>> [ ] +1, Approve the release
>>>>
>>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>>
>>>>
>>>> The complete staging area is available for your review, which includes:
>>>>
>>>> * the official Apache source release to be deployed to dist.apache.org
>>>> [1], which is signed with the key with fingerprint FCFD152811BF1578 [2],
>>>>
>>>> * all artifacts to be deployed to the Maven Central Repository [3],
>>>>
>>>> * commit hash "2d08b32e674a1046ba7be0ae5f1e4b7b05b73488" [4].
>>>>
>>>> The vote will be open for at least 72 hours. It is adopted by majority
>>>> approval, with at least 3 PMC affirmative votes.
>>>>
>>>> Thanks,
>>>>
>>>> Sam
>>>>
>>>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
>>>>
>>>> [2] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>>
>>>> [3]
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1369/
>>>>
>>>> [4]
>>>> https://github.com/apache/beam/commit/2d08b32e674a1046ba7be0ae5f1e4b7b05b73488
>>>>
>>>> [5] https://s.apache.org/beam-release-vendored-artifacts
>>>>
>>>

Re: [ANNOUNCE] New Committer: Svetak Sundhar

[Yi Hu via dev]

Congrats, Svetak!

On Wed, Feb 14, 2024 at 9:50 AM John Casey via dev 
wrote:

> Congrats Svetak!
>
> On Wed, Feb 14, 2024 at 9:00 AM Ahmed Abualsaud 
> wrote:
>
>> Congrats Svetak!
>>
>> On 2024/02/14 02:05:02 Priyans Desai via dev wrote:
>> > Congratulations Svetak!!
>> >
>> > On Tue, Feb 13, 2024 at 8:09 PM Chamikara Jayalath via dev <
>> > dev@beam.apache.org> wrote:
>> >
>> > > Congrats Svetak!
>> > >
>> > > On Tue, Feb 13, 2024 at 4:39 PM Svetak Sundhar via dev <
>> > > dev@beam.apache.org> wrote:
>> > >
>> > >> Thanks everyone!! Looking forward to the continued collaboration :)
>> > >>
>> > >>
>> > >> Svetak Sundhar
>> > >>
>> > >>   Data Engineer
>> > >> s vetaksund...@google.com
>> > >>
>> > >>
>> > >>
>> > >> On Mon, Feb 12, 2024 at 9:58 PM Byron Ellis via dev <
>> dev@beam.apache.org>
>> > >> wrote:
>> > >>
>> > >>> Congrats Svetak!
>> > >>>
>> > >>> On Mon, Feb 12, 2024 at 6:57 PM Shunping Huang via dev <
>> > >>> dev@beam.apache.org> wrote:
>> > >>>
>> >  Congratulations, Svetak!
>> > 
>> >  On Mon, Feb 12, 2024 at 9:50 PM XQ Hu via dev > >
>> >  wrote:
>> > 
>> > > Great job, Svetak! Thanks for all your contributions to Beam!!!
>> > >
>> > > On Mon, Feb 12, 2024 at 4:44 PM Valentyn Tymofieiev via dev <
>> > > dev@beam.apache.org> wrote:
>> > >
>> > >> Congrats, Svetak!
>> > >>
>> > >> On Mon, Feb 12, 2024 at 11:20 AM Kenneth Knowles <
>> k...@apache.org>
>> > >> wrote:
>> > >>
>> > >>> Hi all,
>> > >>>
>> > >>> Please join me and the rest of the Beam PMC in welcoming a new
>> > >>> committer: Svetak Sundhar (sve...@apache.org).
>> > >>>
>> > >>> Svetak has been with Beam since 2021. Svetak has contributed
>> code to
>> > >>> many areas of Beam, including notebooks, Beam Quest,
>> dataframes, and IOs.
>> > >>> We also want to especially highlight the effort Svetak has put
>> into
>> > >>> improving Beam's documentation, participating in release
>> validation, and
>> > >>> evangelizing Beam.
>> > >>>
>> > >>> Considering his contributions to the project over this
>> timeframe,
>> > >>> the Beam PMC trusts Svetak with the responsibilities of a Beam
>> committer.
>> > >>> [1]
>> > >>>
>> > >>> Thank you Svetak! And we are looking to see more of your
>> > >>> contributions!
>> > >>>
>> > >>> Kenn, on behalf of the Apache Beam PMC
>> > >>>
>> > >>> [1]
>> > >>>
>> > >>>
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>> > >>>
>> > >>
>> >
>>
>

Re: [VOTE] Vendored Dependencies Release

[Yi Hu via dev]

Hi,

I am trying to open "
https://repository.apache.org/content/repositories/orgapachebeam-1369/; but
get "[id=orgapachebeam-1369] exists but is not exposed." It seems the
staging repository needs to be closed to have it available to public: [1]

[1]
https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit?disco=vHX80XE

On Mon, Feb 12, 2024 at 1:44 PM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> +1 (binding)
>
> Thanks,
> Cham
>
> On Fri, Feb 9, 2024 at 5:25 AM Sam Whittle  wrote:
>
>> Please review the release of the following artifacts that we vendor,
>> following the process [5]:
>>
>>  * beam-vendor-grpc-1-60-1:0.2
>>
>> Hi everyone,
>>
>> Please review and vote on the release candidate #1 for the version
>> beam-vendor-grpc-1-60-1:0.2 as follows:
>>
>> [ ] +1, Approve the release
>>
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> The complete staging area is available for your review, which includes:
>>
>> * the official Apache source release to be deployed to dist.apache.org
>> [1], which is signed with the key with fingerprint FCFD152811BF1578 [2],
>>
>> * all artifacts to be deployed to the Maven Central Repository [3],
>>
>> * commit hash "2d08b32e674a1046ba7be0ae5f1e4b7b05b73488" [4].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Thanks,
>>
>> Sam
>>
>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
>>
>> [2] https://dist.apache.org/repos/dist/release/beam/KEYS
>>
>> [3]
>> https://repository.apache.org/content/repositories/orgapachebeam-1369/
>>
>> [4]
>> https://github.com/apache/beam/commit/2d08b32e674a1046ba7be0ae5f1e4b7b05b73488
>>
>> [5] https://s.apache.org/beam-release-vendored-artifacts
>>
>

Re: [VOTE] Release 2.54.0, release candidate #2

[Yi Hu via dev]

Also tested with GCP IO performance benchmark [1]. Passed other than
SpannerIO where the benchmark failed due to issues in the test suite itself
[2], not related to Beam.

+1 but I had voted for another validation suite before for this RC

[1]
https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/main/it/google-cloud-platform
[2] https://github.com/GoogleCloudPlatform/DataflowTemplates/issues/1326

On Fri, Feb 9, 2024 at 9:43 AM Valentyn Tymofieiev via dev <
dev@beam.apache.org> wrote:

> +1.
>
> Checked postcommit test results for Python SDK, and exercised a couple of
> Datadow scenarios.
>
> On Thu, Feb 8, 2024, 14:07 Svetak Sundhar via dev 
> wrote:
>
>> +1 (Non-Binding)
>>
>> Tested with Python SDK on DirectRunner and Dataflow Runner
>>
>>
>> Svetak Sundhar
>>
>>   Data Engineer
>> s vetaksund...@google.com
>>
>>
>>
>> On Thu, Feb 8, 2024 at 12:45 PM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>>> +1 (binding)
>>>
>>> Tried out Java/Python multi-lang jobs and upgrading BQ/Kafka transforms
>>> from 2.53.0 to 2.54.0 using the Transform Service.
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Wed, Feb 7, 2024 at 5:52 PM XQ Hu via dev 
>>> wrote:
>>>
>>>> +1 (non-binding)
>>>>
>>>> Validated with a simple RunInference Python pipeline:
>>>> https://github.com/google/dataflow-ml-starter/actions/runs/7821639833/job/21339032997
>>>>
>>>> On Wed, Feb 7, 2024 at 7:10 PM Yi Hu via dev 
>>>> wrote:
>>>>
>>>>> +1 (non-binding)
>>>>>
>>>>> Validated with Dataflow Template:
>>>>> https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/1317
>>>>>
>>>>> Regards,
>>>>>
>>>>> On Wed, Feb 7, 2024 at 11:18 AM Ritesh Ghorse via dev <
>>>>> dev@beam.apache.org> wrote:
>>>>>
>>>>>> +1 (non-binding)
>>>>>>
>>>>>> Ran a few batch and streaming examples for Python SDK on Dataflow
>>>>>> Runner
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> On Wed, Feb 7, 2024 at 4:08 AM Jan Lukavský  wrote:
>>>>>>
>>>>>>> +1 (binding)
>>>>>>>
>>>>>>> Validated Java SDK with Flink runner.
>>>>>>>
>>>>>>>  Jan
>>>>>>> On 2/7/24 06:23, Robert Burke via dev wrote:
>>>>>>>
>>>>>>> Hi everyone,
>>>>>>> Please review and vote on the release candidate #2 for the version
>>>>>>> 2.54.0,
>>>>>>> as follows:
>>>>>>> [ ] +1, Approve the release
>>>>>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>>>>>
>>>>>>>
>>>>>>> Reviewers are encouraged to test their own use cases with the release
>>>>>>> candidate, and vote +1 if
>>>>>>> no issues are found. Only PMC member votes will count towards the
>>>>>>> final
>>>>>>> vote, but votes from all
>>>>>>> community members is encouraged and helpful for finding regressions;
>>>>>>> you
>>>>>>> can either test your own
>>>>>>> use cases [13] or use cases from the validation sheet [10].
>>>>>>>
>>>>>>> The complete staging area is available for your review, which
>>>>>>> includes:
>>>>>>> * GitHub Release notes [1],
>>>>>>> * the official Apache source release to be deployed to
>>>>>>> dist.apache.org [2],
>>>>>>> which is signed with the key with fingerprint D20316F712213422 [3],
>>>>>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>>>>>> * source code tag "v2.54.0-RC2" [5],
>>>>>>> * website pull request listing the release [6], the blog post [6],
>>>>>>> and
>>>>>>> publishing the API reference manual [7].
>>>>>>> * Python artifacts are deployed along with the source release to the
>>>>>>> dist.apache.org [2] and PyPI[8].
>>>>>>> * Go artifacts and documentation are available at pkg.go.dev [9]
>>>>

Re: [VOTE] Release 2.54.0, release candidate #2

[Yi Hu via dev]

+1 (non-binding)

Validated with Dataflow Template:
https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/1317

Regards,

On Wed, Feb 7, 2024 at 11:18 AM Ritesh Ghorse via dev 
wrote:

> +1 (non-binding)
>
> Ran a few batch and streaming examples for Python SDK on Dataflow Runner
>
> Thanks!
>
> On Wed, Feb 7, 2024 at 4:08 AM Jan Lukavský  wrote:
>
>> +1 (binding)
>>
>> Validated Java SDK with Flink runner.
>>
>>  Jan
>> On 2/7/24 06:23, Robert Burke via dev wrote:
>>
>> Hi everyone,
>> Please review and vote on the release candidate #2 for the version 2.54.0,
>> as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> Reviewers are encouraged to test their own use cases with the release
>> candidate, and vote +1 if
>> no issues are found. Only PMC member votes will count towards the final
>> vote, but votes from all
>> community members is encouraged and helpful for finding regressions; you
>> can either test your own
>> use cases [13] or use cases from the validation sheet [10].
>>
>> The complete staging area is available for your review, which includes:
>> * GitHub Release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org
>> [2],
>> which is signed with the key with fingerprint D20316F712213422 [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.54.0-RC2" [5],
>> * website pull request listing the release [6], the blog post [6], and
>> publishing the API reference manual [7].
>> * Python artifacts are deployed along with the source release to the
>> dist.apache.org [2] and PyPI[8].
>> * Go artifacts and documentation are available at pkg.go.dev [9]
>> * Validation sheet with a tab for 2.54.0 release to help with validation
>> [10].
>> * Docker images published to Docker Hub [11].
>> * PR to run tests against release branch [12].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> For guidelines on how to try the release in your projects, check out our
>> RC
>> testing guide [13].
>>
>> Thanks,
>> Robert Burke
>> Beam 2.54.0 Release Manager
>>
>> [1] https://github.com/apache/beam/milestone/18?closed=1
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.54.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1368/
>> [5] https://github.com/apache/beam/tree/v2.54.0-RC2
>> [6] https://github.com/apache/beam/pull/30201
>> [7] https://github.com/apache/beam-site/pull/659
>> [8] https://pypi.org/project/apache-beam/2.54.0rc2/
>> [9]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.54.0-RC2/go/pkg/beam
>> [10]
>>
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=28763708
>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>> [12] https://github.com/apache/beam/pull/30104
>> [13]
>>
>> https://github.com/apache/beam/blob/master/contributor-docs/rc-testing-guide.md
>>
>>

Re: [VOTE] Release 2.54.0, release candidate #1

[Yi Hu via dev]

Thanks for taking care of the release process! After validation two
breaking change was found
(1) Python Xlang Gcp Direct and Python Xlang Gcp Dataflow PostCommit tests
[1, 2]. It affects Python xlang BigQueryIO write (STORAGE_WRITE_API mode)
configuration. Filed [3] and pull request for cherry pick [4].
(2) Validation on Dataflow Template [5] found that OutputReceiver interface
requiring a new method (`outputWindowedValue`) to be implemented. Filed [6]
to determine if this is a release blocker or not, and I am working on a PR
for fix soon.

That said I am -1 to this vote.

[1]
https://github.com/apache/beam/actions/runs/7647377867/job/20838203146?pr=30104
[2]
https://github.com/apache/beam/actions/runs/7647377805/job/20985387650?pr=30104
[3] https://github.com/apache/beam/issues/30159
[4] https://github.com/apache/beam/pull/30189
[5]
https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/7762162776/job/21172087527
[6] https://github.com/apache/beam/issues/30203

On Fri, Feb 2, 2024 at 6:01 PM XQ Hu via dev  wrote:

> +1 validated by running the simple RunInference ML pipeline:
> https://github.com/google/dataflow-ml-starter/actions/runs/7761835540/job/21171080332
>
> On Fri, Feb 2, 2024 at 4:10 PM Robert Burke  wrote:
>
>> Hi everyone,
>> Please review and vote on the release candidate #1 for the version
>> 2.54.0, as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> Reviewers are encouraged to test their own use cases with the release
>> candidate, and vote +1 if
>> no issues are found. Only PMC member votes will count towards the final
>> vote, but votes from all
>> community members is encouraged and helpful for finding regressions; you
>> can either test your own
>> use cases [13] or use cases from the validation sheet [10].
>>
>> The complete staging area is available for your review, which includes:
>> * GitHub Release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org
>> [2], which is signed with the key with fingerprint D20316F712213422 [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.54.0-RC1" [5],
>> * website pull request listing the release [6], the blog post [6], and
>> publishing the API reference manual [7].
>> * Python artifacts are deployed along with the source release to the
>> dist.apache.org [2] and PyPI[8].
>> * Go artifacts and documentation are available at pkg.go.dev [9]
>> * Validation sheet with a tab for 2.54.0 release to help with validation
>> [10].
>> * Docker images published to Docker Hub [11].
>> * PR to run tests against release branch [12].
>>   * Legacy Dataflow Java Worker image has been published, so the failing
>> tests are being re-run.
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> For guidelines on how to try the release in your projects, check out our
>> RC testing guide [13].
>>
>> Thanks,
>> Robert Burke
>> Beam 2.54.0 Release Manager
>>
>> [1] https://github.com/apache/beam/milestone/18?closed=1
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.54.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1367/
>> [5] https://github.com/apache/beam/tree/v2.54.0-RC1
>> [6] https://github.com/apache/beam/pull/30201
>> [7] https://github.com/apache/beam-site/pull/658
>> [8] https://pypi.org/project/apache-beam/2.54.0rc1/
>> [9]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.54.0-RC1/go/pkg/beam
>> [10]
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=28763708
>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>> [12] https://github.com/apache/beam/pull/30104
>> [13]
>> https://github.com/apache/beam/blob/master/contributor-docs/rc-testing-guide.md
>>
>

[Result][VOTE] Vendored Dependencies Release beam-vendor-grpc-1_60_1

[Yi Hu via dev]

Hi everyone,

I'm happy to announce that we have unanimously approved this release.

There are three approving votes, three of which are binding:
* Kenneth Knowles
* Robert Bradshaw
* Chamikara Jayalath

There are no disapproving votes. I will begin finalizing the release.

Thanks everyone!

-- 

Yi Hu, (he/him/his)

Software Engineer

Re: [VOTE] Vendored Dependencies Release

[Yi Hu via dev]

> Notably, the vendored artifact has no impact on the repo until the
version used is also bumped.

That is correct. The PR that actually bump the version then the change
takes effect will be like https://github.com/apache/beam/pull/29976

On Mon, Jan 22, 2024 at 10:11 AM Kenneth Knowles  wrote:

> Notably, the vendored artifact has no impact on the repo until the version
> used is also bumped, right? So the release is very low stakes.
>
> Kenn
>
> On Fri, Jan 19, 2024 at 4:55 PM Robert Bradshaw via dev <
> dev@beam.apache.org> wrote:
>
>> Thanks.
>>
>> +1
>>
>>
>> On Fri, Jan 19, 2024 at 1:24 PM Yi Hu  wrote:
>>
>>> The process I have been following is [1]. I have also suggested edits to
>>> the voting email template to include the self-link. However, does anyone
>>> can edit this doc so the change can be made? Otherwise we might better to
>>> migrate this doc to
>>> https://github.com/apache/beam/tree/master/contributor-docs
>>>
>>> [1] https://s.apache.org/beam-release-vendored-artifacts
>>>
>>> On Thu, Jan 18, 2024 at 2:56 PM Robert Bradshaw via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> Could you explain the process you used to produce these artifacts?
>>>>
>>>> On Thu, Jan 18, 2024 at 11:23 AM Kenneth Knowles 
>>>> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On Wed, Jan 17, 2024 at 6:03 PM Yi Hu via dev 
>>>>> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>>
>>>>>> Please review the release of the following artifacts that we vendor:
>>>>>>
>>>>>>  * beam-vendor-grpc-1_60_1
>>>>>>
>>>>>>
>>>>>> Please review and vote on the release candidate #1 for the version
>>>>>> 0.1, as follows:
>>>>>>
>>>>>> [ ] +1, Approve the release
>>>>>>
>>>>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>>>>
>>>>>>
>>>>>> The complete staging area is available for your review, which
>>>>>> includes:
>>>>>>
>>>>>> * the official Apache source release to be deployed to
>>>>>> dist.apache.org [1], which is signed with the key with fingerprint
>>>>>> 8935B943A188DE65 [2],
>>>>>>
>>>>>> * all artifacts to be deployed to the Maven Central Repository [3],
>>>>>>
>>>>>> * commit hash "52b4a9cb58e486745ded7d53a5b6e2d2312e9551" [4],
>>>>>>
>>>>>> The vote will be open for at least 72 hours. It is adopted by
>>>>>> majority approval, with at least 3 PMC affirmative votes.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Release Manager
>>>>>>
>>>>>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
>>>>>>
>>>>>> [2] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>>>>
>>>>>> [3]
>>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1366/
>>>>>>
>>>>>> [4]
>>>>>> https://github.com/apache/beam/commits/52b4a9cb58e486745ded7d53a5b6e2d2312e9551/
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Yi Hu, (he/him/his)
>>>>>>
>>>>>> Software Engineer
>>>>>>
>>>>>>
>>>>>>

Re: [VOTE] Vendored Dependencies Release

[Yi Hu via dev]

The process I have been following is [1]. I have also suggested edits to
the voting email template to include the self-link. However, does anyone
can edit this doc so the change can be made? Otherwise we might better to
migrate this doc to
https://github.com/apache/beam/tree/master/contributor-docs

[1] https://s.apache.org/beam-release-vendored-artifacts

On Thu, Jan 18, 2024 at 2:56 PM Robert Bradshaw via dev 
wrote:

> Could you explain the process you used to produce these artifacts?
>
> On Thu, Jan 18, 2024 at 11:23 AM Kenneth Knowles  wrote:
>
>> +1
>>
>> On Wed, Jan 17, 2024 at 6:03 PM Yi Hu via dev 
>> wrote:
>>
>>> Hi everyone,
>>>
>>>
>>> Please review the release of the following artifacts that we vendor:
>>>
>>>  * beam-vendor-grpc-1_60_1
>>>
>>>
>>> Please review and vote on the release candidate #1 for the version 0.1,
>>> as follows:
>>>
>>> [ ] +1, Approve the release
>>>
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>>
>>> The complete staging area is available for your review, which includes:
>>>
>>> * the official Apache source release to be deployed to dist.apache.org
>>> [1], which is signed with the key with fingerprint 8935B943A188DE65 [2],
>>>
>>> * all artifacts to be deployed to the Maven Central Repository [3],
>>>
>>> * commit hash "52b4a9cb58e486745ded7d53a5b6e2d2312e9551" [4],
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> Thanks,
>>>
>>> Release Manager
>>>
>>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
>>>
>>> [2] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>
>>> [3]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1366/
>>>
>>> [4]
>>> https://github.com/apache/beam/commits/52b4a9cb58e486745ded7d53a5b6e2d2312e9551/
>>>
>>>
>>> --
>>>
>>> Yi Hu, (he/him/his)
>>>
>>> Software Engineer
>>>
>>>
>>>

[VOTE] Vendored Dependencies Release

[Yi Hu via dev]

Hi everyone,


Please review the release of the following artifacts that we vendor:

 * beam-vendor-grpc-1_60_1


Please review and vote on the release candidate #1 for the version 0.1, as
follows:

[ ] +1, Approve the release

[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:

* the official Apache source release to be deployed to dist.apache.org [1],
which is signed with the key with fingerprint 8935B943A188DE65 [2],

* all artifacts to be deployed to the Maven Central Repository [3],

* commit hash "52b4a9cb58e486745ded7d53a5b6e2d2312e9551" [4],

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,

Release Manager

[1] https://dist.apache.org/repos/dist/dev/beam/vendor/

[2] https://dist.apache.org/repos/dist/release/beam/KEYS

[3] https://repository.apache.org/content/repositories/orgapachebeam-1366/

[4]
https://github.com/apache/beam/commits/52b4a9cb58e486745ded7d53a5b6e2d2312e9551/


-- 

Yi Hu, (he/him/his)

Software Engineer

Re: [PROPOSAL] Upgrade vendor grpc

[Yi Hu via dev]

Thanks for the support. Could a PMC member help copying the GPG pub key
from https://dist.apache.org/repos/dist/dev/beam/KEYS (currently the last
one, after the second-last "END PGP PUBLIC KEY BLOCK") to
https://dist.apache.org/repos/dist/release/beam/KEYS ?

On Fri, Jan 12, 2024 at 11:28 AM Kenneth Knowles  wrote:

> Yes, thank you!
>
> On Thu, Jan 11, 2024 at 8:21 PM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
>
>> Sounds good and thanks for doing this :)
>>
>> - Cham
>>
>> On Thu, Jan 11, 2024 at 8:06 AM Yi Hu via dev 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I would like to volunteer to upgrade the Beam vendored grpc, as
>>> requested by the GitHub Issue [1]. The last update was in Apr 2023 [2].
>>> There have been vulnerabilities in its dependencies as well as potential
>>> oom issues found since then (see [1]), and also to include grpc-alts [2].
>>>
>>> My plan is to follow the release process [3, 4], which involves
>>> preparing for the release, building a candidate, voting and finalizing the
>>> release. Then the vendored artifact is targeted to be integrated by Beam
>>> v2.54.0 onwards (cut date Jan 24, 2024).
>>>
>>> Please let me know if you have any comments/objections/questions.
>>>
>>> Thanks,
>>>
>>> Yi
>>>
>>> [1] https://github.com/apache/beam/issues/29861
>>> [2] https://github.com/apache/beam/issues/25746
>>> [3] https://github.com/apache/beam/tree/master/vendor
>>> [4]
>>> https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit#heading=h.vhcuqlttpnog
>>> --
>>>
>>> Yi Hu, (he/him/his)
>>>
>>> Software Engineer
>>>
>>>
>>>

[PROPOSAL] Upgrade vendor grpc

[Yi Hu via dev]

Hi everyone,

I would like to volunteer to upgrade the Beam vendored grpc, as requested
by the GitHub Issue [1]. The last update was in Apr 2023 [2]. There have
been vulnerabilities in its dependencies as well as potential oom issues
found since then (see [1]), and also to include grpc-alts [2].

My plan is to follow the release process [3, 4], which involves preparing
for the release, building a candidate, voting and finalizing the release.
Then the vendored artifact is targeted to be integrated by Beam v2.54.0
onwards (cut date Jan 24, 2024).

Please let me know if you have any comments/objections/questions.

Thanks,

Yi

[1] https://github.com/apache/beam/issues/29861
[2] https://github.com/apache/beam/issues/25746
[3] https://github.com/apache/beam/tree/master/vendor
[4]
https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit#heading=h.vhcuqlttpnog
-- 

Yi Hu, (he/him/his)

Software Engineer

Re: [VOTE] Release 2.53.0, release candidate #2

[Yi Hu via dev]

+1 (non-binding)

Tested with Beam GCP IOs benchmarking (
https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/main/it/google-cloud-platform
)

On Thu, Dec 28, 2023 at 11:36 AM Svetak Sundhar via dev 
wrote:

> +1 (non binding)
>
> Tested with Healthcare notebooks.
>
>
> Svetak Sundhar
>
>   Data Engineer
> s vetaksund...@google.com
>
>
>
> On Thu, Dec 28, 2023 at 3:52 AM Jan Lukavský  wrote:
>
>> +1 (binding)
>>
>> Tested Java SDK with Flink Runner.
>>
>>  Jan
>> On 12/27/23 14:13, Danny McCormick via dev wrote:
>>
>> +1 (non-binding)
>>
>> Tested with some example ML notebooks.
>>
>> Thanks,
>> Danny
>>
>> On Tue, Dec 26, 2023 at 6:41 PM XQ Hu via dev 
>> wrote:
>>
>>> +1 (non-binding)
>>>
>>> Tested with the simple RunInference pipeline:
>>> https://github.com/google/dataflow-ml-starter/actions/runs/7332832875/job/19967521369
>>>
>>> On Tue, Dec 26, 2023 at 3:29 PM Jack McCluskey via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Happy holidays everyone,

 Please review and vote on the release candidate #2 for the version
 2.53.0, as follows:

 [ ] +1, Approve the release
 [ ] -1, Do not approve the release (please provide specific comments)

 Reviewers are encouraged to test their own use cases with the release
 candidate, and vote +1 if no issues are found. Only PMC member votes will
 count towards the final vote, but votes from all community members are
 encouraged and helpful for finding regressions; you can either test your
 own use cases [13] or use cases from the validation sheet [10].

 The complete staging area is available for your review, which includes:
 * GitHub Release notes [1],
 * the official Apache source release to be deployed to dist.apache.org [2],
 which is signed with the key with fingerprint DF3CBA4F3F4199F4
 (D20316F712213422 if automated) [3],
 * all artifacts to be deployed to the Maven Central Repository [4],
 * source code tag "v1.2.3-RC3" [5],
 * website pull request listing the release [6], the blog post [6], and
 publishing the API reference manual [7].
 * Python artifacts are deployed along with the source release to the
 dist.apache.org [2] and PyPI[8].
 * Go artifacts and documentation are available at pkg.go.dev [9]
 * Validation sheet with a tab for 2.53.0 release to help with
 validation [10].
 * Docker images published to Docker Hub [11].
 * PR to run tests against release branch [12].

 The vote will be open for at least 72 hours. It is adopted by majority
 approval, with at least 3 PMC affirmative votes.

 For guidelines on how to try the release in your projects, check out
 our RC testing guide [13].

 Thanks,

 Jack McCluskey

 [1] https://github.com/apache/beam/milestone/17
 [2] https://dist.apache.org/repos/dist/dev/beam/2.53.0/
 [3] https://dist.apache.org/repos/dist/release/beam/KEYS
 [4]
 https://repository.apache.org/content/repositories/orgapachebeam-1365/
 [5] https://github.com/apache/beam/tree/v2.53.0-RC2
 [6] https://github.com/apache/beam/pull/29856
 [7] https://github.com/apache/beam-site/pull/657
 [8] https://pypi.org/project/apache-beam/2.53.0rc2/
 [9]
 https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.53.0-RC2/go/pkg/beam
 [10]
 https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1290249774
 [11] https://hub.docker.com/search?q=apache%2Fbeam=image
 [12] https://github.com/apache/beam/pull/29758
 [13]
 https://github.com/apache/beam/blob/master/contributor-docs/rc-testing-guide.md


 --


 Jack McCluskey
 SWE - DataPLS PLAT/ Dataflow ML
 RDU
 jrmcclus...@google.com

Re: Disabling Jenkins Jobs

[Yi Hu via dev]

Hi all,

Just another update that we have shutdown all precommit tests on Jenkins.

For now, the remaining tests running on Jenkins are PostCommit suites. They
are also exercised on GitHub Action, but cannot be triggered from a pull
request due to [1]. That said, once the blocker [1] is resolved, we can
fully shutdown the Jenkins server. For now, I plan to stop half of the
Jenkins worker nodes since developers will no longer see Jenkins jobs under
their pull requests unless triggered manually.

[1] https://github.com/apache/beam/issues/28909

Regards,
Yi

Re: [VOTE] Release 2.52.0, release candidate #2

[Yi Hu via dev]

+1 (non-binding) Tested on Java IO load tests (
https://github.com/bvolpato/DataflowTemplates/tree/56d18a31c1c95e58543d7a1656bd83d7e859b482/it)
BigQueryIO, TextIO, BigtableIO, SpannerIO on Dataflow legacy runner and
runner v2

While it was announced there will be an RC3, the RC2 validation for IO
benchmark was still ongoing. I decided to continue with validation as
suggested. Will also run a few pipelines with RC3.

Regards,
Yi

On Wed, Nov 8, 2023 at 10:25 AM Danny McCormick via dev 
wrote:

> Hey everyone, @Ritesh Ghorse  pointed out to me
> that the docker containers were not pushed for RC2, just for RC1. On closer
> inspection, I've realized that I accidentally built the RC from the RC1 tag
> (https://github.com/apache/beam/tree/v2.52.0-RC1) instead of the RC2 tag (
> https://github.com/apache/beam/tree/v2.52.0-RC2), so it is also missing
> an important cherry pick fix to the Datastore IO (
> https://github.com/apache/beam/commit/0fdf404873636d24be50ae8360a08e4dddfae679
> ).
>
> I'm going to move to RC3 and should have that out later today. You're
> still welcome to do more validation on RC2, especially if you're not using
> the Datastore IO. Sorry for the mixup!
>
> Thanks,
> Danny
>
> On Wed, Nov 8, 2023 at 9:27 AM Svetak Sundhar via dev 
> wrote:
>
>> Thanks, Danny!
>>
>> @all: Reminder that if there's anything you think that is worth
>> documenting while RC testing, please feel free to add it here
>> 
>> .
>>
>> We can then use it to update
>> https://github.com/apache/beam/blob/master/contributor-docs/release-guide.md#vote-and-validate-the-release-candidate
>> .
>>
>> Thanks,
>>
>>
>> Svetak Sundhar
>>
>>   Data Engineer
>> s vetaksund...@google.com
>>
>>
>>
>> On Wed, Nov 8, 2023 at 9:04 AM Jean-Baptiste Onofré 
>> wrote:
>>
>>> +1 (binding)
>>>
>>> Regards
>>> JB
>>>
>>> On Wed, Nov 8, 2023 at 12:24 AM Danny McCormick via dev
>>>  wrote:
>>> >
>>> > Hi everyone,
>>> > Please review and vote on the release candidate #2 for the version
>>> 2.52.0, as follows:
>>> > [ ] +1, Approve the release
>>> > [ ] -1, Do not approve the release (please provide specific comments)
>>> >
>>> >
>>> > Reviewers are encouraged to test their own use cases with the release
>>> candidate, and vote +1 if no issues are found. Only PMC member votes will
>>> count towards the final vote, but votes from all community members is
>>> encouraged and helpful for finding regressions; you can either test your
>>> own use cases or use cases from the validation sheet [10].
>>> >
>>> > The complete staging area is available for your review, which includes:
>>> >
>>> > GitHub Release notes [1]
>>> > the official Apache source release to be deployed to dist.apache.org
>>> [2], which is signed with the key with fingerprint D20316F712213422 [3]
>>> > all artifacts to be deployed to the Maven Central Repository [4]
>>> > source code tag "v2.52.0-RC1" [5]
>>> > website pull request listing the release [6], the blog post [6], and
>>> publishing the API reference manual [7]
>>> > Python artifacts are deployed along with the source release to the
>>> dist.apache.org [2] and PyPI[8].
>>> > Go artifacts and documentation are available at pkg.go.dev [9]
>>> > Validation sheet with a tab for 2.52.0 release to help with validation
>>> [10]
>>> > Docker images published to Docker Hub [11]
>>> > PR to run tests against release branch [12]
>>> >
>>> >
>>> > The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>> >
>>> > For guidelines on how to try the release in your projects, check out
>>> our blog post at https://beam.apache.org/blog/validate-beam-release/.
>>> >
>>> > Thanks,
>>> > Danny
>>> >
>>> > [1] https://github.com/apache/beam/milestone/16
>>> > [2] https://dist.apache.org/repos/dist/dev/beam/2.52.0/
>>> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> > [4]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1360/
>>> > [5] https://github.com/apache/beam/tree/v2.52.0-RC2
>>> > [6] https://github.com/apache/beam/pull/29331
>>> > [7] https://github.com/apache/beam-site/pull/652
>>> > [8] https://pypi.org/project/apache-beam/2.52.0rc2/
>>> > [9]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.52.0-RC2/go/pkg/beam
>>> > [10]
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1387982510
>>> > [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>>> > [12] https://github.com/apache/beam/pull/29319
>>>
>>

Re: Disabling Jenkins Jobs

[Yi Hu via dev]

Hi Alexey,

> all Jenkins jobs are stuck and there is a big Build Queue on
https://ci-beam.apache.org/

This is not intentional. This is likely due to INFRA's routine Jenkins
upgrade on Nov 5 and caused this outage. Have created
https://issues.apache.org/jira/projects/INFRA/issues/INFRA-25147?filter=allopenissues


Regards,
Yi

On Tue, Nov 7, 2023 at 10:35 AM Danny McCormick via dev 
wrote:

> I don't think it's related. I noticed the problem half an hour ago; it
> seems there's an expired cert on the Jenkins machines. I'm hoping
> https://github.com/apache/beam/actions/runs/6786537134/job/18447281366
> will fix this since the IO-Datastores cert is the problematic piece I think
> (and that has fixed a similar problem before). I'm not totally confident it
> will though given that the job succeeded last week.
>
> On Tue, Nov 7, 2023 at 10:18 AM Alexey Romanenko 
> wrote:
>
>> Not sure if it’s related but I see that, seems, all Jenkins jobs are
>> stuck and there is a big Build Queue on https://ci-beam.apache.org/
>>
>> Random clicks on jobs show that “"All nodes of label ‘beam’ are offline”
>> message.
>>
>> Is it known problem?
>>
>> —
>> Alexey
>>
>> On 24 Oct 2023, at 21:50, Yi Hu via dev  wrote:
>>
>> Hi all,
>>
>> We have shut down most tests in the Jenkins Load Tests and Performance
>> Tests categories [1, 2], as they have been migrated to GitHub Actions for a
>> while and are continuously publishing the metrics as expected. Please refer
>> to https://github.com/apache/beam/pull/29092 for these tests. Note that
>> pull requests mostly do not involve these tests so that it does not affect
>> the development and release process.
>>
>> For the queueing issue mentioned before, after the self-hosted runners
>> switched back from github webhook scaling to load based scaling, it is back
>> to being stable. The issue was likely due to the webhook scaling on GitHub
>> side.
>>
>> Regards,
>> Yi
>>
>> [1] https://ci-beam.apache.org/view/LoadTests/
>> [2] https://ci-beam.apache.org/view/PerformanceTests/
>>
>>
>>
>>
>>

Re: Disabling Jenkins Jobs

[Yi Hu via dev]

Hi all,

We have shut down most tests in the Jenkins Load Tests and Performance
Tests categories [1, 2], as they have been migrated to GitHub Actions for a
while and are continuously publishing the metrics as expected. Please refer
to https://github.com/apache/beam/pull/29092 for these tests. Note that
pull requests mostly do not involve these tests so that it does not affect
the development and release process.

For the queueing issue mentioned before, after the self-hosted runners
switched back from github webhook scaling to load based scaling, it is back
to being stable. The issue was likely due to the webhook scaling on GitHub
side.

Regards,
Yi

[1] https://ci-beam.apache.org/view/LoadTests/
[2] https://ci-beam.apache.org/view/PerformanceTests/

Re: [VOTE] Release 2.51.0, release candidate #1

[Yi Hu via dev]

> Would it impact all python users including breaking the new user, quick
> start experience? Or would it impact users of a specific IO or
> configuration?
>

It is the latter. It will impact users of Specific IO (BigQueryIO read)
specific configuration (Direct_Read). Note that the default configuration
for BigQueryIO read is EXPORT. So this won't affect "quick-start" examples
having default settings.

It also won't affect users using SDK docker containers (e.g. Dataflow users
and Flink/Spark users running on a remote cluster). It will affect users
running in direct runner, and local portable runners (e.g. Flink local
cluster) with LOOPBACK configuration, which is exactly what our Python
PostComit is doing.

Re: [PROPOSAL] [Nice-to-have] CI job names and commands that match

[Yi Hu via dev]

Thanks for raising this. This generally works, though some jobs run more
than one gradle task (e.g. some IO_Direct_PreCommit run both :build (which
executes unit tests) and :integrationTest).

Another option is to normalize the naming of every job, saying the job name
is X, then workflow name is PreCommit_X or PostCommit_X, and the phrase is
Run X. Currently most PreCommit follow this pattern, but there are also
many outliers. A good start could be to clean up all jobs to follow the
same pattern.


On Tue, Oct 10, 2023 at 9:57 AM Kenneth Knowles  wrote:

> FWIW I aware of the README in
> https://github.com/apache/beam/tree/master/.test-infra/jenkins that lists
> the phrases alongside the jobs. This is just wasted work to maintain IMO.
>
> Kenn
>
> On Tue, Oct 10, 2023 at 9:46 AM Kenneth Knowles  wrote:
>
>> *Proposal:* make all the job names exactly match the GH comment to run
>> them and make it also as close as possible to how to reproduce locally
>>
>> *Example problems*:
>>
>>  - We have really silly redundant jobs results like 'Chicago Taxi Example
>> on Dataflow ("Run Chicago Taxi on Dataflow")' and 'Python_Xlang_IO_Dataflow
>> ("Run Python_Xlang_IO_Dataflow PostCommit")'
>>
>>  - We have jobs that there's no way you could guess the command 'Google
>> Cloud Dataflow Runner V2 Java ValidatesRunner Tests (streaming)'
>>
>>  - (nit) We are weirdly inconsistent about using spaces vs underscores. I
>> don't think any of our infrastructure cares about this.
>>
>> *Extra proposal*: make the job name also the local command, where
>> possible
>>
>> *Example: *
>> https://github.com/apache/beam/blob/master/.github/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow.yml
>>
>>  - This runs :runners:google-cloud-dataflow-java:validatesRunner
>>  - So make the status label
>> ":runners:google-cloud-dataflow-java:validatesRunner"
>>  - "Run :runners:google-cloud-dataflow-java:validatesRunner" as comment
>>
>> If I want to run it locally, yes there are GCP things I have to set up,
>> but I know the gradle command now.
>>
>> *Corollary*: remove "postcommit" and "precommit" from names, because
>> whether a suite runs before merge or after merge is not a property of the
>> suite.
>>
>> *Caveats*: I haven't been that involved. I didn't do this to Jenkins
>> because they are going away. I didn't do anything to GHA because I don't
>> know if they are ready or in flux.
>>
>> I know this is the sort of thing that invites bikeshedding. It just would
>> save me a few minutes when puzzling out what to care about and how to kick
>> jobs on the release branch validation PR.
>>
>> I'm happy to scrape through the existing stuff and align it. Perfect task
>> for when my brain is too tired for other work.
>>
>> Kenn
>>
>

Re: [VOTE] Release 2.51.0, release candidate #1

[Yi Hu via dev]

Yes, and moreover, this specific issue will break the user the same way for
*all* Beam versions (2.50.0, 2.49.0, etc) after Oct 3. That said the issue
is not limited to Beam 2.50.0 though.

On Mon, Oct 9, 2023 at 4:08 PM Kenneth Knowles  wrote:

> If we had closed the release today, this would still have broken all our
> users, correct?
>
> Kenn
>
> On Mon, Oct 9, 2023 at 3:37 PM Anand Inguva via dev 
> wrote:
>
>> There was a regression[1] on fastavro latest release 1.8.4. Fix was
>> merged at https://github.com/apache/beam/pull/28896. The RC1 includes
>> that version in the range for fastavro[2]. I think we need to CP
>> https://github.com/apache/beam/pull/28896 to solve the fastavro
>> regression.
>>
>> [1] https://github.com/apache/beam/issues/28811
>> [2]
>> https://github.com/apache/beam/blob/cd653e33b342bd09c76c2bbaca12597fec5b4a2c/sdks/python/setup.py#L245
>>
>>
>> On Mon, Oct 9, 2023 at 3:15 PM Kenneth Knowles  wrote:
>>
>>> Ran a couple of Java pipelines "as a newb user" to make sure our
>>> instructions weren't out of date. There are some errors in the instructions
>>> but they don't have to do with this release.
>>>
>>> Re-ran mass_comment.py on https://github.com/apache/beam/pull/28663.
>>> There are enough red signals there that some triage is needed. Any help
>>> triaging would be appreciated.
>>>
>>> I'll close the vote once everything is run and examined.
>>>
>>> Kenn
>>>
>>> On Sat, Oct 7, 2023 at 9:58 AM Yi Hu via dev 
>>> wrote:
>>>
>>>> +1 (non-binding) Tested on Java IO load tests (
>>>> https://github.com/bvolpato/DataflowTemplates/tree/56d18a31c1c95e58543d7a1656bd83d7e859b482/it)
>>>> BigQueryIO, TextIO, BigtableIO, SpannerIO on Dataflow legacy runner and
>>>> runner v2
>>>>
>>>>
>>>> On Fri, Oct 6, 2023 at 3:23 PM Kenneth Knowles  wrote:
>>>>
>>>>> Additionally we need https://github.com/apache/beam/pull/28665/files
>>>>> in order to run GHA tests.
>>>>>
>>>>> On Fri, Oct 6, 2023 at 3:19 PM Kenneth Knowles 
>>>>> wrote:
>>>>>
>>>>>> That PR was prior to many cherry-picks so it is not the signal we
>>>>>> need. I have updated it to the tip of the release-2.51.0 branch.
>>>>>>
>>>>>> There were some post-commit tests involving JPMS that I believe need
>>>>>> https://github.com/apache/beam/pull/28726 to pass.
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Fri, Oct 6, 2023 at 2:53 PM Valentyn Tymofieiev via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>>> > PR to run tests against release branch [12].
>>>>>>>
>>>>>>>  https://github.com/apache/beam/pull/28663 is closed and test
>>>>>>> signal is no longer available. did all the tests pass?
>>>>>>>
>>>>>>> On Fri, Oct 6, 2023 at 5:32 AM Alexey Romanenko <
>>>>>>> aromanenko@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1 (binding)
>>>>>>>>
>>>>>>>> —
>>>>>>>> Alexey
>>>>>>>>
>>>>>>>> > On 5 Oct 2023, at 18:38, Jean-Baptiste Onofré 
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> > +1 (binding)
>>>>>>>> >
>>>>>>>> > Thanks !
>>>>>>>> > Regards
>>>>>>>> > JB
>>>>>>>> >
>>>>>>>> > On Tue, Oct 3, 2023 at 7:58 PM Kenneth Knowles 
>>>>>>>> wrote:
>>>>>>>> >>
>>>>>>>> >> Hi everyone,
>>>>>>>> >>
>>>>>>>> >> Please review and vote on the release candidate #1 for the
>>>>>>>> version 2.51.0, as follows:
>>>>>>>> >>
>>>>>>>> >> [ ] +1, Approve the release
>>>>>>>> >> [ ] -1, Do not approve the release (please provide specific
>>>>>>>> comments)
>>>>>>>> >>
>>>>>>>> >> Reviewers are encouraged to test their own use cases with the
>>>>>&g

Re: [VOTE] Release 2.51.0, release candidate #1

[Yi Hu via dev]

+1 (non-binding) Tested on Java IO load tests (
https://github.com/bvolpato/DataflowTemplates/tree/56d18a31c1c95e58543d7a1656bd83d7e859b482/it)
BigQueryIO, TextIO, BigtableIO, SpannerIO on Dataflow legacy runner and
runner v2


On Fri, Oct 6, 2023 at 3:23 PM Kenneth Knowles  wrote:

> Additionally we need https://github.com/apache/beam/pull/28665/files in
> order to run GHA tests.
>
> On Fri, Oct 6, 2023 at 3:19 PM Kenneth Knowles  wrote:
>
>> That PR was prior to many cherry-picks so it is not the signal we need. I
>> have updated it to the tip of the release-2.51.0 branch.
>>
>> There were some post-commit tests involving JPMS that I believe need
>> https://github.com/apache/beam/pull/28726 to pass.
>>
>> Kenn
>>
>> On Fri, Oct 6, 2023 at 2:53 PM Valentyn Tymofieiev via dev <
>> dev@beam.apache.org> wrote:
>>
>>> > PR to run tests against release branch [12].
>>>
>>>  https://github.com/apache/beam/pull/28663 is closed and test signal is
>>> no longer available. did all the tests pass?
>>>
>>> On Fri, Oct 6, 2023 at 5:32 AM Alexey Romanenko <
>>> aromanenko@gmail.com> wrote:
>>>
 +1 (binding)

 —
 Alexey

 > On 5 Oct 2023, at 18:38, Jean-Baptiste Onofré 
 wrote:
 >
 > +1 (binding)
 >
 > Thanks !
 > Regards
 > JB
 >
 > On Tue, Oct 3, 2023 at 7:58 PM Kenneth Knowles 
 wrote:
 >>
 >> Hi everyone,
 >>
 >> Please review and vote on the release candidate #1 for the version
 2.51.0, as follows:
 >>
 >> [ ] +1, Approve the release
 >> [ ] -1, Do not approve the release (please provide specific comments)
 >>
 >> Reviewers are encouraged to test their own use cases with the
 release candidate, and vote +1 if no issues are found. Only PMC member
 votes will count towards the final vote, but votes from all community
 members is encouraged and helpful for finding regressions; you can either
 test your own use cases or use cases from the validation sheet [10].
 >>
 >> The complete staging area is available for your review, which
 includes:
 >>
 >> GitHub Release notes [1],
 >> the official Apache source release to be deployed to dist.apache.org
 [2], which is signed with the key with fingerprint  [3],
 >> all artifacts to be deployed to the Maven Central Repository [4],
 >> source code tag "v1.2.3-RC3" [5],
 >> website pull request listing the release [6], the blog post [6], and
 publishing the API reference manual [7].
 >> Java artifacts were built with Gradle GRADLE_VERSION and
 OpenJDK/Oracle JDK JDK_VERSION.
 >> Python artifacts are deployed along with the source release to the
 dist.apache.org [2] and PyPI[8].
 >> Go artifacts and documentation are available at pkg.go.dev [9]
 >> Validation sheet with a tab for 1.2.3 release to help with
 validation [10].
 >> Docker images published to Docker Hub [11].
 >> PR to run tests against release branch [12].
 >>
 >> The vote will be open for at least 72 hours. It is adopted by
 majority approval, with at least 3 PMC affirmative votes.
 >>
 >> For guidelines on how to try the release in your projects, check out
 our blog post at https://beam.apache.org/blog/validate-beam-release/.
 >>
 >> Thanks,
 >> Kenn
 >>
 >> [1] https://github.com/apache/beam/milestone/15
 >> [2] https://dist.apache.org/repos/dist/dev/beam/2.51.0
 >> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
 >> [4]
 https://repository.apache.org/content/repositories/orgapachebeam-1356/
 >> [5] https://github.com/apache/beam/tree/v2.51.0-RC1
 >> [6] https://github.com/apache/beam/pull/28800
 >> [7] https://github.com/apache/beam-site/pull/649
 >> [8] https://pypi.org/project/apache-beam/2.51.0rc1/
 >> [9]
 https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.51.0-RC1/go/pkg/beam
 >> [10]
 https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=437054928
 >> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
 >> [12] https://github.com/apache/beam/pull/28663

Re: [Proposal] Enable EnricoMi/publish-unit-test-result-action

[Yi Hu via dev]

Thanks for the feedback! A request has been sent to Apache Infra.

I checked that GitHub Action workflows actually published the gradle scan
(done by [1]), but some workflow added after [1] simply missed that
setting. Opened [2] for clean up and improvements.

Best,
Yi

[1] https://github.com/apache/beam/pull/28212
[2] https://github.com/apache/beam/issues/28378

On Tue, Sep 5, 2023 at 12:26 PM Kenneth Knowles  wrote:

> +1 this seems useful.
>
> Some of the same functionality is also done pretty well or even more in
> depth via gradle scan. If I recall, some GHA jobs do not upload those. Is
> that also on the roadmap or is it blocked for some reason?
>
> Kenn
>
> On Tue, Sep 5, 2023 at 11:54 AM Bruno Volpato via dev 
> wrote:
>
>> +1, this is helpful.
>>
>> We had a similar situation with DataflowTemplates
>> <https://github.com/GoogleCloudPlatform/DataflowTemplates>.
>> Even though we used a different repository (mikepenz/action-junit-report
>> <https://github.com/mikepenz/action-junit-report>), this strategy was
>> invaluable to reduce troubleshoot time (sample report
>> <https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/1025/checks?check_run_id=16466087985>
>> ).
>>
>> Thanks Yi!
>>
>>
>>
>> On Tue, Sep 5, 2023 at 11:26 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Thanks Yi, I'm definitely +1 on adding this, this is definitely a gap in
>>> our GitHub Actions infra today.
>>>
>>> Thanks,
>>> Danny
>>>
>>> On Tue, Sep 5, 2023 at 10:35 AM Yi Hu via dev 
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> As you may have noticed, GitHub Action for test suites are setting up
>>>> in the Beam repository. A current gap is that Jenkins has a pretty
>>>> convenient test result page showing all tests / failed tests and stack
>>>> trace, while these are not available in github workflow logs.
>>>>
>>>> Here we propose to introduce EnricoMi/publish-unit-test-result-action (
>>>> https://github.com/EnricoMi/publish-unit-test-result-action) to
>>>> publish Java (and possibly Python in the future) test results. An example
>>>> PR can be found [1] and an INFRA ticket in [2]. Currently both Java and
>>>> Python test reports are supported by this action (note that Jenkins test
>>>> report page is also available only for Java and Python currently).
>>>>
>>>> Please feel free to comment if you have any questions and suggestions.
>>>>
>>>>
>>>> [1] https://github.com/apache/beam/pull/28075
>>>> [2] https://issues.apache.org/jira/browse/INFRA-24950
>>>>
>>>> Regards,
>>>> Yi
>>>> --
>>>>
>>>> Yi Hu, (he/him/his)
>>>>
>>>> Software Engineer
>>>>
>>>>
>>>>

[Proposal] Enable EnricoMi/publish-unit-test-result-action

[Yi Hu via dev]

Hi everyone,

As you may have noticed, GitHub Action for test suites are setting up in
the Beam repository. A current gap is that Jenkins has a pretty convenient
test result page showing all tests / failed tests and stack trace, while
these are not available in github workflow logs.

Here we propose to introduce EnricoMi/publish-unit-test-result-action (
https://github.com/EnricoMi/publish-unit-test-result-action) to publish
Java (and possibly Python in the future) test results. An example PR can be
found [1] and an INFRA ticket in [2]. Currently both Java and Python test
reports are supported by this action (note that Jenkins test report page is
also available only for Java and Python currently).

Please feel free to comment if you have any questions and suggestions.


[1] https://github.com/apache/beam/pull/28075
[2] https://issues.apache.org/jira/browse/INFRA-24950

Regards,
Yi
-- 

Yi Hu, (he/him/his)

Software Engineer

Re: [VOTE] Release 2.50.0, release candidate #2

[Yi Hu via dev]

+1 (non-binding)

Verified Java IO load tests (TextIO, BigQuery, Bigtable) on Dataflow runner
(legacy and V2) using https://github.com/apache/beam/tree/master/it

On Mon, Aug 28, 2023 at 1:13 PM Ahmet Altay via dev 
wrote:

> +1 (binding).
>
> I validated python quick starts on direct and dataflow runners. Thank you
> for working on the release!
>
> On Mon, Aug 28, 2023 at 8:48 AM Robert Burke  wrote:
>
>> Good morning!
>>
>> RC2 validation and vote is still open!
>>
>> On Sun, Aug 27, 2023, 1:28 PM XQ Hu via dev  wrote:
>>
>>> +1
>>> Ran the simple Dataflow ML GPU batch job using
>>> https://github.com/google/dataflow-ml-starter with Python 2.50.0rc2 to
>>> validate the RC works well.
>>>
>>> On Sat, Aug 26, 2023 at 12:16 AM Valentyn Tymofieiev via dev <
>>> dev@beam.apache.org> wrote:
>>>
 +1

 Verified that the issue detected in RC0 has been resolved. Successfully
 ran a Python pipeline on ARM Dataflow workers.

 Noted that Dataflow runner logs became less verbose as the result of
 https://github.com/apache/beam/pull/27788. One line that I often pay
 attention to no longer appears at the default  INFO log level:

 ```
 INFO:apache_beam.runners.dataflow.dataflow_runner:2023-08-26T03:45:35.126Z:
 JOB_MESSAGE_DETAILED: All workers have finished the startup processes and
 began to receive work requests.
 ```

 Dataflow service can be adjusted to compensate for this (internal
 change: http://cl/560265419 ).

 On Fri, Aug 25, 2023 at 3:05 PM Bruno Volpato via dev <
 dev@beam.apache.org> wrote:

> +1 (non-binding).
>
> Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates
> (Java SDK 11, Dataflow runner).
>
> Thanks Robert!
>
> On Thu, Aug 24, 2023 at 7:12 PM Robert Burke 
> wrote:
>
>> Two minor erata from the previous email:
>>
>> The validation spreadsheet link should be:
>>
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=1014811464
>>
>> And the source code tag is: "v2.50.0-RC2"
>>
>> On 2023/08/24 23:09:23 Robert Burke wrote:
>> > Hi everyone,
>> > Please review and vote on the release candidate #2 for the version
>> 2.50.0,
>> > as follows:
>> > [ ] +1, Approve the release
>> > [ ] -1, Do not approve the release (please provide specific
>> comments)
>> >
>> >
>> > Reviewers are encouraged to test their own use cases with the
>> release
>> > candidate, and vote +1 if
>> > no issues are found. Only PMC member votes will count towards the
>> final
>> > vote, but votes from all
>> > community members is encouraged and helpful for finding
>> regressions; you
>> > can either test your own
>> > use cases or use cases from the validation sheet [10].
>> >
>> > Issues noted in RC1 vote proposal [13] have now been resolved.
>> >
>> > The staging area is available for your review, which includes:
>> > * GitHub Release notes [1],
>> > * the official Apache source release to be deployed to
>> dist.apache.org [2],
>> > which is signed with the key with fingerprint 02677FF4371A3756 (
>> > lostl...@apache.org) or D20316F712213422
>> > (GitHub Action automated) [[3],
>> > * all artifacts to be deployed to the Maven Central Repository [4],
>> > * source code tag "v2.50.0-RC2" [5],
>> > * website pull request listing the release [6], the blog post [6],
>> and
>> > publishing the API reference manual [7].
>> > * Java artifacts were built with Gradle 7.5.1 and OpenJDK
>> (Temurin)(build
>> > 1.8.0_382-b05).
>> > * Python artifacts are deployed along with the source release to the
>> > dist.apache.org [2] and PyPI[8].
>> > * Go artifacts and documentation are available at pkg.go.dev [9]
>> > * Validation sheet with a tab for 2.50.0 release to help with
>> validation
>> > [10].
>> > * Docker images published to Docker Hub [11].
>> > * PR to run tests against release branch [12].
>> >
>> > The vote will be open for at least 72 hours. It is adopted by
>> majority
>> > approval, with at least 3 PMC affirmative votes.
>> >
>> > For guidelines on how to try the release in your projects, check
>> out our
>> > blog post at https://beam.apache.org/blog/validate-beam-release/.
>> >
>> > Thanks,
>> > Robert Burke
>> > Apache Beam 2.50.0 Release Manager
>> >
>> > [1] https://github.com/apache/beam/milestone/14
>> > [2] https://dist.apache.org/repos/dist/dev/beam/2.50.0/
>> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> > [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1355/
>> > [5] https://github.com/apache/beam/tree/v2.50.0-RC2
>> > [6] https://github.com/apache/beam/pull/28055
>> > [7] 

Re: [ANNOUNCE] New committer: Ahmed Abualsaud

[Yi Hu via dev]

Congrats Ahmed!

On Fri, Aug 25, 2023 at 10:11 AM Ritesh Ghorse via dev 
wrote:

> Congrats Ahmed!
>
> On Fri, Aug 25, 2023 at 9:53 AM Kerry Donny-Clark via dev <
> dev@beam.apache.org> wrote:
>
>> Well done Ahmed!
>>
>> On Fri, Aug 25, 2023 at 9:17 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congrats Ahmed!
>>>
>>> On Fri, Aug 25, 2023 at 3:16 AM Jan Lukavský  wrote:
>>>
 Congrats Ahmed!
 On 8/25/23 07:56, Anand Inguva via dev wrote:

 Congratulations Ahmed :)

 On Fri, Aug 25, 2023 at 1:17 AM Damon Douglas 
 wrote:

> Well deserved! Congratulations, Ahmed! I'm so happy for you.
>
> On Thu, Aug 24, 2023, 5:46 PM Byron Ellis via dev 
> wrote:
>
>> Congratulations!
>>
>> On Thu, Aug 24, 2023 at 5:34 PM Robert Burke 
>> wrote:
>>
>>> Congratulations Ahmed!!
>>>
>>> On Thu, Aug 24, 2023, 4:08 PM Chamikara Jayalath via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congrats Ahmed!!

 On Thu, Aug 24, 2023 at 4:06 PM Bruno Volpato via dev <
 dev@beam.apache.org> wrote:

> Congratulations, Ahmed!
>
> Very well deserved!
>
>
> On Thu, Aug 24, 2023 at 6:09 PM XQ Hu via dev 
> wrote:
>
>> Congratulations, Ahmed!
>>
>> On Thu, Aug 24, 2023, 5:49 PM Ahmet Altay via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Ahmed Abualsaud (ahmedabuals...@apache.org).
>>>
>>> Ahmed has been part of the Beam community since January 2022,
>>> working mostly on IO connectors, made a large amount of 
>>> contributions to
>>> make Beam IOs more usable, performant, and reliable. And at the 
>>> same time
>>> Ahmed was active in the user list and at the Beam summit helping 
>>> users by
>>> sharing his knowledge.
>>>
>>> Considering their contributions to the project over this
>>> timeframe, the Beam PMC trusts Ahmed with the responsibilities of a 
>>> Beam
>>> committer. [1]
>>>
>>> Thank you Ahmed! And we are looking to see more of your
>>> contributions!
>>>
>>> Ahmet, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>>
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>>

Re: [RESULT] [VOTE] Vendored Dependency guava 32.1.2-jre Release

[Yi Hu via dev]

Hi everyone,

The vendored guava 32.1.2-jre is now released, and beam is switched to use
this version [1]. If you have working pull requests that have called vendor
guava directly (Java SDK), it may need to rebase the PR branch onto the
latest master branch, and switch `guava.v26_0_jre` namespace to
`guava.v32_1_2_jre`. Please do not hesitate to reply to this thread or pin
me (github username Abacn) in your PR if there is any question.

Best,
Yi

[1] https://github.com/apache/beam/pull/27895

On Tue, Aug 8, 2023 at 9:22 AM Yi Hu  wrote:

> Hi,
>
> Could a PMC member please help me finalize the source release? That is
> copy https://dist.apache.org/repos/dist/dev/beam/vendor/guava-32_1_2-jre/
> into
> https://dist.apache.org/repos/dist/release/beam/vendor/guava-32_1_2-jre
> using svn. Thanks!
>
> Regards,
> Yi
>
> On Tue, Aug 8, 2023 at 9:01 AM Yi Hu  wrote:
>
>> I'm happy to announce that we have unanimously approved this release.
>>
>> There are 5 approving votes, 3 of which are binding:
>>
>> * Chamikara Jayalath
>> * Ahmet Altay
>> * Robert Bradshaw
>>
>> There are no disapproving votes.
>>
>> Thanks everyone!
>>
>> --
>>
>> Yi Hu, (he/him/his)
>>
>> Software Engineer
>>
>>
>>

Re: [RESULT] [VOTE] Vendored Dependency guava 32.1.2-jre Release

[Yi Hu via dev]

Hi,

Could a PMC member please help me finalize the source release? That is copy
https://dist.apache.org/repos/dist/dev/beam/vendor/guava-32_1_2-jre/ into
https://dist.apache.org/repos/dist/release/beam/vendor/guava-32_1_2-jre
using svn. Thanks!

Regards,
Yi

On Tue, Aug 8, 2023 at 9:01 AM Yi Hu  wrote:

> I'm happy to announce that we have unanimously approved this release.
>
> There are 5 approving votes, 3 of which are binding:
>
> * Chamikara Jayalath
> * Ahmet Altay
> * Robert Bradshaw
>
> There are no disapproving votes.
>
> Thanks everyone!
>
> --
>
> Yi Hu, (he/him/his)
>
> Software Engineer
>
>
>

[RESULT] [VOTE] Vendored Dependency guava 32.1.2-jre Release

[Yi Hu via dev]

I'm happy to announce that we have unanimously approved this release.

There are 5 approving votes, 3 of which are binding:

* Chamikara Jayalath
* Ahmet Altay
* Robert Bradshaw

There are no disapproving votes.

Thanks everyone!

-- 

Yi Hu, (he/him/his)

Software Engineer

Re: [VOTE] Vendored Dependency guava 32.1.2-jre Release

[Yi Hu via dev]

+1 (non-binding) verified signature of source release, artifacts (jar,
pom); also tested the staged artifact locally with beam java core

On Mon, Aug 7, 2023 at 1:22 PM Ahmet Altay  wrote:

> +1 - I checked the tests on the linked testing PR. Thank you!
>
> On Thu, Aug 3, 2023 at 12:02 PM Yi Hu via dev  wrote:
>
>> Hi everyone,
>>
>>
>> Please review the release of the following artifacts that we vendor:
>>
>>
>> * beam-vendor-guava-32_1_2-jre
>>
>>
>> Please review and vote on the release candidate 1 for the version 0.1, as
>> follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> The complete staging area is available for your review, which includes:
>> * the official Apache source release to be deployed to dist.apache.org
>> [1], which is signed with the key with fingerprint 170405CB [2],
>> * all artifacts to be deployed to the Maven Central Repository [3],
>> * commit hash "ef2ca7a" [4],
>>
>> * Testing PR on the vendored dependency [5]
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Thanks,
>> Release Manager
>>
>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
>> [2] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [3]
>> https://repository.apache.org/content/repositories/orgapachebeam-1350/
>> [4]
>> https://github.com/apache/beam/commit/ef2ca7aa49ec75c3ab4e3a94b7ad8162e1c81c1e
>> [5] https://github.com/apache/beam/pull/27825
>>
>> Regards,
>> Yi
>>
>> --
>>
>> Yi Hu, (he/him/his)
>>
>> Software Engineer
>>
>>
>>

[VOTE] Vendored Dependency guava 32.1.2-jre Release

[Yi Hu via dev]

Hi everyone,


Please review the release of the following artifacts that we vendor:


* beam-vendor-guava-32_1_2-jre


Please review and vote on the release candidate 1 for the version 0.1, as
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* the official Apache source release to be deployed to dist.apache.org [1],
which is signed with the key with fingerprint 170405CB [2],
* all artifacts to be deployed to the Maven Central Repository [3],
* commit hash "ef2ca7a" [4],

* Testing PR on the vendored dependency [5]

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1] https://dist.apache.org/repos/dist/dev/beam/vendor/
[2] https://dist.apache.org/repos/dist/release/beam/KEYS
[3] https://repository.apache.org/content/repositories/orgapachebeam-1350/
[4]
https://github.com/apache/beam/commit/ef2ca7aa49ec75c3ab4e3a94b7ad8162e1c81c1e
[5] https://github.com/apache/beam/pull/27825

Regards,
Yi

-- 

Yi Hu, (he/him/his)

Software Engineer

Re: [DISCUSS] Upgrade vended guava version

[Yi Hu via dev]

Hi Hong,

Thanks for bringing this up. Sure, I would like to volunteer to work as
release manager [1] to vendor guava 32.1.2-jre. Created GitHub Issue for
tracking [2].

Regards,
Yi

[1] https://s.apache.org/beam-release-vendored-artifacts
[2] https://github.com/apache/beam/issues/27801


On Mon, Jul 31, 2023 at 1:08 PM Ahmet Altay via dev 
wrote:

> Hi Hong,
>
> Thank you for reaching out and thank you for offering to help. If you can
> start the PR and do the testing, one of the committers could help with the
> process.
>
> Thank you!
> Ahmet
>
> On Mon, Jul 31, 2023 at 9:13 AM Hong Teoh  wrote:
>
>> Hi all,
>>
>> The current version of guava that is vended in Beam is
>> com.google.guava:guava:26.0-jre.
>>
>> This version is really old, and has active vulnerabilities [1] [2]
>> [1] https://mvnrepository.com/artifact/com.google.guava/guava/26.0-jre
>> [2] CVE-2023-2976
>> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2976
>> [3] CVE-2020-8908
>> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908
>>
>> Is there anyone else keen on upgrading the vended guava version to match
>> the guava version of 32.1.1-jre ? [4]
>> [4]
>> https://github.com/apache/beam/blame/df6964aac62a521081481b21c96ecd506ea3c503/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L542
>>
>> I am happy to contribute the PR to upgrade the guava dependencies in the
>> Beam repository, but I would need a committer to drive the release of the
>> vended version first! [5]
>> [5]
>> https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit#heading=h.vhcuqlttpnog
>>
>>
>> Side question: Does anyone know why we have libraries that use the
>> non-vended guava version? [6]
>> [6]
>> https://github.com/search?q=repo%3Aapache%2Fbeam%20library.java.guava=code
>>
>
> @Kenneth Knowles  - might know.
>
>
>>
>>
>>
>> Regards,
>> Hong
>>
>

Re: [VOTE] Release 2.49.0, release candidate #2

[Yi Hu via dev]

Sorry for bothering,

It appears the email delivery issue still exists. In case the last emails
are not received, check the complete thread here:
https://lists.apache.org/thread/r7r5q5mq7rqjrfbf8nj90smrdkss0sbf

Still waiting for PMC finalization, mainly deploy the source release from
staging (https://dist.apache.org/repos/dist/dev/beam/2.49.0/) to release
(will be https://dist.apache.org/repos/dist/release/beam/2.49.0/). Thanks!

Regards,
Yi

On Mon, Jul 17, 2023 at 10:27 AM Yi Hu  wrote:

> Could a PMC member please help finalizing the release (
> https://beam.apache.org/contribute/release-guide/#pmc-only-finalization),
> mainly deploy the source release from staging (
> https://dist.apache.org/repos/dist/dev/beam/2.49.0/) to release (will be
> https://dist.apache.org/repos/dist/release/beam/2.49.0/). Thanks!
>
>
> On Mon, Jul 17, 2023 at 7:28 AM Yi Hu  wrote:
>
>> I'm happy to announce that we have unanimously approved this release.
>>
>> There are 8 approving votes, 4 of which are binding:
>> * approver 1: Jan Lukavský
>> * approver 2: Robert Bradshaw
>> * approver 3: Chamikara Jayalath
>> * approver 4: Ahmet Altay
>>
>> There are no disapproving votes.
>>
>> Thanks everyone!
>>
>> Note: there is an ongoing issue such that some reply emails not get
>> delivered to certain email address (like gmail). Check the complete thread
>> here: https://lists.apache.org/thread/r7r5q5mq7rqjrfbf8nj90smrdkss0sbf
>>
>>

[ANNOUNCE] Beam 2.49.0 Released

[Yi Hu via dev]

The Apache Beam Team is pleased to announce the release of version 2.49.0.

You can download the release here:

https://beam.apache.org/get-started/downloads/ (website daily update
pending)

This release includes bug fixes, features, and improvements detailed on the
Beam Blog: https://beam.apache.org/blog/beam-2.49.0/ (website daily update
pending)
and the Github release page
https://github.com/apache/beam/releases/tag/v2.49.0

Thanks to everyone who contributed to this release, and we hope you enjoy
using Beam 2.49.0.

-- Yi, on behalf of the Apache Beam Team.


-- 

Yi Hu, (he/him/his)

Software Engineer

Re: [VOTE] Release 2.49.0, release candidate #2

[Yi Hu via dev]

Could a PMC member please help finalizing the release (
https://beam.apache.org/contribute/release-guide/#pmc-only-finalization),
mainly deploy the source release from staging (
https://dist.apache.org/repos/dist/dev/beam/2.49.0/) to release (will be
https://dist.apache.org/repos/dist/release/beam/2.49.0/). Thanks!


On Mon, Jul 17, 2023 at 7:28 AM Yi Hu  wrote:

> I'm happy to announce that we have unanimously approved this release.
>
> There are 8 approving votes, 4 of which are binding:
> * approver 1: Jan Lukavský
> * approver 2: Robert Bradshaw
> * approver 3: Chamikara Jayalath
> * approver 4: Ahmet Altay
>
> There are no disapproving votes.
>
> Thanks everyone!
>
> Note: there is an ongoing issue such that some reply emails not get
> delivered to certain email address (like gmail). Check the complete thread
> here: https://lists.apache.org/thread/r7r5q5mq7rqjrfbf8nj90smrdkss0sbf
>
>

Re: [VOTE] Release 2.49.0, release candidate #2

[Yi Hu via dev]

I'm happy to announce that we have unanimously approved this release.

There are 8 approving votes, 4 of which are binding:
* approver 1: Jan Lukavský
* approver 2: Robert Bradshaw
* approver 3: Chamikara Jayalath
* approver 4: Ahmet Altay

There are no disapproving votes.

Thanks everyone!

Note: there is an ongoing issue such that some reply emails not get
delivered to certain email address (like gmail). Check the complete thread
here: https://lists.apache.org/thread/r7r5q5mq7rqjrfbf8nj90smrdkss0sbf

On Fri, Jul 14, 2023 at 4:50 PM Valentyn Tymofieiev via dev <
dev@beam.apache.org> wrote:

> +1. Tested a few python pipelines on Dataflow Runner V1 and Runner V2.
>
>
>
> On Thu, Jul 13, 2023 at 12:54 PM Svetak Sundhar via dev <
> dev@beam.apache.org> wrote:
>
>> +1 (Non-Binding)
>>
>> Python quickstart Dataflow runner.
>>
>>
>> Svetak Sundhar
>>
>>   Data Engineer
>> s vetaksund...@google.com
>>
>>
>>
>> On Thu, Jul 13, 2023 at 5:03 AM Jan Lukavský  wrote:
>>
>>> +1 (binding)
>>>
>>> Tested Java SDK with FlinkRunner.
>>>
>>>  Jan
>>> On 7/13/23 02:30, Bruno Volpato via dev wrote:
>>>
>>> +1 (non-binding).
>>>
>>> Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates
>>> (Java SDK 11, Dataflow runner).
>>>
>>> Thanks Yi!
>>>
>>> On Tue, Jul 11, 2023 at 4:23 PM Yi Hu via dev 
>>> wrote:
>>>
>>>> Hi everyone,
>>>> Please review and vote on the release candidate #2 for the version
>>>> 2.49.0, as follows:
>>>> [ ] +1, Approve the release
>>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>>
>>>>
>>>> Reviewers are encouraged to test their own use cases with the release
>>>> candidate, and vote +1 if
>>>> no issues are found. Only PMC member votes will count towards the final
>>>> vote, but votes from all
>>>> community members is encouraged and helpful for finding regressions;
>>>> you can either test your own
>>>> use cases or use cases from the validation sheet [10].
>>>>
>>>> The complete staging area is available for your review, which includes:
>>>> * GitHub Release notes [1],
>>>> * the official Apache source release to be deployed to dist.apache.org
>>>> [2], which is signed with the key with
>>>> fingerprint either CB6974C8170405CB (y...@apache.org) or
>>>> D20316F712213422 (GitHub Action automated) [3],
>>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>>> * source code tag "v2.49.0-RC2" [5],
>>>> * website pull request listing the release [6], the blog post [6], and
>>>> publishing the API reference manual [7].
>>>> * Java artifacts were built with Gradle GRADLE_VERSION and
>>>> OpenJDK/Oracle JDK JDK_VERSION.
>>>>
>>> nit: versions were missing.
>
>> * Python artifacts are deployed along with the source release to the
>>>> dist.apache.org [2] and PyPI [8].
>>>> * Go artifacts and documentation are available at pkg.go.dev [9]
>>>> * Validation sheet with a tab for 2.49.0 release to help with
>>>> validation [10].
>>>> * Docker images published to Docker Hub [11].
>>>> * PR to run tests against release branch [12].
>>>>
>>>> The vote will be open for at least 72 hours. It is adopted by majority
>>>> approval, with at least 3 PMC affirmative votes.
>>>>
>>>> For guidelines on how to try the release in your projects, check out
>>>> our blog post at /blog/validate-beam-release/.
>>>>
>>>> Thanks,
>>>> Release Manager
>>>>
>>>> [1] https://github.com/apache/beam/milestone/13
>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.49.0/
>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>> [4]
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1349/
>>>> [5] https://github.com/apache/beam/tree/v2.49.0-RC2
>>>> [6] https://github.com/apache/beam/pull/27374 (unchanged since RC1)
>>>> [7] https://github.com/apache/beam-site/pull/646  (unchanged since RC1)
>>>> [8] https://pypi.org/project/apache-beam/2.49.0rc2/
>>>> [9]
>>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.49.0-RC2/go/pkg/beam
>>>> [10]
>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=934901728
>>>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>>>> [12] https://github.com/apache/beam/pull/27307
>>>>
>>>> --
>>>>
>>>> Yi Hu, (he/him/his)
>>>>
>>>> Software Engineer
>>>>
>>>>
>>>>

Re: [VOTE] Release 2.49.0, release candidate #2

[Yi Hu via dev]

+1 (non-binding)

Tested Java and Python QuickStart on Direct and Dataflow runner; tested
Python Mobile Game on Direct and Dataflow Runner

Yi

On Thu, Jul 13, 2023 at 5:03 AM Jan Lukavský  wrote:

> +1 (binding)
>
> Tested Java SDK with FlinkRunner.
>
>  Jan
> On 7/13/23 02:30, Bruno Volpato via dev wrote:
>
> +1 (non-binding).
>
> Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates
> (Java SDK 11, Dataflow runner).
>
> Thanks Yi!
>
> On Tue, Jul 11, 2023 at 4:23 PM Yi Hu via dev  wrote:
>
>> Hi everyone,
>> Please review and vote on the release candidate #2 for the version
>> 2.49.0, as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> Reviewers are encouraged to test their own use cases with the release
>> candidate, and vote +1 if
>> no issues are found. Only PMC member votes will count towards the final
>> vote, but votes from all
>> community members is encouraged and helpful for finding regressions; you
>> can either test your own
>> use cases or use cases from the validation sheet [10].
>>
>> The complete staging area is available for your review, which includes:
>> * GitHub Release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org
>> [2], which is signed with the key with
>> fingerprint either CB6974C8170405CB (y...@apache.org) or D20316F712213422
>> (GitHub Action automated) [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.49.0-RC2" [5],
>> * website pull request listing the release [6], the blog post [6], and
>> publishing the API reference manual [7].
>> * Java artifacts were built with Gradle GRADLE_VERSION and OpenJDK/Oracle
>> JDK JDK_VERSION.
>> * Python artifacts are deployed along with the source release to the
>> dist.apache.org [2] and PyPI [8].
>> * Go artifacts and documentation are available at pkg.go.dev [9]
>> * Validation sheet with a tab for 2.49.0 release to help with validation
>> [10].
>> * Docker images published to Docker Hub [11].
>> * PR to run tests against release branch [12].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> For guidelines on how to try the release in your projects, check out our
>> blog post at /blog/validate-beam-release/.
>>
>> Thanks,
>> Release Manager
>>
>> [1] https://github.com/apache/beam/milestone/13
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.49.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1349/
>> [5] https://github.com/apache/beam/tree/v2.49.0-RC2
>> [6] https://github.com/apache/beam/pull/27374 (unchanged since RC1)
>> [7] https://github.com/apache/beam-site/pull/646  (unchanged since RC1)
>> [8] https://pypi.org/project/apache-beam/2.49.0rc2/
>> [9]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.49.0-RC2/go/pkg/beam
>> [10]
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=934901728
>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>> [12] https://github.com/apache/beam/pull/27307
>>
>> --
>>
>> Yi Hu, (he/him/his)
>>
>> Software Engineer
>>
>>
>>

[VOTE] Release 2.49.0, release candidate #2

[Yi Hu via dev]

Hi everyone,
Please review and vote on the release candidate #2 for the version 2.49.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


Reviewers are encouraged to test their own use cases with the release
candidate, and vote +1 if
no issues are found. Only PMC member votes will count towards the final
vote, but votes from all
community members is encouraged and helpful for finding regressions; you
can either test your own
use cases or use cases from the validation sheet [10].

The complete staging area is available for your review, which includes:
* GitHub Release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with
fingerprint either CB6974C8170405CB (y...@apache.org) or D20316F712213422
(GitHub Action automated) [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.49.0-RC2" [5],
* website pull request listing the release [6], the blog post [6], and
publishing the API reference manual [7].
* Java artifacts were built with Gradle GRADLE_VERSION and OpenJDK/Oracle
JDK JDK_VERSION.
* Python artifacts are deployed along with the source release to the
dist.apache.org [2] and PyPI [8].
* Go artifacts and documentation are available at pkg.go.dev [9]
* Validation sheet with a tab for 2.49.0 release to help with validation
[10].
* Docker images published to Docker Hub [11].
* PR to run tests against release branch [12].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

For guidelines on how to try the release in your projects, check out our
blog post at /blog/validate-beam-release/.

Thanks,
Release Manager

[1] https://github.com/apache/beam/milestone/13
[2] https://dist.apache.org/repos/dist/dev/beam/2.49.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1349/
[5] https://github.com/apache/beam/tree/v2.49.0-RC2
[6] https://github.com/apache/beam/pull/27374 (unchanged since RC1)
[7] https://github.com/apache/beam-site/pull/646  (unchanged since RC1)
[8] https://pypi.org/project/apache-beam/2.49.0rc2/
[9]
https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.49.0-RC2/go/pkg/beam
[10]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=934901728
[11] https://hub.docker.com/search?q=apache%2Fbeam=image
[12] https://github.com/apache/beam/pull/27307

-- 

Yi Hu, (he/him/his)

Software Engineer

Re: [VOTE] Release 2.49.0, release candidate #1

[Yi Hu via dev]

Sorry for confusion, I meant around July 11.

Thanks,
Release Manager

On Mon, Jul 10, 2023 at 10:36 AM Yi Hu  wrote:

> Hi everyone,
>
> Thanks for helping with the validation of RC1. This is another update for
> the 2.49.0 release candidate validation.
>
> There was another issue added to the milestone [1] since the last update
> regarding java fn-execution, which
> could cause OOM in some use cases [2].
>
> As planned, if there is no outstanding issue in the milestone [1], RC2
> will be tagged around Jun 11.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/beam/milestone/13
> [2] https://github.com/apache/beam/issues/27401
>
>
>
>
> On Fri, Jul 7, 2023 at 9:17 PM Bruno Volpato  wrote:
>
>> +1 (non-binding).
>>
>> Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates
>> (Java SDK 11, Dataflow runner).
>>
>> Thanks Yi!
>>
>> On Fri, Jul 7, 2023 at 10:51 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> +1 (binding)
>>>
>>> Tested with https://github.com/Talend/beam-samples/actions
>>> (Java SDK v8/v11/v17, Spark 3.x runner).
>>>
>>> —
>>> Alexey
>>>
>>> On 6 Jul 2023, at 17:34, Yi Hu via dev  wrote:
>>>
>>> Hi everyone,
>>> Please review and vote on the release candidate #1 for the version
>>> 2.49.0, as follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>>
>>> Reviewers are encouraged to test their own use cases with the release
>>> candidate, and vote +1 if
>>> no issues are found. Only PMC member votes will count towards the final
>>> vote, but votes from all
>>> community members is encouraged and helpful for finding regressions; you
>>> can either test your own
>>> use cases or use cases from the validation sheet [10].
>>>
>>> The complete staging area is available for your review, which includes:
>>> * GitHub Release notes [1],
>>> * the official Apache source release to be deployed to dist.apache.org
>>> [2], which is signed with the key with
>>> fingerprint either CB6974C8170405CB (y...@apache.org) or D20316F712213422
>>> (GitHub Action automated) [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "v2.49.0-RC1" [5],
>>> * website pull request listing the release [6], the blog post [6], and
>>> publishing the API reference manual [7].
>>> * Java artifacts were built with Gradle GRADLE_VERSION and
>>> OpenJDK/Oracle JDK JDK_VERSION.
>>> * Python artifacts are deployed along with the source release to the
>>> dist.apache.org [2] and PyPI [8].
>>> * Go artifacts and documentation are available at pkg.go.dev [9]
>>> * Validation sheet with a tab for 2.49.0 release to help with validation
>>> [10].
>>> * Docker images published to Docker Hub [11].
>>> * PR to run tests against release branch [12].
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> For guidelines on how to try the release in your projects, check out our
>>> blog post at /blog/validate-beam-release/.
>>>
>>> Thanks,
>>> Release Manager
>>>
>>> [1] https://github.com/apache/beam/milestone/13
>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.49.0/
>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [4]
>>> https://repository.apache.org/content/repositories/orgapachebeam-1348/
>>> [5] https://github.com/apache/beam/tree/v2.49.0-RC1
>>> [6] https://github.com/apache/beam/pull/27374
>>> [7] https://github.com/apache/beam-site/pull/646
>>> [8] https://pypi.org/project/apache-beam/2.49.0rc1/
>>> [9]
>>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.49.0-RC1/go/pkg/beam
>>> [10]
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=934901728
>>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>>> [12] https://github.com/apache/beam/pull/27307
>>>
>>> --
>>> Yi Hu, (he/him/his)
>>> Software Engineer
>>>
>>>
>>>

Re: [VOTE] Release 2.49.0, release candidate #1

[Yi Hu via dev]

Hi everyone,

Thanks for helping with the validation of RC1. This is another update for
the 2.49.0 release candidate validation.

There was another issue added to the milestone [1] since the last update
regarding java fn-execution, which
could cause OOM in some use cases [2].

As planned, if there is no outstanding issue in the milestone [1], RC2 will
be tagged around Jun 11.

Thanks,
Release Manager

[1] https://github.com/apache/beam/milestone/13
[2] https://github.com/apache/beam/issues/27401




On Fri, Jul 7, 2023 at 9:17 PM Bruno Volpato  wrote:

> +1 (non-binding).
>
> Tested with https://github.com/GoogleCloudPlatform/DataflowTemplates
> (Java SDK 11, Dataflow runner).
>
> Thanks Yi!
>
> On Fri, Jul 7, 2023 at 10:51 AM Alexey Romanenko 
> wrote:
>
>> +1 (binding)
>>
>> Tested with https://github.com/Talend/beam-samples/actions
>> (Java SDK v8/v11/v17, Spark 3.x runner).
>>
>> —
>> Alexey
>>
>> On 6 Jul 2023, at 17:34, Yi Hu via dev  wrote:
>>
>> Hi everyone,
>> Please review and vote on the release candidate #1 for the version
>> 2.49.0, as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> Reviewers are encouraged to test their own use cases with the release
>> candidate, and vote +1 if
>> no issues are found. Only PMC member votes will count towards the final
>> vote, but votes from all
>> community members is encouraged and helpful for finding regressions; you
>> can either test your own
>> use cases or use cases from the validation sheet [10].
>>
>> The complete staging area is available for your review, which includes:
>> * GitHub Release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org
>> [2], which is signed with the key with
>> fingerprint either CB6974C8170405CB (y...@apache.org) or D20316F712213422
>> (GitHub Action automated) [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.49.0-RC1" [5],
>> * website pull request listing the release [6], the blog post [6], and
>> publishing the API reference manual [7].
>> * Java artifacts were built with Gradle GRADLE_VERSION and OpenJDK/Oracle
>> JDK JDK_VERSION.
>> * Python artifacts are deployed along with the source release to the
>> dist.apache.org [2] and PyPI [8].
>> * Go artifacts and documentation are available at pkg.go.dev [9]
>> * Validation sheet with a tab for 2.49.0 release to help with validation
>> [10].
>> * Docker images published to Docker Hub [11].
>> * PR to run tests against release branch [12].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> For guidelines on how to try the release in your projects, check out our
>> blog post at /blog/validate-beam-release/.
>>
>> Thanks,
>> Release Manager
>>
>> [1] https://github.com/apache/beam/milestone/13
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.49.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1348/
>> [5] https://github.com/apache/beam/tree/v2.49.0-RC1
>> [6] https://github.com/apache/beam/pull/27374
>> [7] https://github.com/apache/beam-site/pull/646
>> [8] https://pypi.org/project/apache-beam/2.49.0rc1/
>> [9]
>> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.49.0-RC1/go/pkg/beam
>> [10]
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=934901728
>> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
>> [12] https://github.com/apache/beam/pull/27307
>>
>> --
>> Yi Hu, (he/him/his)
>> Software Engineer
>>
>>
>>

Re: [VOTE] Release 2.49.0, release candidate #1

[Yi Hu via dev]

Hi everyone,

This is an update for the 2.49.0 release candidate validation.

There was one issue added to the milestone [1] since the creation of this
thread that was regarding TypeScript container [2].
Also, two infrastructure fixes are integrated into the release branch [3,
4], both are part of the ongoing work of automating the
release process and do not involve SDK code changes.

This means there will be an RC2 to be tagged. Because the changes on the
release branch since RC1 tag do not involve
Java, Python, and Go SDK, I encourage continue the current release
validations, and please use the spreadsheet [5] and add
identified issues to the milestone [1] if any.

I plan to tag RC2 3 weekdays after RC1 validation has started (that is
around Jun 11) if there is no outstanding issue in the
milestone [1] then.

Thanks,
Release Manager

[1] https://github.com/apache/beam/milestone/13
[2] https://github.com/apache/beam/pull/27386
[3] https://github.com/apache/beam/pull/26967
[4] https://github.com/apache/beam/pull/27398
[5]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=934901728



On Fri, Jul 7, 2023 at 10:50 AM Alexey Romanenko 
wrote:

> +1 (binding)
>
> Tested with https://github.com/Talend/beam-samples/actions
> (Java SDK v8/v11/v17, Spark 3.x runner).
>
> —
> Alexey
>
> On 6 Jul 2023, at 17:34, Yi Hu via dev  wrote:
>
> Hi everyone,
> Please review and vote on the release candidate #1 for the version 2.49.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> Reviewers are encouraged to test their own use cases with the release
> candidate, and vote +1 if
> no issues are found. Only PMC member votes will count towards the final
> vote, but votes from all
> community members is encouraged and helpful for finding regressions; you
> can either test your own
> use cases or use cases from the validation sheet [10].
>
> The complete staging area is available for your review, which includes:
> * GitHub Release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2], which is signed with the key with
> fingerprint either CB6974C8170405CB (y...@apache.org) or D20316F712213422
> (GitHub Action automated) [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.49.0-RC1" [5],
> * website pull request listing the release [6], the blog post [6], and
> publishing the API reference manual [7].
> * Java artifacts were built with Gradle GRADLE_VERSION and OpenJDK/Oracle
> JDK JDK_VERSION.
> * Python artifacts are deployed along with the source release to the
> dist.apache.org [2] and PyPI [8].
> * Go artifacts and documentation are available at pkg.go.dev [9]
> * Validation sheet with a tab for 2.49.0 release to help with validation
> [10].
> * Docker images published to Docker Hub [11].
> * PR to run tests against release branch [12].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> For guidelines on how to try the release in your projects, check out our
> blog post at /blog/validate-beam-release/.
>
> Thanks,
> Release Manager
>
> [1] https://github.com/apache/beam/milestone/13
> [2] https://dist.apache.org/repos/dist/dev/beam/2.49.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1348/
> [5] https://github.com/apache/beam/tree/v2.49.0-RC1
> [6] https://github.com/apache/beam/pull/27374
> [7] https://github.com/apache/beam-site/pull/646
> [8] https://pypi.org/project/apache-beam/2.49.0rc1/
> [9]
> https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.49.0-RC1/go/pkg/beam
> [10]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=934901728
> [11] https://hub.docker.com/search?q=apache%2Fbeam=image
> [12] https://github.com/apache/beam/pull/27307
>
> --
> Yi Hu, (he/him/his)
> Software Engineer
>
>
>

[VOTE] Release 2.49.0, release candidate #1

[Yi Hu via dev]

Hi everyone,
Please review and vote on the release candidate #1 for the version 2.49.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


Reviewers are encouraged to test their own use cases with the release
candidate, and vote +1 if
no issues are found. Only PMC member votes will count towards the final
vote, but votes from all
community members is encouraged and helpful for finding regressions; you
can either test your own
use cases or use cases from the validation sheet [10].

The complete staging area is available for your review, which includes:
* GitHub Release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with
fingerprint either CB6974C8170405CB (y...@apache.org) or D20316F712213422
(GitHub Action automated) [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.49.0-RC1" [5],
* website pull request listing the release [6], the blog post [6], and
publishing the API reference manual [7].
* Java artifacts were built with Gradle GRADLE_VERSION and OpenJDK/Oracle
JDK JDK_VERSION.
* Python artifacts are deployed along with the source release to the
dist.apache.org [2] and PyPI [8].
* Go artifacts and documentation are available at pkg.go.dev [9]
* Validation sheet with a tab for 2.49.0 release to help with validation
[10].
* Docker images published to Docker Hub [11].
* PR to run tests against release branch [12].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

For guidelines on how to try the release in your projects, check out our
blog post at /blog/validate-beam-release/.

Thanks,
Release Manager

[1] https://github.com/apache/beam/milestone/13
[2] https://dist.apache.org/repos/dist/dev/beam/2.49.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1348/
[5] https://github.com/apache/beam/tree/v2.49.0-RC1
[6] https://github.com/apache/beam/pull/27374
[7] https://github.com/apache/beam-site/pull/646
[8] https://pypi.org/project/apache-beam/2.49.0rc1/
[9]
https://pkg.go.dev/github.com/apache/beam/sdks/v2@v2.49.0-RC1/go/pkg/beam
[10]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=934901728
[11] https://hub.docker.com/search?q=apache%2Fbeam=image
[12] https://github.com/apache/beam/pull/27307

-- 

Yi Hu, (he/him/his)

Software Engineer

Re: [PROPOSAL] Preparing for 2.49.0 Release

[Yi Hu via dev]

Hi everyone,

The release branch for 2.49.0 has been cut. There is currently no
outstanding change that needs to be cherry-picked, and no open milestone
[1]. The verification on the release branch is done on [2]. There were a
few postcommit failures and flaky tests, all known issues identified as not
release blocking (see the comment in that PR for more detail).

As such, I plan to move ahead to building an RC1.

Thanks,
Yi Hu

[1] https://github.com/apache/beam/milestone/13
[2] https://github.com/apache/beam/pull/27307


On Mon, Jun 26, 2023 at 4:34 PM Yi Hu  wrote:

> Hi,
>
> As for preparing the release, could one of the owner of apache-beam
> project in PyPi please add me as a maintainer?
>
> Username: abacn
> email: y...@apache.org
>
> Best,
> Yi
>
> On Thu, Jun 15, 2023 at 10:44 AM Yi Hu  wrote:
>
>> Hey Beam community,
>>
>> The next release (2.49.0) branch cut is scheduled on June 28th, 2023,
>> according to
>> the release calendar [1].
>>
>> I volunteer to perform this release. My plan is to cut the branch on
>> that date, and cherrypick release-blocking fixes afterwards, if any.
>>
>> Please help me make sure the release goes smoothly by:
>> - Making sure that any unresolved release blocking issues for 2.49.0 should
>> have their "Milestone" marked as "2.49.0 Release" as soon as possible.
>> - Reviewing the current release blockers [2] and remove the Milestone if
>> they don't meet the criteria at [3].
>>
>> Let me know if you have any comments/objections/questions.
>>
>> Thanks,
>>
>> Yi
>>
>> [1]
>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>> [2] https://github.com/apache/beam/milestone/13
>> [3] https://beam.apache.org/contribute/release-blocking/
>>
>> --
>>
>> Yi Hu, (he/him/his)
>>
>> Software Engineer
>>
>>
>>

Re: [PROPOSAL] Preparing for 2.49.0 Release

[Yi Hu via dev]

Hi,

As for preparing the release, could one of the owner of apache-beam project
in PyPi please add me as a maintainer?

Username: abacn
email: y...@apache.org

Best,
Yi

On Thu, Jun 15, 2023 at 10:44 AM Yi Hu  wrote:

> Hey Beam community,
>
> The next release (2.49.0) branch cut is scheduled on June 28th, 2023,
> according to
> the release calendar [1].
>
> I volunteer to perform this release. My plan is to cut the branch on that
> date, and cherrypick release-blocking fixes afterwards, if any.
>
> Please help me make sure the release goes smoothly by:
> - Making sure that any unresolved release blocking issues for 2.49.0 should
> have their "Milestone" marked as "2.49.0 Release" as soon as possible.
> - Reviewing the current release blockers [2] and remove the Milestone if
> they don't meet the criteria at [3].
>
> Let me know if you have any comments/objections/questions.
>
> Thanks,
>
> Yi
>
> [1]
> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
> [2] https://github.com/apache/beam/milestone/13
> [3] https://beam.apache.org/contribute/release-blocking/
>
> --
>
> Yi Hu, (he/him/his)
>
> Software Engineer
>
>
>

Re: Jenkins down

[Yi Hu via dev]

Hi,

Jenkins is now back and tests run successfully. Thanks for your patience!

Best,
Yi

On Thu, Jun 15, 2023 at 3:13 PM Yi Hu  wrote:

> Dear Beam developers,
>
> There is currently an outage for Beam repo's Jenkins test server (
> https://github.com/apache/beam/issues/27142 and
> https://issues.apache.org/jira/browse/INFRA-24703) some tests shows
> success status but actually not run anything. Until it get resolved I
> propose a code freeze to avoid untested code get merged.
>
> Best,
> Yi
>
> --
>
> Yi Hu, (he/him/his)
>
> Software Engineer
>
>
>

Jenkins down

[Yi Hu via dev]

Dear Beam developers,

There is currently an outage for Beam repo's Jenkins test server (
https://github.com/apache/beam/issues/27142 and
https://issues.apache.org/jira/browse/INFRA-24703) some tests shows success
status but actually not run anything. Until it get resolved I propose a
code freeze to avoid untested code get merged.

Best,
Yi

-- 

Yi Hu, (he/him/his)

Software Engineer

[PROPOSAL] Preparing for 2.49.0 Release

[Yi Hu via dev]

Hey Beam community,

The next release (2.49.0) branch cut is scheduled on June 28th, 2023,
according to
the release calendar [1].

I volunteer to perform this release. My plan is to cut the branch on that
date, and cherrypick release-blocking fixes afterwards, if any.

Please help me make sure the release goes smoothly by:
- Making sure that any unresolved release blocking issues for 2.49.0 should
have their "Milestone" marked as "2.49.0 Release" as soon as possible.
- Reviewing the current release blockers [2] and remove the Milestone if
they don't meet the criteria at [3].

Let me know if you have any comments/objections/questions.

Thanks,

Yi

[1]
https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
[2] https://github.com/apache/beam/milestone/13
[3] https://beam.apache.org/contribute/release-blocking/

-- 

Yi Hu, (he/him/his)

Software Engineer

[Notice] Jenkins seed job comment trigger no longer working, and possible solutions

[Yi Hu via dev]

Dear Beam Developers,

tl;dr For PRs involving Jenkins task changes authored by Beam committers,
"Run seed job" no longer working due to apache infra change.

It is noted that due to recent Apache Infra change on LDAP server, Beam
Jenkins CI/CD no longer has access to the GitHub username list, and
consequently several Jenkins tasks that used to have triggers enabled by
committers can no longer triggered by commenting phrase against PR (e.g.
"Run seed job")

A full list of affected jobs are


   - seed_00_job
   - seed_job_standalone
   - beam_Publish_Docker_Snapshots
   - beam_Dependency_Check
   - beam_Metrics_Report

Other than the seed job are release related workflows and should not affect
development on code base.

I have created a PR to temporarily remove the step of fetching GitHub
usernames [2] to get the seed job back green. After that, I would like to
ask the community if it is fine to either


   - Leave these jobs have no comment trigger (they can still be manually
   triggered via steps described in [2], besides the scheduled jobs)
   - Maintain a list of committer GitHub usernames manually in
   
https://github.com/apache/beam/blob/master/.test-infra/jenkins/Committers.groovy


Please feel free to share if you have a better idea for fixing this.

See more context on
[1] https://github.com/apache/beam/issues/26602
[2] https://github.com/apache/beam/pull/26652


Regards,
Yi

-- 

Yi Hu, (he/him/his)

Software Engineer

Re: [ANNOUNCE] New committer: Damon Douglas

[Yi Hu via dev]

Congrats Damon!

On Tue, Apr 25, 2023 at 8:55 AM Ritesh Ghorse via dev 
wrote:

> Congratulations Damon!
>
> On Tue, Apr 25, 2023 at 12:03 AM Byron Ellis via dev 
> wrote:
>
>> Congrats Damon!
>>
>> On Mon, Apr 24, 2023 at 8:57 PM Austin Bennett  wrote:
>>
>>> thanks for all you do @Damon Douglas  !
>>>
>>> On Mon, Apr 24, 2023 at 1:00 PM Robert Burke  wrote:
>>>
 Congratulations Damon!!!

 On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles  wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Damon Douglas (damondoug...@apache.org)
>
> Damon has contributed widely: Beam Katas, playground, infrastructure,
> and many IO connectors. Damon does lots of code review in addition to 
> code.
> (yes, you can review code as a non-committer!)
>
> Considering their contributions to the project over this timeframe,
> the Beam PMC trusts Damon with the responsibilities of a Beam committer. 
> [1]
>
> Thank you Damon! And we are looking to see more of your contributions!
>
> Kenn, on behalf of the Apache Beam PMC
>
> [1]
>
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>

Re: [ANNOUNCE] New committer: Anand Inguva

[Yi Hu via dev]

Congrats Anand!

On Fri, Apr 21, 2023 at 3:54 PM Danielle Syse via dev 
wrote:

> Congratulations!
>
> On Fri, Apr 21, 2023 at 3:53 PM Damon Douglas via dev 
> wrote:
>
>> Congratulations Anand!
>>
>> On Fri, Apr 21, 2023 at 12:28 PM Ritesh Ghorse via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congratulations Anand!
>>>
>>> On Fri, Apr 21, 2023 at 3:24 PM Ahmed Abualsaud via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congrats Anand!

 On Fri, Apr 21, 2023 at 3:18 PM Anand Inguva via dev <
 dev@beam.apache.org> wrote:

> Thanks everyone. Really excited to be a part of Beam Committers.
>
> On Fri, Apr 21, 2023 at 3:07 PM XQ Hu via dev 
> wrote:
>
>> Congratulations, Anand!!!
>>
>> On Fri, Apr 21, 2023 at 2:31 PM Jack McCluskey via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congratulations, Anand!
>>>
>>> On Fri, Apr 21, 2023 at 2:28 PM Valentyn Tymofieiev via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congratulations!

 On Fri, Apr 21, 2023 at 8:19 PM Jan Lukavský 
 wrote:

> Congrats Anand!
> On 4/21/23 20:05, Robert Burke wrote:
>
> Congratulations Anand!
>
> On Fri, Apr 21, 2023, 10:55 AM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> Woohoo, congrats Anand! This is very well deserved!
>>
>> On Fri, Apr 21, 2023 at 1:54 PM Chamikara Jayalath <
>> chamik...@apache.org> wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Anand Inguva (ananding...@apache.org)
>>>
>>> Anand has been contributing to Apache Beam for more than a year
>>> and  authored and reviewed more than 100 PRs. Anand has been a core
>>> contributor to Beam Python SDK and drove the efforts to support 
>>> Python 3.10
>>> and Python 3.11.
>>>
>>> Considering their contributions to the project over this
>>> timeframe, the Beam PMC trusts Anand with the responsibilities of a 
>>> Beam
>>> committer. [1]
>>>
>>> Thank you Anand! And we are looking to see more of your
>>> contributions!
>>>
>>> Cham, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>> https://beam.apache.org/contribute/become-a-committer
>>> /#an-apache-beam-committer
>>>
>>

Re: [VOTE] Vendored Dependencies Release

[Yi Hu via dev]

Update: the beam-vendor-grpc-1_54_0 artifact is now released and the
development branch has been switched to use it [1].

[1] https://github.com/apache/beam/pull/26371

Thanks everyone!

On Wed, Apr 19, 2023 at 2:57 PM Yi Hu  wrote:

> I'm happy to announce that we have unanimously approved this release.
>
> There are 3 approving votes, 3 of which are binding:
> * Kenneth Knowles
> * Chamikara Jayalath
> * Robert Bradshaw
>
> There are no disapproving votes.
>
> Thanks everyone!
>
> On Mon, Apr 17, 2023 at 6:11 PM Robert Bradshaw 
> wrote:
>
>> +1
>>
>> On Mon, Apr 17, 2023 at 11:20 AM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>>> +1
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Mon, Apr 17, 2023 at 11:04 AM Kenneth Knowles 
>>> wrote:
>>>
>>>> +1
>>>>
>>>> On Fri, Apr 14, 2023 at 1:30 PM Yi Hu via dev 
>>>> wrote:
>>>>
>>>>> Please review the release of the following artifacts that we vendor:
>>>>>
>>>>>  * beam-vendor-grpc-1_54_0
>>>>>
>>>>>
>>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Please review and vote on the release candidate #1 for the version
>>>>> 0.1, as follows:
>>>>>
>>>>> [ ] +1, Approve the release
>>>>>
>>>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>>>
>>>>>
>>>>> The complete staging area is available for your review, which includes:
>>>>>
>>>>> * the official Apache source release to be deployed to dist.apache.org
>>>>> [1], which is signed with the key with fingerprint
>>>>> 2011EC936303D9A1DB662EE1CB6974C8170405CB [2],
>>>>>
>>>>> * all artifacts to be deployed to the Maven Central Repository [3],
>>>>>
>>>>> * commit hash "a38d9b94a738e4c488e7339ae3710fd5e1dc119e" [4],
>>>>>
>>>>> The vote will be open for at least 72 hours. It is adopted by majority
>>>>> approval, with at least 3 PMC affirmative votes.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Release Manager
>>>>>
>>>>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
>>>>>
>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>>
>>>>> [3]
>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1308/
>>>>>
>>>>> [4]
>>>>> https://github.com/apache/beam/commit/a38d9b94a738e4c488e7339ae3710fd5e1dc119e
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Yi Hu, (he/him/his)
>>>>>
>>>>> Software Engineer
>>>>>
>>>>> 919-641-8436 <(919)%20641-8436>
>>>>>
>>>>>

Re: [VOTE] Vendored Dependencies Release

[Yi Hu via dev]

I'm happy to announce that we have unanimously approved this release.

There are 3 approving votes, 3 of which are binding:
* Kenneth Knowles
* Chamikara Jayalath
* Robert Bradshaw

There are no disapproving votes.

Thanks everyone!

On Mon, Apr 17, 2023 at 6:11 PM Robert Bradshaw  wrote:

> +1
>
> On Mon, Apr 17, 2023 at 11:20 AM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
>
>> +1
>>
>> Thanks,
>> Cham
>>
>> On Mon, Apr 17, 2023 at 11:04 AM Kenneth Knowles  wrote:
>>
>>> +1
>>>
>>> On Fri, Apr 14, 2023 at 1:30 PM Yi Hu via dev 
>>> wrote:
>>>
>>>> Please review the release of the following artifacts that we vendor:
>>>>
>>>>  * beam-vendor-grpc-1_54_0
>>>>
>>>>
>>>>
>>>> Hi everyone,
>>>>
>>>> Please review and vote on the release candidate #1 for the version 0.1,
>>>> as follows:
>>>>
>>>> [ ] +1, Approve the release
>>>>
>>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>>
>>>>
>>>> The complete staging area is available for your review, which includes:
>>>>
>>>> * the official Apache source release to be deployed to dist.apache.org
>>>> [1], which is signed with the key with fingerprint
>>>> 2011EC936303D9A1DB662EE1CB6974C8170405CB [2],
>>>>
>>>> * all artifacts to be deployed to the Maven Central Repository [3],
>>>>
>>>> * commit hash "a38d9b94a738e4c488e7339ae3710fd5e1dc119e" [4],
>>>>
>>>> The vote will be open for at least 72 hours. It is adopted by majority
>>>> approval, with at least 3 PMC affirmative votes.
>>>>
>>>> Thanks,
>>>>
>>>> Release Manager
>>>>
>>>> [1] https://dist.apache.org/repos/dist/dev/beam/vendor/
>>>>
>>>> [2] https://dist.apache.org/repos/dist/dev/beam/KEYS
>>>>
>>>> [3]
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1308/
>>>>
>>>> [4]
>>>> https://github.com/apache/beam/commit/a38d9b94a738e4c488e7339ae3710fd5e1dc119e
>>>>
>>>>
>>>> --
>>>>
>>>> Yi Hu, (he/him/his)
>>>>
>>>> Software Engineer
>>>>
>>>> 919-641-8436 <(919)%20641-8436>
>>>>
>>>>

[VOTE] Vendored Dependencies Release

[Yi Hu via dev]

Please review the release of the following artifacts that we vendor:

 * beam-vendor-grpc-1_54_0



Hi everyone,

Please review and vote on the release candidate #1 for the version 0.1, as
follows:

[ ] +1, Approve the release

[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:

* the official Apache source release to be deployed to dist.apache.org [1],
which is signed with the key with fingerprint
2011EC936303D9A1DB662EE1CB6974C8170405CB [2],

* all artifacts to be deployed to the Maven Central Repository [3],

* commit hash "a38d9b94a738e4c488e7339ae3710fd5e1dc119e" [4],

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,

Release Manager

[1] https://dist.apache.org/repos/dist/dev/beam/vendor/

[2] https://dist.apache.org/repos/dist/dev/beam/KEYS

[3] https://repository.apache.org/content/repositories/orgapachebeam-1308/

[4]
https://github.com/apache/beam/commit/a38d9b94a738e4c488e7339ae3710fd5e1dc119e


-- 

Yi Hu, (he/him/his)

Software Engineer

919-641-8436

Re: [Python SDK] Use pre-released dependencies for Beam python unit testing

[Yi Hu via dev]

Sounds good, thanks!

Best,
Yi

On Wed, Apr 12, 2023 at 2:20 PM Anand Inguva  wrote:

> @Yi Hu  I think adding them to Jenkins or github
> actions is okay with me. With Github actions, since we don't use self
> hosted runners yet, I worry that action workers might get queued up.
>
> Also, I plan to not run these on every commit but run it as a cron
> job(maybe once per day) and also as trigger phrases and only on the lowest
> and highest python version. Also, migrating this workflow to jenkins would
> be trivial in the future once beam starts the migration. For now, I think
> it might be best to run on jenkins.
>
> On Wed, Apr 12, 2023 at 1:32 PM Valentyn Tymofieiev 
> wrote:
>
>> I think case in point dependency that would benefit from this testing is
>> grpcio, which includes pre-releases, and broke us and multiple of it's
>> released versions were yanked. https://pypi.org/project/grpcio/#history .
>>
>> We can look at how grpcio affected Beam previously. Couple of issues:
>>
>> - https://github.com/grpc/grpc/issues/30446 -- affected XLang tests
>> - https://github.com/apache/beam/issues/23734 -- affected MacOS suites
>> - https://github.com/apache/beam/issues/22159 -- (not detected by us,
>> but potentially could have affected a performance test).
>>
>> I'm afraid a dedicated suite may not give us desired test coverage to
>> catch regression at RC stage.
>>
>> On Wed, Apr 12, 2023 at 10:19 AM Yi Hu via dev 
>> wrote:
>>
>>> Thanks Anand,
>>>
>>> This would be very helpful to avoid experiencing multiple time (
>>> https://s.apache.org/beam-python-dependencies-pm). One thing to note is
>>> that Beam Jenkins CI is experiencing many issues recently, mostly due to
>>> that multiple Jenkins plugins does not scale (draining GitHub API call
>>> limit; disk usage, etc) so more PreCommit may add more pressures to Jenkins
>>> if going ahead with Option 1. As we have started GitHub Action migration,
>>> is it considered to add these new tests to GitHub Action?
>>>
>>> Best,
>>> Yi
>>>
>>> On Wed, Apr 12, 2023 at 10:46 AM Danny McCormick via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> Thanks for doing this Anand, I'm +1 on option 1 as well - I think
>>>> having the clear signal of the normal suite succeeding and the prerelease
>>>> one failing would be helpful and there shouldn't be too much additional
>>>> code necessary. That makes it really easy to treat the prerelease suite as
>>>> a (at least temporary) signal on needing upper bounds on our dependencies.
>>>>
>>>> Thanks,
>>>> Danny
>>>>
>>>> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> For Apache Beam Python we are considering using pre-released
>>>>> dependencies for unit testing by using the --pre flag to install
>>>>> pre-released dependencies of packages.
>>>>>
>>>>> We believe that using pre-released dependencies may help us to
>>>>> identify and resolve bugs more quickly, and to take advantage of new
>>>>> features or bug fixes that are not yet available in stable releases.
>>>>> However, we also understand that using pre-released dependencies may
>>>>> introduce new risks and challenges, including potential code duplication
>>>>> and stability issues.
>>>>>
>>>>> Before proceeding, we wanted to get your feedback on this approach.
>>>>>
>>>>> 1. Create a new PreCommit test suite and a PostCommit test suite that
>>>>> runs tests by installing pre-released dependencies.
>>>>>
>>>>> Pros:
>>>>>
>>>>>- stable and pre-released test suites are separate and it will be
>>>>>easier to debug if the pre-released test suite fails.
>>>>>
>>>>> Cons:
>>>>>
>>>>>- More test infra code to maintain. More tests to monitor.
>>>>>
>>>>>
>>>>> 2. Make use of the current PreCommit and PostCommit test suite and
>>>>> modify it so that it installs pre-released dependencies.
>>>>>
>>>>> Pros:
>>>>>
>>>>>- Less infra code and less tests to monitor.
>>>>>
>>>>> Cons:
>>>>>
>>>>>- Leads to noisy test signals if the pre-release candidate is
>>>>>unstable.
>>>>>
>>>>> I am in favor of approach 1 since this approach would ensure that any
>>>>> issues encountered during pre-release testing do not impact the stable
>>>>> release environment, and vice versa.
>>>>>
>>>>> If you have experience or done any testing work using pre-released
>>>>> dependencies, please let me know if you took any different approaches. It
>>>>> will be really helpful.
>>>>>
>>>>> Thanks,
>>>>> Anand
>>>>>
>>>>

Re: [Python SDK] Use pre-released dependencies for Beam python unit testing

[Yi Hu via dev]

Thanks Anand,

This would be very helpful to avoid experiencing multiple time (
https://s.apache.org/beam-python-dependencies-pm). One thing to note is
that Beam Jenkins CI is experiencing many issues recently, mostly due to
that multiple Jenkins plugins does not scale (draining GitHub API call
limit; disk usage, etc) so more PreCommit may add more pressures to Jenkins
if going ahead with Option 1. As we have started GitHub Action migration,
is it considered to add these new tests to GitHub Action?

Best,
Yi

On Wed, Apr 12, 2023 at 10:46 AM Danny McCormick via dev <
dev@beam.apache.org> wrote:

> Thanks for doing this Anand, I'm +1 on option 1 as well - I think having
> the clear signal of the normal suite succeeding and the prerelease one
> failing would be helpful and there shouldn't be too much additional code
> necessary. That makes it really easy to treat the prerelease suite as a (at
> least temporary) signal on needing upper bounds on our dependencies.
>
> Thanks,
> Danny
>
> On Wed, Apr 12, 2023 at 12:36 AM Anand Inguva via dev 
> wrote:
>
>> Hi all,
>>
>> For Apache Beam Python we are considering using pre-released dependencies
>> for unit testing by using the --pre flag to install pre-released
>> dependencies of packages.
>>
>> We believe that using pre-released dependencies may help us to identify
>> and resolve bugs more quickly, and to take advantage of new features or bug
>> fixes that are not yet available in stable releases. However, we also
>> understand that using pre-released dependencies may introduce new risks and
>> challenges, including potential code duplication and stability issues.
>>
>> Before proceeding, we wanted to get your feedback on this approach.
>>
>> 1. Create a new PreCommit test suite and a PostCommit test suite that
>> runs tests by installing pre-released dependencies.
>>
>> Pros:
>>
>>- stable and pre-released test suites are separate and it will be
>>easier to debug if the pre-released test suite fails.
>>
>> Cons:
>>
>>- More test infra code to maintain. More tests to monitor.
>>
>>
>> 2. Make use of the current PreCommit and PostCommit test suite and modify
>> it so that it installs pre-released dependencies.
>>
>> Pros:
>>
>>- Less infra code and less tests to monitor.
>>
>> Cons:
>>
>>- Leads to noisy test signals if the pre-release candidate is
>>unstable.
>>
>> I am in favor of approach 1 since this approach would ensure that any
>> issues encountered during pre-release testing do not impact the stable
>> release environment, and vice versa.
>>
>> If you have experience or done any testing work using pre-released
>> dependencies, please let me know if you took any different approaches. It
>> will be really helpful.
>>
>> Thanks,
>> Anand
>>
>

Re: Jenkins Flakes

[Yi Hu via dev]

+1 Thanks Danny for figuring out a solution.

Best,
Yi

On Tue, Apr 11, 2023 at 10:56 AM Svetak Sundhar via dev 
wrote:

> +1 to the proposal.
>
> Regarding the "(and not guaranteed to work)" part, is the resolution that
> the memory issues may still persist and we restore the normal retention
> limit (and we look for another fix), or that we never restore back to the
> normal retention limit?
>
>
> Svetak Sundhar
>
>   Technical Solutions Engineer, Data
> s vetaksund...@google.com
>
>
>
> On Tue, Apr 11, 2023 at 10:34 AM Jack McCluskey via dev <
> dev@beam.apache.org> wrote:
>
>> +1 for getting Jenkins back into a happier state, getting release
>> blockers resolved ahead of building an RC has been severely hindered by
>> Jenkins not picking up tests or running them properly.
>>
>> On Tue, Apr 11, 2023 at 10:24 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> *;tldr - I want to temporarily reduce the number of builds that we
>>> retain to reduce pressure on Jenkins*
>>>
>>> Hey everyone, over the past few days our Jenkins runs have been
>>> particularly flaky across the board, with errors like the following showing
>>> up all over the place [1]:
>>>
>>> java.nio.file.FileSystemException: 
>>> /home/jenkins/jenkins-home/jobs/beam_PreCommit_Python_Phrase/builds/3352/changelog.xml:
>>>  No space left on device [2]
>>>
>>>
>>> These errors indicate that we're out of space on the Jenkins master
>>> node. After some digging (thanks @Yi Hu  @Ahmet Altay
>>>  and @Bruno Volpato  for
>>> contributing), we've determined that at least one large contributing issue
>>> is that some of our builds are eating up too much space. For example, our
>>> beam_PreCommit_Java_Commit build is taking up 28GB of space by itself (this
>>> is just one example).
>>>
>>> @Yi Hu  found one change around code coverage that is
>>> likely heavily contributing to the problem and rolled that back [3]. We can
>>> continue to find other contributing factors here.
>>>
>>> In the meantime, to get us back to healthy *I propose that we reduce
>>> the number of builds that we are retaining to 40 for all jobs that are
>>> using a large amount of storage (>5GB)*. This will hopefully allow us
>>> to return Jenkins to a normal functioning state, though it will do so at
>>> the cost of a significant amount of build history (right now, for example,
>>> beam_PreCommit_Java_Commit is at 400 retained builds). We could restore the
>>> normal retention limit once the underlying problem is resolved. Given that
>>> this is irreversible (and not guaranteed to work), I wanted to gather
>>> feedback before doing this. Personally, I rarely use builds that old, but
>>> others may feel differently.
>>>
>>> Please let me know if you have any objections or support for this
>>> proposal.
>>>
>>> Thanks,
>>> Danny
>>>
>>> [1] Tracking issue: https://github.com/apache/beam/issues/26197
>>> [2] Example run with this error:
>>> https://ci-beam.apache.org/job/beam_PreCommit_Python_Phrase/3352/console
>>> [3] Rollback PR: https://github.com/apache/beam/pull/26199
>>>
>>

[PROPOSAL] Upgrade vendor grpc

[Yi Hu via dev]

Hi all,

I would like to volunteer to upgrade the Beam vendored grpc, as
requested by the GitHub Issue [1]. I checked the project history that we
did four upgrades in the last 2 years (1.26->1.36->1.43->1.48) and the last
time was in Aug 2022 [2]. There have been vulnerabilities in its
dependencies found since then (see [1]).

My plan is to follow the release process [3, 4], which involves preparing
for the release, building a candidate, voting and finalizing the release.
Then the vendored artifact is targeted to be integrated by Beam v2.48.0
onwards (cut date May 17, 2023).

Please let me know if you have any comments/objections/questions.

Thanks,

Yi

[1] https://github.com/apache/beam/issues/25746
[2] https://github.com/apache/beam/pull/22628
[3] https://github.com/apache/beam/tree/master/vendor
[4]
https://docs.google.com/document/d/1ztEoyGkqq9ie5riQxRtMuBu3vb6BUO91mSMn1PU0pDA/edit#heading=h.vhcuqlttpnog

-- 

Yi Hu, (he/him/his)

Software Engineer

Re: Broken Jenkins jobs

[Yi Hu via dev]

Hi Alexy,

Thanks for raising this. The breaking change is found:
https://github.com/apache/beam/pull/25566

Best,
Yi

On Mon, Feb 20, 2023 at 12:44 PM Alexey Romanenko 
wrote:

> Hi all,
>
> Jenkins jobs “beam_PreCommit_SQL_Java11_Commit” [1] and
> “beam_PreCommit_SQL_Java17_Commit” [2] seems are broken since Feb 17th.
>
> Anyone is looking into this ?
>
> —
> Alexey
>
> [1] https://ci-beam.apache.org/job/beam_PreCommit_SQL_Java11_Commit/
> [2] https://ci-beam.apache.org/job/beam_PreCommit_SQL_Java17_Commit/

Re: Performance and Cost benchmarking

[Yi Hu via dev]

Hi everyone,

Thanks for your attention. Since this last thread there are works
implementing the utilities for the performance and cost benchmarking
framework ongoing, currently under the DataflowTemplate repository (
https://github.com/GoogleCloudPlatform/DataflowTemplates). In order to use
these utilities for IO performance tests hosted in Beam repo, we plan to
migrate these utilities to Beam repository.

I have attached a link [1] describes the migration plan. Please feel free
to comment on the doc with any questions, suggestions, and of course any
concerns.

Best,
Yi


 template-it-to-beam




On Mon, Sep 26, 2022 at 12:07 PM Pranav Bhandari <
bhandari.prana...@gmail.com> wrote:

> Hello,
>
> Hope this email finds you well. I have attached a link to a doc which
> discusses the design for a performance and cost benchmarking framework to
> be used by Beam IOs and Google-provided dataflow templates.
>
> Please feel free to comment on the doc with any questions, concerns or
> ideas you might have.
>
> Thank you,
> Pranav Bhandari
>
>
>
> https://docs.google.com/document/d/14GatBilwuR4jJGb-ZNpYeuB-KkVmDvEm/edit?usp=sharing=102139643796739130048=true=true
>

Re: Refactor Kubernetes Kafka External load balancer dependency in tests

[Yi Hu via dev]

Hi Damon,

Thanks for the proposal! Our k8s infrastructure has long been under
maintained. I agree Public IP exposure is not necessary and it has
triggered security alerts quite often... Would like to help if needed.

Best,
Yi

On Mon, Jan 23, 2023 at 1:25 PM Damon Douglas via dev 
wrote:

> Hello Everyone,
>
> I would like to share with the community a proposal [1] to refactor a
> Kubernetes Kafka External Load Balancer dependency in tests.  It fixes
> [2].  The referenced document summarizes the situation, background,
> assessment, and recommendation.
>
> Best,
>
> Damon
>
> *References*
>
> 1.[Public facing][Beam][issue/25119] Refactor Kubernetes Kafka External
> load balancer dependency in tests
> 
> 2. https://github.com/apache/beam/issues/25119
>

Re: Credentials Rotation Failure on IO-Datastores cluster

[Yi Hu via dev]

Thanks for reporting. I have bumped the pool size of io-datastore as we
have more tests being added and the default-pool frequently becomes
unschedulable due to memory constraints. A simple fix is just rename the
'pool1' back to 'default-pool'.

On Thu, Dec 1, 2022 at 1:26 PM Danny McCormick 
wrote:

> Yes, I was just starting to look into this. Looks like this is the result
> of this job failing -
> https://github.com/apache/beam/blob/ec2a07b38c1f640c62e7c3b96966f18b334a7ce9/.test-infra/jenkins/job_IODatastoresCredentialsRotation.groovy#L49
>
> The error is:
>
> ```
>
> *21:25:58* + gcloud container clusters upgrade io-datastores 
> --node-pool=default-pool --zone=us-central1-a --quiet*21:25:59* ERROR: 
> (gcloud.container.clusters.upgrade) No node pool found matching the name 
> [default-pool].
>
> ```
>
>
> from 
> https://ci-beam.apache.org/job/Rotate%20IO-Datastores%20Cluster%20Credentials/6/console
>
>
> It looks like there's been some change to the cluster that is causing the
> job to fail. If we don't fix this and rerun, the cluster's creds will
> expire (probably in like a monthish). I'm not sure what the impact of that
> would be, I think probably broken IO integration tests.
>
> @John Casey  or @Yi Hu  might
> know more about this, I think the cluster in question is
> https://pantheon.corp.google.com/kubernetes/clusters/details/us-central1-a/io-datastores/details?mods=dataflow_dev=apache-beam-testing
>
> Next steps are:
> 1) figuring out why there's no longer a default-pool
> 2) Either recreating it or modifying the cred rotation logic
> 3) (Minor) Fixing the url in the Jenkins job so it actually points to the
> failing job when we get emails like this
>
> On Thu, Dec 1, 2022 at 1:18 PM Byron Ellis via dev 
> wrote:
>
>> Is there something we need to do here?
>>
>> On Thu, Dec 1, 2022 at 10:10 AM Apache Jenkins Server <
>> jenk...@builds.apache.org> wrote:
>>
>>> Something went wrong during the automatic credentials rotation for
>>> IO-Datastores Cluster, performed at Thu Dec 01 15:00:47 UTC 2022. It may be
>>> necessary to check the state of the cluster certificates. For further
>>> details refer to the following links:
>>>  * https://ci-beam.apache.org/job/beam_SeedJob_Standalone/
>>>  * https://ci-beam.apache.org/.
>>
>>

Re: [ANNOUNCE] New committer: Yi Hu

[Yi Hu via dev]

Thank you for the help of you all over the time, and I am glad to
contribute and help with the community.

Best,
Yi

On Thu, Nov 10, 2022 at 11:29 AM Alexey Romanenko 
wrote:

> Congratulations! Well deserved!
>
> —
> Alexey
>
> On 9 Nov 2022, at 21:01, Tomo Suzuki via dev  wrote:
>
> Congratulations!
>
> On Wed, Nov 9, 2022 at 3:00 PM John Casey via dev 
> wrote:
>
>> Congrats! this is well deserved YI
>>
>> On Wed, Nov 9, 2022 at 2:58 PM Austin Bennett <
>> whatwouldausti...@gmail.com> wrote:
>>
>>> Congrats, and Thanks, Yi!
>>>
>>> On Wed, Nov 9, 2022 at 11:24 AM Valentyn Tymofieiev via dev <
>>> dev@beam.apache.org> wrote:
>>>
 I am with the Beam PMC on this, congratulations and very well deserved,
 Yi!

 On Wed, Nov 9, 2022 at 11:08 AM Byron Ellis via dev <
 dev@beam.apache.org> wrote:

> Congratulations!
>
> On Wed, Nov 9, 2022 at 11:00 AM Pablo Estrada via dev <
> dev@beam.apache.org> wrote:
>
>> +1 thanks Yi : D
>>
>> On Wed, Nov 9, 2022 at 10:47 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congrats Yi! I've really appreciated the ways you've consistently
>>> taken responsibility for improving our team's infra and working through
>>> sharp edges in the codebase that others have ignored. This is definitely
>>> well deserved!
>>>
>>> Thanks,
>>> Danny
>>>
>>> On Wed, Nov 9, 2022 at 1:37 PM Anand Inguva via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congratulations Yi!

 On Wed, Nov 9, 2022 at 1:35 PM Ritesh Ghorse via dev <
 dev@beam.apache.org> wrote:

> Congratulations Yi!
>
> On Wed, Nov 9, 2022 at 1:34 PM Ahmed Abualsaud via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Yi!
>>
>> On Wed, Nov 9, 2022 at 1:33 PM Sachin Agarwal via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congratulations Yi!
>>>
>>> On Wed, Nov 9, 2022 at 10:32 AM Kenneth Knowles 
>>> wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Yi Hu (y...@apache.org)

 Yi started contributing to Beam in early 2022. Yi's
 contributions are very diverse! I/Os, performance tests, Jenkins, 
 support
 for Schema logical types. Not only code but a very large amount of 
 code
 review. Yi is also noted for picking up smaller issues that 
 normally would
 be left on the backburner and filing issues that he finds rather 
 than
 ignoring them.

 Considering their contributions to the project over this
 timeframe, the Beam PMC trusts Yi with the responsibilities of a 
 Beam
 committer. [1]

 Thank you Yi! And we are looking to see more of your
 contributions!

 Kenn, on behalf of the Apache Beam PMC

 [1]

 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

>>>
>
> --
> Regards,
> Tomo
>
>
>

Re: [ANNOUNCE] New committer: Ritesh Ghorse

[Yi Hu via dev]

Congratulations Ritesh!

On Thu, Nov 3, 2022 at 7:23 PM Byron Ellis via dev 
wrote:

> Congratulations!
>
> On Thu, Nov 3, 2022 at 4:21 PM Austin Bennett 
> wrote:
>
>> Congratulations, and Thanks @riteshgho...@apache.org!
>>
>> On Thu, Nov 3, 2022 at 4:17 PM Sachin Agarwal via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congrats Ritesh!
>>>
>>> On Thu, Nov 3, 2022 at 4:16 PM Kenneth Knowles  wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Ritesh Ghorse (riteshgho...@apache.org)

 Ritesh started contributing to Beam in mid-2021 and has contributed
 immensely to bringin the Go SDK to fruition, in addition to contributions
 to Java and Python and release validation.

 Considering their contributions to the project over this timeframe, the
 Beam PMC trusts Ritesh with the responsibilities of a Beam committer. [1]

 Thank you Ritesh! And we are looking to see more of your contributions!

 Kenn, on behalf of the Apache Beam PMC

 [1]

 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

>>>

Re: [ACTION REQUESTED] Triage 3-10 issues (get the labels right, basically)

[Yi Hu via dev]

up to 120, my bad

On Mon, Oct 31, 2022 at 4:29 PM Yi Hu  wrote:

> Thanks for raising this. Also closed a couple of issues. Now the number is
> up to 12.
>
> On Mon, Oct 31, 2022 at 9:24 AM Manu Zhang 
> wrote:
>
>> I closed several issues that have already been fixed. It looks we didn’t
>> reference the issue number in commit message which can auto-close the issue.
>>
>> Kenneth Knowles 于2022年10月31日 周一20:01写道：
>>
>>> Hi all
>>>
>>> We have built up to 138 GitHub Issues with the "awaiting triage" label.
>>> If you can, take just a couple minutes today to grab a couple and triage
>>> them:
>>>
>>>- get the right priority label (
>>>https://beam.apache.org/contribute/issue-priorities/)
>>>- get the right component label (runner, IO, SDK, etc)
>>>- ping someone who may be interested
>>>- remember to remove the "awaiting triage" label
>>>
>>> It really is pretty quick!
>>>
>>> Kenn
>>>
>>

Re: [ACTION REQUESTED] Triage 3-10 issues (get the labels right, basically)

[Yi Hu via dev]

Thanks for raising this. Also closed a couple of issues. Now the number is
up to 12.

On Mon, Oct 31, 2022 at 9:24 AM Manu Zhang  wrote:

> I closed several issues that have already been fixed. It looks we didn’t
> reference the issue number in commit message which can auto-close the issue.
>
> Kenneth Knowles 于2022年10月31日 周一20:01写道：
>
>> Hi all
>>
>> We have built up to 138 GitHub Issues with the "awaiting triage" label.
>> If you can, take just a couple minutes today to grab a couple and triage
>> them:
>>
>>- get the right priority label (
>>https://beam.apache.org/contribute/issue-priorities/)
>>- get the right component label (runner, IO, SDK, etc)
>>- ping someone who may be interested
>>- remember to remove the "awaiting triage" label
>>
>> It really is pretty quick!
>>
>> Kenn
>>
>

Re: Inclusive terminology: "Sickbay" ==> "Disabled test"

[Yi Hu via dev]

(From someone who received education not in English and from another
field). I first assumed this is some terminology just like "flaky", "smock
test" because I came from another world and did not know most of the terms
in this field. That said, I just assumed it was another term used in the
field and just acknowledged that.

Just note that these "sickbayed" tests are not disabled from running. There
are disabled test suites on https://ci-beam.apache.org/ having gray " "
 signs. Those tests are still running on schedule but we "expect" they fail
/ known to fail. If we really want to change. Maybe we could use "broken
test", "unstable test" or something like that to make distinction to the
disabled test suites we have.

Best,
Yi

On Mon, Oct 17, 2022 at 3:26 PM Danny McCormick via dev 
wrote:

> I'm +1 on this, sickbay was a new term for me when I joined the project.
> One thing I will note: we still have plenty of sickbay references in our
> code itself - https://github.com/apache/beam/search?p=1=sickbay - if we
> decide to take this forward we should create an issue to remove those (with
> the "good first issue" label).
>
> > By the way, is there any known reason not to have spaces in GitHub
> Issues tags?
>
> Generally, no. GitHub gives you a label with spaces by default when you
> create a new repo ("good first issue") and we already have multiple labels
> with spaces ("awaiting triage", "good first issue", there are probably
> more).
>
> I personally slightly prefer dashes because spaces make queries
> 
> a little less clean since you need to quote the label (e.g. `is:open
> is:issue *label:"disabled test"*` instead of is:open is:issue
> *label:disabled-test*), but that is not a widely accepted standard.
>
> On Mon, Oct 17, 2022 at 3:10 PM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> I have gotten a lot of questions from people like "what is sickbay?"
>>
>> Because I am a Star Trek enthusiast I easily understood that if I
>> "sickbay the test" means to disable it temporarily. And people on my team
>> are used to this terminology. But this is not all people :-) and there are
>> many name conflicts with products too.
>>
>> So I have edited the GitHub Issues tag "sickbay" to be "disabled test"
>> and I suggest we use this term everywhere.
>>
>> By the way, is there any known reason not to have spaces in GitHub Issues
>> tags?
>>
>> Kenn
>>
>

Re: Performance and Cost benchmarking

[Yi Hu via dev]

Hi Andrew and Pranav,

Thanks for pointing out the current infrastructure we have in Beam. I have
investigated the current Performance tests of Beam IOs in the Beam repo and
summarized the current tools and infrastructure we have in this document:
https://docs.google.com/document/d/11CgNVtyZSipoRiJ2O57hhqShUw_FQDTj6rEOzudwsK4/edit#
. I also put some brief designs about how can we incorporate them into the
test framework proposed by Pranav.

Best,
Yi

On Tue, Sep 27, 2022 at 12:06 PM Alexey Romanenko 
wrote:

> Thanks for raising this topic.
>
> > On 26 Sep 2022, at 23:32, Andrew Pilloud via dev 
> wrote:
> >
> > I left some comments on your design. Your doc discusses a bunch of
> > details about infrastructure such as testing frameworks, automation,
> > and performance databases, but doesn't describe how it will fit in
> > with our existing infrastructure (Load Tests, Nexmark, Jenkins,
> > InfluxDB, Grafina). I would suspect we actually have most of the
> > infrastructure already built?
>
> Right, I’m second on this question. We already have an infrastructure
> ready to run a bunch of different benchmarks/load tests and
> collect/present/analyse the results. Of course, there is a field for
> improvements, but it would be great to take this into account and add the
> details how this benchmark can be integrated into (to avoid a double job
> for further support).
>
>
> —
> Alexey
>
> > On Mon, Sep 26, 2022 at 9:07 AM Pranav Bhandari
> >  wrote:
> >>
> >> Hello,
> >>
> >> Hope this email finds you well. I have attached a link to a doc which
> discusses the design for a performance and cost benchmarking framework to
> be used by Beam IOs and Google-provided dataflow templates.
> >>
> >> Please feel free to comment on the doc with any questions, concerns or
> ideas you might have.
> >>
> >> Thank you,
> >> Pranav Bhandari
> >>
> >>
> >>
> https://docs.google.com/document/d/14GatBilwuR4jJGb-ZNpYeuB-KkVmDvEm/edit?usp=sharing=102139643796739130048=true=true
>
>

Re: Representation of logical type beam:logical_type:datetime:v1

[Yi Hu via dev]

Hi,

It now appears that if we want a clean solution then we have to add a fixed
size primitive type to Beam atomic types. Or, we then have a millis_instant
logical type that does not have to_language_type and to_representation_type
implementations. Any suggestions are welcome!

Best,
Yi

On Thu, Aug 18, 2022 at 10:07 AM Yi Hu  wrote:

>
>
> On Wed, Aug 17, 2022 at 5:14 PM Chamikara Jayalath 
> wrote:
>
>>
>> I think this is fine (even though it would add a small perf hit to
>> JdbcIO.Read). We also probably should make this conversion a utility method
>> that can be used elsewhere when we need to encode datetime fields.
>> We should also document that "beam:logical_type:datetime:v1" is not
>> portable (till we fix the incompatibility).
>>
>>
> +1 for utility method and documentation.
> If we were to change JDBC instead of make  millis_instant compatible to
> InstantCoder, this would only fix JDBC cross-language timestamps. I expect
> for other IO connectors this is still a problem and that is why I would
> like to take a generic approach. In general, inside each sdk we would like
> to follow the language specific convention of that sdk. I remember a
> related  discussion about the timestamp types:
> https://github.com/apache/beam/pull/17380#discussion_r852422314 which
> reached a conclusion that follows the language convention on timestamp
> values, e.g. use milli precision (long backed) Instant in Java; micro
> precision (float backed) timestamp in python.
>
> Best,
> Yi
>

Re: Representation of logical type beam:logical_type:datetime:v1

[Yi Hu via dev]

On Wed, Aug 17, 2022 at 5:14 PM Chamikara Jayalath 
wrote:

>
> I think this is fine (even though it would add a small perf hit to
> JdbcIO.Read). We also probably should make this conversion a utility method
> that can be used elsewhere when we need to encode datetime fields.
> We should also document that "beam:logical_type:datetime:v1" is not
> portable (till we fix the incompatibility).
>
>
+1 for utility method and documentation.
If we were to change JDBC instead of make  millis_instant compatible to
InstantCoder, this would only fix JDBC cross-language timestamps. I expect
for other IO connectors this is still a problem and that is why I would
like to take a generic approach. In general, inside each sdk we would like
to follow the language specific convention of that sdk. I remember a
related  discussion about the timestamp types:
https://github.com/apache/beam/pull/17380#discussion_r852422314 which
reached a conclusion that follows the language convention on timestamp
values, e.g. use milli precision (long backed) Instant in Java; micro
precision (float backed) timestamp in python.

Best,
Yi

Re: Representation of logical type beam:logical_type:datetime:v1

[Yi Hu via dev]

Hi Brian,

Currently Java's JDBCIO does not do special cross-language things. A
DATETIME field type appears because the Row contains an Instant object. And
the sdk will always encode the Instant object using InstantCoder. This is
done for both Java pipelines and cross-language pipelines. To use
millis_instant in JDBCIO and to avoid break Instant used elsewhere (like
watermarks) I will need to change the type of timestamp returned by JDBC
read from Instant to millis_instant.

We could make the name of the logical type to be
"beam:logical_type:millis_instant" which is backed by a big endian int 64.

Best,
Yi


On Fri, Aug 12, 2022 at 5:28 PM Brian Hulette  wrote:

> Ah sorry, I forgot that INT64 is encoded with VarIntCoder, so we can't
> simulate TimestampCoder with a logical type.
>
> I think the ideal end state would be to have a well-defined
> beam:logical_type:millis_instant that we use for cross-language (when
> appropriate), and never use DATETIME at cross-language boundaries. Would it
> be possible to add millis_instant, and use that for JDBC read/write instead
> of DATETIME?
>
> Separately we could consider how to resolve the conflicting definitions of
> beam:logical_type:datetime:v1. I'm not quite sure how/if we can do that
> without breaking pipeline update.
>
> Brian
>
>
> On Fri, Aug 12, 2022 at 7:50 AM Yi Hu via dev  wrote:
>
>> Hi Cham,
>>
>> Thanks for the comments.
>>
>>
>>>
>>>>
>>>> ii. "beam:logical_type:instant:v1" is still backed by INT64, but in
>>>> implementation it will use BigEndianLongCoder to encode/decode the stream.
>>>>
>>>>
>>> Is this to be compatible with the current Java implementation ? And we
>>> have to update other SDKs to use big endian coders when encoding/decoding
>>> the "beam:logical_type:instant:v1" logical type ?
>>>
>>>
>> Yes, and the proposal is aimed to keep the Java SDK change minimal; we
>> have to update other SDKs to make it work. Currently python and go sdk does
>> not implement "beam:logical_type:datetime:v1" (will
>> be "beam:logical_type:instant:v1") at all.
>>
>>
>>>
>>>
>>>> For the second step ii, the problem is that there is a primitive type
>>>> backed by a fixed length integer coder. Currently INT8, INT16, INT32,
>>>> INT64... are all backed by VarInt (and there is ongoing work to use fixed
>>>> size big endian to encode INT8, INT16 (
>>>> https://github.com/apache/beam/issues/19815)). Ideally I would think
>>>> (INT8, INT16, INT32, INT64) are all fixed and having a generic (INT)
>>>> primitive type is backed by VarInt. But this may be a more substantial
>>>> change for the current code base.
>>>>
>>>
>>> I'm a bit confused by this. Did you mean that there's *no* primitive
>>> type backed by a fixed length integer coder ? Also, by primitive, I'm
>>> assuming you mean Beam Schema types here.
>>>
>>>
>> Yes I mean Beam Schema types here. The proto for datetime(instant)
>> logical type is constructed here:
>> https://github.com/apache/beam/blob/cf9ea1f442636f781b9f449e953016bb39622781/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaTranslation.java#L202
>> It is represented by an INT64 atomic type. In cross-language case,
>> another SDK receives proto and decodes the stream according to the proto.
>> Currently I do not see an atomic type that will be decoded using a
>> fixed-length BigEndianLong coder. INT8, ..., INT64 will all be decoded with
>> VarInt.
>>
>> As a workaround in the PR (#22561), in python's RowCoder I explicitly set
>> the coder for "beam:logical_type:datetime:v1" (will
>> be "beam:logical_type:instant:v1") to be TimestampCoder. I do not find a
>> way to keep the logic contained in the logical type implementation, e.g. in
>> to_language_type and to_representation_type method. To do this I will need
>> an atomic type that is decoded using the BigEndianLong coder.
>> Please point out if I was wrong.
>>
>> Best,
>> Yi
>>
>

Re: Representation of logical type beam:logical_type:datetime:v1

[Yi Hu via dev]

Hi Cham,

Thanks for the comments.


>
>>
>> ii. "beam:logical_type:instant:v1" is still backed by INT64, but in
>> implementation it will use BigEndianLongCoder to encode/decode the stream.
>>
>>
> Is this to be compatible with the current Java implementation ? And we
> have to update other SDKs to use big endian coders when encoding/decoding
> the "beam:logical_type:instant:v1" logical type ?
>
>
Yes, and the proposal is aimed to keep the Java SDK change minimal; we have
to update other SDKs to make it work. Currently python and go sdk does not
implement "beam:logical_type:datetime:v1" (will
be "beam:logical_type:instant:v1") at all.


>
>
>> For the second step ii, the problem is that there is a primitive type
>> backed by a fixed length integer coder. Currently INT8, INT16, INT32,
>> INT64... are all backed by VarInt (and there is ongoing work to use fixed
>> size big endian to encode INT8, INT16 (
>> https://github.com/apache/beam/issues/19815)). Ideally I would think
>> (INT8, INT16, INT32, INT64) are all fixed and having a generic (INT)
>> primitive type is backed by VarInt. But this may be a more substantial
>> change for the current code base.
>>
>
> I'm a bit confused by this. Did you mean that there's *no* primitive type
> backed by a fixed length integer coder ? Also, by primitive, I'm assuming
> you mean Beam Schema types here.
>
>
Yes I mean Beam Schema types here. The proto for datetime(instant) logical
type is constructed here:
https://github.com/apache/beam/blob/cf9ea1f442636f781b9f449e953016bb39622781/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaTranslation.java#L202
It is represented by an INT64 atomic type. In cross-language case, another
SDK receives proto and decodes the stream according to the proto. Currently
I do not see an atomic type that will be decoded using a fixed-length
BigEndianLong coder. INT8, ..., INT64 will all be decoded with VarInt.

As a workaround in the PR (#22561), in python's RowCoder I explicitly set
the coder for "beam:logical_type:datetime:v1" (will
be "beam:logical_type:instant:v1") to be TimestampCoder. I do not find a
way to keep the logic contained in the logical type implementation, e.g. in
to_language_type and to_representation_type method. To do this I will need
an atomic type that is decoded using the BigEndianLong coder.
Please point out if I was wrong.

Best,
Yi

Representation of logical type beam:logical_type:datetime:v1

[Yi Hu via dev]

Hi,

tl;dr There are two (or three) different representations used for the URN
"beam:logical_type:datetime:v1" in Beam Java SDK. Clarification or clean up
is needed.

I recently try to resolve a long-time issue
https://github.com/apache/beam/issues/19817 for the datetime logical type
cross-language support. However I notice that currently in Java SDK this
URN is referred in two places:

(1)
https://github.com/apache/beam/blob/cf9ea1f442636f781b9f449e953016bb39622781/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/DateTime.java#L49
where it has a representation of Row {Date: INT64, Time: INT64}

(2)
https://github.com/apache/beam/blob/cf9ea1f442636f781b9f449e953016bb39622781/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaTranslation.java#L67
where it is represented by a single INT64

Moreover, there is a third, "actual" representation that is used when one
encodes a joda.DateTime or joda.Instant object in a Row, which is an int64
type encoded with fixed-size big endian. Note that this is different from
(2) because INT64 will be decoded using a VarInt coder which causes data
corruption when other sdk (e.g. python) pulls rows from java ptransform.

To resolve this I try to implement the "actual" representation of logical
type "beam:logical_type:datetime:v1" (
https://github.com/apache/beam/pull/22561 for contexts). It then becomes
necessary to resolve this inconsistency because we have reached a point of
adding a known logical type in schemas.proto. I am consider some solutions:

i. Rename the logical type of case (2) as "beam:logical_type:instant:v1" as
suggested by Brian in an earlier review.

ii. "beam:logical_type:instant:v1" is still backed by INT64, but in
implementation it will use BigEndianLongCoder to encode/decode the stream.

For the second step ii, the problem is that there is a primitive type
backed by a fixed length integer coder. Currently INT8, INT16, INT32,
INT64... are all backed by VarInt (and there is ongoing work to use fixed
size big endian to encode INT8, INT16 (
https://github.com/apache/beam/issues/19815)). Ideally I would think (INT8,
INT16, INT32, INT64) are all fixed and having a generic (INT) primitive
type is backed by VarInt. But this may be a more substantial change for the
current code base.

I would like to have opinions from the community. Thanks for your attention!

Regards,
Yi

-- 

Yi Hu, (he/him/his)

Software Engineer

Re: Update PR description template

[Yi Hu via dev]

Hi all,

I have drafted a document summarizing the ideas suggested in this thread.

https://docs.google.com/document/d/10FlXOo_hL2QYTPhwS8uHSyJbQCzwC3K3C12tFccANA8/edit#

where I extracted the following action items:

short-term (easy) items
- Split the linting task (e.g. java checkStyle) out from the test (java
percommit) step

mid-to-long-term items
- A single github action that indicates the PR is ready to merge
- Merging the PR through action

I am happy to volunteer to take on this task. Once we have decided the
direction(s) to proceed I am going to detail the design.

Regards,
Yi



On Tue, Jun 28, 2022 at 10:40 PM Ahmet Altay  wrote:

> I am worried about adding our own custom tooling. They require
> maintenance, they introduce new flakiness, and so far we have not been
> great about maintaining custom infra.
>
> On Tue, Jun 28, 2022 at 1:36 PM Kenneth Knowles  wrote:
>
>> Thanks for all the super useful info. I can add some experience from the
>> early days of Beam: Because the GitHub "merge" button did not exist, we
>> rolled our own "merge bot". The pros/cons below are not general to the
>> concept, but to our experience with it:
>>
>> PROs:
>>  - It single-threaded commits to the repo so there weren't race
>> conditions on test results. I think this will be OK for our scale for the
>> foreseeable future.
>>  - It re-ran tests after merge but before pushing to master, which is the
>> second half of eliminating that race condition.
>>
>> CONs:
>>  - It was very flaky and often just didn't stay up. We danced for joy
>> when it was gone.
>>  - It ran a kind of arbitrary set of tests that didn't match the PR
>> statuses. It did not have any filter on which tests were run during merge.
>> We were small enough that mostly just "run the tests" was specific enough.
>>  - It always squashed the commits and then pushed the squashed commit
>> with a comment "this closes #". This messes up PRs that have multiple
>> commits that should remain separate. But more importantly it made it
>> impossible to easily distinguish PRs that were merged versus those that
>> were closed without merge. And of course it is way harder to navigate
>> history when the commits on master have different hashes than the commits
>> that were authored.
>>  - Getting reasonable logs with error messages when a merge failed for
>> whatever reason was hard or impossible IIRC.
>>
>> So I think in what you have said there is an option to get the best of
>> all, something like:
>>
>>  - Do merges through an action. Merge commit or squash-and-merge would be
>> separate labels. Or not bother with squash and merge, instead using
>> heuristics to block PRs with bad commit histories.
>>  - Have the workflow check that all PR statuses are green before
>> continuing to merge.
>>
>
> This sounds like a reasonable process. I assume the only addition in this
> case would be another github action.
>
>
>>
>> Kenn
>>
>> On Tue, Jun 28, 2022 at 8:03 AM Danny McCormick <
>> dannymccorm...@google.com> wrote:
>>
>>> After looking into this a little bit more, I need to revise my opinion
>>> on how we would do this; I don't think it's practical to have all required
>>> status checks via the .asf.yml file because those required checks can't be
>>> filtered by path (for example, if we want to require Python precommit on
>>> Python PRs, it would need to be required on *all *PRs). That's a GitHub
>>> limitation
>>> ,
>>> not an ASF one.
>>>
>>> One option is to write an action that makes sure no checks are failing
>>> (except maybe codecov?) and put a single required check on that. That would
>>> also make it easy to build in logic to override required checks like Robert
>>> suggested ("specific wording would have to be in the last comment"). We
>>> already have logic in the PR bot that does some of this
>>> 
>>> .
>>>
>>> The downside to that approach is that it's not clear what the best way
>>> to trigger that workflow is since it has to run after all other checks have
>>> completed. We could have it trigger on some label (e.g. "ready to merge")
>>> and then automatically merge the PR when it's done or comment and remove
>>> the label if checks are failing/incomplete. This changes the workflow for
>>> committers from "click the merge button" to "add a label", but doesn't
>>> require significantly more action or oversight and is pretty similar to how
>>> Kubernetes  and
>>> some other large repos run things.
>>>
>>> Another option would be to trigger that check at the end of every
>>> Actions run and use the check_suite trigger for external runs
>>> (unfortunately actions doesn't trigger that). That 

Re: [Proposal] Path to Re-enabling Build Comment Triggers

[Yi Hu via dev]

Thanks for the investigation done and disabling the "admins verify patch"
comment. Wondering if there is a way to early return the bot comments for
ghprb (like blacklist)? Besides asf-ci there are also comments
from github-actions (e.g. assign reviewer or "stopping review
notifications"). In the future if we enable more github action items there
might be more bot pr comments.

Best,
Yi

On Fri, Jul 1, 2022 at 8:33 AM Danny McCormick via dev 
wrote:

> Given the early consensus here, I tried disabling the "Can one of the
> admins verify this patch?" messages, and verified that it worked. If you
> disagree with that decision, please let me know - I expect that is the most
> popular part of this proposal though :)
>
> The remaining questions are:
> 1) Are we comfortable adding trusted repeat contributors to the Jenkins
> allow-list?
> 2) Are we ok trying to enable build triggers?
>
> Given that disabling the "Can one of the admins verify this patch?"
> messages worked, I'm not sure that we need to do (1) - I think it would
> help reduce load from the plugin a little bit, but is probably not as
> necessary or helpful as what we've already done. I'm pretty comfortable
> skipping that step.
>
> *If there are no objections, I'd like to try reenabling build comment
> triggers next week.* Because I'll be offline for the first half of next
> week for the American 4th of July holiday, I will plan on trying it next
> Wednesday or Thursday if there are no objections.
>
> Thanks,
> Danny
>
> On Thu, Jun 30, 2022 at 2:54 PM Pablo Estrada  wrote:
>
>> Agreed! I never know what to do about them : )
>>
>> On Thu, Jun 30, 2022 at 10:58 AM Robert Burke  wrote:
>>
>>> +1 to get rid of the "admins very patch" comments if we can. They add no
>>> value at all when commented in quadruplicate immediately after PR creation.
>>>
>>> On Thu, Jun 30, 2022, 8:40 AM Danny McCormick 
>>> wrote:
>>>
 Hey everyone, I've been digging into the issues we've been having with
 Jenkins recently[1] which led to us disabling many of our build
 comment triggers (e.g. "Run " triggering a Jenkins job). Moving
 forward, I'd like to recommend that we:

 1) Try to disable the "Can one of the admins verify this patch?"
 comments via the Jenkins plugin configuration.
 2) Add trusted repeat contributors to the Jenkins allow-list.
 3) Try re-enabling all build triggers.

 *Justification*

 Right now, around 33% of our PR comments are "Can one of the admins
 verify this patch?". Aside from being (IMO) very unhelpful and
 annoying, these are actually causing a significant amount of load on the
 ghprb plugin and indicate that these PRs are more expensive than those from
 allow-listed contributors. Since we believe that the ghprb plugin makes
 calls to GitHub that are roughly proportional to >>> comments>X, reducing our number of issue
 comments and the load per comment should give us enough breathing room to
 enable our triggers again.

 I wrote up a more thorough supporting doc here as well -  -
 https://docs.google.com/document/d/15CILeNjNxCnbigSvxNq4eXPj6x6sn5DGdbTdWu55kCI/edit?usp=sharing

 *Disclaimer*

 It's really hard to empirically prove any of this after the fact - I
 think there's enough evidence to try it, but we should be ready to engage
 Infra to restart Jenkins and ready to revert any triggers we add.

 Thanks,
 Danny

 [1] Context on previous investigation here -
 https://docs.google.com/document/d/10qyUsvB_uVy5jftfTiwohlvN8Qwix5AuadssyoC4JsE/edit?usp=sharing

>>>