Re: [RESULT] [VOTE] Release 2.46.0, release candidate #1

[Danny McCormick via dev]

The release has been finalized, thanks everyone!

Thanks,
Danny

On Fri, Mar 10, 2023 at 2:38 PM Danny McCormick 
wrote:

> We determined that the same issue exists in the 2.45 release, so we are
> going to continue finalizing the release candidate. Thank you for your
> patience.
>
> Thanks,
> Danny
>
> On Wed, Mar 8, 2023 at 2:30 PM Danny McCormick 
> wrote:
>
>> @Reuven Lax  found a new potential regression in
>> BigQuery I/O, so I have paused the release rollout. I had already pushed
>> the Python artifacts and Go tags, but not the Java ones. We have since
>> temporarily yanked  the Python release
>> and deleted the Go tags, they were live for around an hour. The possible
>> regression is in Java, so neither of those releases should be affected, but
>> x-lang may not work properly because it depends on versioning. I will
>> update this thread with next steps when we know more.
>>
>> Thanks,
>> Danny
>>
>> On Wed, Mar 8, 2023 at 1:02 PM Anand Inguva 
>> wrote:
>>
>>> Thanks Danny!!
>>>
>>> On Wed, Mar 8, 2023 at 12:14 PM Danny McCormick via dev <
>>> dev@beam.apache.org> wrote:
>>>
 I'm happy to announce that we have unanimously approved release 2.46.0
 There are 8 approving votes, 5 of which are binding: * Robert Bradshaw
 (binding) * Chamikara Jayalath (binding) * Ahmet Altay (binding) * Alexey
 Romanenko (binding)
 * Jan Lukavský (binding)

 * Bruno Volpato (non-binding)
 * Anand Inguva (non-binding)
 * Ritesh Ghorse (non-binding)
 There are no disapproving votes. I will now begin the process of
 finalizing the release. Thanks everyone!

 - Danny McCormick

>>>

Re: [VOTE] Release 2.46.0, release candidate #1

[Ahmet Altay via dev]

Thank you!

Is there a tracking issue for this known issue? And would the known issues
section of the release notes link to that?


On Fri, Mar 10, 2023 at 11:38 AM Danny McCormick via dev <
dev@beam.apache.org> wrote:

> We determined that the same issue exists in the 2.45 release, so we are
> going to continue finalizing the release candidate. Thank you for your
> patience.
>
> Thanks,
> Danny
>
> On Wed, Mar 8, 2023 at 3:15 PM Reuven Lax  wrote:
>
>> We are trying to reproduce and debug the issue we saw to validate whether
>> it was a real regression or not. Will update when we know more.
>>
>> On Wed, Mar 8, 2023 at 11:31 AM Danny McCormick <
>> dannymccorm...@google.com> wrote:
>>
>>>
>>> @Reuven Lax  found a new potential regression in
>>> BigQuery I/O, so I have paused the release rollout. I had already pushed
>>> the Python artifacts and Go tags, but not the Java ones. We have since
>>> temporarily yanked  the Python release
>>> and deleted the Go tags, they were live for around an hour. The possible
>>> regression is in Java, so neither of those releases should be affected, but
>>> x-lang may not work properly because it depends on versioning. I will
>>> update this thread with next steps when we know more.
>>>
>>> Thanks,
>>> Danny
>>> On Wed, Mar 8, 2023 at 5:59 AM Jan Lukavský  wrote:
>>>
 +1 (binding)

 Tested Java SDK with Flink and Spark 3 runner.

 Thanks,
  Jan

 On 3/8/23 01:53, Valentyn Tymofieiev via dev wrote:

 +1. Verified the composition of Python containers and ran Python
 pipelines on Dataflow runner v1 and runner v2.

 On Tue, Mar 7, 2023 at 4:11 PM Ritesh Ghorse via dev <
 dev@beam.apache.org> wrote:

> +1 (non-binding)
> Validated Go SDK quickstart on direct and dataflow runner
>
> On Tue, Mar 7, 2023 at 10:54 AM Alexey Romanenko <
> aromanenko@gmail.com> wrote:
>
>> +1 (binding)
>>
>> Tested with  https://github.com/Talend/beam-samples/
>> (Java SDK v8/v11/v17, Spark 3.x runner).
>>
>> ---
>> Alexey
>>
>> On 7 Mar 2023, at 07:38, Ahmet Altay via dev 
>> wrote:
>>
>> +1 (binding) - I validated python quickstarts on direct & dataflow
>> runners.
>>
>> Thank you for doing the release!
>>
>> On Sat, Mar 4, 2023 at 8:01 AM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>>> +1 (binding)
>>>
>>> Validated multi-language Java and Python pipelines.
>>>
>>> On Fri, Mar 3, 2023 at 1:59 PM Danny McCormick via dev <
>>> dev@beam.apache.org> wrote:
>>>
 > I have encountered a failure in a Python pipeline running with
 Runner v1:

 > RuntimeError: Beam SDK base version 2.46.0 does not match
 Dataflow Python worker version 2.45.0. Please check Dataflow worker 
 startup
 logs and make sure that correct version of Beam SDK is installed.

 > We should understand why Python ValidatesRunner tests (which have
 passed)  didn't catch this error.

 > This can be remediated in Dataflow containers without  changes to
 the release candidate.

 Good catch! I've kicked off a release to fix this, it should be
 done later this evening - I won't be available when it completes, but I
 would expect it to be around 5:00 PST.

 On Fri, Mar 3, 2023 at 3:49 PM Danny McCormick <
 dannymccorm...@google.com> wrote:

> Hey Reuven, could you provide some more context on the bug/why it
> is important? Does it meet the standard in
> https://beam.apache.org/contribute/release-guide/#7-triage-release-blocking-issues-in-github?
>
>
> The release branch was cut last Wednesday, so that is why it is
> not included.
>

>>> Seems like this was a revert of a previous commit that was also not
>>> included in the 2.46.0 release branch (
>>> https://github.com/apache/beam/pull/25627) ?
>>>
>>> If so we might not need a new RC but good to confirm.
>>>
>>> Thanks,
>>> Cham
>>>
>>>
> On Fri, Mar 3, 2023 at 3:24 PM Reuven Lax 
> wrote:
>
>> If possible, I would like to see if we could include
>> https://github.com/apache/beam/pull/25642 as we believe this bug
>> has been impacting multiple users. This was merged 4 days ago, but 
>> this RC
>> cut does not seem to include it.
>>
>> On Fri, Mar 3, 2023 at 12:18 PM Valentyn Tymofieiev via dev <
>> dev@beam.apache.org> wrote:
>>
>>> I have encountered a failure in a Python pipeline running with
>>> Runner v1:
>>>
>>> RuntimeError: Beam SDK base version 2.46.0 does not match
>>> Dataflow Python worker version 2.45.0. Please check Dataflow worker 
>>>

[GitHub] [beam-site] damccorm merged pull request #641: Publish 2.46.0 release

[via GitHub]

damccorm merged PR #641:
URL: https://github.com/apache/beam-site/pull/641


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam-site] chamikaramj commented on pull request #641: Publish 2.46.0 release

[via GitHub]

chamikaramj commented on PR #641:
URL: https://github.com/apache/beam-site/pull/641#issuecomment-1464371177

   LGTM.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [beam-site] damccorm commented on pull request #641: Publish 2.46.0 release

[via GitHub]

damccorm commented on PR #641:
URL: https://github.com/apache/beam-site/pull/641#issuecomment-1464328242

   R: @johnjcasey 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [RESULT] [VOTE] Release 2.46.0, release candidate #1

[Danny McCormick via dev]

We determined that the same issue exists in the 2.45 release, so we are
going to continue finalizing the release candidate. Thank you for your
patience.

Thanks,
Danny

On Wed, Mar 8, 2023 at 2:30 PM Danny McCormick 
wrote:

> @Reuven Lax  found a new potential regression in
> BigQuery I/O, so I have paused the release rollout. I had already pushed
> the Python artifacts and Go tags, but not the Java ones. We have since
> temporarily yanked  the Python release and
> deleted the Go tags, they were live for around an hour. The possible
> regression is in Java, so neither of those releases should be affected, but
> x-lang may not work properly because it depends on versioning. I will
> update this thread with next steps when we know more.
>
> Thanks,
> Danny
>
> On Wed, Mar 8, 2023 at 1:02 PM Anand Inguva 
> wrote:
>
>> Thanks Danny!!
>>
>> On Wed, Mar 8, 2023 at 12:14 PM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> I'm happy to announce that we have unanimously approved release 2.46.0
>>> There are 8 approving votes, 5 of which are binding: * Robert Bradshaw
>>> (binding) * Chamikara Jayalath (binding) * Ahmet Altay (binding) * Alexey
>>> Romanenko (binding)
>>> * Jan Lukavský (binding)
>>>
>>> * Bruno Volpato (non-binding)
>>> * Anand Inguva (non-binding)
>>> * Ritesh Ghorse (non-binding)
>>> There are no disapproving votes. I will now begin the process of
>>> finalizing the release. Thanks everyone!
>>>
>>> - Danny McCormick
>>>
>>

Re: [VOTE] Release 2.46.0, release candidate #1

[Danny McCormick via dev]

We determined that the same issue exists in the 2.45 release, so we are
going to continue finalizing the release candidate. Thank you for your
patience.

Thanks,
Danny

On Wed, Mar 8, 2023 at 3:15 PM Reuven Lax  wrote:

> We are trying to reproduce and debug the issue we saw to validate whether
> it was a real regression or not. Will update when we know more.
>
> On Wed, Mar 8, 2023 at 11:31 AM Danny McCormick 
> wrote:
>
>>
>> @Reuven Lax  found a new potential regression in
>> BigQuery I/O, so I have paused the release rollout. I had already pushed
>> the Python artifacts and Go tags, but not the Java ones. We have since
>> temporarily yanked  the Python release
>> and deleted the Go tags, they were live for around an hour. The possible
>> regression is in Java, so neither of those releases should be affected, but
>> x-lang may not work properly because it depends on versioning. I will
>> update this thread with next steps when we know more.
>>
>> Thanks,
>> Danny
>> On Wed, Mar 8, 2023 at 5:59 AM Jan Lukavský  wrote:
>>
>>> +1 (binding)
>>>
>>> Tested Java SDK with Flink and Spark 3 runner.
>>>
>>> Thanks,
>>>  Jan
>>>
>>> On 3/8/23 01:53, Valentyn Tymofieiev via dev wrote:
>>>
>>> +1. Verified the composition of Python containers and ran Python
>>> pipelines on Dataflow runner v1 and runner v2.
>>>
>>> On Tue, Mar 7, 2023 at 4:11 PM Ritesh Ghorse via dev <
>>> dev@beam.apache.org> wrote:
>>>
 +1 (non-binding)
 Validated Go SDK quickstart on direct and dataflow runner

 On Tue, Mar 7, 2023 at 10:54 AM Alexey Romanenko <
 aromanenko@gmail.com> wrote:

> +1 (binding)
>
> Tested with  https://github.com/Talend/beam-samples/
> (Java SDK v8/v11/v17, Spark 3.x runner).
>
> ---
> Alexey
>
> On 7 Mar 2023, at 07:38, Ahmet Altay via dev 
> wrote:
>
> +1 (binding) - I validated python quickstarts on direct & dataflow
> runners.
>
> Thank you for doing the release!
>
> On Sat, Mar 4, 2023 at 8:01 AM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
>
>> +1 (binding)
>>
>> Validated multi-language Java and Python pipelines.
>>
>> On Fri, Mar 3, 2023 at 1:59 PM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> > I have encountered a failure in a Python pipeline running with
>>> Runner v1:
>>>
>>> > RuntimeError: Beam SDK base version 2.46.0 does not match Dataflow
>>> Python worker version 2.45.0. Please check Dataflow worker startup logs 
>>> and
>>> make sure that correct version of Beam SDK is installed.
>>>
>>> > We should understand why Python ValidatesRunner tests (which have
>>> passed)  didn't catch this error.
>>>
>>> > This can be remediated in Dataflow containers without  changes to
>>> the release candidate.
>>>
>>> Good catch! I've kicked off a release to fix this, it should be done
>>> later this evening - I won't be available when it completes, but I would
>>> expect it to be around 5:00 PST.
>>>
>>> On Fri, Mar 3, 2023 at 3:49 PM Danny McCormick <
>>> dannymccorm...@google.com> wrote:
>>>
 Hey Reuven, could you provide some more context on the bug/why it
 is important? Does it meet the standard in
 https://beam.apache.org/contribute/release-guide/#7-triage-release-blocking-issues-in-github?


 The release branch was cut last Wednesday, so that is why it is not
 included.

>>>
>> Seems like this was a revert of a previous commit that was also not
>> included in the 2.46.0 release branch (
>> https://github.com/apache/beam/pull/25627) ?
>>
>> If so we might not need a new RC but good to confirm.
>>
>> Thanks,
>> Cham
>>
>>
 On Fri, Mar 3, 2023 at 3:24 PM Reuven Lax  wrote:

> If possible, I would like to see if we could include
> https://github.com/apache/beam/pull/25642 as we believe this bug
> has been impacting multiple users. This was merged 4 days ago, but 
> this RC
> cut does not seem to include it.
>
> On Fri, Mar 3, 2023 at 12:18 PM Valentyn Tymofieiev via dev <
> dev@beam.apache.org> wrote:
>
>> I have encountered a failure in a Python pipeline running with
>> Runner v1:
>>
>> RuntimeError: Beam SDK base version 2.46.0 does not match
>> Dataflow Python worker version 2.45.0. Please check Dataflow worker 
>> startup
>> logs and make sure that correct version of Beam SDK is installed.
>>
>> We should understand why Python ValidatesRunner tests (which have
>> passed)  didn't catch this error.
>>
>> This can be remediated in Dataflow containers without  changes to
>> the release candidate.
>>
>> On Fri, Mar 3, 2023 at 11:

Beam High Priority Issue Report (35)

[beamactions]

This is your daily summary of Beam's current high priority issues that may need 
attention.

See https://beam.apache.org/contribute/issue-priorities for the meaning and 
expectations around issue priorities.

Unassigned P1 Issues:

https://github.com/apache/beam/issues/25675 [Bug]: Reenable 
GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow: causes dataflow suite to 
be permared
https://github.com/apache/beam/issues/24776 [Bug]: Race condition in Python SDK 
Harness ProcessBundleProgress
https://github.com/apache/beam/issues/24389 [Failing Test]: 
HadoopFormatIOElasticTest.classMethod ExceptionInInitializerError 
ContainerFetchException
https://github.com/apache/beam/issues/24313 [Flaky]: 
apache_beam/runners/portability/portable_runner_test.py::PortableRunnerTestWithSubprocesses::test_pardo_state_with_custom_key_coder
https://github.com/apache/beam/issues/24267 [Failing Test]: Timeout waiting to 
lock gradle
https://github.com/apache/beam/issues/23944  beam_PreCommit_Python_Cron 
regularily failing - test_pardo_large_input flaky
https://github.com/apache/beam/issues/23709 [Flake]: Spark batch flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
https://github.com/apache/beam/issues/22961 [Bug]: WriteToBigQuery silently 
skips most of records without job fail
https://github.com/apache/beam/issues/22913 [Bug]: 
beam_PostCommit_Java_ValidatesRunner_Flink is flakes in 
org.apache.beam.sdk.transforms.GroupByKeyTest$BasicTests.testAfterProcessingTimeContinuationTriggerUsingState
https://github.com/apache/beam/issues/22115 [Bug]: 
apache_beam.runners.portability.portable_runner_test.PortableRunnerTestWithSubprocesses
 is flaky
https://github.com/apache/beam/issues/21713 404s in BigQueryIO don't get output 
to Failed Inserts PCollection
https://github.com/apache/beam/issues/21706 Flaky timeout in github Python unit 
test action 
StatefulDoFnOnDirectRunnerTest.test_dynamic_timer_clear_then_set_timer
https://github.com/apache/beam/issues/21695 DataflowPipelineResult does not 
raise exception for unsuccessful states.
https://github.com/apache/beam/issues/21643 FnRunnerTest with non-trivial 
(order 1000 elements) numpy input flakes in non-cython environment
https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: 
Connection refused
https://github.com/apache/beam/issues/21424 Java VR (Dataflow, V2, Streaming) 
failing: ParDoTest$TimestampTests/OnWindowExpirationTests
https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do not 
follow spec
https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit 
data at GC time
https://github.com/apache/beam/issues/21121 
apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it
 flakey
https://github.com/apache/beam/issues/21104 Flaky: 
apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers
https://github.com/apache/beam/issues/20976 
apache_beam.runners.portability.flink_runner_test.FlinkRunnerTestOptimized.test_flink_metrics
 is flaky
https://github.com/apache/beam/issues/20974 Python GHA PreCommits flake with 
grpc.FutureTimeoutError on SDK harness startup
https://github.com/apache/beam/issues/20689 Kafka commitOffsetsInFinalize OOM 
on Flink
https://github.com/apache/beam/issues/20108 Python direct runner doesn't emit 
empty pane when it should
https://github.com/apache/beam/issues/19814 Flink streaming flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful and 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
https://github.com/apache/beam/issues/19465 Explore possibilities to lower 
in-use IP address quota footprint.


P1 Issues with no update in the last week:

https://github.com/apache/beam/issues/25669 [Bug]: Different orderings of 
SchemaAwareExternalTransform() kwargs may result in misplaced arguments 
https://github.com/apache/beam/issues/23875 [Bug]: beam.Row.__eq__ returns true 
for unequal rows
https://github.com/apache/beam/issues/23848 Support for Python 3.11
https://github.com/apache/beam/issues/23525 [Bug]: Default PubsubMessage coder 
will drop message id and orderingKey
https://github.com/apache/beam/issues/22605 [Bug]: Beam Python failure for 
dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest.test_metrics_it
https://github.com/apache/beam/issues/21714 
PulsarIOTest.testReadFromSimpleTopic is very flaky
https://github.com/apache/beam/issues/21708 beam_PostCommit_Java_DataflowV2, 
testBigQueryStorageWrite30MProto failing consistently
https://github.com/apache/beam/issues/21645 
beam_PostCommit_XVR_GoUsingJava_Dataflow fails on some test transforms
https://github.com/apache/beam/issues/21476 WriteToBigQuery Dynamic table 
destinations returns wrong tableId