unsubscribe

[Tongjie Chen]

EOM

Re: [ANNOUNCE] New Committer: Sam Whittle

[Austin Bennett]

Thanks, Sam!

On Mon, Oct 16, 2023 at 12:39 PM XQ Hu via dev  wrote:

> Congratulations!
>
> On Mon, Oct 16, 2023 at 1:58 PM Ahmet Altay via dev 
> wrote:
>
>> Congratulations Sam!
>>
>> On Mon, Oct 16, 2023 at 10:42 AM Byron Ellis via dev 
>> wrote:
>>
>>> Congrats Sam!
>>>
>>> On Mon, Oct 16, 2023 at 10:32 AM Chamikara Jayalath via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congrats Sam!

 On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles 
 wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Sam Whittle (scwhit...@apache.org).
>
> Sam has been contributing to Beam since 2016! In particular, he
> specializes in streaming and the Dataflow Java worker but his 
> contributions
> expand naturally from there to the Java SDK, IOs, and even a bit of Python
> :-). Sam has contributed a ton of code over the years and is generous in
> code review and sharing his expertise.
>
> Considering his contributions to the project over this timeframe, the
> Beam PMC trusts Sam with the responsibilities of a Beam committer. [1]
>
> Thank you Sam! And we are looking to see more of your contributions!
>
> Kenn, on behalf of the Apache Beam PMC
>
> [1]
>
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>

Re: [ANNOUNCE] New Committer: Byron Ellis

[Austin Bennett]

thanks, Byron!

On Mon, Oct 16, 2023 at 12:38 PM XQ Hu via dev  wrote:

> Congratulations!
>
> On Mon, Oct 16, 2023 at 1:58 PM Ahmet Altay via dev 
> wrote:
>
>> Congratulations Byron!
>>
>> On Mon, Oct 16, 2023 at 10:35 AM Tomo Suzuki via dev 
>> wrote:
>>
>>> Congratulations!
>>>
>>>
>>> On Mon, Oct 16, 2023 at 1:33 PM Chamikara Jayalath via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congrats Byron!

 On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles 
 wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Byron Ellis (b...@apache.org).
>
> Byron has been with Beam for over a year now. You may all know him as
> the guy who just decided to write a Swift SDK :-). In addition to that big
> contribution Byron has also fixed plenty of bugs, prototyped DBT-tyle
> pipeline authoring, and participated in our collective decision-making
> process.
>
> Considering his contributions to the project over this timeframe, the
> Beam PMC trusts Byron with the responsibilities of a Beam committer.
> [1]
>
> Thank you Byron! And we are looking to see more of your contributions!
>
> Kenn, on behalf of the Apache Beam PMC
>
> [1]
>
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>

>>>
>>> --
>>> Regards,
>>> Tomo
>>>
>>

Re: Disabling Jenkins Jobs

[Danny McCormick via dev]

The number of runners should autoscale, but it seems that they did not do
so quickly enough in this instance. For the short term, we have set a
higher minimum number of runners (and are currently all caught up on
backlog) and we will investigate the underlying issue.

Thanks for calling this out Alexey!

On Mon, Oct 16, 2023 at 10:48 AM Alexey Romanenko 
wrote:

> Thanks for moving this forward!
>
> On a flip side, I noticed that many PreCommit actions (actually all for my
> PR #27851) are stuck waiting for a runner.
> Is it expected behaviour?
> Should we increase a number a runners while moving all Jenkins Jobs to
> GitHub actions?
>
> —
> Alexey
>
> On 13 Oct 2023, at 17:09, Yi Hu via dev  wrote:
>
> Hi all,
>
> We have shutdown another set of Jenkins PreCommit, see the newly shutdown
> ones in https://github.com/apache/beam/pull/28840 , And a complete list
> of fully migrated tests tracked in
> https://github.com/apache/beam/issues/28426
>
> Regards,
> Yi
>
> On Thu, Sep 14, 2023 at 3:41 PM Yi Hu  wrote:
>
>> Hi all,
>>
>> We just shutdown a second list of Jenkins test suites. The full list is
>> tracked in https://github.com/apache/beam/issues/28426 . In the Issue
>> there is also information about the status of migration as we proceed.
>>
>> Regards,
>> Yi
>>
>> On Thu, Sep 14, 2023 at 12:44 PM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Sure, I added
>>> https://cwiki.apache.org/confluence/display/BEAM/GitHub+Actions+Tips
>>>
>>> On Thu, Sep 14, 2023 at 12:39 PM Ahmet Altay  wrote:
>>>
 That is great. Could we add a link to that README from
 https://cwiki.apache.org/confluence/display/BEAM/Developer+Guides ?
 That will increase discoverability for people like me who use wiki as the
 starting point for finding how to do things.

 On Thu, Sep 14, 2023 at 8:41 AM Danny McCormick <
 dannymccorm...@google.com> wrote:

> Most of our docs for GitHub actions are located here -
> https://github.com/apache/beam/blob/master/.github/workflows/README.md.
> I added https://github.com/apache/beam/pull/28453 to add instructions
> to that page.
>
> Thanks,
> Danny
>
> On Wed, Sep 13, 2023 at 12:31 PM Ahmet Altay via dev <
> dev@beam.apache.org> wrote:
>
>> This is all great. Do you mind documenting the github actions flow
>> for running these jobs on the wiki? Or if it already exists, share a 
>> link.
>> Thank you!
>>
>> On Wed, Sep 13, 2023 at 6:19 AM XQ Hu via dev 
>> wrote:
>>
>>> This is awesome! Thanks, Danny, Yi, Andrey, and Vlado!
>>>
>>> On Wed, Sep 13, 2023 at 8:08 AM Danny McCormick via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Right now, it is just the set of jobs in Yi's PR -
 https://github.com/apache/beam/pull/28316/files but the plan is to
 slowly move jobs over time once we've built confidence that they work.

 There is a set of jobs that are non-idempotent which we are moving
 in one shot (which does include the website publishing job in this
 PR  that I just merged)
 and monitoring closely. The best source of truth is just looking at
 https://ci-beam.apache.org/ though. Any migrated jobs will have
 the disabled symbol next to them - for example, the whitespace jobs 
 have
 now been migrated so they are disabled on Jenkins:

 

 To run a job that has been migrated manually, you can navigate to
 that job in the Actions tab and click "run workflow". So for the 
 website
 publish job you would navigate to
 https://github.com/apache/beam/actions/workflows/beam_PostCommit_Website_Publish.yml
  and
 click this button:


 

 Thanks,
 Danny

 On Tue, Sep 12, 2023 at 6:22 PM Ahmet Altay via dev <
 dev@beam.apache.org> wrote:

> Thank you for doing this.
>
> Is there a list of jobs that will be disabled? I am particularly
> curious about: website publishing job (which I need to use manually
> sometimes) and the job that publishes daily staging builds (which we 
> share
> with users sometimes.)
>
> Thank you.
> Ahmet
>
> On Tue, Sep 12, 2023 at 11:14 AM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> Hey everyone, I wanted to let you know that as part of the
>> migration from Jenkins to GitHub Actions we are going to start 
>> disabling
>> Jenkins jobs if they have a corresponding GitHub Actions job that 
>> has been
>> running successfully for a while. We are starting with Yi's PR here -
>> https://github.com/apache/beam/pull/28316. This is the next step
>> in the process we kicked 

Re: [ANNOUNCE] New Committer: Sam Whittle

[XQ Hu via dev]

Congratulations!

On Mon, Oct 16, 2023 at 1:58 PM Ahmet Altay via dev 
wrote:

> Congratulations Sam!
>
> On Mon, Oct 16, 2023 at 10:42 AM Byron Ellis via dev 
> wrote:
>
>> Congrats Sam!
>>
>> On Mon, Oct 16, 2023 at 10:32 AM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congrats Sam!
>>>
>>> On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles  wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Sam Whittle (scwhit...@apache.org).

 Sam has been contributing to Beam since 2016! In particular, he
 specializes in streaming and the Dataflow Java worker but his contributions
 expand naturally from there to the Java SDK, IOs, and even a bit of Python
 :-). Sam has contributed a ton of code over the years and is generous in
 code review and sharing his expertise.

 Considering his contributions to the project over this timeframe, the
 Beam PMC trusts Sam with the responsibilities of a Beam committer. [1]

 Thank you Sam! And we are looking to see more of your contributions!

 Kenn, on behalf of the Apache Beam PMC

 [1]

 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

>>>

Re: [ANNOUNCE] New Committer: Byron Ellis

[XQ Hu via dev]

Congratulations!

On Mon, Oct 16, 2023 at 1:58 PM Ahmet Altay via dev 
wrote:

> Congratulations Byron!
>
> On Mon, Oct 16, 2023 at 10:35 AM Tomo Suzuki via dev 
> wrote:
>
>> Congratulations!
>>
>>
>> On Mon, Oct 16, 2023 at 1:33 PM Chamikara Jayalath via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congrats Byron!
>>>
>>> On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles  wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Byron Ellis (b...@apache.org).

 Byron has been with Beam for over a year now. You may all know him as
 the guy who just decided to write a Swift SDK :-). In addition to that big
 contribution Byron has also fixed plenty of bugs, prototyped DBT-tyle
 pipeline authoring, and participated in our collective decision-making
 process.

 Considering his contributions to the project over this timeframe, the
 Beam PMC trusts Byron with the responsibilities of a Beam committer. [1]

 Thank you Byron! And we are looking to see more of your contributions!

 Kenn, on behalf of the Apache Beam PMC

 [1]

 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

>>>
>>
>> --
>> Regards,
>> Tomo
>>
>

Re: [ANNOUNCE] New Committer: Sam Whittle

[Ahmet Altay via dev]

Congratulations Sam!

On Mon, Oct 16, 2023 at 10:42 AM Byron Ellis via dev 
wrote:

> Congrats Sam!
>
> On Mon, Oct 16, 2023 at 10:32 AM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Sam!
>>
>> On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Sam Whittle (scwhit...@apache.org).
>>>
>>> Sam has been contributing to Beam since 2016! In particular, he
>>> specializes in streaming and the Dataflow Java worker but his contributions
>>> expand naturally from there to the Java SDK, IOs, and even a bit of Python
>>> :-). Sam has contributed a ton of code over the years and is generous in
>>> code review and sharing his expertise.
>>>
>>> Considering his contributions to the project over this timeframe, the
>>> Beam PMC trusts Sam with the responsibilities of a Beam committer. [1]
>>>
>>> Thank you Sam! And we are looking to see more of your contributions!
>>>
>>> Kenn, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>>
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>

Re: [ANNOUNCE] New Committer: Byron Ellis

[Ahmet Altay via dev]

Congratulations Byron!

On Mon, Oct 16, 2023 at 10:35 AM Tomo Suzuki via dev 
wrote:

> Congratulations!
>
>
> On Mon, Oct 16, 2023 at 1:33 PM Chamikara Jayalath via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Byron!
>>
>> On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Byron Ellis (b...@apache.org).
>>>
>>> Byron has been with Beam for over a year now. You may all know him as
>>> the guy who just decided to write a Swift SDK :-). In addition to that big
>>> contribution Byron has also fixed plenty of bugs, prototyped DBT-tyle
>>> pipeline authoring, and participated in our collective decision-making
>>> process.
>>>
>>> Considering his contributions to the project over this timeframe, the
>>> Beam PMC trusts Byron with the responsibilities of a Beam committer. [1]
>>>
>>> Thank you Byron! And we are looking to see more of your contributions!
>>>
>>> Kenn, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>>
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>
>
> --
> Regards,
> Tomo
>

Re: [ANNOUNCE] New Committer: Sam Whittle

[Byron Ellis via dev]

Congrats Sam!

On Mon, Oct 16, 2023 at 10:32 AM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> Congrats Sam!
>
> On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Sam Whittle (scwhit...@apache.org).
>>
>> Sam has been contributing to Beam since 2016! In particular, he
>> specializes in streaming and the Dataflow Java worker but his contributions
>> expand naturally from there to the Java SDK, IOs, and even a bit of Python
>> :-). Sam has contributed a ton of code over the years and is generous in
>> code review and sharing his expertise.
>>
>> Considering his contributions to the project over this timeframe, the
>> Beam PMC trusts Sam with the responsibilities of a Beam committer. [1]
>>
>> Thank you Sam! And we are looking to see more of your contributions!
>>
>> Kenn, on behalf of the Apache Beam PMC
>>
>> [1]
>>
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>

Re: [ANNOUNCE] New Committer: Byron Ellis

[Tomo Suzuki via dev]

Congratulations!


On Mon, Oct 16, 2023 at 1:33 PM Chamikara Jayalath via dev <
dev@beam.apache.org> wrote:

> Congrats Byron!
>
> On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Byron Ellis (b...@apache.org).
>>
>> Byron has been with Beam for over a year now. You may all know him as the
>> guy who just decided to write a Swift SDK :-). In addition to that big
>> contribution Byron has also fixed plenty of bugs, prototyped DBT-tyle
>> pipeline authoring, and participated in our collective decision-making
>> process.
>>
>> Considering his contributions to the project over this timeframe, the
>> Beam PMC trusts Byron with the responsibilities of a Beam committer. [1]
>>
>> Thank you Byron! And we are looking to see more of your contributions!
>>
>> Kenn, on behalf of the Apache Beam PMC
>>
>> [1]
>>
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>

-- 
Regards,
Tomo

Re: [ANNOUNCE] New Committer: Byron Ellis

[Chamikara Jayalath via dev]

Congrats Byron!

On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles  wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Byron Ellis (b...@apache.org).
>
> Byron has been with Beam for over a year now. You may all know him as the
> guy who just decided to write a Swift SDK :-). In addition to that big
> contribution Byron has also fixed plenty of bugs, prototyped DBT-tyle
> pipeline authoring, and participated in our collective decision-making
> process.
>
> Considering his contributions to the project over this timeframe, the
> Beam PMC trusts Byron with the responsibilities of a Beam committer. [1]
>
> Thank you Byron! And we are looking to see more of your contributions!
>
> Kenn, on behalf of the Apache Beam PMC
>
> [1]
>
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>

Re: [ANNOUNCE] New Committer: Sam Whittle

[Chamikara Jayalath via dev]

Congrats Sam!

On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles  wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Sam Whittle (scwhit...@apache.org).
>
> Sam has been contributing to Beam since 2016! In particular, he
> specializes in streaming and the Dataflow Java worker but his contributions
> expand naturally from there to the Java SDK, IOs, and even a bit of Python
> :-). Sam has contributed a ton of code over the years and is generous in
> code review and sharing his expertise.
>
> Considering his contributions to the project over this timeframe, the
> Beam PMC trusts Sam with the responsibilities of a Beam committer. [1]
>
> Thank you Sam! And we are looking to see more of your contributions!
>
> Kenn, on behalf of the Apache Beam PMC
>
> [1]
>
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>

Re: Enable state cache in Python SDK

[Robert Bradshaw via dev]

+1, we should definitely be enabling at least some caching by default
here. Added some comments to the doc.

On Mon, Oct 16, 2023 at 9:27 AM Anand Inguva via dev
 wrote:
>
> Hello,
>
> In Python SDK, the user state and side input caching is disabled by default 
> for all the runners except FnAPI direct runner(intended for testing 
> purposes). I would like to propose that we enable the state cache for the 
> Python SDK similar to other SDKs. I created a doc[1] on why we need to do it, 
> advantages and disadvantages along with future improvements.
>
> Please take a look and let me know what you think.
>
> Thanks,
> Anand
>
> [1] 
> https://docs.google.com/document/d/1gllYsIFqKt4TWAxQmXU_-sw7SLnur2Q69d__N0XBMdE/edit
>
>

Re: Enable state cache in Python SDK

[Anand Inguva via dev]

My bad! It was on viewer access when I shared. I updated the doc access to
commenters now.

On Mon, Oct 16, 2023 at 12:26 PM Anand Inguva 
wrote:

> Hello,
>
> In Python SDK, the user state and side input caching is disabled by
> default for all the runners except FnAPI direct runner(intended for testing
> purposes). I would like to propose that we enable the state cache for the
> Python SDK similar to other SDKs. I created a doc[1] on why we need to do
> it, advantages and disadvantages along with future improvements.
>
> Please take a look and let me know what you think.
>
> Thanks,
> Anand
>
> [1]
> https://docs.google.com/document/d/1gllYsIFqKt4TWAxQmXU_-sw7SLnur2Q69d__N0XBMdE/edit
>
>
>

[ANNOUNCE] New Committer: Byron Ellis

[Kenneth Knowles]

Hi all,

Please join me and the rest of the Beam PMC in welcoming a new
committer: Byron Ellis (b...@apache.org).

Byron has been with Beam for over a year now. You may all know him as the
guy who just decided to write a Swift SDK :-). In addition to that big
contribution Byron has also fixed plenty of bugs, prototyped DBT-tyle
pipeline authoring, and participated in our collective decision-making
process.

Considering his contributions to the project over this timeframe, the
Beam PMC trusts Byron with the responsibilities of a Beam committer. [1]

Thank you Byron! And we are looking to see more of your contributions!

Kenn, on behalf of the Apache Beam PMC

[1]
https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

[ANNOUNCE] New Committer: Sam Whittle

[Kenneth Knowles]

Hi all,

Please join me and the rest of the Beam PMC in welcoming a new
committer: Sam Whittle (scwhit...@apache.org).

Sam has been contributing to Beam since 2016! In particular, he specializes
in streaming and the Dataflow Java worker but his contributions expand
naturally from there to the Java SDK, IOs, and even a bit of Python :-).
Sam has contributed a ton of code over the years and is generous in code
review and sharing his expertise.

Considering his contributions to the project over this timeframe, the
Beam PMC trusts Sam with the responsibilities of a Beam committer. [1]

Thank you Sam! And we are looking to see more of your contributions!

Kenn, on behalf of the Apache Beam PMC

[1]
https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

Enable state cache in Python SDK

[Anand Inguva via dev]

Hello,

In Python SDK, the user state and side input caching is disabled by default
for all the runners except FnAPI direct runner(intended for testing
purposes). I would like to propose that we enable the state cache for the
Python SDK similar to other SDKs. I created a doc[1] on why we need to do
it, advantages and disadvantages along with future improvements.

Please take a look and let me know what you think.

Thanks,
Anand

[1]
https://docs.google.com/document/d/1gllYsIFqKt4TWAxQmXU_-sw7SLnur2Q69d__N0XBMdE/edit

Re: Disabling Jenkins Jobs

[Alexey Romanenko]

Thanks for moving this forward!

On a flip side, I noticed that many PreCommit actions (actually all for my PR 
#27851) are stuck waiting for a runner. 
Is it expected behaviour? 
Should we increase a number a runners while moving all Jenkins Jobs to GitHub 
actions?

—
Alexey

> On 13 Oct 2023, at 17:09, Yi Hu via dev  wrote:
> 
> Hi all,
> 
> We have shutdown another set of Jenkins PreCommit, see the newly shutdown 
> ones in https://github.com/apache/beam/pull/28840 , And a complete list of 
> fully migrated tests tracked in https://github.com/apache/beam/issues/28426 
> 
> Regards,
> Yi
> 
> On Thu, Sep 14, 2023 at 3:41 PM Yi Hu  > wrote:
>> Hi all,
>> 
>> We just shutdown a second list of Jenkins test suites. The full list is 
>> tracked in https://github.com/apache/beam/issues/28426 . In the Issue there 
>> is also information about the status of migration as we proceed.
>> 
>> Regards,
>> Yi
>> 
>> On Thu, Sep 14, 2023 at 12:44 PM Danny McCormick via dev 
>> mailto:dev@beam.apache.org>> wrote:
>>> Sure, I added 
>>> https://cwiki.apache.org/confluence/display/BEAM/GitHub+Actions+Tips
>>> 
>>> On Thu, Sep 14, 2023 at 12:39 PM Ahmet Altay >> > wrote:
 That is great. Could we add a link to that README from 
 https://cwiki.apache.org/confluence/display/BEAM/Developer+Guides ? That 
 will increase discoverability for people like me who use wiki as the 
 starting point for finding how to do things.
 
 On Thu, Sep 14, 2023 at 8:41 AM Danny McCormick >>> > wrote:
> Most of our docs for GitHub actions are located here - 
> https://github.com/apache/beam/blob/master/.github/workflows/README.md. I 
> added https://github.com/apache/beam/pull/28453 to add instructions to 
> that page.
> 
> Thanks,
> Danny
> 
> On Wed, Sep 13, 2023 at 12:31 PM Ahmet Altay via dev  > wrote:
>> This is all great. Do you mind documenting the github actions flow for 
>> running these jobs on the wiki? Or if it already exists, share a link. 
>> Thank you!
>> 
>> On Wed, Sep 13, 2023 at 6:19 AM XQ Hu via dev > > wrote:
>>> This is awesome! Thanks, Danny, Yi, Andrey, and Vlado!
>>> 
>>> On Wed, Sep 13, 2023 at 8:08 AM Danny McCormick via dev 
>>> mailto:dev@beam.apache.org>> wrote:
 Right now, it is just the set of jobs in Yi's PR - 
 https://github.com/apache/beam/pull/28316/files but the plan is to 
 slowly move jobs over time once we've built confidence that they work.
 
 There is a set of jobs that are non-idempotent which we are moving in 
 one shot (which does include the website publishing job in this PR 
  that I just merged) and 
 monitoring closely. The best source of truth is just looking at 
 https://ci-beam.apache.org/ though. Any migrated jobs will have the 
 disabled symbol next to them - for example, the whitespace jobs have 
 now been migrated so they are disabled on Jenkins:
 
 
 
 To run a job that has been migrated manually, you can navigate to that 
 job in the Actions tab and click "run workflow". So for the website 
 publish job you would navigate to 
 https://github.com/apache/beam/actions/workflows/beam_PostCommit_Website_Publish.yml
  and click this button:
 
 
 
 
 Thanks,
 Danny
 
 On Tue, Sep 12, 2023 at 6:22 PM Ahmet Altay via dev 
 mailto:dev@beam.apache.org>> wrote:
> Thank you for doing this.
> 
> Is there a list of jobs that will be disabled? I am particularly 
> curious about: website publishing job (which I need to use manually 
> sometimes) and the job that publishes daily staging builds (which we 
> share with users sometimes.)
> 
> Thank you.
> Ahmet
> 
> On Tue, Sep 12, 2023 at 11:14 AM Danny McCormick via dev 
> mailto:dev@beam.apache.org>> wrote:
>> Hey everyone, I wanted to let you know that as part of the migration 
>> from Jenkins to GitHub Actions we are going to start disabling 
>> Jenkins jobs if they have a corresponding GitHub Actions job that 
>> has been running successfully for a while. We are starting with Yi's 
>> PR here - https://github.com/apache/beam/pull/28316. This is the 
>> next step in the process we kicked off last year [1] now that 
>> self-hosted runners have been in place and working for a while [2].
>> 
>> We will not migrate jobs until we've confirmed we have parity with 
>> the existing Jenkins implementations (for example, some jobs are 
>> still missing test publishing and we won't 

Re: [DISCUSS] Drop Euphoria extension

[Jan Lukavský]

Sure, that would be probably the preferred way to go. For now, I'm 
trying to get some feedback, if there are some real-world users who 
might miss the API. Currently, the only value I see is that Euphoria 
adds an additional level of indirection for user code. The expansion 
goes like this:


 Euphoria Pipeline -> runtime provided translators -> vanilla Beam 
Pipeline -> runner


Hence code written using Euphoria extension can be modified at runtime 
(Pipeline construction time) using dependency injection, which brings 
the value that users can modify (typically optimize) Pipelines without 
actually modifying the business logic. On the other hand I'm not sure if 
this justifies the complexity of the extension. Were this the only 
value, it should be possible to implement such dynamic expansion either 
into Java SDK core or as a different light-weight extension.


 Jan

On 10/16/23 15:10, Alexey Romanenko wrote:

Can we just deprecate it for a while and then remove completely?

—
Alexey


On 13 Oct 2023, at 18:59, Jan Lukavský  wrote:

Hi,

it has been some time since Euphoria extension [1] has been adopted by Beam as a possible 
"Java 8 API". Beam has evolved from that time a lot, the current API seems 
actually more elegant than the original Euphoria's and last but not least, it has no 
maintainers and no known users. If there are any users, please speak up!

Otherwise I'd like to propose to drop it from codebase, I'll start a vote 
thread during next week, if there are no objections.

Best,

  Jan

[1] https://beam.apache.org/documentation/sdks/java/euphoria/

Re: [DISCUSS] Drop Euphoria extension

[Alexey Romanenko]

Can we just deprecate it for a while and then remove completely?

—
Alexey

> On 13 Oct 2023, at 18:59, Jan Lukavský  wrote:
> 
> Hi,
> 
> it has been some time since Euphoria extension [1] has been adopted by Beam 
> as a possible "Java 8 API". Beam has evolved from that time a lot, the 
> current API seems actually more elegant than the original Euphoria's and last 
> but not least, it has no maintainers and no known users. If there are any 
> users, please speak up!
> 
> Otherwise I'd like to propose to drop it from codebase, I'll start a vote 
> thread during next week, if there are no objections.
> 
> Best,
> 
>  Jan
> 
> [1] https://beam.apache.org/documentation/sdks/java/euphoria/
> 

Beam High Priority Issue Report (43)

[beamactions]

This is your daily summary of Beam's current high priority issues that may need 
attention.

See https://beam.apache.org/contribute/issue-priorities for the meaning and 
expectations around issue priorities.

Unassigned P1 Issues:

https://github.com/apache/beam/issues/28985 [Bug]: Savepoint doesn't work with 
savepointPath specificed
https://github.com/apache/beam/issues/28909 [Stuck Test]: GitHub Action 
issue_comment trigger not scalable
https://github.com/apache/beam/issues/28760 [Bug]: EFO Kinesis IO reader 
provided by apache beam does not pick the event time for watermarking
https://github.com/apache/beam/issues/28703 [Failing Test]: Building a wheel 
for integration tests sometimes times out
https://github.com/apache/beam/issues/28383 [Failing Test]: 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest.testMaxThreadMetric
https://github.com/apache/beam/issues/28339 Fix failing 
"beam_PostCommit_XVR_GoUsingJava_Dataflow" job
https://github.com/apache/beam/issues/28326 Bug: 
apache_beam.io.gcp.pubsublite.ReadFromPubSubLite not working
https://github.com/apache/beam/issues/28142 [Bug]: [Go SDK] Memory seems to be 
leaking on 2.49.0 with Dataflow
https://github.com/apache/beam/issues/27892 [Bug]: ignoreUnknownValues not 
working when using CreateDisposition.CREATE_IF_NEEDED 
https://github.com/apache/beam/issues/27648 [Bug]: Python SDFs (e.g. 
PeriodicImpulse) running in Flink and polling using tracker.defer_remainder 
have checkpoint size growing indefinitely 
https://github.com/apache/beam/issues/27616 [Bug]: Unable to use 
applyRowMutations() in bigquery IO apache beam java
https://github.com/apache/beam/issues/27486 [Bug]: Read from datastore with 
inequality filters
https://github.com/apache/beam/issues/27314 [Failing Test]: 
bigquery.StorageApiSinkCreateIfNeededIT.testCreateManyTables[1]
https://github.com/apache/beam/issues/27238 [Bug]: Window trigger has lag when 
using Kafka and GroupByKey on Dataflow Runner
https://github.com/apache/beam/issues/26981 [Bug]: Getting an error related to 
SchemaCoder after upgrading to 2.48
https://github.com/apache/beam/issues/26911 [Bug]: UNNEST ARRAY with a nested 
ROW (described below)
https://github.com/apache/beam/issues/26343 [Bug]: 
apache_beam.io.gcp.bigquery_read_it_test.ReadAllBQTests.test_read_queries is 
flaky
https://github.com/apache/beam/issues/26329 [Bug]: BigQuerySourceBase does not 
propagate a Coder to AvroSource
https://github.com/apache/beam/issues/26041 [Bug]: Unable to create 
exactly-once Flink pipeline with stream source and file sink
https://github.com/apache/beam/issues/24776 [Bug]: Race condition in Python SDK 
Harness ProcessBundleProgress
https://github.com/apache/beam/issues/24389 [Failing Test]: 
HadoopFormatIOElasticTest.classMethod ExceptionInInitializerError 
ContainerFetchException
https://github.com/apache/beam/issues/24313 [Flaky]: 
apache_beam/runners/portability/portable_runner_test.py::PortableRunnerTestWithSubprocesses::test_pardo_state_with_custom_key_coder
https://github.com/apache/beam/issues/23944  beam_PreCommit_Python_Cron 
regularily failing - test_pardo_large_input flaky
https://github.com/apache/beam/issues/23709 [Flake]: Spark batch flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
https://github.com/apache/beam/issues/23525 [Bug]: Default PubsubMessage coder 
will drop message id and orderingKey
https://github.com/apache/beam/issues/22913 [Bug]: 
beam_PostCommit_Java_ValidatesRunner_Flink is flakes in 
org.apache.beam.sdk.transforms.GroupByKeyTest$BasicTests.testAfterProcessingTimeContinuationTriggerUsingState
https://github.com/apache/beam/issues/22605 [Bug]: Beam Python failure for 
dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest.test_metrics_it
https://github.com/apache/beam/issues/21714 
PulsarIOTest.testReadFromSimpleTopic is very flaky
https://github.com/apache/beam/issues/21706 Flaky timeout in github Python unit 
test action 
StatefulDoFnOnDirectRunnerTest.test_dynamic_timer_clear_then_set_timer
https://github.com/apache/beam/issues/21643 FnRunnerTest with non-trivial 
(order 1000 elements) numpy input flakes in non-cython environment
https://github.com/apache/beam/issues/21476 WriteToBigQuery Dynamic table 
destinations returns wrong tableId
https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: 
Connection refused
https://github.com/apache/beam/issues/21424 Java VR (Dataflow, V2, Streaming) 
failing: ParDoTest$TimestampTests/OnWindowExpirationTests
https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do not 
follow spec
https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit 
data at GC time
https://github.com/apache/beam/issues/21121 
apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it
 flakey
https://github.com/apache/beam/issues/21104 Flaky: