Re: [ANNOUNCE] New committer: Damon Douglas

[Alex Kosolapov]

Congratulations, Damon!

From: Kenneth Knowles 
Reply-To: "dev@beam.apache.org" 
Date: Monday, April 24, 2023 at 12:52 PM
To: "dev@beam.apache.org" 
Subject: [EXTERNAL] [ANNOUNCE] New committer: Damon Douglas

Hi all,

Please join me and the rest of the Beam PMC in welcoming a new committer: Damon 
Douglas (damondoug...@apache.org)
Damon has contributed widely: Beam Katas, playground, infrastructure, and many 
IO connectors. Damon does lots of code review in addition to code. (yes, you 
can review code as a non-committer!)

Considering their contributions to the project over this timeframe, the Beam 
PMC trusts Damon with the responsibilities of a Beam committer. [1]

Thank you Damon! And we are looking to see more of your contributions!

Kenn, on behalf of the Apache Beam PMC

[1]
https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

Re: [ANNOUNCE] New committer: Damon Douglas

[Byron Ellis via dev]

Congrats Damon!

On Mon, Apr 24, 2023 at 8:57 PM Austin Bennett  wrote:

> thanks for all you do @Damon Douglas  !
>
> On Mon, Apr 24, 2023 at 1:00 PM Robert Burke  wrote:
>
>> Congratulations Damon!!!
>>
>> On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Damon Douglas (damondoug...@apache.org)
>>>
>>> Damon has contributed widely: Beam Katas, playground, infrastructure,
>>> and many IO connectors. Damon does lots of code review in addition to code.
>>> (yes, you can review code as a non-committer!)
>>>
>>> Considering their contributions to the project over this timeframe, the
>>> Beam PMC trusts Damon with the responsibilities of a Beam committer. [1]
>>>
>>> Thank you Damon! And we are looking to see more of your contributions!
>>>
>>> Kenn, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>>
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>

Re: [ANNOUNCE] New committer: Anand Inguva

[Byron Ellis via dev]

Congrats Anand!

On Mon, Apr 24, 2023 at 9:54 AM Ahmet Altay via dev 
wrote:

> Congratulations Anand!
>
> On Mon, Apr 24, 2023 at 8:05 AM Kerry Donny-Clark via dev <
> dev@beam.apache.org> wrote:
>
>> Great work Anand, this is well deserved.
>>
>>
>> On Mon, Apr 24, 2023 at 10:35 AM Yi Hu via dev 
>> wrote:
>>
>>> Congrats Anand!
>>>
>>> On Fri, Apr 21, 2023 at 3:54 PM Danielle Syse via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congratulations!

 On Fri, Apr 21, 2023 at 3:53 PM Damon Douglas via dev <
 dev@beam.apache.org> wrote:

> Congratulations Anand!
>
> On Fri, Apr 21, 2023 at 12:28 PM Ritesh Ghorse via dev <
> dev@beam.apache.org> wrote:
>
>> Congratulations Anand!
>>
>> On Fri, Apr 21, 2023 at 3:24 PM Ahmed Abualsaud via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congrats Anand!
>>>
>>> On Fri, Apr 21, 2023 at 3:18 PM Anand Inguva via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Thanks everyone. Really excited to be a part of Beam Committers.

 On Fri, Apr 21, 2023 at 3:07 PM XQ Hu via dev 
 wrote:

> Congratulations, Anand!!!
>
> On Fri, Apr 21, 2023 at 2:31 PM Jack McCluskey via dev <
> dev@beam.apache.org> wrote:
>
>> Congratulations, Anand!
>>
>> On Fri, Apr 21, 2023 at 2:28 PM Valentyn Tymofieiev via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congratulations!
>>>
>>> On Fri, Apr 21, 2023 at 8:19 PM Jan Lukavský 
>>> wrote:
>>>
 Congrats Anand!
 On 4/21/23 20:05, Robert Burke wrote:

 Congratulations Anand!

 On Fri, Apr 21, 2023, 10:55 AM Danny McCormick via dev <
 dev@beam.apache.org> wrote:

> Woohoo, congrats Anand! This is very well deserved!
>
> On Fri, Apr 21, 2023 at 1:54 PM Chamikara Jayalath <
> chamik...@apache.org> wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a
>> new committer: Anand Inguva (ananding...@apache.org)
>>
>> Anand has been contributing to Apache Beam for more than a
>> year and  authored and reviewed more than 100 PRs. Anand has 
>> been a core
>> contributor to Beam Python SDK and drove the efforts to support 
>> Python 3.10
>> and Python 3.11.
>>
>> Considering their contributions to the project over this
>> timeframe, the Beam PMC trusts Anand with the responsibilities 
>> of a Beam
>> committer. [1]
>>
>> Thank you Anand! And we are looking to see more of your
>> contributions!
>>
>> Cham, on behalf of the Apache Beam PMC
>>
>> [1]
>> https://beam.apache.org/contribute/become-a-committer
>> /#an-apache-beam-committer
>>
>

Re: [ANNOUNCE] New committer: Damon Douglas

[Austin Bennett]

thanks for all you do @Damon Douglas  !

On Mon, Apr 24, 2023 at 1:00 PM Robert Burke  wrote:

> Congratulations Damon!!!
>
> On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new committer:
>> Damon Douglas (damondoug...@apache.org)
>>
>> Damon has contributed widely: Beam Katas, playground, infrastructure, and
>> many IO connectors. Damon does lots of code review in addition to code.
>> (yes, you can review code as a non-committer!)
>>
>> Considering their contributions to the project over this timeframe, the
>> Beam PMC trusts Damon with the responsibilities of a Beam committer. [1]
>>
>> Thank you Damon! And we are looking to see more of your contributions!
>>
>> Kenn, on behalf of the Apache Beam PMC
>>
>> [1]
>>
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>

Re: [ANNOUNCE] New committer: Damon Douglas

[Ahmed Abualsaud via dev]

Congrats Damon!

On Mon, Apr 24, 2023 at 5:05 PM Kerry Donny-Clark via dev <
dev@beam.apache.org> wrote:

> Damon, you have done outstanding work to grow and improve Beam and the
> Beam community. Well done, well deserved!
>
> On Mon, Apr 24, 2023 at 4:39 PM XQ Hu via dev  wrote:
>
>> Congrats Damon!!!
>>
>> On Mon, Apr 24, 2023 at 4:34 PM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congrats Damon!
>>>
>>> On Mon, Apr 24, 2023 at 4:03 PM Ahmet Altay via dev 
>>> wrote:
>>>
 Congratulations Damon!

 On Mon, Apr 24, 2023 at 1:00 PM Robert Burke 
 wrote:

> Congratulations Damon!!!
>
> On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles 
> wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Damon Douglas (damondoug...@apache.org)
>>
>> Damon has contributed widely: Beam Katas, playground, infrastructure,
>> and many IO connectors. Damon does lots of code review in addition to 
>> code.
>> (yes, you can review code as a non-committer!)
>>
>> Considering their contributions to the project over this timeframe,
>> the Beam PMC trusts Damon with the responsibilities of a Beam committer. 
>> [1]
>>
>> Thank you Damon! And we are looking to see more of your contributions!
>>
>> Kenn, on behalf of the Apache Beam PMC
>>
>> [1]
>>
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>

Re: [ANNOUNCE] New committer: Damon Douglas

[Kerry Donny-Clark via dev]

Damon, you have done outstanding work to grow and improve Beam and the Beam
community. Well done, well deserved!

On Mon, Apr 24, 2023 at 4:39 PM XQ Hu via dev  wrote:

> Congrats Damon!!!
>
> On Mon, Apr 24, 2023 at 4:34 PM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Damon!
>>
>> On Mon, Apr 24, 2023 at 4:03 PM Ahmet Altay via dev 
>> wrote:
>>
>>> Congratulations Damon!
>>>
>>> On Mon, Apr 24, 2023 at 1:00 PM Robert Burke  wrote:
>>>
 Congratulations Damon!!!

 On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles  wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Damon Douglas (damondoug...@apache.org)
>
> Damon has contributed widely: Beam Katas, playground, infrastructure,
> and many IO connectors. Damon does lots of code review in addition to 
> code.
> (yes, you can review code as a non-committer!)
>
> Considering their contributions to the project over this timeframe,
> the Beam PMC trusts Damon with the responsibilities of a Beam committer. 
> [1]
>
> Thank you Damon! And we are looking to see more of your contributions!
>
> Kenn, on behalf of the Apache Beam PMC
>
> [1]
>
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>

Re: [ANNOUNCE] New committer: Damon Douglas

[Damon Douglas via dev]

Thank you, everyone. Remembering several years ago seeing my first Dataflow
job finally working  to now here, I'm really grateful to help preserve
and grow this community resource.

Best,

Damon

On Mon, Apr 24, 2023 at 1:39 PM XQ Hu via dev  wrote:

> Congrats Damon!!!
>
> On Mon, Apr 24, 2023 at 4:34 PM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Damon!
>>
>> On Mon, Apr 24, 2023 at 4:03 PM Ahmet Altay via dev 
>> wrote:
>>
>>> Congratulations Damon!
>>>
>>> On Mon, Apr 24, 2023 at 1:00 PM Robert Burke  wrote:
>>>
 Congratulations Damon!!!

 On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles  wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Damon Douglas (damondoug...@apache.org)
>
> Damon has contributed widely: Beam Katas, playground, infrastructure,
> and many IO connectors. Damon does lots of code review in addition to 
> code.
> (yes, you can review code as a non-committer!)
>
> Considering their contributions to the project over this timeframe,
> the Beam PMC trusts Damon with the responsibilities of a Beam committer. 
> [1]
>
> Thank you Damon! And we are looking to see more of your contributions!
>
> Kenn, on behalf of the Apache Beam PMC
>
> [1]
>
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>

Re: [ANNOUNCE] New committer: Damon Douglas

[XQ Hu via dev]

Congrats Damon!!!

On Mon, Apr 24, 2023 at 4:34 PM Danny McCormick via dev 
wrote:

> Congrats Damon!
>
> On Mon, Apr 24, 2023 at 4:03 PM Ahmet Altay via dev 
> wrote:
>
>> Congratulations Damon!
>>
>> On Mon, Apr 24, 2023 at 1:00 PM Robert Burke  wrote:
>>
>>> Congratulations Damon!!!
>>>
>>> On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles  wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Damon Douglas (damondoug...@apache.org)

 Damon has contributed widely: Beam Katas, playground, infrastructure,
 and many IO connectors. Damon does lots of code review in addition to code.
 (yes, you can review code as a non-committer!)

 Considering their contributions to the project over this timeframe, the
 Beam PMC trusts Damon with the responsibilities of a Beam committer. [1]

 Thank you Damon! And we are looking to see more of your contributions!

 Kenn, on behalf of the Apache Beam PMC

 [1]

 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

>>>

Re: [ANNOUNCE] New committer: Damon Douglas

[Danny McCormick via dev]

Congrats Damon!

On Mon, Apr 24, 2023 at 4:03 PM Ahmet Altay via dev 
wrote:

> Congratulations Damon!
>
> On Mon, Apr 24, 2023 at 1:00 PM Robert Burke  wrote:
>
>> Congratulations Damon!!!
>>
>> On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Damon Douglas (damondoug...@apache.org)
>>>
>>> Damon has contributed widely: Beam Katas, playground, infrastructure,
>>> and many IO connectors. Damon does lots of code review in addition to code.
>>> (yes, you can review code as a non-committer!)
>>>
>>> Considering their contributions to the project over this timeframe, the
>>> Beam PMC trusts Damon with the responsibilities of a Beam committer. [1]
>>>
>>> Thank you Damon! And we are looking to see more of your contributions!
>>>
>>> Kenn, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>>
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>

Re: [ANNOUNCE] New committer: Damon Douglas

[Chamikara Jayalath via dev]

Congrats Damon!

On Mon, Apr 24, 2023 at 1:03 PM Ahmet Altay via dev 
wrote:

> Congratulations Damon!
>
> On Mon, Apr 24, 2023 at 1:00 PM Robert Burke  wrote:
>
>> Congratulations Damon!!!
>>
>> On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Damon Douglas (damondoug...@apache.org)
>>>
>>> Damon has contributed widely: Beam Katas, playground, infrastructure,
>>> and many IO connectors. Damon does lots of code review in addition to code.
>>> (yes, you can review code as a non-committer!)
>>>
>>> Considering their contributions to the project over this timeframe, the
>>> Beam PMC trusts Damon with the responsibilities of a Beam committer. [1]
>>>
>>> Thank you Damon! And we are looking to see more of your contributions!
>>>
>>> Kenn, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>>
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>

Re: [ANNOUNCE] New committer: Damon Douglas

[Ahmet Altay via dev]

Congratulations Damon!

On Mon, Apr 24, 2023 at 1:00 PM Robert Burke  wrote:

> Congratulations Damon!!!
>
> On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new committer:
>> Damon Douglas (damondoug...@apache.org)
>>
>> Damon has contributed widely: Beam Katas, playground, infrastructure, and
>> many IO connectors. Damon does lots of code review in addition to code.
>> (yes, you can review code as a non-committer!)
>>
>> Considering their contributions to the project over this timeframe, the
>> Beam PMC trusts Damon with the responsibilities of a Beam committer. [1]
>>
>> Thank you Damon! And we are looking to see more of your contributions!
>>
>> Kenn, on behalf of the Apache Beam PMC
>>
>> [1]
>>
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>
>

Re: [ANNOUNCE] New committer: Damon Douglas

[Robert Burke]

Congratulations Damon!!!

On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles  wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new committer:
> Damon Douglas (damondoug...@apache.org)
>
> Damon has contributed widely: Beam Katas, playground, infrastructure, and
> many IO connectors. Damon does lots of code review in addition to code.
> (yes, you can review code as a non-committer!)
>
> Considering their contributions to the project over this timeframe, the
> Beam PMC trusts Damon with the responsibilities of a Beam committer. [1]
>
> Thank you Damon! And we are looking to see more of your contributions!
>
> Kenn, on behalf of the Apache Beam PMC
>
> [1]
>
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>

[ANNOUNCE] New committer: Damon Douglas

[Kenneth Knowles]

Hi all,

Please join me and the rest of the Beam PMC in welcoming a new committer:
Damon Douglas (damondoug...@apache.org)

Damon has contributed widely: Beam Katas, playground, infrastructure, and
many IO connectors. Damon does lots of code review in addition to code.
(yes, you can review code as a non-committer!)

Considering their contributions to the project over this timeframe, the
Beam PMC trusts Damon with the responsibilities of a Beam committer. [1]

Thank you Damon! And we are looking to see more of your contributions!

Kenn, on behalf of the Apache Beam PMC

[1]
https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

Re: Is there any way to set the parallelism of operators like group by, join?

[Kenneth Knowles]

FWIW I think parallelism is close enough to a resource. If you phrased it
like "how many CPUs can work independently" it is more closely related to
resources. Just like how many bits it takes to encode something is a
semantic property, but "RAM" is a resource.

I think a big role of resource hints is to be a bridge between the Beam
Model, which tries hard to only include essential information, to a
particular implementation which may not be able to autotune various
inessential/implementation details. Specifying parallelism to a runner that
still requires manual tuning of that seems like a fine use of this.

Kenn

On Fri, Apr 21, 2023 at 11:30 AM Jan Lukavský  wrote:

> Absolutely agree this is not something that should be part of the model.
> The ResourceHints is good place, but given how Pipeline might get fused
> (and though this might be under the control of a runner, basically all
> runners use the same code, because there is currently no reason why this
> should be runner-specifiic), there is a problem with how to resolve
> conflicting settings. Also it is somewhat questionable if parallelism is a
> "resource". It feels more like a runtime property. I tend to think that
> FlinkPipelineOptions could be a good place for that, because this seems to
> apply (mostly) to Flink batch runner.
> On 4/21/23 19:43, Robert Bradshaw via dev wrote:
>
> +1 to not requiring details like this in the Beam model. There is,
> however, the question of how to pass such implementation-detail specific
> hints to a runner that requires them. Generally that's done via
> ResourceHints or annotations, and while the former seems a good fit it's
> primarily focused on setting up the right context for user code (which GBK
> is not).
>
> A complete hack is to add an experiment like
> flink_parallelism_for_stage=STAGE_NAME:value. It'd be nice to do something
> cleaner.
>
>
> On Fri, Apr 21, 2023 at 10:37 AM Ning Kang via user 
> wrote:
>
>> Hi Jan,
>>
>> To generalize the per-stage parallelism configuration, we should have a
>> FR proposing the capability to explicitly set autoscaling (in this case,
>> fixed size per stage) policy in Beam pipelines.
>>
>> Per-step or per-stage parallelism, or fusion/optimization is not part of
>> the Beam model. They are [Flink] runner implementation details and should
>> be configured for each runner.
>>
>> Also, when building the pipeline, it's not clear what the fusion looks
>> like until the pipeline is submitted to a runner, thus making configuration
>> of the parallelism/worker-per-stage not straightforward.
>> Flink's parallelism settings can be found here
>> ,
>> it's still kind of a black box since you don't really know how many tasks
>> are actually spawned until you run a pipeline.
>>
>> That being said, if we have a general interface controlling how a
>> pipeline scales, each runner could adapt [auto]scaling in their own way.
>> For example, in a Flink job, each operator/stage's task slot is prorated
>> by their key numbers; the maximum parallelism is throttled by task slot
>> utilization.
>> Another example, in a Dataflow job, each stage horizontally scales by CPU
>> utilization; vertically scales by memory/disk utilization.
>>
>> +dev@beam.apache.org 
>> Let's use this thread to discuss how to configure a pipeline for runners
>> so that they can scale workers appropriately without exposing
>> runner-specific details to the Beam model.
>>
>> Ning.
>>
>>
>> On Thu, Apr 20, 2023 at 1:41 PM Jan Lukavský  wrote:
>>
>>> Hi Ning,
>>>
>>> I might have missed that in the discussion, but we talk about batch
>>> execution, am I right? In streaming, all operators (PTransforms) of a
>>> Pipeline are run in the same slots, thus the downsides are limited. You can
>>> enforce streaming mode using --streaming command-line argument. But yes,
>>> this might have other implications. For batch only it obviously makes sense
>>> to limit parallelism of a (fused) 'stage', which is not an transform-level
>>> concept, but rather a more complex union of transforms divided by shuffle
>>> barrier. Would you be willing to start a follow-up thread in @dev mailing
>>> list for this for deeper discussion?
>>>
>>>  Jan
>>> On 4/20/23 19:18, Ning Kang via user wrote:
>>>
>>> Hi Jan,
>>>
>>> The approach works when your pipeline doesn't have too many operators.
>>> And the operator that needs the highest parallelism can only use at most
>>> #total_task_slots / #operators resources available in the cluster.
>>>
>>> Another downside is wasted resources for other smaller operators who
>>> cannot make full use of task slots assigned to them. You might see only
>>> 1/10 tasks running while the other 9/10 tasks idle for an operator with
>>> parallelism 10, especially when it's doing some aggregation like a SUM.
>>>
>>> One redeeming method is that, for operators following another operator
>>> with high fanout, we can explicitly add a 

Re: Testing a pipeline with external transform

[Sahith Nallapareddy via dev]

Sounds good, let me try that thank you again!

Thanks,

Sahith

On Mon, Apr 24, 2023 at 2:11 PM Chamikara Jayalath 
wrote:

>
>
> On Mon, Apr 24, 2023 at 11:06 AM Sahith Nallapareddy 
> wrote:
>
>> Hello,
>>
>> Ah I missed that thank you! I am assuming for java jobs with java
>> external transform, we run a Java Job Service?
>>
>
> No, the job service is tied to the runner not to the pipeline SDK. The
> Java quickstart guide gives instructions for starting up a job service for
> Python portable runner for running Java jobs.
>
> - Cham
>
>
>
>>
>> Thanks,
>>
>> Sahith
>>
>> On Mon, Apr 24, 2023 at 2:01 PM Chamikara Jayalath 
>> wrote:
>>
>>> Have you looked into quickstart guides (Java [1], Python [2]) which also
>>> gives instructions for running using DirectRunner ?
>>>
>>> Thanks,
>>> Cham
>>>
>>> [1]
>>> https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/
>>> [2]
>>> https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/
>>>
>>> On Mon, Apr 24, 2023 at 10:57 AM Sahith Nallapareddy via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Hello,

 I was wondering the best way to test a pipeline with external
 transform. We were trying to use the DirectRunner but it seemed to hang on
 the external transform step. I was wondering if someone could point me to
 somewhere in the beam code where this is done or give some pointers on how
 to test a pipeline with an External step in it? I tried searching a bit but
 was unsuccessful, but maybe I was looking in the wrong place.

 Thanks,

 Sahith

>>>

Re: Testing a pipeline with external transform

[Chamikara Jayalath via dev]

On Mon, Apr 24, 2023 at 11:06 AM Sahith Nallapareddy 
wrote:

> Hello,
>
> Ah I missed that thank you! I am assuming for java jobs with java external
> transform, we run a Java Job Service?
>

No, the job service is tied to the runner not to the pipeline SDK. The Java
quickstart guide gives instructions for starting up a job service for
Python portable runner for running Java jobs.

- Cham



>
> Thanks,
>
> Sahith
>
> On Mon, Apr 24, 2023 at 2:01 PM Chamikara Jayalath 
> wrote:
>
>> Have you looked into quickstart guides (Java [1], Python [2]) which also
>> gives instructions for running using DirectRunner ?
>>
>> Thanks,
>> Cham
>>
>> [1]
>> https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/
>> [2]
>> https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/
>>
>> On Mon, Apr 24, 2023 at 10:57 AM Sahith Nallapareddy via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hello,
>>>
>>> I was wondering the best way to test a pipeline with external transform.
>>> We were trying to use the DirectRunner but it seemed to hang on the
>>> external transform step. I was wondering if someone could point me to
>>> somewhere in the beam code where this is done or give some pointers on how
>>> to test a pipeline with an External step in it? I tried searching a bit but
>>> was unsuccessful, but maybe I was looking in the wrong place.
>>>
>>> Thanks,
>>>
>>> Sahith
>>>
>>

Re: Testing a pipeline with external transform

[Sahith Nallapareddy via dev]

Hello,

Ah I missed that thank you! I am assuming for java jobs with java external
transform, we run a Java Job Service?

Thanks,

Sahith

On Mon, Apr 24, 2023 at 2:01 PM Chamikara Jayalath 
wrote:

> Have you looked into quickstart guides (Java [1], Python [2]) which also
> gives instructions for running using DirectRunner ?
>
> Thanks,
> Cham
>
> [1]
> https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/
> [2]
> https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/
>
> On Mon, Apr 24, 2023 at 10:57 AM Sahith Nallapareddy via dev <
> dev@beam.apache.org> wrote:
>
>> Hello,
>>
>> I was wondering the best way to test a pipeline with external transform.
>> We were trying to use the DirectRunner but it seemed to hang on the
>> external transform step. I was wondering if someone could point me to
>> somewhere in the beam code where this is done or give some pointers on how
>> to test a pipeline with an External step in it? I tried searching a bit but
>> was unsuccessful, but maybe I was looking in the wrong place.
>>
>> Thanks,
>>
>> Sahith
>>
>

Re: Testing a pipeline with external transform

[Chamikara Jayalath via dev]

Have you looked into quickstart guides (Java [1], Python [2]) which also
gives instructions for running using DirectRunner ?

Thanks,
Cham

[1]
https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/
[2]
https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/

On Mon, Apr 24, 2023 at 10:57 AM Sahith Nallapareddy via dev <
dev@beam.apache.org> wrote:

> Hello,
>
> I was wondering the best way to test a pipeline with external transform.
> We were trying to use the DirectRunner but it seemed to hang on the
> external transform step. I was wondering if someone could point me to
> somewhere in the beam code where this is done or give some pointers on how
> to test a pipeline with an External step in it? I tried searching a bit but
> was unsuccessful, but maybe I was looking in the wrong place.
>
> Thanks,
>
> Sahith
>

Testing a pipeline with external transform

[Sahith Nallapareddy via dev]

Hello,

I was wondering the best way to test a pipeline with external transform. We
were trying to use the DirectRunner but it seemed to hang on the external
transform step. I was wondering if someone could point me to somewhere in
the beam code where this is done or give some pointers on how to test a
pipeline with an External step in it? I tried searching a bit but was
unsuccessful, but maybe I was looking in the wrong place.

Thanks,

Sahith

Re: [ANNOUNCE] New committer: Anand Inguva

[Ahmet Altay via dev]

Congratulations Anand!

On Mon, Apr 24, 2023 at 8:05 AM Kerry Donny-Clark via dev <
dev@beam.apache.org> wrote:

> Great work Anand, this is well deserved.
>
>
> On Mon, Apr 24, 2023 at 10:35 AM Yi Hu via dev 
> wrote:
>
>> Congrats Anand!
>>
>> On Fri, Apr 21, 2023 at 3:54 PM Danielle Syse via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congratulations!
>>>
>>> On Fri, Apr 21, 2023 at 3:53 PM Damon Douglas via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congratulations Anand!

 On Fri, Apr 21, 2023 at 12:28 PM Ritesh Ghorse via dev <
 dev@beam.apache.org> wrote:

> Congratulations Anand!
>
> On Fri, Apr 21, 2023 at 3:24 PM Ahmed Abualsaud via dev <
> dev@beam.apache.org> wrote:
>
>> Congrats Anand!
>>
>> On Fri, Apr 21, 2023 at 3:18 PM Anand Inguva via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Thanks everyone. Really excited to be a part of Beam Committers.
>>>
>>> On Fri, Apr 21, 2023 at 3:07 PM XQ Hu via dev 
>>> wrote:
>>>
 Congratulations, Anand!!!

 On Fri, Apr 21, 2023 at 2:31 PM Jack McCluskey via dev <
 dev@beam.apache.org> wrote:

> Congratulations, Anand!
>
> On Fri, Apr 21, 2023 at 2:28 PM Valentyn Tymofieiev via dev <
> dev@beam.apache.org> wrote:
>
>> Congratulations!
>>
>> On Fri, Apr 21, 2023 at 8:19 PM Jan Lukavský 
>> wrote:
>>
>>> Congrats Anand!
>>> On 4/21/23 20:05, Robert Burke wrote:
>>>
>>> Congratulations Anand!
>>>
>>> On Fri, Apr 21, 2023, 10:55 AM Danny McCormick via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Woohoo, congrats Anand! This is very well deserved!

 On Fri, Apr 21, 2023 at 1:54 PM Chamikara Jayalath <
 chamik...@apache.org> wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Anand Inguva (ananding...@apache.org)
>
> Anand has been contributing to Apache Beam for more than a
> year and  authored and reviewed more than 100 PRs. Anand has been 
> a core
> contributor to Beam Python SDK and drove the efforts to support 
> Python 3.10
> and Python 3.11.
>
> Considering their contributions to the project over this
> timeframe, the Beam PMC trusts Anand with the responsibilities of 
> a Beam
> committer. [1]
>
> Thank you Anand! And we are looking to see more of your
> contributions!
>
> Cham, on behalf of the Apache Beam PMC
>
> [1]
> https://beam.apache.org/contribute/become-a-committer
> /#an-apache-beam-committer
>

Re: [ANNOUNCE] New committer: Anand Inguva

[Kerry Donny-Clark via dev]

Great work Anand, this is well deserved.


On Mon, Apr 24, 2023 at 10:35 AM Yi Hu via dev  wrote:

> Congrats Anand!
>
> On Fri, Apr 21, 2023 at 3:54 PM Danielle Syse via dev 
> wrote:
>
>> Congratulations!
>>
>> On Fri, Apr 21, 2023 at 3:53 PM Damon Douglas via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congratulations Anand!
>>>
>>> On Fri, Apr 21, 2023 at 12:28 PM Ritesh Ghorse via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congratulations Anand!

 On Fri, Apr 21, 2023 at 3:24 PM Ahmed Abualsaud via dev <
 dev@beam.apache.org> wrote:

> Congrats Anand!
>
> On Fri, Apr 21, 2023 at 3:18 PM Anand Inguva via dev <
> dev@beam.apache.org> wrote:
>
>> Thanks everyone. Really excited to be a part of Beam Committers.
>>
>> On Fri, Apr 21, 2023 at 3:07 PM XQ Hu via dev 
>> wrote:
>>
>>> Congratulations, Anand!!!
>>>
>>> On Fri, Apr 21, 2023 at 2:31 PM Jack McCluskey via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congratulations, Anand!

 On Fri, Apr 21, 2023 at 2:28 PM Valentyn Tymofieiev via dev <
 dev@beam.apache.org> wrote:

> Congratulations!
>
> On Fri, Apr 21, 2023 at 8:19 PM Jan Lukavský 
> wrote:
>
>> Congrats Anand!
>> On 4/21/23 20:05, Robert Burke wrote:
>>
>> Congratulations Anand!
>>
>> On Fri, Apr 21, 2023, 10:55 AM Danny McCormick via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Woohoo, congrats Anand! This is very well deserved!
>>>
>>> On Fri, Apr 21, 2023 at 1:54 PM Chamikara Jayalath <
>>> chamik...@apache.org> wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Anand Inguva (ananding...@apache.org)

 Anand has been contributing to Apache Beam for more than a year
 and  authored and reviewed more than 100 PRs. Anand has been a core
 contributor to Beam Python SDK and drove the efforts to support 
 Python 3.10
 and Python 3.11.

 Considering their contributions to the project over this
 timeframe, the Beam PMC trusts Anand with the responsibilities of 
 a Beam
 committer. [1]

 Thank you Anand! And we are looking to see more of your
 contributions!

 Cham, on behalf of the Apache Beam PMC

 [1]
 https://beam.apache.org/contribute/become-a-committer
 /#an-apache-beam-committer

>>>

Re: [ANNOUNCE] New committer: Anand Inguva

[Yi Hu via dev]

Congrats Anand!

On Fri, Apr 21, 2023 at 3:54 PM Danielle Syse via dev 
wrote:

> Congratulations!
>
> On Fri, Apr 21, 2023 at 3:53 PM Damon Douglas via dev 
> wrote:
>
>> Congratulations Anand!
>>
>> On Fri, Apr 21, 2023 at 12:28 PM Ritesh Ghorse via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congratulations Anand!
>>>
>>> On Fri, Apr 21, 2023 at 3:24 PM Ahmed Abualsaud via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congrats Anand!

 On Fri, Apr 21, 2023 at 3:18 PM Anand Inguva via dev <
 dev@beam.apache.org> wrote:

> Thanks everyone. Really excited to be a part of Beam Committers.
>
> On Fri, Apr 21, 2023 at 3:07 PM XQ Hu via dev 
> wrote:
>
>> Congratulations, Anand!!!
>>
>> On Fri, Apr 21, 2023 at 2:31 PM Jack McCluskey via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Congratulations, Anand!
>>>
>>> On Fri, Apr 21, 2023 at 2:28 PM Valentyn Tymofieiev via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Congratulations!

 On Fri, Apr 21, 2023 at 8:19 PM Jan Lukavský 
 wrote:

> Congrats Anand!
> On 4/21/23 20:05, Robert Burke wrote:
>
> Congratulations Anand!
>
> On Fri, Apr 21, 2023, 10:55 AM Danny McCormick via dev <
> dev@beam.apache.org> wrote:
>
>> Woohoo, congrats Anand! This is very well deserved!
>>
>> On Fri, Apr 21, 2023 at 1:54 PM Chamikara Jayalath <
>> chamik...@apache.org> wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Anand Inguva (ananding...@apache.org)
>>>
>>> Anand has been contributing to Apache Beam for more than a year
>>> and  authored and reviewed more than 100 PRs. Anand has been a core
>>> contributor to Beam Python SDK and drove the efforts to support 
>>> Python 3.10
>>> and Python 3.11.
>>>
>>> Considering their contributions to the project over this
>>> timeframe, the Beam PMC trusts Anand with the responsibilities of a 
>>> Beam
>>> committer. [1]
>>>
>>> Thank you Anand! And we are looking to see more of your
>>> contributions!
>>>
>>> Cham, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>> https://beam.apache.org/contribute/become-a-committer
>>> /#an-apache-beam-committer
>>>
>>

Beam High Priority Issue Report (30)

[beamactions]

This is your daily summary of Beam's current high priority issues that may need 
attention.

See https://beam.apache.org/contribute/issue-priorities for the meaning and 
expectations around issue priorities.

Unassigned P1 Issues:

https://github.com/apache/beam/issues/26343 [Bug]: 
apache_beam.io.gcp.bigquery_read_it_test.ReadAllBQTests.test_read_queries is 
flaky
https://github.com/apache/beam/issues/26329 [Bug]: BigQuerySourceBase does not 
propagate a Coder to AvroSource
https://github.com/apache/beam/issues/26126 [Failing Test]: 
beam_PostCommit_XVR_Samza permared validatesCrossLanguageRunnerGoUsingJava 
TestDebeziumIO_BasicRead
https://github.com/apache/beam/issues/26041 [Bug]: Unable to create 
exactly-once Flink pipeline with stream source and file sink
https://github.com/apache/beam/issues/25975 [Bug]: Reducing parallelism in 
FlinkRunner leads to a data loss
https://github.com/apache/beam/issues/24776 [Bug]: Race condition in Python SDK 
Harness ProcessBundleProgress
https://github.com/apache/beam/issues/24389 [Failing Test]: 
HadoopFormatIOElasticTest.classMethod ExceptionInInitializerError 
ContainerFetchException
https://github.com/apache/beam/issues/24313 [Flaky]: 
apache_beam/runners/portability/portable_runner_test.py::PortableRunnerTestWithSubprocesses::test_pardo_state_with_custom_key_coder
https://github.com/apache/beam/issues/23944  beam_PreCommit_Python_Cron 
regularily failing - test_pardo_large_input flaky
https://github.com/apache/beam/issues/23709 [Flake]: Spark batch flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
https://github.com/apache/beam/issues/22913 [Bug]: 
beam_PostCommit_Java_ValidatesRunner_Flink is flakes in 
org.apache.beam.sdk.transforms.GroupByKeyTest$BasicTests.testAfterProcessingTimeContinuationTriggerUsingState
https://github.com/apache/beam/issues/22605 [Bug]: Beam Python failure for 
dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest.test_metrics_it
https://github.com/apache/beam/issues/21706 Flaky timeout in github Python unit 
test action 
StatefulDoFnOnDirectRunnerTest.test_dynamic_timer_clear_then_set_timer
https://github.com/apache/beam/issues/21645 
beam_PostCommit_XVR_GoUsingJava_Dataflow fails on some test transforms
https://github.com/apache/beam/issues/21643 FnRunnerTest with non-trivial 
(order 1000 elements) numpy input flakes in non-cython environment
https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: 
Connection refused
https://github.com/apache/beam/issues/21424 Java VR (Dataflow, V2, Streaming) 
failing: ParDoTest$TimestampTests/OnWindowExpirationTests
https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do not 
follow spec
https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit 
data at GC time
https://github.com/apache/beam/issues/21121 
apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it
 flakey
https://github.com/apache/beam/issues/21104 Flaky: 
apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers
https://github.com/apache/beam/issues/20976 
apache_beam.runners.portability.flink_runner_test.FlinkRunnerTestOptimized.test_flink_metrics
 is flaky
https://github.com/apache/beam/issues/20108 Python direct runner doesn't emit 
empty pane when it should
https://github.com/apache/beam/issues/19814 Flink streaming flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful and 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
https://github.com/apache/beam/issues/19465 Explore possibilities to lower 
in-use IP address quota footprint.


P1 Issues with no update in the last week:

https://github.com/apache/beam/issues/26280 [Task]: Allow users to pass their 
own service name for google cloud profiler 
https://github.com/apache/beam/issues/23525 [Bug]: Default PubsubMessage coder 
will drop message id and orderingKey
https://github.com/apache/beam/issues/21714 
PulsarIOTest.testReadFromSimpleTopic is very flaky
https://github.com/apache/beam/issues/21708 beam_PostCommit_Java_DataflowV2, 
testBigQueryStorageWrite30MProto failing consistently
https://github.com/apache/beam/issues/21476 WriteToBigQuery Dynamic table 
destinations returns wrong tableId