Re: [ANNOUNCE] New committer: Ning Kang

Congratulations and thanks for all your contributions, Ning! Kenn On Tue, Mar 23, 2021 at 4:10 PM Valentyn Tymofieiev wrote: > Congratulations, Ning, very well deserved! > > On Tue, Mar 23, 2021 at 2:01 PM Chamikara Jayalath > wrote: > >> Congrats Ning! >> >> On Tue, Mar 23, 2021 at 1:23 PM Ru

Re: Make KafkaIO usable from Dataflow Template?

Is the idea that we should move away from ValueProviders then? *~Vincent* On Tue, Mar 23, 2021 at 4:11 PM Robert Bradshaw wrote: > I would encourage flex templates over further proliferation of > ValueProviders. > > On Tue, Mar 23, 2021 at 12:42 PM Alexey Romanenko < > aromanenko@gmail.com

Re: Null checking in Beam

I moved the checker framework from opt-out to opt-in [1]. You can opt in by passing "-PenableCheckerFramework=true" to the Gradle invocation. The checker still runs on all Jenkins CI builds. [1] https://github.com/apache/beam/pull/14301 On Wed, Mar 17, 2021 at 8:50 AM Kenneth Knowles wrote: > >

Re: Make KafkaIO usable from Dataflow Template?

I would encourage flex templates over further proliferation of ValueProviders. On Tue, Mar 23, 2021 at 12:42 PM Alexey Romanenko wrote: > I think you are right - now with SDF support in KafkaIO it should be > possible to determine the number of splits in the runtime and support > ValueProviders

Re: [ANNOUNCE] New committer: Ning Kang

Congratulations, Ning, very well deserved! On Tue, Mar 23, 2021 at 2:01 PM Chamikara Jayalath wrote: > Congrats Ning! > > On Tue, Mar 23, 2021 at 1:23 PM Rui Wang wrote: > >> Congrats! >> >> >> >> -Rui >> >> On Tue, Mar 23, 2021 at 1:05 PM Yichi Zhang wrote: >> >>> Congratulations Ning! >>> >>

Flaky test issue report

This is your daily summary of Beam's current flaky tests. These are P1 issues because they have a major negative impact on the community and make it hard to determine the quality of the software. BEAM-12020: :sdks:java:container:java8:docker failing missing licenses (https://issues.apache.o

P1 issues report

This is your daily summary of Beam's current P1 issues, not including flaky tests. See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the meaning and expectations around P1 issues. BEAM-12041: Project beam_PostCommit_Java_ValidatesRunner_ULR is failing (https://issues.

Re: [ANNOUNCE] New committer: Ning Kang

Congrats Ning! On Tue, Mar 23, 2021 at 1:23 PM Rui Wang wrote: > Congrats! > > > > -Rui > > On Tue, Mar 23, 2021 at 1:05 PM Yichi Zhang wrote: > >> Congratulations Ning! >> >> On Tue, Mar 23, 2021 at 1:00 PM Robin Qiu wrote: >> >>> Congratulations Ning! >>> >>> On Tue, Mar 23, 2021 at 12:56 PM

Re: [ANNOUNCE] New committer: Ning Kang

Congrats! -Rui On Tue, Mar 23, 2021 at 1:05 PM Yichi Zhang wrote: > Congratulations Ning! > > On Tue, Mar 23, 2021 at 1:00 PM Robin Qiu wrote: > >> Congratulations Ning! >> >> On Tue, Mar 23, 2021 at 12:56 PM Ahmet Altay wrote: >> >>> Congratulations Ning! >>> >>> On Tue, Mar 23, 2021 at 12

Re: [ANNOUNCE] New committer: Ning Kang

Congratulations Ning! On Tue, Mar 23, 2021 at 1:00 PM Robin Qiu wrote: > Congratulations Ning! > > On Tue, Mar 23, 2021 at 12:56 PM Ahmet Altay wrote: > >> Congratulations Ning! >> >> On Tue, Mar 23, 2021 at 12:38 PM Alexey Romanenko < >> aromanenko@gmail.com> wrote: >> >>> Congrats, Ning K

Re: [ANNOUNCE] New committer: Ning Kang

Congratulations Ning! On Tue, Mar 23, 2021 at 12:56 PM Ahmet Altay wrote: > Congratulations Ning! > > On Tue, Mar 23, 2021 at 12:38 PM Alexey Romanenko < > aromanenko@gmail.com> wrote: > >> Congrats, Ning Kang! Well deserved! >> Thank you for your contributions and users support! >> >> Alexe

Re: [ANNOUNCE] New committer: Ning Kang

Congratulations Ning! On Tue, Mar 23, 2021 at 12:38 PM Alexey Romanenko wrote: > Congrats, Ning Kang! Well deserved! > Thank you for your contributions and users support! > > Alexey > > On 23 Mar 2021, at 20:35, Pablo Estrada wrote: > > Hi all, > > Please join me and the rest of the Beam PMC in

Re: Make KafkaIO usable from Dataflow Template?

I think you are right - now with SDF support in KafkaIO it should be possible to determine the number of splits in the runtime and support ValueProviders in that way. CC: Boyuan Zhang > On 23 Mar 2021, at 18:18, Vincent Marquez wrote: > > Hello. I was looking at how to use the KafkaIO from a

Re: [ANNOUNCE] New committer: Ning Kang

Congrats, Ning Kang! Well deserved! Thank you for your contributions and users support! Alexey > On 23 Mar 2021, at 20:35, Pablo Estrada wrote: > > Hi all, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > Ning Kang. > > Ning has been working in Beam for a while

[ANNOUNCE] New committer: Ning Kang

Hi all, Please join me and the rest of the Beam PMC in welcoming a new committer: Ning Kang. Ning has been working in Beam for a while. He has contributed to the interactive experience of the Pyhton SDK, and developed a sidebar component, along with a release process for it. Ning has also helped

Re: Make KafkaIO usable from Dataflow Template?

Hi Vincent, ValueProviders required for Dataflow Classic templates. Dataflow Flex templates supports I/O interfaces beyond ValueProvider, and here are some Kafka examples: * [1] Kafka to BigQuery Dataflow Flex template * [2] Kafka to Pub/Sub Dataflow Flex template * [3] Apache Beam

Re: BEAM-9613 - BigQueryUtils and Avro

Logically, all JSON values are string. We often have put other objects in there, which I believe works simply because of the implicit .toString() method on those objects, but I'm not convinced this is really correct. On Tue, Mar 23, 2021 at 6:38 AM Brian Hulette wrote: > Thank you for digging in

Make KafkaIO usable from Dataflow Template?

Hello. I was looking at how to use the KafkaIO from a Dataflow Template, which requires all configuration options to be ValueProviders, which KafkaIO doesn't support currently. I saw this old PR: https://github.com/apache/beam/pull/6636/files I believe the reason this was never merged was there

Re: Add to Beam contributor list

Done Welcome to Beam! > On 23 Mar 2021, at 16:46, Isidro Martinez > wrote: > > > Hello, my Jira ID is: isidro.martinez > > > On Mon, Mar 22, 2021 at 9:40 PM Chamikara Jayalath > wrote: > What's you Jira ID ? > > On Fri, Mar 19, 2021 at 9:05 PM Isidro Martinez

Re: Add to Beam contributor list

Hello, my Jira ID is: isidro.martinez On Mon, Mar 22, 2021 at 9:40 PM Chamikara Jayalath wrote: > What's you Jira ID ? > > On Fri, Mar 19, 2021 at 9:05 PM Isidro Martinez < > isidro.marti...@wizeline.com> wrote: > >> I want to be added as a contributor to Beam Jira please. >> >> On Fri, Mar 19,

Re: Tutorial - How to run a Beam pipeline with Flink on Kubernetes Natively

See https://beam.apache.org/documentation/programming-guide/#inferring-schemas for details on how to use schemas with various types of Java classes (POJO, Java Beans, AutoValue). On Mon, Mar 22, 2021 at 3:01 PM Robert Bradshaw wrote: > Thanks, that looks quite useful. > > A not on POJOs, Seriali

Re: BEAM-9613 - BigQueryUtils and Avro

Thank you for digging into this and figuring out how this bug was introduced! In the long-term I think it would be preferable to avoid TableRow altogether in order to do a schema-aware read of avro data from BQ. We can go directly from Avro GenericRecord to Beam Rows now [1]. This would also be pr