Re: [Proposal] Add a new Beam example to ingest data from Kafka to Pub/Sub

Just a thought, but what if in the future there were templates for other runners? Then having a template folder would fit nicely no? We could even have a runner specific subfolder and maybe even a shared area for things that could be used by all templates for all runners? On Thu, 15 Oct 2020 at

Re: beam-sdks-java-bom.pom cannot be signed after upgrade to Gradle 6

My knowledge of this stuff has gotten rusty. What I remember: I had to do special work since Gradle didn't have support for generating BOMs. The generation process is kind of finicky because various stages need to run in order and that order isn't obvious based on the structure of the build rules;

Re: [Proposal] Add a new Beam example to ingest data from Kafka to Pub/Sub

Hi Ilya, I have added you to the "Contributors" role on Jira so you can be assigned tickets, and given you the ticket you filed since you are already solving it. Thanks! I have a very high level thought: Since Dataflow's "Flex Templates" feature is just any pipeline, perhaps the main pipeline

Re: beam-sdks-java-bom.pom cannot be signed after upgrade to Gradle 6

+Garrett Jones who appears to have been involved and +Michael Luckey who has touched this build.gradle (according to git) and is a bit of a build wizard. Kenn On Wed, Oct 14, 2020 at 7:34 PM Robin Qiu wrote: > Hi all, > > I am working on creating Beam 2.25.0 RC1. The repo I created ( >

beam-sdks-java-bom.pom cannot be signed after upgrade to Gradle 6

Hi all, I am working on creating Beam 2.25.0 RC1. The repo I created ( https://repository.apache.org/#stagingRepositories) failed to close because Missing Signature: > '/org/apache/beam/beam-sdks-java-bom/2.25.0/beam-sdks-java-bom-2.25.0.pom.asc' > does not exist for

Re: [Proposal] Add a new Beam example to ingest data from Kafka to Pub/Sub

Hi Beam Community, There was no feedback on the proposal, and I would like to submit PR for this proposal. I created a JIRA improvement to track this proposal and now submitting PR in the Beam

Re: Dataflow updates fail with "Coder has changed" error using KafkaIO with SchemaCoder

We are using an Avro Schema Registry and converting these schemas to Beam Schemas with `AvroUtils.toBeamSchema`: https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java#L314

Re: [DISCUSS][BEAM-10670] Migrating BoundedSource/UnboundedSource to execute as a Splittable DoFn for non-portable Java runners

Thanks Alexey, that is correct. On Wed, Oct 14, 2020 at 10:33 AM Alexey Romanenko wrote: > Thanks Luke, just I guess that the proper link should be this one: > > https://docs.google.com/document/d/1kpn0RxqZaoacUPVSMYhhnfmlo8fGT-p50fEblaFr2HE > > On 13 Oct 2020, at 00:23, Luke Cwik wrote: > > I

Re: [DISCUSS][BEAM-10670] Migrating BoundedSource/UnboundedSource to execute as a Splittable DoFn for non-portable Java runners

Thanks Luke, just I guess that the proper link should be this one: https://docs.google.com/document/d/1kpn0RxqZaoacUPVSMYhhnfmlo8fGT-p50fEblaFr2HE > On 13 Oct 2020, at 00:23, Luke Cwik wrote: > > I have a draft[1] off the blog ready. Please take a look. > > 1: >

Re: [DISCUSS] Move Avro dependency out of core Beam

It sounds like there's a consensus around making Beam core work with either Avro 1.8 or 1.9. It looks like +Tomo Suzuki actually accomplished this already back in January for BEAM-9144 [1] and a user has run pipelines on Dataflow with Avro 1.9. Would we need to do anything else to unblock using

Re: [Proposal] Website Revamp project

Hi Everyone, We're ready to start work on the revamp of the website, we'll use the PRD shared in this thread previously. Polidea will be the team working on this revamp and we'll be bringing designs and proposals to the community for review as the project progresses. Thank you! Gris On

Re: Self-checkpoint Support on Portable Flink

Duplicates cannot happen because the state of all operators will be rolled back to the latest checkpoint, in case of failures. On 14.10.20 06:31, Reuven Lax wrote: Does this mean that we have to deal with duplicate messages over the back edge? Or will that not happen, since duplicates mean