[RFC][design/idea] CDAP plugins support in Apache Beam

TL:DR: We want to develop support for Apache CDAP batch and streaming plugins to enrich Apache Beam connectors to external applications. Please review the design[1] to help us bring CDAP plugins integrations into Apache Beam. Hi all, I along with a few community members thought of an idea to

Re: KafkaIO SSL issue

tream used in this case, is it Kafka code, Beam code, or your consumerFactoryFn? Brian On Mon, May 24, 2021 at 4:01 AM Ilya Kozyrev mailto:ilya.kozy...@akvelon.com>> wrote: Hi community, We have an issue with KafkaIO in the case of using a secure connection SASL SSL to the Confluent Kafk

KafkaIO SSL issue

Hi community, We have an issue with KafkaIO in the case of using a secure connection SASL SSL to the Confluent Kafka 5.5.1. When we trying to configure the Kafka consumer using consumerFactoryFn, we have an irregular issue related to certificate reads from the file system. Irregular means,

Re: [Proposal] Add a new Beam example to ingest data from Kafka to Pub/Sub

ya On 26 Oct 2020, 17:25 +0300, Ilya Kozyrev Akvelon , wrote: Hi everyone, We completed the development of the template. Could someone kindly help with reviewing the PR<https://github.com/apache/beam/pull/13112>? Thank you, Ilya On 16 Oct 2020, 17:25 +0300, Ilya Kozyrev Akvelon , wrote: Tha

Re: [Proposal] Add a new Beam example to ingest data from Kafka to Pub/Sub

Hi everyone, We completed the development of the template. Could someone kindly help with reviewing the PR<https://github.com/apache/beam/pull/13112>? Thank you, Ilya On 16 Oct 2020, 17:25 +0300, Ilya Kozyrev Akvelon , wrote: Thanks a lot for your comments! I see your point reg

Re: [Proposal] Add a new Beam example to ingest data from Kafka to Pub/Sub

ts inherit dependencies from their parents. So you could move your subproject under the `examples/java` directory and it should still build exactly the same. On Thu, Oct 15, 2020 at 7:34 AM Ilya Kozyrev mailto:ilya.kozy...@akvelon.com>> wrote: Hi! Thanks for joining the discussion a lot!

Re: [Proposal] Add a new Beam example to ingest data from Kafka to Pub/Sub

stand this is not your primary goal, considering the contribution. I just want to open this for discussion. Kenn *In fact, the JSON is very generic. It is not really "Google specific" in concept, just in practice. On Wed, Oct 14, 2020 at 12:14 PM Ilya Kozyrev mailto:ilya.

Re: [Proposal] Add a new Beam example to ingest data from Kafka to Pub/Sub

thub.com/apache/beam/pull/13112> ? Thank you, Ilya On 7 Oct 2020, at 21:23, Ilya Kozyrev mailto:ilya.kozy...@akvelon.com>> wrote: Hi Beam Community, I have a proposal to add Apache Beam example that is a template to ingest data from Apache Kafka to Google Cloud Pub/Sub. More de

[Proposal] Add a new Beam example to ingest data from Kafka to Pub/Sub

Hi Beam Community, I have a proposal to add Apache Beam example that is a template to ingest data from Apache Kafka to Google Cloud Pub/Sub. More detailed information about the proposed template can be found in

Contributor permission for Beam Jira tickets

Hi! This is Ilya from Akvelon. I working on examples for using Beam with Kafka and Pub/Sub both. Could you please add me as a contributor to Beam's Jira issue tracker? I would like to create/assign tickets. My jira username: IKozyrev Thanks, Ilya