Re: Offset access in Kafka messages in Python

HI Ondrej, You can retrieve metadata info using SDF (here is the video from Israel Herraiz) and the kafka-python PY package. Utkarsh On Tue, Mar 26, 2024 at 8:00 AM Ondřej Pánek wrote: > Hello

Re: Request to join slack channel

Hello, Can someone add me in slack channel as well? Thanks, Utkarsh On Wed, Feb 21, 2024 at 11:10 AM George Dekermenjian wrote: > Me too please - slack channel. > > > On Wed, Feb 21, 2024 at 19:43 Geddy Schellevis > wrote: > >> Hey Valentyn, >> >> can you add me to slack channel as well? >>

Re: Beam 2.53.0 Release

Awesome!! On Fri, Jan 5, 2024 at 2:11 PM XQ Hu via user wrote: > Great! And thank you! > > On Fri, Jan 5, 2024 at 2:49 PM Jack McCluskey via user < > user@beam.apache.org> wrote: > >> We are happy to present the new 2.53.0 release of Beam. >> This release includes both improvements and new

Where to specify trust.jks

Hi, I'm testing a streaming app using kafka, Dafaflow, and Apache beam [Python]. "Error message from worker: org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

Apache beam

Hi, I'm writing a simple streaming beam application. The application job is doing following tasks: 1. Reads data from GCS bucket (project 1) and loads into Kafka topic 2. Reads data from Kafka topic and loads into BigQUery (project 3) Composer running in Project 1 Data Flow running in project 2

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

And I also get this error occasionally when I execute a streaming pipeline with a new cluster instead of an existing cluster. https://issues.apache.org/jira/browse/BEAM-12032?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel On Tue, Feb 1, 2022 at 1:59 PM Utkarsh Parekh wrote

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

emotely the worker process. > > On 1 Feb 2022, at 19:18, Utkarsh Parekh > wrote: > > Sorry I sent the last message in a hurry. Here is the Beam java to kafka: > Is something missing here? > > > org.apache.beam > beam-sdks-java-io-kafka > 2.35.0 >

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Sorry I sent the last message in a hurry. Here is the Beam java to kafka: Is something missing here? org.apache.beam beam-sdks-java-io-kafka 2.35.0 On Tue, Feb 1, 2022 at 9:01 AM Utkarsh Parekh wrote: > Here it is > > > org.apache.kafka > kafka-clie

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Here it is org.apache.kafka kafka-clients 2.8.0 On Tue, Feb 1, 2022 at 8:53 AM Alexey Romanenko wrote: > Hmm, this is strange. Which version of Kafka client do you use while > running it with Beam? > > On 1 Feb 2022, at 16:56, Utkarsh Parekh > wrote: > >

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

t") .load() Utkarsh On Tue, Feb 1, 2022 at 6:20 AM Alexey Romanenko wrote: > Hi Utkarsh, > > Can it be related to this configuration problem? > > https://docs.microsoft.com/en-us/azure/event-hubs/apache-kafka-troubleshooting-guide#no-records-received > > Did you check time

Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Hello, I'm doing POC with KafkaIO and spark runner on Azure Databricks. I'm trying to create a simple streaming app with Apache Beam, where it reads data from an Azure event hub and produces messages into another Azure event hub. I'm creating and running spark jobs on Azure Databricks. The