Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Thank you for quick answers, Utkarsh, but unfortunately, I don’t see the real cause of this right now. Seems like, it will require some remote debugging on your site to see what workers are actually doing. > On 1 Feb 2022, at 22:59, Utkarsh Parekh wrote: > > If you tested earlier with the

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

And I also get this error occasionally when I execute a streaming pipeline with a new cluster instead of an existing cluster. https://issues.apache.org/jira/browse/BEAM-12032?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel On Tue, Feb 1, 2022 at 1:59 PM Utkarsh Parekh wrote:

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

If you tested earlier with the same stack, which version did you use? *Can you enable debug logs to check what’s happening there? *So far the following warning was received from from log4j which I received from log4j on Databricks (no errors other than that). *Can you make sure that there is no

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Well, personally I didn’t test with this version, but it should be fine… Can you enable debug logs to check what’s happening there? Can you make sure that there is no issue with firewall or something? Can you run this pipeline locally against a real Kafka server, not Azure Event Hub, to make

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Sorry I sent the last message in a hurry. Here is the Beam java to kafka: Is something missing here? org.apache.beam beam-sdks-java-io-kafka 2.35.0 On Tue, Feb 1, 2022 at 9:01 AM Utkarsh Parekh wrote: > Here it is > > > org.apache.kafka > kafka-clients > 2.8.0 > >

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Here it is org.apache.kafka kafka-clients 2.8.0 On Tue, Feb 1, 2022 at 8:53 AM Alexey Romanenko wrote: > Hmm, this is strange. Which version of Kafka client do you use while > running it with Beam? > > On 1 Feb 2022, at 16:56, Utkarsh Parekh > wrote: > > Hi Alexey, > > First of

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Hmm, this is strange. Which version of Kafka client do you use while running it with Beam? > On 1 Feb 2022, at 16:56, Utkarsh Parekh wrote: > > Hi Alexey, > > First of all, thank you for the response! Yes I did have it in Consumer > configuration and try to increase "session.timeout". > >

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Hi Alexey, First of all, thank you for the response! Yes I did have it in Consumer configuration and try to increase "session.timeout". >From consumer side so far I've following settings: props.put("sasl.mechanism", SASL_MECHANISM); props.put("security.protocol", SECURITY_PROTOCOL);

Re: Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Hi Utkarsh, Can it be related to this configuration problem? https://docs.microsoft.com/en-us/azure/event-hubs/apache-kafka-troubleshooting-guide#no-records-received Did you check timeout

Running small app using Apache Beam, KafkaIO, Azure EventHuband Databricks Spark

Hello, I'm doing POC with KafkaIO and spark runner on Azure Databricks. I'm trying to create a simple streaming app with Apache Beam, where it reads data from an Azure event hub and produces messages into another Azure event hub. I'm creating and running spark jobs on Azure Databricks. The