[
https://issues.apache.org/jira/browse/KAFKA-19403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17966775#comment-17966775
]
Matthias J. Sax commented on KAFKA-19403:
-----------------------------------------
The provide log snippets do not appear to be Kafka related. There is no classes
`KafkaBatchReaderFactoryWithRowBytesAccumulator` or `KafkaDataConsumer` in the
Kafka code. As a matter of fact, it seems `KafkaDataConsumer` is from Spark (->
`org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer`)
At this point, it's totally unclear if there is actually a bug or if this is a
configuration issue, or Event Hubs, or Databricks issue. – Last but not least,
3.3.1 is an older Kafka versions, so the main question would be, if you can
reproduce the issue in Kafka 4.0.0?
{quote}After conducting initial analysis with the Microsoft support team, they
confirmed that the root cause appears to be related to Kafka
{quote}
This is very vague, and we won't be able to help with a more concrete problem
statement, or other information. How can this issue be reproduced? Is there any
broker / client side ERROR logs or similar that indicate a bug?
If you suspect a issue in client-broker communication, the first step would be
to see if the client did send fetch request to the broker, and if the broker
did receive and process these request. Client/broker side debug (or trace)
level logs should contain this information.
> We're facing issue in Kafka while reading data from Azure event hubs through
> Azure Databricks
> ---------------------------------------------------------------------------------------------
>
> Key: KAFKA-19403
> URL: https://issues.apache.org/jira/browse/KAFKA-19403
> Project: Kafka
> Issue Type: Bug
> Components: consumer, network
> Affects Versions: 3.3.1
> Environment: Production
> Reporter: karthickthavasiraj09
> Priority: Major
>
> * We are experiencing significant slowness while reading data from Azure
> Event Hubs using Azure Databricks. After conducting initial analysis with the
> Microsoft support team, they confirmed that the root cause appears to be
> related to Kafka. We are reaching out for your assistance in investigating
> and resolving this issue.
> Below are the key findings and debug logs provided by the Microsoft team:
> ** The data read operation took *49 minutes* in total.
> ** Out of this, only one task(spark task) {*}Task 143 alone took 46
> minutes{*}, indicating a bottleneck in this specific task.
> ** The job duration was {*}49 minutes and 30 seconds{*}.
> Relevant Log Snippets:
> 25/04/15 14:21:44 INFO KafkaBatchReaderFactoryWithRowBytesAccumulator:
> Creating Kafka reader topicPartition=<topic-name>-0 fromOffset=16511904
> untilOffset=16658164,
> for queryId=dd660d4d-05cc-4a8e-8f93-d202ec78fec3
> runId=af7eb711-7310-4788-85b7-0977fc0756b7 batchId=73 taskId=143 partitionId=0
> 25/04/15 15:07:21 INFO KafkaDataConsumer:
> From Kafka topicPartition=<topic-name>-0
> groupId=spark-kafka-source-da79e0fc-8ee5-40f5-a127-7b31766b3022--1737876659-executor
> read 146260 records through 4314 polls (polled out 146265 records), taking
> 2526471821132 ns,
> over a timespan of 2736294068630 ns.
>
> Additionally, the thread stack trace indicates that the task was mostly
> waiting on Kafka to respond. See the following thread details captured during
> the slowness:
> Executor task launch worker for task 0.0 in stage 147.0 (TID 143)
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> ...
> kafkashaded.org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1759)
> ...
> org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer.fetchRecord(KafkaDataConsumer.scala:517)
> *We kindly request the Kafka team to look into this issue. The task appears
> to be blocked for an extended period during Kafka polling. Any insights into
> why Kafka is not responding promptly or recommendations for configuration
> changes or optimizations would be greatly appreciated.*
> *Please let us know if any additional information or diagnostics are required
> from our end.*
> *Thank you for your support.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
