[
https://issues.apache.org/jira/browse/DRILL-8122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17485802#comment-17485802
]
Cong Luo commented on DRILL-8122:
---------------------------------
[~mrymar] Thank you for the report and assign this task. Do you already use
Kafka 2.x in the production environment?
> Change kafka metadata obtaining due to KAFKA-5697
> -------------------------------------------------
>
> Key: DRILL-8122
> URL: https://issues.apache.org/jira/browse/DRILL-8122
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.17.0, 1.18.0, 1.19.0
> Reporter: Maksym Rymar
> Assignee: Maksym Rymar
> Priority: Major
> Fix For: 1.20.0
>
>
> Starting from kafka 2.0 was changed behavior of the
> [KafkaConsumer#poll|https://kafka.apache.org/11/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll-long-]
> . In Drill was used one hack, where poll() is used only to update consumer
> metadata. But after the next change:
> [https://github.com/apache/kafka/pull/4855] this hack is not working now as
> poll() is not blocked more until metadata update yet more.
> Unfortunately, Kafka doesn't have public method to only update assignment
> metadata in consumer and it's on the discussion now whether to add it or not:
> [KIP-505|https://cwiki.apache.org/confluence/display/KAFKA/KIP-505%3A+Add+new+public+method+to+only+update+assignment+metadata+in+consumer]
> . Once it will be introduced we can use it but until that it needs to
> implement a workaround.
> Code to change:
> https://github.com/apache/drill/blob/15b2f52260e4f0026f2dfafa23c5d32e0fb66502/contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaGroupScan.java#L185
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
