[
https://issues.apache.org/jira/browse/DRILL-8122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489481#comment-17489481
]
ASF GitHub Bot commented on DRILL-8122:
---------------------------------------
rymarm opened a new pull request #2456:
URL: https://github.com/apache/drill/pull/2456
# [DRILL-8122](https://issues.apache.org/jira/browse/DRILL-8122): Change
kafka metadata obtaining due to KAFKA-5697
## Description
[`Consumer#poll(long)`](https://javadoc.io/static/org.apache.kafka/kafka-clients/3.1.0/org/apache/kafka/clients/consumer/Consumer.html#poll-long-)
is deprecated starting from kafka 2.0. In Drill, `Consumer#poll` is used in 2
places:
1. [By its direct purpose
](https://github.com/apache/drill/blob/15b2f52260e4f0026f2dfafa23c5d32e0fb66502/contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/MessageIterator.java#L82)
2. As an only one way to make a Kafka consumer [update metadata
](https://github.com/apache/drill/blob/15b2f52260e4f0026f2dfafa23c5d32e0fb66502/contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaGroupScan.java#L185)
Kafka [hasn't
implemented](https://cwiki.apache.org/confluence/display/KAFKA/KIP-505%3A+Add+new+public+method+to+only+update+assignment+metadata+in+consumer)
a separate method to update metadata. And new implementation
[Consumer#poll(Duration)](https://javadoc.io/static/org.apache.kafka/kafka-clients/3.1.0/org/apache/kafka/clients/consumer/Consumer.html#poll-java.time.Duration-)
doesn't work with a hack that Drill use: `poll(0)`, due to changed logic:
https://github.com/apache/kafka/pull/4855 . That is why I had to use a loop
with a timeout to workaround the absent separate method.
## Documentation
\-
## Testing
Unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Change kafka metadata obtaining due to KAFKA-5697
> -------------------------------------------------
>
> Key: DRILL-8122
> URL: https://issues.apache.org/jira/browse/DRILL-8122
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.17.0, 1.18.0, 1.19.0
> Reporter: Maksym Rymar
> Assignee: Maksym Rymar
> Priority: Major
> Fix For: 1.20.0
>
>
> Starting from kafka 2.0 was changed behavior of the
> [KafkaConsumer#poll|https://kafka.apache.org/11/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll-long-]
> . In Drill was used one hack, where poll(0) is used only to update consumer
> metadata. But after the next change:
> [https://github.com/apache/kafka/pull/4855] this hack is not working now as
> poll() is not blocked more until metadata update.
> Unfortunately, Kafka doesn't have public method to only update assignment
> metadata in consumer and it's on the discussion now whether to add it or not:
> [KIP-505|https://cwiki.apache.org/confluence/display/KAFKA/KIP-505%3A+Add+new+public+method+to+only+update+assignment+metadata+in+consumer]
> . Once it will be introduced we can use it but until that it needs to
> implement a workaround.
> Code to change:
> [https://github.com/apache/drill/blob/15b2f52260e4f0026f2dfafa23c5d32e0fb66502/contrib/storage-kafka/src/main/java/org/apache/drill/exec/store/kafka/KafkaGroupScan.java#L185]
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
