[
https://issues.apache.org/jira/browse/FLINK-27137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17519591#comment-17519591
]
Gyula Fora commented on FLINK-27137:
------------------------------------
[~arvid] "so I'd be inclined to close this ticket with "Won't Do" : let's try
to keep a friendly attitude, I never suggested that you would need to do
anything, I simply want to discuss this :)
As for the context of this ticket. Some managed Kafka environments (including
ours) use custom Kafka client implementations. This might restrict the use of
certain APIs / functions such as the AdminClient. From an architectural
standpoint we should generally try to limit what external APIs we use to
increase compatibility between services. Since Flink will absolutely need the
KafkaConsumer API for the main purpose of consuming / producing data there is
clearly no way around that, but if we can limit the usage of new client
interfaces such as AdminClient that can be beneficial.
Regular kafka consumers and producer applications do not usually use the
AdminClient (as the name also suggests) and one could argue that Flink should
try to behave as much as regular kafka applications to increase compatibility.
Looking at the OffsetRetriever there are 4 requirements here: getting committed
group offsets, getting begin/end offsets and gettings offsets for timestamp.
This could all be implemented using the KafkaConsumer (at least looking at the
newer API
[https://kafka.apache.org/30/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html)]
So if we find that everything that we are trying to achieve can be done by the
APIs that we already use (KafkaConsumer) then it would best to avoid using the
AdminClient which is basically just an extra API dependency at that point
without practical gain.
If we see that there are some future features that cannot be covered by the
KafkaConsumer we should at least make this logic pluggable / optional.
> Remove usage of AdminClient from KafkaSource logic
> --------------------------------------------------
>
> Key: FLINK-27137
> URL: https://issues.apache.org/jira/browse/FLINK-27137
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
> Affects Versions: 1.15.0, 1.14.4
> Reporter: Gyula Fora
> Priority: Major
>
> Parts of the KafkaSource logic (specifically the KafkaSourceEnumerator) uses
> the Kafka AdminClient instead of the KafkaConsumer.
> It seems that the KafkaConsumer already provides all the necessary
> information that the enumerator needs so there is no reason for introducing
> the AdminClient.
> In some environments using the AdminClient can be problematic even if we are
> not using certaing features.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
