[
https://issues.apache.org/jira/browse/FLINK-40396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser reassigned FLINK-40396:
--------------------------------------
Assignee: Edson Wang
> Create SplitReaders on the split fetcher thread instead of the caller thread
> ----------------------------------------------------------------------------
>
> Key: FLINK-40396
> URL: https://issues.apache.org/jira/browse/FLINK-40396
> Project: Flink
> Issue Type: Improvement
> Reporter: Edson Wang
> Assignee: Edson Wang
> Priority: Major
>
> Follow-up promised in
> https://github.com/apache/flink-connector-kafka/pull/290 (FLINK-36434).
> SplitFetcherManager currently invokes the split-reader supplier on the thread
> that creates
> the SplitFetcher (the source reader thread), not on the fetcher thread that
> will use the
> reader. Clients constructed inside a SplitReader therefore capture the wrong
> thread at
> construction time. For Kafka this meant the KafkaConsumer was created on a
> thread it never
> runs on afterwards; FLINK-36434 fixed it locally by making the consumer lazy
> inside
> KafkaPartitionSplitReader.
> Deferring the supplier call itself onto the fetcher thread in
> flink-connector-base would fix
> this class of issue for all connectors, so no connector needs its own
> laziness workaround.
> Sketch: move the supplier invocation from SplitFetcher construction into the
> fetcher
> thread's startup (first run), while keeping construction failures propagating
> through the
> existing fetcher error-handling path (today a failing supplier throws
> synchronously on the
> caller thread — connectors relying on that would see the failure
> asynchronously instead,
> which needs to be called out in the compatibility notes).
> I'm happy to work on this once assigned.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
