[
https://issues.apache.org/jira/browse/FLINK-29617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lijie Wang reassigned FLINK-29617:
----------------------------------
Assignee: luoyuxia
> Cost too much time to start SourceCoordinator of hdfsFileSource when start
> JobMaster
> ------------------------------------------------------------------------------------
>
> Key: FLINK-29617
> URL: https://issues.apache.org/jira/browse/FLINK-29617
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Runtime / Coordination
> Affects Versions: 1.15.3
> Reporter: LI Mingkun
> Assignee: luoyuxia
> Priority: Major
> Labels: coordination, file-system
> Attachments: image-2022-10-13-19-02-18-555.png,
> image-2022-10-13-19-02-29-620.png, image-2022-10-13-19-02-35-422.png
>
>
> h1. Scenario:
> Our user use flink batch to compact small files in one day. Flink version :
> 1.15
> He split pipeline into 24 for each hour. So there are 24 source
>
> I find it costs too much time to start SourceCoordinator of hdfsFileSource
> when start JobMaster
>
> as follow:
>
> !image-2022-10-13-19-02-18-555.png!
>
> h1. Root Cause:
> I got the root cause after check:
> # AbstractFileSource will enumerateSplits when createEnumerator
> # NotSplittingRecursiveEnumerator need to get fileblockLocation of every
> fileblock which is a heavy IO operation
> !image-2022-10-13-19-02-29-620.png!
> !image-2022-10-13-19-02-35-422.png!
>
> h1. Suggestion
> # FileSource add option to disable location fetcher
> # Move location fetcher into IOExecutor
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
