[
https://issues.apache.org/jira/browse/FLINK-29617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-29617:
-----------------------------------
Labels: coordination file-system stale-assigned (was: coordination
file-system)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Cost too much time to start SourceCoordinator of hdfsFileSource when start
> JobMaster
> ------------------------------------------------------------------------------------
>
> Key: FLINK-29617
> URL: https://issues.apache.org/jira/browse/FLINK-29617
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Runtime / Coordination
> Affects Versions: 1.15.3
> Reporter: LI Mingkun
> Assignee: luoyuxia
> Priority: Major
> Labels: coordination, file-system, stale-assigned
> Attachments: image-2022-10-13-19-02-18-555.png,
> image-2022-10-13-19-02-29-620.png, image-2022-10-13-19-02-35-422.png
>
>
> h1. Scenario:
> Our user use flink batch to compact small files in one day. Flink version :
> 1.15
> He split pipeline into 24 for each hour. So there are 24 source
>
> I find it costs too much time to start SourceCoordinator of hdfsFileSource
> when start JobMaster
>
> as follow:
>
> !image-2022-10-13-19-02-18-555.png!
>
> h1. Root Cause:
> I got the root cause after check:
> # AbstractFileSource will enumerateSplits when createEnumerator
> # NotSplittingRecursiveEnumerator need to get fileblockLocation of every
> fileblock which is a heavy IO operation
> !image-2022-10-13-19-02-29-620.png!
> !image-2022-10-13-19-02-35-422.png!
>
> h1. Suggestion
> # FileSource add option to disable location fetcher
> # Move location fetcher into IOExecutor
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
