[
https://issues.apache.org/jira/browse/HUDI-6117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-6117:
----------------------------
Fix Version/s: 0.14.0
> Parallelize creation of initial file groups for MDT partitions
> --------------------------------------------------------------
>
> Key: HUDI-6117
> URL: https://issues.apache.org/jira/browse/HUDI-6117
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Prashant Wason
> Assignee: Prashant Wason
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.14.0
>
>
> When there are large number of file groups in a MDT partition (record index
> having billions of records can have 10k+ partitions, creating the initial log
> files in a for-loop can take a long time (100ms per create = 1000 seconds =
> 16mins) but routinely this latency is as high as 500msec / create.
> The initial file group creation can be optimized to be done parallelly
> speeding up MDT partition initialization.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
