Prashant Wason created HUDI-6117:
------------------------------------
Summary: Parallelize creation of initial file groups for MDT
partitions
Key: HUDI-6117
URL: https://issues.apache.org/jira/browse/HUDI-6117
Project: Apache Hudi
Issue Type: Improvement
Reporter: Prashant Wason
Assignee: Prashant Wason
When there are large number of file groups in a MDT partition (record index
having billions of records can have 10k+ partitions, creating the initial log
files in a for-loop can take a long time (100ms per create = 1000 seconds =
16mins) but routinely this latency is as high as 500msec / create.
The initial file group creation can be optimized to be done parallelly speeding
up MDT partition initialization.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
