Balaji Varadarajan created HUDI-1108:
----------------------------------------
Summary: Allow parallel listing of dataset partitions for various
actions during write
Key: HUDI-1108
URL: https://issues.apache.org/jira/browse/HUDI-1108
Project: Apache Hudi
Issue Type: Improvement
Components: Writer Core
Reporter: Balaji Varadarajan
Fix For: 0.6.0
Currently we rely on FSUtils.getAllPartitionPaths to return all partitions of a
dataset. This implementation is slow for AWS S3 file-systems. We need to
provide option to allow the listing to be parallelizable.
GH Issue : [https://github.com/apache/hudi/issues/1837]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
