[
https://issues.apache.org/jira/browse/HUDI-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-5611:
----------------------------
Description:
We discover a performance issue with savepoint when the metadata table is
enabled. It is due to unnecessary scanning of the metadata table when the
number of partitions is large. When the metadata table is enabled, in the
savepoint operation, for each partition, the metadata table is scanned, which
leads to a lot of S3 requests. The solution is to batch the list calls of all
partitions (HUDI-5485).
We need to revisit metadata-table-based file listing calls in a similar fashion
and replace them with batch lookup if needed.
was:
We discover a performance issue with savepoint when the metadata table is
enabled. It is due to unnecessary scanning of the metadata table when the
number of partitions is large. When the metadata table is enabled, in the
savepoint operation, for each partition, the metadata table is scanned, which
leads to a lot of S3 requests. The solution is to batch the list calls of all
partitions (HUDI-5485).
We need to revisit
> Revisit metadata-table-based file listing calls and use batch lookup instead
> ----------------------------------------------------------------------------
>
> Key: HUDI-5611
> URL: https://issues.apache.org/jira/browse/HUDI-5611
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Ethan Guo
> Priority: Major
>
> We discover a performance issue with savepoint when the metadata table is
> enabled. It is due to unnecessary scanning of the metadata table when the
> number of partitions is large. When the metadata table is enabled, in the
> savepoint operation, for each partition, the metadata table is scanned, which
> leads to a lot of S3 requests. The solution is to batch the list calls of
> all partitions (HUDI-5485).
>
> We need to revisit metadata-table-based file listing calls in a similar
> fashion and replace them with batch lookup if needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
