[
https://issues.apache.org/jira/browse/IMPALA-13254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877881#comment-17877881
]
ASF subversion and git services commented on IMPALA-13254:
----------------------------------------------------------
Commit 53a452af669b022fd2ab19eb58a5dc5ea29aed78 in impala's branch
refs/heads/master from fulili
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=53a452af6 ]
IMPALA-13254: Optimize REFRESH for Iceberg tables
Considering that Iceberg's ContentFile is a collection of immutable
files, the current code logic has been simplified. The optimized
process is as follows:
1. For existing ContentFiles, directly reuse the existing file
descriptors.
2. For newly added ContentFiles that do not support block locations,
directly create file descriptors.
3. For newly added ContentFiles that support block locations,
choose between using a listLocatedStatus operation or calling
getFileBlockLocations one by one, based on the number of files.
A simple performance comparison test has been conducted in a
single-node environment. The test used the following data tables:
- non_partitioned_table: No partitions, containing 10,000 files
- partitioned_table_1: Contains 10,000 partitions, each with 1 file
- partitioned_table_2: Contains 300 partitions, each with 300 files
and scenarios tested:
- FULL: Perform REFRESH after executing INVALIDATE METADATA
- ADD_1_FILES: Insert 1 file using Hive and then perform REFRESH
- ADD_101_FILES: Insert 101 files using Hive and then perform REFRESH
The test results of the new version are as follows:
+------------------------+----------+-------------+----------------+
| Table | FULL | ADD_1_FILES | ADD_101_FILES |
+------------------------+----------+-------------+----------------+
| non_partitioned_table | 356.389ms| 40.015ms | 302.435ms |
| partitioned_table_1 | 288.798ms| 26.667ms | 33.035ms |
| partitioned_table_2 | 1s436ms | 237.057ms | 225.749ms |
+------------------------+----------+-------------+----------------+
The test results of the old version are as follows:
+------------------------+----------+-------------+----------------+
| Table | FULL | ADD_1_FILES | ADD_101_FILES |
+------------------------+----------+-------------+----------------+
| non_partitioned_table | 338ms | 57.156ms | 12s903ms |
| partitioned_table_1 | 281ms | 40.525ms | 12s743ms |
| partitioned_table_2 | 1s397ms | 336.965ms | 1m57s |
+------------------------+----------+-------------+----------------+
It can be observed that when the number of newly added files exceeds
iceberg_reload_new_files_threshold, REFRESH performance improves
significantly, while there is no noticeable change in other scenarios.
Change-Id: I8c99a28eb16275efdff52e0ea2711c0c6036719
Reviewed-on: http://gerrit.cloudera.org:8080/21608
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Optimizing incremental reload performance of Iceberg tables
> -----------------------------------------------------------
>
> Key: IMPALA-13254
> URL: https://issues.apache.org/jira/browse/IMPALA-13254
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Affects Versions: Impala 4.4.0
> Reporter: Fu Lili
> Assignee: Fu Lili
> Priority: Major
>
> When performing a {{REFRESH}} on an Iceberg table, if the number of changed
> files exceeds the {{iceberg_reload_new_files_threshold}} configuration
> (default is 100), a highly inefficient reload operation is triggered.
> The main issue with this code lies in the
> {{IcebergFileMetadataLoader.getFileStatuses}} function. During incremental
> loading, the {{listWithLocations}} parameter is always set to {{{}false{}}},
> resulting in {{fs.getFileStatus}} and {{fs.getFileBlockLocations}} operations
> being performed on each {{contentFile}} sequentially (if the filesystem
> supports {{{}StorageIds{}}}).
> To optimize this logic, the following changes can be made:
> # In the {{IcebergFileMetadataLoader.getFileStatuses}} function, always
> trigger {{parallelListing}} to quickly retrieve {{{}nameToFileStatus{}}},
> avoiding the sequential fetching for each {{{}contentFile{}}}.
> # Increase the default value of {{iceberg_reload_new_files_threshold}} to
> 1000. When changes are fewer than {{{}iceberg_reload_new_files_threshold{}}},
> perform a single RPC for each changed file to get the {{{}FileDescriptor{}}}.
> The average time for a single operation is 1 to 3 milliseconds, so 1000
> operations would take approximately 1 to 3 seconds, which is within a
> reasonable range.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
