Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23906 )
Change subject: IMPALA-13122: Add detailed file metadata statistics to table loading logs ...................................................................... IMPALA-13122: Add detailed file metadata statistics to table loading logs This patch enhances table loading logs to include comprehensive file metadata statistics, making it easier to identify small files issues and diagnose slow storage performance. The following statistics are now logged when loading file metadata: - Number of files and blocks - File sizes (min/avg/max) - Total file size - Modification times (min/max) - Access times (min/max) - Number of host:disk pairs (HDFS/Ozone only) Example log output: Loaded file and block metadata for functional.alltypes partitions: year=2009/month=1, year=2009/month=10, year=2009/month=11, and 21 others. Time taken: 13.474ms. Files: 24, Blocks: 24, Total size: 478.45KB, File sizes (min/avg/max): 18.12KB/19.93KB/20.36KB, Modification times (min/max): 2026-02-17 01:28:17/2026-02-17 01:28:21, Access times (min/max): 2026-02-24 00:58:39/2026-02-24 00:58:39, Hosts: 3, Host:Disk pairs: 3 Testing: - Added Junit tests to verify statistics collection accuracy - Added new python end-to-end tests covering various cases Change-Id: I6f4592f173c047e5064058402f83be6d1f5c9a79 Reviewed-on: http://gerrit.cloudera.org:8080/23906 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java A tests/custom_cluster/test_file_metadata_stats.py 6 files changed, 373 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/23906 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6f4592f173c047e5064058402f83be6d1f5c9a79 Gerrit-Change-Number: 23906 Gerrit-PatchSet: 13 Gerrit-Owner: Arnab Karmakar <[email protected]> Gerrit-Reviewer: Arnab Karmakar <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Pranav Lodha <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
