Arnab Karmakar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23906
Change subject: IMPALA-13122: Add detailed file metadata statistics to table loading logs ...................................................................... IMPALA-13122: Add detailed file metadata statistics to table loading logs This patch enhances table loading logs to include comprehensive file metadata statistics, making it easier to identify small files issues and diagnose slow storage performance. The following statistics are now logged when loading file metadata: - Number of files and blocks - File sizes (min/avg/max) - Total file size - Modification times (min/max) - Access times (min/max) - Number of unique hosts and disks (HDFS/Ozone only) Example log output: Loaded file and block metadata for functional.alltypes partitions: year=2009/month=1, year=2009/month=10, year=2009/month=11, and 21 others. Time taken: 11.749ms. Files: 24, Blocks: 24, Total size: 478.45KB, File sizes (min/avg/max): 18.12KB/19.93KB/20.36KB, Modification times (min/max): 2025-12-17 01:40:53/2025-12-17 01:40:55, Access times (min/max): 2026-01-25 22:09:47/2026-01-25 22:09:48, Hosts: 3, Disks: 3 Testing: - Added Junit tests to verify statistics collection accuracy - Added new python end-to-end tests covering various cases Change-Id: I6f4592f173c047e5064058402f83be6d1f5c9a79 --- M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java A tests/metadata/test_file_metadata_stats.py 6 files changed, 391 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/23906/1 -- To view, visit http://gerrit.cloudera.org:8080/23906 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6f4592f173c047e5064058402f83be6d1f5c9a79 Gerrit-Change-Number: 23906 Gerrit-PatchSet: 1 Gerrit-Owner: Arnab Karmakar <[email protected]>
