Zoltán Borók-Nagy created IMPALA-15428:
------------------------------------------
Summary: HdfsPartition keeps per-partition host and host:disk sets
that are only used for a log line
Key: IMPALA-15428
URL: https://issues.apache.org/jira/browse/IMPALA-15428
Project: IMPALA
Issue Type: Improvement
Components: Catalog
Reporter: Zoltán Borók-Nagy
IMPALA-13122 added two sets to FeFsTable.FileMetadataStats
(FeFsTable.java:149-155):
{{uniqueHostIndices}} and {{uniqueHostDiskPairs}}. A FileMetadataStats object
is stored in
every HdfsPartition. The sets are:
* filled for every file and block during loading;
* deep-copied in set() on every {{HdfsPartition.Builder(partition)}}
(FeFsTable.java:194);
* merged across partitions (FeFsTable.java:208).
Their only consumer is toLogString(), i.e. the host statistics in the
table-loading log
line. The sets are retained in catalogd for the lifetime of the table, and they
cost CPU
on every partition rebuild. This affects HDFS and Ozone tables; object-store
files have
no block locations.
h3. Measurements
Standalone harness using the real classes, retained memory per partition:
||Files per partition||Current||With fix||
|10|3288 B|100 B|
|100|20.9 KB|100 B|
|100 × 4 blocks|51.8 KB|100 B|
These numbers are for a 100-host cluster; with 1000 hosts, 10 files per
partition cost
about 4.3 KB. With 10 files per partition on 100 hosts, this saves about 319 MB
of catalogd heap per 100k
partitions (the 100k-partition run measured 318.8 MB). Builder set() becomes 36x
faster. In the same harness, per-partition catalogd load CPU is 1.2–2.4x lower
and REFRESH
1.7–2.8x lower. With use_local_catalog=false, incremental coordinator updates
become
87x faster (5.5 → 0.06 us per partition).
h3. Proposed fix
* Collect the host statistics in a transient aggregate inside the loader
(ParallelFileMetadataLoader), used only for the log line.
* Strip or skip the sets before the stats object is stored in the partition,
and make
fromThrift()/set()/merge() ignore them.
* Keep the public getters returning counts, in case extensions use them.
The host counts in the log line will then describe the files listed in this
load only.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
