[email protected] has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/24642 )
Change subject: IMPALA-15229: Derive the EC Policy column from file metadata, not RPCs ...................................................................... IMPALA-15229: Derive the EC Policy column from file metadata, not RPCs SHOW TABLE STATS, SHOW PARTITIONS and SHOW FILES report an "EC Policy" column that is populated by FileSystemUtil.getErasureCodingPolicy(), which issues one NameNode RPC per path, uncached, from a serial loop. The statements therefore cost O(#partitions) or O(#files) round trips to populate a display-only column: on a table with ~105k partitions a single SHOW TABLE STATS produced ~105k serial getErasureCodingPolicy RPCs, and NameNode audit logs showed 952k such calls in a five minute window, tripping RPC-spike alerts on clusters that do not use erasure coding at all (where every call returns NONE). The information is already available for free: the directory listing that loads the file metadata returns HdfsFileStatus objects that carry the erasure coding policy of each file. This change records the policy id (a single byte) in the file descriptor at load time and derives the EC Policy column from it, eliminating the per-path RPCs. The recorded value also describes how the data is actually stored, rather than which policy the directory would assign to new files. Details: - FbFileDesc gains 'ec_policy_id: byte = 0 (id: 8)'. 0 is the id of the replication policy, i.e. "not erasure-coded or unknown". The field is additive with an explicit id, so catalog objects remain compatible in both directions and rolling upgrades keep working: descriptors written by older versions read as 0, which makes the new code fall back to the old per-path lookup for erasure-coded files. 'is_ec' is untouched and remains what scheduling relies on. - FileDescriptor records the id extracted from the FileStatus (a purely local call, no RPC) and exposes getEcPolicyId(). - Policy ids resolve to names locally through SystemErasureCodingPolicies for the built-in policies. Names of user-defined policies are fetched with a single getAllErasureCodingPolicies() call per filesystem and cached. The cache is refreshed once when an id missing from it is looked up, so policies added to the NameNode while Impala is running still resolve without falling back to per-path lookups. - SHOW TABLE STATS / SHOW PARTITIONS aggregate the descriptors of the partition: NONE if no file is erasure-coded, the policy name if all files share one policy, and the new label MIXED otherwise, including when an erasure-coded file with an unknown policy id is mixed with plain files. Partitions with no files fall back to the previous per-path lookup, as do partitions where every file is erasure-coded but the policy id of some is unknown (older metadata, or filesystems like Ozone whose listing does not carry the policy). - SHOW FILES resolves the policy from each file's descriptor directly. - DESCRIBE FORMATTED and the Iceberg table-level row keep the existing lookup, which is one RPC per statement. The per-file lookups in the Paimon SHOW FILES path iterate Paimon file handles that have no Impala file descriptor and are left for a follow-up. Behaviour change: for a partition whose directory has an erasure coding policy set but whose files are not erasure-coded (or the reverse), the column now describes the files instead of the directory attribute, and mixed contents report MIXED. Testing: - FileSystemUtilTest covers the policy id extraction from plain and HDFS file statuses and the local resolution of system policy names. - FeFsTableTest covers the aggregation of a partition's descriptors: NONE, a single policy resolved locally, and the MIXED combinations, including erasure-coded files with unknown policy ids. - Existing EC end-to-end tests exercise the single-policy path on an erasure-coded minicluster via $ERASURECODE_POLICY. Change-Id: I35fc413c4a1f4730baab55406b13f5541f379707 --- M common/fbs/CatalogObjects.fbs M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java A fe/src/test/java/org/apache/impala/catalog/FeFsTableTest.java M fe/src/test/java/org/apache/impala/catalog/IcebergContentFileStoreTest.java M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java 8 files changed, 298 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/24642/2 -- To view, visit http://gerrit.cloudera.org:8080/24642 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I35fc413c4a1f4730baab55406b13f5541f379707 Gerrit-Change-Number: 24642 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
