[email protected] has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24642
Change subject: IMPALA-15229: Derive the EC Policy column from file metadata, not RPCs ...................................................................... IMPALA-15229: Derive the EC Policy column from file metadata, not RPCs SHOW TABLE STATS, SHOW PARTITIONS and SHOW FILES report an "EC Policy" column that is populated by FileSystemUtil.getErasureCodingPolicy(), which issues one NameNode RPC per path, uncached, from a serial loop. The statements therefore cost O(#partitions) or O(#files) round trips to populate a display-only column: on a table with ~105k partitions a single SHOW TABLE STATS produced ~105k serial getErasureCodingPolicy RPCs, and NameNode audit logs showed 952k such calls in a five minute window, tripping RPC-spike alerts on clusters that do not use erasure coding at all (where every call returns NONE). The information is already available for free: the directory listing that loads the file metadata returns HdfsFileStatus objects that carry the erasure coding policy of each file. This change records the policy id (a single byte) in the file descriptor at load time and derives the EC Policy column from it, eliminating the per-path RPCs. The recorded value also describes how the data is actually stored, rather than which policy the directory would assign to new files. Details: - FbFileDesc gains 'ec_policy_id: byte = 0 (id: 8)'. 0 is the id of the replication policy, i.e. "not erasure-coded or unknown". The field is additive with an explicit id, so catalog objects remain compatible in both directions and rolling upgrades keep working: descriptors written by older versions read as 0, which makes the new code fall back to the old per-path lookup for erasure-coded files. 'is_ec' is untouched and remains what scheduling relies on. - FileDescriptor records the id extracted from the FileStatus (a purely local call, no RPC) and exposes getEcPolicyId(). - Policy ids resolve to names locally through SystemErasureCodingPolicies for the built-in policies. Names of user-defined policies are fetched with a single getAllErasureCodingPolicies() call per filesystem and cached. - SHOW TABLE STATS / SHOW PARTITIONS aggregate the descriptors of the partition: NONE if no file is erasure-coded, the policy name if all files share one policy, and the new label MIXED otherwise. Partitions with no files, and erasure-coded files whose policy id is unknown (older metadata, or filesystems like Ozone whose listing does not carry the policy), fall back to the previous per-path lookup. - SHOW FILES resolves the policy from each file's descriptor directly. - DESCRIBE FORMATTED and the Iceberg table-level row keep the existing lookup, which is one RPC per statement. The per-file lookups in the Paimon SHOW FILES path iterate Paimon file handles that have no Impala file descriptor and are left for a follow-up. Behaviour change: for a partition whose directory has an erasure coding policy set but whose files are not erasure-coded (or the reverse), the column now describes the files instead of the directory attribute, and mixed contents report MIXED. Testing: - FileSystemUtilTest covers the policy id extraction from plain and HDFS file statuses and the local resolution of system policy names. - Existing EC end-to-end tests exercise the single-policy path on an erasure-coded minicluster via $ERASURECODE_POLICY. Change-Id: I35fc413c4a1f4730baab55406b13f5541f379707 --- M common/fbs/CatalogObjects.fbs M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/catalog/IcebergContentFileStoreTest.java M fe/src/test/java/org/apache/impala/common/FileSystemUtilTest.java 7 files changed, 184 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/24642/1 -- To view, visit http://gerrit.cloudera.org:8080/24642 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I35fc413c4a1f4730baab55406b13f5541f379707 Gerrit-Change-Number: 24642 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward <[email protected]>
