[
https://issues.apache.org/jira/browse/IMPALA-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115180#comment-18115180
]
ASF subversion and git services commented on IMPALA-15229:
----------------------------------------------------------
Commit a921854ac4bd6c2e8a5b2ba896aa639af12f0a08 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a921854ac ]
IMPALA-15291: Fix EC Policy column when block locations are not preloaded
IMPALA-15229 made the "EC Policy" column (SHOW TABLE STATS, SHOW
PARTITIONS, SHOW FILES) derive from FileDescriptor.getIsEc(). That flag
was only set on the block-loading path (create()); createWithNoBlocks(),
used when block-location preloading is disabled, hardcoded is_ec=false,
so erasure-coded files loaded that way reported NONE.
createWithNoBlocks now reads the erasure-coding and encryption flags from
the FileStatus, like create(). Wherever the listing carries a real (HDFS)
FileStatus, getIsEc()/getIsEncrypted() are now correct at no extra cost.
Iceberg is the exception: on the no-preload path it builds the FileStatus
synthetically (size and path only) to avoid a per-file stat RPC, so it
carries no erasure-coding info. Rather than re-add a per-file NameNode RPC
that could still be wrong (a directory's policy is not necessarily the
file's), SHOW FILES reports "N/A" for such descriptors. Knowing the EC
policy in such cases wouldn't have too much added value as we already
schedule blockless splits as remote reads, so EC policy doesn't change
much for query processing. SHOW TABLE STATS is unaffected: it reads the
policy from one RPC on the table location.
Testing
* FeFsTableTest: descriptors from createWithNoBlocks keep the flags; a
partition mixing an EC and a plain file reports MIXED.
* iceberg-insert.test SHOW FILES rows assert $EXPECTED_ERASURECODE_POLICY,
supplied per driver: the real policy in test_iceberg, "N/A" in
TestDisabledBlockLocations.
* test_iceberg_smoke/test_iceberg_v3_smoke in TestDisabledBlockLocations
on an EC build.
Change-Id: Ia7a7c97a20f6ec0efc2150185f4a9e26e3ac43cf
Assisted-by: Claude Opus 4.8 <[email protected]>
Reviewed-on: http://gerrit.cloudera.org:8080/24758
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> SHOW TABLE STATS / SHOW PARTITIONS / SHOW FILES issue one NameNode RPC
> perpartition/file for the display-only EC Policy column
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-15229
> URL: https://issues.apache.org/jira/browse/IMPALA-15229
> Project: IMPALA
> Issue Type: Improvement
> Reporter: YUBI LEE
> Assignee: YUBI LEE
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> {{FileSystemUtil.getErasureCodingPolicy(Path)}} issues one
> {{getErasureCodingPolicy}} RPC per
> path, uncached, from a serial loop, called once per partition by SHOW TABLE
> STATS / SHOW
> PARTITIONS and once per file by SHOW FILES — only to fill a display-only
> column. Nothing
> in planning, scheduling, or the read path consumes it
> ({{FileDescriptor.isEc}}, sourced from
> the listing, covers that separately).
> Production impact (non-EC HDFS cluster): one SHOW TABLE STATS on a
> ~105k-partition table
> generated ~105k serial NameNode RPCs; audit logs showed ~952k calls in a
> 5-minute window,
> tripping RPC-spike alerts and adding tens of seconds of statement latency.
> Every call
> returned NONE.
> The listing that loads file metadata already carries the EC policy in
> {{HdfsFileStatus}}.
> Record the policy id in the file descriptor at load time and derive the
> column from it;
> resolve ids to names locally (system policies) or with one cached
> {{getAllErasureCodingPolicies()}} call per filesystem. Fall back to the
> per-path lookup only
> for empty partitions and metadata written before the upgrade.
> Alternatives considered:
> - Removing the column introduced by IMPALA-9487 altogether. If the community
> feels the
> column does not carry its weight, that is also a valid resolution — but it
> changes the
> output schema of shipped statements that tools and tests parse, so it would
> need a
> deprecation discussion. Deriving the value from metadata we already have
> keeps the
> feature at roughly zero cost, which seemed like the less disruptive fix.
> - A Hadoop configuration switch to skip the lookups (in the style of
> impala.preload-block-locations-for-scheduling). This works as a mitigation
> but leaves
> the default behaviour expensive and drops the information where it is
> disabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
