[
https://issues.apache.org/jira/browse/IMPALA-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100059#comment-18100059
]
Zoltán Borók-Nagy commented on IMPALA-15229:
--------------------------------------------
Thanks for your contribution [~eub], I'll take a look this week.
> SHOW TABLE STATS / SHOW PARTITIONS / SHOW FILES issue one NameNode RPC
> perpartition/file for the display-only EC Policy column
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-15229
> URL: https://issues.apache.org/jira/browse/IMPALA-15229
> Project: IMPALA
> Issue Type: Improvement
> Reporter: YUBI LEE
> Priority: Major
>
> {{FileSystemUtil.getErasureCodingPolicy(Path)}} issues one
> {{getErasureCodingPolicy}} RPC per
> path, uncached, from a serial loop, called once per partition by SHOW TABLE
> STATS / SHOW
> PARTITIONS and once per file by SHOW FILES — only to fill a display-only
> column. Nothing
> in planning, scheduling, or the read path consumes it
> ({{FileDescriptor.isEc}}, sourced from
> the listing, covers that separately).
> Production impact (non-EC HDFS cluster): one SHOW TABLE STATS on a
> ~105k-partition table
> generated ~105k serial NameNode RPCs; audit logs showed ~952k calls in a
> 5-minute window,
> tripping RPC-spike alerts and adding tens of seconds of statement latency.
> Every call
> returned NONE.
> The listing that loads file metadata already carries the EC policy in
> {{HdfsFileStatus}}.
> Record the policy id in the file descriptor at load time and derive the
> column from it;
> resolve ids to names locally (system policies) or with one cached
> {{getAllErasureCodingPolicies()}} call per filesystem. Fall back to the
> per-path lookup only
> for empty partitions and metadata written before the upgrade.
> Alternatives considered:
> - Removing the column introduced by IMPALA-9487 altogether. If the community
> feels the
> column does not carry its weight, that is also a valid resolution — but it
> changes the
> output schema of shipped statements that tools and tests parse, so it would
> need a
> deprecation discussion. Deriving the value from metadata we already have
> keeps the
> feature at roughly zero cost, which seemed like the less disruptive fix.
> - A Hadoop configuration switch to skip the lookups (in the style of
> impala.preload-block-locations-for-scheduling). This works as a mitigation
> but leaves
> the default behaviour expensive and drops the information where it is
> disabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
