[
https://issues.apache.org/jira/browse/HIVE-27190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100172#comment-18100172
]
Denys Kuzmenko commented on HIVE-27190:
---------------------------------------
[~lisoda] i've experienced similar perf degradation caused by:
1. Per-partition stats fetch is O(file) per partition.
With hive.iceberg.stats.source=iceberg, StatsUtils.collectStatistics builds
BasicStats per pruned partition. Each one called getPartishSummary, which opens
the entire partition-stats file from object storage and linearly scans it for
that single partition. N partitions ⇒ N full file opens + O(N²/2) record parses
per TableScan.
2. The stats worker pool can't use the table-metadata cache (HIVE-24313
interaction). The per-partition work runs on the dedicated basic-stats-*
ForkJoin pool, whose thread factory doesn't propagate the SessionState
ThreadLocal. Without it, SessionStateUtil.getQueryState(conf) is empty, so the
handler's getTable() takes skipCache = orElse(true) — a full Catalogs.loadTable
per partition (HMS RPC + metadata.json read from storage). That's each
"Refreshing table metadata" line. The session-level cache is both unreadable
and unpopulatable from those threads, so it never warms.
> Fix cache-key collisions for time-travel queries on Iceberg
> -----------------------------------------------------------
>
> Key: HIVE-27190
> URL: https://issues.apache.org/jira/browse/HIVE-27190
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
> Affects Versions: 4.1.0, 4.2.0
> Reporter: Simhadri Govindappa
> Assignee: Denys Kuzmenko
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
