katty he created HIVE-29766:
-------------------------------
Summary: [Hive 4] HS2 handlers wait indefinitely in
StatsUtils.collectStatistics while basic-stats ForkJoinPool workers are idle
Key: HIVE-29766
URL: https://issues.apache.org/jira/browse/HIVE-29766
Project: Hive
Issue Type: Bug
Affects Versions: 4.0.1
Reporter: katty he
We observed an HS2 availability incident in a Hive 4.0.1-derived build running
on JDK 11.0.17.13. This is a report of the observed behavior; we have not yet
found a deterministic reproducer.
For a partitioned-table query, HS2 handlers block during compilation in:
StatsUtils.collectStatistics
-> statsForkJoinPool.submit(...)
-> ForkJoinTask.get()
The relevant code is:
partStats = statsForkJoinPool.submit(() ->
partList.getNotDeniedPartns().parallelStream()
.map(p -> basicStatsFactory.build(Partish.buildFor(table, p)))
.collect(Collectors.toList())
).get();
At the time of the thread dump:
* 91 HiveServer2 Handler threads were waiting in ForkJoinTask.get(), with the
stack leading to StatsUtils.collectStatistics (the partitioned-table path
around line 338).
* The basic-stats-ForkJoinPool worker threads visible in the same dump were all
parked in ForkJoinPool.runWorker, waiting for work. No worker was observed
blocked in HDFS, Hive Metastore, or a BasicStats/DataSizeEstimator frame at
that instant.
* The affected queries had not reached "Semantic Analysis Completed",
"Completed compiling", Tez DAG submission, or execution. Thus the issue
occurred before Tez submission.
* The HS2 open-connection count gradually increased to approximately 91 and
recovered immediately after the affected HS2 process was restarted.
* Other HS2 instances in the same cluster did not show the problem. This
suggests a process-local state in the static statsForkJoinPool or its
task-completion chain.
Some affected queries were simple partition-filtered SELECT ... LIMIT queries.
The last compiler-side message for most of them was a fetch threshold decision
similar to:
Data set size=<value>, threshold=268435456
The tables/partitions often lacked basic statistics. When hive.stats.estimate
is enabled, BasicStats.DataSizeEstimator may fall back to recursive
FileSystem.listFiles(path, true). However, we cannot conclude from the
available dump whether a particular build() call, filesystem operation, task
scheduling operation, or Future completion state was the original trigger.
Expected behavior
-----------------
Statistics collection must not allow an individual query, or a stalled
ForkJoinPool task-completion chain, to block HS2 Handler threads indefinitely.
At minimum, a timeout, diagnostic logging/metrics, or a safe fallback would
help prevent HS2 connection exhaustion.
Actual behavior
---------------
Handler threads wait indefinitely in Future.get(). Once enough concurrent
queries enter the same path, HS2 becomes unable to compile/submit further
requests until the process is restarted.
Questions
---------
1. Is this a known issue in the Hive 4 StatsUtils parallelStream / custom
ForkJoinPool implementation?
2. Are there known JDK 11 ForkJoinPool issues that could leave externally
submitted tasks incomplete while workers appear idle?
3. Is there a recommended Hive configuration to bypass or bound this
statistics-annotation path for Tez compilation?
4. Would a patch adding a bounded wait and diagnostic information (query id,
table, partition count, pool queued/active counts, and per-partition build
timing) be acceptable? We can prepare one if maintainers agree.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
