skywalker0618 opened a new pull request, #19718:
URL: https://github.com/apache/hudi/pull/19718
### Describe the issue this Pull Request addresses
HiveQL sync mode leaks a Hive `Driver` on every sync.
`Driver.compile()` registers a shutdown hook with `ShutdownHookManager`.
`Driver.close()` does not remove it, only `Driver.destroy()` does, and Hive's
own comment on `destroy()` notes it "is usually called after close() ... to end
the driver life cycle". Since a fresh `HiveQueryDDLExecutor` (and, with
batching enabled, a fresh `HiveDriverPool`) is built per sync, every sync
permanently adds Drivers to the JVM-global hook set. Each retained Driver keeps
its `QueryState`, semantic analyzer, and the `Table` / `ReadEntity` /
`FieldSchema` objects of its last statement alive, so any process that syncs
repeatedly in one JVM grows without bound.
This was found on a long-running Flink streaming job syncing every 10
minutes. Its JobManager heap climbed roughly 1 GB/hour and OOMed every few
hours, on both a 4 GB and a 12 GB heap, more memory only bought time. Eclipse
MAT on the resulting 8 GB dump attributed **3.78 GB, 81.08% of the heap**, to a
single accumulation point:
AppClassLoader -> Object[] -> ShutdownHookManager (static MGR)
-> SynchronizedSet -> HashSet -> HashMap$Node[4096] 3.78 GB
Held there were 2,252 `ShutdownHookManager$HookEntry` retaining 2,292
`org.apache.hadoop.hive.ql.Driver`, which in turn retained 114,373
`hive.ql.metadata.Table`, 338,755 `StorageDescriptor`, and 12,296,367
`FieldSchema`. The Driver count matched the number of DDL statements the job
had executed, confirming one permanently retained Driver per Driver ever
created.
Note on scale: that dump comes from a deployment that constructs one Driver
per statement, which is why the counts are so large. With `HiveDriverPool` the
same defect leaks one Driver per pool worker per sync plus the session Driver,
far slower, but still unbounded for a streaming sync loop.
### Summary and Changelog
Drivers are now destroyed, not just closed, so nothing is left behind in
`ShutdownHookManager` after a sync completes.
- `HiveDriverPool` teardown: call `destroy()` after `close()` on each pooled
worker Driver.
- `HiveQueryDDLExecutor.close()`: call `destroy()` after `close()` on the
session Driver.
- `HiveQueryDDLExecutor` constructor failure path: same, so a partially
constructed executor does not leak the Driver it already built.
- New test `TestHiveDriverPool#closeDestroysEachPooledDriver`: injects mock
Drivers through the existing package-private `DriverFactory` seam and verifies
every pooled Driver is both closed and destroyed on pool close.
`destroy()` is safe on a Driver that never compiled: `shutdownRunner` is
null in that case and `ShutdownHookManager.removeShutdownHook` documents
returning false for a null hook. That matters on the constructor path, where a
throwing `destroy()` would otherwise mask the original construction failure.
### Impact
No public API or behavior change. HiveQL sync memory stays flat across syncs
instead of growing with each one. This mainly benefits long-running processes
such as Flink or Spark streaming jobs that sync repeatedly in the same JVM;
one-shot sync jobs are unaffected in practice since the JVM exits regardless.
### Risk Level
Low
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
