GitHub user steveloughran opened a pull request:
https://github.com/apache/spark/pull/22186
[SPARK-25183][SQL] Spark HiveServer2 to use Spark ShutdownHookManager
## What changes were proposed in this pull request?
Switch `org.apache.hive.service.server.HiveServer2` to register its
shutdown callback with Spark's `ShutdownHookManager`, rather than direct with
the Java Runtime callback.
This avoids race conditions in shutdown where the filesystem is shutdown
before the flush/write/rename of the event log is completed, particularly on
object stores where the write and rename can be slow.
## How was this patch tested?
There's no explicit unit for test this, which is consistent with every
other shutdown hook in the codebase.
* There's an implicit test when the scalatest process is halted.
* More manual/integration testing is needed.
HADOOP-15679 has added the ability to explicitly execute the hadoop
shutdown hook sequence which spark uses; that could be stabilized for testing
if desired, after which all the spark hooks could be tested. Until then:
external system tests only.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/steveloughran/spark BUG/SPARK-25183-shutdown
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22186.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22186
----
commit 8fbeb59c8a96dd7f7ed6982009bac59ab3fa87ce
Author: Steve Loughran <stevel@...>
Date: 2018-08-22T17:13:19Z
SPARK-25183 Spark HiveServer2 to use Spark ShutdownHookManager for shutdown
hook
Change-Id: I9a0885660efda4ec6277e0237ca7eada0b43533f
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
