Nilesh Rathi created LIVY-1060:
----------------------------------
Summary: Livy Spark sessions fail to connect to Hive Metastore
over SSL with keystore password error
Key: LIVY-1060
URL: https://issues.apache.org/jira/browse/LIVY-1060
Project: Livy
Issue Type: Bug
Reporter: Nilesh Rathi
When Apache Livy launches Spark sessions (interactive SQL sessions or batch
jobs) against a TLS-enabled Hive Metastore (HMS), the Spark driver fails during
Hive Metastore client initialization. This prevents any workload that requires
HMS access — such as querying Hive tables, accessing Iceberg tables via the
Hive catalog, or running SQL sessions with
{{livy.repl.enable-hive-context=true}}.
h3. Problem
The failure is caused by an SSL/TLS handshake error between the Spark driver
(running inside a Livy-managed session) and the Hive Metastore. The Java
keystore or truststore used for the SSL connection cannot be loaded, typically
with a password verification error.
h3. Steps to Reproduce
# Deploy Livy on a cluster where Hive Metastore is configured to use SSL/TLS.
# Ensure {{livy.repl.enable-hive-context=true}} and that {{hive-site.xml}}
(with HMS SSL settings) is available on the Livy server classpath.
# Create a Livy interactive SQL session or submit a batch job that accesses
Hive/Iceberg tables via the Hive Metastore.
# Observe the Spark driver logs during session startup or first HMS access.
h3. Expected Behavior
Livy should propagate the required Hive Metastore SSL/TLS configuration
(keystore/truststore locations, passwords, and related settings) to the Spark
driver so that the Hive Metastore client can establish a secure connection
successfully.
h3. Actual Behavior
The Spark driver fails to instantiate the Hive Metastore client with
SSL-related keystore errors.
h3. Error Trace
{code}
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:108)
...
Caused by: java.io.IOException: Keystore was tampered with, or password was
incorrect
at
java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:813)
at
org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java:200)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
{code}
h3. Proposed Fix
Ensure that Hive Metastore SSL/TLS configuration is correctly propagated from
Livy to Spark sessions. This may include:
* Distributing {{hive-site.xml}} (or equivalent HMS SSL properties) to the
Spark driver via {{spark.files}} when Hive context is enabled.
* Ensuring keystore/truststore paths referenced in the configuration are
accessible from the Spark driver container.
* Resolving keystore/truststore passwords via the Hadoop Credential Provider
API where applicable, consistent with existing Livy SSL support (see LIVY-475,
LIVY-705).
h3. Environment
* Livy with Hive context enabled ({{livy.repl.enable-hive-context=true}})
* Hive Metastore configured with SSL/TLS
* Spark deployed in YARN cluster mode (typical Livy deployment)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
