[
https://issues.apache.org/jira/browse/LIVY-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nilesh Rathi updated LIVY-1060:
-------------------------------
Description:
When Apache Livy launches Spark sessions (interactive SQL sessions or batch
jobs) against a TLS-enabled Hive Metastore (HMS), the Spark driver fails during
Hive Metastore client initialization. This prevents any workload that requires
HMS access — such as querying Hive tables, accessing Iceberg tables via the
Hive catalog, or running SQL sessions with
{{{}livy.repl.enable-hive-context=true{}}}.
h3. Problem
The failure is caused by an SSL/TLS handshake error between the Spark driver
(running inside a Livy-managed session) and the Hive Metastore. The Java
keystore or truststore used for the SSL connection cannot be loaded, typically
with a password verification error.
h3. Steps to Reproduce
# Deploy Livy on a cluster where Hive Metastore is configured to use SSL/TLS.
# Ensure {{livy.repl.enable-hive-context=true}} and that {{hive-site.xml}}
(with HMS SSL settings) is available on the Livy server classpath.
# Create a Livy interactive SQL session or submit a batch job that accesses
Hive/Iceberg tables via the Hive Metastore.
# Observe the Spark driver logs during session startup or first HMS access.
h3. Expected Behavior
Livy should propagate the required Hive Metastore SSL/TLS configuration
(keystore/truststore locations, passwords, and related settings) to the Spark
driver so that the Hive Metastore client can establish a secure connection
successfully.
h3. Actual Behavior
The Spark driver fails to instantiate the Hive Metastore client with
SSL-related keystore errors.
h3. Error Trace
{code:java}
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:108)
...
Caused by: java.io.IOException: Keystore was tampered with, or password was
incorrect
at
java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:813)
at
org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java:200)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
{code}
was:
When Apache Livy launches Spark sessions (interactive SQL sessions or batch
jobs) against a TLS-enabled Hive Metastore (HMS), the Spark driver fails during
Hive Metastore client initialization. This prevents any workload that requires
HMS access — such as querying Hive tables, accessing Iceberg tables via the
Hive catalog, or running SQL sessions with
{{livy.repl.enable-hive-context=true}}.
h3. Problem
The failure is caused by an SSL/TLS handshake error between the Spark driver
(running inside a Livy-managed session) and the Hive Metastore. The Java
keystore or truststore used for the SSL connection cannot be loaded, typically
with a password verification error.
h3. Steps to Reproduce
# Deploy Livy on a cluster where Hive Metastore is configured to use SSL/TLS.
# Ensure {{livy.repl.enable-hive-context=true}} and that {{hive-site.xml}}
(with HMS SSL settings) is available on the Livy server classpath.
# Create a Livy interactive SQL session or submit a batch job that accesses
Hive/Iceberg tables via the Hive Metastore.
# Observe the Spark driver logs during session startup or first HMS access.
h3. Expected Behavior
Livy should propagate the required Hive Metastore SSL/TLS configuration
(keystore/truststore locations, passwords, and related settings) to the Spark
driver so that the Hive Metastore client can establish a secure connection
successfully.
h3. Actual Behavior
The Spark driver fails to instantiate the Hive Metastore client with
SSL-related keystore errors.
h3. Error Trace
{code}
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:108)
...
Caused by: java.io.IOException: Keystore was tampered with, or password was
incorrect
at
java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:813)
at
org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java:200)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
{code}
h3. Proposed Fix
Ensure that Hive Metastore SSL/TLS configuration is correctly propagated from
Livy to Spark sessions. This may include:
* Distributing {{hive-site.xml}} (or equivalent HMS SSL properties) to the
Spark driver via {{spark.files}} when Hive context is enabled.
* Ensuring keystore/truststore paths referenced in the configuration are
accessible from the Spark driver container.
* Resolving keystore/truststore passwords via the Hadoop Credential Provider
API where applicable, consistent with existing Livy SSL support (see LIVY-475,
LIVY-705).
h3. Environment
* Livy with Hive context enabled ({{livy.repl.enable-hive-context=true}})
* Hive Metastore configured with SSL/TLS
* Spark deployed in YARN cluster mode (typical Livy deployment)
> Livy Spark sessions fail to connect to Hive Metastore over SSL with keystore
> password error
> -------------------------------------------------------------------------------------------
>
> Key: LIVY-1060
> URL: https://issues.apache.org/jira/browse/LIVY-1060
> Project: Livy
> Issue Type: Bug
> Reporter: Nilesh Rathi
> Priority: Major
>
> When Apache Livy launches Spark sessions (interactive SQL sessions or batch
> jobs) against a TLS-enabled Hive Metastore (HMS), the Spark driver fails
> during Hive Metastore client initialization. This prevents any workload that
> requires HMS access — such as querying Hive tables, accessing Iceberg tables
> via the Hive catalog, or running SQL sessions with
> {{{}livy.repl.enable-hive-context=true{}}}.
> h3. Problem
> The failure is caused by an SSL/TLS handshake error between the Spark driver
> (running inside a Livy-managed session) and the Hive Metastore. The Java
> keystore or truststore used for the SSL connection cannot be loaded,
> typically with a password verification error.
> h3. Steps to Reproduce
> # Deploy Livy on a cluster where Hive Metastore is configured to use SSL/TLS.
> # Ensure {{livy.repl.enable-hive-context=true}} and that {{hive-site.xml}}
> (with HMS SSL settings) is available on the Livy server classpath.
> # Create a Livy interactive SQL session or submit a batch job that accesses
> Hive/Iceberg tables via the Hive Metastore.
> # Observe the Spark driver logs during session startup or first HMS access.
> h3. Expected Behavior
> Livy should propagate the required Hive Metastore SSL/TLS configuration
> (keystore/truststore locations, passwords, and related settings) to the Spark
> driver so that the Hive Metastore client can establish a secure connection
> successfully.
> h3. Actual Behavior
> The Spark driver fails to instantiate the Hive Metastore client with
> SSL-related keystore errors.
> h3. Error Trace
> {code:java}
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
> Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> ...
> org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:108)
> ...
> Caused by: java.io.IOException: Keystore was tampered with, or password was
> incorrect
> at
> java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:813)
> at
> org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java:200)
> Caused by: java.security.UnrecoverableKeyException: Password verification
> failed
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
