[
https://issues.apache.org/jira/browse/LIVY-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nilesh Rathi updated LIVY-1060:
-------------------------------
Description:
When Apache Livy launches Spark sessions (interactive SQL sessions or batch
jobs) against a TLS-enabled Hive Metastore (HMS), the Spark driver fails during
Hive Metastore client initialization. This prevents any workload that requires
HMS access — such as querying Hive tables, accessing Iceberg tables via the
Hive catalog, or running SQL sessions with
{{{}livy.repl.enable-hive-context=true{}}}.
h3. Problem
The failure is caused by an SSL/TLS handshake error between the Spark driver
(running inside a Livy-managed session) and the Hive Metastore. The Java
keystore or truststore used for the SSL connection cannot be loaded, typically
with a password verification error.
h3. Steps to Reproduce
# Deploy Livy on a cluster where Hive Metastore is configured to use SSL/TLS.
# Ensure {{livy.repl.enable-hive-context=true}} and that {{hive-site.xml}}
(with HMS SSL settings) is available on the Livy server classpath.
# Create a Livy interactive SQL session or submit a batch job that accesses
Hive/Iceberg tables via the Hive Metastore.
# Observe the Spark driver logs during session startup or first HMS access.
h3. Expected Behavior
Livy should propagate the required Hive Metastore SSL/TLS configuration
(keystore/truststore locations, passwords, and related settings) to the Spark
driver so that the Hive Metastore client can establish a secure connection
successfully.
h3. Actual Behavior
The Spark driver fails to instantiate the Hive Metastore client with
SSL-related keystore errors.
h3. Possible Error Trace
{code:java}
HiveException: Unable to instantiate SessionHiveMetaStoreClient
...
Caused by: java.io.IOException: Keystore was tampered with, or password was
incorrect
Caused by: java.security.UnrecoverableKeyException: Password verification
failed {code}
was:
When Apache Livy launches Spark sessions (interactive SQL sessions or batch
jobs) against a TLS-enabled Hive Metastore (HMS), the Spark driver fails during
Hive Metastore client initialization. This prevents any workload that requires
HMS access — such as querying Hive tables, accessing Iceberg tables via the
Hive catalog, or running SQL sessions with
{{livy.repl.enable-hive-context=true}}.
h3. Problem
The failure is caused by an SSL/TLS handshake error between the Spark driver
(running inside a Livy-managed session) and the Hive Metastore. The Java
keystore or truststore used for the SSL connection cannot be loaded, typically
with a password verification error.
h3. Steps to Reproduce
# Deploy Livy on a cluster where Hive Metastore is configured to use SSL/TLS.
# Ensure {{livy.repl.enable-hive-context=true}} and that {{hive-site.xml}}
(with HMS SSL settings) is available on the Livy server classpath.
# Create a Livy interactive SQL session or submit a batch job that accesses
Hive/Iceberg tables via the Hive Metastore.
# Observe the Spark driver logs during session startup or first HMS access.
h3. Expected Behavior
Livy should propagate the required Hive Metastore SSL/TLS configuration
(keystore/truststore locations, passwords, and related settings) to the Spark
driver so that the Hive Metastore client can establish a secure connection
successfully.
h3. Actual Behavior
The Spark driver fails to instantiate the Hive Metastore client with
SSL-related keystore errors.
h3. Error Trace
{code}
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
Caused by: java.io.IOException: Keystore was tampered with, or password was
incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java)
at
org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
{code}
> Livy Spark sessions fail to connect to Hive Metastore over SSL with keystore
> password error
> -------------------------------------------------------------------------------------------
>
> Key: LIVY-1060
> URL: https://issues.apache.org/jira/browse/LIVY-1060
> Project: Livy
> Issue Type: Bug
> Reporter: Nilesh Rathi
> Priority: Major
>
> When Apache Livy launches Spark sessions (interactive SQL sessions or batch
> jobs) against a TLS-enabled Hive Metastore (HMS), the Spark driver fails
> during Hive Metastore client initialization. This prevents any workload that
> requires HMS access — such as querying Hive tables, accessing Iceberg tables
> via the Hive catalog, or running SQL sessions with
> {{{}livy.repl.enable-hive-context=true{}}}.
> h3. Problem
> The failure is caused by an SSL/TLS handshake error between the Spark driver
> (running inside a Livy-managed session) and the Hive Metastore. The Java
> keystore or truststore used for the SSL connection cannot be loaded,
> typically with a password verification error.
> h3. Steps to Reproduce
> # Deploy Livy on a cluster where Hive Metastore is configured to use SSL/TLS.
> # Ensure {{livy.repl.enable-hive-context=true}} and that {{hive-site.xml}}
> (with HMS SSL settings) is available on the Livy server classpath.
> # Create a Livy interactive SQL session or submit a batch job that accesses
> Hive/Iceberg tables via the Hive Metastore.
> # Observe the Spark driver logs during session startup or first HMS access.
> h3. Expected Behavior
> Livy should propagate the required Hive Metastore SSL/TLS configuration
> (keystore/truststore locations, passwords, and related settings) to the Spark
> driver so that the Hive Metastore client can establish a secure connection
> successfully.
> h3. Actual Behavior
> The Spark driver fails to instantiate the Hive Metastore client with
> SSL-related keystore errors.
> h3. Possible Error Trace
> {code:java}
> HiveException: Unable to instantiate SessionHiveMetaStoreClient
> ...
> Caused by: java.io.IOException: Keystore was tampered with, or password was
> incorrect
> Caused by: java.security.UnrecoverableKeyException: Password verification
> failed {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
