Sunwoo-Shin opened a new issue, #7433: URL: https://github.com/apache/kyuubi/issues/7433
### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug In environments with **multiple Hive Metastore (HMS) instances backed by different Kerberos principals**, Spark engines launched by Kyuubi cannot authenticate against non-default HMS instances. Catalog operations against those HMSes fail with `DIGEST-MD5: IO error acquiring password`. The root cause is in the engine-side credential push path, `SparkTBinaryFrontendService#addHiveToken`. The current logic assumes a **single HMS** and selects only tokens whose `getService()` is empty: ```scala val newToken = newTokens .find { case (uris, token) => val matched = uris.toString.split(",").exists(uriSet.contains) && token.getService == new Text() ... } ``` Any Hive delegation token that has its `service` field populated — the natural way to disambiguate per-HMS tokens that share a single `Credentials` — is **silently dropped**. As a result, tokens for non-default HMSes never reach the engine UGI even when a (custom) provider has issued them correctly. ### Affects Version(s) 1.11.1 ### Kyuubi Server Log Output ```logtalk ``` ### Kyuubi Engine Log Output ```logtalk Authentication fails on the engine side because the token for HMS B never reached the engine UGI: Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: Peer indicated failure: DIGEST-MD5: IO error acquiring password at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(...) at org.apache.thrift.transport.TSaslTransport.open(...) at org.apache.thrift.transport.TSaslClientTransport.open(...) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(...) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(...) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(...) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(...) ``` ### Kyuubi Server Configurations ```yaml ``` ### Kyuubi Engine Configurations ```yaml A Spark session configured with **two Iceberg catalogs, each backed by a separate HMS with different Kerberos principals** — a common setup for domain-partitioned warehouses: # Catalog A — HMS A spark.sql.catalog.warehouse_a org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.warehouse_a.type hive spark.sql.catalog.warehouse_a.uri thrift://hms-a.example.com:9083 spark.sql.catalog.warehouse_a.warehouse s3a://warehouse-a/warehouse spark.sql.catalog.warehouse_a.io-impl org.apache.iceberg.aws.s3.S3FileIO spark.sql.catalog.warehouse_a.hadoop.hive.metastore.uris thrift://hms-a.example.com:9083 spark.sql.catalog.warehouse_a.hadoop.hive.metastore.sasl.enabled true spark.sql.catalog.warehouse_a.hadoop.hive.metastore.kerberos.principal hive-a/[email protected] # Tells the HMS client which delegation token to pick from UGI credentials. # Must match the alias used when the token was added to Credentials (= instance name). spark.sql.catalog.warehouse_a.hadoop.hive.metastore.token.signature warehouse_a # Catalog B — HMS B (different Kerberos principal) spark.sql.catalog.warehouse_b org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.warehouse_b.type hive spark.sql.catalog.warehouse_b.uri thrift://hms-b.example.com:9083 spark.sql.catalog.warehouse_b.warehouse s3a://warehouse-b/warehouse spark.sql.catalog.warehouse_b.io-impl org.apache.iceberg.aws.s3.S3FileIO spark.sql.catalog.warehouse_b.hadoop.hive.metastore.uris thrift://hms-b.example.com:9083 spark.sql.catalog.warehouse_b.hadoop.hive.metastore.sasl.enabled true spark.sql.catalog.warehouse_b.hadoop.hive.metastore.kerberos.principal hive-b/[email protected] spark.sql.catalog.warehouse_b.hadoop.hive.metastore.token.signature warehouse_b ``` ### Additional context - This bug is layered *under* #1091 — fixing the provider does not eliminate the engine-side drop. ### Are you willing to submit PR? - [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix. - [ ] No. I cannot submit a PR at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
