ericyangliu opened a new issue, #7515: URL: https://github.com/apache/kyuubi/issues/7515
### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct). ### Search before asking - [x] I have searched the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug Kyuubi fails when returning result set metadata for a Spark SQL result containing a `VARIANT` column. The SQL operation itself completes successfully, and row fetching can succeed, but the REST metadata endpoint fails when Kyuubi converts the Spark result schema to HiveServer2/Thrift metadata. The failure comes from: ```text org.apache.kyuubi.engine.spark.schema.SchemaHelper$.toTTypeId(SchemaHelper.scala:58) java.lang.IllegalArgumentException: Unrecognized type name: variant ``` `SchemaHelper.toTTypeId` handles common Spark SQL types but does not appear to handle Spark 4 `VariantType`. Spark 4.0+ supports `VariantType`, and `parse_json(...)` returns a `VARIANT`. ### Steps to reproduce Run Kyuubi with a Spark 4.x engine, then execute a query that returns a `VARIANT` column: ```sql SELECT parse_json('{"a":1,"b":0.8}') AS v; ``` Then request result set metadata for the completed operation, for example through the Kyuubi Web UI or REST API: ```text GET /api/v1/operations/<operationHandle>/resultsetmetadata ``` ### Expected behavior Kyuubi should return result set metadata successfully. If HiveServer2/Thrift has no native `VARIANT` type, Kyuubi should expose Spark `VariantType` as a compatible metadata type, such as `STRING_TYPE` or another appropriate fallback, instead of failing the metadata request. ### Actual behavior The query reaches `FINISHED_STATE`, but metadata retrieval fails: ```text Error getting result set metadata for operation handle <operationHandle> org.apache.kyuubi.KyuubiSQLException: Unrecognized type name: variant Caused by: java.lang.IllegalArgumentException: Unrecognized type name: variant at org.apache.kyuubi.engine.spark.schema.SchemaHelper$.toTTypeId(SchemaHelper.scala:58) at org.apache.kyuubi.engine.spark.schema.SchemaHelper$.toTTypeDesc(SchemaHelper.scala:77) at org.apache.kyuubi.engine.spark.schema.SchemaHelper$.toTColumnDesc(SchemaHelper.scala:87) at org.apache.kyuubi.engine.spark.schema.SchemaHelper$.toTTableSchema(SchemaHelper.scala:95) ``` ### Affects Version(s) Observed on Kyuubi `1.11.1` with Spark `4.1.1`. The same `SchemaHelper.toTTypeId` mapping appears to be present on current `master`, so `master` may also be affected. ### Kyuubi Server Log Output ```logtalk Processing query[<operationHandle>]: RUNNING_STATE -> FINISHED_STATE GET /api/v1/operations/<operationHandle>/rowset ... status=200 ERROR org.apache.kyuubi.server.api.ApiUtils: Error getting result set metadata for operation handle <operationHandle> org.apache.kyuubi.KyuubiSQLException: Unrecognized type name: variant Caused by: java.lang.IllegalArgumentException: Unrecognized type name: variant at org.apache.kyuubi.engine.spark.schema.SchemaHelper$.toTTypeId(SchemaHelper.scala:58) ``` ### Kyuubi Engine Log Output No Spark query execution failure was observed. The query completed successfully; the failure occurs during Kyuubi result set metadata conversion. ### Kyuubi Server Configurations ```yaml kyuubi.engine.type: SPARK_SQL kyuubi.frontend.protocols: THRIFT_BINARY,REST ``` ### Kyuubi Engine Configurations The issue should reproduce without extra engine configuration using: ```sql SELECT parse_json('{"a":1,"b":0.8}') AS v; ``` ### Additional context Spark 4.0 introduced `VariantType`. Kyuubi already stringifies non-primitive row values in rowset generation, but result metadata conversion currently throws before clients can render metadata for a result containing `VARIANT`. A likely fix is to add Spark `VariantType` handling in Kyuubi's Spark schema conversion path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
