David Tagatac created SPARK-55645:
-------------------------------------
Summary: Include Hive Metastore SerDeInfo field NAME when
converting between HiveTableRelation and LogicalRelation
Key: SPARK-55645
URL: https://issues.apache.org/jira/browse/SPARK-55645
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.5.6
Reporter: David Tagatac
The [Hive Metastore
API|https://github.com/apache/hive/blob/5160d3af392248255f68e41e1e0557eae4d95273/metastore/if/hive_metastore.thrift#L260]
SerDeInfo struct includes a `name` field. However, this field is not included
in the [data structure used to represent the API
response|https://github.com/apache/spark/blob/ac4e7947e4efa4ca2484d9393b14322f4e636df5/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala#L143-L149]
{code:java}
org.apache.spark.sql.catalyst.catalog.CatalogStorageFormat{code}
As a result, extension developers do not have a good way to use this field
without storing it somewhere unintuitive like `CatalogStorageFormat.properties`
with some arbitrary key like "serde.name". Including this field as a
corresponding field in the `org.apache.spark.sql.catalyst.catalog` data model
will allow for a better representation of Hive Metastore tables in Spark.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
