[
https://issues.apache.org/jira/browse/SPARK-50541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amanda Liu updated SPARK-50541:
-------------------------------
Summary: Support structured JSON output for Spark SQL metadata commands
(was: Describe Table As JSON)
> Support structured JSON output for Spark SQL metadata commands
> --------------------------------------------------------------
>
> Key: SPARK-50541
> URL: https://issues.apache.org/jira/browse/SPARK-50541
> Project: Spark
> Issue Type: Epic
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Amanda Liu
> Assignee: Amanda Liu
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Support DESCRIBE TABLE ... [AS JSON] option to display table metadata in
> JSON format.
>
> *Context:*
> The Spark SQL command DESCRIBE TABLE displays table metadata in a DataFrame
> format geared toward human consumption. This format causes parsing
> challenges, e.g. if fields contain special characters or the format changes
> as new features are added.
> The new AS JSON option would return the table metadata as a JSON string that
> supports parsing via machine, while being extensible with a minimized risk of
> breaking changes. It is not meant to be human-readable.
>
> *SQL Ref Spec:*
> { DESC | DESCRIBE } [ TABLE ] [ EXTENDED | FORMATTED ] table_name \{ [
> PARTITION clause ] | [ column_name ] } [ AS JSON ]
>
> *JSON Schema:*
> ```
> {
> "table_name": "<table_name>",
> "catalog_name": [...],
> "database_name": [...],
> "qualified_name": "<qualified_name>"
> "type": "<table_type>",
> "provider": "<provider>",
> "columns": [
> { "id": 1, "name": "<name>", "type": <type_json>, "comment": "<comment>",
> "default": "<default_val>" }
> ],
> "partition_values":
> { "<col_name>": "<val>" }
> ,
> "location": "<path>",
> "view_definition": "<view_defn>",
> "owner": "<owner>",
> "comment": "<comment>",
> "table_properties":
> { "property1": "<property1>", "property2": "<property2>" }
> ,
> "storage_properties":
> { "property1": "<property1>", "property2": "<property2>" }
> ,
> "serde_library": "<serde_library>",
> "inputformat": "<input_format>",
> "outputformt": "<output_format>",
> "bucket_columns": [<col_name>],
> "sort_columns": [<col_name>],
> "created_time": "<timestamp>",
> "last_access": "<timestamp>",
> "partition_provider": "<partition_provider>"
> }
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
