[
https://issues.apache.org/jira/browse/SPARK-57449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiwon Park updated SPARK-57449:
-------------------------------
Description:
Currently, for complex type columns (ARRAY, MAP, STRUCT),
{{SparkConnectResultSet.getObject}} returns native Scala objects
({{scala.collection.Seq}}, {{scala.collection.Map}},
{{org.apache.spark.sql.Row}}), and {{getArray()}} throws
{{SQLFeatureNotSupportedException}}.
BI tools (e.g. DataGrip) fetch complex columns via
{{getObject()}}/{{getArray()}}, not {{getString()}}. Because the returned
objects are not standard JDBC types, the tools fall back to
{{Object.toString()}}, producing unreadable output such as {{ArraySeq(1, 2,
3)}} for arrays and {{[1,x]}} for structs (losing field names).
This proposes exposing complex types via the standard JDBC interfaces:
* {{getObject}}: ARRAY -> {{java.sql.Array}}, STRUCT -> {{java.sql.Struct}},
MAP -> {{java.util.Map}}
* implement {{getArray()}}
* align {{getColumnClassName}} accordingly ({{java.sql.Array}} /
{{java.sql.Struct}} / {{java.util.Map}})
DataGrip renders {{java.sql.Struct}} as nested columns and {{java.sql.Array}}
in a dedicated value editor. The implementations should also provide a
JSON-like {{toString}} as a fallback for clients that stringify the object
directly (this subsumes the originally-scoped {{getString}} JSON rendering).
Summary: Return java.sql.Array and java.sql.Struct from
ResultSet.getObject for complex types in the Connect JDBC driver (was: Render
complex type values as JSON in ResultSet.getString for the Connect JDBC driver)
> Return java.sql.Array and java.sql.Struct from ResultSet.getObject for
> complex types in the Connect JDBC driver
> ---------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-57449
> URL: https://issues.apache.org/jira/browse/SPARK-57449
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Jiwon Park
> Priority: Major
>
> Currently, for complex type columns (ARRAY, MAP, STRUCT),
> {{SparkConnectResultSet.getObject}} returns native Scala objects
> ({{scala.collection.Seq}}, {{scala.collection.Map}},
> {{org.apache.spark.sql.Row}}), and {{getArray()}} throws
> {{SQLFeatureNotSupportedException}}.
> BI tools (e.g. DataGrip) fetch complex columns via
> {{getObject()}}/{{getArray()}}, not {{getString()}}. Because the returned
> objects are not standard JDBC types, the tools fall back to
> {{Object.toString()}}, producing unreadable output such as {{ArraySeq(1, 2,
> 3)}} for arrays and {{[1,x]}} for structs (losing field names).
> This proposes exposing complex types via the standard JDBC interfaces:
> * {{getObject}}: ARRAY -> {{java.sql.Array}}, STRUCT -> {{java.sql.Struct}},
> MAP -> {{java.util.Map}}
> * implement {{getArray()}}
> * align {{getColumnClassName}} accordingly ({{java.sql.Array}} /
> {{java.sql.Struct}} / {{java.util.Map}})
> DataGrip renders {{java.sql.Struct}} as nested columns and {{java.sql.Array}}
> in a dedicated value editor. The implementations should also provide a
> JSON-like {{toString}} as a fallback for clients that stringify the object
> directly (this subsumes the originally-scoped {{getString}} JSON rendering).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
