[
https://issues.apache.org/jira/browse/SPARK-41919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655346#comment-17655346
]
Martin Grund commented on SPARK-41919:
--------------------------------------
There is a compatible way of doing this by adding another field to the oneof
that represents the string schema and marking the other fields as reserved.
> Unify the schema or datatype in protos
> --------------------------------------
>
> Key: SPARK-41919
> URL: https://issues.apache.org/jira/browse/SPARK-41919
> Project: Spark
> Issue Type: Sub-task
> Components: Connect
> Affects Versions: 3.4.0
> Reporter: Ruifeng Zheng
> Priority: Major
>
> this ticket only focus on the protos sent from client to server.
> we normally use
> {code:java}
> oneof schema {
> DataType datatype = 2;
> // Server will use Catalyst parser to parse this string to DataType.
> string datatype_str = 3;
> }
> {code}
> to represent a schema or datatype.
> actually, we can simplify it with just a string. In the server, we can easily
> parse a DDL-formatted schema or a JSON formatted one.
> {code:java}
> // (Optional) The schema of local data.
> // It should be either a DDL-formatted type string or a JSON string.
> //
> // The server side will update the column names and data types according to
> this schema.
> // If the 'data' is not provided, then this schema will be required.
> optional string schema = 2;
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
