zhengruifeng commented on code in PR #38979:
URL: https://github.com/apache/spark/pull/38979#discussion_r1044259093
##########
connector/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -305,6 +306,17 @@ message LocalRelation {
// Local collection data serialized into Arrow IPC streaming format which
contains
// the schema of the data.
bytes data = 1;
+
+ // (Optional) The user provided schema.
+ //
+ // The Sever side will update the column names and data types according to
this schema.
+ oneof schema {
+
+ DataType datatype = 2;
+
+ // Server will use Catalyst parser to parse this string to DataType.
+ string datatype_str = 3;
Review Comment:
I'm think adding support for `_parse_datatype_string` in AnalyzePlan, then
we don't need to add `datatype` and `datatype_str` in `LocalRelation` at all.
Then the implementation will be like this (after we implement
[`DataFrame.to`](https://github.com/apache/spark/blob/master/python/pyspark/sql/dataframe.py#L1913-L1968)):
```
schema = _parse_datatype_string(schema_str)
return DataFrame.withPlan(LocalRelation(table),
self).toDF(*schema.fieldNames).to(schema)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
