[
https://issues.apache.org/jira/browse/SPARK-61031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61031:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Take the lists of paths of the readers and of columns of bucketBy and sortBy,
> and the arguments of mode and createTable, in the Python client as PySpark
> does
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61031
> URL: https://issues.apache.org/jira/browse/SPARK-61031
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, {{spark.read.load(path)}} and the readers of formats, such as
> {{csv}}, {{json}}, {{text}}, {{orc}} and {{xml}}, take a path or a list of
> paths. {{df.write.bucketBy(numBuckets, col, *cols)}} and {{sortBy(col,
> *cols)}} take the columns as names or as a list or tuple of them, and raise
> {{NOT_EXPECTED_TYPE}} for a {{numBuckets}} that is not an int and for columns
> that are not strings. {{df.write.mode(None)}} keeps the default mode.
> {{spark.catalog.createTable}} raises {{NOT_EXPECTED_TYPE}} for a schema that
> is not a StructType, and {{tableExists(tableName, dbName)}} warns that
> {{dbName}} is deprecated.
> The Rust-backed Python client raises a TypeError for a list of paths, as
> {{spark.read.csv([path])}}, for the list of columns of {{bucketBy}} and
> {{sortBy}}, and for {{mode(None)}}. It raises TypeErrors rather than
> PySpark's errors for the arguments of {{bucketBy}}, sends a schema of
> {{createTable}} that is not a StructType to the server, and does not warn of
> {{dbName}}. Take the arguments as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
