[
https://issues.apache.org/jira/browse/SPARK-61103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61103.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 303
[https://github.com/apache/spark-connect-rust/pull/303]
> Create a DataFrame of a numpy array in createDataFrame in the Python client
> as PySpark does
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-61103
> URL: https://issues.apache.org/jira/browse/SPARK-61103
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{spark.createDataFrame}} takes a numpy array of 1 or 2
> dimensions. It builds an Arrow table of a column of each column of the array,
> and sends it with the schema, if there is one, for the server to cast to.
> The columns are named:
> * by a list or tuple of names, if one is given
> * {{value}}, for a 1-dimensional array or an array of one column
> * {{_1}}, {{_2}}, ... otherwise
> It raises:
> * {{INVALID_NDARRAY_DIMENSION}} for an array of another dimension
> * {{AXIS_LENGTH_MISMATCH}} when the names, or the fields of a schema, do not
> match the columns of the array
> The Python client instead took a numpy array as rows of Python values, which
> it could not infer a type of. {{spark.createDataFrame(np.array([1, 2]))}}
> raised {{CANNOT_INFER_TYPE_FOR_FIELD}}, and an array of 3 dimensions raised
> the same error.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
