Hyukjin Kwon created SPARK-61103:
------------------------------------
Summary: Create a DataFrame of a numpy array in createDataFrame in
the Python client as PySpark does
Key: SPARK-61103
URL: https://issues.apache.org/jira/browse/SPARK-61103
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{spark.createDataFrame}} takes a numpy array of 1 or 2
dimensions. It builds an Arrow table of a column of each column of the array,
and sends it with the schema, if there is one, for the server to cast to.
The columns are named:
* by a list or tuple of names, if one is given
* {{value}}, for a 1-dimensional array or an array of one column
* {{_1}}, {{_2}}, ... otherwise
It raises:
* {{INVALID_NDARRAY_DIMENSION}} for an array of another dimension
* {{AXIS_LENGTH_MISMATCH}} when the names, or the fields of a schema, do not
match the columns of the array
The Python client instead took a numpy array as rows of Python values, which it
could not infer a type of. {{spark.createDataFrame(np.array([1, 2]))}} raised
{{CANNOT_INFER_TYPE_FOR_FIELD}}, and an array of 3 dimensions raised the same
error.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
