Hyukjin Kwon created SPARK-61335:
------------------------------------
Summary: Send the fill values of fillna_value and fillna_map as
PySpark sends them, and reject the others, in the Rust API
Key: SPARK-61335
URL: https://issues.apache.org/jira/browse/SPARK-61335
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
DataFrame::fillna_value and fillna_map of the Rust API sent a value of a type
other than the ones PySpark's fillna takes as the string of its Debug text:
Value::Null filled string columns with "Null", Value::Binary with "Binary([1,
2])", and List, Map, Struct and Time likewise. And Value::Integer and
Value::Float, which PySpark's fillna(5) and fillna(1.5) send as a long and a
double, were sent as an int and a float, which the server's fill rejects
("Unsupported value type java.lang.Integer (5).").
As PySpark's fillna (NAFill._convert_value), the fix sends a bool as a boolean,
an integer as a long, a float as a double and a string as a string, and rejects
any other value with NOT_EXPECTED_TYPE and an empty map with CANNOT_BE_EMPTY:
fillna_value and fillna_map now return a Result, as dropna does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]