[ 
https://issues.apache.org/jira/browse/SPARK-61335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61335:
-----------------------------------
    Labels: pull-request-available  (was: )

> Send the fill values of fillna_value and fillna_map as PySpark sends them, 
> and reject the others, in the Rust API
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61335
>                 URL: https://issues.apache.org/jira/browse/SPARK-61335
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> DataFrame::fillna_value and fillna_map of the Rust API sent a value of a type 
> other than the ones PySpark's fillna takes as the string of its Debug text: 
> Value::Null filled string columns with "Null", Value::Binary with "Binary([1, 
> 2])", and List, Map, Struct and Time likewise. And Value::Integer and 
> Value::Float, which PySpark's fillna(5) and fillna(1.5) send as a long and a 
> double, were sent as an int and a float, which the server's fill rejects 
> ("Unsupported value type java.lang.Integer (5).").
> As PySpark's fillna (NAFill._convert_value), the fix sends a bool as a 
> boolean, an integer as a long, a float as a double and a string as a string, 
> and rejects any other value with NOT_EXPECTED_TYPE and an empty map with 
> CANNOT_BE_EMPTY: fillna_value and fillna_map now return a Result, as dropna 
> does.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to