[ 
https://issues.apache.org/jira/browse/SPARK-61405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61405:
---------------------------------
    Fix Version/s: connect-rust-4.3.0

> Draw a seed of the client for sample_by without one, take PySpark's strata 
> types, and fill the columns of a map in fillna_value, in the Rust API as 
> PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61405
>                 URL: https://issues.apache.org/jira/browse/SPARK-61405
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> Three differences of the Rust API's DataFrameStatFunctions.sampleBy and 
> fillna from PySpark's:
> 1. StatFunctions::sample_by(col, fractions, None) sends no seed. The server 
> then draws one (Utils.random.nextLong) each time it plans the relation, so 
> one DataFrame returns different rows in different actions: count() changes 
> from call to call and disagrees with collect(), and a self-join pairs two 
> different samples. PySpark draws the seed on the client (random.randint(0, 
> sys.maxsize)), as the Rust API's sample and random_split do since 
> SPARK-61329's siblings.
> 2. sample_by takes a stratum literal of any type, but the server matches 
> date, timestamp, decimal and binary strata against no row, so such a sample 
> is silently empty. PySpark takes only float, int (and bool) and str keys and 
> raises DISALLOWED_TYPE_FOR_CONTAINER for others.
> 3. fillna_value(Value::Map(..)) fails with "[NOT_EXPECTED_TYPE] Argument 
> `value` should be bool, dict, float, int or str, got dict.", which names dict 
> as both accepted and rejected. PySpark's fillna takes a dict and fills its 
> columns with its values, ignoring subset.
> Fix: draw the seed on the client when none is given; take only the strata 
> types of PySpark, with its error; fill the columns of a map of string keys as 
> fillna_map does, with PySpark's errors for a key that is not a string, a 
> value of another type and an empty map.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to