[
https://issues.apache.org/jira/browse/SPARK-61288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Meehan updated SPARK-61288:
-------------------------------
Fix Version/s: connect-rust-4.3.0
> Draw the seeds of sample, randomSplit and sampleBy from Python's random in
> the Python client, as PySpark does
> -------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61288
> URL: https://issues.apache.org/jira/browse/SPARK-61288
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Tim Meehan
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Without a seed, PySpark's sample, randomSplit and sampleBy draw one with
> random.randint(0, sys.maxsize) (_preapare_args_for_sample in
> pyspark/sql/dataframe.py:2165, and pyspark/sql/connect/dataframe.py:1124 and
> :1844), so after random.seed(...) they return the same rows in every process.
> In the Python client of spark-connect-rust, sample and randomSplit take their
> seed from a UUID, and sampleBy sends none, so random.seed() has no effect on
> them.
> {code:python}
> import random
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
> random.seed(42)
> print(sorted(r.id for r in spark.range(40).sample(0.5).collect()))
> {code}
> Run twice, in a new process each time, against an OSS Spark 4.3.0 Connect
> server on loopback:
> * pyspark-client 4.3.0: \[1, 2, 3, 8, 9, 10, 11, 14, 15, 16, 17, 20, 22, 23,
> 24, 26, 27, 29, 30, 32, 33, 37\] both times. randomSplit(\[0.5, 0.5\]) and
> sampleBy also return the same rows in both processes.
> * spark-connect-rust master 8768cc4: \[3, 5, 8, 11, 12, 13, 14, 24, 26, 27,
> 28, 29, 30, 31, 32, 33, 35, 38, 39\], then \[0, 4, 8, 11, 13, 14, 15, 19, 20,
> 21, 22, 24, 25, 29, 32, 34, 37, 38, 39\]. randomSplit and sampleBy also
> differ between the two processes.
> From source: sample and randomSplit fall back to random_seed
> (crates/spark-connect/src/dataframe.rs:898, :2004), which derives the seed
> from uuid::Uuid::new_v4() (:2561), and StatSampleBy sets a seed only when one
> is given (crates/spark-connect/src/plan.rs:1700).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
