Tim Meehan created SPARK-61288:
----------------------------------
Summary: Draw the seeds of sample, randomSplit and sampleBy from
Python's random in the Python client, as PySpark does
Key: SPARK-61288
URL: https://issues.apache.org/jira/browse/SPARK-61288
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
Without a seed, PySpark's sample, randomSplit and sampleBy draw one with
random.randint(0, sys.maxsize) (_preapare_args_for_sample in
pyspark/sql/dataframe.py:2165, and pyspark/sql/connect/dataframe.py:1124 and
:1844), so after random.seed(...) they return the same rows in every process.
In the Python client of spark-connect-rust, sample and randomSplit take their
seed from a UUID, and sampleBy sends none, so random.seed() has no effect on
them.
{code:python}
import random
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
random.seed(42)
print(sorted(r.id for r in spark.range(40).sample(0.5).collect()))
{code}
Run twice, in a new process each time, against an OSS Spark 4.3.0 Connect
server on loopback:
* pyspark-client 4.3.0: \[1, 2, 3, 8, 9, 10, 11, 14, 15, 16, 17, 20, 22, 23,
24, 26, 27, 29, 30, 32, 33, 37\] both times. randomSplit(\[0.5, 0.5\]) and
sampleBy also return the same rows in both processes.
* spark-connect-rust master 8768cc4: \[3, 5, 8, 11, 12, 13, 14, 24, 26, 27, 28,
29, 30, 31, 32, 33, 35, 38, 39\], then \[0, 4, 8, 11, 13, 14, 15, 19, 20, 21,
22, 24, 25, 29, 32, 34, 37, 38, 39\]. randomSplit and sampleBy also differ
between the two processes.
>From source: sample and randomSplit fall back to random_seed
>(crates/spark-connect/src/dataframe.rs:898, :2004), which derives the seed
>from uuid::Uuid::new_v4() (:2561), and StatSampleBy sets a seed only when one
>is given (crates/spark-connect/src/plan.rs:1700).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
