[
https://issues.apache.org/jira/browse/SPARK-61329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61329:
------------------------------------
Assignee: Hyukjin Kwon
> Draw a new seed for each call of the random functions of the Rust API, as
> PySpark does
> --------------------------------------------------------------------------------------
>
> Key: SPARK-61329
> URL: https://issues.apache.org/jira/browse/SPARK-61329
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The random functions of the Rust API (spark_connect::functions::rand, randn,
> uuid, shuffle, uniform, randstr and count_min_sketch) send the same fixed
> seed literal in every call, in every query, session and process. So uuid()
> returns the same UUIDs in every run (surrogate keys collide across appends),
> two rand() columns of one select are equal, and orderBy(rand()) or
> filter(rand() < p) pick the same rows every time.
> PySpark 4.3.0 (pyspark.sql.connect.functions.builtin) draws a new seed with
> random.randint(0, sys.maxsize) for each call without one, and accepts an
> explicit seed; the Scala client does the same with random.nextLong. The
> Python drop-in is not affected: its functions draw from Python's random.
> The fix draws a new non-negative seed for each call, adds _with_seed variants
> for an explicit seed (Long for rand and randn, a Column for the others, as
> the Scala client takes them), and adds random and random_with_seed, PySpark's
> alias of rand.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]