[ 
https://issues.apache.org/jira/browse/SPARK-61561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61561:
-----------------------------------
    Labels: pull-request-available  (was: )

> TABLESAMPLE REPEATABLE (and seeded Dataset.sample) over ORDER BY returns 
> different rows on every execution because RangePartitioner seeds bounds with 
> the RDD id
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61561
>                 URL: https://issues.apache.org/jira/browse/SPARK-61561
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 5.0.0
>            Reporter: Linhong Liu
>            Priority: Major
>              Labels: pull-request-available
>
> RangePartitioner derives the seeds for the samples that pick its range bounds 
> from the RDD id (RangePartitioner.sketch uses shift = rdd.id, and the 
> imbalanced re-sample uses byteswap32(-rdd.id - 1)). A global sort builds a 
> new RDD, with a new id, on every execution, so the range bounds of the same 
> data move between runs. A seeded TABLESAMPLE ... REPEATABLE (or a seeded 
> Dataset.sample) over an ORDER BY therefore returns different rows each time.
> Fix: derive the sketch and re-sample seeds from the partition index only, not 
> the RDD id. A new conf spark.rdd.rangePartitioner.legacyRddIdSeed (default 
> false) restores the previous behavior. Found via correctness analysis.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to