Linhong Liu created SPARK-61561:
-----------------------------------

             Summary: TABLESAMPLE REPEATABLE (and seeded Dataset.sample) over 
ORDER BY returns different rows on every execution because RangePartitioner 
seeds bounds with the RDD id
                 Key: SPARK-61561
                 URL: https://issues.apache.org/jira/browse/SPARK-61561
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 5.0.0
            Reporter: Linhong Liu


RangePartitioner derives the seeds for the samples that pick its range bounds 
from the RDD id (RangePartitioner.sketch uses shift = rdd.id, and the 
imbalanced re-sample uses byteswap32(-rdd.id - 1)). A global sort builds a new 
RDD, with a new id, on every execution, so the range bounds of the same data 
move between runs. A seeded TABLESAMPLE ... REPEATABLE (or a seeded 
Dataset.sample) over an ORDER BY therefore returns different rows each time.

Fix: derive the sketch and re-sample seeds from the partition index only, not 
the RDD id. A new conf spark.rdd.rangePartitioner.legacyRddIdSeed (default 
false) restores the previous behavior. Found via correctness analysis.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to