Raviston6296 commented on issue #59434: URL: https://github.com/apache/spark/issues/59434#issuecomment-6570979658
This is the same root cause as SPARK-57956 (#57347): during AQE re-optimization, `LogicalQueryStage.maxRows` used a cost estimate of a not-yet-materialized stage as a hard upper bound, so `EliminateLimits` could drop a LIMIT. That fix is in `master`, `branch-4.x` and `branch-4.3`, but not in `branch-4.2` or `branch-4.1`. I reproduced the query from this issue on `branch-4.2` and `branch-4.1` (RDD-backed tables, `local[1]`, `spark.sql.shuffle.partitions=2`, 200 runs with AQE on): 19 and 18 runs returned 5 rows. With SPARK-57956 backported, all 200 runs return 3 rows on both branches. Current master also returns 3 rows in every run. Backports: #60017 (`branch-4.2`) and #60018 (`branch-4.1`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
