cloud-fan commented on issue #26946: [SPARK-30036][SQL] Fix: REPARTITION hint does not work with order by URL: https://github.com/apache/spark/pull/26946#issuecomment-580861982 After more thoughts, I think it's wrong to use optimization to fix a bug. Looking into the bug, the issue is: the `Repartition` operator added by the hint is under the `Sort` operator, not above it. This is because our parser treats ORDER BY as the last clause, while the hint is associated with the SELECT clause. The parser rule is like `SELECT ... UNION/INTERSECT SELECT ... ORDER BY`. That's why we add the `Sort` operator at the end. I think #27096 is in the right way to optimize redundant shuffles, but we still need to fix the bug about how to handle hints in the parser. I'm reverting this. Let's fix the bug in the parser.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
