RexXiong commented on PR #3279: URL: https://github.com/apache/celeborn/pull/3279#issuecomment-2909969138
> @saurabhd336 Hi, thanks for your enthusiasm. Have you noticed a PR(#2373) that aims to eliminate the sort? Thanks @saurabhd336, and your idea, Frankly speaking, we have considered this approach previously. While it does eliminate shuffle file sorting, it also introduces some side effects. Therefore, we opted for another solution, as mentioned by @FMX in [CELEBORN-1319](https://issues.apache.org/jira/browse/CELEBORN-1319). This solution completely eliminates partition file sorting and performs well in handling AQE skew cases. For a detailed design, please refer to [CIP-18](https://cwiki.apache.org/confluence/display/CELEBORN/CIP-18+Support+read+skew+partition+by+Celeborn+chunks+offsets). Could you please check if it meets your needs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
