cloud-fan commented on pull request #34785: URL: https://github.com/apache/spark/pull/34785#issuecomment-1002582437
Can you update the PR description and fix the code conflicts? Seems we don't need to change `RepartitionByExpression` anymore. IIUC, the new proposal of this PR is: For distribution required by the v2 sink, we use Rebalance operator instead of `RepartitionByExpression`. This looks reasonable, as it's for performance and best-effort partitioning is good enough. AFAFIK @wangyum is working on something similar for file sources. We should update the DS v2 API doc though. To mention that the required distribution is just a best effort. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
