aokolnychyi commented on pull request #31083: URL: https://github.com/apache/spark/pull/31083#issuecomment-767099741
@cloud-fan, I tried to implement the suggestion to let AQE set the number of partitions but it changes the output partitioning to be `UnknownPartitioning` due to the logic in `CustomShuffleReaderExec` and `CoalescedPartitionSpec`. I created SPARK-34230 to reconsider this but will keep the current logic for now. I'd appreciate more input on AQE. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
