wangshisan opened a new pull request #23627: [SPARK-26672] SinglePartition may not satisfies HashClusteredDistribu… URL: https://github.com/apache/spark/pull/23627 …tion/OrderedDistribution [SPARK-26672] SinglePartition may not satisfies HashClusteredDistribution/OrderedDistribution ## What changes were proposed in this pull request? Fix a bug which may result lack of shuffle. Say, If we are loading data to a bucketed table TEST_TABLE of which bucket number is not 1, from another table SRC_TABLE(bucketed or not) with sql: insert overwrite table TEST_TABLE select * from SRC_TABLE limit 1000. Data inserted into TEST_TABLE will not be bucketed since after LimitExec the output partitioning will be SinglePartition, and in current logic, it satisfies the HashClusteredDistribution, so no shuffle will be added. ## How was this patch tested? unit tests,
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
