maryannxue commented on PR #43435: URL: https://github.com/apache/spark/pull/43435#issuecomment-1785553724
@eejbyfeldt Can you briefly describe the triggering condition of this bug? Does it only occur when coalescing happens to produce just the exact number of partitions as the other side of the join? In the meantime, I'm wondering if it would be better to: 1. not coalesce for the top/last shuffle of the physical plan of InMemoryTableScan 2. have coalesce rule deal with `InMemoryTableScan` from the caller side (user of the cache) This PR, just to address the correctness issue, only needs to do 1. And we can do 2 (a little trickier I suppose) for performance improvement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
