JkSelf commented on pull request #32683: URL: https://github.com/apache/spark/pull/32683#issuecomment-850179735
At present, AQE uses the compressed size to do conditional judgment. In some extreme cases, such as when the compression ratio is relatively high, if smj is converted to shj when enable AQE, it is very likely that the oom issue will happen when building the hash table in shj. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
