JkSelf commented on pull request #32683:
URL: https://github.com/apache/spark/pull/32683#issuecomment-850227847
- In the production environment, we have encountered oom issue when AQE is
turned on to convert smj to bhj. The reason is that the compression ratio of
the data is too large, the size after compression is 16M, but the data after
decompression is 2GB when building the hash table. I think this concern still
exists in shj.
- Also this is a big changes and it is better to have a config to control
this feature. So even if OOM appears, users can bypass this problem by
disabling this switch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
