Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/19560
@wzhfy
Thanks for comment;
I know your point.
In my cluster, namenode is under heavy pressure. Errors in stats happen
with big chance. Users always do not know there's error in stats. That's why I
propose this config. Users can chose to turn on this config by default. Yes, it
will hit performance. But it's only when Join and the `totalSize` from
metastore is below `spark.sql.autoBroadcastJoinThreshold`, which I think is
acceptable.
Like I mentioned in description, this is for defense. Help to avoid OOM
caused by broadcast join. It's for stability.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
