Hi,
I'm using structured streaming to count unique visits of our website. I use spark on yarn mode with 4 executor instances and from 2 cores * 5g memory to 4 cores * 10g memory for each executor, but there are frequent full gc, and once the count raises to about more than 4.5 millions the application will be blocked and finally crash in OOM. It's kind of unreasonable. So is there any suggestion to optimize the memory consumption of SS? Thanks.
