Can you show the full query you are running? On Tue, Sep 12, 2017 at 10:11 AM, 张万新 <kevinzwx1...@gmail.com> wrote:
> Hi, > > I'm using structured streaming to count unique visits of our website. I > use spark on yarn mode with 4 executor instances and from 2 cores * 5g > memory to 4 cores * 10g memory for each executor, but there are frequent > full gc, and once the count raises to about more than 4.5 millions the > application will be blocked and finally crash in OOM. It's kind of > unreasonable. So is there any suggestion to optimize the memory consumption > of SS? Thanks. >
