Hey Yana,

I think you posted screenshots, but they are not visible in the email.
Probably better to upload them and post links.

Are you using StreamingContext.getOrCreate? If that is being used, then it
will recreate the SparkContext with SparkConf having whatever configuration
is present in the existing checkpoint files. It may so happen that the
existing checkpoint files were from an old run which had 512 configured. So
the SparkConf in the restarted SparkContext/StremingContext is accidentally
picking up the old configuration. Deleting the checkpoint files avoided a
restart, and the new config took affect. Maybe. :)

TD

On Sat, Feb 21, 2015 at 7:30 PM, Yana Kadiyska <[email protected]>
wrote:

> Hi all,
>
> I had a streaming application and midway through things decided to up the
> executor memory. I spent a long time launching like this:
>
> ~/spark-1.2.0-bin-cdh4/bin/spark-submit --class StreamingTest
> --executor-memory 2G --master...
>
> and observing the executor memory is still at old 512 setting
>
> I was about to ask if this is a bug when I decided to delete the
> checkpoints. Sure enough the setting took after that.
>
> So my question is -- why is it required to remove checkpoints to increase
> memory allowed on an executor? This seems pretty un-intuitive to me.
>
> Thanks for any insights.
>

Reply via email to