Tathagata, yes, I was using StreamingContext.getOrCreate. My question is about the design decision here. I was expecting that if I have a streaming application that say crashed, and I wanted to give the executors more memory, I would be able to restart, using the checkpointed RDD but with more memory.
I thought deleting the checkpoints in a checkpointed application is the last thing that you want to do (as you lose all state). Seems a bit harsh to have to do this just to increase the amount of memory? On Mon, Feb 23, 2015 at 11:12 PM, Tathagata Das <t...@databricks.com> wrote: > Hey Yana, > > I think you posted screenshots, but they are not visible in the email. > Probably better to upload them and post links. > > Are you using StreamingContext.getOrCreate? If that is being used, then it > will recreate the SparkContext with SparkConf having whatever configuration > is present in the existing checkpoint files. It may so happen that the > existing checkpoint files were from an old run which had 512 configured. So > the SparkConf in the restarted SparkContext/StremingContext is accidentally > picking up the old configuration. Deleting the checkpoint files avoided a > restart, and the new config took affect. Maybe. :) > > TD > > On Sat, Feb 21, 2015 at 7:30 PM, Yana Kadiyska <yana.kadiy...@gmail.com> > wrote: > >> Hi all, >> >> I had a streaming application and midway through things decided to up the >> executor memory. I spent a long time launching like this: >> >> ~/spark-1.2.0-bin-cdh4/bin/spark-submit --class StreamingTest >> --executor-memory 2G --master... >> >> and observing the executor memory is still at old 512 setting >> >> I was about to ask if this is a bug when I decided to delete the >> checkpoints. Sure enough the setting took after that. >> >> So my question is -- why is it required to remove checkpoints to increase >> memory allowed on an executor? This seems pretty un-intuitive to me. >> >> Thanks for any insights. >> > >
