Currently configuration is a part of checkpoint data, and when recovering from failure, Spark Streaming will fetch the configuration from checkpoint data, so even if you change the configuration file, recovered Spark Streaming application will not use it. So from my understanding currently there's no way to handle your situation.
Thanks Saisai On Tue, Mar 15, 2016 at 5:12 PM, Ewan Leith <ewan.le...@realitymine.com> wrote: > Has anyone seen a way of updating the Spark streaming job configuration > while retaining the existing data in the write ahead log? > > > > e.g. if you’ve launched a job without enough executors and a backlog has > built up in the WAL, can you increase the number of executors without > losing the WAL data? > > > > Thanks, > > Ewan >
