Re: Configure spark.kryoserializer.buffer.max at runtime does not take effect

yeah I feel like this is a bug since you can't really modify the settings once you were given spark session or spark context. so the work around would be to use --conf. In your case it would be like this ./spark-shell --conf spark.kryoserializer.buffer.max=1g On Thu, Nov 17, 2016 at 1:59 PM,

Re: Configure spark.kryoserializer.buffer.max at runtime does not take effect

getOrCreate uses existing SparkSession if available, in which case the settings will be ignored On Wed, Nov 16, 2016 at 10:55 PM, bluishpenguin wrote: > Hi all, > I would like to configure the following setting during runtime as below: > > spark = (SparkSession >

Configure spark.kryoserializer.buffer.max at runtime does not take effect

Hi all, I would like to configure the following setting during runtime as below: spark = (SparkSession .builder .appName("ElasticSearchIndex") .config("spark.kryoserializer.buffer.max", "1g") .getOrCreate()) But I still hit error, Caused by: org.apache.spark.SparkException: