Hi all,
I would like to configure the following setting during runtime as below:
spark = (SparkSession
.builder
.appName("ElasticSearchIndex")
.config("spark.kryoserializer.buffer.max", "1g")
.getOrCreate())
But I still hit error,
Caused by: org.apache.spark.SparkException: Kryo serialization failed:
Buffer overflow. Available: 0, required: 1614707. To avoid this, increase
spark.kryoserializer.buffer.max value.
It works when configure it along with the spark-submit command as below:
spark-submit pyspark-shell-main --name "PySparkShell" --conf
spark.kryoserializer.buffer.max=1g
Any idea what have I done wrong?
Thank you.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Configure-spark-kryoserializer-buffer-max-at-runtime-does-not-take-effect-tp28094.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
