0
I get the following error on executors while running my spark job. I am
reading data from Database. The data has string in UTF8
Iterator t.next().getString(row.fieldIndex("short_name"));
ERROR org.apache.spark.util.SparkUncaughtExceptionHandler - Uncaught
exception in thread Thread[Executor task launch worker for task 359,5,main]
java.lang.OutOfMemoryError: Java heap space at
org.apache.spark.unsafe.types.UTF8String.fromAddress(UTF8String.java:135)
at
org.apache.spark.sql.catalyst.expressions.UnsafeRow.getUTF8String(UnsafeRow.java:419)
at
org.apache.spark.sql.execution.columnar.STRING$.getField(ColumnType.scala:452)
at
org.apache.spark.sql.execution.columnar.STRING$.getField(ColumnType.scala:424)
at
org.apache.spark.sql.execution.columnar.compression.RunLengthEncoding$Encoder.gatherCompressibilityStats(compressionSchemes.scala:194)
at
org.apache.spark.sql.execution.columnar.compression.CompressibleColumnBuilder$$anonfun$gatherCompressibilityStats$1.apply(CompressibleColumnBuilder.scala:74)
at
org.apache.spark.sql.execution.columnar.compression.CompressibleColumnBuilder$$anonfun$gatherCompressibilityStats$1.apply(CompressibleColumnBuilder.scala:74)
at scala.collection.immutable.List.foreach(List.scala:392) at
org.apache.spark.sql.execution.columnar.compression.CompressibleColumnBuilder$class.gatherCompressibilityStats(CompressibleColumnBuilder.scala:74)
I am processing 100 GB of data with 10 executors of 14G. I startted with
12G executors and I get the same error even with 14G and 3G Overhead memory.
Thanks,
Nimmi
