The following Scala (Spark 1.6) code for reading a value from a Row fails
with a NullPointerException when the value is null.
val test = row.getAs[Int]("ColumnName").toString
while this works fine
val test1 = row.getAs[Int]("ColumnName") // returns 0 for nullval
test2 = test1.toString // converts to String fine
What is causing NullPointerException and what is the recommended way to
handle such cases?
PS: getting row from DataFrame as follows:
val myRDD = myDF
.repartition(partitions)
.mapPartitions {
rows =>
rows.flatMap {
row =>
functionWithRows(row) //has above logic to read null
column which fails
}
}
functionWithRows has then above mentioned NullPointerException
MyDF schema:
root
|-- LDID: string (nullable = true)
|-- KTAG: string (nullable = true)
|-- ColumnName: integer (nullable = true)
