Here is a code snippet in my spark job. I added the numbers at the start of
code lines to show the relevant line numbers in exception.
45 val windowedEventCounts = events.reduceByKeyAndWindow(_ + _, _ - _, 30,
5, filterFunc = filterFunction)
val usefulEvents = windowedEventCounts.filter { case (event, count) =>
{ count > requestThreshold } }
70 usefulEvents.foreachRDD(events => { ... })
Every once in a while, I see this error in my log files.
org.apache.spark.SparkException: Checkpoint RDD ReliableCheckpointRDD[28052] at
foreachRDD at EventProcessor.scala:70(103) has different number of partitions
from original RDD MapPartitionsRDD[28050] at reduceByKeyAndWindow at
EventProcessor.scala:45(108)
Has anyone seen this issue and under what circumstances will it exception occur?
Thanks,RK
