The error is in your code, which you don't show. You are almost certainly incorrectly referencing something like a SparkContext in a Spark task.
On Wed, Jun 30, 2021 at 3:48 PM Amit Sharma <resolve...@gmail.com> wrote: > Hi , I am using spark 2.7 version with scala. I am calling a method as > below > > 1. val rddBacklog = spark.sparkContext.parallelize(MAs) // MA is list of say > city > > 2. rddBacklog.foreach(ma => doAlloc3Daily(ma, fteReview.forecastId, > startYear, endYear)) > > 3.doAlloc3Daily method just doing a database call and doing some scala > calculation (no rdd or dataframe) > > > Line number 2 I am getting below nullpointer intermittently on cluster but > never on local. > > java.lang.NullPointerException > at > sparkStreaming.CalculateFteReview.doAlloc3Daily(CalculateFteReview.scala:1307) > at > sparkStreaming.CalculateFteReview$$anonfun$getNewDistribution$2.apply(CalculateFteReview.scala:1199) > at > sparkStreaming.CalculateFteReview$$anonfun$getNewDistribution$2.apply(CalculateFteReview.scala:1199) > at scala.collection.Iterator$class.foreach(Iterator.scala:893) > at > org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) > at > org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:927) > at > org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:927) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:109) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > > Thanks > > Amit > > > > > >
