[Spark Streaming] Runtime Error in call to max function for JavaPairRDD

[Nipun Arora]

Hi,

I have the following piece of code, where I am trying to transform a spark
stream and add min and max to it of eachRDD. However, I get an error saying
max call does not exist, at run-time (compiles properly). I am using
spark-1.4

I have added the question to stackoverflow as well:
http://stackoverflow.com/questions/30902090/adding-max-and-min-in-spark-stream-in-java/30909796#30909796

Any help is greatly appreciated :)

Thanks
Nipun

JavaPairDStream<Tuple2<Long, Integer>, Tuple3<Integer,Long,Long>>
sortedtsStream = transformedMaxMintsStream.transformToPair(new
Sort2());

sortedtsStream.foreach(
        new Function<JavaPairRDD<Tuple2<Long, Integer>,
Tuple3<Integer, Long, Long>>, Void>() {
            @Override
            public Void call(JavaPairRDD<Tuple2<Long, Integer>,
Tuple3<Integer, Long, Long>> tuple2Tuple3JavaPairRDD) throws Exception
{
                List<Tuple2<Tuple2<Long, Integer>,
Tuple3<Integer,Long,Long>> >templist =
tuple2Tuple3JavaPairRDD.collect();
                for(Tuple2<Tuple2<Long,Integer>,
Tuple3<Integer,Long,Long>> tuple :templist){

                    Date date = new Date(tuple._1._1);
                    int pattern = tuple._1._2;
                    int count = tuple._2._1();
                    Date maxDate = new Date(tuple._2._2());
                    Date minDate = new Date(tuple._2._2());
                    System.out.println("TimeSlot: " + date.toString()
+ " Pattern: " + pattern + " Count: " + count + " Max: " +
maxDate.toString() + " Min: " + minDate.toString());

                }
                return null;
            }
        }
);

Error:


15/06/18 11:05:06 INFO BlockManagerInfo: Added input-0-1434639906000
in memory on localhost:42829 (size: 464.0 KB, free: 264.9 MB)15/06/18
11:05:06 INFO BlockGenerator: Pushed block
input-0-1434639906000Exception in thread "JobGenerator"
java.lang.NoSuchMethodError:
org.apache.spark.api.java.JavaPairRDD.max(Ljava/util/Comparator;)Lscala/Tuple2;
        at 
org.necla.ngla.spark_streaming.MinMax.call(Type4ViolationChecker.java:346)
        at 
org.necla.ngla.spark_streaming.MinMax.call(Type4ViolationChecker.java:340)
        at 
org.apache.spark.streaming.api.java.JavaDStreamLike$class.scalaTransform$3(JavaDStreamLike.scala:360)
        at 
org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$transformToPair$1.apply(JavaDStreamLike.scala:361)
        at 
org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$transformToPair$1.apply(JavaDStreamLike.scala:361)
        at 
org.apache.spark.streaming.dstream.DStream$$anonfun$transform$1$$anonf