Re: KryoSerializer exception in Spark Streaming JAVA

[bonnahu]

class MyRegistrator implements KryoRegistrator { 

public void registerClasses(Kryo kryo) { 
kryo.register(ImpressionFactsValue.class); 
} 
  
} 

change this class to public and give a try 



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/KryoSerializer-exception-in-Spark-Streaming-JAVA-tp15479p20647.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

KryoSerializer exception in Spark Streaming JAVA

[Mudassar Sarwar]

Hi,

I'm implementing KryoSerializer for my custom class. Here is class

public class ImpressionFactsValue implements KryoSerializable {

private int hits;

public ImpressionFactsValue() {

}
 
public int getHits() {
return hits;
}

public void setHits(int hits) {
this.hits = hits;
}
public void read(Kryo kryo, Input input) {
 this.hits = input.readInt();
 //kryo.readClass(input);
}

public void write(Kryo arg0, Output output) {
 output.writeInt(this.hits); 
 //arg0.writeClass(output, ImpressionFactsValue.class);
 
}
}

Here is my KryoRegistrator 

class MyRegistrator implements KryoRegistrator {

public void registerClasses(Kryo kryo) {
kryo.register(ImpressionFactsValue.class);
}
  
}

Here is spark conf

conf.set(spark.serializer, org.apache.spark.serializer.KryoSerializer); 
 
conf.set(spark.kryo.registrator, com.MyRegistrator);

Getting exception

14/10/01 17:15:38 ERROR serializer.KryoSerializer: Failed to run
spark.kryo.registrator
java.lang.IllegalAccessException: Class
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2 can not access
a member of class com.MyRegistrator with modifiers 
at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:95)
at java.lang.Class.newInstance0(Class.java:366)
at java.lang.Class.newInstance(Class.java:325)
at
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2.apply(KryoSerializer.scala:78)
at
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2.apply(KryoSerializer.scala:76)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:76)
at
org.apache.spark.serializer.KryoSerializerInstance.init(KryoSerializer.scala:133)
at
org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:95)
at
org.apache.spark.util.collection.ExternalAppendOnlyMap.init(ExternalAppendOnlyMap.scala:109)
at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:57)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$1.apply(PairRDDFunctions.scala:96)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$1.apply(PairRDDFunctions.scala:95)
at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:594)
at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:594)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:158)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.Task.run(Task.scala:51)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

Please help to resolve this





--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/KryoSerializer-exception-in-Spark-Streaming-JAVA-tp15479.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org