com.esotericsoftware.kryo.KryoException: Buffer overflow.

2014-10-21 Thread nitinkak001 
I am running a simple rdd filter command. What does it mean? 
Here is the full stack trace(and code below it): 

com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0,
required: 133
at com.esotericsoftware.kryo.io.Output.require(Output.java:138)
at
com.esotericsoftware.kryo.io.Output.writeString_slow(Output.java:420)
at com.esotericsoftware.kryo.io.Output.writeString(Output.java:326)
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$StringArraySerializer.write(DefaultArraySerializers.java:274)
at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$StringArraySerializer.write(DefaultArraySerializers.java:262)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at
org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:138)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

*Here is the code of the main function:*

/String comparisonFieldIndexes = 16,18;
String segmentFieldIndexes = 14,15;
String comparisonFieldWeights = 50, 50;
String delimiter = +'\001';

PartitionDataOnColumn parOnCol = new PartitionDataOnColumn(70,
comparisonFieldIndexes, comparisonFieldWeights, segmentFieldIndexes,
delimiter);

JavaRDDString filtered_rdd = origRDD.filter(parOnCol.new
FilterEmptyFields(parOnCol.fieldIndexes, parOnCol.DELIMITER) );

parOnCol.printRDD(filtered_rdd);/


*Here is the FilterEmptyFields class:*

/public class FilterEmptyFields implements FunctionString, Boolean
{

final int[] nonEmptyFields;
final String DELIMITER;

public FilterEmptyFields(int[] nonEmptyFields, String 
delimiter){
this.nonEmptyFields = nonEmptyFields;
this.DELIMITER = delimiter;
}

@Override
public Boolean call(String s){

String[] fields = s.split(DELIMITER);

for(int i=0; inonEmptyFields.length; i++){
if(fields[nonEmptyFields[i]] == null  ||
fields[nonEmptyFields[i]].isEmpty()){
return false;
}
}

return true;
}

}lt;/i

Any suggestions guys?



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/com-esotericsoftware-kryo-KryoException-Buffer-overflow-tp16947.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: com.esotericsoftware.kryo.KryoException: Buffer overflow.

2014-10-21 Thread Koert Kuipers 
you ran out of kryo buffer. are you using spark 1.1 (which supports buffer
resizing) or spark 1.0 (which has a fixed size buffer)?
On Oct 21, 2014 5:30 PM, nitinkak001 nitinkak...@gmail.com wrote:

 I am running a simple rdd filter command. What does it mean?
 Here is the full stack trace(and code below it):

 com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0,
 required: 133
 at com.esotericsoftware.kryo.io.Output.require(Output.java:138)
 at
 com.esotericsoftware.kryo.io.Output.writeString_slow(Output.java:420)
 at com.esotericsoftware.kryo.io.Output.writeString(Output.java:326)
 at

 com.esotericsoftware.kryo.serializers.DefaultArraySerializers$StringArraySerializer.write(DefaultArraySerializers.java:274)
 at

 com.esotericsoftware.kryo.serializers.DefaultArraySerializers$StringArraySerializer.write(DefaultArraySerializers.java:262)
 at
 com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
 at

 org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:138)
 at
 org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
 at

 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at

 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:744)

 *Here is the code of the main function:*

 /String comparisonFieldIndexes = 16,18;
 String segmentFieldIndexes = 14,15;
 String comparisonFieldWeights = 50, 50;
 String delimiter = +'\001';

 PartitionDataOnColumn parOnCol = new PartitionDataOnColumn(70,
 comparisonFieldIndexes, comparisonFieldWeights, segmentFieldIndexes,
 delimiter);

 JavaRDDString filtered_rdd = origRDD.filter(parOnCol.new
 FilterEmptyFields(parOnCol.fieldIndexes, parOnCol.DELIMITER) );

 parOnCol.printRDD(filtered_rdd);/


 *Here is the FilterEmptyFields class:*

 /public class FilterEmptyFields implements FunctionString,
 Boolean
 {

 final int[] nonEmptyFields;
 final String DELIMITER;

 public FilterEmptyFields(int[] nonEmptyFields, String
 delimiter){
 this.nonEmptyFields = nonEmptyFields;
 this.DELIMITER = delimiter;
 }

 @Override
 public Boolean call(String s){

 String[] fields = s.split(DELIMITER);

 for(int i=0; inonEmptyFields.length; i++){
 if(fields[nonEmptyFields[i]] == null  ||
 fields[nonEmptyFields[i]].isEmpty()){
 return false;
 }
 }

 return true;
 }

 }lt;/i

 Any suggestions guys?



 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/com-esotericsoftware-kryo-KryoException-Buffer-overflow-tp16947.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org