Hi,
I'm trying to create a custom version of saveAsObject(). however, I do not
seem to be able to use SequenceFileRDDFunctions in my package.
I simply copy/pasted saveAsObject() body to my funtion:
out.mapPartitions(iter => iter.grouped(10).map(_.toArray))
.map(x => (NullWritable.get(), new BytesWritable(serialize(x))))
.*saveAsSequenceFile*("output")
But that gives me this error:
value saveAsSequenceFile is not a member of
org.apache.spark.rdd.RDD[(org.apache.hadoop.io.NullWritable,
org.apache.hadoop.io.BytesWritable)]
possible cause: maybe a semicolon is missing before `value
saveAsSequenceFile'?
.saveAsSequenceFile("output")
^
Scala implicit conversion error is not of any help here. So I tried to
apply explicit conversion:
*org.apache.spark.rdd.SequenceFileRDDFunctions[(NullWritable,
BytesWritable)](*out.mapPartitions(iter => iter.grouped(10).map(_.toArray))
.map(x => (NullWritable.get(), new BytesWritable(serialize(x))))*)*
.saveAsSequenceFile("output")
Giving me this error:
object SequenceFileRDDFunctions is not a member of package
org.apache.spark.rdd
*Note: class SequenceFileRDDFunctions exists, but it has no companion
object.*
org.apache.spark.rdd.SequenceFileRDDFunctions[(NullWritable,
BytesWritable)](out.mapPartitions(iter => iter.grouped(10).map(_.toArray))
^
Is this scala compiler version mismatch hell?
