Awesome. Thanks so much for sharing the solution! It's very appreciated.
On Apr 29, 2015 6:14 AM, "madhvi" <[email protected]> wrote:
> Hi Josh,
>
> The classes which are not serialized are automatically serialized by
> kryo.It is used as follows while making the sparkConf object:
> val conf = new SparkConf()
>
> .setAppName(detail)
>
> .set("spark.serializer",
> "org.apache.spark.serializer.KryoSerializer")
>
> .set("spark.driver.maxResultSize", arguments.get("maxResultSize"
> ).get)
> .registerKryoClasses(Array(classOf[org.apache.accumulo.core.data.Key]))
>
> Thanks
> Madhvi
> On Tuesday 28 April 2015 11:32 PM, Josh Elser wrote:
>
> Thanks for the report back, Vaibhav.
>
> To clarify, did you have to write code to do the serialization with Kryo
> or was this something that Kryo could do automatically.
>
> If you could expand on the steps you took, that'd be awesome as I assume
> it will be extremely helpful in the mail archives for others who run into
> this problem.
>
> vaibhav thapliyal wrote:
>
> Hi josh,
>
> We solved it using the kryo serializer library to serialise the key class.
>
> Thanks
> vaibhav
>
> On 28-Apr-2015 11:14 pm, "Josh Elser" <[email protected]
> <mailto:[email protected]> <[email protected]>> wrote:
>
> Hi Madhvi,
>
> Thanks for posting this. I'm not super familiar, but my hunch is
> that Spark requires objects that it works with to implement the Java
> Serializable interface.
>
> Accumulo deals with Key (and Value) through Hadoop's Writable
> interface (technically WritableComparable, but still stems from
> Writable). I'm not sure if there's a way that you can inform Spark
> to use the Writable interface methods instead of Serializable.
>
> If there isn't a way to do this, I don't see any reason why we
> couldn't make Key (and Value) also implement Serializable for this
> use case. Please open an issue on JIRA for this -- we can track the
> investigation there.
>
> Thanks!
>
> - Josh
>
> madhvi wrote:
>
> Hi,
>
> While connecting to accumulo through spark by making sparkRDD I am
> getting the following error:
> object not serializable (class: org.apache.accumulo.core.data.Key)
>
> This is due to the 'key' class of accumulo which does not
> implement
> serializable interface.How it can be solved and accumulo can be
> used
> with spark
>
> Thanks
> Madhvi
>
>
>
