[
https://issues.apache.org/jira/browse/JENA-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15474301#comment-15474301
]
Rob Vesse commented on JENA-1233:
---------------------------------
Even if you use Elephas it won't work entirely out-of-the-box if memory serves
because I don't think {{Writable}} extends {{Serializable}}
I experimented a bit with RDF and spark using Elephas and ended up creating
some Scala wrapper classes around the Elephas {{Writable}} implementations to
make Spark happy. This is however all internal prototype code and not open
source anywhere.
The other thing you can try is configuring spark to use the Kryo serialiser
which will serialise objects that are not {{Serializable}} though I'm not sure
is that gets invoked by the {{saveAsObjectFile()}} method.
> Make RDF primitives Serializable
> --------------------------------
>
> Key: JENA-1233
> URL: https://issues.apache.org/jira/browse/JENA-1233
> Project: Apache Jena
> Issue Type: Improvement
> Components: Elephas
> Affects Versions: Jena 3.1.0
> Reporter: Itsuki Toyota
>
> I always use Jena when I handle RDF data with Apache Spark.
> However, when I want to store resulting RDD data (ex. RDD[Triple]) in binary
> format, I can't call RDD.saveAsObjectFile method.
> It's because RDD.saveAsObjectFile requires java.io.Serializable interface.
> See the following code.
> https://github.com/apache/spark/blob/v1.6.0/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L1469
> https://github.com/apache/spark/blob/v1.6.0/core/src/main/scala/org/apache/spark/util/Utils.scala#L79-L86
> You can see that
> 1) RDD.saveAsObjectFile calls Util.serialize method
> 2) Util.serialize method requires the RDD-wrapped object implementing
> java.io.Serializable interface. For example, if you want to save a
> RDD[Triple] object, Triple must implements java.io.Serializable.
> So why not implement java.io.Serializable ?
> I think it will improve the usability in Apache Spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
