Hi Jason On 08/06/2019 11:49, Scarlet Remilia wrote:
Hello everyone,I changed model to triples in RDD/Dataset, but there is a question. I have triples in Dataset of Spark now, and I need to put them into a Model or something else ,then output them into a file or TDB or somewhere else. As Dan mentioned before, is there any binary syntax for RDF?
https://jena.apache.org/documentation/io/rdf-binary.html org.apache.jena.riot.thrift.*
Or Is Jena supported distributed model to handling billions triples?(supporting parsing triples into a RDF file is OK).TDB’s MRSW is a quite problem for me.
(It's MR+SW - multiple reader AND single writer) Are you wanting to load smallish units of triples from multiple sources?Maybe you want to have all the streams send their output to a queue (in blocks, not triple by triple) and have TDB load from that queue. Multiple StreamRDF to a single StreamRDF, load the StreamTDB.
There is the TDB2 parallel loader - that is, loading from a single source using internal parallelism, not loading from parallel inputs. (It's 5 threads for triples, more for quads). It load from a StreamRDF.
NB - it can consume all the server's I/O bandwidth and a lot of CPU to make the machine unusable for anything else. It is quite hardware dependent.
Andy
Thank you very much! Jason Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 ________________________________ From: Andy Seaborne <[email protected]> Sent: Thursday, June 6, 2019 6:35:41 PM To: [email protected] Subject: Re: Jena Model is serializable in Java? On 06/06/2019 08:57, Scarlet Remilia wrote:Hello everyone, My use case is a r2rml implementation, which could support millions or billions rows from RDBMS and distributed parse them into RDF. For now, We try to setup some small models in different spark executors to parse individually, and finally union them all.That sounds more like a stream usage. Jena's StreamRDF and collect to a set (model or graph don't sound like they do anything for your application - sound like you are just using them as container of triples to move around.I think RDD[Triple] is a good idea, but I need to review exist code to change model into triples. an RDF syntax and write-then-read the RDF is also a resolution but is too loose. It’s very hard to manage these files, especially there are too many small models mentioned above. Thanks, Jason Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: Lorenz B.<mailto:[email protected]> Sent: Thursday, June 6, 2019 15:32 To: [email protected]<mailto:[email protected]> Subject: Re: Jena Model is serializable in Java? I don't see why one would want to share Model instances via Spark. I mean, it's possible via wrapping it inside an object which is serializable or some other wrapper method: object ModelWrapper extends Serializable { lazy val model = ... } rdd.map(s => ModelWrapper.model. ... ) This makes the model being attached to some static code that can't be changed during runtime and that's what Spark needs. Ideally, you'd use some broadcast variable, but indeed those are just use to share smaller entities among the different Spark workers. For smaller models like a schema this would work and is supposed to be more efficient than having joins etc. (yes, there are also broadcast joins in Spark, still data would be distributed during processing) - but it depends ... I don't know your use-case nor why you need a Model, but what we did when using Jena on Spark was to use RDD (or Dataset) of Triple objects, i.e. RDD[Triple]. RDD is the fundamental shared datastructure of Spark and this is the only way to scale when using very large datasets. Parsing RDF triples from e.g. N-Triples directly into RDD[Triple] is pretty easy. For Dataset you have to define a custom encoder (Kryo encoder works though). But as already mentioned, your use-case or application would be needed to give further advice if necessary.Jason, I would argue that you should exchange a Set of triples, so you can take advantage of Spark's distributed nature. Your logic can materialize that list into a Graph or Model when needed to operate on it. Andy is right about being careful about the size - you may want to build a specialized set that throws if the set is too large, and you may want to experiment with it. Andy, Does Jena Riot (or contrib) provide a binary syntax for RDF that is optimal for fast parse? I'm recalling Michael Stonebraker's response to the BigTable paper - https://pdfs.semanticscholar.org/08d1/2e771d811bcd0d4bc81fa3993563efbaeadb.pdf, and also gSOAP and other binary XML formats. To this paper, the Google BigTable authors then responded that they don't use loose serializations such as provided by HDFS, but instead use structured data. This is hugely important to Jason's question because this is one of the benefits of using Spark instead of HDFS - Spark will handle distributing a huge dataset to multiple systems so that algorithm authors can operate on a vector (of Jena models?) far too large to fit in one machine. On Wed, Jun 5, 2019 at 4:40 PM Andy Seaborne <[email protected]> wrote:Hi Jason, Models aren't serializable, nor are Graphs (the more system oriented view of RDF) through Triples, Quads and Node are serializable. You can send a list of triples. Or use an RDF syntax and write-then-read the RDF. But are the models small? RDF graph aren't always small so moving them around may be expensive. Andy On 05/06/2019 17:59, Scarlet Remilia wrote:Hello everyone, I get a problem about Jena and Spark. I use Jena Model to handle some RDF models in my spark executor, but Iget a error:java.io.NotSerializableException:org.apache.jena.rdf.model.impl.ModelComSerialization stack: - object not serializable (class:org.apache.jena.rdf.model.impl.ModelCom)- field (class: org.nari.r2rml.entities.Template, name: model,type: interface org.apache.jena.rdf.model.Model)- object (class org.nari.r2rml.entities.Template,org.nari.r2rml.entities.Template@23dc70c1)- field (class: org.nari.r2rml.entities.PredicateObjectMap,name: objectTemplate, type: class org.nari.r2rml.entities.Template)- object (class org.nari.r2rml.entities.PredicateObjectMap,org.nari.r2rml.entities.PredicateObjectMap@2de96eba)- writeObject data (class: java.util.ArrayList) - object (class java.util.ArrayList,[org.nari.r2rml.entities.PredicateObjectMap@2de96eba])- field (class: org.nari.r2rml.entities.LogicalTableMapping,name: predicateObjectMaps, type: class java.util.ArrayList)- object (class org.nari.r2rml.entities.LogicalTableMapping,org.nari.r2rml.entities.LogicalTableMapping@8e00c02)- field (class: org.nari.r2rml.beans.Impl.EachPartitonFunction,name: logicalTableMapping, type: class org.nari.r2rml.entities.LogicalTableMapping)- object (class org.nari.r2rml.beans.Impl.EachPartitonFunction,org.nari.r2rml.beans.Impl.EachPartitonFunction@1e14b269)- field (class:org.apache.spark.sql.Dataset$$anonfun$foreachPartition$2, name: func$4, type: interface org.apache.spark.api.java.function.ForeachPartitionFunction)- object (classorg.apache.spark.sql.Dataset$$anonfun$foreachPartition$2, <function1>)atorg.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)atorg.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46)atorg.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)atorg.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:400)... 33 more All these classes implement serializable interface. So how could I serialize Jena model java object? Thanks very much! Jason Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> forWindows 10-- Lorenz Bühmann AKSW group, University of Leipzig Group: http://aksw.org - semantic web research center
