Even if I am using try catch to catch RiotException but my code still gets 
terminated on this exception 😞


Regards,

Samita Bai

________________________________
From: Samita Bai / PhD CS Scholar @ City Campus
Sent: 16 April 2018 22:32:26
To: users@jena.apache.org
Subject: Re: TDB 2 Store Parameters


Yes I am using the same data but of Feb, 2018 as I started experimenting that 
time. For example for  the following piece of code I am getting the error as 
shown below.


public class ReadQuadInJena {

public static void main(String[] args) {
// TODO Auto-generated method stub
String FileName = "/home/samita/Dyldo_DS_4Feb2018/data.nq";
DatasetGraph dsg = RDFDataMgr.loadDatasetGraph(FileName);
//System.out.println(node);
Iterator<Quad> iterQuad = dsg.find();
while(iterQuad.hasNext()){
System.out.println(iterQuad.next());
}






Exception in thread "main" org.apache.jena.riot.RiotException: [line: 89841, 
col: 232] Illegal character in IRI (codepoint 0x7C, '|'): 
<http://fonts.googleapis.com/css?family=Nunito[|]...>
at 
org.apache.jena.riot.system.ErrorHandlerFactory$ErrorHandlerStd.fatal(ErrorHandlerFactory.java:147)
at org.apache.jena.riot.lang.LangEngine.raiseException(LangEngine.java:148)
at org.apache.jena.riot.lang.LangEngine.nextToken(LangEngine.java:105)
at org.apache.jena.riot.lang.LangNQuads.parseOne(LangNQuads.java:67)
at org.apache.jena.riot.lang.LangNQuads.runParser(LangNQuads.java:54)
at org.apache.jena.riot.lang.LangBase.parse(LangBase.java:41)
at 
org.apache.jena.riot.RDFParserRegistry$ReaderRIOTLang.read(RDFParserRegistry.java:195)
at org.apache.jena.riot.RDFParser.read(RDFParser.java:334)
at org.apache.jena.riot.RDFParser.parseURI(RDFParser.java:303)
at org.apache.jena.riot.RDFParser.parse(RDFParser.java:277)
at org.apache.jena.riot.RDFParserBuilder.parse(RDFParserBuilder.java:498)
at org.apache.jena.riot.RDFDataMgr.parseFromURI(RDFDataMgr.java:890)
at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:519)
at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:486)
at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:439)
at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:419)
at org.apache.jena.riot.RDFDataMgr.loadDatasetGraph(RDFDataMgr.java:392)
at ldbqPack.ReadQuadInJena.main(ReadQuadInJena.java:19)



________________________________
From: Andy Seaborne <a...@apache.org>
Sent: 16 April 2018 22:13:36
To: users@jena.apache.org
Subject: Re: TDB 2 Store Parameters

I downlaoded

http://swse.deri.org/dyldo/data/2016-03-27/data.nq.gz

(the latest I could find)

and used tdblaoder.

Is that the data you are using?

     Andy

On 16/04/18 17:32, ajs6f wrote:
> You should be able to check the validity of any of your files just by running 
> them through Jena's `riot` command.
>
> You can try loading them into a TDB1 or TDB2 db by using the `tdbloader` or 
> `tdb2.tdbloader` commands.
>
> ajs6f
>
>> On Apr 16, 2018, at 12:28 PM, Samita Bai / PhD CS Scholar @ City Campus 
>> <s...@iba.edu.pk> wrote:
>>
>> OK Andy I got your point. Can you please share the code that you used to 
>> read the Dynamic Linked Data Observatory dataset?
>>
>>
>>
>> Regards,
>>
>> Samita Bai
>>
>> ________________________________
>> From: Andy Seaborne <a...@apache.org>
>> Sent: 16 April 2018 15:34:07
>> To: users@jena.apache.org
>> Subject: Re: TDB 2 Store Parameters
>>
>> If you wish to prcoess the data as it is parsed, then see StreamRDF and
>> either
>>
>> NxParser, which is not part of Jena, is not a validating parser.
>>
>> If the data is not valid, then you will have problems at some point,
>> either loading, querying or outputting later.
>>
>> Adam has explained that TDB2 inxexes heavily so that querying is well
>> severed.
>>
>> We can't help with the parser errors without knowing what they are.
>>
>> Which files from Dynamic Linked Data Observatory are you processing?
>> Don't the later ones replace the earlier ones?
>>
>> I found that the last n-quads file was 42 million triples and all valid.
>>
>>      Andy
>>
>> On 16/04/18 11:05, ajs6f wrote:
>>> Is there are syntax errors in your RDF (and it sounds like that is why Jena 
>>> will not read it directly) you are doing yourself no service by taking 
>>> unusual pains to force TDB to ingest your data.
>>>
>>> Please show us the errors that Jena is throwing trying to read your data 
>>> and an appropriate sample of the data in question.
>>>
>>>
>>> ajs6f
>>>
>>>> On Apr 16, 2018, at 4:42 AM, Samita Bai / PhD CS Scholar @ City Campus 
>>>> <s...@iba.edu.pk> wrote:
>>>>
>>>> In addition to previous query. It is taking a lot of time to first parse 
>>>> the dataset using NXParser then checking for object, and creating quad 
>>>> again and storing in TDB. It could be very simple if we can take the quad 
>>>> check its object and insert it in TDB.
>>>>
>>>>
>>>> But Jena is not helping me with this 😞
>>>>
>>>>
>>>> So I have to create quads again and store it in TDB.
>>>>
>>>>
>>>> Any help is surely appreciated.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Samita Bai
>>>>
>>>> ________________________________
>>>> From: Samita Bai / PhD CS Scholar @ City Campus
>>>> Sent: 16 April 2018 13:33:51
>>>> To: users@jena.apache.org
>>>> Subject: Re: TDB 2 Store Parameters
>>>>
>>>>
>>>> Thank you Andy and Adam for the help. Actually, I am just indexing the 
>>>> quads where object is either literal or foreign URI (i.e. Object belonging 
>>>> to different dataset than subject), I am using NXParser (as Jena is giving 
>>>> various parsing errors) to parse the dataset and then I am storing in TDB2 
>>>> in the following manner.
>>>>
>>>>
>>>>
>>>> public  void SetQuadsList(String sub, String pred, String obj, String 
>>>> context) {
>>>> Node subjects = NodeFactory.createURI(sub);
>>>> Node objects = NodeFactory.createURI(obj);
>>>> Node contexts =NodeFactory.createURI(context);
>>>> //Node rdfSeeAlso = RDFS.seeAlso.asNode();
>>>>
>>>> Node predicates =NodeFactory.createURI(pred);
>>>>
>>>> //Quad quads = Quad.create(contexts, objects, rdfSeeAlso, subjects);
>>>>
>>>> Quad quads = Quad.create(contexts, subjects, predicates, objects);
>>>>
>>>> QuadList.add(quads);
>>>>
>>>> //System.out.println("Number of backlinks:" + QuadList.size());
>>>>
>>>> //System.out.println("quad written");
>>>>
>>>> //System.out.println("Quad"+quads.toString());
>>>>
>>>> }
>>>> public List<Quad> GetQuadsList(){
>>>> return QuadList;
>>>> }
>>>> public void QuadsToTDB(List<Quad> quadList) {
>>>> final String DATASET_DIR_NAME = "DyLDO1000K_Index";
>>>>         Dataset dataset = TDB2Factory.connectDataset ( DATASET_DIR_NAME );
>>>>
>>>>
>>>>         dataset.begin ( ReadWrite.WRITE );
>>>>         try {
>>>>         DatasetGraph dsg = dataset.asDatasetGraph();
>>>>             Iterator<Quad> quads = quadList.iterator();
>>>>             System.out.println("Size of Quad List: "+quadList.size());
>>>>             while ( quads.hasNext() ) {
>>>>             //System.out.println("here");
>>>>                 Quad quad = quads.next();
>>>>                 dsg.add(quad);
>>>>                 //System.out.println(quad.toString()+ "added");
>>>>                 //RDFDataMgr.writeQuads(System.out, quads);
>>>>               //  RDFDataMgr.write(System.out, dsg, Lang.NQUADS);
>>>>
>>>>             }
>>>>             System.out.println("dsg created of size "+dsg.size());
>>>>             //RDFDataMgr.write(System.out, dsg, Lang.NQUADS);
>>>>             System.out.println("written dsg using datamgr.");
>>>>
>>>>
>>>>             //System.out.println(dataset.isEmpty());
>>>>             //RDFDataMgr.write(System.out, dsg, Lang.NQUADS);
>>>>             dataset.commit();
>>>>
>>>>             System.out.println("committed dataset.");
>>>>
>>>>
>>>>         } catch ( Exception e ) {
>>>>             e.printStackTrace(System.err);
>>>>             //dataset.abort();
>>>>         } finally {
>>>>         //RDFDataMgr.write(System.out, dsg, Lang.NQUADS);
>>>>             dataset.end();
>>>>
>>>>         }
>>>>         System.out.println("end method.");
>>>> }}
>>>>
>>>>
>>>> I have indexed 40,000 files (as I have spilited the dataset into files 
>>>> according to context) and the index size has become 120 GB. I have a total 
>>>> of 1,35,600 files whose total size is 19.8 GB only.
>>>>
>>>>
>>>> Why the TDB is making such BIG index size. I am confused :( is there any 
>>>> problem in my code.
>>>>
>>>>
>>>> Please suggest me if there can be some improvements.
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Samita Bai
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: ajs6f <aj...@apache.org>
>>>> Sent: 15 April 2018 03:07:59
>>>> To: users@jena.apache.org
>>>> Subject: Re: TDB 2 Store Parameters
>>>>
>>>> 42 million quads is nothing like so many that either TDB version should 
>>>> have any problem doing normal indexing (assuming very little in the way of 
>>>> hardware-- I ingest datasets like that on my laptop all the time).
>>>>
>>>> Do you have some extraordinary hardware limitations?
>>>>
>>>> Adam
>>>>
>>>>> On Apr 14, 2018, at 11:42 AM, Andy Seaborne <a...@apache.org> wrote:
>>>>>
>>>>> Hi Samita,
>>>>>
>>>>> Firstly - as Adam points out - if theer are no indexes then access to the 
>>>>> data will be very slow.  For a GSPO index,  that means squeries must be 
>>>>> "GRAPH <uri> { ... }" and probably "GRAPH <uri> { <fixedSubject>.. }".
>>>>>
>>>>> GSPO means lookup by G then S within those G and the same for P then O.
>>>>>
>>>>> I looked at the data and it seems to be able 42 million quads.
>>>>>
>>>>> Using TDB1 (the loader is faster at this scale currently) is likely to be 
>>>>> a better choice.
>>>>>
>>>>> Looking at StoreParams in TDB2:
>>>>>
>>>>> The code below creates the database at TDB2Factory.connectDataset so any 
>>>>> StoreParams after that do not affect indexing.
>>>>>
>>>>> I tried to make it work in the release but the code ignores provided 
>>>>> StoreParams - sorry.  Even if it did work, it hits a test to make sure 
>>>>> there are basic indexing (Adam's point).
>>>>>
>>>>>    Andy
>>>>>
>>>>>
>>>>> On 13/04/18 13:42, Samita Bai  / PhD CS Scholar @ City Campus wrote:
>>>>>> I wrote the following code to build only one type of triple and quad 
>>>>>> index but it is still creating all indexes 😞
>>>>>> package ldbqPack;
>>>>>> import org.apache.jena.query.Dataset;
>>>>>> import org.apache.jena.tdb2.TDB2Factory;
>>>>>> import org.apache.jena.tdb2.setup.StoreParams;
>>>>>> import org.apache.jena.tdb2.sys.DatabaseConnection;
>>>>>> import org.apache.jena.dboe.base.block.FileMode;
>>>>>> import org.apache.jena.dboe.base.file.Location;
>>>>>> import org.apache.jena.tdb2.setup.StoreParamsFactory;
>>>>>> public class StrPrms {
>>>>>> static String[] tindexes= {"SPO"};
>>>>>> static String[] qindexes= {"GSPO"};
>>>>>> static String[] pindexes= {"GPU"};
>>>>>> static final StoreParams pApp = StoreParams.builder()
>>>>>>        .blockSize(12)              // Not dynamic
>>>>>>        .nodeMissCacheSize(12)      // Dynamic
>>>>>>        .build();
>>>>>>    static final StoreParams pLoc = StoreParams.builder()
>>>>>>        .blockSize(0)
>>>>>>        .nodeMissCacheSize(0).build();
>>>>>>    static final StoreParams pDft = StoreParams.builder()
>>>>>>     .fileMode(FileMode.mapped)
>>>>>>     .blockSize(8192)
>>>>>>     .blockReadCacheSize(5000)
>>>>>>     .blockWriteCacheSize(1000)
>>>>>>     .node2NodeIdCacheSize(200000)
>>>>>>     .nodeId2NodeCacheSize(750000)
>>>>>>     .nodeMissCacheSize(1000)
>>>>>>     .nodeTableBaseName("nodes")
>>>>>>     .primaryIndexTriples("SPO")
>>>>>>     .tripleIndexes(tindexes)
>>>>>>     .primaryIndexQuads("GSPO")
>>>>>>     .quadIndexes(qindexes)
>>>>>>     .prefixTableBaseName("prefixes")
>>>>>>     .primaryIndexPrefix("GPU")
>>>>>>     .prefixIndexes(pindexes)
>>>>>>     .build();
>>>>>> public static void main(String[] args) {
>>>>>> // TODO Auto-generated method stub
>>>>>> final String DATASET_DIR_NAME = "DyLDO100";
>>>>>>         Dataset dataset = TDB2Factory.connectDataset ( DATASET_DIR_NAME 
>>>>>> );
>>>>>>         Location location = Location.create(DATASET_DIR_NAME);
>>>>>>         StoreParams custom_params = 
>>>>>> StoreParamsFactory.decideStoreParams(location, true, pApp, pLoc,  pDft);
>>>>>>        DatabaseConnection.connectCreate(location, custom_params);
>>>>>>        StoreParams params = StoreParams.getSmallStoreParams();
>>>>>>         System.out.println(params);
>>>>>> }
>>>>>> }
>>>>>> Please help.
>>>>>> Regards,
>>>>>> Samita Bai
>>>>>> ________________________________
>>>>>> P : Please consider the environment before printing this e-mail
>>>>>> ________________________________
>>>>>> CONFIDENTIALITY / DISCLAIMER NOTICE: This e-mail and any attachments may 
>>>>>> contain confidential and privileged information. If you are not the 
>>>>>> intended recipient, please notify the sender immediately by return 
>>>>>> e-mail, delete this e-mail and destroy any copies. Any dissemination or 
>>>>>> use of this information by a person other than the intended recipient is 
>>>>>> unauthorized and may be illegal.
>>>>>> ________________________________
>>>>
>>>>
>>>> P : Please consider the environment before printing this e-mail
>>>>
>>>> ________________________________
>>>>
>>>> CONFIDENTIALITY / DISCLAIMER NOTICE: This e-mail and any attachments may 
>>>> contain confidential and privileged information. If you are not the 
>>>> intended recipient, please notify the sender immediately by return e-mail, 
>>>> delete this e-mail and destroy any copies. Any dissemination or use of 
>>>> this information by a person other than the intended recipient is 
>>>> unauthorized and may be illegal.
>>>>
>>>> ________________________________
>>>
>>
>> P : Please consider the environment before printing this e-mail
>>
>> ________________________________
>>
>> CONFIDENTIALITY / DISCLAIMER NOTICE: This e-mail and any attachments may 
>> contain confidential and privileged information. If you are not the intended 
>> recipient, please notify the sender immediately by return e-mail, delete 
>> this e-mail and destroy any copies. Any dissemination or use of this 
>> information by a person other than the intended recipient is unauthorized 
>> and may be illegal.
>>
>> ________________________________
>

P : Please consider the environment before printing this e-mail

________________________________

CONFIDENTIALITY / DISCLAIMER NOTICE: This e-mail and any attachments may 
contain confidential and privileged information. If you are not the intended 
recipient, please notify the sender immediately by return e-mail, delete this 
e-mail and destroy any copies. Any dissemination or use of this information by 
a person other than the intended recipient is unauthorized and may be illegal.

________________________________

Reply via email to