Re: tdbloader2 issues

[Michael Brunnbauer]

Hello Andy,

[tdbloader2 disk access pattern]
>Lots of unique nodes can slow things down because of all the node writing.

And there is no way to convert this algorithm to sequential access?

[tdbloader2 parser]
> >>But also no " { } | ^ ` if I read that right? tdbloader2 accepts those in 
> >>IRIs.
> 
> Could you provide a set of data with one feature per NTriple line,marking in
> a comment what you expect, and I'll check each one and add them to the test
> suite.

See attachment. I would consider all triples in it illegal according to the
n triples spec. 

If I allow these characters that RFC 1738 calls "unsafe", why then not allow
CR, LF and TAB? And why then allow \\ but not \", which seems to be sanctioned 
by older versions of the spec:

 http://www.w3.org/2001/sw/RDFCore/ntriples/#character

I found 752 triples with \" IRIs in the Wikidata dump and 94 triples with \n
IRIs, e.G.:

<http://www.wikidata.org/entity/P1348v> 
<http://www.algaebase.org/search/species/detail/?species_id=26717\n> .
<http://www.wikidata.org/entity/Q181274S0B6CB54F-C792-4A12-B20E-A165B91BB46D> 
<http://www.wikidata.org/entity/P18v> 
<http://commons.wikimedia.org/wiki/File:George_\"Corpsegrinder\"_Fisher_of_Cannibal_Corpse.jpg>
 .

This trial and error cleaning of data dumps with self made scripts and days 
between each try is very straining and probably a big deterrent for newcomers.
I had it with DBpedia and now I have it with Wikidata all over again (with
new syntax problems).

Regards,

Michael Brunnbauer

-- 
++  Michael Brunnbauer
++  netEstate GmbH
++  Geisenhausener Straße 11a
++  81379 München
++  Tel +49 89 32 19 77 80
++  Fax +49 89 32 19 77 89 
++  E-Mail bru...@netestate.de
++  http://www.netestate.de/
++
++  Sitz: München, HRB Nr.142452 (Handelsregister B München)
++  USt-IdNr. DE221033342
++  Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++  Prokurist: Dipl. Kfm. (Univ.) Markus Hendel

<http://test/> <http://test/> <http://test/";> .
<http://test/> <http://test/> <http://test/{> .
<http://test/> <http://test/> <http://test/}> .
<http://test/> <http://test/> <http://test/|> .
<http://test/> <http://test/> <http://test/^> .
<http://test/> <http://test/> <http://test/`> .

pgpYKhMUw6INA.pgp
Description: PGP signature