Perhaps a bit tangential but this is somehow related to how HDT stores its data (I've run some tests with Fuseki + HDT store instead of TDB). Basically, they assign each subject, predicate, and object an integer value. It keeps an index to map integers with the corresponding string (of the original value), and then they store every triple using integers instead of strings (something like "1 2 9. 8 2 1 ." and so forth. The drawback I think is that they have to translate indices/strings back and forth at each query, nonetheless the response time is still impressive (milliseconds), and it compresses the original file *a lot*. By a lot I mean that for Wikidata (not the full file though, but one with about 2.3 billion triples) the HDT is more or less 40GB, and gz-compressed about 10GB. The problem is that their rdf2hdt tool is so inefficient that it does everything in RAM, so to convert something like wikidata you'd need at least a machine with 512GB of ram (or swap if you have a fast enough swap :D). Also the tool looks like it can't handle files with more than 2^32 triples, although HDT (the format) does handle them. So as long as you can handle the conversion, if you want to save space you could benefit from using a HDT store rather than using TDB.
Sent: Sunday, November 26, 2017 at 5:30 AM From: "Andrew U. Frank" <fr...@geoinfo.tuwien.ac.at> To: users@jena.apache.org Subject: Re: Estimating TDB2 size i have specific questiosn in relation to what ajs6f said: i have a TDB store with 1/3 triples with very small literals (3-5 char), where the same sequence is often repeated. would i get smaller store and better performance if these were URI of the character sequence (stored once for each repeated case)? any guess how much I could improve? does the size of the URI play a role in the amount of storage used. i observe that i have for 33 M triples a TDB size (files) of 13 GB, which means about 300 byte per triple. the literals are all short (very seldom more than 10 char, mostly 5 - words from english text). is is a named graph, if this makes a difference. thank you! andrew
