of course, my comparison was naive - and I should have known better,
working with databases back in the 1980s. still it surprises when a 200
mb files becomes 13 gb - given that diskspace and memory is inexpensive
compared to human time waiting for responses, the design choices are
amply justified. i will buy more memory (;-)
andrew
On 11/26/2017 11:20 AM, ajs6f wrote:
You have to start with the understanding that the indexes in a database are not
the same thing nor for the same purpose as a simple file of triples or quads.
TDB1 and 2 store the same triple several times in different orders (SPO, OPS,
etc.) in order to be able to answer arbitrary queries with good performance,
which is a common technique.
There is no reason to expect a database that is capable of answering arbitrary
queries with good performance to be as small as a file, which is not.
ajs6f
On Nov 26, 2017, at 11:17 AM, Andrew U. Frank <fr...@geoinfo.tuwien.ac.at>
wrote:
thank you for the explanations:
to laura: i guess HDT would reduce the size of my files considerably. where
could i find information how to use fuseki with HDT? i might be worth trying
and see how response time changes.
to andy: am i correct to understand that a triple (uri p literal) is translated
in two triples (uri p uriX) and a second one (uriX s literal) for some
properties p and s? is there any reuse of existing literals? that would give
for each literal triple approx. 60 bytes?
i still do not undestand how a triple needs about 300 bytes of storage? (or how
an nt.gzip file of 219 M igives a TDB database of 13 GB)
size of the database is of concern to me and I think it influences performance
through the use of IO time.
thank you all very much for the clarifications!
andrew
On 11/26/2017 07:30 AM, Andy Seaborne wrote:
Every RDFTerm gets a NodeId in TDB. A triple is 3 NodeIds.
There is a big cache, NodeId->RDFTerm.
In TDB1 and TDB2, a NodeId is stored as 8 bytes. TDB2 design is an int and long
(96 bits) - the current implementation is using 64 bits.
It is very common as a design to dictionary (intern) terms because joins can be
done by comparing a integers, not testing whether two strings are the same,
which is much more expensive.
In addition TDBx inlines numbers integers and date/times and some others.
https://jena.apache.org/documentation/tdb/architecture.html
TDBx could, but doesn't, store compressed data on disk. There are pros and cons
of this.
Andy
On 26/11/17 08:30, Laura Morales wrote:
Perhaps a bit tangential but this is somehow related to how HDT stores its data (I've run
some tests with Fuseki + HDT store instead of TDB). Basically, they assign each subject,
predicate, and object an integer value. It keeps an index to map integers with the
corresponding string (of the original value), and then they store every triple using
integers instead of strings (something like "1 2 9. 8 2 1 ." and so forth. The
drawback I think is that they have to translate indices/strings back and forth at each
query, nonetheless the response time is still impressive (milliseconds), and it
compresses the original file *a lot*. By a lot I mean that for Wikidata (not the full
file though, but one with about 2.3 billion triples) the HDT is more or less 40GB, and
gz-compressed about 10GB. The problem is that their rdf2hdt tool is so inefficient that
it does everything in RAM, so to convert something like wikidata you'd need at least a
machine with 512GB of ram (or swap if you have a fast enough swap :D). Also the tool
looks like it can't handle files with more than 2^32 triples, although HDT (the format)
does handle them. So as long as you can handle the conversion, if you want to save space
you could benefit from using a HDT store rather than using TDB.
Sent: Sunday, November 26, 2017 at 5:30 AM
From: "Andrew U. Frank" <fr...@geoinfo.tuwien.ac.at>
To: users@jena.apache.org
Subject: Re: Estimating TDB2 size
i have specific questiosn in relation to what ajs6f said:
i have a TDB store with 1/3 triples with very small literals (3-5 char),
where the same sequence is often repeated. would i get smaller store and
better performance if these were URI of the character sequence (stored
once for each repeated case)? any guess how much I could improve?
does the size of the URI play a role in the amount of storage used. i
observe that i have for 33 M triples a TDB size (files) of 13 GB, which
means about 300 byte per triple. the literals are all short (very seldom
more than 10 char, mostly 5 - words from english text). is is a named
graph, if this makes a difference.
thank you!
andrew
--
em.o.Univ.Prof. Dr. sc.techn. Dr. h.c. Andrew U. Frank
+43 1 58801 12710 direct
Geoinformation, TU Wien +43 1 58801 12700 office
Gusshausstr. 27-29 +43 1 55801 12799 fax
1040 Wien Austria +43 676 419 25 72 mobil
--
em.o.Univ.Prof. Dr. sc.techn. Dr. h.c. Andrew U. Frank
+43 1 58801 12710 direct
Geoinformation, TU Wien +43 1 58801 12700 office
Gusshausstr. 27-29 +43 1 55801 12799 fax
1040 Wien Austria +43 676 419 25 72 mobil
