A related question to ask is whether the data can be split into separate TDB datasets (different folders, different files) yet still be able to combine the data in a reasonably efficient manner.
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Zhiyun Qian Sent: Friday, October 04, 2013 10:42 AM To: [email protected] Subject: Re: jena TDB scalability Thanks very much for the explanation, Andy. I am curious about the case where I divide my data into separate graphs/models. Let's say I have 10B triples into 10 graphs, each has 1B triples. If most of my query can be specified for each graph, is it practically the same (in terms of scalability and performance) between organizing the 10B in a single DB and separate DBs (where each DB has 1B)? The reason I may still need to have them in one DB is because I have some (small number of) queries that may need to go over the boundaries of graphs. Best, -Zhiyun On Fri, Oct 4, 2013 at 5:26 AM, Andy Seaborne <[email protected]> wrote: > On 03/10/13 15:01, Zhiyun Qian wrote: > >> Hi there, >> >> I'm looking for some clues on the scalability of jena TDB. It looks >> like our requirement would be at least 1B - 10B triples. From what I >> can find online (which seems to be dated back in 2008), the max >> number ever put into TDB is 1.7B [1]. I wonder if there's any more >> recent number on this. >> >> I'm also curious about whether the scalability is primarily measured >> on the union of all the graphs or individual graphs. In other words, >> whether a "Dataset" (regardless of how many graphs/models in it) can >> only scale up to a given number (let's say 1.7B) or an individual >> graph/model can scale to a given number. Since our data naturally can >> be divided into different graphs (with limited relationship across >> graphs), most queries can be performed on a single graph at a time >> (we need some hacks to query the relationship across graphs but I >> assume it is possible). >> >> My understanding is that if we simply query one graph out of the many >> in a dataset, it does not matter much how many triples there are in >> other graphs. Is this correct? >> >> [1]. >> http://www.w3.org/wiki/**LargeTripleStores<http://www.w3.org/wiki/Lar >> geTripleStores> >> >> Best, >> -Zhiyun >> >> > Theer isn't a hard cutoff point whereby it works at X but not at X+1. > There are no particular built-in assumptions like that (the nearest is > that nodes have unique hashes - but the node hash is 128 bits so you > can do some maths about that; things like undetected memory corruption are > more likely). > > 10B triples is beyond the practical limits. 1B will need a big > machine and not too complicated queries. > > As the database gets larger, the practical queries that can be > executed become more limited. Loading also becomes an issue. > > If you are just doing URI->some properties and a bit of filtering on > the retrieved values, then huge databases are possible. > > But as soon as general patterns, or group-aggregates or complicated > combinations of patterns, OPTIONALs and UNIONS and NOT EXISTS then it > will be impractically slow. ARQ/TDB uses an evaluation strategy [*] > that uses temporary RAM only at a few points, so it does not run out of > memory easily. > > Loading takes a long time - more hardware, specifically, more RAM, > makes a big difference. > > Andy > > [*] currently, in the released code. >
