Thanks very much for the explanation, Andy. I am curious about the case where I divide my data into separate graphs/models. Let's say I have 10B triples into 10 graphs, each has 1B triples. If most of my query can be specified for each graph, is it practically the same (in terms of scalability and performance) between organizing the 10B in a single DB and separate DBs (where each DB has 1B)? The reason I may still need to have them in one DB is because I have some (small number of) queries that may need to go over the boundaries of graphs.
Best, -Zhiyun On Fri, Oct 4, 2013 at 5:26 AM, Andy Seaborne <[email protected]> wrote: > On 03/10/13 15:01, Zhiyun Qian wrote: > >> Hi there, >> >> I'm looking for some clues on the scalability of jena TDB. It looks like >> our requirement would be at least 1B - 10B triples. From what I can find >> online (which seems to be dated back in 2008), the max number ever put >> into >> TDB is 1.7B [1]. I wonder if there's any more recent number on this. >> >> I'm also curious about whether the scalability is primarily measured on >> the >> union of all the graphs or individual graphs. In other words, whether a >> "Dataset" (regardless of how many graphs/models in it) can only scale up >> to >> a given number (let's say 1.7B) or an individual graph/model can scale to >> a >> given number. Since our data naturally can be divided into different >> graphs >> (with limited relationship across graphs), most queries can be performed >> on >> a single graph at a time (we need some hacks to query the relationship >> across graphs but I assume it is possible). >> >> My understanding is that if we simply query one graph out of the many in a >> dataset, it does not matter much how many triples there are in other >> graphs. Is this correct? >> >> [1]. >> http://www.w3.org/wiki/**LargeTripleStores<http://www.w3.org/wiki/LargeTripleStores> >> >> Best, >> -Zhiyun >> >> > Theer isn't a hard cutoff point whereby it works at X but not at X+1. > There are no particular built-in assumptions like that (the nearest is that > nodes have unique hashes - but the node hash is 128 bits so you can do some > maths about that; things like undetected memory corruption are more likely). > > 10B triples is beyond the practical limits. 1B will need a big machine > and not too complicated queries. > > As the database gets larger, the practical queries that can be executed > become more limited. Loading also becomes an issue. > > If you are just doing URI->some properties and a bit of filtering on the > retrieved values, then huge databases are possible. > > But as soon as general patterns, or group-aggregates or complicated > combinations of patterns, OPTIONALs and UNIONS and NOT EXISTS then it will > be impractically slow. ARQ/TDB uses an evaluation strategy [*] that uses > temporary RAM only at a few points, so it does not run out of memory easily. > > Loading takes a long time - more hardware, specifically, more RAM, makes a > big difference. > > Andy > > [*] currently, in the released code. >
