42 million quads is nothing like so many that either TDB version should have any problem doing normal indexing (assuming very little in the way of hardware-- I ingest datasets like that on my laptop all the time).
Do you have some extraordinary hardware limitations? Adam > On Apr 14, 2018, at 11:42 AM, Andy Seaborne <[email protected]> wrote: > > Hi Samita, > > Firstly - as Adam points out - if theer are no indexes then access to the > data will be very slow. For a GSPO index, that means squeries must be > "GRAPH <uri> { ... }" and probably "GRAPH <uri> { <fixedSubject>.. }". > > GSPO means lookup by G then S within those G and the same for P then O. > > I looked at the data and it seems to be able 42 million quads. > > Using TDB1 (the loader is faster at this scale currently) is likely to be a > better choice. > > Looking at StoreParams in TDB2: > > The code below creates the database at TDB2Factory.connectDataset so any > StoreParams after that do not affect indexing. > > I tried to make it work in the release but the code ignores provided > StoreParams - sorry. Even if it did work, it hits a test to make sure there > are basic indexing (Adam's point). > > Andy > > > On 13/04/18 13:42, Samita Bai / PhD CS Scholar @ City Campus wrote: >> I wrote the following code to build only one type of triple and quad index >> but it is still creating all indexes 😞 >> package ldbqPack; >> import org.apache.jena.query.Dataset; >> import org.apache.jena.tdb2.TDB2Factory; >> import org.apache.jena.tdb2.setup.StoreParams; >> import org.apache.jena.tdb2.sys.DatabaseConnection; >> import org.apache.jena.dboe.base.block.FileMode; >> import org.apache.jena.dboe.base.file.Location; >> import org.apache.jena.tdb2.setup.StoreParamsFactory; >> public class StrPrms { >> static String[] tindexes= {"SPO"}; >> static String[] qindexes= {"GSPO"}; >> static String[] pindexes= {"GPU"}; >> static final StoreParams pApp = StoreParams.builder() >> .blockSize(12) // Not dynamic >> .nodeMissCacheSize(12) // Dynamic >> .build(); >> static final StoreParams pLoc = StoreParams.builder() >> .blockSize(0) >> .nodeMissCacheSize(0).build(); >> static final StoreParams pDft = StoreParams.builder() >> .fileMode(FileMode.mapped) >> .blockSize(8192) >> .blockReadCacheSize(5000) >> .blockWriteCacheSize(1000) >> .node2NodeIdCacheSize(200000) >> .nodeId2NodeCacheSize(750000) >> .nodeMissCacheSize(1000) >> .nodeTableBaseName("nodes") >> .primaryIndexTriples("SPO") >> .tripleIndexes(tindexes) >> .primaryIndexQuads("GSPO") >> .quadIndexes(qindexes) >> .prefixTableBaseName("prefixes") >> .primaryIndexPrefix("GPU") >> .prefixIndexes(pindexes) >> .build(); >> public static void main(String[] args) { >> // TODO Auto-generated method stub >> final String DATASET_DIR_NAME = "DyLDO100"; >> Dataset dataset = TDB2Factory.connectDataset ( DATASET_DIR_NAME ); >> Location location = Location.create(DATASET_DIR_NAME); >> StoreParams custom_params = >> StoreParamsFactory.decideStoreParams(location, true, pApp, pLoc, pDft); >> DatabaseConnection.connectCreate(location, custom_params); >> StoreParams params = StoreParams.getSmallStoreParams(); >> System.out.println(params); >> } >> } >> Please help. >> Regards, >> Samita Bai >> ________________________________ >> P : Please consider the environment before printing this e-mail >> ________________________________ >> CONFIDENTIALITY / DISCLAIMER NOTICE: This e-mail and any attachments may >> contain confidential and privileged information. If you are not the intended >> recipient, please notify the sender immediately by return e-mail, delete >> this e-mail and destroy any copies. Any dissemination or use of this >> information by a person other than the intended recipient is unauthorized >> and may be illegal. >> ________________________________
