Hi, just wanted to confirm that version 3.9.0 solves the spatial indexing issue:
$ java -cp /home/process/fuseki-3.9.0/fuseki-server.jar jena.spatialindexer --desc=/home/process/fuseki-3.9.0/run/configuration/mm.ttl ... INFO 539607479 (490106 per second) properties indexed Spatial queries work like a charm now. Many thanks for the quick fix. Kind regards Markus On 2018/09/19 14:25:41, Marco Neumann <[email protected]> wrote: > indeed Andy, I got confused with the naming here. same in the code> > actually. it might be a good idea to come up with a new name for TDB2 to> > reduce scope for confusion.> > > > On Wed, Sep 19, 2018 at 1:47 PM Andy Seaborne <[email protected]> wrote:> > > >> > >> > > On 19/09/18 11:24, Marco Neumann wrote:> > > > On Wed, Sep 19, 2018 at 11:04 AM Andy Seaborne <[email protected]> wrote:> > > >> > > >>> > > >>> > > >> On 17/09/18 15:26, ajs6f wrote:> > > >>> Okay, I think we've got it. Nice debugging, Markus!> > > >>>> > > >>> If you look at the line of the spatial indexer code that is throwing > > >>> an> > > >> exception:> > > >>>> > > >>>> > > >>> > > https://github.com/apache/jena/blob/master/jena-spatial/src/main/java/jena/spatialindexer.java#L110> > > > > >>>> > > >>> indeed, it does not start a transaction against the dataset before> > > >> starting to read quads to index them. Now, in TDB1 this isn't> > > inherently a> > > >> problem-- it's possible to work against a dataset without transactions.> > > >> (Although it's generally not a good idea at all.) But IIRC, TDB2> > > requires> > > >> transactions. If that's true (and Andy can verify that for us) then> > > this is> > > >> a bug and we can fix it in a straightforward way.> > > >>>> > > >>> Andy, can you confirm that diagnosis?> > > >>> > > >> Yes - TDB2 only works transactionally.> > > >>> > > >>> > > > Andy--> > > >> > > > I am just looking at the file system, I noticed that there is no binary> > > > difference between tdb and tdb2 indexes. would it be fair to describe> > > tdb2> > > > as a tdb+txn with different access methods?> > >> > > Are you looking at tdblaod vs tdbloader2 (unfortunately named> > >> > > The TBD2 loader is tdb2.tdbloader.> > >> > > The disk format for TDB2 is different - it is pushed down one level plus> > > the node table and the indexes are different byte structures. The> > > indexes are nearly the same but not quite (there is no link field in the> > > leaf blocks; tere is an extra control file ".bpt" per index).> > >> > > The journal in TDB1 can grow big, the journal in TDB2 is small - a few> > > hundrad bytes maximum.> > >> > > And (by accident!) you can have both in the same directory. This is not> > > a good idea.> > >> > > Andy> > >> > >> > > TDB2:> > > DB2:> > > Data-0001/ tdb.lock> > >> > > DB2/Data-0001:> > > GOSP.bpt GSPO.bpt OSP.bpt POSG.dat SPO.dat> > > GOSP.dat GSPO.dat OSP.dat POSG.idn SPOG.bpt> > > GOSP.idn GSPO.idn OSPG.bpt POS.idn SPOG.dat> > > GPOS.bpt journal.jrnl OSPG.dat prefixes.bpt SPOG.idn> > > GPOS.dat nodes.bpt OSPG.idn prefixes.dat SPO.idn> > > GPOS.idn nodes.dat OSP.idn prefixes-data.bdf tdb.lock> > > GPU.bpt nodes-data.bdf POS.bpt prefixes-data.obj> > > GPU.dat nodes-data.obj POS.dat prefixes.idn> > > GPU.idn nodes.idn POSG.bpt SPO.bpt> > >> > > TDB1:> > > DB1:> > > GOSP.dat journal.jrnl OSPG.idn prefix2id.dat SPOG.dat> > > GOSP.idn node2id.dat OSP.idn prefix2id.idn SPOG.idn> > > GPOS.dat node2id.idn POS.dat prefixes.dat SPO.idn> > > GPOS.idn nodes.dat POSG.dat prefixIdx.dat stats.opt> > > GSPO.dat OSP.dat POSG.idn prefixIdx.idn> > > GSPO.idn OSPG.dat POS.idn SPO.dat> > >> > >> > > >> > > >> > > >>> > > >>> If so, I'll open a ticket and write a PR to use a Txn method to fix> > > this.> > > >>> > > >> which ajs6f has merged into the development system.> > > >>> > > >>>> > > >>> ajs6f> > > >>>> > > >>>> On Sep 17, 2018, at 10:14 AM, Markus Neumann <> > > [email protected]>> > > >> wrote:> > > >>>>> > > >>>> It seems I made a mess with absolute and relative paths all together.> > > >> So I created a minimal setup for testing:> > > >>>> (I include the file-contents at the bottom)> > > >>>>> > > >>>> $ export JENA_HOME=/srv/linked_data_store/apache-jena-3.8.0> > > >>>> $ export PATH=$PATH:$JENA_HOME/bin> > > >>>> $ tdb2.tdbloader> > > >> --desc=/srv/linked_data_store/fuseki-server/run/configuration/mm.ttl> > > >> /srv/linked_data_store/rdf_python_connector/turtle_files/mm-loc-mini.ttl> > > >> > > >> /srv/linked_data_store/rdf_python_connector/turtle_files/mm-rdf.ttl> > > >>>> 13:50:54 INFO loader :: Loader = LoaderPhased> > > >>>> 13:50:54 INFO loader :: Data:> > > >>> > > /srv/linked_data_store/rdf_python_connector/turtle_files/mm-loc-mini.ttl:> > > >> 13 mm-loc-mini.ttl 0.06s (Avg: 232)> > > >>>> 13:50:54 INFO loader :: Data:> > > >> /srv/linked_data_store/rdf_python_connector/turtle_files/mm-rdf.ttl: 50> > > >> mm-rdf.ttl 0.01s (Avg: 6,250)> > > >>>> 13:50:55 INFO loader :: Finish - index SPO> > > >>>> 13:50:55 INFO loader :: Start replay index SPO> > > >>>> 13:50:55 INFO loader :: Index set: SPO => SPO->POS,> > > >> SPO->OSP> > > >>>> 13:50:55 INFO loader :: Index set: SPO => SPO->POS,> > > >> SPO->OSP [63 items, 0.0 seconds]> > > >>>> 13:50:55 INFO loader :: Finish - index POS> > > >>>> 13:50:55 INFO loader :: Finish - index OSP> > > >>>> $ java -cp> > > >>> > > /srv/linked_data_store/fuseki-server/fuseki-server.jar:/srv/linked_data_store/apache-jena-3.8.0/lib/*> > > > > >> jena.spatialindexer> > > >> --desc=/srv/linked_data_store/fuseki-server/run/configuration/mm.ttl> > > >>>> SLF4J: Class path contains multiple SLF4J bindings.> > > >>>> SLF4J: Found binding in> > > >>> > > [jar:file:/srv/linked_data_store/fuseki-server/fuseki-server.jar!/org/slf4j/impl/StaticLoggerBinder.class]> > > > > >>>> SLF4J: Found binding in> > > >>> > > [jar:file:/srv/linked_data_store/apache-jena-3.8.0/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]> > > > > >>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an> > > >> explanation.> > > >>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]> > > >>>> org.apache.jena.dboe.transaction.txn.TransactionException: Not in a> > > >> transaction> > > >>>> at> > > >>> > > org.apache.jena.dboe.transaction.txn.TransactionalComponentLifecycle.checkTxn(TransactionalComponentLifecycle.java:417)> > > > > >>>> at> > > >>> > > org.apache.jena.dboe.trans.bplustree.BPlusTree.getRootRead(BPlusTree.java:159)> > > > > >>>> at> > > >>> > > org.apache.jena.dboe.trans.bplustree.BPlusTree.iterator(BPlusTree.java:348)> > > > > >>>> at> > > >>> > > org.apache.jena.tdb2.store.tupletable.TupleIndexRecord.all(TupleIndexRecord.java:251)> > > > > >>>> at> > > >>> > > org.apache.jena.tdb2.store.tupletable.TupleTable.find(TupleTable.java:148)> > > >>>> at> > > >>> > > org.apache.jena.tdb2.store.nodetupletable.NodeTupleTableConcrete.find(NodeTupleTableConcrete.java:161)> > > > > >>>> at> > > >>> > > org.apache.jena.tdb2.store.nodetupletable.NodeTupleTableConcrete.find(NodeTupleTableConcrete.java:150)> > > > > >>>> at> > > >>> > > org.apache.jena.tdb2.store.nodetupletable.NodeTupleTableConcrete.findAsNodeIds(NodeTupleTableConcrete.java:141)> > > > > >>>> at> > > org.apache.jena.tdb2.store.TripleTable.find(TripleTable.java:64)> > > >>>> at> > > >>> > > org.apache.jena.tdb2.store.DatasetGraphTDB.findInDftGraph(DatasetGraphTDB.java:104)> > > > > >>>> at> > > >>> > > org.apache.jena.sparql.core.DatasetGraphBaseFind.findAny(DatasetGraphBaseFind.java:72)> > > > > >>>> at> > > >>> > > org.apache.jena.sparql.core.DatasetGraphBaseFind.find(DatasetGraphBaseFind.java:50)> > > > > >>>> at> > > >>> > > org.apache.jena.sparql.core.DatasetGraphWrapper.find(DatasetGraphWrapper.java:152)> > > > > >>>> at> > > >>> > > org.apache.jena.sparql.core.DatasetGraphWrapper.find(DatasetGraphWrapper.java:152)> > > > > >>>> at jena.spatialindexer.exec(spatialindexer.java:110)> > > >>>> at jena.cmd.CmdMain.mainMethod(CmdMain.java:93)> > > >>>> at jena.cmd.CmdMain.mainRun(CmdMain.java:58)> > > >>>> at jena.cmd.CmdMain.mainRun(CmdMain.java:45)> > > >>>> at jena.spatialindexer.main(spatialindexer.java:53)> > > >>>>> > > >>>> I can get rid of the SLF4J warnings:> > > >>>> $ java -cp /srv/linked_data_store/fuseki-server/fuseki-server.jar> > > >> jena.spatialindexer> > > >> --desc=/srv/linked_data_store/fuseki-server/run/configuration/mm.ttl> > > >>>> org.apache.jena.dboe.transaction.txn.TransactionException: Not in a> > > >> transaction> > > >>>> at> > > >>> > > org.apache.jena.dboe.transaction.txn.TransactionalComponentLifecycle.checkTxn(TransactionalComponentLifecycle.java:417)> > > > > >>>> at> > > >>> > > org.apache.jena.dboe.trans.bplustree.BPlusTree.getRootRead(BPlusTree.java:159)> > > > > >>>> at> > > >>> > > org.apache.jena.dboe.trans.bplustree.BPlusTree.iterator(BPlusTree.java:348)> > > > > >>>> at> > > >>> > > org.apache.jena.tdb2.store.tupletable.TupleIndexRecord.all(TupleIndexRecord.java:251)> > > > > >>>> at> > > >>> > > org.apache.jena.tdb2.store.tupletable.TupleTable.find(TupleTable.java:148)> > > >>>> at> > > >>> > > org.apache.jena.tdb2.store.nodetupletable.NodeTupleTableConcrete.find(NodeTupleTableConcrete.java:161)> > > > > >>>> at> > > >>> > > org.apache.jena.tdb2.store.nodetupletable.NodeTupleTableConcrete.find(NodeTupleTableConcrete.java:150)> > > > > >>>> at> > > >>> > > org.apache.jena.tdb2.store.nodetupletable.NodeTupleTableConcrete.findAsNodeIds(NodeTupleTableConcrete.java:141)> > > > > >>>> at> > > org.apache.jena.tdb2.store.TripleTable.find(TripleTable.java:64)> > > >>>> at> > > >>> > > org.apache.jena.tdb2.store.DatasetGraphTDB.findInDftGraph(DatasetGraphTDB.java:104)> > > > > >>>> at> > > >>> > > org.apache.jena.sparql.core.DatasetGraphBaseFind.findAny(DatasetGraphBaseFind.java:72)> > > > > >>>> at> > > >>> > > org.apache.jena.sparql.core.DatasetGraphBaseFind.find(DatasetGraphBaseFind.java:50)> > > > > >>>> at> > > >>> > > org.apache.jena.sparql.core.DatasetGraphWrapper.find(DatasetGraphWrapper.java:152)> > > > > >>>> at> > > >>> > > org.apache.jena.sparql.core.DatasetGraphWrapper.find(DatasetGraphWrapper.java:152)> > > > > >>>> at jena.spatialindexer.exec(spatialindexer.java:110)> > > >>>> at jena.cmd.CmdMain.mainMethod(CmdMain.java:93)> > > >>>> at jena.cmd.CmdMain.mainRun(CmdMain.java:58)> > > >>>> at jena.cmd.CmdMain.mainRun(CmdMain.java:45)> > > >>>> at jena.spatialindexer.main(spatialindexer.java:53)> > > >>>>> > > >>>> There is no other java instance running at the time when I run those> > > >> commands.> > > >>>> All paths are fully qualified and double checked. The destination> > > >> directory and the Lucene directory were cleared just before.> > > >>>>> > > >>>> The server does start and serve the data without errors or warnings > > >>>> in> > > >> the log, but spatial queries do not work.> > > >>>>> > > >>>> Does anyone see a reason for the TransactionException?> > > >>>> Thanks and best regards> > > >>>> Markus> > > >>>>> > > >>>> Attachments:> > > >>>> /srv/linked_data_store/rdf_python_connector/turtle_files/mm-rdf.ttl:> > > >>>> @prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> .> > > >>>> @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .> > > >>>> @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .> > > >>>> @prefix owl: <http://www.w3.org/2002/07/owl#> .> > > >>>> @prefix dc: <http://purl.org/dc/elements/1.1/> .> > > >>>> @prefix dcterms: <http://purl.org/dc/terms/> .> > > >>>> @prefix xsd: <http://www.w3.org/2001/XMLSchema#>> > > >>>> @prefix mm: <http://rdf.meteomatics.com/mm/> .> > > >>>>> > > >>>> <http://rdf.meteomatics.com/mm/>> > > >>>> dcterms:publisher <http://api.meteomatics.com/Imprint.html> ;> > > >>>> dcterms:title "Meteomatics RDF Vocabulary"@en .> > > >>>>> > > >>>> mm:location a rdf:Property ;> > > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;> > > >>>> rdfs:label "location" ;> > > >>>> dcterms:description "Location of a measurement"@en .> > > >>>>> > > >>>> mm:elevation a rdf:Property ;> > > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;> > > >>>> rdfs:label "elevation" ;> > > >>>> dcterms:description "Elevation of a location"@en .> > > >>>>> > > >>>> mm:validdate a rdf:Property ;> > > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;> > > >>>> rdfs:label "validdate" ;> > > >>>> dcterms:description "Temporal information describing when a> > > >> Measurement is/was valid"@en .> > > >>>>> > > >>>> mm:value a rdf:Property ;> > > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;> > > >>>> rdfs:label "value" ;> > > >>>> dcterms:description "The (observed or predicted) value of a> > > >> Measurement"@en .> > > >>>>> > > >>>> mm:parameter a rdf:Property ;> > > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;> > > >>>> rdfs:label "parameter" ;> > > >>>> dcterms:description "what was measured"@en .> > > >>>>> > > >>>> mm:station_name a rdf:Property ;> > > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;> > > >>>> rdfs:label "station_name" ;> > > >>>> dcterms:description "Name of a station"@en .> > > >>>>> > > >>>> mm:abbreviation a rdf:Property ;> > > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;> > > >>>> rdfs:label "abbreviation" ;> > > >>>> dcterms:description "Abbreviation of a station name"@en .> > > >>>>> > > >>>> mm:didok_id a rdf:Property ;> > > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;> > > >>>> rdfs:label "didok_id" ;> > > >>>> dcterms:description "didok id of a station"@en .> > > >>>>> > > >>>> mm:unit_symbol a rdf:Property ;> > > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;> > > >>>> rdfs:label "unit symbol" ;> > > >>>> dcterms:description "unit symbol of a parameter"@en .> > > >>>>> > > >>>> mm:Measurement a rdf:Class ;> > > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;> > > >>>> rdfs:label "Measurement" ;> > > >>>> dcterms:description "Measurement of a specific weather parameter> > > at> > > >> specific location at a specific date (both future or past)" .> > > >>>>> > > >>>> mm:Location a rdf:Class ;> > > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;> > > >>>> rdfs:label "Location" ;> > > >>>> dcterms:description "A Point on the globe where we observe> > > >> weather"@en .> > > >>>>> > > >>>> mm:Parameter a rdf:Class ;> > > >>>> rdfs:isDefinedBy <http://rdf.meteomatics.com/mm/> ;> > > >>>> rdfs:label "Parameter" ;> > > >>>> dcterms:description "A parameter describing an observable aspect> > > of> > > >> the weather"@en .> > > >>>>> > > >>>>> > > /srv/linked_data_store/rdf_python_connector/turtle_files/mm-loc-mini.ttl> > > >>>> @prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> .> > > >>>> @prefix geosparql: <http://www.opengis.net/ont/geosparql#> .> > > >>>> @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .> > > >>>> @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .> > > >>>> @prefix owl: <http://www.w3.org/2002/07/owl#> .> > > >>>> @prefix dc: <http://purl.org/dc/elements/1.1/> .> > > >>>> @prefix dcterms: <http://purl.org/dc/terms/> .> > > >>>> @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .> > > >>>> @prefix mm: <http://rdf.meteomatics.com/mm/> .> > > >>>> @prefix mm-measurements: <http://rdf.meteomatics.com/mm/measurements/> > > >> > > >> .> > > >>>> @prefix mm-locations: <http://rdf.meteomatics.com/mm/locations/> .> > > >>>>> > > >>>>> > > >>>> mm-locations:8500015 a mm:Location ;> > > >>>> a geosparql:Geometry ;> > > >>>> mm:dido [message truncated...]
