Re: empty backup files
Dan,
The exceptions don't correspond to problems with the indexing of
triples; they come from the node table. The query
SELECT (count(*) AS ?C) { { ?s ?p ?o } UNION { GRAPH ?g { ?s ?p ?o } } }
only counts the triple and quad indexes and not the node table so
> "C": { "type": "literal" , "datatype":
> "http://www.w3.org/2001/XMLSchema#integer"; , "value": "0" }
suggests the indexes are empty (specifically indexes SPO and GSPO).
Could moving the databases around have got files mixed up?
In the backup log, there wasn't a NodeTable exception.
Andy
On 15/05/18 22:36, Dan Pritts wrote:
...turns out that the server that I was working on is known to have a
corrupted index - we'd left it running for no good reason. Meanwhile,
development work, and the associated CNAME, had moved on to another
server. So i was querying the good server's web interface, but
logging in to the bad server to attempt backups.
I don't know what caused the index to be corrupted on the bad server -
the person involved is one of those who is unavailable today. If it
seems like it might have been errors in fuseki (as opposed to operator
error) I'll let you know.
For the record, on the bad machine, the count(*) query returns:
"C": { "type": "literal" , "datatype":
"http://www.w3.org/2001/XMLSchema#integer"; , "value": "0" }
the tdbdump command returns with no output. So the indices are so
borked that the code believed there was nothing there. Which fits with
a size 0 backup.
Meanwhile, the "working" server, where we have the object we had trouble
deleting, throws the following error when I try to back it up. It only
backs up a couple hundred (out of 20 million) entries before it croaks.
[2018-05-15 16:35:56] Admin INFO [156546] 200 OK (12 ms)
[2018-05-15 16:35:56] TDB ERROR
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]:
Impossibly large object : 1013478516 bytes >
filesize-(loc+SizeOfInt)=17445826
[2018-05-15 16:35:56] Log4jLoggerAdapter WARN Exception in backup
org.apache.jena.tdb.base.file.FileException:
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]:
Impossibly large object : 1013478516 bytes >
filesize-(loc+SizeOfInt)=17445826
at
org.apache.jena.tdb.base.objectfile.ObjectFileStorage.read(ObjectFileStorage.java:348)
at
org.apache.jena.tdb.base.objectfile.ObjectFileWrapper.read(ObjectFileWrapper.java:57)
at org.apache.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:78)
at
org.apache.jena.tdb.store.nodetable.NodeTableNative.readNodeFromTable(NodeTableNative.java:186)
at
org.apache.jena.tdb.store.nodetable.NodeTableNative._retrieveNodeByNodeId(NodeTableNative.java:111)
at
org.apache.jena.tdb.store.nodetable.NodeTableNative.getNodeForNodeId(NodeTableNative.java:70)
at
org.apache.jena.tdb.store.nodetable.NodeTableCache._retrieveNodeByNodeId(NodeTableCache.java:128)
at
org.apache.jena.tdb.store.nodetable.NodeTableCache.getNodeForNodeId(NodeTableCache.java:82)
at
org.apache.jena.tdb.store.nodetable.NodeTableWrapper.getNodeForNodeId(NodeTableWrapper.java:50)
at
org.apache.jena.tdb.store.nodetable.NodeTableInline.getNodeForNodeId(NodeTableInline.java:67)
The configuration is identical - it is a clone made from an AWS snapshot
In case it's useful:
count query: "C": { "type": "literal" , "datatype":
"http://www.w3.org/2001/XMLSchema#integer"; , "value": "20371945" }
tdbdump dumps what looks like the same few hundred entries, then throws
a similar stack trace:
17:31:28 ERROR TDB ::
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]:
Impossibly large object : 1013478516 bytes >
filesize-(loc+SizeOfInt)=17445826
org.apache.jena.tdb.base.file.FileException:
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]:
Impossibly large object : 1013478516 bytes >
filesize-(loc+SizeOfInt)=17445826
at
org.apache.jena.tdb.base.objectfile.ObjectFileStorage.read(ObjectFileStorage.java:348)
at org.apache.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:78)
at
org.apache.jena.tdb.store.nodetable.NodeTableNative.readNodeFromTable(NodeTableNative.java:186)
at
org.apache.jena.tdb.store.nodetable.NodeTableNative._retrieveNodeByNodeId(NodeTableNative.java:111)
at
org.apache.jena.tdb.store.nodetable.NodeTableNative.getNodeForNodeId(NodeTableNative.java:70)
at
org.apache.jena.tdb.store.nodetable.NodeTableCache._retrieveNodeByNodeId(NodeTableCache.java:128)
I don't know what we did to muck up the indices; again, the appropriate
guy is unavailable. I doubt we used anything other than the http sparql
interface, but i could definitely be wrong. I know he was making very
significant changes, but i think that was on yet another development
server.
Andy Seaborne
Re: empty backup files
...turns out that the server that I was working on is known to have a
corrupted index - we'd left it running for no good reason. Meanwhile,
development work, and the associated CNAME, had moved on to another
server.So i was querying the good server's web interface, but
logging in to the bad server to attempt backups.
I don't know what caused the index to be corrupted on the bad server -
the person involved is one of those who is unavailable today. If it
seems like it might have been errors in fuseki (as opposed to operator
error) I'll let you know.
For the record, on the bad machine, the count(*) query returns:
"C": { "type": "literal" , "datatype":
"http://www.w3.org/2001/XMLSchema#integer"; , "value": "0" }
the tdbdump command returns with no output. So the indices are so
borked that the code believed there was nothing there. Which fits with
a size 0 backup.
Meanwhile, the "working" server, where we have the object we had trouble
deleting, throws the following error when I try to back it up. It only
backs up a couple hundred (out of 20 million) entries before it croaks.
[2018-05-15 16:35:56] Admin INFO [156546] 200 OK (12 ms)
[2018-05-15 16:35:56] TDBERROR
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]:
Impossibly large object : 1013478516 bytes >
filesize-(loc+SizeOfInt)=17445826
[2018-05-15 16:35:56] Log4jLoggerAdapter WARN Exception in backup
org.apache.jena.tdb.base.file.FileException:
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]:
Impossibly large object : 1013478516 bytes >
filesize-(loc+SizeOfInt)=17445826
at
org.apache.jena.tdb.base.objectfile.ObjectFileStorage.read(ObjectFileStorage.java:348)
at
org.apache.jena.tdb.base.objectfile.ObjectFileWrapper.read(ObjectFileWrapper.java:57)
at org.apache.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:78)
at
org.apache.jena.tdb.store.nodetable.NodeTableNative.readNodeFromTable(NodeTableNative.java:186)
at
org.apache.jena.tdb.store.nodetable.NodeTableNative._retrieveNodeByNodeId(NodeTableNative.java:111)
at
org.apache.jena.tdb.store.nodetable.NodeTableNative.getNodeForNodeId(NodeTableNative.java:70)
at
org.apache.jena.tdb.store.nodetable.NodeTableCache._retrieveNodeByNodeId(NodeTableCache.java:128)
at
org.apache.jena.tdb.store.nodetable.NodeTableCache.getNodeForNodeId(NodeTableCache.java:82)
at
org.apache.jena.tdb.store.nodetable.NodeTableWrapper.getNodeForNodeId(NodeTableWrapper.java:50)
at
org.apache.jena.tdb.store.nodetable.NodeTableInline.getNodeForNodeId(NodeTableInline.java:67)
The configuration is identical - it is a clone made from an AWS snapshot
In case it's useful:
count query: "C": { "type": "literal" , "datatype":
"http://www.w3.org/2001/XMLSchema#integer"; , "value": "20371945" }
tdbdump dumps what looks like the same few hundred entries, then throws
a similar stack trace:
17:31:28 ERROR TDB ::
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]:
Impossibly large object : 1013478516 bytes >
filesize-(loc+SizeOfInt)=17445826
org.apache.jena.tdb.base.file.FileException:
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]:
Impossibly large object : 1013478516 bytes >
filesize-(loc+SizeOfInt)=17445826
at
org.apache.jena.tdb.base.objectfile.ObjectFileStorage.read(ObjectFileStorage.java:348)
at org.apache.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:78)
at
org.apache.jena.tdb.store.nodetable.NodeTableNative.readNodeFromTable(NodeTableNative.java:186)
at
org.apache.jena.tdb.store.nodetable.NodeTableNative._retrieveNodeByNodeId(NodeTableNative.java:111)
at
org.apache.jena.tdb.store.nodetable.NodeTableNative.getNodeForNodeId(NodeTableNative.java:70)
at
org.apache.jena.tdb.store.nodetable.NodeTableCache._retrieveNodeByNodeId(NodeTableCache.java:128)
I don't know what we did to muck up the indices; again, the appropriate
guy is unavailable. I doubt we used anything other than the http sparql
interface, but i could definitely be wrong.I know he was making very
significant changes, but i think that was on yet another development
server.
Andy Seaborne
May 15, 2018 at 4:42 PM
Dan,
Could you try this query:
SELECT (count(*) AS ?C) { { ?s ?p ?o } UNION { GRAPH ?g { ?s ?p ?o } }
for the list archives, if anyone ever wants to try this, it's missing a
trailing }
thanks
danno
--
Dan Pritts
ICPSR Computing & Network Services
University of Michigan
Re: empty backup files
Dan,
Could you try this query:
SELECT (count(*) AS ?C) { { ?s ?p ?o } UNION { GRAPH ?g { ?s ?p ?o } }
and also stop the server and tdbdump --loc
(Both use mechanism quite similar to how backup is performed.)
Andy
On 15/05/18 19:46, Dan Pritts wrote:
Hi,
I'm attempting to run a backup of a fuseki database. I've done this
successfully many times on other systems, which I have configured
identically (AFAIK; it is all done by hand, so an inconsistency is
possible).
Based on the logs, fuseki appears to run through all the motions, but
the file that's written is a 20-byte .gz file (i.e., an empty file that
got gzip'd). Logs and info below.
I was running 3.6.0 standalone. I tried upgrading to 3.7.0 standalone,
which didn't fix it.
Am i doing something wrong, or is this a bug?
In the spirit of full disclosure...I think it's probably fine, but It's
possible that there is some corruption in the database.
In general queries work fine. We have one triple with a null value
that we can't figure out how to delete with sparql. Our two experts are
unavailable today and the rest of us are sparql-incompetent. I don't
need help with that, we will figure it out when one of them returns
and/or by reading some more docs.
This led me to the ham-fisted idea of backing up, deleting the offending
line, and restoring. I agree with any suggestion that it's a stupid
solution to this problem - but I still want to be able to back up this
database, it's inconvenient to recreate it.
database has about 20 million triples.
Java:
log/archonnex/fuseki@acxcore-awstest0% java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
fuseki command line:
fuseki 16548 1.1 0.6 12341352 209636 ? Sl 13:26
00:00:04 /etc/alternatives/java_sdk_1.8.0/bin/java -Xmx8G
-Dlog4j.configuration=file:/etc/archonnex/fuseki/log4j.properties
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100m
-XX:+PrintTenuringDistribution
-Xloggc:/var/log/archonnex/fuseki/gc-%t.log -jar
/usr/local/apache-jena-fuseki-3.7.0/fuseki-server.jar
--config=/etc/archonnex/fuseki/fcrepo.ttl
ttl config:
# cat fcrepo.ttl
@prefix : .
@prefix tdb: .
@prefix rdf: .
@prefix ja: .
@prefix rdfs: .
@prefix fuseki: .
:service_tdb_all a fuseki:Service ;
rdfs:label "TDB fcrepo" ;
fuseki:dataset :tdb_dataset_readwrite ;
fuseki:name "fcrepo" ;
fuseki:serviceQuery "query" , "sparql" ;
fuseki:serviceReadGraphStore "get" ;
fuseki:serviceReadWriteGraphStore
"data" ;
fuseki:serviceUpdate "update" ;
fuseki:serviceUpload "upload" .
:tdb_dataset_readwrite
a tdb:DatasetTDB ;
tdb:location "/var/archonnex/fuseki/data/fcrepo" .
log file:
[2018-05-15 14:18:35] Admin INFO [1] POST
http://127.0.0.1:3030/$/backup/fcrepo
[2018-05-15 14:18:35] Admin INFO [1] Backup dataset /fcrepo
[2018-05-15 14:18:35] Server INFO Task : 1 : backup
[2018-05-15 14:18:35] Server INFO [Task 1] starts : backup
[2018-05-15 14:18:35] Backup INFO [1] Start backup /fcrepo ->
/var/archonnex/fuseki/backups/fcrepo_2018-05-15_14-18-35
[2018-05-15 14:18:35] TDB DEBUG Txn[2]/R: begin$
[2018-05-15 14:18:35] info DEBUG Triple table: SPO :: SPO,POS,OSP
[2018-05-15 14:18:35] info DEBUG Quad table: GSPO ::
GSPO,GPOS,GOSP,POSG,OSPG,SPOG
[2018-05-15 14:18:35] info DEBUG Prefixes: GPU :: GPU
[2018-05-15 14:18:35] LockMRSW DEBUG Lock : pool-2-thread-1
[2018-05-15 14:18:35] TDB DEBUG Txn[2]/R: start
[2018-05-15 14:18:35] LockMRSW DEBUG pool-2-thread-1 >> enterCS:
Thread R/W: 0/0 :: Model R/W: 0/0 (thread: pool-2-thread-1)
[2018-05-15 14:18:35] LockMRSW DEBUG pool-2-thread-1 << enterCS:
Thread R/W: 1/0 :: Model R/W: 1/0 (thread: pool-2-thread-1)
[2018-05-15 14:18:35] Admin INFO [1] 200 OK (8 ms)
[2018-05-15 14:18:35] BlockAccessMapped DEBUG Segment: 0
[2018-05-15 14:18:35] BlockAccessMapped DEBUG Segment: 0
[2018-05-15 14:18:35] BlockAccessMapped DEBUG Segment: 0
[2018-05-15 14:18:35] BlockAccessMapped DEBUG Segment: 0
[2018-05-15 14:18:35] BlockAccessMapped DEBUG Segment: 0
[2018-05-15 14:18:35] BlockAccessMapped DEBUG Segment: 0
[2018-05-15 14:18:35] TDB DEBUG Txn[2]/R: finish
[2018-05-15 14:18:35] LockMRSW DEBUG pool-2-thread-1 >> leaveCS:
Thread R/W: 1/0 :: Model R/W: 1/0 (thread: pool-2-thread-1)
[2018-05-15 14:18:35] LockMRSW DEBUG pool-2-thread-1 << leaveCS:
Thread R/W: 0/0 :: Model R/W: 0/0 (thread: pool-2-thread-1)
[2018-05-15 14:18:35] Backup INFO [1]
