Re: empty backup files

2018-05-17 Thread Andy Seaborne

Dan,

The exceptions don't correspond to problems with the indexing of 
triples; they come from the node table.  The query


SELECT (count(*) AS ?C) { { ?s ?p ?o } UNION { GRAPH ?g { ?s ?p ?o } } }

only counts the triple and quad indexes and not the node table so

> "C": { "type": "literal" , "datatype":
> "http://www.w3.org/2001/XMLSchema#integer; , "value": "0" }

suggests the indexes are empty (specifically indexes SPO and GSPO).

Could moving the databases around have got files mixed up?

In the backup log, there wasn't a NodeTable exception.


Andy


On 15/05/18 22:36, Dan Pritts wrote:
...turns out that the server that I was working on is known to have a 
corrupted index - we'd left it running for no good reason.  Meanwhile, 
development work, and the associated CNAME, had moved on to another 
server.    So i was querying the good server's web interface, but 
logging in to the bad server to attempt backups.




I don't know what caused the index to be corrupted on the bad server - 
the person involved is one of those who is unavailable today.  If it 
seems like it might have been errors in fuseki (as opposed to operator 
error) I'll let you know.


For the record, on the bad machine,  the count(*) query returns:

"C": { "type": "literal" , "datatype": 
"http://www.w3.org/2001/XMLSchema#integer; , "value": "0" }


the tdbdump command returns with no output.  So the indices are so 
borked that the code believed there was nothing there.  Which fits with 
a size 0 backup.





Meanwhile, the "working" server, where we have the object we had trouble 
deleting, throws the following error when I try to back it up.  It only 
backs up a couple hundred (out of 20 million) entries before it croaks.


[2018-05-15 16:35:56] Admin  INFO  [156546] 200 OK (12 ms)
[2018-05-15 16:35:56] TDB    ERROR 
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]: 
Impossibly large object : 1013478516 bytes > 
filesize-(loc+SizeOfInt)=17445826

[2018-05-15 16:35:56] Log4jLoggerAdapter WARN  Exception in backup
org.apache.jena.tdb.base.file.FileException: 
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]: 
Impossibly large object : 1013478516 bytes > 
filesize-(loc+SizeOfInt)=17445826
     at 
org.apache.jena.tdb.base.objectfile.ObjectFileStorage.read(ObjectFileStorage.java:348)
     at 
org.apache.jena.tdb.base.objectfile.ObjectFileWrapper.read(ObjectFileWrapper.java:57)

     at org.apache.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:78)
     at 
org.apache.jena.tdb.store.nodetable.NodeTableNative.readNodeFromTable(NodeTableNative.java:186)
     at 
org.apache.jena.tdb.store.nodetable.NodeTableNative._retrieveNodeByNodeId(NodeTableNative.java:111)
     at 
org.apache.jena.tdb.store.nodetable.NodeTableNative.getNodeForNodeId(NodeTableNative.java:70)
     at 
org.apache.jena.tdb.store.nodetable.NodeTableCache._retrieveNodeByNodeId(NodeTableCache.java:128)
     at 
org.apache.jena.tdb.store.nodetable.NodeTableCache.getNodeForNodeId(NodeTableCache.java:82)
     at 
org.apache.jena.tdb.store.nodetable.NodeTableWrapper.getNodeForNodeId(NodeTableWrapper.java:50)
     at 
org.apache.jena.tdb.store.nodetable.NodeTableInline.getNodeForNodeId(NodeTableInline.java:67)



The  configuration is identical - it is a clone made from an AWS snapshot

In case it's useful:

count query:      "C": { "type": "literal" , "datatype": 
"http://www.w3.org/2001/XMLSchema#integer; , "value": "20371945" }


tdbdump dumps what looks like the same few hundred entries, then throws 
a similar stack trace:


17:31:28 ERROR TDB  :: 
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]: 
Impossibly large object : 1013478516 bytes > 
filesize-(loc+SizeOfInt)=17445826
org.apache.jena.tdb.base.file.FileException: 
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]: 
Impossibly large object : 1013478516 bytes > 
filesize-(loc+SizeOfInt)=17445826
     at 
org.apache.jena.tdb.base.objectfile.ObjectFileStorage.read(ObjectFileStorage.java:348)

     at org.apache.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:78)
     at 
org.apache.jena.tdb.store.nodetable.NodeTableNative.readNodeFromTable(NodeTableNative.java:186)
     at 
org.apache.jena.tdb.store.nodetable.NodeTableNative._retrieveNodeByNodeId(NodeTableNative.java:111)
     at 
org.apache.jena.tdb.store.nodetable.NodeTableNative.getNodeForNodeId(NodeTableNative.java:70)
     at 
org.apache.jena.tdb.store.nodetable.NodeTableCache._retrieveNodeByNodeId(NodeTableCache.java:128)




I  don't know what we did to muck up the indices; again, the appropriate 
guy is unavailable.  I doubt we used anything other than the http sparql 
interface, but i could definitely be wrong.    I know he was making very 
significant changes, but i think that was on yet another development 
server.



Andy Seaborne 

Re: empty backup files

2018-05-15 Thread Dan Pritts
...turns out that the server that I was working on is known to have a 
corrupted index - we'd left it running for no good reason.  Meanwhile, 
development work, and the associated CNAME, had moved on to another 
server.So i was querying the good server's web interface, but 
logging in to the bad server to attempt backups.




I don't know what caused the index to be corrupted on the bad server - 
the person involved is one of those who is unavailable today.  If it 
seems like it might have been errors in fuseki (as opposed to operator 
error) I'll let you know.


For the record, on the bad machine,  the count(*) query returns:

"C": { "type": "literal" , "datatype": 
"http://www.w3.org/2001/XMLSchema#integer; , "value": "0" }


the tdbdump command returns with no output.  So the indices are so 
borked that the code believed there was nothing there.  Which fits with 
a size 0 backup.





Meanwhile, the "working" server, where we have the object we had trouble 
deleting, throws the following error when I try to back it up.  It only 
backs up a couple hundred (out of 20 million) entries before it croaks.


[2018-05-15 16:35:56] Admin  INFO  [156546] 200 OK (12 ms)
[2018-05-15 16:35:56] TDBERROR 
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]: 
Impossibly large object : 1013478516 bytes > 
filesize-(loc+SizeOfInt)=17445826

[2018-05-15 16:35:56] Log4jLoggerAdapter WARN  Exception in backup
org.apache.jena.tdb.base.file.FileException: 
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]: 
Impossibly large object : 1013478516 bytes > 
filesize-(loc+SizeOfInt)=17445826
at 
org.apache.jena.tdb.base.objectfile.ObjectFileStorage.read(ObjectFileStorage.java:348)
at 
org.apache.jena.tdb.base.objectfile.ObjectFileWrapper.read(ObjectFileWrapper.java:57)

at org.apache.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:78)
at 
org.apache.jena.tdb.store.nodetable.NodeTableNative.readNodeFromTable(NodeTableNative.java:186)
at 
org.apache.jena.tdb.store.nodetable.NodeTableNative._retrieveNodeByNodeId(NodeTableNative.java:111)
at 
org.apache.jena.tdb.store.nodetable.NodeTableNative.getNodeForNodeId(NodeTableNative.java:70)
at 
org.apache.jena.tdb.store.nodetable.NodeTableCache._retrieveNodeByNodeId(NodeTableCache.java:128)
at 
org.apache.jena.tdb.store.nodetable.NodeTableCache.getNodeForNodeId(NodeTableCache.java:82)
at 
org.apache.jena.tdb.store.nodetable.NodeTableWrapper.getNodeForNodeId(NodeTableWrapper.java:50)
at 
org.apache.jena.tdb.store.nodetable.NodeTableInline.getNodeForNodeId(NodeTableInline.java:67)



The  configuration is identical - it is a clone made from an AWS snapshot

In case it's useful:

count query:  "C": { "type": "literal" , "datatype": 
"http://www.w3.org/2001/XMLSchema#integer; , "value": "20371945" }


tdbdump dumps what looks like the same few hundred entries, then throws 
a similar stack trace:


17:31:28 ERROR TDB  :: 
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]: 
Impossibly large object : 1013478516 bytes > 
filesize-(loc+SizeOfInt)=17445826
org.apache.jena.tdb.base.file.FileException: 
ObjectFileStorage.read[nodes](595777248)[filesize=613223078][file.size()=613223078]: 
Impossibly large object : 1013478516 bytes > 
filesize-(loc+SizeOfInt)=17445826
at 
org.apache.jena.tdb.base.objectfile.ObjectFileStorage.read(ObjectFileStorage.java:348)

at org.apache.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:78)
at 
org.apache.jena.tdb.store.nodetable.NodeTableNative.readNodeFromTable(NodeTableNative.java:186)
at 
org.apache.jena.tdb.store.nodetable.NodeTableNative._retrieveNodeByNodeId(NodeTableNative.java:111)
at 
org.apache.jena.tdb.store.nodetable.NodeTableNative.getNodeForNodeId(NodeTableNative.java:70)
at 
org.apache.jena.tdb.store.nodetable.NodeTableCache._retrieveNodeByNodeId(NodeTableCache.java:128)




I  don't know what we did to muck up the indices; again, the appropriate 
guy is unavailable.  I doubt we used anything other than the http sparql 
interface, but i could definitely be wrong.I know he was making very 
significant changes, but i think that was on yet another development 
server.



Andy Seaborne 
May 15, 2018 at 4:42 PM
Dan,

Could you try this query:

SELECT (count(*) AS ?C) { { ?s ?p ?o } UNION { GRAPH ?g { ?s ?p ?o } }


for the list archives, if anyone ever wants to try this, it's missing a 
trailing }



thanks
danno
--
Dan Pritts
ICPSR Computing & Network Services
University of Michigan



Re: empty backup files

2018-05-15 Thread Andy Seaborne

Dan,

Could you try this query:

SELECT (count(*) AS ?C) { { ?s ?p ?o } UNION { GRAPH ?g { ?s ?p ?o } }

and also stop the server and tdbdump --loc 

(Both use mechanism quite similar to how backup is performed.)

Andy

On 15/05/18 19:46, Dan Pritts wrote:

Hi,

I'm attempting to run a backup of a fuseki database. I've done this 
successfully many times on other systems, which I have configured 
identically (AFAIK; it is all done by hand, so an inconsistency is 
possible).


Based on the logs, fuseki appears to run through all the motions, but 
the file that's written is a 20-byte .gz file (i.e., an empty file that 
got gzip'd).  Logs and info below.


I was running 3.6.0 standalone.  I tried upgrading to 3.7.0 standalone, 
which didn't fix it.


Am i doing something wrong, or is this a bug?



In the spirit of full disclosure...I think it's probably fine, but It's 
possible that there is some corruption in the database.


In general queries work fine.   We have one triple with a null value 
that we can't figure out how to delete with sparql.  Our two experts are 
unavailable today and the rest of us are sparql-incompetent.    I don't 
need help with that, we will figure it out when one of them returns 
and/or by reading some more docs.


This led me to the ham-fisted idea of backing up, deleting the offending 
line, and restoring.    I agree with any suggestion that it's a stupid 
solution to this problem - but I still want to be able to back up this 
database, it's inconvenient to recreate it.





database has about 20 million triples.

Java:

log/archonnex/fuseki@acxcore-awstest0% java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)

fuseki command line:

fuseki 16548  1.1  0.6 12341352   209636 ?    Sl   13:26 
00:00:04 /etc/alternatives/java_sdk_1.8.0/bin/java -Xmx8G 
-Dlog4j.configuration=file:/etc/archonnex/fuseki/log4j.properties 
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps 
-XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation 
-XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100m 
-XX:+PrintTenuringDistribution 
-Xloggc:/var/log/archonnex/fuseki/gc-%t.log -jar 
/usr/local/apache-jena-fuseki-3.7.0/fuseki-server.jar 
--config=/etc/archonnex/fuseki/fcrepo.ttl


ttl config:

# cat fcrepo.ttl
@prefix :   .
@prefix tdb:  .
@prefix rdf:  .
@prefix ja:  .
@prefix rdfs:  .
@prefix fuseki:  .

:service_tdb_all  a fuseki:Service ;
     rdfs:label "TDB fcrepo" ;
     fuseki:dataset :tdb_dataset_readwrite ;
     fuseki:name "fcrepo" ;
     fuseki:serviceQuery "query" , "sparql" ;
     fuseki:serviceReadGraphStore "get" ;
fuseki:serviceReadWriteGraphStore
     "data" ;
     fuseki:serviceUpdate "update" ;
     fuseki:serviceUpload "upload" .

:tdb_dataset_readwrite
     a tdb:DatasetTDB ;
     tdb:location "/var/archonnex/fuseki/data/fcrepo" .


log file:

[2018-05-15 14:18:35] Admin  INFO [1] POST 
http://127.0.0.1:3030/$/backup/fcrepo

[2018-05-15 14:18:35] Admin  INFO [1] Backup dataset /fcrepo
[2018-05-15 14:18:35] Server INFO Task : 1 : backup
[2018-05-15 14:18:35] Server INFO [Task 1] starts : backup
[2018-05-15 14:18:35] Backup INFO [1]  Start backup /fcrepo -> 
/var/archonnex/fuseki/backups/fcrepo_2018-05-15_14-18-35

[2018-05-15 14:18:35] TDB    DEBUG Txn[2]/R: begin$
[2018-05-15 14:18:35] info   DEBUG Triple table: SPO :: SPO,POS,OSP
[2018-05-15 14:18:35] info   DEBUG Quad table: GSPO :: 
GSPO,GPOS,GOSP,POSG,OSPG,SPOG

[2018-05-15 14:18:35] info   DEBUG Prefixes: GPU :: GPU
[2018-05-15 14:18:35] LockMRSW   DEBUG Lock : pool-2-thread-1
[2018-05-15 14:18:35] TDB    DEBUG Txn[2]/R: start
[2018-05-15 14:18:35] LockMRSW   DEBUG pool-2-thread-1 >> enterCS: 
Thread R/W: 0/0 :: Model R/W: 0/0 (thread: pool-2-thread-1)
[2018-05-15 14:18:35] LockMRSW   DEBUG pool-2-thread-1 << enterCS: 
Thread R/W: 1/0 :: Model R/W: 1/0 (thread: pool-2-thread-1)

[2018-05-15 14:18:35] Admin  INFO [1] 200 OK (8 ms)
[2018-05-15 14:18:35] BlockAccessMapped DEBUG Segment: 0
[2018-05-15 14:18:35] BlockAccessMapped DEBUG Segment: 0
[2018-05-15 14:18:35] BlockAccessMapped DEBUG Segment: 0
[2018-05-15 14:18:35] BlockAccessMapped DEBUG Segment: 0
[2018-05-15 14:18:35] BlockAccessMapped DEBUG Segment: 0
[2018-05-15 14:18:35] BlockAccessMapped DEBUG Segment: 0
[2018-05-15 14:18:35] TDB    DEBUG Txn[2]/R: finish
[2018-05-15 14:18:35] LockMRSW   DEBUG pool-2-thread-1 >> leaveCS: 
Thread R/W: 1/0 :: Model R/W: 1/0 (thread: pool-2-thread-1)
[2018-05-15 14:18:35] LockMRSW   DEBUG pool-2-thread-1 << leaveCS: 
Thread R/W: 0/0 :: Model R/W: 0/0 (thread: pool-2-thread-1)
[2018-05-15 14:18:35] Backup INFO [1]