On 27/02/14 20:11, Steve Groeger wrote:
Hi Andy,
I ran the test again with the additional code you requested. It took a
long time to fail, over 4.3 million reads but it eventually did. The error
we saw was:
reader3: Models do not compare expected size: 1000 was: 778
java.lang.IllegalArgumentException: ObjectFile.read[nodes]: Bad read: -1
at com.hp.hpl.jena.tdb.base.objectfile.ObjectFileStorage.read(
ObjectFileStorage.java:311)
at com.hp.hpl.jena.tdb.lib.NodeLib.fetchDecode(NodeLib.java:79)
...
Hope this allows you to move closer to determining the issue.
The fact it takes such a long time is a bit of a nuisanse. I hoped that
nulling out would generate an earlier error with "read -1" (it might
have been reading valid but deleted NodeIds before : -1 is not a valid id).
We need to converge on exactly the same code to be testing - things are
a bit clearer but it will need some staring at the code.
Andy
Thanks
Steve Groeger
From: Andy Seaborne <[email protected]>
To: [email protected],
Date: 27/02/2014 15:09
Subject: Re: BlockException: No such block
On 27/02/14 13:50, Steve Groeger wrote:
Hi Andy,
All the latest tests I was running were setting SystemTDB.NullOut =
true;
I will add the code you specified to our tests and re-run them and will
send you our results shortly.
Thanks - if send the code as well, I'll try to run it at my end. Even
though I wasn't managing to catch this nearly as frequently as you, this
new situation might be better.
Andy
Thanks
Steve Groeger
IBM WebSphere Service Registry Development
MP 211, Hursley
Tel: (44) 1962 816911 Mobex: 279990 Mobile: 07718 517 129
Fax (44) 1962 816800
Lotus Notes: Steve Groeger/UK/IBM
Internet: [email protected]
From: Andy Seaborne <[email protected]>
To: [email protected],
Date: 27/02/2014 13:36
Subject: Re: BlockException: No such block
Hi Steve,
Thanks for the update.
On 25/02/14 15:15, Steve Groeger wrote:
Hi Andy,
I picked up a snapshot of 2.11.2 and tried running our test using these
libraries.
Adding the changes you suggested seems to have stopped the exception "
BlockException: No such block", the but we are now seeing other
symptoms,
which we saw previously but not as frequently as we are now.
The change in behaviour seems to be result of the SystemTDB.NullOut =
true
; as I also tried various version of the listStatements and this gave
the
same results, What does the SystemTDB.NullOut=true do?
NullOut makes the B+Tree code overwrite areas of blocks that are now
unused because of a delete with some value (FillByte = 0xFF). In normal
operation, it just leaves it as junk, but of NullOut is set it goes and
sets the bytes to a specific value, and one which does not make sense.
I would not expect it to change the frequency of errors, but it would
change the symptom.
What we are now seeing is data not being returned correctly.
We initially populate the database with a Resource with 1000
Properties,
then in the ReaderThread we query the database, using the following:
StmtIterator found = model.listStatements((Resource)null,
(Property)null, (RDFNode)null);
we are expecting to get back 1000 statements when we perform the
count = found.toList().size();
however we occasionally get a batch of results that show 0 or non 1000
statements being returned, then after several more reads this goes back
to
getting 1000 statements and everything continues as normal.
I'm not clear whether that is with or without NullOut set true.
We need to determine why we occasionally dont get the 1000 statements
returned that we are expecting. Any advice you caould provide would be
great.
It looks as if under some unusual conditions, either the B+Tree branch
node or leaf node is not being updated correctly for deletions.
("unusual" as this code is not new and for anyone doing deletes, heavily
used).
That gives us something to go on.
Could you please add some code that not only counts the StmtIterator but
also touches every part of the statement. Execute with
SystemTDB.NullOut=true
There is some delayed evaluation under the covers and this forces that
evaluation.
StmtIterator found = ....
long count = 0 ;
while(found.hasNext()) {
count++ ;
Statement s = found.next();
try {
Node subj = s.getSubject().asNode() ;
Node pred = s.getPredicate().asNode() ;
Node obj = s.getObject().asNode() ;
} catch (Exception ex) {
ex.printStackTrace(System.err) ;
System.exit(99) ;
}
}
Warning - this may slow the execution down, or not, depending on cache
efficiencies.
Andy
Thanks
Steve Groeger
IBM WebSphere Service Registry Development
MP 211, Hursley
Tel: (44) 1962 816911 Mobex: 279990 Mobile: 07718 517 129
Fax (44) 1962 816800
Lotus Notes: Steve Groeger/UK/IBM
Internet: [email protected]
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
