Andy,
I apologise for the mismatch between the code and the stack trace. I
captured the stack trace earlier and later modified the code to make it
easier to post, but the posted code still gives the same stack trace from
the:
at com.hp.hpl.jena.rdf.model.impl.ModelCom.add(ModelCom.java:202)
.
The checkModel method basically contained the following code:
StmtIterator found = model.listStatements(new
SimpleSelector((Resource)null, (Property)null, (RDFNode)null));
int count = 0;
count = found.toList().size();
if (count != NUM_PROPS) {
System.out.println("Models do not compare expected size: "
+ NUM_PROPS + " was: " + count);
}
found.close();
which I moved into the main test method (for ease of posting). If needed
I can run my test aagin with the posted code and then post the stack
trace.
When you say you fixed up the code, what changes did you make?
With our test we sometimes had to run it for at least 50000 iterations of
the writer thread loop before it failed. How many iterations did you run
the code for?
The other thing we are seeing, which may be expected is that the TDB
database gets very big (18Gb+ on my machine) even though we are adding and
then deleting the objects in the writer thread.
Is this normal behaviour for the TDB database to do this?
Thanks
Steve Groeger
From: Andy Seaborne <[email protected]>
To: [email protected],
Date: 23/01/2014 18:49
Subject: Re: BlockException: No such block
Steve,
The stacktrace does not correspond to the test code posted - there's no
class CreateThread nor methods .checkModel or .getModelCopy.
I have fixed up the code in your message and run it and it has been OK
for me over a couple of extended runs.
Andy
On 23/01/14 13:04, Steve Groeger wrote:
> We are using Jena 2.11.0 and trying to do some concurrency stress
testing.
> When running our tests, after a while we get the following error.
>
> com.hp.hpl.jena.tdb.base.block.BlockException: No such block
> at
> com.hp.hpl.jena.tdb.base.recordbuffer.RecordRangeIterator.iterator(
> RecordRangeIterator.java:39)
> at com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.iterator(
> BPlusTree.java:383)
> at com.hp.hpl.jena.tdb.index.bplustree.BPlusTree.iterator(
> BPlusTree.java:366)
> at com.hp.hpl.jena.tdb.index.TupleIndexRecord.findWorker(
> TupleIndexRecord.java:164)
> at com.hp.hpl.jena.tdb.index.TupleIndexRecord.findOrScan(
> TupleIndexRecord.java:84)
> at com.hp.hpl.jena.tdb.index.TupleIndexRecord.performFind(
> TupleIndexRecord.java:78)
> at com.hp.hpl.jena.tdb.index.TupleIndexBase.find(
> TupleIndexBase.java:91)
> at
com.hp.hpl.jena.tdb.index.TupleTable.find(TupleTable.java:197)
> at com.hp.hpl.jena.tdb.nodetable.NodeTupleTableConcrete.find(
> NodeTupleTableConcrete.java:169)
> at com.hp.hpl.jena.tdb.nodetable.NodeTupleTableConcrete.find(
> NodeTupleTableConcrete.java:157)
> at
> com.hp.hpl.jena.tdb.nodetable.NodeTupleTableConcrete.findAsNodeIds(
> NodeTupleTableConcrete.java:146)
> at com.hp.hpl.jena.tdb.store.QuadTable.find(QuadTable.java:87)
> at
> com.hp.hpl.jena.tdb.store.DatasetGraphTDB.findInSpecificNamedGraph(
> DatasetGraphTDB.java:89)
> at com.hp.hpl.jena.sparql.core.DatasetGraphBaseFind.findNG(
> DatasetGraphBaseFind.java:59)
> at com.hp.hpl.jena.sparql.core.DatasetGraphBaseFind.find(
> DatasetGraphBaseFind.java:48)
> at com.hp.hpl.jena.sparql.core.GraphView.graphBaseFind(
> GraphView.java:110)
> at com.hp.hpl.jena.sparql.core.GraphView.graphBaseFind(
> GraphView.java:104)
> at
com.hp.hpl.jena.graph.impl.GraphBase.find(GraphBase.java:268)
> at com.hp.hpl.jena.graph.GraphUtil.findAll(GraphUtil.java:128)
> at com.hp.hpl.jena.graph.GraphUtil.addInto(GraphUtil.java:183)
> at
com.hp.hpl.jena.rdf.model.impl.ModelCom.add(ModelCom.java:208)
> at
com.hp.hpl.jena.rdf.model.impl.ModelCom.add(ModelCom.java:202)
> at com.ibm.jenatest.CreateThread.getModelCopy(
> CreateThread.java:154)
> at
com.ibm.jenatest.CreateThread.checkModel(CreateThread.java:125)
> at com.ibm.jenatest.CreateThread.test(CreateThread.java:86)
> at com.ibm.jenatest.CreateThread.run(CreateThread.java:52)
>
> Can anyone please explain why we are getting this error, ie if this is a
> bug or we are doing something wrong in our test.
>
> Occasionally we get other issues in the reader thread when we retrieve
the
> statements from the resource, the number that are returned do not match
> the number that are expected (ie 1000) but if we get the statements
again
> all 1000 are there. Hopefully there is an explanation for each of these
> issues.
>
> We are running out test on a RHEL system using the IBM JVM, but have
also
> used other JVMs and we get the same issues.
>
> Our test has a writer thread and several (currently 2) reader threads.
The
> number of iterations necessary for the test to fail vary greatly but it
> does fail at some point.
>
> Here are the classes for our test and the classes for the 2 threads:
>
> public class ReaderWriterTest {
>
> public static void main(String[] args) {
> int i = 1;
> GenericTestThread writer = null;
> writer = new WriterThread("writer" + (i++), args[0] +
"jenatest");
> List<ReaderThread> readers = new ArrayList<ReaderThread>();
> // setup the reader threads
> readers.add(new ReaderThread("reader" + (i++), args[0] +
> "jenatest"));
> readers.add(new ReaderThread("reader" + (i++), args[0] +
> "jenatest"));
>
> if (writer!=null)
> writer.setReaderThreads(readers);
>
> for (GenericTestThread reader : readers) {
> if (writer!=null)
> reader.setWriterThread(writer);
>
> // create the specific data for each of the reader threads
to
> read
> reader.setReaderThreads(readers);
> Dataset createSession = TDBFactory.createDataset
> (((ReaderThread)reader).location);
> createSession.begin(ReadWrite.WRITE);
> String uri = "http://www.ibm.com/"; + reader.getName() + "/"
+
> 0;
>
> // Create Jena model with a number of properties
> Model model = createSession.getNamedModel(uri);
> Resource res = model.createResource(uri);
> for (int j = 0; j < 1000; j++) {
> Property p = model.createProperty("
> http://www.ibm.com/prop/"; + reader.getName() + "/" + j);
> res.addProperty(p, model.createTypedLiteral("Property"
+
> reader.getName() + "/" + j));
> }
> model.close();
> createSession.commit();
> createSession.end();
> createSession.close();
> }
> // start the writer thread
> if (writer!=null)
> writer.start();
>
> // start the reader threads
> for (GenericTestThread reader : readers) {
> reader.start();
> }
> }
> }
>
>
> public class WriterThread extends GenericTestThread {
>
> private static final int NUM_PROPS = 1000;
> private String location;
>
> public WriterThread(String name, String location) {
> super(name);
> this.location = location;
> }
>
> @Override
> protected void test() {
> for (int i = 0; true; i++) {
> String uri = "http://www.ibm.com/"; + getName() + "/" + i;
> checkStop(uri);
>
> Dataset createSession = TDBFactory.createDataset(location);
> createSession.begin(ReadWrite.WRITE);
> info(""+i+" size="+createSession.asDatasetGraph().size());
>
> // Create Jena model with a number of properties
> Model model = createSession.getNamedModel(uri);
> Resource res = model.createResource(uri);
> for (int i1 = 0; i1 < NUM_PROPS; i1++) {
> Property p = model.createProperty("
> http://www.ibm.com/prop/"; + getName() + "/" + i1);
> res.addProperty(p, model.createTypedLiteral("Property"
+
> getName() + "/" + i1));
> }
> createSession.commit();
> createSession.end();
> model.close();
>
> createSession.begin(ReadWrite.WRITE);
> model = createSession.getNamedModel(uri);
> model.removeAll();
> createSession.removeNamedModel(uri);
> createSession.commit();
> createSession.end();
> model.close();
> createSession.close();
> }
> }
> }
>
> public class ReaderThread extends GenericTestThread {
>
> private static final int NUM_PROPS = 1000;
> protected String location;
>
> public ReaderThread(String name, String location) {
> super(name);
> this.location = location;
> }
>
> @Override
> protected void test() {
> String uri = "http://www.ibm.com/"; + getName() + "/" + 0;
> int iteration = 0;
> while (true) {
> if (((iteration++) % 500) == 0) {
> info("check: "+iteration);
> // need to add a sleep to the thread otherwise
jena
> can never write the data and the journal.jml file
> // gets excessively large and we run out of
memory
> in the JVM
> try {
> Thread.sleep(1000);
> } catch (InterruptedException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
> }
> Dataset checkSession = TDBFactory.createDataset(location);
> checkSession.begin(ReadWrite.READ);
> Model model = checkSession.getNamedModel(uri);
> StmtIterator found = model.listStatements(new
> SimpleSelector((Resource)null, (Property)null, (RDFNode)null));
> Int count = found.toList().size();
> if (count != NUM_PROPS) {
> System.out.println("Models do not compare expected
size: "
> + NUM_PROPS + " was: " + count);
> }
> found.close();
> checkSession.end();
> // model.close(); // removed as causes errors with jena
> saying we have multiple writers !!!!!!
> checkSession.close();
> }
> }
> }
>
> public abstract class GenericTestThread extends Thread {
>
> private GenericTestThread writerThread;
> private volatile boolean stopped = false;
> private List<ReaderThread> readers;
>
> public GenericTestThread(String name) {
> setName(name);
> }
>
> protected abstract void test();
>
> public void run() {
> try {
> info("Running thread");
> test();
> info("Thread done");
> } catch (Exception e) {// STYLE reason: fault barrier
> synchronized (WriterThread.class) {
> snapshotWriterThread();
> for (GenericTestThread reader : readers) {
> reader.shutdown();
> }
> e.printStackTrace(System.out);// STYLE reason: unit
test
> // System.exit(0); // STYLE reason: TODO
> }
> }
> }
>
> protected void snapshotWriterThread() {
> if (writerThread != null) {
> info(writerThread.getStackTrace());
> writerThread.shutdown();
> }
> }
>
> private void shutdown() {
> stopped = true;
> }
>
> private void info(StackTraceElement[] stackTraceElements) {
> for (StackTraceElement trace : stackTraceElements) {
> info(" " + trace);
> }
> }
>
> protected void info(Exception e) {
> System.out.println(getName() + ": " + e.toString());// STYLE
> reason: unit test
> }
>
> protected void info(String message) {
> System.out.println(getName() + ": " + message);// STYLE reason:
> unit test
> }
>
> protected void checkStop(String message) {
> if (stopped) {
> throw new RuntimeException("thread stopped: " + message);
> }
> }
>
> public void setWriterThread(GenericTestThread other) {
> this.writerThread = other;
> }
>
> public GenericTestThread getWriterThread() {
> return this.writerThread;
> }
>
> public void setReaderThreads(List<ReaderThread> readers) {
> this.readers = readers;
> }
>
> public List<ReaderThread> getReaderThreads() {
> return this.readers;
> }
> }
>
> Thanks
> Steve Groeger
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
>
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
