Actually, I take back the comment about it working if you synchronize on QueryExecutionFactory.create() method. The accumulators are created lazily, so mutations to the internal buckets can occur at any point during the query execution.
You may just want to avoid caching those Group By queries. -Stephen > -----Original Message----- > From: Stephen Allen [mailto:[email protected]] > Sent: Tuesday, September 20, 2011 11:55 AM > To: [email protected] > Subject: RE: Aggregators and concurrent use of Query object > > Hi Holger, > > I believe you are correct that Query objects with aggregators cannot be > reused by different threads. They *can* be reused by the same thread > or by > different threads that synchronize the compile step, but even then > there is > a problem with the Query object hanging onto references to a new > aggregator > for each query execution. > > The thing causing this appears to be in AlgebraGenerator.java line 562, > where the aggregators added to a Query object are referenced directly > by the > compiled query plan. Instead, we should make a copy of the aggregators > so > that the original Query object remains immutable. > > I've created a JIRA issue and submitted a patch, JENA-120: > https://issues.apache.org/jira/browse/JENA-120 > > As a work-around until the patch is applied, I think you can > synchronize > around the QueryExecutionFactory.create() method. Or, you can decide > not to > cache Group By queries (test for this with Query.hasGroupBy()). > > I don't know if there are other issues that may prevent reusing Query > objects, maybe Andy can chime in here. > > -Stephen > > P.S. Your strategy of caching Query objects does avoid having to > reparse > the query string, which can be quite beneficial. Along these same > lines, a > better enhancement to ARQ would be a mechanism to cache the query plans > after the optimizer step. Query optimization itself can get quite > expensive > (n! for left-deep trees, and even worse for bushy trees). > > > > > -----Original Message----- > > From: Holger Knublauch [mailto:[email protected]] > > Sent: Tuesday, September 20, 2011 1:14 AM > > To: [email protected] > > Subject: Aggregators and concurrent use of Query object > > > > Hi Andy, > > > > we have (unreliably) run into exceptions like the one below, and my > > suspicion is that the ARQ Query class is not meant to be re-used by > > multiple threads. Although each step in the Query is converted into a > > corresponding Algebra objects for execution, the Aggregators seem to > be > > shared between multiple objects. Is this correct and do I need to > > create a new Query each time I want a QueryExecution? This would slow > > down things quite a lot, as we currently cache all Queries that were > > created from string representation. If this is the case, are there > any > > ways to tell which particular queries are not thread-safe, e.g. all > > queries involving aggregations? > > > > If I am totally off the mark, do you know what else could cause the > > exception below, only sometimes in multi-threading conditions? > > > > Thank you, > > Holger > > > > > > com.hp.hpl.jena.sparql.ARQInternalErrorException: Null for > accumulator > > at > > > com.hp.hpl.jena.sparql.expr.aggregate.AggregatorBase.getValue(Aggregato > > rBase.java:61) > > at > > > com.hp.hpl.jena.sparql.engine.iterator.QueryIterGroup.calc(QueryIterGro > > up.java:121) > > at > > > com.hp.hpl.jena.sparql.engine.iterator.QueryIterGroup.<init>(QueryIterG > > roup.java:32) > > at > > > com.hp.hpl.jena.sparql.engine.main.OpExecutor.execute(OpExecutor.java:4 > > 13) > > at > > > com.hp.hpl.jena.sparql.engine.main.ExecutionDispatch.visit(ExecutionDis > > patch.java:255) > > at > > com.hp.hpl.jena.sparql.algebra.op.OpGroup.visit(OpGroup.java:37) > > at > > > com.hp.hpl.jena.sparql.engine.main.ExecutionDispatch.exec(ExecutionDisp > > atch.java:33) > > at > > > com.hp.hpl.jena.sparql.engine.main.OpExecutor.executeOp(OpExecutor.java > > :107) > > at > > > com.hp.hpl.jena.sparql.engine.main.OpExecutor.execute(OpExecutor.java:4 > > 41) > > at > > > com.hp.hpl.jena.sparql.engine.main.ExecutionDispatch.visit(ExecutionDis > > patch.java:241) > > at > > com.hp.hpl.jena.sparql.algebra.op.OpExtend.visit(OpExtend.java:107) > > at > > > com.hp.hpl.jena.sparql.engine.main.ExecutionDispatch.exec(ExecutionDisp > > atch.java:33) > > at > > > com.hp.hpl.jena.sparql.engine.main.OpExecutor.executeOp(OpExecutor.java > > :107) > > at > > > com.hp.hpl.jena.sparql.engine.main.OpExecutor.execute(OpExecutor.java:3 > > 93) > > at > > > com.hp.hpl.jena.sparql.engine.main.ExecutionDispatch.visit(ExecutionDis > > patch.java:213) > > at > > com.hp.hpl.jena.sparql.algebra.op.OpProject.visit(OpProject.java:34) > > at > > > com.hp.hpl.jena.sparql.engine.main.ExecutionDispatch.exec(ExecutionDisp > > atch.java:33) > > at > > > com.hp.hpl.jena.sparql.engine.main.OpExecutor.executeOp(OpExecutor.java > > :107) > > at > > > com.hp.hpl.jena.sparql.engine.main.OpExecutor.execute(OpExecutor.java:8 > > 0) > > at com.hp.hpl.jena.sparql.engine.main.QC.execute(QC.java:40) > > at > > > com.hp.hpl.jena.sparql.engine.main.QueryEngineMain.eval(QueryEngineMain > > .java:52) > > at > > > com.hp.hpl.jena.sparql.engine.QueryEngineBase.evaluate(QueryEngineBase. > > java:138) > > at > > > com.hp.hpl.jena.sparql.engine.QueryEngineBase.createPlan(QueryEngineBas > > e.java:109) > > at > > > com.hp.hpl.jena.sparql.engine.QueryEngineBase.getPlan(QueryEngineBase.j > > ava:97) > > at > > > com.hp.hpl.jena.sparql.engine.main.QueryEngineMain$1.create(QueryEngine > > Main.java:91) > > at > > > com.hp.hpl.jena.sparql.engine.QueryExecutionBase.getPlan(QueryExecution > > Base.java:266) > > at > > > com.hp.hpl.jena.sparql.engine.QueryExecutionBase.startQueryIterator(Que > > ryExecutionBase.java:243) > > at > > > com.hp.hpl.jena.sparql.engine.QueryExecutionBase.execResultSet(QueryExe > > cutionBase.java:248) > > at > > > com.hp.hpl.jena.sparql.engine.QueryExecutionBase.execSelect(QueryExecut > > ionBase.java:94) > > at > > > org.topbraid.spin.arq.SPINARQFunction.executeBody(SPINARQFunction.java: > > 121)
