Hi Rupert, Thanks for the modifications I will port the changes to my "scalable" version.
How long does it take for the changes to arrive in https://github.com/apache/clerezza? Regards, Minto Op 18-3-2013 8:38, Rupert Westenthaler schreef: > Hi Minto > > Created CLEREZZA-745 describing this issue and provided a fix with > revision 1457660 [2]. You will need to port the fix to your "scalable" > version (CLEREZZA-736). While I do have adapted the > MultiThreadedSingleTdbDatasetTest to explicitly validate your usage > scenario it would still be nice if you could validate the fix against > your usage scenario > > best > Rupert > > > [1] https://issues.apache.org/jira/browse/CLEREZZA-745 > [2] http://svn.apache.org/r1457660 > > On Thu, Mar 14, 2013 at 2:43 PM, Andy Seaborne <[email protected]> wrote: >> On 14/03/13 09:39, Minto van der Sluis wrote: >>> Rupert, >>> >>> Thanks for the additional explanation. >>> >>> Regards, >>> >>> Minto >>> >>> Op 14-3-2013 10:31, Rupert Westenthaler schreef: >>>> Hi Minto >>>> >>>> I am traveling this week and do not have time to work on this until >>>> the weekend but I will have a look into this. >>>> >>>> Let me try to explain my concern again and make it more clear: >>>> >>>> The Jena TDB named graphs are hold in a single quad store table (SPOC >>>> - Subject Predicate Object Context). On the Clerezza side you have a >>>> TripleCollections (SPO) with a name (C). What that means is that all >>>> Clerezza TripleCollections provided by the same >>>> SingleTdbDatasetTcProvider do share the same SPOC table. meaning that >>>> a change of any of those TripleCollections will cause a modification >>>> in the Jena TDB Backend. This means that Iterators of all >>>> TripleCollections need to make a ReadLock on the SPOC table (and not >>>> only on the SPO section represented by the TripleCollection). >>>> >>>> While Clerezza allows to build a LockableMGraphWrapper over an MGrpah >>>> this is not sufficient for the SingleTdbDatasetTcProvider as this will >>>> only protect the SPO section and not the SPOC table used by the >>>> backend. So changes in other graphs - or the creation of a new graph - >>>> are still possible and will cause ConcurrentModificationExceptions as >>>> reported. >>>> >>>> To solve this issue one needs to ensure that a single ReadWrite lock >>>> is used for all TripleCollections provided by the >>>> SingleTdbDatasetTcProvider as this will allow users to lock the whole >>>> SPOC table of the backend when they perform operations on the Clerezza >>>> TripleCollections. >> >> A TDB dataset provides a single Lock you can reuse/wrap so all the graph >> locks are related when needed. The GraphTDB.getLock() is the dataset lock. >> >> Transactions would be better. Better concurrency (concurrent writer and >> multiple readers). >> >> Andy >> >> >>>> best >>>> Rupert >>>> >>>> >>>> On Thu, Mar 14, 2013 at 9:50 AM, Minto van der Sluis <[email protected]> >>>> wrote: >>>>> Hi, >>>>> >>>>> Half of what the 2 of you write is not very clear to me. Probably due to >>>>> being a novice when it comes to Clerezza internals. >>>>> >>>>> Maybe I will start with giving CLEREZZA-726 another try and then check >>>>> if I still get these exceptions. >>>>> >>>>> Regard, >>>>> >>>>> Minto >>>>> >>>>> Op 13-3-2013 18:35, Reto Bachmann-Gmür schreef: >>>>>> On Wed, Mar 13, 2013 at 6:04 PM, Rupert Westenthaler < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I think that this is cased by the fact that if you create a >>>>>>> LockableMGraph over MGraphs provided by the SingleTdbDatasetTcProvider >>>>>>> you end up in a situation where you have multiple ReadWrite Locks on >>>>>>> the same quad store (the Jena TDB dataset). This means that acquiring >>>>>>> a write lock on one MGraph will not prohibit changes in other graphs - >>>>>>> or the creation of new graphs. Because of that you will end up with >>>>>>> ConcurrentModificationException when using iterators over triples >>>>>>> (such as going over SPARQL results). >>>>>>> >>>>>> True. But where is the graph locked in the first place? It should >>>>>> aquire a >>>>>> lock before iterating though the graph, does this happen? >>>>>> >>>>>> cheers, >>>>>> reto >>>>>> >>>>>>> The solution would be to >>>>>>> >>>>>>> * create a single ReadWirte lock for the SingleTdbDatasetTcProvider >>>>>>> * replace all synchronized(dataset){..} block with read/wirte locks >>>>>>> * all methods returning MGraphs need to return LockableMGraph >>>>>>> instances that do use the ReadWrite lock used by the >>>>>>> SingleTdbDatasetTcProvider >>>>>>> * users would than need to use the LockableMGraph instance provided by >>>>>>> the provider and NOT wrap those with an other LockableMGraph instance >>>>>>> (e.g. the LockableMGraphWrapper). >>>>>>> >>>>>>> best >>>>>>> Rupert >>>>>>> >>>>>>> >>>>>>> On Wed, Mar 13, 2013 at 5:31 PM, Minto van der Sluis <[email protected]> >>>>>>> wrote: >>>>>>>> Hi Folks, >>>>>>>> >>>>>>>> I ran into an issue is both the existing SingleTdbDatasetTcProvider >>>>>>>> and >>>>>>>> my customized version (see CLEREZZA-736). >>>>>>>> >>>>>>>> How to reproduce: >>>>>>>> 1) Have some process constantly inject new named graphs (I had a >>>>>>>> process >>>>>>>> injecting 1000 named graphs) >>>>>>>> 2) perform a query while 1 is still running. I used the following >>>>>>>> query: >>>>>>>> >>>>>>>> SELECT ?graphName WHERE { GRAPH ?graphName {} } LIMIT 10 >>>>>>>> OFFSET 0 >>>>>>>> >>>>>>>> 3) repeat step 2 a number of times (since the error does not always >>>>>>> occur) >>>>>>>> This results in a ConcurrentModificationException (see stacktrace >>>>>>>> below). I am not sure whether this is a Clerezza or Jena issue. >>>>>>>> >>>>>>>> Anyone an idea what is causing this? Or more importantly how to fix >>>>>>>> it? >>>>>>>> >>>>>>>> Should I create a Jira issue for this? >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> -- >>>>>>>> ir. ing. Minto van der Sluis >>>>>>>> Software innovator / renovator >>>>>>>> Xup BV >>>>>>>> >>>>>>>> >>>>>>>> Stacktrace: >>>>>>>> java.util.ConcurrentModificationException: Iterator: started at 7103, >>>>>>> now 7105 >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.tdb.sys.DatasetControlMRSW.policyError(DatasetControlMRSW.java:157) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.tdb.sys.DatasetControlMRSW.access$000(DatasetControlMRSW.java:32) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.tdb.sys.DatasetControlMRSW$IteratorCheckNotConcurrent.checkCourrentModification(DatasetControlMRSW.java:110) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.tdb.sys.DatasetControlMRSW$IteratorCheckNotConcurrent.hasNext(DatasetControlMRSW.java:118) >>>>>>>> at org.openjena.atlas.iterator.Iter$4.hasNext(Iter.java:295) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.tdb.store.GraphTDBBase$ProjectQuadsToTriples.hasNext(GraphTDBBase.java:173) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.util.iterator.WrappedIterator.hasNext(WrappedIterator.java:76) >>>>>>>> at >>>>>>> >>>>>>> org.apache.clerezza.rdf.jena.storage.JenaGraphAdaptor$1.hasNext(JenaGraphAdaptor.java:106) >>>>>>>> at >>>>>>> >>>>>>> org.apache.clerezza.rdf.core.impl.AbstractTripleCollection$1.hasNext(AbstractTripleCollection.java:78) >>>>>>>> at >>>>>>> >>>>>>> org.apache.clerezza.rdf.core.access.LockingIterator.hasNext(LockingIterator.java:47) >>>>>>>> at >>>>>>> >>>>>>> org.apache.clerezza.rdf.jena.facade.JenaGraph$1.hasNext(JenaGraph.java:95) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.util.iterator.WrappedIterator.hasNext(WrappedIterator.java:76) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIterTriplePattern$TripleMapper.hasNextBinding(QueryIterTriplePattern.java:151) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:112) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIterRepeatApply.hasNextBinding(QueryIterRepeatApply.java:79) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:112) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIterBlockTriples.hasNextBinding(QueryIterBlockTriples.java:64) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:112) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.main.iterator.QueryIterGraph$QueryIterGraphInner.hasNextBinding(QueryIterGraph.java:123) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:112) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIterRepeatApply.hasNextBinding(QueryIterRepeatApply.java:79) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:112) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIterConvert.hasNextBinding(QueryIterConvert.java:59) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:112) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIterSlice.hasNextBinding(QueryIterSlice.java:76) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:112) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:112) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorWrapper.hasNextBinding(QueryIteratorWrapper.java:40) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.iterator.QueryIteratorBase.hasNext(QueryIteratorBase.java:112) >>>>>>>> at >>>>>>> >>>>>>> com.hp.hpl.jena.sparql.engine.ResultSetStream.hasNext(ResultSetStream.java:72) >>>>>>>> at >>>>>>> >>>>>>> org.apache.clerezza.rdf.jena.sparql.ResultSetWrapper.<init>(ResultSetWrapper.java:39) >>>>>>>> at >>>>>>> >>>>>>> org.apache.clerezza.rdf.jena.sparql.JenaSparqlEngine.execute(JenaSparqlEngine.java:68) >>>>>>>> at >>>>>>> >>>>>>> org.apache.clerezza.rdf.core.access.TcManager.executeSparqlQuery(TcManager.java:272) >>>>>>>> ... >>>>>>>> >>>>>>> -- >>>>>>> | Rupert Westenthaler [email protected] >>>>>>> | Bodenlehenstraße 11 ++43-699-11108907 >>>>>>> | A-5500 Bischofshofen >>>>>>> >>>>> -- >>>>> ir. ing. Minto van der Sluis >>>>> Software innovator / renovator >>>>> Xup BV >>>>> >>>>> Mobiel: +31 (0) 626 014541 >>>>> >>>> >>>> -- >>>> | Rupert Westenthaler [email protected] >>>> | Bodenlehenstraße 11 ++43-699-11108907 >>>> | A-5500 Bischofshofen >>>> >>>> >>> > > > -- > | Rupert Westenthaler [email protected] > | Bodenlehenstraße 11 ++43-699-11108907 > | A-5500 Bischofshofen > > -- ir. ing. Minto van der Sluis Software innovator / renovator Xup BV Mobiel: +31 (0) 626 014541
