Tracked in JENA-321.
On Fri, Sep 14, 2012 at 11:05 AM, Stephen Allen <[email protected]> wrote: > Michael, > > The log actually is very helpful as the stacktrace seems to be the > point where it is using up all the memory (this is not always the > case!). From what I see, I am guessing your have a very large number > of named graphs in your store. > > What appears to be happening is that before the update starts, > UpdateEngineMain attempts to fire notification events to listeners > that an update is about to occur. Unfortunately, it tries to fire an > event for each named graph in the system. Because TDB represents > named graphs as quads, the only way to get a list of all the named > graphs to fire an event for is to perform an entire table scan, > project just the graph part of the quad and then perform a distinct > operation. > > There are a few problems with this approach: > 1) This is pretty dang inefficient, as the entire database is > scanned on every update query > 2) With a large number of named graphs, you have to fire a lot of > events, which is also inefficient > 3) If you have a lot of named graphs, the distinct operation has to > store every graph name in an in-memory hashset > > You are running into issue 3). The underlying cause seems to be a > mismatch in the design of the graph notification. This needs to be > redesigned to fire a single event for the entire graphstore. > > -Stephen > > P.S. Problematic code is in DatasetGraphTDB.java (line 262). > > > On Fri, Sep 14, 2012 at 5:12 AM, Michael Brunnbauer <[email protected]> > wrote: >> >> Hello Andy, >> >> On Fri, Sep 14, 2012 at 12:11:41PM +0100, Andy Seaborne wrote: >>> What I don't understand is where the garbage is coming from. >>> It may be the queries, and not the update. >> >> The queries are on another TDB. I do nothing with the updated TDB except the >> DROP. >> >>> So does the log provide any clues? (Running with -v provides more >>> details - including the updates). >> >> See the attached log. The exception trace at the end may provide hints. >> >> Regards, >> >> Michael Brunnbauer >>
