Hi Nikos, This will be fixed in 1.9.8, which is the next thing we'll do once 2.1.2 is out, which is soon. The fix is already in our 1.9-maint branch.
-- Chris Vest System Engineer, Neo Technology [ skype: mr.chrisvest, twitter: chvest ] On 09 Jun 2014, at 12:56, Nikos <[email protected]> wrote: > > Hello again, > I am posting an update to this. > We upgraded to 1.9.7; after reading the release notes I had hoped that the > problem might have been fixed, but it is still there. > > > I was able to pinpoint the problem more accurately, in the code that > allocates memory: > In class > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool > and method > boolean allocateNewWindow( BrickElement brick ) > ... > while ( true ) { > > there is a busy-wait loop that expects a lock in a BrickElement to be kept > for a very short time > > > /* > * This is a busy wait, given that rows are kept for a very > short time. What we do is lock the brick so > > * no rows can be mapped over it, then we wait for every row > mapped over it to be released (which can > > * happen because releasing the row does not acquire a lock). > ... > > > */ > > > Unfortunately I am seeing cases where the thread is trapped in the loop > forever.. > Since that thread holds another lock (on a node) already, it is only a > matter of time for threads needing the lock to that node to commit their > transaction > to get blocked and then then we get the 'concertina effect' until the > system becomes unresponsive and needs a hard kill. > > The problem appears only when there is a lot of contention in writing to > the graph. > > I am wondering if it has to do with my MMIO settings... > As per neo4j docs, I am setting those to the datastore file sizes plus 10% > > Any thoughts are welcome! > Thanks > Best Regards > Nk > > > > > On Friday, 21 March 2014 17:04:10 UTC, Michael Hunger wrote: > Hey Nikos, > > sorry for the delay. I talked to the development team and it seems that you > found a bug in our transaction synchronization. > We will fix this issue. A long running read operation shouldn't affect other > operations like that. > > Cheers, > > Michael > > > Am 20.03.2014 um 12:36 schrieb Nikos <[email protected]>: > >> Hi Michael, >> thanks for your swift reply! >> There is a good mix of Java RW transactions and Cypher RW transactions in >> this >> >> After careful study of the thread dumps, I was able to narrow the problem >> down to this: >> The thread holding the TxManager lock was waiting on another lock (an >> instance org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool) held by a >> thread doing what was a long running Cypher query (several minutes)... >> So, all the new Read (or Write) requests coming in waited on the commit of >> TxManager and could not make progress until that other - long running Cypher >> query, had finished. >> >> This query is an aggregation one, that I run periodically to gather some >> stats. Since I have disabled this one, the system runs well. >> >> I have also verified that , basically any long-running query degrades >> performance of simple read queries in the same area of the graph by a factor >> of about a 1000! >> I have annotated some of these queries with @Transactional (Spring), I am >> not sure if that is required for all cases in order not to get 'dirty reads' >> >> I guess maybe this happens since all threads wait on the same instance of >> TxManager, as is described in the comments found in the code: >> >> "..There is some performance degradation related to this, since now we hold >> a lock over commit() for (potentially) all resource managers.." >> >> I did notice that the code works differently in 2.0 >> >> Until then I guess the best strategy is to avoid long-running queries & >> fine grain the transaction boundaries or if you can perhaps advice on a >> better use of the @Transactional annotation? >> >> Many thanks! >> Best Regards >> Nk. >> >> PS. System is Ubuntu, 2.6.32 Kernel >> >> >> >> >> >> >> >> On Tuesday, 18 March 2014 08:21:40 UTC, Michael Hunger wrote: >> Nikos, >> >> Are these Java-code read or write transactions or Cypher read or write TX >> that you see the behavior with? >> >> Michael >> >> Am 17.03.2014 um 11:31 schrieb Nikos <[email protected]>: >> >>> Hello, >>> I am using Neo4j1.9.5 community edition on a test-drive basis to assess >>> its performance and gain experience. >>> I have a graph that is both written to and read from; size is about 10M >>> nodes, 100M relationships, about 14 Gb. >>> >>> In the 1.8 version I was getting a lot of deadlocks, which ended up in >>> countless restarts, but after moving to 1.9.5 things have been much more >>> stable, >>> until recently, that is, where I started getting these BLOCKED threads >>> problem. >>> I dug in the code and it seems that all these thread are waiting on the >>> same instance of org.neo4j.kernel.impl.transaction.TxManager >>> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:344) >>> >>> I have seen another forum post about this but there has been no reply. >>> Any ideas welcome! >>> >>> Thanks! >>> Nk >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Neo4j" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
