This new GC seems to work better actually. Queries that failed before now seem to work. I give it 128 mb. The "tons" were 512 mb. (My graph has 1.3 M nodes and 10.5 M props, 0 rels, the data folder is ~780 mb.) Querying is though considerably slower now when using 128 rather than 512 mb, but I guess this has to do with how much of the graph fits in the given heap, vs how often it needs to go to disk.
Although it works now, perhaps this is only for these particular queries. I am trying to create a rather general server plugin which is supposed to work for a general configuration. I would like to be able to avoid this exception also with my former configuration. Any more ideas on what might be causing this? How are the returned Representations from the plugins handled and GC:d? It seems this could be critical for this kind of exception, see e.g.: http://stackoverflow.com/questions/1393486/what-does-the-error-message-java-lang-outofmemoryerror-gc-overhead-limit-excee > From: david.mon...@neotechnology.com > Date: Tue, 15 Nov 2011 07:50:46 -0800 > To: user@lists.neo4j.org > Subject: Re: [Neo4j] Server plugin running into memory limits > > Hi Anders, > > How much heap are you giving the JVM? When it worked with "tons" of memory, > how much heap did you give? > > Maybe you could also try changing the garbage collector used in > conf/neo4j-wrapper.conf. Just add these lines: > > wrapper.java.additional.20=-XX:+UseConcMarkSweepGC > wrapper.java.additional.21=-verbose:gc > > David > > 2011/11/15 Anders Lindström <andli...@hotmail.com> > > > > > Hi all, > > I'm currently writing a server plugin. I need it to make some specialized > > queries that are not supported by the standard REST API. The important > > methods I expose are 'query' and 'get_next_page', the latter to support > > results pagination (i.e. the plugin is stateful). > > In 'query', I run my query against the Neo4j backend, and store a Node > > iterator to the query results (this is either an iterator originating from > > 'getAllNodes', or a Lucene IndexHits<Node> instance). In 'get_next_page', I > > run through the next N items of the iterator and return these as a > > ListRepresentation. The same iterator object is kept across all page > > retrievals, but of course stepped forward N steps for every invocation. > > After having gone through all pages, the reference to the Node iterator is > > removed. > > Now, as I understand it, all the heap space I should be concerned about > > using, is the one I allocate locally in my methods, since the referenced > > stored to the iterator object is just a tiny reference, and iterator > > results are fetched lazily (i.e., even though the iterator covers a result > > set greater than the allotted heap size, I shall be able to page through it > > within given heap space if the page size is small enough). But when I run > > my plugin, this does not seem to be the case. I can make several successful > > calls in a row to 'get_next_page', but then after a while bump into "GC > > overhead limit exceeded" which I cannot quite understand. I am rather > > certain the size of each page returned is within the allotted heap size. > > For some reason the heap usage seems to grow with the calls to > > 'get_next_page' which I cannot understand, given my understanding of the > > Node iterators from Neo4j. > > How do I avoid hitting this GC overhead limit? Am I missing something? > > (And yes, I've tried using different values of the allowed heap space by > > fiddling in the conf-files, and sure I can give tons of memory to the > > instance, and then it works, but I shouldn't have to give more heap space > > than what Neo4j "needs", plus my page size). > > Thanks! > > Regards,Anders > > > > _______________________________________________ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > David Montag <david.mon...@neotechnology.com> > Neo Technology, www.neotechnology.com > Cell: 650.556.4411 > Skype: ddmontag > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
