Re: [Neo4j] Neo4j Java API: GC overhead limit exceeded after running multiple queries

[Clark Richey]

At a brief glance I think you issue might be related to the fact that neo 
doesn’t support nested transactions. So, while you are committing txQ 
EVERYTHING is happening inside the single txG transaction. This is taking up a 
lot of memory to hold the entire transaction state.



> On Mar 8, 2016, at 9:19 PM, Mohammad Hossain Namaki  
> wrote:
> 
> I’m using Neo4j 2.3.0 in Java. I have 16 GB RAM, running the code on MAC OSX 
> Laptop, using "-Xmx12g -Xms12g" as VM arguments.
> 
> I’ve encountered a “GC overhead limit exceeded” problem in Neo4j Java API.
> 
> 
> 
> In order to do experiments with lots of queries, I have a program which opens 
> a transaction over different query.db's and get the answers of that from my 
> own framework which is wrapped in an object (It runs a query and print its 
> running time in a file).
> 
> 
> 
> So, for running the query, I don’t use Cypher.
> 
> 
> 
> For each query I open two transactions over a query.db and a data.db, 
> initialize my framework and run it. The memory usage slightly increases and 
> the “GC overhead” finally happens.
> 
> 
> 
> try (Transaction txG = knowledgeGraph.beginTx()) {
>  try (Transaction txQ = queryGraph.beginTx()) {
>   MyObj myFramework = new MyObj();
>   printTheResultsIntoTheFile(framework.run());
>   myFramework =null;
>   txQ.success();
>   txQ.close();
> 
> 
> These are some of my endeavors to get rid of this error:
> 
> 1. After I’ve used a monitoring program to dump the heap, I’ve found that 
> there is some problem with this 
> “org.neo4j.io.pagecache.impl.muninn.MuninnPageCache” So, I’ve tried to set 
> the page cache size and limit it to a small value:
> 
> dataGraph = new 
> GraphDatabaseFactory().newEmbeddedDatabaseBuilder(MODELGRAPH_DB_PATH)
> .setConfig(GraphDatabaseSettings.pagecache_memory, 
> "500M").newGraphDatabase();
> 
> However, still the "memory leakage” problem exists.
> 
> 
> 
> 2. After tx.success(), I called the tx.close() to make sure that it doesn’t 
> use the memory.
> 
> 3. After using my framework(object) to find the answers of a query, I 
> explicitly set it to null. topkFramework=null;
> 
> 4. I called System.gc(); and System.runFinalization();
> 
> 5. I changed all of my static variables like MyCacheServer or 
> MyNeighborIndexer to non-static ones and in each query, I made them clear, 
> and explicitly set them to null.
> 
> queryNodeIdSet.clear();
> queryNodeIdSet = null;
> queryNodeIdSet = new HashSet();
> 
> I've used MemoryAnalyzer (MAT) for eclipse. The biggest object is 
> org.neo4j.io.pagecache.impl.muninn.MuninnPage which got 3.54 GB of memory. 
> Other objects are ok in my case because of some nodes/relationships id 
> indexing. (The dataset itself is 3.15 GB, I really don't want this much of 
> caching)
> Could you please help me on this? I've asked this question 
> 
>  in stackoverflow also.
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to neo4j+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Neo4j] Neo4j Java API: GC overhead limit exceeded after running multiple queries

[Mohammad Hossain Namaki]

I’m using *Neo4j 2.3.0* in *Java*. I have 16 GB RAM, running the code on 
MAC OSX Laptop, using "*-Xmx12g -Xms12g*" as VM arguments.

I’ve encountered a “GC overhead limit exceeded” problem in Neo4j Java API.


In order to do experiments with lots of queries, I have a program which 
opens a transaction over different query.db's and get the answers of that 
from my own framework which is wrapped in an object (It runs a query and 
print its running time in a file).


So, for running the query, I don’t use *Cypher*.


For each query I open *two transactions* over a query.db and a data.db, 
initialize my framework and run it. The memory usage slightly increases and 
the “GC overhead” finally happens.


try (Transaction txG = knowledgeGraph.beginTx()) {
 try (Transaction txQ = queryGraph.beginTx()) {
  MyObj myFramework = new MyObj();
  printTheResultsIntoTheFile(framework.run());
  myFramework =null;
  txQ.success();
  txQ.close();


*These are some of my endeavors to get rid of this error:*

   1. 
   
   1. After I’ve used a monitoring program to dump the heap, I’ve found 
   that there is some problem with this “
   *org.neo4j.io.pagecache.impl.muninn.MuninnPageCache*” So, I’ve tried to 
   set the page cache size and limit it to a small value:
   2. 
  1. 
  
  dataGraph = new 
  GraphDatabaseFactory().newEmbeddedDatabaseBuilder(MODELGRAPH_DB_PATH) 
  .setConfig(GraphDatabaseSettings.pagecache_memory, 
  "500M").newGraphDatabase();
  
   However, still the "memory leakage” problem exists.
   1. 
  
  
  2. 
  
  2. After tx.success(), I called the tx.close() to make sure that it 
  doesn’t use the memory.
  3. 
  
  3. After using my framework(object) to find the answers of a query, I 
  explicitly set it to null. topkFramework=null;
  4. 
  
  4. I called System.gc(); and System.runFinalization();
  5. 
  
  5. I changed all of my static variables like MyCacheServer or 
  MyNeighborIndexer to non-static ones and in each query, I made them 
clear, 
  and explicitly set them to null.
  
  queryNodeIdSet.clear(); queryNodeIdSet = null; queryNodeIdSet = new 
  HashSet();
  3. I've used MemoryAnalyzer (MAT) for eclipse. The biggest object is 
   *org.neo4j.io.pagecache.impl.muninn.MuninnPage* which got 3.54 GB of 
   memory. Other objects are ok in my case because of some nodes/relationships 
   id indexing. (The dataset itself is 3.15 GB, I really don't want this much 
   of caching)
   
Could you please help me on this? I've asked this question 

 
in stackoverflow also.


-- 
You received this message because you are subscribed to the Google Groups 
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.