Re: [Neo4j] OutOfMemory while populating large graph
Great, so maybe neo4j-index should be updated to depend on Lucene 2.9.3. 2010/7/9 Bill Janssen jans...@parc.com Note that a couple of memory issues are fixed in Lucene 2.9.3. Leaking when indexing big docs, and indolent reclamation of space from the FieldCache. Bill Arijit Mukherjee ariji...@gmail.com wrote: I've a similar problem. Although I'm not going out of memory yet, I can see the heap constantly growing, and JProfiler says most of it is due to the Lucene indexing. And even if I do the commit after every X transactions, once the population is finished, the final commit is done, and the graph db closed - the heap stays like that - almost full. An explicit gc will clean up some part, but not fully. Arijit On 9 July 2010 17:00, Mattias Persson matt...@neotechnology.com wrote: 2010/7/9 Marko Rodriguez okramma...@gmail.com Hi, Would it actually be worth something to be able to begin a transaction which auto-committs stuff every X write operation, like a batch inserter mode which can be used in normal EmbeddedGraphDatabase? Kind of like: graphDb.beginTx( Mode.BATCH_INSERT ) ...so that you can start such a transaction and then just insert data without having to care about restarting it now and then? Thats cool! Does that already exist? In my code (like others on the list it seems) I have a counter++ that every 20,000 inserts (some made up number that is not going to throw an OutOfMemory) commits and the reopens a new transaction. Sorta sux. No it doesn't, I just wrote stuff which I though someone could think of as useful. A cool thing with just telling it to do a batch insert mode transaction (not the actual commit interval) is that it could look at how much memory it had to play around with and commit whenever it would be the most efficient, even having the ability to change the limit on the fly if the memory suddenly ran out. Thanks, Marko. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- And when the night is cloudy, There is still a light that shines on me, Shine on until tomorrow, let it be. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Neo4j Tuning for specific application
Hi,
I have a GraphDB with the following attributes:
Number of nodes: 3.6M
Number of relation types: 2
Total size of DB: 9GB
lucene : 160MB
neostore.nodestore.db : 31MB
neostore.propertystore.db : 2GB
neostore.propertystore.db.strings : 4GB
neostore.relationshipstore.db : 1.5GB
Machine characteristics:
vm.dirty_background_ratio = 50
vm.dirty_ratio = 80
OS: Ubuntu x64
CPU: Corei7
MEM: 12GB
The following is our running scenario (The source code is attached):
1. Iterate over all nodes and extract a list of node IDs (fillNodes function).
2. For each node ID, initiate a worker thread that process the following items
(8 threads are executed in parallel using a pool - walk function):
-extract relationships of this node.
-perform a light processing.
-update results (in a ConcurrentHashMap).
Note that:
-The above scenario is iterative. Roughly it runs 10 times.
-No update is applied to the DB during running (read only).
After running the application:
-Less than 4GB/12GB of memory is occupied. It seems that Neo4j is leveraged
only 2GB of memory.
-The hard disk is overloaded.
-Only less than 20% of 8 cores is utilized in average.
Some documents are available in the wiki regarding performance (Performance
Guide, Configuration Settings, Linux Performance Guide). They are so general.
Would you please instruct me to have a better memory map and speed up my
application?
I can benchmark different configurations and reflect the results in the wiki
for
future users.
Kind regards,
Amir
package wikipedia.wnng;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.TimeUnit;
import org.neo4j.graphdb.Direction;
import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.Node;
import org.neo4j.graphdb.Relationship;
import org.neo4j.graphdb.RelationshipType;
import org.neo4j.graphdb.Transaction;
import org.neo4j.index.IndexService;
public class RandomWalker {
public enum NodeRelationshipTypes implements RelationshipType {
go_to, related_to
}
class WalkerThread extends Thread {
private String cui;
public WalkerThread(String str) {
super(str);
cui = str;
}
public void run() {
// create current row of transition matrix
MapString, Double Pi = new HashMapString, Double();
Transaction tx = graphDb.beginTx();
try {
Node current_node = index.getSingleNode(cui,
cui);
for (IteratorRelationship itr =
current_node.getRelationships(NodeRelationshipTypes.go_to,
Direction.INCOMING).iterator();
itr.hasNext();) {
Relationship rel = itr.next();
Node income_node = rel.getStartNode();
Pi.put(income_node.getProperty(cui).toString(),
Double.valueOf(rel.getProperty(weight)
.toString()));
}
tx.success();
} finally {
tx.finish();
}
// calculate v2 = alpha * Pi * v1 + (1 - alpha) * v1
// Pi * v1
Double result = new Double(0.0);
for (String index : Pi.keySet()) {
if (v1.containsKey(index))
result += Pi.get(index) * v1.get(index);
}
// alpha * Pi * v1
result *= alpha;
// (1 - alpha) * v1
Double temp = v1.get(cui);
temp = (temp == null) ? 0.0 : (1 - alpha) * temp;
// result = alpha * Pi * v1 + (1 - alpha) * v1[cui]
result += temp;
// update v2 with new value
v2.put(cui, result);
}
}
private HashSetString nodes;
private final double alpha = 0.5;
private final double epsilon = 0.1;
private ConcurrentMapString, Double v1;
private ConcurrentMapString, Double v2;
private final GraphDatabaseService graphDb;
private final IndexService index;
public RandomWalker() {
v1 = new ConcurrentHashMapString, Double(); // feature vector
1
v2 = new ConcurrentHashMapString, Double(); // feature vector
2
Re: [Neo4j] Neo4j Tuning for specific application
Hi Amir, I'm just starting with neo4j, but saw some issues with your code from a normal java-standpoint. Please note, some of them are just micro-optimizations that may not matter much. But a lot of them are in your critical path, so perhaps they're worth a look. On 10-7-2010 17:59 Amir Hossein Jadidinejad wrote: Hi, I have a GraphDB with the following attributes: Number of nodes: 3.6M Number of relation types: 2 Total size of DB: 9GB lucene : 160MB neostore.nodestore.db : 31MB neostore.propertystore.db : 2GB neostore.propertystore.db.strings : 4GB neostore.relationshipstore.db : 1.5GB Machine characteristics: vm.dirty_background_ratio = 50 vm.dirty_ratio = 80 OS: Ubuntu x64 CPU: Corei7 MEM: 12GB The following is our running scenario (The source code is attached): 1. Iterate over all nodes and extract a list of node IDs (fillNodes function). 2. For each node ID, initiate a worker thread that process the following items (8 threads are executed in parallel using a pool - walk function): -extract relationships of this node. -perform a light processing. -update results (in a ConcurrentHashMap). Note that: -The above scenario is iterative. Roughly it runs 10 times. -No update is applied to the DB during running (read only). After running the application: -Less than 4GB/12GB of memory is occupied. It seems that Neo4j is leveraged only 2GB of memory. What jvm-flags did you specify? I take it, you didn't forget to include a high -Xmx, to allow more memory and perhaps the parallel 'old generation' garbage collector to allow more throughput. Otherwise, most 64-bit jvm's start with system-dependent maximums (afaik at most 2GB). -The hard disk is overloaded. -Only less than 20% of 8 cores is utilized in average. What is your disk doing? Reading, writing, seeking? (see iostat, iotop or similar tools, if its seeking, you see just a few mb/sec reads and no writes). You have actually 4 real cores, the other 4 are just hyperthread-cores which may alleviate some of the work-load if you're cpu-bound. If you're disk-bound, you may actually overwhelm your disk even further with all the additional threads. Some documents are available in the wiki regarding performance (Performance Guide, Configuration Settings, Linux Performance Guide). They are so general. Would you please instruct me to have a better memory map and speed up my application? I can benchmark different configurations and reflect the results in the wiki for future users. Kind regards, Amir Have you checked which parts of your application take a long time? Is it the fillNodes as well as the walk-methods? Or only the walk-variant. A decent profile may be useful. Are you sure you need to first retrieve all nodes, than store the cui-property in a hashset and than re-retrieve that same node via the index? It sounds to me, it should be possible to actually start working on the node right away? Or are your multiple threads (and thus the separate transactions) working against you here? Apart from that, I see a few things that may actually cost a bit of performance: - You're storing unique (?) values in a hashset, to iterate them later. An arraylist is faster for this scenario and uses less memory. - You're boxing and unboxing Double's continuously to and from double's (for instance your 'temp' and 'result'). I don't know how many of these the jvm is able to optimize away, but preventing them to begin with may save a few cpu-cycles per iteration. - You're recalculating 1 - alpha needlessly. - You're using Math.pow rather than diff = v1 - v2; diff_value += diff * diff, the latter has no serious mathematical side-effects (afaik) and should be a bit faster. - You're starting many transactions (for each cui you process), without modifying your graph. I've no idea how heavy these are (relative to the rest of your application), so you may or may not have a need to reduce the amount of transactions. With the above mention of a list it should be relatively easy to adjust your walkerthread to process several cui's rather than just one by using sublist's. - You're retrieving the reference node for each iteration, rather than just once outside the loop in fillNodes. - Why are you converting the weight-property to a string, to then convert it to a Double? If its stored as a string, perhaps it'd be a good idea to change it to a Double? - Perhaps the cui-value can also be stored in a more efficient storage format (long?), thus saving space and memory. - Why are you filling v_star if you're not using the result? Best regards and good luck, Arjen PS, shouldn't a random walk do some random stuff? ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4j Tuning for specific application
Are you using kernel/index version 1.0? Regarding the index lookups (each lookup in its own separate transaction): I think there's a bug in neo4j-index 1.0 which causes such a transaction (which contains a call to index.getNodes) to write stuff to and flush the logical log, which of course is completely unnecessary. That may very well be the cause of the disk being so heavily used. What you could try is to update to latest kernel/index version 1.1-SNAPSHOT where this problem have been fixed, also in that version you aren't forced to wrap reads in transactions. If you cannot update to latest 1.1-SNAPSHOT then try to do more cuis in the each transaction. 2010/7/10 Arjen van der Meijden acmmail...@tweakers.net Hi Amir, I'm just starting with neo4j, but saw some issues with your code from a normal java-standpoint. Please note, some of them are just micro-optimizations that may not matter much. But a lot of them are in your critical path, so perhaps they're worth a look. On 10-7-2010 17:59 Amir Hossein Jadidinejad wrote: Hi, I have a GraphDB with the following attributes: Number of nodes: 3.6M Number of relation types: 2 Total size of DB: 9GB lucene : 160MB neostore.nodestore.db : 31MB neostore.propertystore.db : 2GB neostore.propertystore.db.strings : 4GB neostore.relationshipstore.db : 1.5GB Machine characteristics: vm.dirty_background_ratio = 50 vm.dirty_ratio = 80 OS: Ubuntu x64 CPU: Corei7 MEM: 12GB The following is our running scenario (The source code is attached): 1. Iterate over all nodes and extract a list of node IDs (fillNodes function). 2. For each node ID, initiate a worker thread that process the following items (8 threads are executed in parallel using a pool - walk function): -extract relationships of this node. -perform a light processing. -update results (in a ConcurrentHashMap). Note that: -The above scenario is iterative. Roughly it runs 10 times. -No update is applied to the DB during running (read only). After running the application: -Less than 4GB/12GB of memory is occupied. It seems that Neo4j is leveraged only 2GB of memory. What jvm-flags did you specify? I take it, you didn't forget to include a high -Xmx, to allow more memory and perhaps the parallel 'old generation' garbage collector to allow more throughput. Otherwise, most 64-bit jvm's start with system-dependent maximums (afaik at most 2GB). -The hard disk is overloaded. -Only less than 20% of 8 cores is utilized in average. What is your disk doing? Reading, writing, seeking? (see iostat, iotop or similar tools, if its seeking, you see just a few mb/sec reads and no writes). You have actually 4 real cores, the other 4 are just hyperthread-cores which may alleviate some of the work-load if you're cpu-bound. If you're disk-bound, you may actually overwhelm your disk even further with all the additional threads. Some documents are available in the wiki regarding performance (Performance Guide, Configuration Settings, Linux Performance Guide). They are so general. Would you please instruct me to have a better memory map and speed up my application? I can benchmark different configurations and reflect the results in the wiki for future users. Kind regards, Amir Have you checked which parts of your application take a long time? Is it the fillNodes as well as the walk-methods? Or only the walk-variant. A decent profile may be useful. Are you sure you need to first retrieve all nodes, than store the cui-property in a hashset and than re-retrieve that same node via the index? It sounds to me, it should be possible to actually start working on the node right away? Or are your multiple threads (and thus the separate transactions) working against you here? Apart from that, I see a few things that may actually cost a bit of performance: - You're storing unique (?) values in a hashset, to iterate them later. An arraylist is faster for this scenario and uses less memory. - You're boxing and unboxing Double's continuously to and from double's (for instance your 'temp' and 'result'). I don't know how many of these the jvm is able to optimize away, but preventing them to begin with may save a few cpu-cycles per iteration. - You're recalculating 1 - alpha needlessly. - You're using Math.pow rather than diff = v1 - v2; diff_value += diff * diff, the latter has no serious mathematical side-effects (afaik) and should be a bit faster. - You're starting many transactions (for each cui you process), without modifying your graph. I've no idea how heavy these are (relative to the rest of your application), so you may or may not have a need to reduce the amount of transactions. With the above mention of a list it should be relatively easy to adjust your walkerthread to process several cui's rather than just one by using
Re: [Neo4j] Neo4j with Amazon EC2 Setup
hi, Thanks for the advice, Would it be better to use Amazon S3 or EBS storage to mount the neo4j database? thanks Paddy On Thu, Jul 8, 2010 at 6:24 AM, Dave butlerdi butle...@gmail.com wrote: We have just started putting the NK stuff together as they recently moved to V4 and it is quite a curve for us. (Also not much time due to August release). Have been using the NK stuff for 8 years or so and it a real good platform for handling cache, execution scheduling et al. Real smart guys over there. Over the years we have written many accessors (XMPP, JXTA, JINI (Spaces) and many other protocols as well as ASN1(PER) libraries to replace XML in messaging. Now just doing the Wave, Neo4j and Allegrograph stuff at present. As soon as we have something that is readable we will send a message. You will be able to install from the NK installer and the jars (as is common in NK will contain scripts, docs, code ...). On 8 July 2010 15:18, Peter Neubauer peter.neuba...@neotechnology.com wrote: Dave, sounds very interesting to have a netkernel based REST support for Neo4j. Would be great to get some details on that - is the code accessible somewhere to learn more about it? This might be an interesting programming model even for others ... Cheers, /peter neubauer COO and Sales, Neo Technology GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Thu, Jul 8, 2010 at 2:59 PM, Dave butlerdi butle...@gmail.com wrote: Yep, that is fine. Just using Neo for an RDF store (dense tripple). It works ok but we need to do a lot more work, so still using Allegrograph more (3/4). OWL2 and Reasoner support for dynamic reasoning is really cool for us. . At present we are witing a NetKernel Module for Neo that will give us a very stable and fast neo Rest service as well as provide easy scripting of apps (DPML, Groovy JS, Ruby, PHP, Python ...). Will post when we get done (mid August most likeley). At present Neo is used to store RDF ised messages from barcode and RFID readers in GS1 formats , events are then passed to Robots which handle messaging and update Google Waves. It is all M2M except for dashboard. Sail and all really need better docs. We have worked most of it out and when we finish we will do some docs for internal use. You may also find some use for them. On 8 July 2010 14:41, Peter Neubauer peter.neuba...@neotechnology.com wrote: Dave, you working on the http://pharm2phork.org project? Seems awesome! Are you using the Neo4j RDF components in there? Would be nice to get some feedback on your experience in working with them! I added the project to http://wiki.neo4j.org/content/Neo4j_In_The_Wild#Other if that is ok? Cheers, /peter neubauer COO and Sales, Neo Technology GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Thu, Jul 8, 2010 at 2:30 PM, Dave butlerdi butle...@gmail.com wrote: There was a Cent OS AMI which contained Jetty in the EU West zone. Or else just use a Tomcat instance like ami-45e7002c on US East. Both were fine with Neo4j. You can also just embed jetty and neo into a wrapper and run. On 8 July 2010 14:22, Peter Neubauer peter.neuba...@neotechnology.com wrote: Paddy, yes, this is about the things to consider. I think you should be fine. You might need to tweak some filesystem stuff and the memory mapping settings later, according to the Performance Guidelines, http://wiki.neo4j.org/content/Neo4j_Performance_Guide . Let us know how things progress for you! Cheers, /peter neubauer COO and Sales, Neo Technology GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Thu, Jul 8, 2010 at 3:25 AM, Paddy paddyf...@gmail.com wrote: Hi, I'm trying to configure neo4j to run on an amazon ec2 instance. This is my first time using ec2, I'd like to deploy a neo4j web app war file to Jetty. So far I have been reading
Re: [Neo4j] Neo4j with Amazon EC2 Setup
also can i load the configurations from the neo4j_config.props file if the GraphDatabaseService is injected by Spring? The Configuration Settings wiki outlines the setup as : MapString,String configuration = EmbeddedGraphDatabase.loadConfigurations( neo4j_config.props ); GraphDatabaseService graphDb = new EmbeddedGraphDatabase( my-neo4j-db/, configuration ); my graphDbService is configured the in app-config.xml: bean id=graphDbService class=org.neo4j.kernel.EmbeddedGraphDatabase init-method=enableRemoteShell destroy-method=shutdown constructor-arg index=0 value=/home/neo/var/neo4j-db/ /bean and auto-wired using the @Autowired annotation: @Autowired private GraphDatabaseService graphDbService; Thanks a lot Paddy On Sat, Jul 10, 2010 at 3:38 PM, Paddy paddyf...@gmail.com wrote: hi, Thanks for the advice, Would it be better to use Amazon S3 or EBS storage to mount the neo4j database? thanks Paddy On Thu, Jul 8, 2010 at 6:24 AM, Dave butlerdi butle...@gmail.com wrote: We have just started putting the NK stuff together as they recently moved to V4 and it is quite a curve for us. (Also not much time due to August release). Have been using the NK stuff for 8 years or so and it a real good platform for handling cache, execution scheduling et al. Real smart guys over there. Over the years we have written many accessors (XMPP, JXTA, JINI (Spaces) and many other protocols as well as ASN1(PER) libraries to replace XML in messaging. Now just doing the Wave, Neo4j and Allegrograph stuff at present. As soon as we have something that is readable we will send a message. You will be able to install from the NK installer and the jars (as is common in NK will contain scripts, docs, code ...). On 8 July 2010 15:18, Peter Neubauer peter.neuba...@neotechnology.com wrote: Dave, sounds very interesting to have a netkernel based REST support for Neo4j. Would be great to get some details on that - is the code accessible somewhere to learn more about it? This might be an interesting programming model even for others ... Cheers, /peter neubauer COO and Sales, Neo Technology GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Thu, Jul 8, 2010 at 2:59 PM, Dave butlerdi butle...@gmail.com wrote: Yep, that is fine. Just using Neo for an RDF store (dense tripple). It works ok but we need to do a lot more work, so still using Allegrograph more (3/4). OWL2 and Reasoner support for dynamic reasoning is really cool for us. . At present we are witing a NetKernel Module for Neo that will give us a very stable and fast neo Rest service as well as provide easy scripting of apps (DPML, Groovy JS, Ruby, PHP, Python ...). Will post when we get done (mid August most likeley). At present Neo is used to store RDF ised messages from barcode and RFID readers in GS1 formats , events are then passed to Robots which handle messaging and update Google Waves. It is all M2M except for dashboard. Sail and all really need better docs. We have worked most of it out and when we finish we will do some docs for internal use. You may also find some use for them. On 8 July 2010 14:41, Peter Neubauer peter.neuba...@neotechnology.com wrote: Dave, you working on the http://pharm2phork.org project? Seems awesome! Are you using the Neo4j RDF components in there? Would be nice to get some feedback on your experience in working with them! I added the project to http://wiki.neo4j.org/content/Neo4j_In_The_Wild#Other if that is ok? Cheers, /peter neubauer COO and Sales, Neo Technology GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Thu, Jul 8, 2010 at 2:30 PM, Dave butlerdi butle...@gmail.com wrote: There was a Cent OS AMI which contained Jetty in the EU West zone. Or else just use a Tomcat instance like ami-45e7002c on US East. Both were fine with Neo4j. You can also just embed jetty and neo into a wrapper and run. On 8 July 2010 14:22, Peter Neubauer peter.neuba...@neotechnology.com wrote: Paddy, yes, this is about the things to consider. I think you should be fine. You might need to tweak some filesystem stuff and the memory mapping settings later, according to the Performance Guidelines,
