Miguel, from your code, it seems that you are trying to read up a lot of relationships from the nodespace into RAM by populating the
Set<String> ltra = new HashSet<String>(); I cannot test the code, but you should probably check the size of that construct. Neo4j itself should degrade performance gracefully if the memory is not sufficient to hold all node in cache. Do some profiling on the different constructs that hold data in memory, I strongly suspect ltra. Cheers, /peter neubauer COO and Sales, Neo Technology GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.tinkerpop.com - Processing for Internet-scale graphs. http://www.thoughtmade.com - Scandinavias coolest Bring-a-Thing party. 2010/3/9 Miguel Ángel Águila <[email protected]>: > Ok, > > here is the final code. First of all, i walk around all the nodes and > edges, and finally I do a Traverse for storing some information. > > //Query 1 > public static void getNodeMaxOutDegreeNeo(final > EmbeddedGraphDatabase neo, > final IndexService indexService) { > Node idNode; > long degree; > long maxDegree = 0; > IndexHits<Node> nodes; > long idDegree; > long idMaxDegree; > Transaction tx = neo.beginTx(); > try { > nodes = indexService.getNodes("TYPE", 1); > Iterator<Node> it = nodes.iterator(); > Node idNodeMaxDegree=it.next(); > Iterator<Relationship> relIterator = > idNodeMaxDegree.getRelationships( > NeoDataBase.MyRelationshipTypes.REF, > Direction.OUTGOING).iterator(); > > while (relIterator.hasNext()) { > relIterator.next(); > maxDegree++; > } > long counter = 0; > while (it.hasNext()) { > if ( ++counter % 50000 == 0 ) { > tx.success(); > tx.finish(); > tx = neo.beginTx(); > System.out.println("1.1 Porta "+counter); > } > idNode = it.next(); > relIterator = > idNode.getRelationships(NeoDataBase.MyRelationshipTypes.REF, > Direction.OUTGOING).iterator(); > degree = 0; > while (relIterator.hasNext()) { > relIterator.next(); > degree++; > } > if (degree >= maxDegree) { > if(degree==maxDegree) { > idDegree=(Long)idNode.getProperty("ID_TITLE"); > idMaxDegree= > (Long)idNodeMaxDegree.getProperty("ID_TITLE"); > if(idDegree>idMaxDegree) { > idNodeMaxDegree = idNode; > } > } > else { > maxDegree = degree; > idNodeMaxDegree = idNode; > } > } > } > > System.out.println("OId = > "+(Long)idNodeMaxDegree.getProperty("ID_TITLE")); > System.out.println("Title = > "+(String)idNodeMaxDegree.getProperty("NAME")); > System.out.println("#refs = "+maxDegree); > //Result: > //OId = 176122 > //Title = List of years > //#refs = 5301 > > //Parte 1.2 > tx.success(); > tx.finish(); > tx = neo.beginTx(); > > > Traverser tr = > idNodeMaxDegree.traverse(Traverser.Order.BREADTH_FIRST, > StopEvaluator.END_OF_GRAPH, > ReturnableEvaluator.ALL, > MyRelationshipTypes.REF, Direction.OUTGOING); > int i=0; > Set<String> ltra = new HashSet<String>(); > String id; > counter=0; > for (Node n: tr) { > if ( ++counter % 50000 == 0 ) { > tx.success(); > tx.finish(); > tx = neo.beginTx(); > System.out.println("1.2 Porta "+counter); > } > relIterator = > n.getRelationships(NeoDataBase.MyRelationshipTypes.REF, > Direction.OUTGOING).iterator(); > while (relIterator.hasNext()) { > id=Long.toString(relIterator.next().getId()); > ltra.add(id); > } > if(i!=0) { > > id=Long.toString(tr.currentPosition().lastRelationshipTraversed().getId()); > if(ltra.contains(id)){ > ltra.remove(id); > } > } > i++; > } > System.out.println("Nodes = "+i); > System.out.println("Arestes = "+(i-1)); > System.out.println("Traversals= "+ltra.size()); > System.out.println("Nivells = > "+tr.currentPosition().depth()); > tx.success(); > } > finally > { > tx.finish(); > } > > > } > > Mike. > > > El mar, 09-03-2010 a las 12:40 +0100, Peter Neubauer escribió: >> Mike, >> great you had configuration working out! Your graph is quite big so an >> exhaustive walk will take some time. Still I suspect that 9h is quite >> long for this. Do you have the code somewhere so we could take a look >> at maybe get performance up a notch? >> >> Cheers, >> >> /peter neubauer >> >> COO and Sales, Neo Technology >> >> GTalk: neubauer.peter >> Skype peter.neubauer >> Phone +46 704 106975 >> LinkedIn http://www.linkedin.com/in/neubauer >> Twitter http://twitter.com/peterneubauer >> >> http://www.neo4j.org - Your high performance graph database. >> http://www.tinkerpop.com - Processing for Internet-scale graphs. >> http://www.thoughtmade.com - Scandinavias coolest Bring-a-Thing party. >> >> >> >> 2010/3/9 Miguel Ángel Águila <[email protected]>: >> > Ok, >> > >> > finally the problem was that I had my neo.props bad configurated. This >> > is the good configuration: >> > >> > neostore.nodestore.db.mapped_memory=913M >> > neostore.relationshipstore.db.mapped_memory=11G >> > neostore.propertystore.db.mapped_memory=50M >> > neostore.propertystore.db.strings.mapped_memory=100M >> > neostore.propertystore.db.arrays.mapped_memory=0M >> > >> > Also I changed the type String for a int and the identifier for a long. >> > It made that the database decrease around 10 gb . >> > >> > Now, walking for all the nodes and edges spends 9 hours more or less. >> > >> > Thank you very much! >> > >> > Mike >> > >> > >> > >> > El vie, 26-02-2010 a las 18:19 +0100, Miguel Angel Aguila escribió: >> >> Ok, thank you. >> >> I've been thinking and I can change the String of node type for an >> >> integer. >> >> I'll change it and some other things and will try to execute another time, >> >> when the trials finishes I will comment you how have it gone. >> >> >> >> Mike >> >> >> >> 2010/2/26 Mattias Persson <[email protected]> >> >> >> >> > If it would be possible to have the "node type" as an integer it'd be >> >> > more space efficient. Also if you just need the "node type" for >> >> > lookups you don't need to set such property on the node, it could be >> >> > enough to just index it. >> >> > >> >> > With those data you provided your string property store should be 28G, >> >> > I'm guessing the rest is String properties on relationships? >> >> > >> >> > 2010/2/26 Miguel Ángel Águila <[email protected]>: >> >> > > Yes, one of them is "node type", but I need because I also use this >> >> > > property for indexing depend on the kind of node. >> >> > > >> >> > > >> >> > > >> >> > > El vie, 26-02-2010 a las 13:11 +0100, Mattias Persson escribió: >> >> > >> 2010/2/26 Miguel Ángel Águila <[email protected]>: >> >> > >> > I will explain me better. >> >> > >> > Every node has: >> >> > >> > - One String that have 5 chars. >> >> > >> > - One String that have 3 chars. >> >> > >> > - One String that have 4 chars. >> >> > >> > - One String that have 100 chars. >> >> > >> Is one of them any kind of "node type"? Because often you can infer a >> >> > >> type from its relationships or surrounding environment, making that >> >> > >> property redundant in a way. (nodes can often represent one or more >> >> > >> logical entities, hence a single type wouldn't suffice and that's why >> >> > >> it isn't built in to the API). >> >> > >> > >> >> > >> > Mike >> >> > >> > >> >> > >> > El vie, 26-02-2010 a las 10:05 +0100, Miguel Ángel Águila Lorente >> >> > >> > escribió: >> >> > >> >> I don't know if I'm answering your question but every node has 4 >> >> > pairs >> >> > >> >> of String, therefore 4 pairs of char(15). >> >> > >> >> >> >> > >> >> Mike >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> El jue, 25-02-2010 a las 07:18 -0700, >> >> > >> >> [email protected] escribió: >> >> > >> >> > Yes, exactly the question. If it's a big string, it is quite >> >> > possible >> >> > >> >> > to have the file grow this large. >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > Miguel, approximately how long is the property value? >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > -------- Original Message -------- >> >> > >> >> > Subject: Re: [Neo] Java outof 64 GB ram >> >> > >> >> > From: Johan Svensson <[email protected]> >> >> > >> >> > Date: Thu, February 25, 2010 7:13 am >> >> > >> >> > To: Neo user discussions <[email protected]> >> >> > >> >> > If you store 322M strings (one string property/relationship), >> >> > were >> >> > >> >> > each string is 200 bytes, the string store will be 60GB+ in >> >> > size. >> >> > >> >> > How large are the strings you are storing? >> >> > >> >> > -Johan >> >> > >> >> > On Thu, Feb 25, 2010 at 2:55 PM, < >> >> > [email protected]> >> >> > >> >> > wrote: >> >> > >> >> > > The string propertystore file seems really, really large >> >> > >> >> > to >> >> > me, >> >> > >> >> > based >> >> > >> >> > > on the # of nodes/relationships/properties. The reason I >> >> > ask is >> >> > >> >> > that I >> >> > >> >> > > saw similar behavior in some of our early testing. >> >> > Extremely large >> >> > >> >> > > string propertystores file. It would be helpful to >> >> > understand why >> >> > >> >> > the >> >> > >> >> > > file gets that large, and whether it is related to some >> >> > >> >> > of >> >> > the >> >> > >> >> > memory >> >> > >> >> > > stress issues. >> >> > >> >> > > >> >> > >> >> > > >> >> > >> >> > > >> >> > >> >> > > >> >> > >> >> > > >> >> > >> >> > > -------- Original Message -------- >> >> > >> >> > > Subject: Re: [Neo] Java outof 64 GB ram >> >> > >> >> > > From: Miguel ngel_guila Lorente <[email protected]> >> >> > >> >> > > Date: Thu, February 25, 2010 1:57 am >> >> > >> >> > > To: Neo user discussions <[email protected]> >> >> > >> >> > > There are 322 million relationships, no 57 million >> >> > relationships. >> >> > >> >> > The >> >> > >> >> > > only propierty I save in the relationship is a String. In >> >> > this 322 >> >> > >> >> > > million relationships there are 3 diferents types of >> >> > relations. >> >> > >> >> > > Do you think that 13 GB is a strange number? Why? >> >> > >> >> > _______________________________________________ >> >> > >> >> > Neo mailing list >> >> > >> >> > [email protected] >> >> > >> >> > [1]https://lists.neo4j.org/mailman/listinfo/user >> >> > >> >> > >> >> > >> >> > References >> >> > >> >> > >> >> > >> >> > 1. https://lists.neo4j.org/mailman/listinfo/user >> >> > >> >> > _______________________________________________ >> >> > >> >> > Neo mailing list >> >> > >> >> > [email protected] >> >> > >> >> > https://lists.neo4j.org/mailman/listinfo/user >> >> > >> >> >> >> > >> >> >> >> > >> >> _______________________________________________ >> >> > >> >> Neo mailing list >> >> > >> >> [email protected] >> >> > >> >> https://lists.neo4j.org/mailman/listinfo/user >> >> > >> > >> >> > >> > >> >> > >> > _______________________________________________ >> >> > >> > Neo mailing list >> >> > >> > [email protected] >> >> > >> > https://lists.neo4j.org/mailman/listinfo/user >> >> > >> > >> >> > >> >> >> > >> >> >> > >> >> >> > > >> >> > > >> >> > > _______________________________________________ >> >> > > Neo mailing list >> >> > > [email protected] >> >> > > https://lists.neo4j.org/mailman/listinfo/user >> >> > > >> >> > >> >> > >> >> > >> >> > -- >> >> > Mattias Persson, [[email protected]] >> >> > Neo Technology, www.neotechnology.com >> >> > _______________________________________________ >> >> > Neo mailing list >> >> > [email protected] >> >> > https://lists.neo4j.org/mailman/listinfo/user >> >> > >> >> _______________________________________________ >> >> Neo mailing list >> >> [email protected] >> >> https://lists.neo4j.org/mailman/listinfo/user >> > >> > >> > _______________________________________________ >> > Neo mailing list >> > [email protected] >> > https://lists.neo4j.org/mailman/listinfo/user >> > >> _______________________________________________ >> Neo mailing list >> [email protected] >> https://lists.neo4j.org/mailman/listinfo/user > > > _______________________________________________ > Neo mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
