Re: [Neo4j] Stumped by performance issue in traversal - would take a month to run!
Good Jeff, let the community know how that goes! Cheers, /peter neubauer COO and Sales, Neo Technology GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Sat, Aug 7, 2010 at 6:40 AM, Jeff Klann wrote: > To anyone interested, after reading Martin's e-mail I ran a test of a small > (75mb) Neo4J graph and traversed it (~500k edges) with memory mapping turned > off in about 1.5 minutes on my hard drive and 30sec when copied to a > (probably slow) flash drive. Which incidentally is the same time it took > with memory mapping turned on with the HDD. Very nice. I think I will pick > up a larger flash drive (or SSD if I can find one inexpensively) and try it > out on a more sizable graph. > > Thanks, > Jeff Klann > > On Thu, Aug 5, 2010 at 7:13 PM, Martin Neumann > wrote: > >> > > >> > > - Martin, I'm confused a bit about SSDs. I read up on them after I read >> > > your >> > > post. You said flash drives are best, but I read that even the highest >> > > performing flash drives are about 30MB/s read, whereas modern hard >> drives >> > > are at least 50MB/s. True SSDs claim to be 50MB/s too but they're quite >> > > expensive. So why is a flash drive best? I could definitely spring for >> > one >> > > big enough to hold my db if it'd help a lot, but it has that slower >> read >> > > speed. Does the faster seek time really make that much of a difference? >> > Any >> > > brands you'd recommend? >> > > >> >> Neo4j stores the data as Graph on HD. >> >> An example: e = (n1,n2) >> e at location 1000 >> n1 at location 1 >> n2 at location 5 >> >> A traversal, assuming nothing is cached, would result in moving the head to >> 1 then to 1000 then back to 5. >> Normal HD take a while to move to the locations before it can start to read >> data. SSD does not have these delays. If you read little data that is >> spread >> widely over the storage, like in a traversal, SSD are much faster then HD >> even if they are slower to retrieve the data. >> I don't have performance data on that myself but I heard rumors of around >> 20-40 times speedup. >> >> cheers Martin >> >> >> On Thu, Aug 5, 2010 at 9:02 PM, Jeff Klann wrote: >> >> > Thanks for the answers. >> > >> > Yes, I can do online updates of the datastore, but while this is in R&D I >> > will need to rerun the main loop when I change the algorithm and just for >> > personal benefit I don't want to wait hours to see the changes. Seems to >> be >> > running acceptably now, though. However, I haven't benchmarked it against >> > doing JOINS in Postgres. Are there any good performance stats out there? >> > The >> > speed is about the same as I'd expect from SQL. >> > >> > The graph will probably be nearly a complete graph in the end. The edges >> > between orders will eventually store various stats on the relationships >> > between pairs of items. It'd be nice if I can query an index for outgoing >> > edges from nodes with certain properties. Is this possible? I'll have a >> > look >> > at the edge indexer component. >> > >> > Thanks, >> > Jeff Klann >> > >> > On Mon, Aug 2, 2010 at 2:40 PM, David Montag < >> > david.mon...@neotechnology.com >> > > wrote: >> > >> > > Hi Jeff, >> > > >> > > Please see answers below. >> > > >> > > On Mon, Aug 2, 2010 at 5:47 PM, Jeff Klann wrote: >> > > >> > > > Thank you all for your continued interest in helping me. I tweaked >> the >> > > code >> > > > more to minimize writes to the database and it now looks like: >> > > > For each item A >> > > > For each customer that purchased A >> > > > For each item B (with id>A) that A purchased >> > > > Increment (in memory) the weight of (A-B) >> > > > Write out the edges [(A-B):weight] to disk and clear the in-memory >> > map >> > > > >> > > > This actually (if I'm not mistaken) covers all relationships and does >> > > 7500 >> > > > items in about 45 minutes! Not too bad, especially due to (I think) >> > > > avoiding >> > > > edge-checking, and I think it's usable for my application, though >> it's >> > > > still >> > > > ~200k traversals/sec on average, which is a few times slower than I >> > > hoped. >> > > > I >> > > > doubt that's much faster than a two-table join in SQL, though deeper >> > > > traversals should show benefits. >> > > > >> > > >> > > Do you need to do this computation on the full graph all the time? >> Maybe >> > it >> > > would be enough to do it once, and then update it when a customer buys >> > > something? Usually, high one-time costs can be tolerated, and with >> Neo4j >> > > you >> > > can actually do the updating for a customer performing a purchase at >> > > runtime >> > > without performance problems. >> > > >> > > >> > > > >> > > > - David, thank you for your answers on traversers vs
Re: [Neo4j] Best way to visualize?
Jeff, it should be relatively easy to use the JUNG visualizations. However, I think the adapter in Blueprints is trying to project the multilabel Neo4j graph onto a unlabeled JUNG graph, thus removing information, since JUNG seems only to handle unlabeled graphs. Cheers, /peter neubauer COO and Sales, Neo Technology GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Sat, Aug 7, 2010 at 6:36 AM, Jeff Klann wrote: > Oh, interesting, didn't realize it did visualization. I'll check it out but > is it relatively easy to use Jung visualizations on a Neo4J database through > Blueprints? > > Thanks, > Jeff Klann > > On Thu, Aug 5, 2010 at 6:36 PM, Paul A. Jackson wrote: > >> There is also Jung. >> http://jung.sourceforge.net/ >> >> >> -Original Message- >> From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] >> On Behalf Of Jeff Klann >> Sent: Thursday, August 05, 2010 3:15 PM >> To: Neo4j user discussions >> Subject: [Neo4j] Best way to visualize? >> >> Hi all, there have been snippets of discussion on this but not a full >> answer. I'm looking for the best way to quickly pull up part of my graph to >> visualize. Optimally I'd like to visualize programmatically - e.g., write a >> little code that would throw up some subset of nodes and edges on the >> screen >> in a nicely laid-out way. The tools I've found: >> >> * Neoclipse: Seems super-slow and I can't look up a node either in an index >> or by ID, which makes it basically unusable for me. The search function, >> whatever it does, just runs and runs. >> * iGraph: Looks like a cool tool, but I can't figure out a way to interact >> with it in the JVM. Couldn't even figure out how to install it in Jython. >> * Gephi: Way too buggy. Looks like it will be cool eventually. The alpha >> version with the half-finished Neo4J import didn't get all my edges >> properly, and even after that most things worked erratically. The output >> looks awesome, though. >> * Cytoscape: Actually the best thing I've come across, though a little >> heavyweight for just throwing some stuff on the screen and I haven't >> learned >> the API well enough to get it to interact directly with my Neo4J code. It >> wasn't too hard to export part of my graph to XGMML via the Cytoscape API >> and then import it into the Cytoscape GUI, though it does seem an >> unnecessary extra step. >> >> So can anyone help with: >> - Neoclipse, can I look up a node by index value or at least by ID? >> - iGraph, how to integrate with the JVM? >> - Cytoscape, how can I tighten my integration (rather than through XGMML)? >> - Is there a tool I've missed? >> >> Thanks! >> Jeff Klann >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Stumped by performance issue in traversal - would take a month to run!
To anyone interested, after reading Martin's e-mail I ran a test of a small (75mb) Neo4J graph and traversed it (~500k edges) with memory mapping turned off in about 1.5 minutes on my hard drive and 30sec when copied to a (probably slow) flash drive. Which incidentally is the same time it took with memory mapping turned on with the HDD. Very nice. I think I will pick up a larger flash drive (or SSD if I can find one inexpensively) and try it out on a more sizable graph. Thanks, Jeff Klann On Thu, Aug 5, 2010 at 7:13 PM, Martin Neumann wrote: > > > > > > - Martin, I'm confused a bit about SSDs. I read up on them after I read > > > your > > > post. You said flash drives are best, but I read that even the highest > > > performing flash drives are about 30MB/s read, whereas modern hard > drives > > > are at least 50MB/s. True SSDs claim to be 50MB/s too but they're quite > > > expensive. So why is a flash drive best? I could definitely spring for > > one > > > big enough to hold my db if it'd help a lot, but it has that slower > read > > > speed. Does the faster seek time really make that much of a difference? > > Any > > > brands you'd recommend? > > > > > Neo4j stores the data as Graph on HD. > > An example: e = (n1,n2) > e at location 1000 > n1 at location 1 > n2 at location 5 > > A traversal, assuming nothing is cached, would result in moving the head to > 1 then to 1000 then back to 5. > Normal HD take a while to move to the locations before it can start to read > data. SSD does not have these delays. If you read little data that is > spread > widely over the storage, like in a traversal, SSD are much faster then HD > even if they are slower to retrieve the data. > I don't have performance data on that myself but I heard rumors of around > 20-40 times speedup. > > cheers Martin > > > On Thu, Aug 5, 2010 at 9:02 PM, Jeff Klann wrote: > > > Thanks for the answers. > > > > Yes, I can do online updates of the datastore, but while this is in R&D I > > will need to rerun the main loop when I change the algorithm and just for > > personal benefit I don't want to wait hours to see the changes. Seems to > be > > running acceptably now, though. However, I haven't benchmarked it against > > doing JOINS in Postgres. Are there any good performance stats out there? > > The > > speed is about the same as I'd expect from SQL. > > > > The graph will probably be nearly a complete graph in the end. The edges > > between orders will eventually store various stats on the relationships > > between pairs of items. It'd be nice if I can query an index for outgoing > > edges from nodes with certain properties. Is this possible? I'll have a > > look > > at the edge indexer component. > > > > Thanks, > > Jeff Klann > > > > On Mon, Aug 2, 2010 at 2:40 PM, David Montag < > > david.mon...@neotechnology.com > > > wrote: > > > > > Hi Jeff, > > > > > > Please see answers below. > > > > > > On Mon, Aug 2, 2010 at 5:47 PM, Jeff Klann wrote: > > > > > > > Thank you all for your continued interest in helping me. I tweaked > the > > > code > > > > more to minimize writes to the database and it now looks like: > > > > For each item A > > > > For each customer that purchased A > > > > For each item B (with id>A) that A purchased > > > > Increment (in memory) the weight of (A-B) > > > > Write out the edges [(A-B):weight] to disk and clear the in-memory > > map > > > > > > > > This actually (if I'm not mistaken) covers all relationships and does > > > 7500 > > > > items in about 45 minutes! Not too bad, especially due to (I think) > > > > avoiding > > > > edge-checking, and I think it's usable for my application, though > it's > > > > still > > > > ~200k traversals/sec on average, which is a few times slower than I > > > hoped. > > > > I > > > > doubt that's much faster than a two-table join in SQL, though deeper > > > > traversals should show benefits. > > > > > > > > > > Do you need to do this computation on the full graph all the time? > Maybe > > it > > > would be enough to do it once, and then update it when a customer buys > > > something? Usually, high one-time costs can be tolerated, and with > Neo4j > > > you > > > can actually do the updating for a customer performing a purchase at > > > runtime > > > without performance problems. > > > > > > > > > > > > > > - David, thank you for your answers on traversers vs. > getRelationships > > > and > > > > on property-loading. I imported some properties I don't really need, > > > > perhaps > > > > if I delete them it'll speed things up? Also I'm using the old > > > > Node.traverse(). How is the new framework better? I expect it has a > > nicer > > > > syntax, which I would like to try, but does it improve performance > too? > > > > > > > > > > Well, depending on your setup you should be able to theoretically > improve > > > performance compared to the old traversal framework. The old framework > > > keeps > > > track of visited nodes, so that you don't traverse to the same
Re: [Neo4j] Best way to visualize?
Oh, interesting, didn't realize it did visualization. I'll check it out but is it relatively easy to use Jung visualizations on a Neo4J database through Blueprints? Thanks, Jeff Klann On Thu, Aug 5, 2010 at 6:36 PM, Paul A. Jackson wrote: > There is also Jung. > http://jung.sourceforge.net/ > > > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > On Behalf Of Jeff Klann > Sent: Thursday, August 05, 2010 3:15 PM > To: Neo4j user discussions > Subject: [Neo4j] Best way to visualize? > > Hi all, there have been snippets of discussion on this but not a full > answer. I'm looking for the best way to quickly pull up part of my graph to > visualize. Optimally I'd like to visualize programmatically - e.g., write a > little code that would throw up some subset of nodes and edges on the > screen > in a nicely laid-out way. The tools I've found: > > * Neoclipse: Seems super-slow and I can't look up a node either in an index > or by ID, which makes it basically unusable for me. The search function, > whatever it does, just runs and runs. > * iGraph: Looks like a cool tool, but I can't figure out a way to interact > with it in the JVM. Couldn't even figure out how to install it in Jython. > * Gephi: Way too buggy. Looks like it will be cool eventually. The alpha > version with the half-finished Neo4J import didn't get all my edges > properly, and even after that most things worked erratically. The output > looks awesome, though. > * Cytoscape: Actually the best thing I've come across, though a little > heavyweight for just throwing some stuff on the screen and I haven't > learned > the API well enough to get it to interact directly with my Neo4J code. It > wasn't too hard to export part of my graph to XGMML via the Cytoscape API > and then import it into the Cytoscape GUI, though it does seem an > unnecessary extra step. > > So can anyone help with: > - Neoclipse, can I look up a node by index value or at least by ID? > - iGraph, how to integrate with the JVM? > - Cytoscape, how can I tighten my integration (rather than through XGMML)? > - Is there a tool I've missed? > > Thanks! > Jeff Klann > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] neo4j REST server configuration
Hi, I'm running the standalone neo4j REST server on a 64 bit linux machine with 64GB RAM and am trying to configure the following memory settings through the wrapper.conf file: wrapper.java.initmemory=16144 wrapper.java.maxmemory=16144 However when I restart the server, JMX shows me the following VM arguments: -Dcom.sun.management.jmxremote -Xms4096m -Xmx4096m -Djava.library.path=lib -Dwrapper.key=q8W6vP8LS9mj0ekz -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=27943 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 Another unrelated issue is that JMX Mbeans shows configuration attributes as unavailable when I attach to the REST wrapper. The reason I am looking into modifying the configuration is that my client servers seem to be timing out. The server cannot handle more than 5 concurrent transactions, so I want to tweak the heap size and see if that helps. Thanks, ~Mohit ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] LuceneIndexService: NoSuchMethodError
For future reference, if anybody else is in the situation I was in,
once solution is to package up your library with dependency classes
rolled up into the jar file.
The following added to the POM.xml under plugins accomplishes this:
maven-assembly-plugin
package
attached
jar-with-dependencies
Alex
On Thu, Aug 5, 2010 at 8:25 PM, Alex D'Amour wrote:
> Hi all,
>
> I'm getting this same error in an environment where a class I
> implemented using Neo4j and the LuceneIndexService is called by an
> application that's using Lucene 3.0. The application server is
> unfortunately above my abstraction layer (I'm just implementing the
> back end in neo4j), so I can't change the version of lucene that it's
> including. Previous messages have suggested that the indexing
> component should work using Lucene 3.0, but is there an easy way for
> me to remvoe this version conflict without manually editing the
> pom.xml in the indexing component source?
>
> Thanks,
> Alex
>
> On Mon, Aug 2, 2010 at 11:07 AM, Peter Neubauer
> wrote:
>> Max,
>> since you are using neo4j-index, you should not be importing Lucene
>> again, since it already is a dependency of the index components (and I
>> think the version is higher there).
>>
>> So, upgrading to 1.1 and removing the Lucene dependency should fix it:
>>
>>
>> org.neo4j
>> neo4j-kernel
>> 1.1
>>
>>
>> org.neo4j
>> neo4j-index
>> 1.1
>>
>>
>> Cheers,
>>
>> /peter neubauer
>>
>> COO and Sales, Neo Technology
>>
>> GTalk: neubauer.peter
>> Skype peter.neubauer
>> Phone +46 704 106975
>> LinkedIn http://www.linkedin.com/in/neubauer
>> Twitter http://twitter.com/peterneubauer
>>
>> http://www.neo4j.org - Your high performance graph database.
>> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.
>>
>>
>>
>> On Mon, Aug 2, 2010 at 3:11 PM, Max Jakob wrote:
>>> Hi Peter,
>>>
this sounds like a version clash on Lucene. Can you check what
version(s) of Lucene (and Neo4j-Index) you are running in the two
scenarios?
>>>
>>> That would make sense to me as well. But like I said, on the first
>>> run, the method is found. Running the exact same code a second time,
>>> without any changes, it complains that the method is not found. (?!)
>>>
>>> Here the versions I use (for both runs) from my pom.xml:
>>>
>>> org.neo4j
>>> neo4j-kernel
>>> 1.1-SNAPSHOT
>>>
>>>
>>> org.neo4j
>>> neo4j-index
>>> 1.1-SNAPSHOT
>>>
>>>
>>> org.apache.lucene
>>> lucene-highlighter
>>> 2.9.1
>>>
>>>
>>> Cheers,
>>> Max
>>>
>>>
>>> On Mon, Aug 2, 2010 at 3:01 PM, Peter Neubauer
>>> wrote:
Max,
this sounds like a version clash on Lucene. Can you check what
version(s) of Lucene (and Neo4j-Index) you are running in the two
scenarios?
Cheers,
/peter neubauer
COO and Sales, Neo Technology
GTalk: neubauer.peter
Skype peter.neubauer
Phone +46 704 106975
LinkedIn http://www.linkedin.com/in/neubauer
Twitter http://twitter.com/peterneubauer
http://www.neo4j.org - Your high performance graph database.
http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.
On Mon, Aug 2, 2010 at 2:38 PM, Max Jakob wrote:
> Hi,
>
> I have a problem with the LuceneIndexService. When I create an indexed
> graph base and I commit it to disk, next time I want to use it, I get
> a NoSuchMethodError for LuceneIndexService.getSingleNode:
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.lucene.search.IndexSearcher.search(Lorg/apache/lucene/search/Query;)Lorg/apache/lucene/search/Hits;
> at
> org.neo4j.index.lucene.LuceneIndexService.searchForNodes(LuceneIndexService.java:430)
> at
> org.neo4j.index.lucene.LuceneIndexService.getNodes(LuceneIndexService.java:310)
> at
> org.neo4j.index.lucene.LuceneIndexService.getSingleNode(LuceneIndexService.java:469)
> at
> org.neo4j.index.lucene.LuceneIndexService.getSingleNode(LuceneIndexService.java:461)
>
> To illustrate this in more detail: if I run the code below for the
> first time, everything goes fine. On a second run I get the exception.
> Could somebody give me a hint where I'm going wrong? (re-indexing does
> not work) Do I have to initialize the LuceneIndexService differently
> if the index is already on disk?
>
> Thanks in advance for any advice you have,
> Max
>
>
> GraphDatabaseService graphDb = new EmbeddedGraphDatabase("gbDir");
> IndexService index = new LuceneIndex
Re: [Neo4j] Best way to visualize?
Hi all, as for Cytoscape and Gephi - a very useful feature which I don't know if any of these tools can support would be to be able to define a "radius", that is, a filter that reaches out for instance depth 2, filtering certain relatoinships, from a set of start nodes, and only load that. that would make it possible to move around in very big graphs without having to load all data and take full advantage of the underlying store like Neo4j and the limited visualization load on the VizTool? Cheers, /peter neubauer COO and Sales, Neo Technology GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Thu, Aug 5, 2010 at 11:28 PM, Keiichiro Ono wrote: > Hi Jeff. > >> * Cytoscape: Actually the best thing I've come across, though a little >> heavyweight for just throwing some stuff on the screen and I haven't learned >> the API well enough to get it to interact directly with my Neo4J code. It >> wasn't too hard to export part of my graph to XGMML via the Cytoscape API >> and then import it into the Cytoscape GUI, though it does seem an >> unnecessary extra step. > > You can write a small plugin code to do this. If you simply want to > visualize subgraph of the gigantic Neo4j backend graph, you can do the > following: > > 1. Fetch set of nodes/edges. If you use Blueprints, create Set of > Vertices/Edges you want to visualize. > 2. Create a CyNetwork object (Cytoscape.createNetwork()), then build > CyNode/CyEdge objects from the set created in step1. > 3. Add CyNode and CyEdge to the CyNetwork and perform layout to the view. > > >> - Cytoscape, how can I tighten my integration (rather than through XGMML)? > > I've not released yet, but I'm also working on simple plugin to > visualize Neo4j data in Cytoscape. It's my hobby project (at least > for now), so I cannot tell the exact date of release, but if you have > questions please let us know. (we have Google Groups for the > users/developers). > > Thanks, > Kei > Cytoscape Core Developer > > >> - Is there a tool I've missed? >> >> Thanks! >> Jeff Klann >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> > > > > -- > Keiichiro Ono http://www.keiono.net/ > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
