Why not use a graph based index? Properly structured it should provide fast reads all the time (property structured kind-of means you have the most likely reads in cache, which is dependant on the type of data).
On Tue, Mar 23, 2010 at 10:56 AM, Peter Neubauer <[email protected]>wrote: > Awh, > kinda not very fast inserting then. From looking at the code, this > kinda sux for fast inserting since you need to go in and out of > indexes. Not sure how to speed that up. Could you keep some of the > index in memory maybe? > > Cheers, > > /peter neubauer > > COO and Sales, Neo Technology > > GTalk: neubauer.peter > Skype peter.neubauer > Phone +46 704 106975 > LinkedIn http://www.linkedin.com/in/neubauer > Twitter http://twitter.com/peterneubauer > > http://www.neo4j.org - Your high performance graph database. > http://www.tinkerpop.com - Processing for Internet-scale graphs. > http://www.thoughtmade.com - Scandinavias coolest Bring-a-Thing party. > > > > On Tue, Mar 23, 2010 at 10:49 AM, Laurent Laborde <[email protected]> > wrote: > > nope. > > "The LuceneIndexBatchInserter is designed for being performant when > > inserting large amounts of data with minimal lookups from the index > > during that time." > > > > problem : i do a lot of lookup while inserting :) > > > > -- > > Ker2x > > > > > > On Tue, Mar 23, 2010 at 10:43 AM, Peter Neubauer > > <[email protected]> wrote: > >> Laurent, > >> have you looked at the BatchInserter for doing big initial > >> populations? It is MUCH faster than the normal Neo4j transactional > >> approach, see http://wiki.neo4j.org/content/Indexing_with_BatchInserter > >> > >> Would that help? > >> > >> Cheers, > >> > >> /peter neubauer > >> > >> COO and Sales, Neo Technology > >> > >> GTalk: neubauer.peter > >> Skype peter.neubauer > >> Phone +46 704 106975 > >> LinkedIn http://www.linkedin.com/in/neubauer > >> Twitter http://twitter.com/peterneubauer > >> > >> http://www.neo4j.org - Your high performance graph > database. > >> http://www.tinkerpop.com - Processing for Internet-scale graphs. > >> http://www.thoughtmade.com - Scandinavias coolest Bring-a-Thing party. > >> > >> > >> > >> On Mon, Mar 22, 2010 at 11:35 PM, Laurent Laborde <[email protected]> > wrote: > >>> On Fri, Mar 19, 2010 at 10:20 AM, Laurent Laborde <[email protected]> > wrote: > >>>> thank you peter. > >>>> Following the various link, i found http://www.cytoscape.org/ > >>>> Look very promising !! i'll try this weekend :) > >>> > >>> 3 days later, i'm still populating neo4j like a crazy, trying to > >>> compute the collatz conjecture up to 100 millions. > >>> i'm at 63.5 millions, but it's now much much slower than at the > beginning. > >>> > >>> probably because of the insane numbers of index.getSingleNode on > >>> lucene on a growing index. > >>> i'm expecting something like a billion of node when it will reach 100 > millions. > >>> > >>> the index is 6GB and the base is 9GB. > >>> The cpu usage is now ~10% instead of 25%, i have a grand total of 9+6 > >>> GB of data in the neo4j drectory with only 8GB on ram on my windows 7 > >>> i'm now (random) IO bound. > >>> > >>> My SATA velociraptor 10krpm is fighting at 1MB/s (not so bad; > >>> considering it's highly mixed R/W and purely random IO) > >>> > >>> time to buy a 32GB SSD :) > >>> > >>> -- > >>> Keru > >>> > >>> > >>> -- > >>> Laurent "ker2x" Laborde > >>> Sysadmin & DBA at http://www.over-blog.com/ > >>> _______________________________________________ > >>> Neo mailing list > >>> [email protected] > >>> https://lists.neo4j.org/mailman/listinfo/user > >>> > >> _______________________________________________ > >> Neo mailing list > >> [email protected] > >> https://lists.neo4j.org/mailman/listinfo/user > >> > > > > > > > > -- > > Laurent "ker2x" Laborde > > Sysadmin & DBA at http://www.over-blog.com/ > > _______________________________________________ > > Neo mailing list > > [email protected] > > https://lists.neo4j.org/mailman/listinfo/user > > > _______________________________________________ > Neo mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > _______________________________________________ Neo mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
