Hello, I am having a hard time to follow what the problems really are since conversation is split up in several threads.
Pablo, you had a problem with batch inserter throwing an exception upon shutdown that I suspected was due to not enough available disk space. Then there was the the "to many open files". Are you still experiencing problems with that? Massimo, you had problems with injection that created duplicates due to a synchronization issue. That issue has been resolved and now you are experiencing a slowdown during batch inserter injection? Mark, I had a quick look at the code you posted and as I understand it you are saying that it is the index lookups that are taking to long time? That could very well be the case. If the index lookup has to go down to disk the lookup time will be a few ms and that will kill any large data injection. For example inserting 500M relationships requiring 1B index lookups (one for each node) with an avg index lookup time of 1ms is 11 days worth of index lookup time... We have come up with some ideas to avoid the index lookup problem when it can't fit in RAM (only a problem during big data injections). It would be possible to write a tool that splits the injection process up into several steps converting the data so we don't have to do index lookups while injecting relationships. However, before we write such a tool it would be great if you people could investigate some more what the problem really is and verify it is linked to index lookups and not something else. Regards, Johan On Thu, Feb 17, 2011 at 2:40 PM, Massimo Lusetti <[email protected]> wrote: > On Thu, Feb 17, 2011 at 12:54 PM, Pablo Pareja <[email protected]> wrote: > >> Hi Massimo, >> >> It's too bad you are running into the same kind of situations, (specially >> when >> the conclusion you came up to is that Neo4j just degrades...). >> However, did you already try dividing "the" big insertion process into >> smaller >> steps? > > Well I do "big transactions" since the BatchInserter (from the wiki) > is not an option for me and I'm doing 10000 insert per transaction but > as soon as the db grows performance drops inexorably. > > Here it a summary of the latest results which store nodes with only > one String (IPv4 address) property each, it starts fom taking 1.05ms > to insert a Node within a db with 440744 nodes and it end taking > 8.75ms to insert a Node within a db with 12545155 nodes. > > The final DB size is: 2.9G since i tweaked the sintrg_block_size at > graphdb creation time to 60bytes instead of 120... > > If anyone is interested I could provide the complete table of progression... > >> I mean, do you think Neo4j degradation is just proportional to DB size ? > > It seems or at least to the Node's number but that is understandable, > what make me think is that perfomance are so bad that they compromise > usability, but I understand I could do something wrong. > >> or rather just to the amount of data being inserted in the same Batch >> Insertion? > > As I said I use Big transaction pattern from the wiki not the Batch insert > > If anyone is interested I could provide more data... let me know, I > hope to be able to use neo4j for this kind of work. > -- > Massimo > http://meridio.blogspot.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
