Hi, Luca. I've now tried OGraphBatchInsert. It is indeed much faster at about 4.5 hours for the billion payments. Slower than Neo but we can live with that.
However, I'm having trouble getting a full run. I'm getting OutOfMemory errors with -XX:MaxDirectMemorySize=512G and combinations of: -Xmx51443M -Dstorage.diskCache.bufferSize=60059 -Xmx90G -Dstorage.diskCache.bufferSize=10240 -Xmx90G -Dstorage.diskCache.bufferSize=8192 So, I've now increased MaxDirectMemorySize to 999G but with no success (this box has 128GB of memory). On a box with about 250GB where I can increase the heap, it ran to the end. There was some SEVEREs at the beginning that said "Previous maximum cache size was 3474813 current maximum cache is 278528. Cache state for storage /home/d3956122/OrientDB/databases/MyPayments3 will not be restored" and some "Exception during commit of active transaction... Database /home/d3956122/OrientDB/databases/MyPayments3 is closed". But after that things seem to go well. I hope these initial errors are not too serious? Unfortunately, there was a hiccup in my coding where the app doesn't naturally die when all the writing is over (I never stop the thread pool - oops). However, it was a day or two later when I killed it. After I killed the processes, I was surprised to see only a few hundred vertices in the plocal database when I was expecting to see one million (the number of edges was much closer to what I expected). At what point are the vertices flushed? Can I flush them via the API? Regards, Phillip On Friday, September 23, 2016 at 6:27:02 PM UTC+1, l.garulli wrote: > > On 23 September 2016 at 11:23, Phillip Henry <[email protected] > <javascript:>> wrote: > >> > will there not be potential contention when the "to" vertex is updated? >> >> Ah, just re-read your post and you've already answered this. My apologies. >> > > Yes, the idea is that with millions and mullions of vertices, the chance > to have a collision with the target nodes is very low, unless you have > supernodes that recurs in most of the relationships. > > >> >>> > OGraphBatchInsert ... keeps everything in RAM before flushing >>> >>> I assume I will still have to write retry code in the event of a >>> collision (see above)? >>> >> > No in this case, the batch insert will manage this for you. > > > Luca > > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
