The total between DiskCache and Heap must not be more than physical RAM otherwise you go in Swap and performance drops down.
If you have 37G free, then I suggest to give 34G to disk cache and only 3G of heap. Lvca Sent from Mobile device Il 10/Set/2014 09:30 "'Curtis Mosters' via OrientDB" < [email protected]> ha scritto: > Will test it with the next import. But I think I have already tested this, > too. > > Am Mittwoch, 10. September 2014 09:06:03 UTC+2 schrieb Andrey Lomakin: >> >> Hi, >> I meant storage.diskCache.bufferSize not -Xmx we do not use heap we use >> direct memory to avoid GC pauses. >> Could you run with -Dstorage.diskCache.bufferSize=37000 ? >> >> On Thu, Sep 4, 2014 at 2:10 PM, 'Curtis Mosters' via OrientDB < >> [email protected]> wrote: >> >>> Here is the whole test: >>> >>> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) { >>> for (Vertex appln : noTxGraph.getVertices("Appln.ID", splitted[1])) >>> { >>> Edge edge = noTxGraph.addEdge("class:WROTE", person, appln, >>> "WROTE"); >>> edge.setProperty("appltSeqNr", splitted[2]); >>> edge.setProperty("intrSeqNr", splitted[3]); >>> } >>> } >>> >>> 10000.0 on 400.0 entry/sec >>> 20000.0 on 512.8205 entry/sec >>> 30000.0 on 447.7612 entry/sec >>> 40000.0 on 500.0 entry/sec >>> 50000.0 on 561.7977 entry/sec >>> 60000.0 on 566.0377 entry/sec >>> 70000.0 on 514.7059 entry/sec >>> 80000.0 on 444.44446 entry/sec >>> 90000.0 on 379.74683 entry/sec >>> 100000.0 on 324.67532 entry/sec >>> 110000.0 on 279.89822 entry/sec >>> 120000.0 on 244.39919 entry/sec >>> 130000.0 on 215.23178 entry/sec >>> 140000.0 on 191.25684 entry/sec >>> 150000.0 on 171.8213 entry/sec >>> >>> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) { >>> for (Vertex appln : noTxGraph.getVertices("Appln.ID", splitted[1])) >>> { >>> Edge edge = noTxGraph.addEdge(null, person, appln, "WROTE"); >>> edge.setProperty("appltSeqNr", splitted[2]); >>> edge.setProperty("intrSeqNr", splitted[3]); >>> } >>> } >>> >>> 10000.0 on 454.54544 entry/sec >>> 20000.0 on 571.4286 entry/sec >>> 30000.0 on 461.53845 entry/sec >>> 40000.0 on 512.8205 entry/sec >>> 50000.0 on 574.71265 entry/sec >>> 60000.0 on 582.5243 entry/sec >>> 70000.0 on 522.38806 entry/sec >>> 80000.0 on 449.4382 entry/sec >>> 90000.0 on 379.74683 entry/sec >>> 100000.0 on 324.67532 entry/sec >>> >>> Not working with: >>> >>> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) { >>> for (Vertex appln : noTxGraph.getVertices( >>> "Appln.ID", splitted[1])) { >>> Edge edge = noTxGraph.addEdge(null, >>> person, appln, "WROTE"); >>> } >>> } >>> >>> or >>> >>> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) { >>> for (Vertex appln : noTxGraph.getVertices( >>> "Appln.ID", splitted[1])) { >>> noTxGraph.addEdge(null, person, appln, >>> "WROTE"); >>> } >>> } >>> >>> >>> Here I get the following exception: >>> >>> com.orientechnologies.orient.core.exception.ODatabaseException: Error >>> on retrieving record #16:15860 (cluster: >>> >>> >>> >>> Am Donnerstag, 4. September 2014 12:56:46 UTC+2 schrieb Curtis Mosters: >>>> >>>> I set this here on the Run Configuration in Eclipse >>>> >>>> -Xmx37000M -Xms250m -XX:MaxPermSize=10000m >>>> >>>> Is that bad or nearly the same? >>>> >>>> And how can I find you setting? >>>> >>>> Btw: I have 15 GB RAM (but yeah just 3 GB are used while that process >>>> is running, so the Run Configuration seems to be a bad place or wrong >>>> parameters) >>>> >>>> >>>> Am Donnerstag, 4. September 2014 12:49:23 UTC+2 schrieb Andrey Lomakin: >>>>> >>>>> Hi, >>>>> What RAM sized do you have ? >>>>> Just set storage.diskCache.bufferSize to the (RAM_size_in_GB - 1.5GB) >>>>> * 1024. >>>>> >>>>> And you should see speed up of test if your ram size is bigger than >>>>> 6GB. >>>>> >>>>> >>>>> On Thu, Sep 4, 2014 at 1:42 PM, 'Curtis Mosters' via OrientDB < >>>>> [email protected]> wrote: >>>>> >>>>>> Well I'm currently in a good state of inserting huhge amounts of data. >>>>>> >>>>>> My database has already 180mio vertices. The whole proces took round >>>>>> about 6 hours. >>>>>> >>>>>> My current taks is to join, or let's say create edges von 2 Classes. >>>>>> I do that with >>>>>> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) >>>>>> { >>>>>> for (Vertex appln : noTxGraph.getVertices("Appln.ID", splitted[1 >>>>>> ])) { >>>>>> Edge edge = noTxGraph.addEdge("class:WROTE", person, appln, >>>>>> "WROTE"); >>>>>> edge.setProperty("appltSeqNr", splitted[2]); >>>>>> edge.setProperty("intrSeqNr", splitted[3]); >>>>>> } >>>>>> } >>>>>> >>>>>> While I had for all inserting tasks like *20k vertices/sec*. I here >>>>>> have *200 vertices/sec*. So 100 times sloewr >>>>>> >>>>>> The files having ~60 mio lines. >>>>>> >>>>>> So some math: >>>>>> >>>>>> - 12.000 every minute >>>>>> - 720.000 every hour >>>>>> >>>>>> So this would take > 3 days. >>>>>> >>>>>> Really need this till tomorrow. Hopefully someone has a performance >>>>>> improvement. >>>>>> >>>>>> Thank you. >>>>>> >>>>>> -- >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "OrientDB" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Best regards, >>>>> Andrey Lomakin. >>>>> >>>>> Orient Technologies >>>>> the Company behind OrientDB >>>>> >>>>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "OrientDB" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Best regards, >> Andrey Lomakin. >> >> Orient Technologies >> the Company behind OrientDB >> >> -- > > --- > You received this message because you are subscribed to the Google Groups > "OrientDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
