So for my next test how would the best Setting look like? I have 30 GB RAM and yeah maybe 1-2 GB for the system.
So -Xmx28000M or -Dstorage.diskCache.bufferSize=28000 ? Am Mittwoch, 10. September 2014 09:34:26 UTC+2 schrieb Lvc@: > > The total between DiskCache and Heap must not be more than physical RAM > otherwise you go in Swap and performance drops down. > > If you have 37G free, then I suggest to give 34G to disk cache and only 3G > of heap. > > Lvca > > Sent from Mobile device > Il 10/Set/2014 09:30 "'Curtis Mosters' via OrientDB" < > [email protected] <javascript:>> ha scritto: > >> Will test it with the next import. But I think I have already tested >> this, too. >> >> Am Mittwoch, 10. September 2014 09:06:03 UTC+2 schrieb Andrey Lomakin: >>> >>> Hi, >>> I meant storage.diskCache.bufferSize not -Xmx we do not use heap we use >>> direct memory to avoid GC pauses. >>> Could you run with -Dstorage.diskCache.bufferSize=37000 ? >>> >>> On Thu, Sep 4, 2014 at 2:10 PM, 'Curtis Mosters' via OrientDB < >>> [email protected]> wrote: >>> >>>> Here is the whole test: >>>> >>>> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) { >>>> for (Vertex appln : noTxGraph.getVertices("Appln.ID", >>>> splitted[1])) { >>>> Edge edge = noTxGraph.addEdge("class:WROTE", person, appln, >>>> "WROTE"); >>>> edge.setProperty("appltSeqNr", splitted[2]); >>>> edge.setProperty("intrSeqNr", splitted[3]); >>>> } >>>> } >>>> >>>> 10000.0 on 400.0 entry/sec >>>> 20000.0 on 512.8205 entry/sec >>>> 30000.0 on 447.7612 entry/sec >>>> 40000.0 on 500.0 entry/sec >>>> 50000.0 on 561.7977 entry/sec >>>> 60000.0 on 566.0377 entry/sec >>>> 70000.0 on 514.7059 entry/sec >>>> 80000.0 on 444.44446 entry/sec >>>> 90000.0 on 379.74683 entry/sec >>>> 100000.0 on 324.67532 entry/sec >>>> 110000.0 on 279.89822 entry/sec >>>> 120000.0 on 244.39919 entry/sec >>>> 130000.0 on 215.23178 entry/sec >>>> 140000.0 on 191.25684 entry/sec >>>> 150000.0 on 171.8213 entry/sec >>>> >>>> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) { >>>> for (Vertex appln : noTxGraph.getVertices("Appln.ID", splitted[1])) >>>> { >>>> Edge edge = noTxGraph.addEdge(null, person, appln, "WROTE"); >>>> edge.setProperty("appltSeqNr", splitted[2]); >>>> edge.setProperty("intrSeqNr", splitted[3]); >>>> } >>>> } >>>> >>>> 10000.0 on 454.54544 entry/sec >>>> 20000.0 on 571.4286 entry/sec >>>> 30000.0 on 461.53845 entry/sec >>>> 40000.0 on 512.8205 entry/sec >>>> 50000.0 on 574.71265 entry/sec >>>> 60000.0 on 582.5243 entry/sec >>>> 70000.0 on 522.38806 entry/sec >>>> 80000.0 on 449.4382 entry/sec >>>> 90000.0 on 379.74683 entry/sec >>>> 100000.0 on 324.67532 entry/sec >>>> >>>> Not working with: >>>> >>>> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) { >>>> for (Vertex appln : noTxGraph.getVertices( >>>> "Appln.ID", splitted[1])) { >>>> Edge edge = noTxGraph.addEdge(null, >>>> person, appln, "WROTE"); >>>> } >>>> } >>>> >>>> or >>>> >>>> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) { >>>> for (Vertex appln : noTxGraph.getVertices( >>>> "Appln.ID", splitted[1])) { >>>> noTxGraph.addEdge(null, person, appln, >>>> "WROTE"); >>>> } >>>> } >>>> >>>> >>>> Here I get the following exception: >>>> >>>> com.orientechnologies.orient.core.exception.ODatabaseException: Error >>>> on retrieving record #16:15860 (cluster: >>>> >>>> >>>> >>>> Am Donnerstag, 4. September 2014 12:56:46 UTC+2 schrieb Curtis Mosters: >>>>> >>>>> I set this here on the Run Configuration in Eclipse >>>>> >>>>> -Xmx37000M -Xms250m -XX:MaxPermSize=10000m >>>>> >>>>> Is that bad or nearly the same? >>>>> >>>>> And how can I find you setting? >>>>> >>>>> Btw: I have 15 GB RAM (but yeah just 3 GB are used while that process >>>>> is running, so the Run Configuration seems to be a bad place or wrong >>>>> parameters) >>>>> >>>>> >>>>> Am Donnerstag, 4. September 2014 12:49:23 UTC+2 schrieb Andrey Lomakin: >>>>>> >>>>>> Hi, >>>>>> What RAM sized do you have ? >>>>>> Just set storage.diskCache.bufferSize to the (RAM_size_in_GB - 1.5GB) >>>>>> * 1024. >>>>>> >>>>>> And you should see speed up of test if your ram size is bigger than >>>>>> 6GB. >>>>>> >>>>>> >>>>>> On Thu, Sep 4, 2014 at 1:42 PM, 'Curtis Mosters' via OrientDB < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Well I'm currently in a good state of inserting huhge amounts of >>>>>>> data. >>>>>>> >>>>>>> My database has already 180mio vertices. The whole proces took round >>>>>>> about 6 hours. >>>>>>> >>>>>>> My current taks is to join, or let's say create edges von 2 Classes. >>>>>>> I do that with >>>>>>> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0 >>>>>>> ])) { >>>>>>> for (Vertex appln : noTxGraph.getVertices("Appln.ID", splitted[1 >>>>>>> ])) { >>>>>>> Edge edge = noTxGraph.addEdge("class:WROTE", person, appln, >>>>>>> "WROTE"); >>>>>>> edge.setProperty("appltSeqNr", splitted[2]); >>>>>>> edge.setProperty("intrSeqNr", splitted[3]); >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> While I had for all inserting tasks like *20k vertices/sec*. I here >>>>>>> have *200 vertices/sec*. So 100 times sloewr >>>>>>> >>>>>>> The files having ~60 mio lines. >>>>>>> >>>>>>> So some math: >>>>>>> >>>>>>> - 12.000 every minute >>>>>>> - 720.000 every hour >>>>>>> >>>>>>> So this would take > 3 days. >>>>>>> >>>>>>> Really need this till tomorrow. Hopefully someone has a performance >>>>>>> improvement. >>>>>>> >>>>>>> Thank you. >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "OrientDB" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best regards, >>>>>> Andrey Lomakin. >>>>>> >>>>>> Orient Technologies >>>>>> the Company behind OrientDB >>>>>> >>>>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "OrientDB" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Best regards, >>> Andrey Lomakin. >>> >>> Orient Technologies >>> the Company behind OrientDB >>> >>> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "OrientDB" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
