Will test it with the next import. But I think I have already tested this, too.
Am Mittwoch, 10. September 2014 09:06:03 UTC+2 schrieb Andrey Lomakin: > > Hi, > I meant storage.diskCache.bufferSize not -Xmx we do not use heap we use > direct memory to avoid GC pauses. > Could you run with -Dstorage.diskCache.bufferSize=37000 ? > > On Thu, Sep 4, 2014 at 2:10 PM, 'Curtis Mosters' via OrientDB < > [email protected] <javascript:>> wrote: > >> Here is the whole test: >> >> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) { >> for (Vertex appln : noTxGraph.getVertices("Appln.ID", splitted[1])) { >> Edge edge = noTxGraph.addEdge("class:WROTE", person, appln, >> "WROTE"); >> edge.setProperty("appltSeqNr", splitted[2]); >> edge.setProperty("intrSeqNr", splitted[3]); >> } >> } >> >> 10000.0 on 400.0 entry/sec >> 20000.0 on 512.8205 entry/sec >> 30000.0 on 447.7612 entry/sec >> 40000.0 on 500.0 entry/sec >> 50000.0 on 561.7977 entry/sec >> 60000.0 on 566.0377 entry/sec >> 70000.0 on 514.7059 entry/sec >> 80000.0 on 444.44446 entry/sec >> 90000.0 on 379.74683 entry/sec >> 100000.0 on 324.67532 entry/sec >> 110000.0 on 279.89822 entry/sec >> 120000.0 on 244.39919 entry/sec >> 130000.0 on 215.23178 entry/sec >> 140000.0 on 191.25684 entry/sec >> 150000.0 on 171.8213 entry/sec >> >> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) { >> for (Vertex appln : noTxGraph.getVertices("Appln.ID", splitted[1])) { >> Edge edge = noTxGraph.addEdge(null, person, appln, "WROTE"); >> edge.setProperty("appltSeqNr", splitted[2]); >> edge.setProperty("intrSeqNr", splitted[3]); >> } >> } >> >> 10000.0 on 454.54544 entry/sec >> 20000.0 on 571.4286 entry/sec >> 30000.0 on 461.53845 entry/sec >> 40000.0 on 512.8205 entry/sec >> 50000.0 on 574.71265 entry/sec >> 60000.0 on 582.5243 entry/sec >> 70000.0 on 522.38806 entry/sec >> 80000.0 on 449.4382 entry/sec >> 90000.0 on 379.74683 entry/sec >> 100000.0 on 324.67532 entry/sec >> >> Not working with: >> >> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) { >> for (Vertex appln : noTxGraph.getVertices( >> "Appln.ID", splitted[1])) { >> Edge edge = noTxGraph.addEdge(null, >> person, appln, "WROTE"); >> } >> } >> >> or >> >> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) { >> for (Vertex appln : noTxGraph.getVertices( >> "Appln.ID", splitted[1])) { >> noTxGraph.addEdge(null, person, appln, >> "WROTE"); >> } >> } >> >> >> Here I get the following exception: >> >> com.orientechnologies.orient.core.exception.ODatabaseException: Error on >> retrieving record #16:15860 (cluster: >> >> >> >> Am Donnerstag, 4. September 2014 12:56:46 UTC+2 schrieb Curtis Mosters: >>> >>> I set this here on the Run Configuration in Eclipse >>> >>> -Xmx37000M -Xms250m -XX:MaxPermSize=10000m >>> >>> Is that bad or nearly the same? >>> >>> And how can I find you setting? >>> >>> Btw: I have 15 GB RAM (but yeah just 3 GB are used while that process is >>> running, so the Run Configuration seems to be a bad place or wrong >>> parameters) >>> >>> >>> Am Donnerstag, 4. September 2014 12:49:23 UTC+2 schrieb Andrey Lomakin: >>>> >>>> Hi, >>>> What RAM sized do you have ? >>>> Just set storage.diskCache.bufferSize to the (RAM_size_in_GB - 1.5GB) * >>>> 1024. >>>> >>>> And you should see speed up of test if your ram size is bigger than 6GB. >>>> >>>> >>>> On Thu, Sep 4, 2014 at 1:42 PM, 'Curtis Mosters' via OrientDB < >>>> [email protected]> wrote: >>>> >>>>> Well I'm currently in a good state of inserting huhge amounts of data. >>>>> >>>>> My database has already 180mio vertices. The whole proces took round >>>>> about 6 hours. >>>>> >>>>> My current taks is to join, or let's say create edges von 2 Classes. I >>>>> do that with >>>>> for (Vertex person : noTxGraph.getVertices("Person.ID", splitted[0])) >>>>> { >>>>> for (Vertex appln : noTxGraph.getVertices("Appln.ID", splitted[1])) >>>>> { >>>>> Edge edge = noTxGraph.addEdge("class:WROTE", person, appln, >>>>> "WROTE"); >>>>> edge.setProperty("appltSeqNr", splitted[2]); >>>>> edge.setProperty("intrSeqNr", splitted[3]); >>>>> } >>>>> } >>>>> >>>>> While I had for all inserting tasks like *20k vertices/sec*. I here >>>>> have *200 vertices/sec*. So 100 times sloewr >>>>> >>>>> The files having ~60 mio lines. >>>>> >>>>> So some math: >>>>> >>>>> - 12.000 every minute >>>>> - 720.000 every hour >>>>> >>>>> So this would take > 3 days. >>>>> >>>>> Really need this till tomorrow. Hopefully someone has a performance >>>>> improvement. >>>>> >>>>> Thank you. >>>>> >>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "OrientDB" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Best regards, >>>> Andrey Lomakin. >>>> >>>> Orient Technologies >>>> the Company behind OrientDB >>>> >>>> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "OrientDB" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Best regards, > Andrey Lomakin. > > Orient Technologies > the Company behind OrientDB > > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
