Andrey, If you will have time to do insertion benchmark on plocal embedded storage could you also set following parameters
OGlobalConfiguration.CACHE_LEVEL2_ENABLED.setValue(false); OGlobalConfiguration.CACHE_LEVEL2_SIZE.setValue(0); and do not forget to set OGlobalConfiguration.DISK_CACHE_SIZE(amout of ram in megabytes to consume by disk cache) it should be done before you open database in embedded mode. On Sat, Feb 8, 2014 at 8:46 AM, Andrey Lomakin <[email protected]>wrote: > Hi Andrey, > Several notes. > > 1. According to your previous answers I suppose that you did not set > amount of memory which is consumed by disk cache (the more the better of > course) , default value is 4Gb so *you do not use 75% of your memory*during > this benchmark, amount of memory for disk cache has the same > importance as -Xmx for Java. To set this parameter the simplest way is to > provide system property in server.sh so if you have 16 Gb of RAM then I > think you can freely use 14GB on Windows environment without any problems. > So to set this parameter set system property in server.sh > *-Dstorage.diskCache.bufferSize=14336* . After start open console and > issue command "config get storage.diskCache.bufferSize" (after connect as > root of course) to check that this parameter was set correctly. > 2. Your note about bottleneck of insertion of edges with properties is not > completely true. I looked into your code. > You do following: > > OrientEdge e = db.addEdge(null, v1, v2, "E"); > for (int i = 0; i < numberOfProperties; i++) > e.setProperty("property" + i, "value" + i); > > So you add edge at first and then properties. > In such case following steps are performed: > a. Lightweight edge is created. It means direct link between 2 vertexes. > b. Lightweight edge is removed (which means that links on each of 2 > vertexes is removed), document to hold properties is created, new links are > added. > > So we have to perform 3 times more operations on collection of links than > we do without properties on edges. > To avoid this either do not use lightweight edges (typically users do not > have properties on edges so it is good choice to use lightweight edges), or > modify your code as following: > > Map<String, String> properties = new HashMap<String, String>(); > for (int i = 0; i < numberOfProperties; i++) > properties.put("property" + i, "value" + i); > > OrientEdge e = db.addEdge(null, v1, v2, "E", properties); > > In such case document for edge will be created at the same time as you > create edge so we will add links to edge document at once, and will not do > add/remove/add operations instead. > > 3. You use following code > OrientVertex vertex = db.addVertex("OGraphVertex",null); > But you did not create schema which means that new cluster is created > during benchmark and full db flush and fsync is performed which provides > good system slowdown: > > So could you either change code to the following one: > a. OrientVertex vertex = db.addVertex(null); > b. or create "OGraphVertex" class before benchmark. > > And at last about your note that multi threaded test is not much faster, > sorry to agree with this but plocal storage is very new, give us several > months and we will provide multicore support > https://github.com/orientechnologies/orientdb/issues/1678 . > > BTW may I ask you to run the same tests not on remote but on plocal > storage and publish difference ? > > Could you change your benchmark according suggestions above and rerun > test ? > > > > > On Sat, Feb 8, 2014 at 5:36 AM, Andrey Yesyev <[email protected]>wrote: > >> Here is some more results. >> The main change there, I got rid of properties on edges, because I don't >> really need that in my project. >> >> Also, I figured out that >> >> db.getRawGraph().declareIntent(new OIntentMassiveInsert()); >> >> actually slow down insertion rate.... What's the purpose of it then? >> >> >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | Test inserting VERTICES_AND_EDGES threads 4 >> | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | test | documents | >> vertices | edges | time (ms) | avg (doc/sec) | >> | ---------------------------------------- | --------------- | >> --------------- | --------------- | -------------------- | --------------- | >> | 1000 documents with 2 fields each | 1000 | >> 500 | 500 | 37 | 27027.03 | >> | 1000 documents with 5 fields each | 1000 | >> 500 | 500 | 71 | 14084.51 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 5000 documents with 2 fields each | 5000 | >> 2500 | 2500 | 86 | 58139.53 | >> | 5000 documents with 5 fields each | 5000 | >> 2500 | 2500 | 99 | 50505.05 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 10000 documents with 2 fields each | 10000 | >> 5000 | 5000 | 149 | 67114.09 | >> | 10000 documents with 5 fields each | 10000 | >> 5000 | 5000 | 180 | 55555.56 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 50000 documents with 2 fields each | 50000 | >> 25000 | 25000 | 813 | 61500.62 | >> | 50000 documents with 5 fields each | 50000 | >> 25000 | 25000 | 806 | 62034.74 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 10000 documents with 2 fields each | 10000 | >> 5000 | 5000 | 154 | 64935.06 | >> | 10000 documents with 5 fields each | 10000 | >> 5000 | 5000 | 180 | 55555.56 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 50000 documents with 2 fields each | 50000 | >> 25000 | 25000 | 746 | 67024.13 | >> | 50000 documents with 5 fields each | 50000 | >> 25000 | 25000 | 819 | 61050.06 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 100000 documents with 2 fields each | 100000 | >> 50000 | 50000 | 1440 | 69444.44 | >> | 100000 documents with 5 fields each | 100000 | >> 50000 | 50000 | 1780 | 56179.78 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 500000 documents with 2 fields each | 500000 | >> 250000 | 250000 | 7576 | 65997.89 | >> | 500000 documents with 5 fields each | 500000 | >> 250000 | 250000 | 10560 | 47348.48 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 1000000 documents with 2 fields each | 1000000 | >> 500000 | 500000 | 16061 | 62262.62 | >> | 1000000 documents with 5 fields each | 1000000 | >> 500000 | 500000 | 21154 | 47272.38 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> >> As you can see, the performance is way better. >> This is the results of test where vertices don't have properties and >> edges have 2 or 5 properties. >> >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | Test inserting VERTICES_AND_EDGES threads 1 >> | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | test | documents | >> vertices | edges | time (ms) | avg (doc/sec) | >> | ---------------------------------------- | --------------- | >> --------------- | --------------- | -------------------- | --------------- | >> | 1000 documents with 2 fields each | 1000 | >> 500 | 500 | 287 | 3484.32 | >> | 1000 documents with 5 fields each | 1000 | >> 500 | 500 | 215 | 4651.16 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 5000 documents with 2 fields each | 5000 | >> 2500 | 2500 | 723 | 6915.63 | >> | 5000 documents with 5 fields each | 5000 | >> 2500 | 2500 | 596 | 8389.26 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 10000 documents with 2 fields each | 10000 | >> 5000 | 5000 | 977 | 10235.41 | >> | 10000 documents with 5 fields each | 10000 | >> 5000 | 5000 | 1041 | 9606.15 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 50000 documents with 2 fields each | 50000 | >> 25000 | 25000 | 12470 | 4009.62 | >> | 50000 documents with 5 fields each | 50000 | >> 25000 | 25000 | 11266 | 4438.13 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 100000 documents with 2 fields each | 100000 | >> 50000 | 50000 | 78769 | 1269.53 | >> | 100000 documents with 5 fields each | 100000 | >> 50000 | 50000 | 68145 | 1467.46 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 500000 documents with 2 fields each | 374500 | >> 250000 | 124500 | (timeout) 617106 | 606.86 | >> | 500000 documents with 5 fields each | 370100 | >> 250000 | 120100 | (timeout) 620044 | 596.89 | >> >> |-----------------------------------------------------------------------------------------------------------------------------------------| >> | 1000000 documents with 2 fields each | 614100 | >> 500000 | 114100 | (timeout) 626038 | 980.93 | >> >> The result is way worse. >> Starting with 500000 docs, test stopped by timeout, so I didn't wait for >> the whole test suit to complete. >> I think we found the bottle neck of OrientDB. >> Avoid properties in edges! >> >> Would be great to hear why edges properties are so slow! >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "OrientDB" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > > > -- > Best regards, > Andrey Lomakin. > > Orient Technologies > the Company behind OrientDB > > -- Best regards, Andrey Lomakin. Orient Technologies the Company behind OrientDB -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
