Hi Andrey, I tried to imlement similar loader using .NET and binary protocol. The algorithm is following:
1. Read bunch of lines related to the same DbPedia Resource 2. Identify existing objects (simple SELECT FROM DbResource WHERE ResourceId IN ['AWK','ResId'.....] 3. Add lightweigt edge to existing objects 4. Create new non existing resources and add lightweight edge 5. Commit all changes in one optimistic commit One thread is able to save 8000-10000 items per second. It's quite ok. Obviously I tried to implement it in multithread scenario. Step 1 is done in one thread and steps 2-5 in other threads pool using work issued from Step 1. Unfortunately performance with two threads is really bad (~270 items per second). Commits get *com.orientechnologies.common.concur.OTimeoutException. Timeout on acquiring exclusive lock against resource of class: class com.orientechnologies.orient.core.index.OIndexNotUnique with timeout=5000.* Seems that index locking is real bottleneck. I was playing with unique, hash, not unique but always result was similar. Best regards, Pawel On Monday, February 10, 2014 1:24:54 PM UTC+1, Pawel K. wrote: > > Thanks! I will look at it. > Pawel > > On Monday, February 10, 2014 10:32:48 AM UTC+1, Andrey Lomakin wrote: >> >> Hi, >> Here you are https://github.com/laa/orientdb-wikipedia-benchmark . >> >> >> On Wed, Feb 5, 2014 at 11:16 PM, Andrey Lomakin <[email protected]>wrote: >> >>> Hi Pawel, >>> >>> I have merged it in develop branch couple of hours ago. >>> >>> About your questions. >>> >>> 1. I will share tomorrow. >>> 2. sorry you should use java remote client it is new data structure, we >>> will publish specification for it so non java users can use it too. >>> >>> The main parameter of plocal is amount of RAM for disk cache, so as you >>> could see db is used 24 GB of RAM so you should >>> set storage.diskCache.bufferSize parameter in megabytes. >>> >>> >>> >>> >>> On Wed, Feb 5, 2014 at 10:38 PM, Pawel K. <[email protected]> wrote: >>> >>>> Andrey, >>>> >>>> Can you share benchmark sources? I am trying to repeat your scenario >>>> and results. >>>> >>>> Can I achieve similar performance boost using binary protocol with >>>> non-Java client? >>>> >>>> Best regards, >>>> Pawel >>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "OrientDB" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> >>> >>> -- >>> Best regards, >>> Andrey Lomakin. >>> >>> Orient Technologies >>> the Company behind OrientDB >>> >>> >> >> >> -- >> Best regards, >> Andrey Lomakin. >> >> Orient Technologies >> the Company behind OrientDB >> >> -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
