Could you run a profiler and collect a GC paths for the most consuming objects?
Best regards, Artem Orobets * Orient Technologiesthe Company behind OrientDB* 2014-06-25 13:15 GMT+03:00 Андрей Логинов <[email protected]>: > Hi Artem, > > Thanks for response. > > We did try it in both ways (tx and notx), and we faced that issue that too > many objects are loaded in memory (both scenarios). I will execute the same > operation with notx and will provide new charts... > > среда, 25 июня 2014 г., 12:10:56 UTC+3 пользователь Artem Orobets написал: >> >> Hi Andrey, >> >> I took a look at your charts and it looks like you have too much records >> loaded in memory. >> I suppose the problem is that transaction becomes too big. So try to use >> OrientGraphNoTx for massive insert. If you still need to do an insertion in >> transaction you could divide it on several transactions by doing a commit() >> sometimes. >> >> Best regards, >> Artem Orobets >> >> * Orient Technologiesthe Company behind OrientDB* >> >> >> 2014-06-25 9:56 GMT+03:00 Андрей Логинов <[email protected]>: >> >> Hello, Artem, >>> >>> >>> >>> *>> Could you be more specific? What class instance do you mean? And >>> where it is kept? It would be great if you could point me at code, probably >>> we could avoid such behavior.* >>> Seems like I made a mistake by claiming this. I haven't found anything >>> suspicious in code. However I still face the memory issue.. I spent some >>> time profiling the application and here are my results with allocated 2GB >>> memory for heap. There is definitely something that cannot by collected by >>> GC. >>> Please take a look at the memory usage and GC activity: >>> http://snag.gy/kfFLH.jpg >>> And here is heap gistogram: http://snag.gy/rkRUb.jpg >>> >>> Please take a look at LinkedHashMap$entry it is increasing all the time. >>> It is not used in our code so I assume this is something from orient-core. >>> Do you have any ideas what it could be? >>> >>> >>> >>> *>> Have you tried to check if value is set correctly with >>> isValidataionEnabled call? * >>> Yes I tries, it returns 'false' after disabling it. So seems working >>> fine. However I couldn't notice that it helped a lot... So another question >>> is: is validation performed when I'm using blueprints API? If so, does it >>> help when I disable it using .getRawGraph().set... method and then continue >>> using blueprint API to add vertexes and edges? >>> >>> One more note, after disabling validation, I started getting next >>> messages in console (somewhere about 10-15 messages during the whole >>> process): >>> 2014 9:53:42 AM com.orientechnologies.common.log.OLogManager log >>> WARNING: Update of collection pointer is received but collection is not >>> registered >>> >>> >>> Maybe there is anything else I could disable to perform massive >>> insertion without allocating additional GBs heap memory? >>> >>> >>> вторник, 24 июня 2014 г., 8:52:43 UTC+3 пользователь Artem Orobets >>> написал: >>>> >>>> Hi Andrey, >>>> >>>> We found, that after adding a new vertex, a new class instance is >>>>> created each time to validate the vertex. However it is not realized after >>>>> all. >>>> >>>> Could you be more specific? What class instance do you mean? And where >>>> it is kept? It would be great if you could point me at code, probably we >>>> could avoid such behavior. >>>> >>>> 2) I'm afraid Blueprints API doesn't aware of schema, it is more orient >>>> specific, so that API doesn't allow to configure it. >>>> >>>> 3) The setValidationEnabled method change the state of db where it was >>>> invoked. So *<OrientBaseGraph>.getRawGraph(* >>>> *).setValidationEnabled(false);* should work. Have you tried to check >>>> if value is set correctly with isValidataionEnabled call? >>>> >>>> Best regards, >>>> Artem Orobets >>>> >>>> * Orient Technologiesthe Company behind OrientDB* >>>> >>>> >>>> 2014-06-23 18:30 GMT+03:00 Андрей Логинов <[email protected]>: >>>> >>>>> Good day, >>>>> >>>>> We've faced several issue while trying massive insertions (900k >>>>> vertexes, millions of edges): >>>>> It is becoming possible only after adding additional memory to >>>>> client's VM (up to 4GB) . We found, that after adding a new vertex, a new >>>>> class instance is created each time to validate the vertex. However it is >>>>> not realized after all. >>>>> >>>>> At the same time I've found next article in wiki: https://github.com/ >>>>> orientechnologies/orientdb/wiki/Performance-Tuning-Graph. I noticed >>>>> that there is special method: >>>>> >>>>> graph.setValidationEnabled(false); >>>>> >>>>> The thing is - I'm using blueprint API and can access this method only >>>>> via <OrientBaseGraph>.getRawGraph().setValidationEnabled(false); >>>>> >>>>> >>>>> >>>>> However it seems like that I should work with returned object further in >>>>> order to make this change work. >>>>> >>>>> >>>>> So my questions are: >>>>> 1) how to make orientDB client (via blueprint API) work without adding >>>>> additional memory? >>>>> 2) how to disable validation using this API? >>>>> 3) how to convert result of setValidationEnabled back to OrientBaseGraph? >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Thanks in advance. >>>>> >>>>> >>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "OrientDB" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "OrientDB" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > > --- > You received this message because you are subscribed to the Google Groups > "OrientDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
