Hi John, At OrientDB startup, haven't you seen a warning in the log about this missing setting?
Best Regards, Luca Garulli Founder & CEO OrientDB LTD <http://orientdb.com/> On 18 May 2017 at 09:07, Andrey Lomakin <[email protected]> wrote: > If you want to calculate how much memory should be consumed by OrientDB > please look at com.orientechnologies.common.directmemory:type= > OByteBufferPoolMXBean JMX bean it contains such attribute as > "preAllocationLimit" which will suggest you how much direct memory now is > set to be used by the server. > > If you need to be sure that you consume not more than you intended to > provide for ODB server, please set system property " > storage.diskCache.bufferSize" to the amount of direct memory in megabytes > which you wish to allow to allocate by the server. > > To be sure that no more direct memory than intended is allocated by ODB > server. > Please run the test and after 24 hours (I suppose that is enough) please > send us values of attributes of following JMX beans: > > 1. com.orientechnologies.common.directmemory:type=OByteBufferPoolMXBean > 2. java.nio:type=BufferPool,name=direct > > So we can check whether ODB consumes direct memory according to > parameters which you set. > > P.S. I do not think that there will be any additional value to switch to > the server with a bigger amount of memory. I suppose would be better to set > server settings accordingly. > > > On Thu, May 18, 2017 at 9:55 AM Andrey Lomakin <[email protected]> > wrote: > >> Hi John, >> >> I suppose you did not set -XX:MaxDirectMemorySize=512g parameter. >> >> >> On Wed, May 17, 2017 at 7:07 PM John J. Szucs <[email protected]> >> wrote: >> >>> After 70 hours on a 32GB VM, ODB 2.2.20, JRE 8u131, the job failed with >>> a direct buffer memory exception. Given the complications I mentioned >>> above, my next step is going to be to get a high-RAM AWS EC2 instance and >>> run this there. However, as I mentioned above, my leadership is getting >>> frustrated with this situation. >>> >>> -- John >>> >>> 'Battle of banja luka'. >>> com.orientechnologies.orient.core.exception.ODatabaseException: >>> Error on retrieving record #63:19090001 (cluster: xlink_simple_2) >>> >>> DB name="kb" >>> at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx. >>> executeReadRecord(ODatabaseDocumentTx.java:2050) >>> at com.orientechnologies.orient.core.tx.OTransactionOptimistic. >>> loadRecord(OTransactionOptimistic.java:187) >>> at com.orientechnologies.orient.core.tx.OTransactionOptimistic. >>> loadRecord(OTransactionOptimistic.java:162) >>> at com.orientechnologies.orient.core.tx.OTransactionOptimistic. >>> loadRecord(OTransactionOptimistic.java:291) >>> at com.orientechnologies.orient.core.db.document. >>> ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1729) >>> at com.orientechnologies.orient.core.db.document. >>> ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:102) >>> at com.orientechnologies.orient.core.id.ORecordId.getRecord( >>> ORecordId.java:329) >>> at com.tinkerpop.blueprints.impls.orient.OrientEdgeIterator. >>> createGraphElement(OrientEdgeIterator.java:72) >>> at com.tinkerpop.blueprints.impls.orient.OrientEdgeIterator. >>> createGraphElement(OrientEdgeIterator.java:44) >>> at com.orientechnologies.orient.core.iterator. >>> OLazyWrapperIterator.hasNext(OLazyWrapperIterator.java:93) >>> at com.orientechnologies.common.collection.OMultiCollectionIterator. >>> hasNextInternal(OMultiCollectionIterator.java:97) >>> at com.orientechnologies.common.collection.OMultiCollectionIterator. >>> hasNext(OMultiCollectionIterator.java:78) >>> at com.lusidity.mind.model.Node.getLinks(Node.java:308) >>> at com.lusidity.mind.model.Node.hasLink(Node.java:435) >>> at com.lusidity.mind.etl.providers.mediawiki.BaseMediaWikiPage. >>> loadHyperlinks(BaseMediaWikiPage.java:401) >>> at com.lusidity.mind.etl.providers.mediawiki.BaseMediaWikiPage.link( >>> BaseMediaWikiPage.java:260) >>> at com.lusidity.mind.etl.providers.mediawiki.BaseMediaWikiPage.load( >>> BaseMediaWikiPage.java:240) >>> at com.lusidity.mind.etl.providers.mediawiki.BaseMediaWikiPage.process( >>> BaseMediaWikiPage.java:98) >>> at com.lusidity.mind.etl.providers.mediawiki.ArticleHandler.process( >>> ArticleHandler.java:113) >>> at com.lusidity.mind.etl.providers.mediawiki.ArticleHandler.process( >>> ArticleHandler.java:75) >>> at info.bliki.wiki.dump.WikiXMLParser.endElement(WikiXMLParser.java:155) >>> at com.sun.org.apache.xerces.internal.parsers. >>> AbstractSAXParser.endElement(AbstractSAXParser.java:609) >>> at com.sun.org.apache.xerces.internal.impl. >>> XMLDocumentFragmentScannerImpl.scanEndElement( >>> XMLDocumentFragmentScannerImpl.java:1782) >>> at com.sun.org.apache.xerces.internal.impl. >>> XMLDocumentFragmentScannerImpl$FragmentContentDriver.next( >>> XMLDocumentFragmentScannerImpl.java:2967) >>> at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next( >>> XMLDocumentScannerImpl.java:602) >>> at com.sun.org.apache.xerces.internal.impl. >>> XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112) >>> at com.sun.org.apache.xerces.internal.impl. >>> XMLDocumentFragmentScannerImpl.scanDocument( >>> XMLDocumentFragmentScannerImpl.java:505) >>> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse( >>> XML11Configuration.java:841) >>> at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse( >>> XML11Configuration.java:770) >>> at com.sun.org.apache.xerces.internal.parsers.XMLParser. >>> parse(XMLParser.java:141) >>> at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse( >>> AbstractSAXParser.java:1213) >>> at info.bliki.wiki.dump.WikiXMLParser.parse(WikiXMLParser.java:194) >>> at com.lusidity.mind.etl.providers.mediawiki.MediaWiki. >>> run(MediaWiki.java:133) >>> at com.lusidity.mind.etl.providers.mediawiki.MediaWiki. >>> run(MediaWiki.java:110) >>> at com.lusidity.mind.shell.commands.ImportCommand. >>> execute(ImportCommand.java:105) >>> at com.lusidity.mind.shell.Shell.execute(Shell.java:265) >>> at com.lusidity.mind.shell.Shell.execute(Shell.java:214) >>> at com.lusidity.mind.shell.commands.ExecCommand.lambda$ >>> execute$0(ExecCommand.java:82) >>> at java.util.Iterator.forEachRemaining(Iterator.java:116) >>> at java.util.Spliterators$IteratorSpliterator. >>> forEachRemaining(Spliterators.java:1801) >>> at java.util.stream.ReferencePipeline$Head.forEachOrdered( >>> ReferencePipeline.java:590) >>> at com.lusidity.mind.shell.commands.ExecCommand.execute( >>> ExecCommand.java:78) >>> at com.lusidity.mind.shell.Shell.execute(Shell.java:265) >>> at com.lusidity.mind.shell.Shell.execute(Shell.java:214) >>> at com.lusidity.mind.shell.Shell.run(Shell.java:173) >>> at com.lusidity.mind.Program.runInteractive(Program.java:209) >>> at com.lusidity.mind.Program.run(Program.java:170) >>> at com.lusidity.mind.Program.main(Program.java:102) >>> Caused by: java.lang.OutOfMemoryError: Direct buffer memory >>> at java.nio.Bits.reserveMemory(Bits.java:694) >>> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) >>> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) >>> at com.orientechnologies.common.directmemory.OByteBufferPool. >>> allocateBuffer(OByteBufferPool.java:328) >>> at com.orientechnologies.common.directmemory.OByteBufferPool. >>> acquireDirect(OByteBufferPool.java:279) >>> at com.orientechnologies.orient.core.storage.cache.local. >>> OWOWCache.cacheFileContent(OWOWCache.java:1280) >>> at com.orientechnologies.orient.core.storage.cache.local. >>> OWOWCache.load(OWOWCache.java:656) >>> at com.orientechnologies.orient.core.storage.cache.local.twoq. >>> O2QCache.updateCache(O2QCache.java:1102) >>> at com.orientechnologies.orient.core.storage.cache.local.twoq. >>> O2QCache.doLoad(O2QCache.java:353) >>> at com.orientechnologies.orient.core.storage.cache.local.twoq. >>> O2QCache.load(O2QCache.java:298) >>> at com.orientechnologies.orient.core.storage.impl.local.paginated.base. >>> ODurableComponent.loadPage(ODurableComponent.java:148) >>> at com.orientechnologies.orient.core.storage.impl.local. >>> paginated.OPaginatedCluster.readRecordBuffer(OPaginatedCluster.java:691) >>> at com.orientechnologies.orient.core.storage.impl.local. >>> paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:667) >>> at com.orientechnologies.orient.core.storage.impl.local. >>> paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:646) >>> at com.orientechnologies.orient.core.storage.impl.local. >>> OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage. >>> java:3260) >>> at com.orientechnologies.orient.core.storage.impl.local. >>> OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage. >>> java:2879) >>> at com.orientechnologies.orient.core.storage.impl.local. >>> OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage. >>> java:1064) >>> at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$ >>> SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3436) >>> at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx. >>> executeReadRecord(ODatabaseDocumentTx.java:2012) >>> ... 47 common frames omitted >>> >>> >>> On Tue, May 16, 2017 at 11:42 AM, John J. Szucs <[email protected]> >>> wrote: >>> >>>> I've had some "complications" (namely, being hospitalized for a medical >>>> issue), but I am running the job right now with OrientDB 2.2.20 and JRE >>>> 8u131. It's only a 32GB VM for now, but it's almost 50% complete and the >>>> results are good so far. >>>> >>>> On Mon, May 15, 2017 at 10:29 AM, Claudio Massi < >>>> [email protected]> wrote: >>>> >>>>> Hi John, >>>>> if you have 64gb ram, to avoid swapping jvm, try to keep process >>>>> size below 64gb, so use Xmx + MaxDirectMemorySize below the available ram >>>>> >>>>> Try orientdb 2.2.20 with java 8u131-b11 , if using G1GC >>>>> >>>>> Monitor heap usage with: jstat -gc pid 120s 9999999 >>>>> >>>>> Monitor direct memory usage with any jmx tool (see >>>>> http://andreylomakin.blogspot.it/2016/05/how-to-calculate- >>>>> maximum-amount-of.html ) >>>>> - use jconsole, section MBeans, choose >>>>> com.orientechnologies.common.directmemory >>>>> -> OByteBufferPoolMXBean -> Attribute >>>>> - use MonBuffers.java (Source from Alan B. in https://gist.github.com/ >>>>> t3rmin4t0r/1a753ccdcfa8d111f07c then increment Thread.sleep(2000), >>>>> and run adding tools.jar in classpath ) >>>>> - use jmxterm (http://wiki.cyclopsgroup.org/jmxterm/) >>>>> ... >>>>> >>>>> Claudio >>>>> >>>>> Il giorno venerdì 5 maggio 2017 18:57:26 UTC+2, John J. Szucs ha >>>>> scritto: >>>>>> >>>>>> Andrey, >>>>>> >>>>>> THANK YOU! I will give this a try as soon as I can. >>>>>> >>>>>> I will also do some JVM profi >>>>>> >>>>>> — John >>>>>> >>>>>> On May 5, 2017, at 05:05, Andrey Lomakin <[email protected]> >>>>>> wrote: >>>>>> >>>>>> Hi John, >>>>>> If you wish you could use this build till we will do official release >>>>>> https://drive.google.com/file/d/0B2oZq2xVp841T2diVGtTcmZ5OTQ/ >>>>>> view?usp=sharing >>>>>> >>>>>> On Fri, May 5, 2017 at 11:58 AM Andrey Lomakin <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> HI John, >>>>>>> >>>>>>> I suppose you encountered issue https://github.com/ >>>>>>> orientechnologies/orientdb/issues/7390 >>>>>>> We will provide release soon. >>>>>>> >>>>>>> Also please do not use such huge heap size we use heap only to keep >>>>>>> temporary data, so I suggest you lower heap size to get ODB the chance >>>>>>> to >>>>>>> use more direct memory. >>>>>>> >>>>>>> On Fri, May 5, 2017 at 10:51 AM Luigi Dell'Aquila < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi John, >>>>>>>> >>>>>>>> How are you doing the import? Are you working in transaction? Some >>>>>>>> code will help us understand where the problem is >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> Luigi >>>>>>>> >>>>>>>> >>>>>>>> 2017-05-05 3:53 GMT+02:00 John J. Szucs <[email protected]>: >>>>>>>> >>>>>>>>> Hello, OrientDB community! It's me again with another question. >>>>>>>>> >>>>>>>>> I am still working on my project and have encountered another >>>>>>>>> serious challenge: it seems that writing to indices (especially edge >>>>>>>>> indices?) can cause OrientDB's direct (non-JVM) memory usage to grow >>>>>>>>> without bounds until the system effectively grinds to a halt due to >>>>>>>>> swap. >>>>>>>>> >>>>>>>>> The specific use case is building a graph based on (English) >>>>>>>>> Wikipedia. There are approximately 17.4M vertices representing pages >>>>>>>>> (including articles, categories, and various meta pages). These >>>>>>>>> vertices >>>>>>>>> are connected by approximately 65M (at last count) edges. There are a >>>>>>>>> few >>>>>>>>> super-nodes. For example, the vertex representing https://en. >>>>>>>>> wikipedia.org/wiki/United_States has (at last count) 306K >>>>>>>>> incoming edges and 822 outgoing edges. However, the degree of the >>>>>>>>> vertices >>>>>>>>> roughly follows a Zipf distribution and the vast majority of vertices >>>>>>>>> have >>>>>>>>> only a few (<10) total (in and out) edges. There are also some other >>>>>>>>> vertex >>>>>>>>> and edge types for lexical data, but I think those are secondary to >>>>>>>>> the >>>>>>>>> issue. >>>>>>>>> >>>>>>>>> Per previous discussion here and on StackOverflow, I have added >>>>>>>>> automatic edge indices on in, out, or the composite of the two to >>>>>>>>> optimize >>>>>>>>> edge queries. When I run the process to extract, transform, and load >>>>>>>>> the >>>>>>>>> data from Wikipedia's XML dumps (using my own ETL code, not >>>>>>>>> OrientDB's), >>>>>>>>> after 24-48 hours, the Linux System Monitor shows that physical memory >>>>>>>>> usage has reached 99.9% and then swap usage begins to grow. At this >>>>>>>>> point, >>>>>>>>> the process is effectively halted by swap thrashing. >>>>>>>>> >>>>>>>>> I am running this on a Fedora 25 Linux VM with 64GB RAM and 16 CPU >>>>>>>>> cores allocated. The JVM settings are as follows: >>>>>>>>> >>>>>>>>> -Xmx32g -Xms32g -server -XX:+PerfDisableSharedMem -XX:+UseG1GC >>>>>>>>> -XX:MaxDirectMemorySize=64413m -Dstorage.wal.syncOnPageFlush=false >>>>>>>>> >>>>>>>>> The MaxDirectMemorySize parameter is recommended by OrientDB >>>>>>>>> itself, during start-up with the "out-of-memory errors" warning. It >>>>>>>>> does >>>>>>>>> seem odd to me that Xmx+MaxDirectMemorySize>available RAM, but >>>>>>>>> I'm more of a deep R&D (not DevOps) guy, so I'm just accepting that >>>>>>>>> unless >>>>>>>>> someone advises me otherwise. >>>>>>>>> >>>>>>>>> If I disable the edge indices, then the process runs fine and >>>>>>>>> completes in a "reasonable" (for it) amount of time: 2-3 days. Of >>>>>>>>> course, >>>>>>>>> if I do this, my run-time performance suffers intolerably. >>>>>>>>> >>>>>>>>> I am running this with OrientDB 2.2.19. I was able to quickly get >>>>>>>>> my code to build with 3.0 M1, but some of the unit tests fail and I am >>>>>>>>> under far too much pressure about this issue from my leadership to >>>>>>>>> try to >>>>>>>>> troubleshoot them right now. >>>>>>>>> >>>>>>>>> What can I do to solve this issue? Thanks in advance for your help! >>>>>>>>> >>>>>>>>> -- John >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "OrientDB" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "OrientDB" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> -- >>>>>>> Best regards, >>>>>>> Andrey Lomakin, R&D lead. >>>>>>> OrientDB Ltd >>>>>>> >>>>>>> twitter: @Andrey_Lomakin >>>>>>> linkedin: https://ua.linkedin.com/in/andreylomakin >>>>>>> blogger: http://andreylomakin.blogspot.com/ >>>>>>> >>>>>> -- >>>>>> Best regards, >>>>>> Andrey Lomakin, R&D lead. >>>>>> OrientDB Ltd >>>>>> >>>>>> twitter: @Andrey_Lomakin >>>>>> linkedin: https://ua.linkedin.com/in/andreylomakin >>>>>> blogger: http://andreylomakin.blogspot.com/ >>>>>> >>>>>> -- >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "OrientDB" group. >>>>>> To unsubscribe from this topic, visit https://groups.google. >>>>>> com/d/topic/orient-database/p0JF5IGsqcs/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "OrientDB" group. >>>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>>> topic/orient-database/p0JF5IGsqcs/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "OrientDB" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> Best regards, >> Andrey Lomakin, R&D lead. >> OrientDB Ltd >> >> twitter: @Andrey_Lomakin >> linkedin: https://ua.linkedin.com/in/andreylomakin >> blogger: http://andreylomakin.blogspot.com/ >> > -- > Best regards, > Andrey Lomakin, R&D lead. > OrientDB Ltd > > twitter: @Andrey_Lomakin > linkedin: https://ua.linkedin.com/in/andreylomakin > blogger: http://andreylomakin.blogspot.com/ > > -- > > --- > You received this message because you are subscribed to the Google Groups > "OrientDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
