Hi Jing i Use Yourkit java profiler
are you using Transactions? how many records in insert? 2015-01-05 18:27 GMT+01:00 Jing Chen <[email protected]>: > Hi Enrico, > > Yes, we are executing massive insert using the Java API directly. > > What profiling tool do you use? > > Thanks, > Jing > > On Tuesday, December 30, 2014 10:47:23 AM UTC-8, Enrico Risa wrote: >> >> Hi Jing >> >> are you executing massive insert behind >> AdminService.processTasks >> >> Can you send me some profiling session? >> >> 2014-12-30 19:05 GMT+01:00 Jing Chen <[email protected]>: >> >>> Thanks for the quick response Enrico. Our application is a tomcat web >>> application running on a Amazon Redhat 7 instance which has 3.5G memory. We >>> have 3G allocated to tomcat JVM. Here are the error message from the tomcat >>> log: >>> >>> 2014-12-21 05:32:19:694 INFO Deleting partial network >>> 31fdf743-88f5-11e4-a88d-0aa4c1de39d1 in order to rollback in response >>> to error [NdexPersistenceService]Exception in thread "Lucene Merge >>> Thread #11" org.apache.lucene.index.MergePolicy$MergeException: >>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>> >>> at org.apache.lucene.index.ConcurrentMergeScheduler. >>> handleMergeException(ConcurrentMergeScheduler.java:545) >>> >>> at org.apache.lucene.index.ConcurrentMergeScheduler$ >>> MergeThread.run(ConcurrentMergeScheduler.java:518) >>> >>> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded >>> >>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC >>> overhead limit exceeded >>> >>> at java.util.concurrent.FutureTask.report(Unknown Source) >>> >>> at java.util.concurrent.FutureTask.get(Unknown Source) >>> >>> at org.ndexbio.task.NdexTaskExecutor.call( >>> NdexTaskExecutor.java:69) >>> >>> at org.ndexbio.task.NdexTaskExecutor.call( >>> NdexTaskExecutor.java:32) >>> >>> at java.util.concurrent.FutureTask.run(Unknown Source) >>> >>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown >>> Source) >>> >>> at java.util.concurrent.FutureTask.run(Unknown Source) >>> >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown >>> Source) >>> >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >>> Source) >>> >>> at java.lang.Thread.run(Unknown Source) >>> >>> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded >>> >>> java.util.concurrent.ExecutionException: java.lang.Exception: Error >>> occured when executing task. java.lang.OutOfMemoryError: GC overhead limit >>> exceeded >>> >>> at java.util.concurrent.FutureTask.report(Unknown Source) >>> >>> at java.util.concurrent.FutureTask.get(Unknown Source) >>> >>> at org.ndexbio.task.NdexQueuedTaskProcessor.processQueuedTasks( >>> NdexQueuedTaskProcessor.java:101) >>> >>> at org.ndexbio.task.NdexQueuedTaskProcessor.processAll( >>> NdexQueuedTaskProcessor.java:176) >>> >>> at org.ndexbio.rest.services.AdminService.processTasks( >>> AdminService.java:103) >>> >>> at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) >>> >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >>> Source) >>> >>> at java.lang.reflect.Method.invoke(Unknown Source) >>> >>> at org.jboss.resteasy.core.MethodInjectorImpl.invoke( >>> MethodInjectorImpl.java:137) >>> >>> at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget( >>> ResourceMethodInvoker.java:296) >>> >>> at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( >>> ResourceMethodInvoker.java:250) >>> >>> at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( >>> ResourceMethodInvoker.java:237) >>> >>> at org.jboss.resteasy.core.SynchronousDispatcher.invoke( >>> SynchronousDispatcher.java:356) >>> >>> at org.jboss.resteasy.core.SynchronousDispatcher.invoke( >>> SynchronousDispatcher.java:179) >>> >>> at org.jboss.resteasy.plugins.server.servlet. >>> ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) >>> >>> at org.jboss.resteasy.plugins.server.servlet. >>> HttpServletDispatcher.service(HttpServletDispatcher.java:56) >>> >>> at org.jboss.resteasy.plugins.server.servlet. >>> HttpServletDispatcher.service(HttpServletDispatcher.java:51) >>> >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) >>> >>> at org.apache.catalina.core.ApplicationFilterChain. >>> internalDoFilter(ApplicationFilterChain.java:303) >>> >>> at org.apache.catalina.core.ApplicationFilterChain.doFilter( >>> ApplicationFilterChain.java:208) >>> >>> at org.apache.tomcat.websocket.server.WsFilter.doFilter( >>> WsFilter.java:52) >>> >>> at org.apache.catalina.core.ApplicationFilterChain. >>> internalDoFilter(ApplicationFilterChain.java:241) >>> >>> at org.apache.catalina.core.ApplicationFilterChain.doFilter( >>> ApplicationFilterChain.java:208) >>> >>> at org.apache.catalina.core.StandardWrapperValve.invoke( >>> StandardWrapperValve.java:220) >>> >>> at org.apache.catalina.core.StandardContextValve.invoke( >>> StandardContextValve.java:122) >>> >>> at org.apache.catalina.authenticator.AuthenticatorBase.invoke( >>> AuthenticatorBase.java:501) >>> >>> at org.apache.catalina.core.StandardHostValve.invoke( >>> StandardHostValve.java:171) >>> >>> at org.apache.catalina.valves.ErrorReportValve.invoke( >>> ErrorReportValve.java:103) >>> >>> at org.apache.catalina.valves.AccessLogValve.invoke( >>> AccessLogValve.java:950) >>> >>> at org.apache.catalina.core.StandardEngineValve.invoke( >>> StandardEngineValve.java:116) >>> >>> at org.apache.catalina.connector.CoyoteAdapter.service( >>> CoyoteAdapter.java:408) >>> >>> at org.apache.coyote.http11.AbstractHttp11Processor.process( >>> AbstractHttp11Processor.java:1070) >>> >>> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler. >>> process(AbstractProtocol.java:611) >>> >>> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor. >>> run(JIoEndpoint.java:314) >>> >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown >>> Source) >>> >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >>> Source) >>> >>> ... 1 more >>> >>> java.lang.IllegalStateException: this writer hit an OutOfMemoryError; >>> cannot commit >>> >>> at org.apache.lucene.index.IndexWriter.prepareCommitInternal( >>> IndexWriter.java:2842) >>> >>> at org.apache.lucene.index.IndexWriter.commitInternal( >>> IndexWriter.java:3022) >>> >>> at org.apache.lucene.index.IndexWriter.commit( >>> IndexWriter.java:2989) >>> >>> at com.orientechnologies.lucene.manager. >>> OLuceneIndexManagerAbstract.commit(OLuceneIndexManagerAbstract.java:118) >>> >>> at com.orientechnologies.lucene.OLuceneIndexEngine.closeDb( >>> OLuceneIndexEngine.java:139) >>> >>> at com.orientechnologies.orient.core.index.OIndexAbstract. >>> onClose(OIndexAbstract.java:802) >>> >>> at com.orientechnologies.orient.core.db.record. >>> ODatabaseRecordAbstract.callOnCloseListeners( >>> ODatabaseRecordAbstract.java:1536) >>> >>> at com.orientechnologies.orient.core.db.document. >>> ODatabaseDocumentTxPooled.close(ODatabaseDocumentTxPooled.java:112) >>> >>> at org.ndexbio.common.access.NdexDatabase.close( >>> NdexDatabase.java:82) >>> >>> at org.ndexbio.rest.services.AdminService.processTasks( >>> AdminService.java:109) >>> >>> at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) >>> >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >>> Source) >>> >>> at java.lang.reflect.Method.invoke(Unknown Source) >>> >>> at org.jboss.resteasy.core.MethodInjectorImpl.invoke( >>> MethodInjectorImpl.java:137) >>> >>> at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget( >>> ResourceMethodInvoker.java:296) >>> >>> at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( >>> ResourceMethodInvoker.java:250) >>> >>> at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( >>> ResourceMethodInvoker.java:237) >>> >>> at org.jboss.resteasy.core.SynchronousDispatcher.invoke( >>> SynchronousDispatcher.java:356) >>> >>> at org.jboss.resteasy.core.SynchronousDispatcher.invoke( >>> SynchronousDispatcher.java:179) >>> >>> at org.jboss.resteasy.plugins.server.servlet. >>> ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) >>> >>> at org.jboss.resteasy.plugins.server.servlet. >>> HttpServletDispatcher.service(HttpServletDispatcher.java:56) >>> >>> at org.jboss.resteasy.plugins.server.servlet. >>> HttpServletDispatcher.service(HttpServletDispatcher.java:51) >>> >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) >>> >>> at org.apache.catalina.core.ApplicationFilterChain. >>> internalDoFilter(ApplicationFilterChain.java:303) >>> >>> at org.apache.catalina.core.ApplicationFilterChain.doFilter( >>> ApplicationFilterChain.java:208) >>> >>> at org.apache.tomcat.websocket.server.WsFilter.doFilter( >>> WsFilter.java:52) >>> >>> at org.apache.catalina.core.ApplicationFilterChain. >>> internalDoFilter(ApplicationFilterChain.java:241) >>> >>> at org.apache.catalina.core.ApplicationFilterChain.doFilter( >>> ApplicationFilterChain.java:208) >>> >>> at org.apache.catalina.core.StandardWrapperValve.invoke( >>> StandardWrapperValve.java:220) >>> >>> at org.apache.catalina.core.StandardContextValve.invoke( >>> StandardContextValve.java:122) >>> >>> at org.apache.catalina.authenticator.AuthenticatorBase.invoke( >>> AuthenticatorBase.java:501) >>> >>> at org.apache.catalina.core.StandardHostValve.invoke( >>> StandardHostValve.java:171) >>> >>> at org.apache.catalina.valves.ErrorReportValve.invoke( >>> ErrorReportValve.java:103) >>> >>> at org.apache.catalina.valves.AccessLogValve.invoke( >>> AccessLogValve.java:950) >>> >>> at org.apache.catalina.core.StandardEngineValve.invoke( >>> StandardEngineValve.java:116) >>> >>> at org.apache.catalina.connector.CoyoteAdapter.service( >>> CoyoteAdapter.java:408) >>> >>> at org.apache.coyote.http11.AbstractHttp11Processor.process( >>> AbstractHttp11Processor.java:1070) >>> >>> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler. >>> process(AbstractProtocol.java:611) >>> >>> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor. >>> run(JIoEndpoint.java:314) >>> >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown >>> Source) >>> >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >>> Source) >>> >>> at org.apache.tomcat.util.threads.TaskThread$ >>> WrappingRunnable.run(TaskThread.java:61) >>> >>> at java.lang.Thread.run(Unknown Source) >>> >>> java.lang.IllegalStateException: this writer hit an OutOfMemoryError; >>> cannot commit >>> >>> at org.apache.lucene.index.IndexWriter.prepareCommitInternal( >>> IndexWriter.java:2842) >>> >>> at org.apache.lucene.index.IndexWriter.commitInternal( >>> IndexWriter.java:3022) >>> >>> at org.apache.lucene.index.IndexWriter.commit( >>> IndexWriter.java:2989) >>> >>> at com.orientechnologies.lucene.manager. >>> OLuceneIndexManagerAbstract.commit(OLuceneIndexManagerAbstract.java:118) >>> >>> at com.orientechnologies.lucene.OLuceneIndexEngine.closeDb( >>> OLuceneIndexEngine.java:139) >>> >>> at com.orientechnologies.orient.core.index.OIndexAbstract. >>> onClose(OIndexAbstract.java:802) >>> >>> at com.orientechnologies.orient.core.db.record. >>> ODatabaseRecordAbstract.callOnCloseListeners( >>> ODatabaseRecordAbstract.java:1536) >>> >>> at com.orientechnologies.orient.core.db.document. >>> ODatabaseDocumentTxPooled.close(ODatabaseDocumentTxPooled.java:112) >>> >>> at org.ndexbio.rest.filters.BasicAuthenticationFilter.filter( >>> BasicAuthenticationFilter.java:75) >>> >>> at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget( >>> ResourceMethodInvoker.java:264) >>> >>> at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( >>> ResourceMethodInvoker.java:250) >>> >>> at org.jboss.resteasy.core.ResourceMethodInvoker.invoke( >>> ResourceMethodInvoker.java:237) >>> >>> at org.jboss.resteasy.core.SynchronousDispatcher.invoke( >>> SynchronousDispatcher.java:356) >>> >>> at org.jboss.resteasy.core.SynchronousDispatcher.invoke( >>> SynchronousDispatcher.java:179) >>> >>> at org.jboss.resteasy.plugins.server.servlet. >>> ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) >>> >>> at org.jboss.resteasy.plugins.server.servlet. >>> HttpServletDispatcher.service(HttpServletDispatcher.java:56) >>> >>> at org.jboss.resteasy.plugins.server.servlet. >>> HttpServletDispatcher.service(HttpServletDispatcher.java:51) >>> >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) >>> >>> at org.apache.catalina.core.ApplicationFilterChain. >>> internalDoFilter(ApplicationFilterChain.java:303) >>> >>> at org.apache.catalina.core.ApplicationFilterChain.doFilter( >>> ApplicationFilterChain.java:208) >>> >>> at org.apache.tomcat.websocket.server.WsFilter.doFilter( >>> WsFilter.java:52) >>> >>> at org.apache.catalina.core.ApplicationFilterChain. >>> internalDoFilter(ApplicationFilterChain.java:241) >>> >>> at org.apache.catalina.core.ApplicationFilterChain.doFilter( >>> ApplicationFilterChain.java:208) >>> >>> at org.apache.catalina.core.StandardWrapperValve.invoke( >>> StandardWrapperValve.java:220) >>> >>> at org.apache.catalina.core.StandardContextValve.invoke( >>> StandardContextValve.java:122) >>> >>> at org.apache.catalina.authenticator.AuthenticatorBase.invoke( >>> AuthenticatorBase.java:501) >>> >>> at org.apache.catalina.core.StandardHostValve.invoke( >>> StandardHostValve.java:171) >>> >>> at org.apache.catalina.valves.ErrorReportValve.invoke( >>> ErrorReportValve.java:103) >>> >>> at org.apache.catalina.valves.AccessLogValve.invoke( >>> AccessLogValve.java:950) >>> >>> at org.apache.catalina.core.StandardEngineValve.invoke( >>> StandardEngineValve.java:116) >>> >>> at org.apache.catalina.connector.CoyoteAdapter.service( >>> CoyoteAdapter.java:408) >>> >>> at org.apache.coyote.http11.AbstractHttp11Processor.process( >>> AbstractHttp11Processor.java:1070) >>> >>> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler. >>> process(AbstractProtocol.java:611) >>> >>> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor. >>> run(JIoEndpoint.java:314) >>> >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown >>> Source) >>> >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >>> Source) >>> >>> at org.apache.tomcat.util.threads.TaskThread$ >>> WrappingRunnable.run(TaskThread.java:61) >>> >>> at java.lang.Thread.run(Unknown Source) >>> >>> java.lang.IllegalStateException: this writer hit an OutOfMemoryError; >>> cannot commit >>> >>> at org.apache.lucene.index.IndexWriter.prepareCommitInternal( >>> IndexWriter.java:2842) >>> >>> at org.apache.lucene.index.IndexWriter.commitInternal( >>> IndexWriter.java:3022) >>> ... >>> >>> >>> >>> On Tuesday, December 30, 2014 12:52:50 AM UTC-7, Enrico Risa wrote: >>>> >>>> Hi Jin Chen >>>> >>>> can you tell me the configuration of your application? >>>> >>>> Do you have some log or profiling session? >>>> >>>> >>>> >>>> 2014-12-30 5:49 GMT+01:00 Jing Chen <[email protected]>: >>>> >>>>> Hi there, >>>>> >>>>> We are using OrientDb 1.7.10 with Orient-Lucene index. We only use >>>>> lucene text index. We use Orientdb as an embedded server in our web >>>>> application. The application ran out of memory a few times recently. Our >>>>> log shows that it was because Lucene can't allocate more memory. Is there >>>>> a >>>>> way to estimate the memory footprint of Orientdb-lucene index? When our >>>>> database grows, does the memory requirement of Lucene index grow as well? >>>>> >>>>> Thanks advance for your answers. >>>>> >>>>> Jing Chen >>>>> >>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "OrientDB" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "OrientDB" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > > --- > You received this message because you are subscribed to the Google Groups > "OrientDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
