Hi Jing are you executing massive insert behind AdminService.processTasks
Can you send me some profiling session? 2014-12-30 19:05 GMT+01:00 Jing Chen <[email protected]>: > Thanks for the quick response Enrico. Our application is a tomcat web > application running on a Amazon Redhat 7 instance which has 3.5G memory. We > have 3G allocated to tomcat JVM. Here are the error message from the tomcat > log: > > 2014-12-21 05:32:19:694 INFO Deleting partial network > 31fdf743-88f5-11e4-a88d-0aa4c1de39d1 in order to rollback in response to > error [NdexPersistenceService]Exception in thread "Lucene Merge Thread #11" > org.apache.lucene.index.MergePolicy$MergeException: > java.lang.OutOfMemoryError: GC overhead limit exceeded > > at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518) > > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded > > java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC > overhead limit exceeded > > at java.util.concurrent.FutureTask.report(Unknown Source) > > at java.util.concurrent.FutureTask.get(Unknown Source) > > at org.ndexbio.task.NdexTaskExecutor.call(NdexTaskExecutor.java:69) > > at org.ndexbio.task.NdexTaskExecutor.call(NdexTaskExecutor.java:32) > > at java.util.concurrent.FutureTask.run(Unknown Source) > > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown > Source) > > at java.util.concurrent.FutureTask.run(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > > at java.lang.Thread.run(Unknown Source) > > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded > > java.util.concurrent.ExecutionException: java.lang.Exception: Error > occured when executing task. java.lang.OutOfMemoryError: GC overhead limit > exceeded > > at java.util.concurrent.FutureTask.report(Unknown Source) > > at java.util.concurrent.FutureTask.get(Unknown Source) > > at > org.ndexbio.task.NdexQueuedTaskProcessor.processQueuedTasks(NdexQueuedTaskProcessor.java:101) > > at > org.ndexbio.task.NdexQueuedTaskProcessor.processAll(NdexQueuedTaskProcessor.java:176) > > at > org.ndexbio.rest.services.AdminService.processTasks(AdminService.java:103) > > at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at > org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137) > > at > org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296) > > at > org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250) > > at > org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237) > > at > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356) > > at > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179) > > at > org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) > > at > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) > > at > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > > at > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) > > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) > > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) > > at > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070) > > at > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) > > at > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > > ... 1 more > > java.lang.IllegalStateException: this writer hit an OutOfMemoryError; > cannot commit > > at > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2842) > > at > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) > > at > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989) > > at > com.orientechnologies.lucene.manager.OLuceneIndexManagerAbstract.commit(OLuceneIndexManagerAbstract.java:118) > > at > com.orientechnologies.lucene.OLuceneIndexEngine.closeDb(OLuceneIndexEngine.java:139) > > at > com.orientechnologies.orient.core.index.OIndexAbstract.onClose(OIndexAbstract.java:802) > > at > com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callOnCloseListeners(ODatabaseRecordAbstract.java:1536) > > at > com.orientechnologies.orient.core.db.document.ODatabaseDocumentTxPooled.close(ODatabaseDocumentTxPooled.java:112) > > at > org.ndexbio.common.access.NdexDatabase.close(NdexDatabase.java:82) > > at > org.ndexbio.rest.services.AdminService.processTasks(AdminService.java:109) > > at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > > at java.lang.reflect.Method.invoke(Unknown Source) > > at > org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137) > > at > org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296) > > at > org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250) > > at > org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237) > > at > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356) > > at > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179) > > at > org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) > > at > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) > > at > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > > at > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) > > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) > > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) > > at > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070) > > at > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) > > at > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > > at java.lang.Thread.run(Unknown Source) > > java.lang.IllegalStateException: this writer hit an OutOfMemoryError; > cannot commit > > at > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2842) > > at > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) > > at > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989) > > at > com.orientechnologies.lucene.manager.OLuceneIndexManagerAbstract.commit(OLuceneIndexManagerAbstract.java:118) > > at > com.orientechnologies.lucene.OLuceneIndexEngine.closeDb(OLuceneIndexEngine.java:139) > > at > com.orientechnologies.orient.core.index.OIndexAbstract.onClose(OIndexAbstract.java:802) > > at > com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.callOnCloseListeners(ODatabaseRecordAbstract.java:1536) > > at > com.orientechnologies.orient.core.db.document.ODatabaseDocumentTxPooled.close(ODatabaseDocumentTxPooled.java:112) > > at > org.ndexbio.rest.filters.BasicAuthenticationFilter.filter(BasicAuthenticationFilter.java:75) > > at > org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:264) > > at > org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250) > > at > org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237) > > at > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356) > > at > org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179) > > at > org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) > > at > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) > > at > org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > > at > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) > > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) > > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) > > at > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070) > > at > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) > > at > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > > at java.lang.Thread.run(Unknown Source) > > java.lang.IllegalStateException: this writer hit an OutOfMemoryError; > cannot commit > > at > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2842) > > at > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) > ... > > > > On Tuesday, December 30, 2014 12:52:50 AM UTC-7, Enrico Risa wrote: >> >> Hi Jin Chen >> >> can you tell me the configuration of your application? >> >> Do you have some log or profiling session? >> >> >> >> 2014-12-30 5:49 GMT+01:00 Jing Chen <[email protected]>: >> >>> Hi there, >>> >>> We are using OrientDb 1.7.10 with Orient-Lucene index. We only use >>> lucene text index. We use Orientdb as an embedded server in our web >>> application. The application ran out of memory a few times recently. Our >>> log shows that it was because Lucene can't allocate more memory. Is there a >>> way to estimate the memory footprint of Orientdb-lucene index? When our >>> database grows, does the memory requirement of Lucene index grow as well? >>> >>> Thanks advance for your answers. >>> >>> Jing Chen >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "OrientDB" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > > --- > You received this message because you are subscribed to the Google Groups > "OrientDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
