Hi all, Manifold CF crashes. Raising this issue again, after enhancing memory and otherwise, still facing the same issue. TRied every possible solution If anybody can help , please suggest me some solution. *Problem:-* Manifoldcf crashes while crawling data from an intranet URL hinting out below error in logs:-
agents process ran out of memory - shutting down java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.HashMap.resize(HashMap.java:704) at java.util.HashMap.putVal(HashMap.java:629) at java.util.HashMap.put(HashMap.java:612) at org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndEx ecute(CacheManager.java:135) at org.apache.manifoldcf.agents.transformationconnection.TransformationC onnectionManager.loadMultipleInternal(TransformationConnectionManager.java:265) at org.apache.manifoldcf.agents.transformationconnection.TransformationC onnectionManager.loadMultiple(TransformationConnectionManager.java:246) agents process could not start - shutting down agents process ran out of memory - shutting down java.lang.OutOfMemoryError: GC overhead limit exceeded at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:11 93) at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:15 77) at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:36 7) at org.apache.manifoldcf.core.database.Database.execute(Database.java:87 3) at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(D atabase.java:696) agents process ran out of memory - shutting down java.lang.OutOfMemoryError: GC overhead limit exceeded [Thread-491] INFO org.eclipse.jetty.server.ServerConnector - Stopped ServerConne ctor@3a4621bd{HTTP/1.1}{0.0.0.0:8345} agents process ran out of memory - shutting down java.lang.OutOfMemoryError: GC overhead limit exceeded [Thread-491] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e. j.w.WebAppContext@6a57ae10{/mcf-api-service,file:/tmp/jetty-0.0.0.0-8345-mcf-api -service.war-_mcf-api-service-any-2010273384908625074.dir/webapp/,UNAVAILABLE}{/ usr/share/manifoldcf/example/./../web/war/mcf-api-service.war} [Thread-491] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e. j.w.WebAppContext@51c693d{/mcf-authority-service,file:/tmp/jetty-0.0.0.0-8345-mc f-authority-service.war-_mcf-authority-service-any-1431262612693374489.dir/webap p/,UNAVAILABLE}{/usr/share/manifoldcf/example/./../web/war/mcf-authority-service .war} agents process ran out of memory - shutting down java.lang.OutOfMemoryError: Java heap space agents process ran out of memory - shutting down java.lang.OutOfMemoryError: Java heap space agents process ran out of memory - shutting down java.lang.OutOfMemoryError: Java heap space Exception in thread "PostgreSQL-JDBC-SharedTimer-1" java.lang.OutOfMemoryError: Java heap space All configuration are:- I have -Xms1024m ,-Xmx1024m memory allocated in *start-options.env.unix, start-options.env.win file.* Also Configuration:- 1) *For Crawler server *- 16 GB RAM and 8-Core Intel(R) Xeon(R) CPU E5-2660v3 @ 2.60GHz and 2) *For Elasticsearch server* - 48GB and 16-Core Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz and i am using postgres as database. 3) Database as PostgreSQL 4) No of worker threads:-20. 5) manifold Version :-2.13 Can anybody please suggest some solution to get rid of this error and crash, as whenever this istutaion comes, manifoldCF UI results me this:_ [image: image.png] Any suggestion would be highly appreciated. Thanks Priya On Fri, Aug 16, 2019 at 6:41 PM Priya Arora <pr...@smartshore.nl> wrote: > Sure I will try to do that.. and will get back to you > > Sent from my iPhone > > > On 16-Aug-2019, at 6:23 PM, Michael Cizmar <mich...@michaelcizmar.com> > wrote: > > > > Priya - Was this right? > > > > 48GB and 1-Core Intel(R) Xeon(R) CPU > > > > While not directly related to an out of memory issue, you should have > more > > cores allocated to ES. > > > >> On Fri, Aug 16, 2019 at 1:09 AM Priya Arora <pr...@smartshore.nl> > wrote: > >> > >> *Existing Threads/connections configuration is :-* > >> > >> How many worker threads do you have? - 15 worker threads has been > >> allocated(in properties.xml file). > >> And the Tika Extractor connections -10 connections are defined. > >> > >> Is this suggested to reduce the number more. > >> If not, what else can be a solution > >> > >> Thanks > >> Priya > >> > >> > >> > >>> On Wed, Aug 14, 2019 at 5:32 PM Karl Wright <daddy...@gmail.com> > wrote: > >>> > >>> How many worker threads do you have? > >>> Even if each worker thread is constrained in memory, and they should > be, > >>> you can easily cause things to run out of memory by giving too many > >> worker > >>> threads. Another way to keep Tika's usage constrained would be to > reduce > >>> the number of Tika Extractor connections, because that effectively > limits > >>> the number of extractions that can be going on at the same time. > >>> > >>> Karl > >>> > >>> > >>>> On Wed, Aug 14, 2019 at 7:23 AM Priya Arora <pr...@smartshore.nl> > wrote: > >>>> > >>>> Yes , I am using Tika Extractor. And the version used for manifold is > >>> 2.13. > >>>> Also I am using postgres as database. > >>>> > >>>> I have 4 types of jobs > >>>> One is accessing/re crawling data from a public site. Other three are > >>>> accessing intranet site. > >>>> Out of which two are giving me correct output-without any error and > >> third > >>>> one which is having data more than the other two , and giving me this > >>>> error. > >>>> > >>>> Is there any possibility with site accessibility issue. Can you please > >>>> suggest some solution > >>>> Thanks and regards > >>>> Priya > >>>> > >>>> On Wed, Aug 14, 2019 at 3:11 PM Karl Wright <daddy...@gmail.com> > >> wrote: > >>>> > >>>>> I will need to know more. Do you have the tika extractor in your > >>>>> pipeline? If so, what version of ManifoldCF are you using? Tika has > >>> had > >>>>> bugs related to memory consumption in the past; the out of memory > >>>> exception > >>>>> may be coming from it and therefore a stack trace is critical to > >> have. > >>>>> > >>>>> Alternatively, you can upgrade to the latest version of MCF (2.13) > >> and > >>>> that > >>>>> has a newer version of Tika without those problem. But you may need > >> to > >>>> get > >>>>> the agents process more memory. > >>>>> > >>>>> Another possible cause is that you're using hsqldb in production. > >>> HSQLDB > >>>>> keeps all of its tables in memory. If you have a large crawl, you do > >>> not > >>>>> want to use HSQLDB. > >>>>> > >>>>> Thanks, > >>>>> Karl > >>>>> > >>>>> > >>>>> On Wed, Aug 14, 2019 at 3:41 AM Priya Arora <pr...@smartshore.nl> > >>> wrote: > >>>>> > >>>>>> Hi Karl, > >>>>>> > >>>>>> Manifold CF logs hints out me an error like : > >>>>>> agents process ran out of memory - shutting down > >>>>>> java.lang.OutOfMemoryError: Java heap space > >>>>>> > >>>>>> Also I have -Xms1024m ,-Xmx1024m memory allocated in > >>>>>> start-options.env.unix, start-options.env.win file. > >>>>>> Also Configuration:- > >>>>>> 1) For Crawler server - 16 GB RAM and 8-Core Intel(R) Xeon(R) CPU > >>>> E5-2660 > >>>>>> v3 @ 2.60GHz and > >>>>>> > >>>>>> 2) For Elasticsearch server - 48GB and 1-Core Intel(R) Xeon(R) CPU > >>>>> E5-2660 > >>>>>> v3 @ 2.60GHz and i am using postgres as database. > >>>>>> > >>>>>> Can you please help me out, what to do in this case. > >>>>>> > >>>>>> Thanks > >>>>>> Priya > >>>>>> > >>>>>> > >>>>>> On Wed, Aug 14, 2019 at 12:33 PM Karl Wright <daddy...@gmail.com> > >>>> wrote: > >>>>>> > >>>>>>> The error occurs, I believe, as the result of basic connection > >>>>> problems, > >>>>>>> e.g. the connection is getting rejected. You can find more > >>>> information > >>>>>> in > >>>>>>> the simple history, and in the manifoldcf log. > >>>>>>> > >>>>>>> I would like to know the underlying cause, since the connector > >>> should > >>>>> be > >>>>>>> resilient against errors of this kind. > >>>>>>> > >>>>>>> Karl > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Aug 14, 2019, 1:46 AM Priya Arora <pr...@smartshore.nl> > >>>> wrote: > >>>>>>> > >>>>>>>> Hi Karl, > >>>>>>>> > >>>>>>>> I have an web Repository connector(Seeds:- an intranet Site)., > >>> and > >>>>> job > >>>>>> i > >>>>>>>> son Production server. > >>>>>>>> > >>>>>>>> When i ran job on PROD, the job stops itself 2 times with and > >>>>>>> error:Error: > >>>>>>>> Unexpected HTTP result code: -1: null. > >>>>>>>> > >>>>>>>> > >>>>>>>> Can you please provide me an idea, in which it happens so? > >>>>>>>> > >>>>>>>> Thanks and regards > >>>>>>>> Priya Arora > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> >