' LOG: incomplete message from client' This shows a network issue. Did your network configuration change recently?
Karl On Mon, Jul 30, 2018 at 9:59 AM Mike Hugo <[email protected]> wrote: > Tried a postgres vacuum and also a restart, but the problem persists. > Here's the log again with some additional logging details added (below) > > I tried running the last query from the logs against the database and it > works fine - I modified it to return a count and that also works. > > SELECT count(*) FROM jobs t1 WHERE EXISTS(SELECT 'x' FROM jobpipelines > WHERE t1.id=ownerid AND transformationname='Tika'); > count > ------- > 39941 > (1 row) > > > Is 39k jobs a high number? I've run some other instances of Manifold with > more like 1,000 jobs and those seem to be working fine. That's the only > thing I can think of that's different between this instance that won't > start and the others. Any ideas? > > Thanks for your help! > > Mike > > LOG: duration: 0.079 ms parse <unnamed>: SELECT connectionname FROM > transformationconnections WHERE classname=$1 > LOG: duration: 0.079 ms bind <unnamed>: SELECT connectionname FROM > transformationconnections WHERE classname=$1 > DETAIL: parameters: $1 = > 'org.apache.manifoldcf.agents.transformation.tika.TikaExtractor' > LOG: duration: 0.017 ms execute <unnamed>: SELECT connectionname FROM > transformationconnections WHERE classname=$1 > DETAIL: parameters: $1 = > 'org.apache.manifoldcf.agents.transformation.tika.TikaExtractor' > LOG: duration: 0.039 ms parse <unnamed>: SELECT * FROM agents > LOG: duration: 0.040 ms bind <unnamed>: SELECT * FROM agents > LOG: duration: 0.010 ms execute <unnamed>: SELECT * FROM agents > LOG: duration: 0.084 ms parse <unnamed>: SELECT id FROM jobs t1 WHERE > EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND > transformationname=$1) > LOG: duration: 0.359 ms bind <unnamed>: SELECT id FROM jobs t1 WHERE > EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND > transformationname=$1) > DETAIL: parameters: $1 = 'Tika' > LOG: duration: 77.622 ms execute <unnamed>: SELECT id FROM jobs t1 WHERE > EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND > transformationname=$1) > DETAIL: parameters: $1 = 'Tika' > LOG: incomplete message from client > LOG: disconnection: session time: 0:00:06.574 user=REMOVED > database=REMOVED host=127.0.0.1 port=45356 > >2018-07-30 12:36:09,415 [main] ERROR org.apache.manifoldcf.root - > Exception: This connection has been closed. > org.apache.manifoldcf.core.interfaces.ManifoldCFException: This connection > has been closed. > at > org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.reinterpretException(DBInterfacePostgreSQL.java:627) > ~[mcf-core.jar:?] > at > org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) > ~[mcf-core.jar:?] > at > org.apache.manifoldcf.core.database.Database.endTransaction(Database.java:368) > ~[mcf-core.jar:?] > at > org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.endTransaction(DBInterfacePostgreSQL.java:1236) > ~[mcf-core.jar:?] > at > org.apache.manifoldcf.crawler.system.ManifoldCF.registerConnectors(ManifoldCF.java:605) > ~[mcf-pull-agent.jar:?] > at > org.apache.manifoldcf.crawler.system.ManifoldCF.reregisterAllConnectors(ManifoldCF.java:160) > ~[mcf-pull-agent.jar:?] > at > org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main(ManifoldCFJettyRunner.java:239) > [mcf-jetty-runner.jar:?] > Caused by: org.postgresql.util.PSQLException: This connection has been > closed. > at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) > ~[postgresql-42.1.3.jar:42.1.3] > at > org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) > ~[postgresql-42.1.3.jar:42.1.3] > at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) > ~[postgresql-42.1.3.jar:42.1.3] > at org.apache.manifoldcf.core.database.Database.execute(Database.java:873) > ~[mcf-core.jar:?] > at > org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696) > ~[mcf-core.jar:?] > org.apache.manifoldcf.core.interfaces.ManifoldCFException: This connection > has been closed. > at > org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.reinterpretException(DBInterfacePostgreSQL.java:627) > at > org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) > at > org.apache.manifoldcf.core.database.Database.endTransaction(Database.java:368) > at > org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.endTransaction(DBInterfacePostgreSQL.java:1236) > at > org.apache.manifoldcf.crawler.system.ManifoldCF.registerConnectors(ManifoldCF.java:605) > at > org.apache.manifoldcf.crawler.system.ManifoldCF.reregisterAllConnectors(ManifoldCF.java:160) > at > org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main(ManifoldCFJettyRunner.java:239) > Caused by: org.postgresql.util.PSQLException: This connection has been > closed. > at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) > at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) > at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) > at org.apache.manifoldcf.core.database.Database.execute(Database.java:873) > at > org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696) > LOG: disconnection: session time: 0:00:10.677 user=postgres > database=template1 host=127.0.0.1 port=45354 > > > > On Sun, Jul 29, 2018 at 8:09 AM, Karl Wright <[email protected]> wrote: > >> It looks to me like your database server is not happy. Maybe it's out of >> resources? Not sure but a restart may be in order. >> >> Karl >> >> >> On Sun, Jul 29, 2018 at 9:06 AM Mike Hugo <[email protected]> wrote: >> >>> Recently we started seeing this error when Manifold CF starts up. We >>> had been running Manifold CF with many web connectors and a few RSS feeds >>> for a while and it had been working fine. The server got rebooted and >>> since then we started seeing this error. I'm not sure exactly what >>> changed. Any ideas as to where to start looking and how to fix this? >>> >>> Thanks! >>> >>> Mike >>> >>> >>> Initial repository connections already created. >>> Configuration file successfully read >>> Successfully unregistered all domains >>> Successfully unregistered all output connectors >>> Successfully unregistered all transformation connectors >>> Successfully unregistered all mapping connectors >>> Successfully unregistered all authority connectors >>> Successfully unregistered all repository connectors >>> WARNING: there is already a transaction in progress >>> WARNING: there is no transaction in progress >>> Successfully registered output connector >>> 'org.apache.manifoldcf.agents.output.solr.SolrConnector' >>> WARNING: there is already a transaction in progress >>> WARNING: there is no transaction in progress >>> Successfully registered output connector >>> 'org.apache.manifoldcf.agents.output.searchblox.SearchBloxConnector' >>> WARNING: there is already a transaction in progress >>> WARNING: there is no transaction in progress >>> Successfully registered output connector >>> 'org.apache.manifoldcf.agents.output.opensearchserver.OpenSearchServerConnector' >>> WARNING: there is already a transaction in progress >>> WARNING: there is no transaction in progress >>> Successfully registered output connector >>> 'org.apache.manifoldcf.agents.output.nullconnector.NullConnector' >>> WARNING: there is already a transaction in progress >>> WARNING: there is no transaction in progress >>> Successfully registered output connector >>> 'org.apache.manifoldcf.agents.output.kafka.KafkaOutputConnector' >>> WARNING: there is already a transaction in progress >>> WARNING: there is no transaction in progress >>> Successfully registered output connector >>> 'org.apache.manifoldcf.agents.output.hdfs.HDFSOutputConnector' >>> WARNING: there is already a transaction in progress >>> WARNING: there is no transaction in progress >>> Successfully registered output connector >>> 'org.apache.manifoldcf.agents.output.gts.GTSConnector' >>> WARNING: there is already a transaction in progress >>> WARNING: there is no transaction in progress >>> Successfully registered output connector >>> 'org.apache.manifoldcf.agents.output.filesystem.FileOutputConnector' >>> WARNING: there is already a transaction in progress >>> WARNING: there is no transaction in progress >>> Successfully registered output connector >>> 'org.apache.manifoldcf.agents.output.elasticsearch.ElasticSearchConnector' >>> WARNING: there is already a transaction in progress >>> WARNING: there is no transaction in progress >>> Successfully registered output connector >>> 'org.apache.manifoldcf.agents.output.amazoncloudsearch.AmazonCloudSearchConnector' >>> WARNING: there is already a transaction in progress >>> WARNING: there is no transaction in progress >>> Successfully registered transformation connector >>> 'org.apache.manifoldcf.agents.transformation.tikaservice.TikaExtractor' >>> WARNING: there is already a transaction in progress >>> LOG: incomplete message from client >>> >2018-07-29 13:02:06,659 [main] ERROR org.apache.manifoldcf.root - >>> Exception: This connection has been closed. >>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >>> connection has been closed. >>> at >>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.reinterpretException(DBInterfacePostgreSQL.java:627) >>> ~[mcf-core.jar:?] >>> at >>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>> ~[mcf-core.jar:?] >>> at >>> org.apache.manifoldcf.core.database.Database.endTransaction(Database.java:368) >>> ~[mcf-core.jar:?] >>> at >>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.endTransaction(DBInterfacePostgreSQL.java:1236) >>> ~[mcf-core.jar:?] >>> at >>> org.apache.manifoldcf.crawler.system.ManifoldCF.registerConnectors(ManifoldCF.java:605) >>> ~[mcf-pull-agent.jar:?] >>> at >>> org.apache.manifoldcf.crawler.system.ManifoldCF.reregisterAllConnectors(ManifoldCF.java:160) >>> ~[mcf-pull-agent.jar:?] >>> at >>> org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main(ManifoldCFJettyRunner.java:239) >>> [mcf-jetty-runner.jar:?] >>> Caused by: org.postgresql.util.PSQLException: This connection has been >>> closed. >>> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >>> ~[postgresql-42.1.3.jar:42.1.3] >>> at >>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) >>> ~[postgresql-42.1.3.jar:42.1.3] >>> at >>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) >>> ~[postgresql-42.1.3.jar:42.1.3] >>> at >>> org.apache.manifoldcf.core.database.Database.execute(Database.java:873) >>> ~[mcf-core.jar:?] >>> at >>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696) >>> ~[mcf-core.jar:?] >>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >>> connection has been closed. >>> at >>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.reinterpretException(DBInterfacePostgreSQL.java:627) >>> at >>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>> at >>> org.apache.manifoldcf.core.database.Database.endTransaction(Database.java:368) >>> at >>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.endTransaction(DBInterfacePostgreSQL.java:1236) >>> at >>> org.apache.manifoldcf.crawler.system.ManifoldCF.registerConnectors(ManifoldCF.java:605) >>> at >>> org.apache.manifoldcf.crawler.system.ManifoldCF.reregisterAllConnectors(ManifoldCF.java:160) >>> at >>> org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main(ManifoldCFJettyRunner.java:239) >>> Caused by: org.postgresql.util.PSQLException: This connection has been >>> closed. >>> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >>> at >>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) >>> at >>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) >>> at >>> org.apache.manifoldcf.core.database.Database.execute(Database.java:873) >>> at >>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696) >>> >> >
