Tried a postgres vacuum and also a restart, but the problem persists. Here's the log again with some additional logging details added (below)
I tried running the last query from the logs against the database and it works fine - I modified it to return a count and that also works. SELECT count(*) FROM jobs t1 WHERE EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND transformationname='Tika'); count ------- 39941 (1 row) Is 39k jobs a high number? I've run some other instances of Manifold with more like 1,000 jobs and those seem to be working fine. That's the only thing I can think of that's different between this instance that won't start and the others. Any ideas? Thanks for your help! Mike LOG: duration: 0.079 ms parse <unnamed>: SELECT connectionname FROM transformationconnections WHERE classname=$1 LOG: duration: 0.079 ms bind <unnamed>: SELECT connectionname FROM transformationconnections WHERE classname=$1 DETAIL: parameters: $1 = 'org.apache.manifoldcf.agents.transformation.tika. TikaExtractor' LOG: duration: 0.017 ms execute <unnamed>: SELECT connectionname FROM transformationconnections WHERE classname=$1 DETAIL: parameters: $1 = 'org.apache.manifoldcf.agents.transformation.tika. TikaExtractor' LOG: duration: 0.039 ms parse <unnamed>: SELECT * FROM agents LOG: duration: 0.040 ms bind <unnamed>: SELECT * FROM agents LOG: duration: 0.010 ms execute <unnamed>: SELECT * FROM agents LOG: duration: 0.084 ms parse <unnamed>: SELECT id FROM jobs t1 WHERE EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND transformationname=$1) LOG: duration: 0.359 ms bind <unnamed>: SELECT id FROM jobs t1 WHERE EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND transformationname=$1) DETAIL: parameters: $1 = 'Tika' LOG: duration: 77.622 ms execute <unnamed>: SELECT id FROM jobs t1 WHERE EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND transformationname=$1) DETAIL: parameters: $1 = 'Tika' LOG: incomplete message from client LOG: disconnection: session time: 0:00:06.574 user=REMOVED database=REMOVED host=127.0.0.1 port=45356 >2018-07-30 12:36:09,415 [main] ERROR org.apache.manifoldcf.root - Exception: This connection has been closed. org.apache.manifoldcf.core.interfaces.ManifoldCFException: This connection has been closed. at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. reinterpretException(DBInterfacePostgreSQL.java:627) ~[mcf-core.jar:?] at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) ~[mcf-core.jar:?] at org.apache.manifoldcf.core.database.Database. endTransaction(Database.java:368) ~[mcf-core.jar:?] at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.endTransaction( DBInterfacePostgreSQL.java:1236) ~[mcf-core.jar:?] at org.apache.manifoldcf.crawler.system.ManifoldCF. registerConnectors(ManifoldCF.java:605) ~[mcf-pull-agent.jar:?] at org.apache.manifoldcf.crawler.system.ManifoldCF.reregisterAllConnectors(ManifoldCF.java:160) ~[mcf-pull-agent.jar:?] at org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main( ManifoldCFJettyRunner.java:239) [mcf-jetty-runner.jar:?] Caused by: org.postgresql.util.PSQLException: This connection has been closed. at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) ~[postgresql-42.1.3.jar:42.1.3] at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) ~[postgresql-42.1.3.jar:42.1.3] at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) ~[postgresql-42.1.3.jar:42.1.3] at org.apache.manifoldcf.core.database.Database.execute(Database.java:873) ~[mcf-core.jar:?] at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696) ~[mcf-core.jar:?] org.apache.manifoldcf.core.interfaces.ManifoldCFException: This connection has been closed. at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. reinterpretException(DBInterfacePostgreSQL.java:627) at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) at org.apache.manifoldcf.core.database.Database. endTransaction(Database.java:368) at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.endTransaction( DBInterfacePostgreSQL.java:1236) at org.apache.manifoldcf.crawler.system.ManifoldCF. registerConnectors(ManifoldCF.java:605) at org.apache.manifoldcf.crawler.system.ManifoldCF.reregisterAllConnectors( ManifoldCF.java:160) at org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main( ManifoldCFJettyRunner.java:239) Caused by: org.postgresql.util.PSQLException: This connection has been closed. at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) at org.apache.manifoldcf.core.database.Database.execute(Database.java:873) at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run( Database.java:696) LOG: disconnection: session time: 0:00:10.677 user=postgres database=template1 host=127.0.0.1 port=45354 On Sun, Jul 29, 2018 at 8:09 AM, Karl Wright <[email protected]> wrote: > It looks to me like your database server is not happy. Maybe it's out of > resources? Not sure but a restart may be in order. > > Karl > > > On Sun, Jul 29, 2018 at 9:06 AM Mike Hugo <[email protected]> wrote: > >> Recently we started seeing this error when Manifold CF starts up. We had >> been running Manifold CF with many web connectors and a few RSS feeds for a >> while and it had been working fine. The server got rebooted and since then >> we started seeing this error. I'm not sure exactly what changed. Any ideas >> as to where to start looking and how to fix this? >> >> Thanks! >> >> Mike >> >> >> Initial repository connections already created. >> Configuration file successfully read >> Successfully unregistered all domains >> Successfully unregistered all output connectors >> Successfully unregistered all transformation connectors >> Successfully unregistered all mapping connectors >> Successfully unregistered all authority connectors >> Successfully unregistered all repository connectors >> WARNING: there is already a transaction in progress >> WARNING: there is no transaction in progress >> Successfully registered output connector 'org.apache.manifoldcf.agents. >> output.solr.SolrConnector' >> WARNING: there is already a transaction in progress >> WARNING: there is no transaction in progress >> Successfully registered output connector 'org.apache.manifoldcf.agents. >> output.searchblox.SearchBloxConnector' >> WARNING: there is already a transaction in progress >> WARNING: there is no transaction in progress >> Successfully registered output connector 'org.apache.manifoldcf.agents. >> output.opensearchserver.OpenSearchServerConnector' >> WARNING: there is already a transaction in progress >> WARNING: there is no transaction in progress >> Successfully registered output connector 'org.apache.manifoldcf.agents. >> output.nullconnector.NullConnector' >> WARNING: there is already a transaction in progress >> WARNING: there is no transaction in progress >> Successfully registered output connector 'org.apache.manifoldcf.agents. >> output.kafka.KafkaOutputConnector' >> WARNING: there is already a transaction in progress >> WARNING: there is no transaction in progress >> Successfully registered output connector 'org.apache.manifoldcf.agents. >> output.hdfs.HDFSOutputConnector' >> WARNING: there is already a transaction in progress >> WARNING: there is no transaction in progress >> Successfully registered output connector 'org.apache.manifoldcf.agents. >> output.gts.GTSConnector' >> WARNING: there is already a transaction in progress >> WARNING: there is no transaction in progress >> Successfully registered output connector 'org.apache.manifoldcf.agents. >> output.filesystem.FileOutputConnector' >> WARNING: there is already a transaction in progress >> WARNING: there is no transaction in progress >> Successfully registered output connector 'org.apache.manifoldcf.agents. >> output.elasticsearch.ElasticSearchConnector' >> WARNING: there is already a transaction in progress >> WARNING: there is no transaction in progress >> Successfully registered output connector 'org.apache.manifoldcf.agents. >> output.amazoncloudsearch.AmazonCloudSearchConnector' >> WARNING: there is already a transaction in progress >> WARNING: there is no transaction in progress >> Successfully registered transformation connector >> 'org.apache.manifoldcf.agents.transformation.tikaservice.TikaExtractor' >> WARNING: there is already a transaction in progress >> LOG: incomplete message from client >> >2018-07-29 13:02:06,659 [main] ERROR org.apache.manifoldcf.root - >> Exception: This connection has been closed. >> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >> connection has been closed. >> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >> reinterpretException(DBInterfacePostgreSQL.java:627) ~[mcf-core.jar:?] >> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >> ~[mcf-core.jar:?] >> at org.apache.manifoldcf.core.database.Database. >> endTransaction(Database.java:368) ~[mcf-core.jar:?] >> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >> endTransaction(DBInterfacePostgreSQL.java:1236) ~[mcf-core.jar:?] >> at org.apache.manifoldcf.crawler.system.ManifoldCF. >> registerConnectors(ManifoldCF.java:605) ~[mcf-pull-agent.jar:?] >> at org.apache.manifoldcf.crawler.system.ManifoldCF. >> reregisterAllConnectors(ManifoldCF.java:160) ~[mcf-pull-agent.jar:?] >> at org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main( >> ManifoldCFJettyRunner.java:239) [mcf-jetty-runner.jar:?] >> Caused by: org.postgresql.util.PSQLException: This connection has been >> closed. >> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >> ~[postgresql-42.1.3.jar:42.1.3] >> at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) >> ~[postgresql-42.1.3.jar:42.1.3] >> at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) >> ~[postgresql-42.1.3.jar:42.1.3] >> at org.apache.manifoldcf.core.database.Database.execute(Database.java:873) >> ~[mcf-core.jar:?] >> at >> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696) >> ~[mcf-core.jar:?] >> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >> connection has been closed. >> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >> reinterpretException(DBInterfacePostgreSQL.java:627) >> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >> at org.apache.manifoldcf.core.database.Database. >> endTransaction(Database.java:368) >> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >> endTransaction(DBInterfacePostgreSQL.java:1236) >> at org.apache.manifoldcf.crawler.system.ManifoldCF. >> registerConnectors(ManifoldCF.java:605) >> at org.apache.manifoldcf.crawler.system.ManifoldCF. >> reregisterAllConnectors(ManifoldCF.java:160) >> at org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main( >> ManifoldCFJettyRunner.java:239) >> Caused by: org.postgresql.util.PSQLException: This connection has been >> closed. >> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >> at org.postgresql.jdbc.PgConnection.createStatement( >> PgConnection.java:1576) >> at org.postgresql.jdbc.PgConnection.createStatement( >> PgConnection.java:367) >> at org.apache.manifoldcf.core.database.Database.execute( >> Database.java:873) >> at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run( >> Database.java:696) >> >
