No, and manifold and postgres run on the same host. On Mon, Jul 30, 2018 at 9:35 AM, Karl Wright <[email protected]> wrote:
> ' LOG: incomplete message from client' > > This shows a network issue. Did your network configuration change > recently? > > Karl > > > On Mon, Jul 30, 2018 at 9:59 AM Mike Hugo <[email protected]> wrote: > >> Tried a postgres vacuum and also a restart, but the problem persists. >> Here's the log again with some additional logging details added (below) >> >> I tried running the last query from the logs against the database and it >> works fine - I modified it to return a count and that also works. >> >> SELECT count(*) FROM jobs t1 WHERE EXISTS(SELECT 'x' FROM jobpipelines >> WHERE t1.id=ownerid AND transformationname='Tika'); >> count >> ------- >> 39941 >> (1 row) >> >> >> Is 39k jobs a high number? I've run some other instances of Manifold >> with more like 1,000 jobs and those seem to be working fine. That's the >> only thing I can think of that's different between this instance that won't >> start and the others. Any ideas? >> >> Thanks for your help! >> >> Mike >> >> LOG: duration: 0.079 ms parse <unnamed>: SELECT connectionname FROM >> transformationconnections WHERE classname=$1 >> LOG: duration: 0.079 ms bind <unnamed>: SELECT connectionname FROM >> transformationconnections WHERE classname=$1 >> DETAIL: parameters: $1 = 'org.apache.manifoldcf.agents. >> transformation.tika.TikaExtractor' >> LOG: duration: 0.017 ms execute <unnamed>: SELECT connectionname FROM >> transformationconnections WHERE classname=$1 >> DETAIL: parameters: $1 = 'org.apache.manifoldcf.agents. >> transformation.tika.TikaExtractor' >> LOG: duration: 0.039 ms parse <unnamed>: SELECT * FROM agents >> LOG: duration: 0.040 ms bind <unnamed>: SELECT * FROM agents >> LOG: duration: 0.010 ms execute <unnamed>: SELECT * FROM agents >> LOG: duration: 0.084 ms parse <unnamed>: SELECT id FROM jobs t1 WHERE >> EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND >> transformationname=$1) >> LOG: duration: 0.359 ms bind <unnamed>: SELECT id FROM jobs t1 WHERE >> EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND >> transformationname=$1) >> DETAIL: parameters: $1 = 'Tika' >> LOG: duration: 77.622 ms execute <unnamed>: SELECT id FROM jobs t1 >> WHERE EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND >> transformationname=$1) >> DETAIL: parameters: $1 = 'Tika' >> LOG: incomplete message from client >> LOG: disconnection: session time: 0:00:06.574 user=REMOVED >> database=REMOVED host=127.0.0.1 port=45356 >> >2018-07-30 12:36:09,415 [main] ERROR org.apache.manifoldcf.root - >> Exception: This connection has been closed. >> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >> connection has been closed. >> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >> reinterpretException(DBInterfacePostgreSQL.java:627) ~[mcf-core.jar:?] >> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >> ~[mcf-core.jar:?] >> at org.apache.manifoldcf.core.database.Database. >> endTransaction(Database.java:368) ~[mcf-core.jar:?] >> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >> endTransaction(DBInterfacePostgreSQL.java:1236) ~[mcf-core.jar:?] >> at org.apache.manifoldcf.crawler.system.ManifoldCF. >> registerConnectors(ManifoldCF.java:605) ~[mcf-pull-agent.jar:?] >> at org.apache.manifoldcf.crawler.system.ManifoldCF. >> reregisterAllConnectors(ManifoldCF.java:160) ~[mcf-pull-agent.jar:?] >> at org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main( >> ManifoldCFJettyRunner.java:239) [mcf-jetty-runner.jar:?] >> Caused by: org.postgresql.util.PSQLException: This connection has been >> closed. >> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >> ~[postgresql-42.1.3.jar:42.1.3] >> at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) >> ~[postgresql-42.1.3.jar:42.1.3] >> at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) >> ~[postgresql-42.1.3.jar:42.1.3] >> at org.apache.manifoldcf.core.database.Database.execute(Database.java:873) >> ~[mcf-core.jar:?] >> at >> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696) >> ~[mcf-core.jar:?] >> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >> connection has been closed. >> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >> reinterpretException(DBInterfacePostgreSQL.java:627) >> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >> at org.apache.manifoldcf.core.database.Database. >> endTransaction(Database.java:368) >> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >> endTransaction(DBInterfacePostgreSQL.java:1236) >> at org.apache.manifoldcf.crawler.system.ManifoldCF. >> registerConnectors(ManifoldCF.java:605) >> at org.apache.manifoldcf.crawler.system.ManifoldCF. >> reregisterAllConnectors(ManifoldCF.java:160) >> at org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main( >> ManifoldCFJettyRunner.java:239) >> Caused by: org.postgresql.util.PSQLException: This connection has been >> closed. >> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >> at org.postgresql.jdbc.PgConnection.createStatement( >> PgConnection.java:1576) >> at org.postgresql.jdbc.PgConnection.createStatement( >> PgConnection.java:367) >> at org.apache.manifoldcf.core.database.Database.execute( >> Database.java:873) >> at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run( >> Database.java:696) >> LOG: disconnection: session time: 0:00:10.677 user=postgres >> database=template1 host=127.0.0.1 port=45354 >> >> >> >> On Sun, Jul 29, 2018 at 8:09 AM, Karl Wright <[email protected]> wrote: >> >>> It looks to me like your database server is not happy. Maybe it's out >>> of resources? Not sure but a restart may be in order. >>> >>> Karl >>> >>> >>> On Sun, Jul 29, 2018 at 9:06 AM Mike Hugo <[email protected]> wrote: >>> >>>> Recently we started seeing this error when Manifold CF starts up. We >>>> had been running Manifold CF with many web connectors and a few RSS feeds >>>> for a while and it had been working fine. The server got rebooted and >>>> since then we started seeing this error. I'm not sure exactly what >>>> changed. Any ideas as to where to start looking and how to fix this? >>>> >>>> Thanks! >>>> >>>> Mike >>>> >>>> >>>> Initial repository connections already created. >>>> Configuration file successfully read >>>> Successfully unregistered all domains >>>> Successfully unregistered all output connectors >>>> Successfully unregistered all transformation connectors >>>> Successfully unregistered all mapping connectors >>>> Successfully unregistered all authority connectors >>>> Successfully unregistered all repository connectors >>>> WARNING: there is already a transaction in progress >>>> WARNING: there is no transaction in progress >>>> Successfully registered output connector 'org.apache.manifoldcf.agents. >>>> output.solr.SolrConnector' >>>> WARNING: there is already a transaction in progress >>>> WARNING: there is no transaction in progress >>>> Successfully registered output connector 'org.apache.manifoldcf.agents. >>>> output.searchblox.SearchBloxConnector' >>>> WARNING: there is already a transaction in progress >>>> WARNING: there is no transaction in progress >>>> Successfully registered output connector 'org.apache.manifoldcf.agents. >>>> output.opensearchserver.OpenSearchServerConnector' >>>> WARNING: there is already a transaction in progress >>>> WARNING: there is no transaction in progress >>>> Successfully registered output connector 'org.apache.manifoldcf.agents. >>>> output.nullconnector.NullConnector' >>>> WARNING: there is already a transaction in progress >>>> WARNING: there is no transaction in progress >>>> Successfully registered output connector 'org.apache.manifoldcf.agents. >>>> output.kafka.KafkaOutputConnector' >>>> WARNING: there is already a transaction in progress >>>> WARNING: there is no transaction in progress >>>> Successfully registered output connector 'org.apache.manifoldcf.agents. >>>> output.hdfs.HDFSOutputConnector' >>>> WARNING: there is already a transaction in progress >>>> WARNING: there is no transaction in progress >>>> Successfully registered output connector 'org.apache.manifoldcf.agents. >>>> output.gts.GTSConnector' >>>> WARNING: there is already a transaction in progress >>>> WARNING: there is no transaction in progress >>>> Successfully registered output connector 'org.apache.manifoldcf.agents. >>>> output.filesystem.FileOutputConnector' >>>> WARNING: there is already a transaction in progress >>>> WARNING: there is no transaction in progress >>>> Successfully registered output connector 'org.apache.manifoldcf.agents. >>>> output.elasticsearch.ElasticSearchConnector' >>>> WARNING: there is already a transaction in progress >>>> WARNING: there is no transaction in progress >>>> Successfully registered output connector 'org.apache.manifoldcf.agents. >>>> output.amazoncloudsearch.AmazonCloudSearchConnector' >>>> WARNING: there is already a transaction in progress >>>> WARNING: there is no transaction in progress >>>> Successfully registered transformation connector >>>> 'org.apache.manifoldcf.agents.transformation.tikaservice.TikaExtractor' >>>> WARNING: there is already a transaction in progress >>>> LOG: incomplete message from client >>>> >2018-07-29 13:02:06,659 [main] ERROR org.apache.manifoldcf.root - >>>> Exception: This connection has been closed. >>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >>>> connection has been closed. >>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>> reinterpretException(DBInterfacePostgreSQL.java:627) ~[mcf-core.jar:?] >>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>>> ~[mcf-core.jar:?] >>>> at org.apache.manifoldcf.core.database.Database. >>>> endTransaction(Database.java:368) ~[mcf-core.jar:?] >>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>> endTransaction(DBInterfacePostgreSQL.java:1236) ~[mcf-core.jar:?] >>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>> registerConnectors(ManifoldCF.java:605) ~[mcf-pull-agent.jar:?] >>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>> reregisterAllConnectors(ManifoldCF.java:160) ~[mcf-pull-agent.jar:?] >>>> at org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main( >>>> ManifoldCFJettyRunner.java:239) [mcf-jetty-runner.jar:?] >>>> Caused by: org.postgresql.util.PSQLException: This connection has been >>>> closed. >>>> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >>>> ~[postgresql-42.1.3.jar:42.1.3] >>>> at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) >>>> ~[postgresql-42.1.3.jar:42.1.3] >>>> at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) >>>> ~[postgresql-42.1.3.jar:42.1.3] >>>> at org.apache.manifoldcf.core.database.Database.execute(Database.java:873) >>>> ~[mcf-core.jar:?] >>>> at >>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696) >>>> ~[mcf-core.jar:?] >>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >>>> connection has been closed. >>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>> reinterpretException(DBInterfacePostgreSQL.java:627) >>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>>> at org.apache.manifoldcf.core.database.Database. >>>> endTransaction(Database.java:368) >>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>> endTransaction(DBInterfacePostgreSQL.java:1236) >>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>> registerConnectors(ManifoldCF.java:605) >>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>> reregisterAllConnectors(ManifoldCF.java:160) >>>> at org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main( >>>> ManifoldCFJettyRunner.java:239) >>>> Caused by: org.postgresql.util.PSQLException: This connection has been >>>> closed. >>>> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >>>> at org.postgresql.jdbc.PgConnection.createStatement( >>>> PgConnection.java:1576) >>>> at org.postgresql.jdbc.PgConnection.createStatement( >>>> PgConnection.java:367) >>>> at org.apache.manifoldcf.core.database.Database.execute( >>>> Database.java:873) >>>> at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run( >>>> Database.java:696) >>>> >>> >>
