Well, I have absolutely no idea what is wrong and I've never seen anything like that before. But postgres is complaining because the communication with the JDBC client is being interrupted by something.
Karl On Mon, Jul 30, 2018 at 10:39 AM Mike Hugo <[email protected]> wrote: > No, and manifold and postgres run on the same host. > > On Mon, Jul 30, 2018 at 9:35 AM, Karl Wright <[email protected]> wrote: > >> ' LOG: incomplete message from client' >> >> This shows a network issue. Did your network configuration change >> recently? >> >> Karl >> >> >> On Mon, Jul 30, 2018 at 9:59 AM Mike Hugo <[email protected]> wrote: >> >>> Tried a postgres vacuum and also a restart, but the problem persists. >>> Here's the log again with some additional logging details added (below) >>> >>> I tried running the last query from the logs against the database and it >>> works fine - I modified it to return a count and that also works. >>> >>> SELECT count(*) FROM jobs t1 WHERE EXISTS(SELECT 'x' FROM jobpipelines >>> WHERE t1.id=ownerid AND transformationname='Tika'); >>> count >>> ------- >>> 39941 >>> (1 row) >>> >>> >>> Is 39k jobs a high number? I've run some other instances of Manifold >>> with more like 1,000 jobs and those seem to be working fine. That's the >>> only thing I can think of that's different between this instance that won't >>> start and the others. Any ideas? >>> >>> Thanks for your help! >>> >>> Mike >>> >>> LOG: duration: 0.079 ms parse <unnamed>: SELECT connectionname FROM >>> transformationconnections WHERE classname=$1 >>> LOG: duration: 0.079 ms bind <unnamed>: SELECT connectionname FROM >>> transformationconnections WHERE classname=$1 >>> DETAIL: parameters: $1 = >>> 'org.apache.manifoldcf.agents.transformation.tika.TikaExtractor' >>> LOG: duration: 0.017 ms execute <unnamed>: SELECT connectionname FROM >>> transformationconnections WHERE classname=$1 >>> DETAIL: parameters: $1 = >>> 'org.apache.manifoldcf.agents.transformation.tika.TikaExtractor' >>> LOG: duration: 0.039 ms parse <unnamed>: SELECT * FROM agents >>> LOG: duration: 0.040 ms bind <unnamed>: SELECT * FROM agents >>> LOG: duration: 0.010 ms execute <unnamed>: SELECT * FROM agents >>> LOG: duration: 0.084 ms parse <unnamed>: SELECT id FROM jobs t1 WHERE >>> EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND >>> transformationname=$1) >>> LOG: duration: 0.359 ms bind <unnamed>: SELECT id FROM jobs t1 WHERE >>> EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND >>> transformationname=$1) >>> DETAIL: parameters: $1 = 'Tika' >>> LOG: duration: 77.622 ms execute <unnamed>: SELECT id FROM jobs t1 >>> WHERE EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND >>> transformationname=$1) >>> DETAIL: parameters: $1 = 'Tika' >>> LOG: incomplete message from client >>> LOG: disconnection: session time: 0:00:06.574 user=REMOVED >>> database=REMOVED host=127.0.0.1 port=45356 >>> >2018-07-30 12:36:09,415 [main] ERROR org.apache.manifoldcf.root - >>> Exception: This connection has been closed. >>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >>> connection has been closed. >>> at >>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.reinterpretException(DBInterfacePostgreSQL.java:627) >>> ~[mcf-core.jar:?] >>> at >>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>> ~[mcf-core.jar:?] >>> at >>> org.apache.manifoldcf.core.database.Database.endTransaction(Database.java:368) >>> ~[mcf-core.jar:?] >>> at >>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.endTransaction(DBInterfacePostgreSQL.java:1236) >>> ~[mcf-core.jar:?] >>> at >>> org.apache.manifoldcf.crawler.system.ManifoldCF.registerConnectors(ManifoldCF.java:605) >>> ~[mcf-pull-agent.jar:?] >>> at >>> org.apache.manifoldcf.crawler.system.ManifoldCF.reregisterAllConnectors(ManifoldCF.java:160) >>> ~[mcf-pull-agent.jar:?] >>> at >>> org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main(ManifoldCFJettyRunner.java:239) >>> [mcf-jetty-runner.jar:?] >>> Caused by: org.postgresql.util.PSQLException: This connection has been >>> closed. >>> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >>> ~[postgresql-42.1.3.jar:42.1.3] >>> at >>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) >>> ~[postgresql-42.1.3.jar:42.1.3] >>> at >>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) >>> ~[postgresql-42.1.3.jar:42.1.3] >>> at >>> org.apache.manifoldcf.core.database.Database.execute(Database.java:873) >>> ~[mcf-core.jar:?] >>> at >>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696) >>> ~[mcf-core.jar:?] >>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >>> connection has been closed. >>> at >>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.reinterpretException(DBInterfacePostgreSQL.java:627) >>> at >>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>> at >>> org.apache.manifoldcf.core.database.Database.endTransaction(Database.java:368) >>> at >>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.endTransaction(DBInterfacePostgreSQL.java:1236) >>> at >>> org.apache.manifoldcf.crawler.system.ManifoldCF.registerConnectors(ManifoldCF.java:605) >>> at >>> org.apache.manifoldcf.crawler.system.ManifoldCF.reregisterAllConnectors(ManifoldCF.java:160) >>> at >>> org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main(ManifoldCFJettyRunner.java:239) >>> Caused by: org.postgresql.util.PSQLException: This connection has been >>> closed. >>> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >>> at >>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) >>> at >>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) >>> at >>> org.apache.manifoldcf.core.database.Database.execute(Database.java:873) >>> at >>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696) >>> LOG: disconnection: session time: 0:00:10.677 user=postgres >>> database=template1 host=127.0.0.1 port=45354 >>> >>> >>> >>> On Sun, Jul 29, 2018 at 8:09 AM, Karl Wright <[email protected]> wrote: >>> >>>> It looks to me like your database server is not happy. Maybe it's out >>>> of resources? Not sure but a restart may be in order. >>>> >>>> Karl >>>> >>>> >>>> On Sun, Jul 29, 2018 at 9:06 AM Mike Hugo <[email protected]> wrote: >>>> >>>>> Recently we started seeing this error when Manifold CF starts up. We >>>>> had been running Manifold CF with many web connectors and a few RSS feeds >>>>> for a while and it had been working fine. The server got rebooted and >>>>> since then we started seeing this error. I'm not sure exactly what >>>>> changed. Any ideas as to where to start looking and how to fix this? >>>>> >>>>> Thanks! >>>>> >>>>> Mike >>>>> >>>>> >>>>> Initial repository connections already created. >>>>> Configuration file successfully read >>>>> Successfully unregistered all domains >>>>> Successfully unregistered all output connectors >>>>> Successfully unregistered all transformation connectors >>>>> Successfully unregistered all mapping connectors >>>>> Successfully unregistered all authority connectors >>>>> Successfully unregistered all repository connectors >>>>> WARNING: there is already a transaction in progress >>>>> WARNING: there is no transaction in progress >>>>> Successfully registered output connector >>>>> 'org.apache.manifoldcf.agents.output.solr.SolrConnector' >>>>> WARNING: there is already a transaction in progress >>>>> WARNING: there is no transaction in progress >>>>> Successfully registered output connector >>>>> 'org.apache.manifoldcf.agents.output.searchblox.SearchBloxConnector' >>>>> WARNING: there is already a transaction in progress >>>>> WARNING: there is no transaction in progress >>>>> Successfully registered output connector >>>>> 'org.apache.manifoldcf.agents.output.opensearchserver.OpenSearchServerConnector' >>>>> WARNING: there is already a transaction in progress >>>>> WARNING: there is no transaction in progress >>>>> Successfully registered output connector >>>>> 'org.apache.manifoldcf.agents.output.nullconnector.NullConnector' >>>>> WARNING: there is already a transaction in progress >>>>> WARNING: there is no transaction in progress >>>>> Successfully registered output connector >>>>> 'org.apache.manifoldcf.agents.output.kafka.KafkaOutputConnector' >>>>> WARNING: there is already a transaction in progress >>>>> WARNING: there is no transaction in progress >>>>> Successfully registered output connector >>>>> 'org.apache.manifoldcf.agents.output.hdfs.HDFSOutputConnector' >>>>> WARNING: there is already a transaction in progress >>>>> WARNING: there is no transaction in progress >>>>> Successfully registered output connector >>>>> 'org.apache.manifoldcf.agents.output.gts.GTSConnector' >>>>> WARNING: there is already a transaction in progress >>>>> WARNING: there is no transaction in progress >>>>> Successfully registered output connector >>>>> 'org.apache.manifoldcf.agents.output.filesystem.FileOutputConnector' >>>>> WARNING: there is already a transaction in progress >>>>> WARNING: there is no transaction in progress >>>>> Successfully registered output connector >>>>> 'org.apache.manifoldcf.agents.output.elasticsearch.ElasticSearchConnector' >>>>> WARNING: there is already a transaction in progress >>>>> WARNING: there is no transaction in progress >>>>> Successfully registered output connector >>>>> 'org.apache.manifoldcf.agents.output.amazoncloudsearch.AmazonCloudSearchConnector' >>>>> WARNING: there is already a transaction in progress >>>>> WARNING: there is no transaction in progress >>>>> Successfully registered transformation connector >>>>> 'org.apache.manifoldcf.agents.transformation.tikaservice.TikaExtractor' >>>>> WARNING: there is already a transaction in progress >>>>> LOG: incomplete message from client >>>>> >2018-07-29 13:02:06,659 [main] ERROR org.apache.manifoldcf.root - >>>>> Exception: This connection has been closed. >>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >>>>> connection has been closed. >>>>> at >>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.reinterpretException(DBInterfacePostgreSQL.java:627) >>>>> ~[mcf-core.jar:?] >>>>> at >>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>>>> ~[mcf-core.jar:?] >>>>> at >>>>> org.apache.manifoldcf.core.database.Database.endTransaction(Database.java:368) >>>>> ~[mcf-core.jar:?] >>>>> at >>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.endTransaction(DBInterfacePostgreSQL.java:1236) >>>>> ~[mcf-core.jar:?] >>>>> at >>>>> org.apache.manifoldcf.crawler.system.ManifoldCF.registerConnectors(ManifoldCF.java:605) >>>>> ~[mcf-pull-agent.jar:?] >>>>> at >>>>> org.apache.manifoldcf.crawler.system.ManifoldCF.reregisterAllConnectors(ManifoldCF.java:160) >>>>> ~[mcf-pull-agent.jar:?] >>>>> at >>>>> org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main(ManifoldCFJettyRunner.java:239) >>>>> [mcf-jetty-runner.jar:?] >>>>> Caused by: org.postgresql.util.PSQLException: This connection has been >>>>> closed. >>>>> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >>>>> ~[postgresql-42.1.3.jar:42.1.3] >>>>> at >>>>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) >>>>> ~[postgresql-42.1.3.jar:42.1.3] >>>>> at >>>>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) >>>>> ~[postgresql-42.1.3.jar:42.1.3] >>>>> at >>>>> org.apache.manifoldcf.core.database.Database.execute(Database.java:873) >>>>> ~[mcf-core.jar:?] >>>>> at >>>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696) >>>>> ~[mcf-core.jar:?] >>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >>>>> connection has been closed. >>>>> at >>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.reinterpretException(DBInterfacePostgreSQL.java:627) >>>>> at >>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>>>> at >>>>> org.apache.manifoldcf.core.database.Database.endTransaction(Database.java:368) >>>>> at >>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.endTransaction(DBInterfacePostgreSQL.java:1236) >>>>> at >>>>> org.apache.manifoldcf.crawler.system.ManifoldCF.registerConnectors(ManifoldCF.java:605) >>>>> at >>>>> org.apache.manifoldcf.crawler.system.ManifoldCF.reregisterAllConnectors(ManifoldCF.java:160) >>>>> at >>>>> org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main(ManifoldCFJettyRunner.java:239) >>>>> Caused by: org.postgresql.util.PSQLException: This connection has been >>>>> closed. >>>>> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >>>>> at >>>>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) >>>>> at >>>>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) >>>>> at >>>>> org.apache.manifoldcf.core.database.Database.execute(Database.java:873) >>>>> at >>>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696) >>>>> >>>> >>> >
