I'm not sure what the solution is yet, but I think I may have found the culprit:
JobManager.noteTransformationConnectionRegistration(List<String> list) is creating a pretty big query: SELECT id,status FROM jobs WHERE (id=? OR id=? OR id=? OR id=? ........ OR id=?) FOR UPDATE replace the elipsis with as list of 39,941 ids (it's a huge query when it prints out) It seems that the database doesn't like that query and closes the connection before returning with a response. As I mentioned this instance of manifold has nearly 40,000 web crawlers. is that a high number for Manifold to handle? On Mon, Jul 30, 2018 at 10:58 AM, Karl Wright <[email protected]> wrote: > Well, I have absolutely no idea what is wrong and I've never seen anything > like that before. But postgres is complaining because the communication > with the JDBC client is being interrupted by something. > > Karl > > > On Mon, Jul 30, 2018 at 10:39 AM Mike Hugo <[email protected]> wrote: > >> No, and manifold and postgres run on the same host. >> >> On Mon, Jul 30, 2018 at 9:35 AM, Karl Wright <[email protected]> wrote: >> >>> ' LOG: incomplete message from client' >>> >>> This shows a network issue. Did your network configuration change >>> recently? >>> >>> Karl >>> >>> >>> On Mon, Jul 30, 2018 at 9:59 AM Mike Hugo <[email protected]> wrote: >>> >>>> Tried a postgres vacuum and also a restart, but the problem persists. >>>> Here's the log again with some additional logging details added (below) >>>> >>>> I tried running the last query from the logs against the database and >>>> it works fine - I modified it to return a count and that also works. >>>> >>>> SELECT count(*) FROM jobs t1 WHERE EXISTS(SELECT 'x' FROM jobpipelines >>>> WHERE t1.id=ownerid AND transformationname='Tika'); >>>> count >>>> ------- >>>> 39941 >>>> (1 row) >>>> >>>> >>>> Is 39k jobs a high number? I've run some other instances of Manifold >>>> with more like 1,000 jobs and those seem to be working fine. That's the >>>> only thing I can think of that's different between this instance that won't >>>> start and the others. Any ideas? >>>> >>>> Thanks for your help! >>>> >>>> Mike >>>> >>>> LOG: duration: 0.079 ms parse <unnamed>: SELECT connectionname FROM >>>> transformationconnections WHERE classname=$1 >>>> LOG: duration: 0.079 ms bind <unnamed>: SELECT connectionname FROM >>>> transformationconnections WHERE classname=$1 >>>> DETAIL: parameters: $1 = 'org.apache.manifoldcf.agents. >>>> transformation.tika.TikaExtractor' >>>> LOG: duration: 0.017 ms execute <unnamed>: SELECT connectionname FROM >>>> transformationconnections WHERE classname=$1 >>>> DETAIL: parameters: $1 = 'org.apache.manifoldcf.agents. >>>> transformation.tika.TikaExtractor' >>>> LOG: duration: 0.039 ms parse <unnamed>: SELECT * FROM agents >>>> LOG: duration: 0.040 ms bind <unnamed>: SELECT * FROM agents >>>> LOG: duration: 0.010 ms execute <unnamed>: SELECT * FROM agents >>>> LOG: duration: 0.084 ms parse <unnamed>: SELECT id FROM jobs t1 WHERE >>>> EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND >>>> transformationname=$1) >>>> LOG: duration: 0.359 ms bind <unnamed>: SELECT id FROM jobs t1 WHERE >>>> EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND >>>> transformationname=$1) >>>> DETAIL: parameters: $1 = 'Tika' >>>> LOG: duration: 77.622 ms execute <unnamed>: SELECT id FROM jobs t1 >>>> WHERE EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid AND >>>> transformationname=$1) >>>> DETAIL: parameters: $1 = 'Tika' >>>> LOG: incomplete message from client >>>> LOG: disconnection: session time: 0:00:06.574 user=REMOVED >>>> database=REMOVED host=127.0.0.1 port=45356 >>>> >2018-07-30 12:36:09,415 [main] ERROR org.apache.manifoldcf.root - >>>> Exception: This connection has been closed. >>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >>>> connection has been closed. >>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>> reinterpretException(DBInterfacePostgreSQL.java:627) ~[mcf-core.jar:?] >>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>>> ~[mcf-core.jar:?] >>>> at org.apache.manifoldcf.core.database.Database. >>>> endTransaction(Database.java:368) ~[mcf-core.jar:?] >>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>> endTransaction(DBInterfacePostgreSQL.java:1236) ~[mcf-core.jar:?] >>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>> registerConnectors(ManifoldCF.java:605) ~[mcf-pull-agent.jar:?] >>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>> reregisterAllConnectors(ManifoldCF.java:160) ~[mcf-pull-agent.jar:?] >>>> at org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main( >>>> ManifoldCFJettyRunner.java:239) [mcf-jetty-runner.jar:?] >>>> Caused by: org.postgresql.util.PSQLException: This connection has been >>>> closed. >>>> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >>>> ~[postgresql-42.1.3.jar:42.1.3] >>>> at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) >>>> ~[postgresql-42.1.3.jar:42.1.3] >>>> at org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) >>>> ~[postgresql-42.1.3.jar:42.1.3] >>>> at org.apache.manifoldcf.core.database.Database.execute(Database.java:873) >>>> ~[mcf-core.jar:?] >>>> at >>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:696) >>>> ~[mcf-core.jar:?] >>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >>>> connection has been closed. >>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>> reinterpretException(DBInterfacePostgreSQL.java:627) >>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>>> at org.apache.manifoldcf.core.database.Database. >>>> endTransaction(Database.java:368) >>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>> endTransaction(DBInterfacePostgreSQL.java:1236) >>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>> registerConnectors(ManifoldCF.java:605) >>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>> reregisterAllConnectors(ManifoldCF.java:160) >>>> at org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main( >>>> ManifoldCFJettyRunner.java:239) >>>> Caused by: org.postgresql.util.PSQLException: This connection has been >>>> closed. >>>> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >>>> at org.postgresql.jdbc.PgConnection.createStatement( >>>> PgConnection.java:1576) >>>> at org.postgresql.jdbc.PgConnection.createStatement( >>>> PgConnection.java:367) >>>> at org.apache.manifoldcf.core.database.Database.execute( >>>> Database.java:873) >>>> at org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run( >>>> Database.java:696) >>>> LOG: disconnection: session time: 0:00:10.677 user=postgres >>>> database=template1 host=127.0.0.1 port=45354 >>>> >>>> >>>> >>>> On Sun, Jul 29, 2018 at 8:09 AM, Karl Wright <[email protected]> >>>> wrote: >>>> >>>>> It looks to me like your database server is not happy. Maybe it's out >>>>> of resources? Not sure but a restart may be in order. >>>>> >>>>> Karl >>>>> >>>>> >>>>> On Sun, Jul 29, 2018 at 9:06 AM Mike Hugo <[email protected]> wrote: >>>>> >>>>>> Recently we started seeing this error when Manifold CF starts up. We >>>>>> had been running Manifold CF with many web connectors and a few RSS feeds >>>>>> for a while and it had been working fine. The server got rebooted and >>>>>> since then we started seeing this error. I'm not sure exactly what >>>>>> changed. Any ideas as to where to start looking and how to fix this? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Mike >>>>>> >>>>>> >>>>>> Initial repository connections already created. >>>>>> Configuration file successfully read >>>>>> Successfully unregistered all domains >>>>>> Successfully unregistered all output connectors >>>>>> Successfully unregistered all transformation connectors >>>>>> Successfully unregistered all mapping connectors >>>>>> Successfully unregistered all authority connectors >>>>>> Successfully unregistered all repository connectors >>>>>> WARNING: there is already a transaction in progress >>>>>> WARNING: there is no transaction in progress >>>>>> Successfully registered output connector >>>>>> 'org.apache.manifoldcf.agents.output.solr.SolrConnector' >>>>>> WARNING: there is already a transaction in progress >>>>>> WARNING: there is no transaction in progress >>>>>> Successfully registered output connector >>>>>> 'org.apache.manifoldcf.agents.output.searchblox.SearchBloxConnector' >>>>>> WARNING: there is already a transaction in progress >>>>>> WARNING: there is no transaction in progress >>>>>> Successfully registered output connector >>>>>> 'org.apache.manifoldcf.agents.output.opensearchserver. >>>>>> OpenSearchServerConnector' >>>>>> WARNING: there is already a transaction in progress >>>>>> WARNING: there is no transaction in progress >>>>>> Successfully registered output connector >>>>>> 'org.apache.manifoldcf.agents.output.nullconnector.NullConnector' >>>>>> WARNING: there is already a transaction in progress >>>>>> WARNING: there is no transaction in progress >>>>>> Successfully registered output connector >>>>>> 'org.apache.manifoldcf.agents.output.kafka.KafkaOutputConnector' >>>>>> WARNING: there is already a transaction in progress >>>>>> WARNING: there is no transaction in progress >>>>>> Successfully registered output connector >>>>>> 'org.apache.manifoldcf.agents.output.hdfs.HDFSOutputConnector' >>>>>> WARNING: there is already a transaction in progress >>>>>> WARNING: there is no transaction in progress >>>>>> Successfully registered output connector >>>>>> 'org.apache.manifoldcf.agents.output.gts.GTSConnector' >>>>>> WARNING: there is already a transaction in progress >>>>>> WARNING: there is no transaction in progress >>>>>> Successfully registered output connector >>>>>> 'org.apache.manifoldcf.agents.output.filesystem.FileOutputConnector' >>>>>> WARNING: there is already a transaction in progress >>>>>> WARNING: there is no transaction in progress >>>>>> Successfully registered output connector >>>>>> 'org.apache.manifoldcf.agents.output.elasticsearch. >>>>>> ElasticSearchConnector' >>>>>> WARNING: there is already a transaction in progress >>>>>> WARNING: there is no transaction in progress >>>>>> Successfully registered output connector >>>>>> 'org.apache.manifoldcf.agents.output.amazoncloudsearch. >>>>>> AmazonCloudSearchConnector' >>>>>> WARNING: there is already a transaction in progress >>>>>> WARNING: there is no transaction in progress >>>>>> Successfully registered transformation connector >>>>>> 'org.apache.manifoldcf.agents.transformation.tikaservice. >>>>>> TikaExtractor' >>>>>> WARNING: there is already a transaction in progress >>>>>> LOG: incomplete message from client >>>>>> >2018-07-29 13:02:06,659 [main] ERROR org.apache.manifoldcf.root - >>>>>> Exception: This connection has been closed. >>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >>>>>> connection has been closed. >>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>> reinterpretException(DBInterfacePostgreSQL.java:627) >>>>>> ~[mcf-core.jar:?] >>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>>>>> ~[mcf-core.jar:?] >>>>>> at org.apache.manifoldcf.core.database.Database. >>>>>> endTransaction(Database.java:368) ~[mcf-core.jar:?] >>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>> endTransaction(DBInterfacePostgreSQL.java:1236) ~[mcf-core.jar:?] >>>>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>>>> registerConnectors(ManifoldCF.java:605) ~[mcf-pull-agent.jar:?] >>>>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>>>> reregisterAllConnectors(ManifoldCF.java:160) ~[mcf-pull-agent.jar:?] >>>>>> at org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main( >>>>>> ManifoldCFJettyRunner.java:239) [mcf-jetty-runner.jar:?] >>>>>> Caused by: org.postgresql.util.PSQLException: This connection has >>>>>> been closed. >>>>>> at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >>>>>> ~[postgresql-42.1.3.jar:42.1.3] >>>>>> at >>>>>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) >>>>>> ~[postgresql-42.1.3.jar:42.1.3] >>>>>> at >>>>>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) >>>>>> ~[postgresql-42.1.3.jar:42.1.3] >>>>>> at >>>>>> org.apache.manifoldcf.core.database.Database.execute(Database.java:873) >>>>>> ~[mcf-core.jar:?] >>>>>> at org.apache.manifoldcf.core.database.Database$ >>>>>> ExecuteQueryThread.run(Database.java:696) ~[mcf-core.jar:?] >>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: This >>>>>> connection has been closed. >>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>> reinterpretException(DBInterfacePostgreSQL.java:627) >>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>>>>> at org.apache.manifoldcf.core.database.Database. >>>>>> endTransaction(Database.java:368) >>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>> endTransaction(DBInterfacePostgreSQL.java:1236) >>>>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>>>> registerConnectors(ManifoldCF.java:605) >>>>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>>>> reregisterAllConnectors(ManifoldCF.java:160) >>>>>> at org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner.main( >>>>>> ManifoldCFJettyRunner.java:239) >>>>>> Caused by: org.postgresql.util.PSQLException: This connection has >>>>>> been closed. >>>>>> at org.postgresql.jdbc.PgConnection.checkClosed( >>>>>> PgConnection.java:766) >>>>>> at org.postgresql.jdbc.PgConnection.createStatement( >>>>>> PgConnection.java:1576) >>>>>> at org.postgresql.jdbc.PgConnection.createStatement( >>>>>> PgConnection.java:367) >>>>>> at org.apache.manifoldcf.core.database.Database.execute( >>>>>> Database.java:873) >>>>>> at org.apache.manifoldcf.core.database.Database$ >>>>>> ExecuteQueryThread.run(Database.java:696) >>>>>> >>>>> >>>> >>
