Thanks Karl! I applied both patches and we're back up and running! Thanks so much for the help and the patches!
Mike On Mon, Jul 30, 2018 at 3:32 PM, Karl Wright <[email protected]> wrote: > Ok, attached a second fix. > > Karl > > > On Mon, Jul 30, 2018 at 4:09 PM Karl Wright <[email protected]> wrote: > >> Yes, of course. I overlooked that. Will fix. >> >> Karl >> >> >> On Mon, Jul 30, 2018 at 3:54 PM Mike Hugo <[email protected]> wrote: >> >>> That limit only applies to the list of transformations, not the list of >>> job IDs. If you follow the code into the next method >>> >>> >>>>>> >>> /** Note registration for a batch of transformation connection names. >>> */ >>> protected void noteTransformationConnectionRegistration(List<String> >>> list) >>> throws ManifoldCFException >>> { >>> // Query for the matching jobs, and then for each job potentially >>> adjust the state >>> Long[] jobIDs = jobs.findJobsMatchingTransformations(list); >>> <<<<<< >>> >>> Even if "list" is only 1 item, findJobsMatchingTransformations may >>> return thousands of jobIDs, which is then passed to the query a few lines >>> later: >>> >>> >>>>>> >>> query.append("SELECT ").append(jobs.idField).append(",").append(jobs. >>> statusField) >>> .append(" FROM ").append(jobs.getTableName()).append(" WHERE ") >>> .append(database.buildConjunctionClause(newList,new >>> ClauseDescription[]{ >>> new MultiClause(jobs.idField,jobIDs)})) >>> .append(" FOR UPDATE"); >>> <<<<<< >>> >>> Which generates a query with a large OR clause >>> >>> >>> Mike >>> >>> On Mon, Jul 30, 2018 at 2:44 PM, Karl Wright <[email protected]> wrote: >>> >>>> The limit is applied in the method that calls >>>> noteTransformationConnectionRegistration. >>>> >>>> Here it is: >>>> >>>> >>>>>> >>>> /** Note the registration of a transformation connector used by the >>>> specified connections. >>>> * This method will be called when a connector is registered, on which >>>> the specified >>>> * connections depend. >>>> *@param connectionNames is the set of connection names. >>>> */ >>>> @Override >>>> public void noteTransformationConnectorRegistration(String[] >>>> connectionNames) >>>> throws ManifoldCFException >>>> { >>>> // For each connection, find the corresponding list of jobs. From >>>> these jobs, we want the job id and the status. >>>> List<String> list = new ArrayList<String>(); >>>> int maxCount = database.findConjunctionClauseMax(new >>>> ClauseDescription[]{}); >>>> int currentCount = 0; >>>> int i = 0; >>>> while (i < connectionNames.length) >>>> { >>>> if (currentCount == maxCount) >>>> { >>>> noteTransformationConnectionRegistration(list); >>>> list.clear(); >>>> currentCount = 0; >>>> } >>>> >>>> list.add(connectionNames[i++]); >>>> currentCount++; >>>> } >>>> if (currentCount > 0) >>>> noteTransformationConnectionRegistration(list); >>>> } >>>> <<<<<< >>>> >>>> It looks correct now. Do you see an issue with it? >>>> >>>> Karl >>>> >>>> >>>> On Mon, Jul 30, 2018 at 3:28 PM Mike Hugo <[email protected]> wrote: >>>> >>>>> Nice catch Karl! >>>>> >>>>> I applied that patch, but I'm still getting the same error. >>>>> >>>>> I think the problem is in JobManager.noteTransformationConnectionRe >>>>> gistration >>>>> >>>>> If jobs.findJobsMatchingTransformations(list); returns a large list >>>>> of ids (like it is doing in our case - 39,941 ids ), the generated >>>>> query string still has a large OR clause in it. I don't see >>>>> getMaxOrClause >>>>> applied to the query being built inside noteTransformationConnectionRe >>>>> gistration >>>>> >>>>> >>>>>> >>>>> protected void noteTransformationConnectionRegistration(List<String> >>>>> list) >>>>> throws ManifoldCFException >>>>> { >>>>> // Query for the matching jobs, and then for each job potentially >>>>> adjust the state >>>>> Long[] jobIDs = jobs.findJobsMatchingTransformations(list); >>>>> if (jobIDs.length == 0) >>>>> return; >>>>> >>>>> StringBuilder query = new StringBuilder(); >>>>> ArrayList newList = new ArrayList(); >>>>> >>>>> query.append("SELECT ").append(jobs.idField). >>>>> append(",").append(jobs.statusField) >>>>> .append(" FROM ").append(jobs.getTableName()).append(" WHERE ") >>>>> * .append(database.buildConjunctionClause(newList,new >>>>> ClauseDescription[]{* >>>>> * new MultiClause(jobs.idField,jobIDs)}))* >>>>> .append(" FOR UPDATE"); >>>>> IResultSet set = database.performQuery(query. >>>>> toString(),newList,null,null); >>>>> int i = 0; >>>>> while (i < set.getRowCount()) >>>>> { >>>>> IResultRow row = set.getRow(i++); >>>>> Long jobID = (Long)row.getValue(jobs.idField); >>>>> int statusValue = jobs.stringToStatus((String) >>>>> row.getValue(jobs.statusField)); >>>>> jobs.noteTransformationConnectorRegistration(jobID,statusValue); >>>>> } >>>>> } >>>>> <<<<<< >>>>> >>>>> >>>>> On Mon, Jul 30, 2018 at 1:55 PM, Karl Wright <[email protected]> >>>>> wrote: >>>>> >>>>>> The Postgresql driver supposedly limits this to 25 clauses at a pop: >>>>>> >>>>>> >>>>>> >>>>>> @Override >>>>>> public int getMaxOrClause() >>>>>> { >>>>>> return 25; >>>>>> } >>>>>> >>>>>> /* Calculate the number of values a particular clause can have, >>>>>> given the values for all the other clauses. >>>>>> * For example, if in the expression x AND y AND z, x has 2 values >>>>>> and z has 1, find out how many values x can legally have >>>>>> * when using the buildConjunctionClause() method below. >>>>>> */ >>>>>> @Override >>>>>> public int findConjunctionClauseMax(ClauseDescription[] >>>>>> otherClauseDescriptions) >>>>>> { >>>>>> // This implementation uses "OR" >>>>>> return getMaxOrClause(); >>>>>> } >>>>>> <<<<<< >>>>>> >>>>>> The problem is that there was a cut-and-paste error, with just >>>>>> transformation connections, that defeated the limit. I'll create a >>>>>> ticket >>>>>> and attach a patch. CONNECTORS-1520. >>>>>> >>>>>> Karl >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Jul 30, 2018 at 2:29 PM Karl Wright <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Mike, >>>>>>> >>>>>>> This might be the issue indeed. I'll look into it. >>>>>>> >>>>>>> Karl >>>>>>> >>>>>>> >>>>>>> On Mon, Jul 30, 2018 at 2:26 PM Mike Hugo <[email protected]> wrote: >>>>>>> >>>>>>>> I'm not sure what the solution is yet, but I think I may have found >>>>>>>> the culprit: >>>>>>>> >>>>>>>> JobManager.noteTransformationConnectionRegistration(List<String> >>>>>>>> list) is creating a pretty big query: >>>>>>>> >>>>>>>> SELECT id,status FROM jobs WHERE (id=? OR id=? OR id=? OR id=? >>>>>>>> ........ OR id=?) FOR UPDATE >>>>>>>> >>>>>>>> replace the elipsis with as list of 39,941 ids (it's a huge query >>>>>>>> when it prints out) >>>>>>>> >>>>>>>> It seems that the database doesn't like that query and closes the >>>>>>>> connection before returning with a response. >>>>>>>> >>>>>>>> As I mentioned this instance of manifold has nearly 40,000 web >>>>>>>> crawlers. is that a high number for Manifold to handle? >>>>>>>> >>>>>>>> On Mon, Jul 30, 2018 at 10:58 AM, Karl Wright <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Well, I have absolutely no idea what is wrong and I've never seen >>>>>>>>> anything like that before. But postgres is complaining because the >>>>>>>>> communication with the JDBC client is being interrupted by something. >>>>>>>>> >>>>>>>>> Karl >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Jul 30, 2018 at 10:39 AM Mike Hugo <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> No, and manifold and postgres run on the same host. >>>>>>>>>> >>>>>>>>>> On Mon, Jul 30, 2018 at 9:35 AM, Karl Wright <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> ' LOG: incomplete message from client' >>>>>>>>>>> >>>>>>>>>>> This shows a network issue. Did your network configuration >>>>>>>>>>> change recently? >>>>>>>>>>> >>>>>>>>>>> Karl >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Jul 30, 2018 at 9:59 AM Mike Hugo <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Tried a postgres vacuum and also a restart, but the problem >>>>>>>>>>>> persists. Here's the log again with some additional logging >>>>>>>>>>>> details added >>>>>>>>>>>> (below) >>>>>>>>>>>> >>>>>>>>>>>> I tried running the last query from the logs against the >>>>>>>>>>>> database and it works fine - I modified it to return a count and >>>>>>>>>>>> that also >>>>>>>>>>>> works. >>>>>>>>>>>> >>>>>>>>>>>> SELECT count(*) FROM jobs t1 WHERE EXISTS(SELECT 'x' FROM >>>>>>>>>>>> jobpipelines WHERE t1.id=ownerid AND >>>>>>>>>>>> transformationname='Tika'); >>>>>>>>>>>> count >>>>>>>>>>>> ------- >>>>>>>>>>>> 39941 >>>>>>>>>>>> (1 row) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Is 39k jobs a high number? I've run some other instances of >>>>>>>>>>>> Manifold with more like 1,000 jobs and those seem to be working >>>>>>>>>>>> fine. >>>>>>>>>>>> That's the only thing I can think of that's different between this >>>>>>>>>>>> instance >>>>>>>>>>>> that won't start and the others. Any ideas? >>>>>>>>>>>> >>>>>>>>>>>> Thanks for your help! >>>>>>>>>>>> >>>>>>>>>>>> Mike >>>>>>>>>>>> >>>>>>>>>>>> LOG: duration: 0.079 ms parse <unnamed>: SELECT >>>>>>>>>>>> connectionname FROM transformationconnections WHERE classname=$1 >>>>>>>>>>>> LOG: duration: 0.079 ms bind <unnamed>: SELECT connectionname >>>>>>>>>>>> FROM transformationconnections WHERE classname=$1 >>>>>>>>>>>> DETAIL: parameters: $1 = 'org.apache.manifoldcf.agents. >>>>>>>>>>>> transformation.tika.TikaExtractor' >>>>>>>>>>>> LOG: duration: 0.017 ms execute <unnamed>: SELECT >>>>>>>>>>>> connectionname FROM transformationconnections WHERE classname=$1 >>>>>>>>>>>> DETAIL: parameters: $1 = 'org.apache.manifoldcf.agents. >>>>>>>>>>>> transformation.tika.TikaExtractor' >>>>>>>>>>>> LOG: duration: 0.039 ms parse <unnamed>: SELECT * FROM agents >>>>>>>>>>>> LOG: duration: 0.040 ms bind <unnamed>: SELECT * FROM agents >>>>>>>>>>>> LOG: duration: 0.010 ms execute <unnamed>: SELECT * FROM >>>>>>>>>>>> agents >>>>>>>>>>>> LOG: duration: 0.084 ms parse <unnamed>: SELECT id FROM jobs >>>>>>>>>>>> t1 WHERE EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid >>>>>>>>>>>> AND transformationname=$1) >>>>>>>>>>>> LOG: duration: 0.359 ms bind <unnamed>: SELECT id FROM jobs >>>>>>>>>>>> t1 WHERE EXISTS(SELECT 'x' FROM jobpipelines WHERE t1.id=ownerid >>>>>>>>>>>> AND transformationname=$1) >>>>>>>>>>>> DETAIL: parameters: $1 = 'Tika' >>>>>>>>>>>> LOG: duration: 77.622 ms execute <unnamed>: SELECT id FROM >>>>>>>>>>>> jobs t1 WHERE EXISTS(SELECT 'x' FROM jobpipelines WHERE >>>>>>>>>>>> t1.id=ownerid >>>>>>>>>>>> AND transformationname=$1) >>>>>>>>>>>> DETAIL: parameters: $1 = 'Tika' >>>>>>>>>>>> LOG: incomplete message from client >>>>>>>>>>>> LOG: disconnection: session time: 0:00:06.574 user=REMOVED >>>>>>>>>>>> database=REMOVED host=127.0.0.1 port=45356 >>>>>>>>>>>> >2018-07-30 12:36:09,415 [main] ERROR >>>>>>>>>>>> org.apache.manifoldcf.root - Exception: This connection has been >>>>>>>>>>>> closed. >>>>>>>>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: >>>>>>>>>>>> This connection has been closed. >>>>>>>>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>>>>>>>> reinterpretException(DBInterfacePostgreSQL.java:627) >>>>>>>>>>>> ~[mcf-core.jar:?] >>>>>>>>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>>>>>>>> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>>>>>>>>>>> ~[mcf-core.jar:?] >>>>>>>>>>>> at org.apache.manifoldcf.core.database.Database. >>>>>>>>>>>> endTransaction(Database.java:368) ~[mcf-core.jar:?] >>>>>>>>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>>>>>>>> endTransaction(DBInterfacePostgreSQL.java:1236) >>>>>>>>>>>> ~[mcf-core.jar:?] >>>>>>>>>>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>>>>>>>>>> registerConnectors(ManifoldCF.java:605) ~[mcf-pull-agent.jar:?] >>>>>>>>>>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>>>>>>>>>> reregisterAllConnectors(ManifoldCF.java:160) >>>>>>>>>>>> ~[mcf-pull-agent.jar:?] >>>>>>>>>>>> at org.apache.manifoldcf.jettyrunner. >>>>>>>>>>>> ManifoldCFJettyRunner.main(ManifoldCFJettyRunner.java:239) >>>>>>>>>>>> [mcf-jetty-runner.jar:?] >>>>>>>>>>>> Caused by: org.postgresql.util.PSQLException: This connection >>>>>>>>>>>> has been closed. >>>>>>>>>>>> at >>>>>>>>>>>> org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >>>>>>>>>>>> ~[postgresql-42.1.3.jar:42.1.3] >>>>>>>>>>>> at >>>>>>>>>>>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) >>>>>>>>>>>> ~[postgresql-42.1.3.jar:42.1.3] >>>>>>>>>>>> at >>>>>>>>>>>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) >>>>>>>>>>>> ~[postgresql-42.1.3.jar:42.1.3] >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.manifoldcf.core.database.Database.execute(Database.java:873) >>>>>>>>>>>> ~[mcf-core.jar:?] >>>>>>>>>>>> at org.apache.manifoldcf.core.database.Database$ >>>>>>>>>>>> ExecuteQueryThread.run(Database.java:696) ~[mcf-core.jar:?] >>>>>>>>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: >>>>>>>>>>>> This connection has been closed. >>>>>>>>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>>>>>>>> reinterpretException(DBInterfacePostgreSQL.java:627) >>>>>>>>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>>>>>>>> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>>>>>>>>>>> at org.apache.manifoldcf.core.database.Database. >>>>>>>>>>>> endTransaction(Database.java:368) >>>>>>>>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>>>>>>>> endTransaction(DBInterfacePostgreSQL.java:1236) >>>>>>>>>>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>>>>>>>>>> registerConnectors(ManifoldCF.java:605) >>>>>>>>>>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>>>>>>>>>> reregisterAllConnectors(ManifoldCF.java:160) >>>>>>>>>>>> at org.apache.manifoldcf.jettyrunner. >>>>>>>>>>>> ManifoldCFJettyRunner.main(ManifoldCFJettyRunner.java:239) >>>>>>>>>>>> Caused by: org.postgresql.util.PSQLException: This connection >>>>>>>>>>>> has been closed. >>>>>>>>>>>> at org.postgresql.jdbc.PgConnection.checkClosed( >>>>>>>>>>>> PgConnection.java:766) >>>>>>>>>>>> at org.postgresql.jdbc.PgConnection.createStatement( >>>>>>>>>>>> PgConnection.java:1576) >>>>>>>>>>>> at org.postgresql.jdbc.PgConnection.createStatement( >>>>>>>>>>>> PgConnection.java:367) >>>>>>>>>>>> at org.apache.manifoldcf.core.database.Database.execute( >>>>>>>>>>>> Database.java:873) >>>>>>>>>>>> at org.apache.manifoldcf.core.database.Database$ >>>>>>>>>>>> ExecuteQueryThread.run(Database.java:696) >>>>>>>>>>>> LOG: disconnection: session time: 0:00:10.677 user=postgres >>>>>>>>>>>> database=template1 host=127.0.0.1 port=45354 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Jul 29, 2018 at 8:09 AM, Karl Wright < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> It looks to me like your database server is not happy. Maybe >>>>>>>>>>>>> it's out of resources? Not sure but a restart may be in order. >>>>>>>>>>>>> >>>>>>>>>>>>> Karl >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Jul 29, 2018 at 9:06 AM Mike Hugo <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Recently we started seeing this error when Manifold CF starts >>>>>>>>>>>>>> up. We had been running Manifold CF with many web connectors >>>>>>>>>>>>>> and a few RSS >>>>>>>>>>>>>> feeds for a while and it had been working fine. The server got >>>>>>>>>>>>>> rebooted >>>>>>>>>>>>>> and since then we started seeing this error. I'm not sure >>>>>>>>>>>>>> exactly what >>>>>>>>>>>>>> changed. Any ideas as to where to start looking and how to fix >>>>>>>>>>>>>> this? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Mike >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Initial repository connections already created. >>>>>>>>>>>>>> Configuration file successfully read >>>>>>>>>>>>>> Successfully unregistered all domains >>>>>>>>>>>>>> Successfully unregistered all output connectors >>>>>>>>>>>>>> Successfully unregistered all transformation connectors >>>>>>>>>>>>>> Successfully unregistered all mapping connectors >>>>>>>>>>>>>> Successfully unregistered all authority connectors >>>>>>>>>>>>>> Successfully unregistered all repository connectors >>>>>>>>>>>>>> WARNING: there is already a transaction in progress >>>>>>>>>>>>>> WARNING: there is no transaction in progress >>>>>>>>>>>>>> Successfully registered output connector >>>>>>>>>>>>>> 'org.apache.manifoldcf.agents.output.solr.SolrConnector' >>>>>>>>>>>>>> WARNING: there is already a transaction in progress >>>>>>>>>>>>>> WARNING: there is no transaction in progress >>>>>>>>>>>>>> Successfully registered output connector >>>>>>>>>>>>>> 'org.apache.manifoldcf.agents.output.searchblox. >>>>>>>>>>>>>> SearchBloxConnector' >>>>>>>>>>>>>> WARNING: there is already a transaction in progress >>>>>>>>>>>>>> WARNING: there is no transaction in progress >>>>>>>>>>>>>> Successfully registered output connector >>>>>>>>>>>>>> 'org.apache.manifoldcf.agents.output.opensearchserver. >>>>>>>>>>>>>> OpenSearchServerConnector' >>>>>>>>>>>>>> WARNING: there is already a transaction in progress >>>>>>>>>>>>>> WARNING: there is no transaction in progress >>>>>>>>>>>>>> Successfully registered output connector >>>>>>>>>>>>>> 'org.apache.manifoldcf.agents.output.nullconnector. >>>>>>>>>>>>>> NullConnector' >>>>>>>>>>>>>> WARNING: there is already a transaction in progress >>>>>>>>>>>>>> WARNING: there is no transaction in progress >>>>>>>>>>>>>> Successfully registered output connector >>>>>>>>>>>>>> 'org.apache.manifoldcf.agents.output.kafka. >>>>>>>>>>>>>> KafkaOutputConnector' >>>>>>>>>>>>>> WARNING: there is already a transaction in progress >>>>>>>>>>>>>> WARNING: there is no transaction in progress >>>>>>>>>>>>>> Successfully registered output connector >>>>>>>>>>>>>> 'org.apache.manifoldcf.agents.output.hdfs. >>>>>>>>>>>>>> HDFSOutputConnector' >>>>>>>>>>>>>> WARNING: there is already a transaction in progress >>>>>>>>>>>>>> WARNING: there is no transaction in progress >>>>>>>>>>>>>> Successfully registered output connector >>>>>>>>>>>>>> 'org.apache.manifoldcf.agents.output.gts.GTSConnector' >>>>>>>>>>>>>> WARNING: there is already a transaction in progress >>>>>>>>>>>>>> WARNING: there is no transaction in progress >>>>>>>>>>>>>> Successfully registered output connector >>>>>>>>>>>>>> 'org.apache.manifoldcf.agents.output.filesystem. >>>>>>>>>>>>>> FileOutputConnector' >>>>>>>>>>>>>> WARNING: there is already a transaction in progress >>>>>>>>>>>>>> WARNING: there is no transaction in progress >>>>>>>>>>>>>> Successfully registered output connector >>>>>>>>>>>>>> 'org.apache.manifoldcf.agents.output.elasticsearch. >>>>>>>>>>>>>> ElasticSearchConnector' >>>>>>>>>>>>>> WARNING: there is already a transaction in progress >>>>>>>>>>>>>> WARNING: there is no transaction in progress >>>>>>>>>>>>>> Successfully registered output connector >>>>>>>>>>>>>> 'org.apache.manifoldcf.agents.output.amazoncloudsearch. >>>>>>>>>>>>>> AmazonCloudSearchConnector' >>>>>>>>>>>>>> WARNING: there is already a transaction in progress >>>>>>>>>>>>>> WARNING: there is no transaction in progress >>>>>>>>>>>>>> Successfully registered transformation connector >>>>>>>>>>>>>> 'org.apache.manifoldcf.agents.transformation.tikaservice. >>>>>>>>>>>>>> TikaExtractor' >>>>>>>>>>>>>> WARNING: there is already a transaction in progress >>>>>>>>>>>>>> LOG: incomplete message from client >>>>>>>>>>>>>> >2018-07-29 13:02:06,659 [main] ERROR >>>>>>>>>>>>>> org.apache.manifoldcf.root - Exception: This connection has been >>>>>>>>>>>>>> closed. >>>>>>>>>>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: >>>>>>>>>>>>>> This connection has been closed. >>>>>>>>>>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>>>>>>>>>> reinterpretException(DBInterfacePostgreSQL.java:627) >>>>>>>>>>>>>> ~[mcf-core.jar:?] >>>>>>>>>>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>>>>>>>>>> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>>>>>>>>>>>>> ~[mcf-core.jar:?] >>>>>>>>>>>>>> at org.apache.manifoldcf.core.database.Database. >>>>>>>>>>>>>> endTransaction(Database.java:368) ~[mcf-core.jar:?] >>>>>>>>>>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>>>>>>>>>> endTransaction(DBInterfacePostgreSQL.java:1236) >>>>>>>>>>>>>> ~[mcf-core.jar:?] >>>>>>>>>>>>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>>>>>>>>>>>> registerConnectors(ManifoldCF.java:605) >>>>>>>>>>>>>> ~[mcf-pull-agent.jar:?] >>>>>>>>>>>>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>>>>>>>>>>>> reregisterAllConnectors(ManifoldCF.java:160) >>>>>>>>>>>>>> ~[mcf-pull-agent.jar:?] >>>>>>>>>>>>>> at org.apache.manifoldcf.jettyrunner. >>>>>>>>>>>>>> ManifoldCFJettyRunner.main(ManifoldCFJettyRunner.java:239) >>>>>>>>>>>>>> [mcf-jetty-runner.jar:?] >>>>>>>>>>>>>> Caused by: org.postgresql.util.PSQLException: This >>>>>>>>>>>>>> connection has been closed. >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:766) >>>>>>>>>>>>>> ~[postgresql-42.1.3.jar:42.1.3] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:1576) >>>>>>>>>>>>>> ~[postgresql-42.1.3.jar:42.1.3] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:367) >>>>>>>>>>>>>> ~[postgresql-42.1.3.jar:42.1.3] >>>>>>>>>>>>>> at >>>>>>>>>>>>>> org.apache.manifoldcf.core.database.Database.execute(Database.java:873) >>>>>>>>>>>>>> ~[mcf-core.jar:?] >>>>>>>>>>>>>> at org.apache.manifoldcf.core.database.Database$ >>>>>>>>>>>>>> ExecuteQueryThread.run(Database.java:696) ~[mcf-core.jar:?] >>>>>>>>>>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: >>>>>>>>>>>>>> This connection has been closed. >>>>>>>>>>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>>>>>>>>>> reinterpretException(DBInterfacePostgreSQL.java:627) >>>>>>>>>>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>>>>>>>>>> rollbackCurrentTransaction(DBInterfacePostgreSQL.java:1296) >>>>>>>>>>>>>> at org.apache.manifoldcf.core.database.Database. >>>>>>>>>>>>>> endTransaction(Database.java:368) >>>>>>>>>>>>>> at org.apache.manifoldcf.core.database.DBInterfacePostgreSQL. >>>>>>>>>>>>>> endTransaction(DBInterfacePostgreSQL.java:1236) >>>>>>>>>>>>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>>>>>>>>>>>> registerConnectors(ManifoldCF.java:605) >>>>>>>>>>>>>> at org.apache.manifoldcf.crawler.system.ManifoldCF. >>>>>>>>>>>>>> reregisterAllConnectors(ManifoldCF.java:160) >>>>>>>>>>>>>> at org.apache.manifoldcf.jettyrunner. >>>>>>>>>>>>>> ManifoldCFJettyRunner.main(ManifoldCFJettyRunner.java:239) >>>>>>>>>>>>>> Caused by: org.postgresql.util.PSQLException: This >>>>>>>>>>>>>> connection has been closed. >>>>>>>>>>>>>> at org.postgresql.jdbc.PgConnection.checkClosed( >>>>>>>>>>>>>> PgConnection.java:766) >>>>>>>>>>>>>> at org.postgresql.jdbc.PgConnection.createStatement( >>>>>>>>>>>>>> PgConnection.java:1576) >>>>>>>>>>>>>> at org.postgresql.jdbc.PgConnection.createStatement( >>>>>>>>>>>>>> PgConnection.java:367) >>>>>>>>>>>>>> at org.apache.manifoldcf.core.database.Database.execute( >>>>>>>>>>>>>> Database.java:873) >>>>>>>>>>>>>> at org.apache.manifoldcf.core.database.Database$ >>>>>>>>>>>>>> ExecuteQueryThread.run(Database.java:696) >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>> >>>
