[
https://issues.apache.org/jira/browse/CONNECTORS-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135407#comment-13135407
]
Karl Wright commented on CONNECTORS-279:
----------------------------------------
With the 9.1 JDBC driver, things are even worse. The crawl does not complete
because 3 queries seem to get forever "stuck". They are not the same query
either:
{code}
"Worker thread '16'" daemon prio=6 tid=0x0547f000 nid=0x1b90 in Object.wait()
[0x0609f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1143)
- locked <0x29c514c0> (a
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
at java.lang.Thread.join(Thread.java:1196)
at
org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:453)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.startATransaction(DBInterfacePostgreSQL.java:1134)
at
org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:235)
at
org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:218)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1140)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:639)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.beginTransaction(DBInterfacePostgreSQL.java:1072)
at
org.apache.manifoldcf.crawler.jobs.JobManager.finishDocuments(JobManager.java:3902)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:567)
{code}
{code}
"Worker thread '12'" daemon prio=6 tid=0x0547d800 nid=0x1170 in Object.wait()
[0x05f5f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1143)
- locked <0x29c51608> (a
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
at java.lang.Thread.join(Thread.java:1196)
at
org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:453)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.startATransaction(DBInterfacePostgreSQL.java:1134)
at
org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:235)
at
org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:218)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1140)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:811)
at
org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:229)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.noteDocumentIngest(IncrementalIngester.java:1358)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:495)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:364)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1577)
at
org.apache.manifoldcf.crawler.connectors.rss.RSSConnector.processDocuments(RSSConnector.java:1470)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:561)
{code}
> Postgresql load test job delete document cleanup fails sometimes and leaves
> orphaned documents
> ----------------------------------------------------------------------------------------------
>
> Key: CONNECTORS-279
> URL: https://issues.apache.org/jira/browse/CONNECTORS-279
> Project: ManifoldCF
> Issue Type: Bug
> Components: Framework crawler agent
> Affects Versions: ManifoldCF 0.4
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 0.4
>
>
> Running the postgresql load test on my laptop, I was surprised when the test
> did not finish. The UI indicated that the job was being deleted, but there
> were 49,000 documents and that number was not moving. Further inspection
> yielded the following:
> - Job was in the "DELETING" state
> - Documents were in the "BEINGDELETED" state
> - No activity of any kind ongoing
> The log had no errors.
> It was impossible to get a thread dump, but a cursory inspection of the code
> indicated that either the documents were being marked as "BEINGDELETED" but
> were not actually being placed on the in-memory queue, or the delete threads
> were picking up the documents and somehow avoiding marking them as being
> processed.
> Also, probably unrelated, the Document Status report listed these documents
> as having a status of "Being removed" and a state of "Unknown". The
> "Unknown" should have been a "Deleting". Since the extended WHEN... ELSE
> clause has a reasonable condition for the "Deleting" answer, it's hard to see
> how this could have occurred either.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
