[jira] [Commented] (CONNECTORS-279) Postgresql load test job delete document cleanup fails sometimes and leaves orphaned documents
[ https://issues.apache.org/jira/browse/CONNECTORS-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134796#comment-13134796 ] Karl Wright commented on CONNECTORS-279: Postgresql logging shows nothing unusual at the time the reindex fails to complete. Postgresql load test job delete document cleanup fails sometimes and leaves orphaned documents -- Key: CONNECTORS-279 URL: https://issues.apache.org/jira/browse/CONNECTORS-279 Project: ManifoldCF Issue Type: Bug Components: Framework crawler agent Affects Versions: ManifoldCF 0.4 Reporter: Karl Wright Assignee: Karl Wright Fix For: ManifoldCF 0.4 Running the postgresql load test on my laptop, I was surprised when the test did not finish. The UI indicated that the job was being deleted, but there were 49,000 documents and that number was not moving. Further inspection yielded the following: - Job was in the DELETING state - Documents were in the BEINGDELETED state - No activity of any kind ongoing The log had no errors. It was impossible to get a thread dump, but a cursory inspection of the code indicated that either the documents were being marked as BEINGDELETED but were not actually being placed on the in-memory queue, or the delete threads were picking up the documents and somehow avoiding marking them as being processed. Also, probably unrelated, the Document Status report listed these documents as having a status of Being removed and a state of Unknown. The Unknown should have been a Deleting. Since the extended WHEN... ELSE clause has a reasonable condition for the Deleting answer, it's hard to see how this could have occurred either. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CONNECTORS-282) We see a lot of unique constraint collisions trying to insert into the ingeststatus table
We see a lot of unique constraint collisions trying to insert into the ingeststatus table - Key: CONNECTORS-282 URL: https://issues.apache.org/jira/browse/CONNECTORS-282 Project: ManifoldCF Issue Type: Bug Components: Framework agents process Affects Versions: ManifoldCF 0.4 Reporter: Karl Wright Priority: Minor Fix For: ManifoldCF 0.4 Postgresql logs the following over and over: 2011-10-24 23:48:56 EDT STATEMENT: INSERT INTO ingeststatus (id,changecount,dockey,lastversion,firstingest,connectionname,authorityname,urihash,lastoutputversion,lastingest,docuri) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11) This is clearly part of a failback sequence designed to prevent duplicates, but we seem to be erring by trying the least probable thing first. This no doubt impacts performance as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CONNECTORS-283) Upgrade to jcifs.jar 1.3.17
Upgrade to jcifs.jar 1.3.17 --- Key: CONNECTORS-283 URL: https://issues.apache.org/jira/browse/CONNECTORS-283 Project: ManifoldCF Issue Type: Improvement Components: Build, JCIFS connector Affects Versions: ManifoldCF 0.4 Reporter: Shinichiro Abe Assignee: Shinichiro Abe Priority: Minor Fix For: ManifoldCF 0.4 Attachments: CONNECTORS-283.patch I upgrade new jcifs version because of missing link of previous version 1.3.16. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CONNECTORS-283) Upgrade to jcifs.jar 1.3.17
[ https://issues.apache.org/jira/browse/CONNECTORS-283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinichiro Abe updated CONNECTORS-283: -- Attachment: CONNECTORS-283.patch Upgrade to jcifs.jar 1.3.17 --- Key: CONNECTORS-283 URL: https://issues.apache.org/jira/browse/CONNECTORS-283 Project: ManifoldCF Issue Type: Improvement Components: Build, JCIFS connector Affects Versions: ManifoldCF 0.4 Reporter: Shinichiro Abe Assignee: Shinichiro Abe Priority: Minor Fix For: ManifoldCF 0.4 Attachments: CONNECTORS-283.patch I upgrade new jcifs version because of missing link of previous version 1.3.16. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-283) Upgrade to jcifs.jar 1.3.17
[ https://issues.apache.org/jira/browse/CONNECTORS-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13134898#comment-13134898 ] Karl Wright commented on CONNECTORS-283: Looks good to me. Upgrade to jcifs.jar 1.3.17 --- Key: CONNECTORS-283 URL: https://issues.apache.org/jira/browse/CONNECTORS-283 Project: ManifoldCF Issue Type: Improvement Components: Build, JCIFS connector Affects Versions: ManifoldCF 0.4 Reporter: Shinichiro Abe Assignee: Shinichiro Abe Priority: Minor Fix For: ManifoldCF 0.4 Attachments: CONNECTORS-283.patch I upgrade new jcifs version because of missing link of previous version 1.3.16. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CONNECTORS-282) We see a lot of unique constraint collisions trying to insert into the ingeststatus table
[ https://issues.apache.org/jira/browse/CONNECTORS-282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wright resolved CONNECTORS-282. Resolution: Fixed Assignee: Karl Wright r1188611. We see a lot of unique constraint collisions trying to insert into the ingeststatus table - Key: CONNECTORS-282 URL: https://issues.apache.org/jira/browse/CONNECTORS-282 Project: ManifoldCF Issue Type: Bug Components: Framework agents process Affects Versions: ManifoldCF 0.4 Reporter: Karl Wright Assignee: Karl Wright Priority: Minor Fix For: ManifoldCF 0.4 Postgresql logs the following over and over: 2011-10-24 23:48:56 EDT STATEMENT: INSERT INTO ingeststatus (id,changecount,dockey,lastversion,firstingest,connectionname,authorityname,urihash,lastoutputversion,lastingest,docuri) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11) This is clearly part of a failback sequence designed to prevent duplicates, but we seem to be erring by trying the least probable thing first. This no doubt impacts performance as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-279) Postgresql load test job delete document cleanup fails sometimes and leaves orphaned documents
[
https://issues.apache.org/jira/browse/CONNECTORS-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13135407#comment-13135407
]
Karl Wright commented on CONNECTORS-279:
With the 9.1 JDBC driver, things are even worse. The crawl does not complete
because 3 queries seem to get forever stuck. They are not the same query
either:
{code}
Worker thread '16' daemon prio=6 tid=0x0547f000 nid=0x1b90 in Object.wait()
[0x0609f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1143)
- locked 0x29c514c0 (a
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
at java.lang.Thread.join(Thread.java:1196)
at
org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:453)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.startATransaction(DBInterfacePostgreSQL.java:1134)
at
org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:235)
at
org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:218)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1140)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:639)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.beginTransaction(DBInterfacePostgreSQL.java:1072)
at
org.apache.manifoldcf.crawler.jobs.JobManager.finishDocuments(JobManager.java:3902)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:567)
{code}
{code}
Worker thread '12' daemon prio=6 tid=0x0547d800 nid=0x1170 in Object.wait()
[0x05f5f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1143)
- locked 0x29c51608 (a
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread)
at java.lang.Thread.join(Thread.java:1196)
at
org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:453)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.startATransaction(DBInterfacePostgreSQL.java:1134)
at
org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:235)
at
org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:218)
at
org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1140)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:811)
at
org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:229)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.noteDocumentIngest(IncrementalIngester.java:1358)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:495)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:364)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1577)
at
org.apache.manifoldcf.crawler.connectors.rss.RSSConnector.processDocuments(RSSConnector.java:1470)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:561)
{code}
Postgresql load test job delete document cleanup fails sometimes and leaves
orphaned documents
--
Key: CONNECTORS-279
URL: https://issues.apache.org/jira/browse/CONNECTORS-279
Project: ManifoldCF
Issue Type: Bug
Components: Framework crawler agent
Affects Versions: ManifoldCF 0.4
Reporter: Karl Wright
Assignee: Karl Wright
Fix For: ManifoldCF 0.4
Running the postgresql load test on my laptop, I was surprised when the test
did not finish. The UI indicated that the job was being deleted, but there
were 49,000 documents and that number was not moving. Further inspection
yielded the following:
- Job was in the DELETING state
- Documents were in the BEINGDELETED state
- No activity of any kind ongoing
The log had no errors.
It was impossible to get a
