Thanks for the confirmation. So if I have code to set dataManager to
null in clearThreadContext and create a dataManager in
setThreadContext. Why do I need the getSession method in
addOrReplaceDocument method? From what I learnt about ManifoldCF
architecture, setThreadContext will get called before
addOrReplaceDocument. This was something you recommended when I was
asking about the third party repository.
Farzad.
On 6/7/2011 4:35 PM, Karl Wright wrote:
It sounds like you are on the right track for fixing all of these problems.
Karl
On Tue, Jun 7, 2011 at 4:38 PM, Farzad Valad<[email protected]> wrote:
I think I found the problem. I should be tearing down the dataManager and
recreating it between clear and set thread context calls, because it has a
thread context. I'm not doing that. I guess I did learn something reading
: ) let me know if you believe otherwise. Also do you think this is why the
bad transaction id is happening? Thanks!
IDBInterface databaseHandle =
DBInterfaceFactory.make(currentContext, ManifoldCF.getMasterDatabaseName(),
ManifoldCF.getMasterDatabaseUsername(),
ManifoldCF.getMasterDatabasePassword());
dataManager = new DataManager(currentContext, databaseHandle);
On 6/7/2011 12:42 PM, Farzad Valad wrote:
So I think I figured it out. For some reason I'm getting a db error, bad
transaction id, which then kills my dataManager object, or I should say the
framework is setting it to null. What does a Bad transaction ID mean?
Thoughts? This happened after I did a LockClean and restart both the agent
and Tomcat. Thanks, Farzad.
ERROR 2011-06-07 11:44:56,365 [Worker thread '90'] (CacheManager.java:621)
- Thread[Worker thread '90',5,main]: invalidateKeys: 1307465096157:
org.apache.manifoldcf.core.cachemanager.CacheManager@13b0c258: Transaction
hash =
{1307465096144=org.apache.manifoldcf.core.cachemanager.CacheManager$CacheTransactionHandle@39a72981}
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Bad transaction
ID!
at
org.apache.manifoldcf.core.cachemanager.CacheManager.invalidateKeys(CacheManager.java:620)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:175)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:637)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:191)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:76)
at
org.apache.manifoldcf.agents.output.dupfinder.DataManager.insertData(DataManager.java:115)
at
org.apache.manifoldcf.agents.output.dupfinder.DupFinderConnector.addOrReplaceDocument(DupFinderConnector.java:158)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.addOrReplaceDocument(IncrementalIngester.java:1433)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:418)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:313)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1565)
at
org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:275)
at
org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:564)
FATAL 2011-06-07 11:44:56,583 [Worker thread '32']
(DupFinderConnector.java:155) - DATAMANAGER IS NULL!!!!
ERROR 2011-06-07 11:44:56,599 [Worker thread '90'] (WorkerThread.java:893)
- Exception tossed: Bad transaction ID!
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Bad transaction
ID!
at
org.apache.manifoldcf.core.cachemanager.CacheManager.invalidateKeys(CacheManager.java:620)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:175)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:637)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:191)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:76)
at
org.apache.manifoldcf.agents.output.dupfinder.DataManager.insertData(DataManager.java:115)
at
org.apache.manifoldcf.agents.output.dupfinder.DupFinderConnector.addOrReplaceDocument(DupFinderConnector.java:158)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.addOrReplaceDocument(IncrementalIngester.java:1433)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:418)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:313)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1565)
at
org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:275)
at
org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:564)
FATAL 2011-06-07 11:44:56,614 [Worker thread '32'] (WorkerThread.java:955)
- Error tossed: null
java.lang.NullPointerException
at
org.apache.manifoldcf.agents.output.dupfinder.DupFinderConnector.addOrReplaceDocument(DupFinderConnector.java:158)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.addOrReplaceDocument(IncrementalIngester.java:1433)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:418)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:313)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1565)
at
org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:275)
at
org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:564)
INFO 2011-06-07 11:44:56,645 [Worker thread '92']
(DupFinderConnector.java:251) - Attempting to initialize dataManager(null)
and ciConnector(null)
On 6/7/2011 9:20 AM, Farzad Valad wrote:
Lately when I issue an abort on a crawl job (click abort in UI), it gets
stuck, meaning the UI doesn't show any new info on subsequent refreshes. It
just says Aborting, the start time, no end time, shows # of documents,
active, and processed. I restarted Tomcat, but still stuck in Aborting
state. Restarting the Agent process doesn't have any affect. But now if
you kill the agent process and issue lock clean, then start the Agent
Process, it will show an Error in the Status column, but no end time.
Ironically, this time the problem was a bad transaction id. The last time
it was a connection refusal to my repository. Thoughts?
PS. Previous problem, you were right, dataManager is going null for some
reason, actually debugging for dataManager I ran into this one : )
ERROR 2011-06-07 08:50:01,416 [Worker thread '64']
(CacheManager.java:621) - Thread[Worker thread '64',5,main]: invalidateKeys:
1307454600471: org.apache.manifoldcf.core.cachemanager.CacheManager@39d7af3:
Transaction hash = {}
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Bad
transaction ID!
at
org.apache.manifoldcf.core.cachemanager.CacheManager.invalidateKeys(CacheManager.java:620)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:175)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:637)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:191)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:76)
at
org.apache.manifoldcf.agents.output.dupfinder.DataManager.insertData(DataManager.java:115)
at
org.apache.manifoldcf.agents.output.dupfinder.DupFinderConnector.addOrReplaceDocument(DupFinderConnector.java:162)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.addOrReplaceDocument(IncrementalIngester.java:1433)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:418)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:313)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1565)
at
org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:275)
at
org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:564)
ERROR 2011-06-07 08:50:01,510 [Worker thread '64']
(WorkerThread.java:893) - Exception tossed: Bad transaction ID!
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Bad
transaction ID!
at
org.apache.manifoldcf.core.cachemanager.CacheManager.invalidateKeys(CacheManager.java:620)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:175)
at
org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:637)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:191)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:76)
at
org.apache.manifoldcf.agents.output.dupfinder.DataManager.insertData(DataManager.java:115)
at
org.apache.manifoldcf.agents.output.dupfinder.DupFinderConnector.addOrReplaceDocument(DupFinderConnector.java:162)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.addOrReplaceDocument(IncrementalIngester.java:1433)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:418)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:313)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1565)
at
org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:275)
at
org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:564)
