Re: Data Manager Null

Tue, 07 Jun 2011 17:50:39 -0700

I think I get it now why you said put getSession in addOrReplaceDocument. This way you construct the dataManager when you need it as oppose to each set and clear pair : ) 

On 6/7/2011 5:10 PM, Farzad Valad wrote:
I don't fully understand how a connector instance can be used by multiple threads without each thread calling setThread. Here is what I think I know. 


The contract does say that addOrReplaceDocument is only called after 
setTC, right?  Because you first have to have connection handle before 
a particular manifold thread can use it.  So if I create my 
dataManager in set, all will be well in addOrReplaceDocument.  The 
other caveat is that I'll make dataManager a class variable, instead 
of static.  So each object would have its own instance with its TC, 
and in clearTC they'd be nulling their version an not anyone else's.


Do I get it?

On 6/7/2011 5:00 PM, Karl Wright wrote:

The recommendation to have getSession be called in
addOrReplaceDocument is because there is nothing in the contract which
states that the connector instance will switch threads between calls.
Therefore there is no guarantee that
clearThreadContext/setThreadContext will be called right prior to
addOrReplaceDocument.  The two aspects of the interface are therefore
independent of one another, and it would be poor coding to presume
that you could assume something in the contract that was not there.

Karl

On Tue, Jun 7, 2011 at 5:52 PM, Farzad Valad<[email protected]>  wrote:

Thanks for the confirmation.  So if I have code to set dataManager 
to null
in clearThreadContext and create a dataManager in setThreadContext.  
Why do
I need the getSession method in addOrReplaceDocument method?  From 
what I

learnt about ManifoldCF architecture, setThreadContext will get called

before addOrReplaceDocument.  This was something you recommended 
when I was

asking about the third party repository.

Farzad.

On 6/7/2011 4:35 PM, Karl Wright wrote:

It sounds like you are on the right track for fixing all of these
problems.

Karl


On Tue, Jun 7, 2011 at 4:38 PM, Farzad Valad<[email protected]>    
wrote:

I think I found the problem.  I should be tearing down the 
dataManager

and

recreating it between clear and set thread context calls, because 
it has

a
thread context.  I'm not doing that.  I guess I did learn something
reading

: ) let me know if you believe otherwise.  Also do you think this 
is why

the
bad transaction id is happening?  Thanks!

            IDBInterface databaseHandle =
DBInterfaceFactory.make(currentContext,
ManifoldCF.getMasterDatabaseName(),
ManifoldCF.getMasterDatabaseUsername(),
ManifoldCF.getMasterDatabasePassword());

            dataManager = new DataManager(currentContext, 
databaseHandle);




On 6/7/2011 12:42 PM, Farzad Valad wrote:

So I think I figured it out.  For some reason I'm getting a db 
error,

bad

transaction id, which then kills my dataManager object, or I 
should say

the

framework is setting it to null.  What does a Bad transaction ID 
mean?
  Thoughts?  This happened after I did a LockClean and restart 
both the

agent
and Tomcat.  Thanks, Farzad.

ERROR 2011-06-07 11:44:56,365 [Worker thread '90']
(CacheManager.java:621)
- Thread[Worker thread '90',5,main]: invalidateKeys: 1307465096157:
org.apache.manifoldcf.core.cachemanager.CacheManager@13b0c258:
Transaction
hash =


{1307465096144=org.apache.manifoldcf.core.cachemanager.CacheManager$CacheTransactionHandle@39a72981} 


org.apache.manifoldcf.core.interfaces.ManifoldCFException: Bad
transaction
ID!
    at


org.apache.manifoldcf.core.cachemanager.CacheManager.invalidateKeys(CacheManager.java:620) 


    at


org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:175) 


    at


org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168) 


    at


org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:637) 


    at


org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:191) 


    at


org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:76) 


    at


org.apache.manifoldcf.agents.output.dupfinder.DataManager.insertData(DataManager.java:115) 


    at


org.apache.manifoldcf.agents.output.dupfinder.DupFinderConnector.addOrReplaceDocument(DupFinderConnector.java:158) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.addOrReplaceDocument(IncrementalIngester.java:1433) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:418) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:313) 


    at


org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1565) 


    at


org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:275) 


    at


org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423) 


    at


org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:564) 


FATAL 2011-06-07 11:44:56,583 [Worker thread '32']
(DupFinderConnector.java:155) - DATAMANAGER IS NULL!!!!
ERROR 2011-06-07 11:44:56,599 [Worker thread '90']
(WorkerThread.java:893)
- Exception tossed: Bad transaction ID!
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Bad
transaction
ID!
    at


org.apache.manifoldcf.core.cachemanager.CacheManager.invalidateKeys(CacheManager.java:620) 


    at


org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:175) 


    at


org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168) 


    at


org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:637) 


    at


org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:191) 


    at


org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:76) 


    at


org.apache.manifoldcf.agents.output.dupfinder.DataManager.insertData(DataManager.java:115) 


    at


org.apache.manifoldcf.agents.output.dupfinder.DupFinderConnector.addOrReplaceDocument(DupFinderConnector.java:158) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.addOrReplaceDocument(IncrementalIngester.java:1433) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:418) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:313) 


    at


org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1565) 


    at


org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:275) 


    at


org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423) 


    at


org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:564) 


FATAL 2011-06-07 11:44:56,614 [Worker thread '32']
(WorkerThread.java:955)
- Error tossed: null
java.lang.NullPointerException
    at


org.apache.manifoldcf.agents.output.dupfinder.DupFinderConnector.addOrReplaceDocument(DupFinderConnector.java:158) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.addOrReplaceDocument(IncrementalIngester.java:1433) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:418) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:313) 


    at


org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1565) 


    at


org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:275) 


    at


org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423) 


    at


org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:564) 


  INFO 2011-06-07 11:44:56,645 [Worker thread '92']
(DupFinderConnector.java:251) - Attempting to initialize
dataManager(null)
and ciConnector(null)


On 6/7/2011 9:20 AM, Farzad Valad wrote:

Lately when I issue an abort on a crawl job (click abort in UI), it
gets
stuck, meaning the UI doesn't show any new info on subsequent
refreshes.  It

just says Aborting, the start time, no end time, shows # of 
documents,
active, and processed.  I restarted Tomcat, but still stuck in 
Aborting
state.  Restarting the Agent process doesn't have any affect.  
But now

if

you kill the agent process and issue lock clean, then start the 
Agent
Process, it will show an Error in the Status column, but no end 
time.
  Ironically, this time the problem was a bad transaction id.  
The last

time
it was a connection refusal to my repository.  Thoughts?


PS.  Previous problem, you were right, dataManager is going null 
for

some
reason, actually debugging for dataManager I ran into this one : )

ERROR 2011-06-07 08:50:01,416 [Worker thread '64']
(CacheManager.java:621) - Thread[Worker thread '64',5,main]:
invalidateKeys:
1307454600471:
org.apache.manifoldcf.core.cachemanager.CacheManager@39d7af3:
Transaction hash = {}
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Bad
transaction ID!
    at


org.apache.manifoldcf.core.cachemanager.CacheManager.invalidateKeys(CacheManager.java:620) 


    at


org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:175) 


    at


org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168) 


    at


org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:637) 


    at


org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:191) 


    at


org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:76) 


    at


org.apache.manifoldcf.agents.output.dupfinder.DataManager.insertData(DataManager.java:115) 


    at


org.apache.manifoldcf.agents.output.dupfinder.DupFinderConnector.addOrReplaceDocument(DupFinderConnector.java:162) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.addOrReplaceDocument(IncrementalIngester.java:1433) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:418) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:313) 


    at


org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1565) 


    at


org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:275) 


    at


org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423) 


    at


org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:564) 


ERROR 2011-06-07 08:50:01,510 [Worker thread '64']
(WorkerThread.java:893) - Exception tossed: Bad transaction ID!
org.apache.manifoldcf.core.interfaces.ManifoldCFException: Bad
transaction ID!
    at


org.apache.manifoldcf.core.cachemanager.CacheManager.invalidateKeys(CacheManager.java:620) 


    at


org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:175) 


    at


org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:168) 


    at


org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:637) 


    at


org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:191) 


    at


org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:76) 


    at


org.apache.manifoldcf.agents.output.dupfinder.DataManager.insertData(DataManager.java:115) 


    at


org.apache.manifoldcf.agents.output.dupfinder.DupFinderConnector.addOrReplaceDocument(DupFinderConnector.java:162) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.addOrReplaceDocument(IncrementalIngester.java:1433) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:418) 


    at


org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:313) 


    at


org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1565) 


    at


org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:275) 


    at


org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423) 


    at


org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:564) 































					Reply via email to