The problem is fixed and I see a few tries up to 5 or so. Question,
instead of my assumed 50, can I use ManifoldCF.getMaxWorkerThreads().
That would return the maximum concurrent threads that can be ingesting
at one time, right?
On 7/5/2011 11:28 AM, Farzad Valad wrote:
Thanks. That is actually an incremented number for each unique
hashsum to track the dups. I made a new set of test data with
repeating content, and hence more than one thread is trying to insert
the same doc. They both grab the highest number in the db and try to
insert, hence the last one trying fails.
My solution, was to make a loop, initially infinite, but then I
decided to put in a counter of 50 tries. The chances that 50 threads
at the same time try to insert same hashsum should be less than me
winning the lottery : )
On 7/5/2011 11:20 AM, Karl Wright wrote:
Also, if you need a unique ID, I suggest that you call ManifoldCF's
unique ID generator.
Karl
On Tue, Jul 5, 2011 at 10:29 AM, Karl Wright<[email protected]> wrote:
It does seem to be in your code.
Try psql. The \d<tablename> command should list indexes.
Karl
On Tue, Jul 5, 2011 at 10:26 AM, Farzad Valad<[email protected]> wrote:
Just got my week started, was out of town with the family. Here is
the
stack trace, coming back to this fresh, I think this is a problem
in my
code. I'm not considering the multi-threaded nature of the
manifoldcf when
trying to find the next unique number to use. Is there a way to
find out
what "i1307564876133" constraint is? I tried to use pgAdmin, but
it didn't
show anything in table properties. My code is adding three
indexes, file
content hash sum, a unique number, and document URI.
ERROR 2011-07-01 18:49:09,935 [Worker thread '33']
(WorkerThread.java:893) -
Exception tossed: ERROR: duplicate key value violates unique
constraint
"i1307564876133"
org.apache.manifoldcf.core.interfaces.ManifoldCFException: ERROR:
duplicate
key value violates unique constraint "i1307564876133"
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.reinterpretException(DBInterfacePostgreSQL.java:621)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:641)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:191)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:76)
at
org.apache.manifoldcf.agents.output.dupfinder.DataManager.insertData(DataManager.java:143)
at
org.apache.manifoldcf.agents.output.dupfinder.DupFinderConnector.addOrReplaceDocument(DupFinderConnector.java:161)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.addOrReplaceDocument(IncrementalIngester.java:1433)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:418)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:313)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1565)
at
org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:275)
at
org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:564)
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key
value
violates unique constraint "i1307564876133"
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1548)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1316)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:191)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:452)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:351)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:305)
at
org.apache.manifoldcf.core.database.Database.execute(Database.java:606)
at
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:421)
On 7/5/2011 2:27 AM, Karl Wright wrote:
Hi Farzad - any luck on getting that stack trace?
Karl
On Sat, Jul 2, 2011 at 1:09 PM,
[email protected]<[email protected]>
wrote:
The unique key violation is not expected - if you could send along a
complete stack trace that would be good.
The lock clean procedure is to shut down all mcf processes,
execute the
lock clean, and start them up again.
Karl
Sent from my Nokia phone
-----Original Message-----
From: Farzad Valad
Sent: 01/07/2011, 12:23 PM
To: [email protected]
Subject: Sync Dir
In a multi process setup, I've noticed if you stop the agent but
keep
Tomcat going, you most like will hit a key violation error like
the one
below. I remember reading they don't have to be both restarted, but
what if they do? What if I need to issue a LockClean in between?
Error: ERROR: duplicate key value violates unique constraint
"i1307564876133"
