Thanks. That is actually an incremented number for each unique hashsum
to track the dups. I made a new set of test data with repeating
content, and hence more than one thread is trying to insert the same
doc. They both grab the highest number in the db and try to insert,
hence the last one trying fails.
My solution, was to make a loop, initially infinite, but then I decided
to put in a counter of 50 tries. The chances that 50 threads at the
same time try to insert same hashsum should be less than me winning the
lottery : )
On 7/5/2011 11:20 AM, Karl Wright wrote:
Also, if you need a unique ID, I suggest that you call ManifoldCF's
unique ID generator.
Karl
On Tue, Jul 5, 2011 at 10:29 AM, Karl Wright<[email protected]> wrote:
It does seem to be in your code.
Try psql. The \d<tablename> command should list indexes.
Karl
On Tue, Jul 5, 2011 at 10:26 AM, Farzad Valad<[email protected]> wrote:
Just got my week started, was out of town with the family. Here is the
stack trace, coming back to this fresh, I think this is a problem in my
code. I'm not considering the multi-threaded nature of the manifoldcf when
trying to find the next unique number to use. Is there a way to find out
what "i1307564876133" constraint is? I tried to use pgAdmin, but it didn't
show anything in table properties. My code is adding three indexes, file
content hash sum, a unique number, and document URI.
ERROR 2011-07-01 18:49:09,935 [Worker thread '33'] (WorkerThread.java:893) -
Exception tossed: ERROR: duplicate key value violates unique constraint
"i1307564876133"
org.apache.manifoldcf.core.interfaces.ManifoldCFException: ERROR: duplicate
key value violates unique constraint "i1307564876133"
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.reinterpretException(DBInterfacePostgreSQL.java:621)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:641)
at
org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:191)
at
org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:76)
at
org.apache.manifoldcf.agents.output.dupfinder.DataManager.insertData(DataManager.java:143)
at
org.apache.manifoldcf.agents.output.dupfinder.DupFinderConnector.addOrReplaceDocument(DupFinderConnector.java:161)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.addOrReplaceDocument(IncrementalIngester.java:1433)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:418)
at
org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:313)
at
org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1565)
at
org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:275)
at
org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423)
at
org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:564)
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value
violates unique constraint "i1307564876133"
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1548)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1316)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:191)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:452)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:351)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:305)
at
org.apache.manifoldcf.core.database.Database.execute(Database.java:606)
at
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:421)
On 7/5/2011 2:27 AM, Karl Wright wrote:
Hi Farzad - any luck on getting that stack trace?
Karl
On Sat, Jul 2, 2011 at 1:09 PM, [email protected]<[email protected]>
wrote:
The unique key violation is not expected - if you could send along a
complete stack trace that would be good.
The lock clean procedure is to shut down all mcf processes, execute the
lock clean, and start them up again.
Karl
Sent from my Nokia phone
-----Original Message-----
From: Farzad Valad
Sent: 01/07/2011, 12:23 PM
To: [email protected]
Subject: Sync Dir
In a multi process setup, I've noticed if you stop the agent but keep
Tomcat going, you most like will hit a key violation error like the one
below. I remember reading they don't have to be both restarted, but
what if they do? What if I need to issue a LockClean in between?
Error: ERROR: duplicate key value violates unique constraint
"i1307564876133"
