You can call it but at the risk of making your design more fragile in the face of future changes. There is no guarantee that that method will remain unchanged in the future.
Karl On Tue, Jul 5, 2011 at 2:03 PM, Farzad Valad <[email protected]> wrote: > The problem is fixed and I see a few tries up to 5 or so. Question, instead > of my assumed 50, can I use ManifoldCF.getMaxWorkerThreads(). That would > return the maximum concurrent threads that can be ingesting at one time, > right? > > On 7/5/2011 11:28 AM, Farzad Valad wrote: >> >> Thanks. That is actually an incremented number for each unique hashsum to >> track the dups. I made a new set of test data with repeating content, and >> hence more than one thread is trying to insert the same doc. They both grab >> the highest number in the db and try to insert, hence the last one trying >> fails. >> >> My solution, was to make a loop, initially infinite, but then I decided to >> put in a counter of 50 tries. The chances that 50 threads at the same time >> try to insert same hashsum should be less than me winning the lottery : ) >> >> On 7/5/2011 11:20 AM, Karl Wright wrote: >>> >>> Also, if you need a unique ID, I suggest that you call ManifoldCF's >>> unique ID generator. >>> >>> Karl >>> >>> On Tue, Jul 5, 2011 at 10:29 AM, Karl Wright<[email protected]> wrote: >>>> >>>> It does seem to be in your code. >>>> >>>> Try psql. The \d<tablename> command should list indexes. >>>> >>>> Karl >>>> >>>> On Tue, Jul 5, 2011 at 10:26 AM, Farzad Valad<[email protected]> wrote: >>>>> >>>>> Just got my week started, was out of town with the family. Here is the >>>>> stack trace, coming back to this fresh, I think this is a problem in my >>>>> code. I'm not considering the multi-threaded nature of the manifoldcf >>>>> when >>>>> trying to find the next unique number to use. Is there a way to find >>>>> out >>>>> what "i1307564876133" constraint is? I tried to use pgAdmin, but it >>>>> didn't >>>>> show anything in table properties. My code is adding three indexes, >>>>> file >>>>> content hash sum, a unique number, and document URI. >>>>> >>>>> ERROR 2011-07-01 18:49:09,935 [Worker thread '33'] >>>>> (WorkerThread.java:893) - >>>>> Exception tossed: ERROR: duplicate key value violates unique constraint >>>>> "i1307564876133" >>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: ERROR: >>>>> duplicate >>>>> key value violates unique constraint "i1307564876133" >>>>> at >>>>> >>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.reinterpretException(DBInterfacePostgreSQL.java:621) >>>>> at >>>>> >>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:641) >>>>> at >>>>> >>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:191) >>>>> at >>>>> >>>>> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:76) >>>>> at >>>>> >>>>> org.apache.manifoldcf.agents.output.dupfinder.DataManager.insertData(DataManager.java:143) >>>>> at >>>>> >>>>> org.apache.manifoldcf.agents.output.dupfinder.DupFinderConnector.addOrReplaceDocument(DupFinderConnector.java:161) >>>>> at >>>>> >>>>> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.addOrReplaceDocument(IncrementalIngester.java:1433) >>>>> at >>>>> >>>>> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.performIngestion(IncrementalIngester.java:418) >>>>> at >>>>> >>>>> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:313) >>>>> at >>>>> >>>>> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocument(WorkerThread.java:1565) >>>>> at >>>>> >>>>> org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:275) >>>>> at >>>>> >>>>> org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector.processDocuments(BaseRepositoryConnector.java:423) >>>>> at >>>>> >>>>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:564) >>>>> Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key >>>>> value >>>>> violates unique constraint "i1307564876133" >>>>> at >>>>> >>>>> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1548) >>>>> at >>>>> >>>>> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1316) >>>>> at >>>>> >>>>> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:191) >>>>> at >>>>> >>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:452) >>>>> at >>>>> >>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:351) >>>>> at >>>>> >>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:305) >>>>> at >>>>> org.apache.manifoldcf.core.database.Database.execute(Database.java:606) >>>>> at >>>>> >>>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:421) >>>>> >>>>> On 7/5/2011 2:27 AM, Karl Wright wrote: >>>>>> >>>>>> Hi Farzad - any luck on getting that stack trace? >>>>>> Karl >>>>>> >>>>>> On Sat, Jul 2, 2011 at 1:09 PM, [email protected]<[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> The unique key violation is not expected - if you could send along a >>>>>>> complete stack trace that would be good. >>>>>>> >>>>>>> The lock clean procedure is to shut down all mcf processes, execute >>>>>>> the >>>>>>> lock clean, and start them up again. >>>>>>> >>>>>>> Karl >>>>>>> Sent from my Nokia phone >>>>>>> -----Original Message----- >>>>>>> From: Farzad Valad >>>>>>> Sent: 01/07/2011, 12:23 PM >>>>>>> To: [email protected] >>>>>>> Subject: Sync Dir >>>>>>> >>>>>>> >>>>>>> In a multi process setup, I've noticed if you stop the agent but keep >>>>>>> Tomcat going, you most like will hit a key violation error like the >>>>>>> one >>>>>>> below. I remember reading they don't have to be both restarted, but >>>>>>> what if they do? What if I need to issue a LockClean in between? >>>>>>> >>>>>>> Error: ERROR: duplicate key value violates unique constraint >>>>>>> "i1307564876133" >>>>>>> >>>>> >> >> > >
