Ok, after applying your patch, it seems that the problem with duplicate values is solved. Nice! However, there are two new warnings that I observed in log:
1. WARN 2012-06-05 09:58:03,822 (Worker thread '33') - Pre-ingest service interruption reported for job 1338817906566 connection 'Wiki': Job no longer active This one is always in series with different numbers of threads. It seems that I'm getting it after pausing a job. Restarting the job does not help. The only way to get rid of those warnings is to abort a job and then start it again. 2. WARN 2012-06-05 10:16:52,967 (Worker thread '5') - Found a long-running query (62959 ms): [UPDATE hopcount SET deathmark=?,distance=? WHERE id IN(SELECT ownerid FROM hopdeletedeps t0 WHERE t0.jobid=? AND t0.childidhash=? AND EXISTS(SELECT 'x' FROM intrinsiclink t1 WHERE t1.jobid=t0.jobid AND t1.linktype=t0.linktype AND t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash AND t1.isnew=?))] WARN 2012-06-05 10:16:52,968 (Worker thread '5') - Parameter 0: 'D' WARN 2012-06-05 10:16:52,968 (Worker thread '5') - Parameter 1: '-1' WARN 2012-06-05 10:16:52,968 (Worker thread '5') - Parameter 2: '1338817906566' WARN 2012-06-05 10:16:52,968 (Worker thread '5') - Parameter 3: '9C9DE99F3162AC861D4B92764B9F3CC47F120704' WARN 2012-06-05 10:16:52,968 (Worker thread '5') - Parameter 4: 'B' This one also goes in series. Usually > 15 threads reporting the same warning one after another. I'm not really sure when it happens though... Marcin -----Original Message----- From: Karl Wright [mailto:[email protected]] Sent: Monday, June 04, 2012 3:03 PM To: [email protected] Subject: Re: Derby DB up-to-date? Also, for what it is worth, we're planning to release 0.6 sometime around June 30, provided all the various tickets are completed as expected. Karl On Mon, Jun 4, 2012 at 8:17 AM, Karl Wright <[email protected]> wrote: > Generally we don't release binaries except as a point release. That > takes at least a four-day process. I can, of course, build the > binaries here for you but then I'd have no means of transfering them > to you. > > Can you describe the problems you are having building? I suspect you > need to download the -lib package and put that in place first. The > steps are: > > (1) download the -src package > (2) download the -lib package > (3) unpack them both and place the contents of the -lib package into > the "lib" directory under the root directory of the -src package > (where LICENSE.txt is found) > (4) type "ant build". > > There are clear directions in the README.txt file for each of the two > packages, if you need further help. > > Thanks, > Karl > > > On Mon, Jun 4, 2012 at 8:10 AM, Marcin Goss <[email protected]> wrote: >> Thanks. Any chance of getting binaries with this fix? I have some problems >> building everything with the patch applied. >> >> -----Original Message----- >> From: Karl Wright [mailto:[email protected]] >> Sent: Monday, June 04, 2012 2:05 PM >> To: [email protected] >> Subject: Re: Derby DB up-to-date? >> >> I committed the pertinent change (without debugging output) to trunk as >> well. So if you want to upgrade to 0.6-dev, you will get the same fix. >> >> Karl >> >> >> On Mon, Jun 4, 2012 at 6:48 AM, Karl Wright <[email protected]> wrote: >>> I've attached a test patch to the CONNECTORS-478 ticket that you can >>> download and use to patch 0.5.1 sources. The test patch *should* >>> fix the problem, but if it doesn't, it will certainly log more >>> diagnostics, enough so I can see what the change in behavior has >>> been for Derby in this release. >>> >>> FWIW, it looks like Derby is throwing a slightly different kind of >>> exception for this case than it did before. >>> >>> Can you please try it out and let me know what happens? >>> >>> Thanks! >>> Karl >>> >>> >>> On Mon, Jun 4, 2012 at 6:29 AM, Karl Wright <[email protected]> wrote: >>>> Thanks, this is exactly what I was looking for. >>>> >>>> I'll put it in the ticket and have a look at this later today. >>>> >>>> Karl >>>> >>>> >>>> On Mon, Jun 4, 2012 at 6:20 AM, Marcin Goss <[email protected]> >>>> wrote: >>>>> This is all I've got. I am sorry for the polish fragments, I can't seem >>>>> to change it anywhere. Anyway, hopefully you should be able to get all >>>>> the information you need. >>>>> >>>>> ERROR 2012-06-04 11:22:25,449 (Worker thread '42') - Worker thread >>>>> aborting and restarting due to database connection reset: Database >>>>> exception: Exception doing query: Instrukcja została przerwana, ponieważ >>>>> mogła spowodować powstanie duplikatu wartości klucza w regule >>>>> ograniczającej klucz unikalny lub podstawowy albo w unikalnym indeksie >>>>> identyfikowanym przez obiekt 'C1338469165883' zdefiniowany dla 'EVENTS'. >>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database >>>>> exception: Exception doing query: Instrukcja została przerwana, ponieważ >>>>> mogła spowodować powstanie duplikatu wartości klucza w regule >>>>> ograniczającej klucz unikalny lub podstawowy albo w unikalnym indeksie >>>>> identyfikowanym przez obiekt 'C1338469165883' zdefiniowany dla 'EVENTS'. >>>>> at >>>>> org.apache.manifoldcf.core.database.Database.executeViaThread(Data >>>>> ba >>>>> se.java:680) >>>>> at >>>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery( >>>>> Da >>>>> tabase.java:726) >>>>> at >>>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.cr >>>>> ea >>>>> te(Database.java:1393) >>>>> at >>>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAn >>>>> dE >>>>> xecute(CacheManager.java:144) >>>>> at >>>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database >>>>> .j >>>>> ava:185) >>>>> at >>>>> org.apache.manifoldcf.core.database.DBInterfaceDerby.performModifi >>>>> ca >>>>> tion(DBInterfaceDerby.java:831) >>>>> at >>>>> org.apache.manifoldcf.core.database.DBInterfaceDerby.performInsert >>>>> (D >>>>> BInterfaceDerby.java:222) >>>>> at >>>>> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTa >>>>> bl >>>>> e.java:68) >>>>> at >>>>> org.apache.manifoldcf.crawler.jobs.EventManager.createEvent(EventM >>>>> an >>>>> ager.java:123) >>>>> at >>>>> org.apache.manifoldcf.crawler.jobs.JobManager.beginEventSequence(J >>>>> ob >>>>> Manager.java:4217) >>>>> at >>>>> org.apache.manifoldcf.crawler.system.WorkerThread$VersionActivity. >>>>> be >>>>> ginEventSequence(WorkerThread.java:1236) >>>>> at >>>>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConn >>>>> ec >>>>> tor.lookupIPAddress(WebcrawlerConnector.java:4737) >>>>> at >>>>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConn >>>>> ec >>>>> tor.getDocumentVersions(WebcrawlerConnector.java:634) >>>>> at >>>>> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread >>>>> .j >>>>> ava:318) Caused by: >>>>> java.sql.SQLIntegrityConstraintViolationException: Instrukcja została >>>>> przerwana, ponieważ mogła spowodować powstanie duplikatu wartości klucza >>>>> w regule ograniczającej klucz unikalny lub podstawowy albo w unikalnym >>>>> indeksie identyfikowanym przez obiekt 'C1338469165883' zdefiniowany dla >>>>> 'EVENTS'. >>>>> at >>>>> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(U >>>>> nk >>>>> nown Source) >>>>> at >>>>> org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown >>>>> Source) >>>>> at >>>>> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLExcept >>>>> io >>>>> n(Unknown Source) >>>>> at >>>>> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException >>>>> (U >>>>> nknown Source) >>>>> at >>>>> org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown >>>>> Source) >>>>> at >>>>> org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown >>>>> Source) >>>>> at >>>>> org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown >>>>> Source) >>>>> at >>>>> org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement >>>>> (U >>>>> nknown Source) >>>>> at >>>>> org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Un >>>>> kn >>>>> own Source) >>>>> at >>>>> org.apache.manifoldcf.core.database.Database.execute(Database.java >>>>> :8 >>>>> 39) >>>>> at >>>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.ru >>>>> n( >>>>> Database.java:640) Caused by: java.sql.SQLException: Instrukcja >>>>> została przerwana, ponieważ mogła spowodować powstanie duplikatu wartości >>>>> klucza w regule ograniczającej klucz unikalny lub podstawowy albo w >>>>> unikalnym indeksie identyfikowanym przez obiekt 'C1338469165883' >>>>> zdefiniowany dla 'EVENTS'. >>>>> at >>>>> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unk >>>>> no >>>>> wn Source) >>>>> at >>>>> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransp >>>>> or >>>>> tAcrossDRDA(Unknown Source) >>>>> ... 11 more >>>>> Caused by: ERROR 23505: Instrukcja została przerwana, ponieważ mogła >>>>> spowodować powstanie duplikatu wartości klucza w regule ograniczającej >>>>> klucz unikalny lub podstawowy albo w unikalnym indeksie identyfikowanym >>>>> przez obiekt 'C1338469165883' zdefiniowany dla 'EVENTS'. >>>>> at >>>>> org.apache.derby.iapi.error.StandardException.newException(Unknown >>>>> Source) >>>>> at >>>>> org.apache.derby.impl.sql.execute.IndexChanger.insertAndCheckDups( >>>>> Un >>>>> known Source) >>>>> at >>>>> org.apache.derby.impl.sql.execute.IndexChanger.doInsert(Unknown >>>>> Source) >>>>> at >>>>> org.apache.derby.impl.sql.execute.IndexChanger.insert(Unknown >>>>> Source) >>>>> at >>>>> org.apache.derby.impl.sql.execute.IndexSetChanger.insert(Unknown >>>>> Source) >>>>> at >>>>> org.apache.derby.impl.sql.execute.RowChangerImpl.insertRow(Unknown >>>>> Source) >>>>> at >>>>> org.apache.derby.impl.sql.execute.InsertResultSet.normalInsertCore >>>>> (U >>>>> nknown Source) >>>>> at >>>>> org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown >>>>> Source) >>>>> at >>>>> org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unk >>>>> no >>>>> wn Source) >>>>> at >>>>> org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown >>>>> Source) >>>>> ... 5 more >>>>> >>>>> >>>>> /Marcin >>>>> >>>>> -----Original Message----- >>>>> From: Karl Wright [mailto:[email protected]] >>>>> Sent: Monday, June 04, 2012 12:17 PM >>>>> To: [email protected] >>>>> Subject: Re: Derby DB up-to-date? >>>>> >>>>> I've created CONNECTORS-478 to track this issue. But please do send >>>>> along the rest of the stack trace context because without that I cannot >>>>> know where, exactly, the constraint violation is not getting caught. >>>>> >>>>> Thanks, >>>>> Karl >>>>> >>>>> On Mon, Jun 4, 2012 at 6:08 AM, Karl Wright <[email protected]> wrote: >>>>>> Can you provide the rest of the exception? (The "caused by" >>>>>> clauses >>>>>> etc.?) >>>>>> >>>>>> The fact is that we discovered that 10.8.2.2 throws deadlock and >>>>>> constraint violation exceptions in different places (as compared >>>>>> to >>>>>> 10.7.1.1) under multi-threaded conditions. That was part of the >>>>>> reason why the 0.5.1 release was necessary. It's of course >>>>>> possible that we missed some places that (for instance) the wiki >>>>>> connector especially tweaks. >>>>>> >>>>>> Karl >>>>>> >>>>>> >>>>>> On Mon, Jun 4, 2012 at 5:29 AM, Marcin Goss <[email protected]> >>>>>> wrote: >>>>>>> Hello Karl, >>>>>>> >>>>>>> Thank you for answering. >>>>>>> I am using Manifold 0.5.1 and what I am doing is a simple indexing of >>>>>>> a wiki page. The error that I am getting looks like this: >>>>>>> >>>>>>> Caused by: java.sql.SQLIntegrityConstraintViolationException: The >>>>>>> statement was aborted because it would have caused a duplicate key >>>>>>> value in a unique or primary key constraint or unique index identified >>>>>>> by 'XXXXXXXX' defined on 'EVENTS'. >>>>>>> at >>>>>>> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException >>>>>>> (U >>>>>>> nkn >>>>>>> own Source) >>>>>>> at >>>>>>> org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown >>>>>>> Source) >>>>>>> at >>>>>>> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLExce >>>>>>> pt >>>>>>> ion >>>>>>> (Unknown Source) >>>>>>> at >>>>>>> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleExcepti >>>>>>> on >>>>>>> (Un >>>>>>> known Source) >>>>>>> at >>>>>>> org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unkno >>>>>>> wn >>>>>>> Source) >>>>>>> at >>>>>>> org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unkno >>>>>>> wn >>>>>>> Source) >>>>>>> at >>>>>>> org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unkno >>>>>>> wn >>>>>>> Source) >>>>>>> >>>>>>> /Marcin >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Karl Wright [mailto:[email protected]] >>>>>>> Sent: Wednesday, May 30, 2012 12:42 PM >>>>>>> To: [email protected] >>>>>>> Subject: Re: Derby DB up-to-date? >>>>>>> >>>>>>> Also, the ManifoldCF tests passed for these releases, so I'm wondering >>>>>>> exactly what you are seeing. Can you provide more details, including >>>>>>> exceptions? >>>>>>> >>>>>>> Karl >>>>>>> >>>>>>> On Wed, May 30, 2012 at 6:38 AM, Karl Wright <[email protected]> wrote: >>>>>>>> Which version of ManifoldCF are you using? The 0.5 release uses >>>>>>>> 10.8.2.2. >>>>>>>> >>>>>>>> Karl >>>>>>>> >>>>>>>> On Wed, May 30, 2012 at 5:21 AM, Marcin Goss >>>>>>>> <[email protected]> wrote: >>>>>>>>> I'm getting an error in a default manifoldCF setup (jetty >>>>>>>>> server and derby >>>>>>>>> db) that is probably caused (my supposition) by this bug in Derby: >>>>>>>>> https://issues.apache.org/jira/browse/DERBY-3343. The bug was >>>>>>>>> fixed in January 2011. Which version of Derby db is coming with >>>>>>>>> ManifoldCF? >>>>>>>>> Is it the one with this bug fixed? >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >> >>
