Hi Karl, Interesting If one job has this error then all jobs including newly created jobs are we never reaching completed status. Your fix worked for us.
Thanks, Jitu On Wed, Nov 19, 2014 at 3:41 PM, Karl Wright <[email protected]> wrote: > Hi Jitu, > > It means that, due to an error, your job tried to abort while it was > shutting down (and cleaning up unreachable documents). The reason for such > an abort is not clear but you will find out if you look at the logs and/or > the job's status when the job finally succeeds in aborting. > > Karl > > > On Wed, Nov 19, 2014 at 4:45 AM, Jitu <[email protected]> wrote: > >> Hi Karl, >> >> Today i faced with the same issue. not able to understand why it started >> happening suddenly. Glad to see fix already available. Thanks to all who >> reported and supported for the fix. >> >> >> Log: >> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected >> value for job status: 37 >> at >> org.apache.manifoldcf.crawler.jobs.Jobs.finishStopJob(Jobs.java:2973) >> at >> org.apache.manifoldcf.crawler.jobs.JobManager.finishJobStops(JobManager.java:8236) >> at >> org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:69) >> [2014-11-19 15:01:43,638][DEBUG][org.apache.manifoldcf.db ] Requested >> query: [SELECT id FROM jobs WHERE status IN (?,?,?,?,?,?,?,?,?,?)] >> [2014-11-19 15:01:43,638][DEBUG][org.apache.manifoldcf.db ] Actual query: >> [SELECT id FROM jobs WHERE status IN (?,?,?,?,?,?,?,?,?,?)] >> [2014-11-19 15:01:43,638][DEBUG][org.apache.manifoldcf.db ] Parameter >> 0: 'X' >> [2014-11-19 15:01:43,638][DEBUG][org.apache.manifoldcf.db ] Parameter >> 1: 'Y' >> [2014-11-19 15:01:43,639][DEBUG][org.apache.manifoldcf.db ] Parameter >> 2: 'M' >> [2014-11-19 15:01:43,639][DEBUG][org.apache.manifoldcf.db ] Parameter >> 3: 'v' >> [2014-11-19 15:01:43,639][DEBUG][org.apache.manifoldcf.db ] Parameter >> 4: 'F' >> [2014-11-19 15:01:43,639][DEBUG][org.apache.manifoldcf.db ] Parameter >> 5: 'f' >> [2014-11-19 15:01:43,639][DEBUG][org.apache.manifoldcf.db ] Parameter >> 6: 'H' >> [2014-11-19 15:01:43,639][DEBUG][org.apache.manifoldcf.db ] Parameter >> 7: 'h' >> [2014-11-19 15:01:43,639][DEBUG][org.apache.manifoldcf.db ] Parameter >> 8: 'G' >> [2014-11-19 15:01:43,639][DEBUG][org.apache.manifoldcf.db ] Parameter >> 9: 'g' >> [2014-11-19 15:01:43,639][DEBUG][org.apache.manifoldcf.db ] Done actual >> query (1ms): [SELECT id FROM jobs WHERE status IN (?,?,?,?,?,?,?,?,?,?)] >> [2014-11-19 15:01:43,640][DEBUG][org.apache.manifoldcf.db ] Requested >> query: [SELECT id FROM jobqueue WHERE jobid=? AND status IN (?,?,?,?,?) >> LIMIT 1] >> [2014-11-19 15:01:43,640][DEBUG][org.apache.manifoldcf.db ] Actual query: >> [SELECT id FROM jobqueue WHERE jobid=? AND status IN (?,?,?,?,?) LIMIT 1] >> [2014-11-19 15:01:43,640][DEBUG][org.apache.manifoldcf.db ] Parameter >> 0: '1416219496151' >> [2014-11-19 15:01:43,640][DEBUG][org.apache.manifoldcf.db ] Parameter >> 1: 'A' >> [2014-11-19 15:01:43,640][DEBUG][org.apache.manifoldcf.db ] Parameter >> 2: 'F' >> [2014-11-19 15:01:43,640][DEBUG][org.apache.manifoldcf.db ] Parameter >> 3: 'a' >> [2014-11-19 15:01:43,640][DEBUG][org.apache.manifoldcf.db ] Parameter >> 4: 'f' >> [2014-11-19 15:01:43,640][DEBUG][org.apache.manifoldcf.db ] Parameter >> 5: 'd' >> [2014-11-19 15:01:43,641][DEBUG][org.apache.manifoldcf.db ] Done actual >> query (1ms): [SELECT id FROM jobqueue WHERE jobid=? AND status IN >> (?,?,?,?,?) LIMIT 1] >> [2014-11-19 15:01:43,641][DEBUG][org.apache.manifoldcf.db ] Requested >> query: [UPDATE jobqueue SET docpriority=?,priorityset=NULL WHERE jobid=?] >> [2014-11-19 15:01:43,642][DEBUG][org.apache.manifoldcf.db ] Actual query: >> [UPDATE jobqueue SET docpriority=?,priorityset=NULL WHERE jobid=?] >> [2014-11-19 15:01:43,642][DEBUG][org.apache.manifoldcf.db ] Parameter >> 0: '1.000000001E9' >> [2014-11-19 15:01:43,642][DEBUG][org.apache.manifoldcf.db ] Parameter >> 1: '1416219496151' >> [2014-11-19 15:01:43,643][DEBUG][org.apache.manifoldcf.db ] Done actual >> query (1ms): [UPDATE jobqueue SET docpriority=?,priorityset=NULL WHERE >> jobid=?] >> [2014-11-19 15:01:43,643][DEBUG][org.apache.manifoldcf.db ] Beginning >> transaction of type 1 >> [2014-11-19 15:01:43,643][DEBUG][org.apache.manifoldcf.db ] Ending >> transaction >> [2014-11-19 15:01:43,643][DEBUG][org.apache.manifoldcf.db ] Beginning >> transaction of type 1 >> [2014-11-19 15:01:43,643][DEBUG][org.apache.manifoldcf.db ] Requested >> query: [SELECT status FROM jobs WHERE id=? FOR UPDATE] >> [2014-11-19 15:01:43,645][DEBUG][org.apache.manifoldcf.db ] Actual query: >> [START TRANSACTION] >> [2014-11-19 15:01:43,646][DEBUG][org.apache.manifoldcf.db ] Done actual >> query (1ms): [START TRANSACTION] >> [2014-11-19 15:01:43,646][DEBUG][org.apache.manifoldcf.db ] Actual query: >> [SELECT status FROM jobs WHERE id=? FOR UPDATE] >> [2014-11-19 15:01:43,646][DEBUG][org.apache.manifoldcf.db ] Parameter >> 0: '1416219496151' >> [2014-11-19 15:01:43,647][DEBUG][org.apache.manifoldcf.db ] Done actual >> query (1ms): [SELECT status FROM jobs WHERE id=? FOR UPDATE] >> [2014-11-19 15:01:43,647][DEBUG][org.apache.manifoldcf.db ] Ending >> transaction >> [2014-11-19 15:01:43,647][DEBUG][org.apache.manifoldcf.db ] Rolling >> transaction back! >> [2014-11-19 15:01:43,648][DEBUG][org.apache.manifoldcf.db ] Actual query: >> [ROLLBACK] >> [2014-11-19 15:01:43,648][DEBUG][org.apache.manifoldcf.db ] Done actual >> query (0ms): [ROLLBACK] >> [2014-11-19 15:01:43,648][ERROR][org.apache.manifoldcf.crawlerthreads] >> Exception tossed: Unexpected value for job status: 37 >> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected >> value for job status: 37 >> at >> org.apache.manifoldcf.crawler.jobs.Jobs.finishStopJob(Jobs.java:2973) >> at >> org.apache.manifoldcf.crawler.jobs.JobManager.finishJobStops(JobManager.java:8236) >> at >> org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:69) >> [2014-11-19 15:01:43,649][DEBUG][org.apache.manifoldcf.db ] Requested >> query: [SELECT id FROM jobs WHERE status IN (?,?,?,?,?,?,?,?,?,?)] >> [2014-11-19 15:01:43,649][DEBUG][org.apache.manifoldcf.db ] Actual query: >> [SELECT id FROM jobs WHERE status IN (?,?,?,?,?,?,?,?,?,?)] >> [2014-11-19 15:01:43,649][DEBUG][org.apache.manifoldcf.db ] Parameter >> 0: 'X' >> [2014-11-19 15:01:43,649][DEBUG][org.apache.manifoldcf.db ] Parameter >> 1: 'Y' >> [2014-11-19 15:01:43,649][DEBUG][org.apache.manifoldcf.db ] Parameter >> 2: 'M' >> [2014-11-19 15:01:43,649][DEBUG][org.apache.manifoldcf.db ] Parameter >> 3: 'v' >> [2014-11-19 15:01:43,649][DEBUG][org.apache.manifoldcf.db ] Parameter >> 4: 'F' >> [2014-11-19 15:01:43,649][DEBUG][org.apache.manifoldcf.db ] Parameter >> 5: 'f' >> [2014-11-19 15:01:43,649][DEBUG][org.apache.manifoldcf.db ] Parameter >> 6: 'H' >> [2014-11-19 15:01:43,649][DEBUG][org.apache.manifoldcf.db ] Parameter >> 7: 'h' >> [2014-11-19 15:01:43,649][DEBUG][org.apache.manifoldcf.db ] Parameter >> 8: 'G' >> [2014-11-19 15:01:43,649][DEBUG][org.apache.manifoldcf.db ] Parameter >> 9: 'g' >> [2014-11-19 15:01:43,650][DEBUG][org.apache.manifoldcf.db ] Done actual >> query (1ms): [SELECT id FROM jobs WHERE status IN (?,?,?,?,?,?,?,?,?,?)] >> [2014-11-19 15:01:43,650][DEBUG][org.apache.manifoldcf.db ] Requested >> query: [SELECT id FROM jobqueue WHERE jobid=? AND status IN (?,?,?,?,?) >> LIMIT 1] >> [2014-11-19 15:01:43,651][DEBUG][org.apache.manifoldcf.db ] Actual query: >> [SELECT id FROM jobqueue WHERE jobid=? AND status IN (?,?,?,?,?) LIMIT 1] >> [2014-11-19 15:01:43,651][DEBUG][org.apache.manifoldcf.db ] Parameter >> 0: '1416219496151' >> [2014-11-19 15:01:43,651][DEBUG][org.apache.manifoldcf.db ] Parameter >> 1: 'A' >> [2014-11-19 15:01:43,651][DEBUG][org.apache.manifoldcf.db ] Parameter >> 2: 'F' >> [2014-11-19 15:01:43,651][DEBUG][org.apache.manifoldcf.db ] Parameter >> 3: 'a' >> [2014-11-19 15:01:43,651][DEBUG][org.apache.manifoldcf.db ] Parameter >> 4: 'f' >> [2014-11-19 15:01:43,651][DEBUG][org.apache.manifoldcf.db ] Parameter >> 5: 'd' >> [2014-11-19 15:01:43,652][DEBUG][org.apache.manifoldcf.db ] Done actual >> query (1ms): [SELECT id FROM jobqueue WHERE jobid=? AND status IN >> (?,?,?,?,?) LIMIT 1] >> [2014-11-19 15:01:43,652][DEBUG][org.apache.manifoldcf.db ] Requested >> query: [UPDATE jobqueue SET docpriority=?,priorityset=NULL WHERE jobid=?] >> [2014-11-19 15:01:43,652][DEBUG][org.apache.manifoldcf.db ] Actual query: >> [UPDATE jobqueue SET docpriority=?,priorityset=NULL WHERE jobid=?] >> [2014-11-19 15:01:43,652][DEBUG][org.apache.manifoldcf.db ] Parameter >> 0: '1.000000001E9' >> [2014-11-19 15:01:43,652][DEBUG][org.apache.manifoldcf.db ] Parameter >> 1: '1416219496151' >> [2014-11-19 15:01:43,653][DEBUG][org.apache.manifoldcf.db ] Done actual >> query (1ms): [UPDATE jobqueue SET docpriority=?,priorityset=NULL WHERE >> jobid=?] >> [2014-11-19 15:01:43,654][DEBUG][org.apache.manifoldcf.db ] Beginning >> transaction of type 1 >> [2014-11-19 15:01:43,654][DEBUG][org.apache.manifoldcf.db ] Ending >> transaction >> [2014-11-19 15:01:43,654][DEBUG][org.apache.manifoldcf.db ] Beginning >> transaction of type 1 >> [2014-11-19 15:01:43,654][DEBUG][org.apache.manifoldcf.db ] Requested >> query: [SELECT status FROM jobs WHERE id=? FOR UPDATE] >> [2014-11-19 15:01:43,654][DEBUG][org.apache.manifoldcf.db ] Actual query: >> [START TRANSACTION] >> [2014-11-19 15:01:43,654][DEBUG][org.apache.manifoldcf.db ] Done actual >> query (0ms): [START TRANSACTION] >> [2014-11-19 15:01:43,655][DEBUG][org.apache.manifoldcf.db ] Actual query: >> [SELECT status FROM jobs WHERE id=? FOR UPDATE] >> [2014-11-19 15:01:43,655][DEBUG][org.apache.manifoldcf.db ] Parameter >> 0: '1416219496151' >> [2014-11-19 15:01:43,656][DEBUG][org.apache.manifoldcf.db ] Done actual >> query (1ms): [SELECT status FROM jobs WHERE id=? FOR UPDATE] >> [2014-11-19 15:01:43,656][DEBUG][org.apache.manifoldcf.db ] Ending >> transaction >> [2014-11-19 15:01:43,656][DEBUG][org.apache.manifoldcf.db ] Rolling >> transaction back! >> [2014-11-19 15:01:43,656][DEBUG][org.apache.manifoldcf.db ] Actual query: >> [ROLLBACK] >> [2014-11-19 15:01:43,657][DEBUG][org.apache.manifoldcf.db ] Done actual >> query (1ms): [ROLLBACK] >> [2014-11-19 15:01:43,657][ERROR][org.apache.manifoldcf.crawlerthreads] >> Exception tossed: Unexpected value for job status: 37 >> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected >> value for job status: 37 >> at >> org.apache.manifoldcf.crawler.jobs.Jobs.finishStopJob(Jobs.java:2973) >> at >> org.apache.manifoldcf.crawler.jobs.JobManager.finishJobStops(JobManager.java:8236) >> at >> org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:69) >> [2014-11-19 15:01:43,658][DEBUG][org.apache.manifoldcf.db ] Requested >> query: [SELECT id FROM jobs WHERE status IN (?,?,?,?,?,?,?,?,?,?)] >> >> Thanks, >> Jitu >> >> On Tue, Nov 18, 2014 at 5:55 PM, Karl Wright <[email protected]> wrote: >> >>> I looked at this in more detail; the problem is a dangling state in the >>> framework. I should have a patch shortly. See CONNECTORS-1106. >>> >>> Karl >>> >>> >>> On Tue, Nov 18, 2014 at 7:18 AM, Karl Wright <[email protected]> wrote: >>> >>>> Please let me know if you were able to get past this. MCF's database >>>> states are quite resilient, so I'd be interested to see what kinds of >>>> problems you have. >>>> >>>> Karl >>>> >>>> >>>> On Tue, Nov 18, 2014 at 6:21 AM, Karl Wright <[email protected]> >>>> wrote: >>>> >>>>> If you are using zookeeper, you may also want to purge the zookeeper >>>>> disk area as well, before starting any MCF processes. >>>>> >>>>> Karl >>>>> >>>>> >>>>> On Tue, Nov 18, 2014 at 6:19 AM, Karl Wright <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Kamil, >>>>>> >>>>>> I really don't know the failure modes of postgresql when it runs out >>>>>> of disk space. But if transactional integrity is maintained, it should >>>>>> be >>>>>> sufficient to shut down all agents processes and start them back up. >>>>>> >>>>>> Karl >>>>>> >>>>>> >>>>>> On Tue, Nov 18, 2014 at 4:24 AM, Kamil Żyta <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> I have problem with fixing manifoldcf after postgresql run without >>>>>>> disk space, >>>>>>> I resize postgresql partition x2 but when i start agents log is full >>>>>>> of: >>>>>>> >>>>>>> ERROR 2014-11-18 10:18:01,269 (Job reset thread) - Exception tossed: >>>>>>> Unexpected value for job status: 37 >>>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: >>>>>>> Unexpected value for job status: 37 >>>>>>> at >>>>>>> org.apache.manifoldcf.crawler.jobs.Jobs.finishStopJob(Jobs.java:2973) >>>>>>> at >>>>>>> org.apache.manifoldcf.crawler.jobs.JobManager.finishJobStops(JobManager.java:8236) >>>>>>> at >>>>>>> org.apache.manifoldcf.crawler.system.JobResetThread.run(JobResetThread.java:69) >>>>>>> >>>>>>> is there a quick way to fix this without purge all postgresql db? >>>>>>> >>>>>>> K >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
