Hi,
I am running a job to ingest documents from Documentum. Things are generally
working, but lately the jobs seems to hang for hours in the Starting Up status.
The version of ManifoldCF is 2.1.
Here is what I see in Postgres and a bit of the log file, which seems to show a
cycle over and over trying to pick up jobs, but not getting the ones with
status B.
Is there anything I can do to make the job change state to Active?
Thank you!Mike
dbname=# select id, description, status, startmethod from jobs; id |
description | status |
startmethod---------------+--------------------------+--------+-------------
1442492810054 | full-crawl-production-06 | B | D
dbname=# select j.description, j.status, jq.status, jq.seedingprocessid, count
(*) from jobqueue jq right join jobs j on j.id = jq.jobid group by
j.description, j.status, jq.status, jq.seedingprocessid order by j.description,
j.status, jq.status; description | status | status |
seedingprocessid |
count--------------------------+--------+--------+------------------+--------
full-crawl-production-06 | B | P | A | 12795
DEBUG 2015-09-20 10:07:07,780 (Delete startup thread) -
org.apache.manifoldcf.db - Requested query: [SELECT id FROM jobs WHERE status=?
FOR UPDATE]DEBUG 2015-09-20 10:07:07,780 (Thread-31855) -
org.apache.manifoldcf.db - Actual query: [START TRANSACTION]DEBUG 2015-09-20
10:07:07,780 (Thread-31855) - org.apache.manifoldcf.db - Done actual query
(0ms): [START TRANSACTION]DEBUG 2015-09-20 10:07:07,781 (Thread-31856) -
org.apache.manifoldcf.db - Actual query: [SELECT id FROM jobs WHERE status=?
FOR UPDATE]DEBUG 2015-09-20 10:07:07,781 (Thread-31856) -
org.apache.manifoldcf.db - Parameter 0: 'E'DEBUG 2015-09-20 10:07:07,781
(Thread-31856) - org.apache.manifoldcf.db - Done actual query (0ms): [SELECT id
FROM jobs WHERE status=? FOR UPDATE]DEBUG 2015-09-20 10:07:07,781 (Delete
startup thread) - org.apache.manifoldcf.db - Committing transaction!DEBUG
2015-09-20 10:07:07,781 (Thread-31857) - org.apache.manifoldcf.db - Actual
query: [COMMIT]DEBUG 2015-09-20 10:07:07,781 (Thread-31857) -
org.apache.manifoldcf.db - Done actual query (0ms): [COMMIT]DEBUG 2015-09-20
10:07:07,782 (Delete startup thread) - org.apache.manifoldcf.db - Ending
transactionDEBUG 2015-09-20 10:07:07,860 (Job reset thread) -
org.apache.manifoldcf.db - Requested query: [SELECT id FROM jobs WHERE
(status=? OR status=? OR status=? OR status=? OR status=? OR status=? OR
status=? OR status=? OR status=? OR status=?)]DEBUG 2015-09-20 10:07:07,860
(Thread-31858) - org.apache.manifoldcf.db - Actual query: [SELECT id FROM jobs
WHERE (status=? OR status=? OR status=? OR status=? OR status=? OR status=? OR
status=? OR status=? OR status=? OR status=?)]DEBUG 2015-09-20 10:07:07,860
(Thread-31858) - org.apache.manifoldcf.db - Parameter 0: 'X'DEBUG 2015-09-20
10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - Parameter 1:
'Y'DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db -
Parameter 2: 'M'DEBUG 2015-09-20 10:07:07,860 (Thread-31858) -
org.apache.manifoldcf.db - Parameter 3: 'v'DEBUG 2015-09-20 10:07:07,860
(Thread-31858) - org.apache.manifoldcf.db - Parameter 4: 'F'DEBUG 2015-09-20
10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - Parameter 5:
'f'DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db -
Parameter 6: 'H'DEBUG 2015-09-20 10:07:07,860 (Thread-31858) -
org.apache.manifoldcf.db - Parameter 7: 'h'DEBUG 2015-09-20 10:07:07,860
(Thread-31858) - org.apache.manifoldcf.db - Parameter 8: 'G'DEBUG 2015-09-20
10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - Parameter 9:
'g'DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db -
Done actual query (0ms): [SELECT id FROM jobs WHERE (status=? OR status=? OR
status=? OR status=? OR status=? OR status=? OR status=? OR status=? OR
status=? OR status=?)]DEBUG 2015-09-20 10:07:07,861 (Job reset thread) -
org.apache.manifoldcf.db - Requested query: [SELECT id FROM jobs WHERE
(status=? OR status=?)]DEBUG 2015-09-20 10:07:07,861 (Thread-31859) -
org.apache.manifoldcf.db - Actual query: [SELECT id FROM jobs WHERE (status=?
OR status=?)]DEBUG 2015-09-20 10:07:07,861 (Thread-31859) -
org.apache.manifoldcf.db - Parameter 0: 'I'DEBUG 2015-09-20 10:07:07,861
(Thread-31859) - org.apache.manifoldcf.db - Parameter 1: 'i'DEBUG 2015-09-20
10:07:07,861 (Thread-31859) - org.apache.manifoldcf.db - Done actual query
(0ms): [SELECT id FROM jobs WHERE (status=? OR status=?)]DEBUG 2015-09-20
10:07:07,861 (Job reset thread) - org.apache.manifoldcf.db - Requested query:
[SELECT id FROM jobs WHERE status=?]DEBUG 2015-09-20 10:07:07,862
(Thread-31860) - org.apache.manifoldcf.db - Actual query: [SELECT id FROM jobs
WHERE status=?]DEBUG 2015-09-20 10:07:07,862 (Thread-31860) -
org.apache.manifoldcf.db - Parameter 0: 'S'DEBUG 2015-09-20 10:07:07,862
(Thread-31860) - org.apache.manifoldcf.db - Done actual query (0ms): [SELECT id
FROM jobs WHERE status=?]DEBUG 2015-09-20 10:07:07,979 (Job notification
thread) - org.apache.manifoldcf.db - Beginning transaction of type 1DEBUG
2015-09-20 10:07:07,979 (Job notification thread) - org.apache.manifoldcf.db -
Requested query: [SELECT id,failtime,failcount FROM jobs WHERE status=? FOR
UPDATE]DEBUG 2015-09-20 10:07:07,979 (Thread-31861) - org.apache.manifoldcf.db
- Actual query: [START TRANSACTION]DEBUG 2015-09-20 10:07:07,980 (Thread-31861)
- org.apache.manifoldcf.db - Done actual query (1ms): [START TRANSACTION]DEBUG
2015-09-20 10:07:07,980 (Thread-31862) - org.apache.manifoldcf.db - Actual
query: [SELECT id,failtime,failcount FROM jobs WHERE status=? FOR UPDATE]DEBUG
2015-09-20 10:07:07,980 (Thread-31862) - org.apache.manifoldcf.db - Parameter
0: 's'DEBUG 2015-09-20 10:07:07,980 (Thread-31862) - org.apache.manifoldcf.db -
Done actual query (0ms): [SELECT id,failtime,failcount FROM jobs WHERE status=?
FOR UPDATE]DEBUG 2015-09-20 10:07:07,980 (Job notification thread) -
org.apache.manifoldcf.db - Committing transaction!DEBUG 2015-09-20 10:07:07,981
(Thread-31863) - org.apache.manifoldcf.db - Actual query: [COMMIT]DEBUG
2015-09-20 10:07:07,981 (Thread-31863) - org.apache.manifoldcf.db - Done actual
query (0ms): [COMMIT]DEBUG 2015-09-20 10:07:07,981 (Job notification thread) -
org.apache.manifoldcf.db - Ending transactionDEBUG 2015-09-20 10:07:07,981 (Job
notification thread) - org.apache.manifoldcf.db - Beginning transaction of type
1DEBUG 2015-09-20 10:07:07,981 (Job notification thread) -
org.apache.manifoldcf.db - Requested query: [SELECT id,failtime,failcount FROM
jobs WHERE status=? FOR UPDATE]DEBUG 2015-09-20 10:07:07,981 (Thread-31864) -
org.apache.manifoldcf.db - Actual query: [START TRANSACTION]DEBUG 2015-09-20
10:07:07,981 (Thread-31864) - org.apache.manifoldcf.db - Done actual query
(0ms): [START TRANSACTION]DEBUG 2015-09-20 10:07:07,982 (Thread-31865) -
org.apache.manifoldcf.db - Actual query: [SELECT id,failtime,failcount FROM
jobs WHERE status=? FOR UPDATE]DEBUG 2015-09-20 10:07:07,982 (Thread-31865) -
org.apache.manifoldcf.db - Parameter 0: 'd'DEBUG 2015-09-20 10:07:07,982
(Thread-31865) - org.apache.manifoldcf.db - Done actual query (0ms): [SELECT
id,failtime,failcount FROM jobs WHERE status=? FOR UPDATE]DEBUG 2015-09-20
10:07:07,982 (Job notification thread) - org.apache.manifoldcf.db - Committing
transaction!DEBUG 2015-09-20 10:07:07,982 (Thread-31866) -
org.apache.manifoldcf.db - Actual query: [COMMIT]DEBUG 2015-09-20 10:07:07,982
(Thread-31866) - org.apache.manifoldcf.db - Done actual query (0ms):
[COMMIT]DEBUG 2015-09-20 10:07:07,982 (Job notification thread) -
org.apache.manifoldcf.db - Ending transactionDEBUG 2015-09-20 10:07:07,991 (Job
start thread) - org.apache.manifoldcf.crawlerthreads - Job start thread -
checking for jobs to start at 1442743627991DEBUG 2015-09-20 10:07:07,991 (Job
start thread) - org.apache.manifoldcf.db - Beginning transaction of type 1DEBUG
2015-09-20 10:07:07,991 (Job start thread) - org.apache.manifoldcf.db -
Requested query: [SELECT id,lasttime,status,startmethod,connectionname FROM
jobs WHERE (status=? OR status=? OR status=? OR status=? OR status=?) AND
startmethod!=? FOR UPDATE]DEBUG 2015-09-20 10:07:07,991 (Thread-31867) -
org.apache.manifoldcf.db - Actual query: [START TRANSACTION]DEBUG 2015-09-20
10:07:07,991 (Thread-31867) - org.apache.manifoldcf.db - Done actual query
(0ms): [START TRANSACTION]DEBUG 2015-09-20 10:07:07,992 (Thread-31868) -
org.apache.manifoldcf.db - Actual query: [SELECT
id,lasttime,status,startmethod,connectionname FROM jobs WHERE (status=? OR
status=? OR status=? OR status=? OR status=?) AND startmethod!=? FOR
UPDATE]DEBUG 2015-09-20 10:07:07,992 (Thread-31868) - org.apache.manifoldcf.db
- Parameter 0: 'N'DEBUG 2015-09-20 10:07:07,992 (Thread-31868) -
org.apache.manifoldcf.db - Parameter 1: 'W'DEBUG 2015-09-20 10:07:07,992
(Thread-31868) - org.apache.manifoldcf.db - Parameter 2: 'w'DEBUG 2015-09-20
10:07:07,992 (Thread-31868) - org.apache.manifoldcf.db - Parameter 3:
'Z'DEBUG 2015-09-20 10:07:07,992 (Thread-31868) - org.apache.manifoldcf.db -
Parameter 4: 'z'DEBUG 2015-09-20 10:07:07,992 (Thread-31868) -
org.apache.manifoldcf.db - Parameter 5: 'D'DEBUG 2015-09-20 10:07:07,992
(Thread-31868) - org.apache.manifoldcf.db - Done actual query (0ms): [SELECT
id,lasttime,status,startmethod,connectionname FROM jobs WHERE (status=? OR
status=? OR status=? OR status=? OR status=?) AND startmethod!=? FOR
UPDATE]DEBUG 2015-09-20 10:07:07,992 (Job start thread) -
org.apache.manifoldcf.db - Beginning transaction of type 1DEBUG 2015-09-20
10:07:07,992 (Job start thread) - org.apache.manifoldcf.db - Ending
transactionDEBUG 2015-09-20 10:07:07,992 (Job start thread) -
org.apache.manifoldcf.db - Committing transaction!DEBUG 2015-09-20 10:07:07,992
(Thread-31869) - org.apache.manifoldcf.db - Actual query: [COMMIT]DEBUG
2015-09-20 10:07:07,992 (Thread-31869) - org.apache.manifoldcf.db - Done actual
query (0ms): [COMMIT]DEBUG 2015-09-20 10:07:07,993 (Job start thread) -
org.apache.manifoldcf.db - Ending transactionDEBUG 2015-09-20 10:07:07,993 (Job
start thread) - org.apache.manifoldcf.db - Beginning transaction of type 1DEBUG
2015-09-20 10:07:07,993 (Job start thread) - org.apache.manifoldcf.db -
Requested query: [SELECT id,status FROM jobs WHERE (status=? OR status=? OR
status=? OR status=? OR status=? OR status=?) AND windowend<? FOR UPDATE]DEBUG
2015-09-20 10:07:07,993 (Thread-31870) - org.apache.manifoldcf.db - Actual
query: [START TRANSACTION]DEBUG 2015-09-20 10:07:07,993 (Thread-31870) -
org.apache.manifoldcf.db - Done actual query (0ms): [START TRANSACTION]DEBUG
2015-09-20 10:07:07,993 (Thread-31871) - org.apache.manifoldcf.db - Actual
query: [SELECT id,status FROM jobs WHERE (status=? OR status=? OR status=? OR
status=? OR status=? OR status=?) AND windowend<? FOR UPDATE]DEBUG 2015-09-20
10:07:07,993 (Thread-31871) - org.apache.manifoldcf.db - Parameter 0:
'A'DEBUG 2015-09-20 10:07:07,993 (Thread-31871) - org.apache.manifoldcf.db -
Parameter 1: 'a'DEBUG 2015-09-20 10:07:07,993 (Thread-31871) -
org.apache.manifoldcf.db - Parameter 2: 'R'DEBUG 2015-09-20 10:07:07,993
(Thread-31871) - org.apache.manifoldcf.db - Parameter 3: 'r'DEBUG 2015-09-20
10:07:07,993 (Thread-31871) - org.apache.manifoldcf.db - Parameter 4:
'P'DEBUG 2015-09-20 10:07:07,993 (Thread-31871) - org.apache.manifoldcf.db -
Parameter 5: 'p'DEBUG 2015-09-20 10:07:07,993 (Thread-31871) -
org.apache.manifoldcf.db - Parameter 6: '1442743627991'DEBUG 2015-09-20
10:07:07,994 (Thread-31871) - org.apache.manifoldcf.db - Done actual query
(1ms): [SELECT id,status FROM jobs WHERE (status=? OR status=? OR status=? OR
status=? OR status=? OR status=?) AND windowend<? FOR UPDATE]DEBUG 2015-09-20
10:07:07,994 (Job start thread) - org.apache.manifoldcf.db - Ending
transactionDEBUG 2015-09-20 10:07:07,994 (Job start thread) -
org.apache.manifoldcf.db - Committing transaction!DEBUG 2015-09-20 10:07:07,994
(Thread-31872) - org.apache.manifoldcf.db - Actual query: [COMMIT]DEBUG
2015-09-20 10:07:07,994 (Thread-31872) - org.apache.manifoldcf.db - Done actual
query (0ms): [COMMIT]DEBUG 2015-09-20 10:07:08,274 (Stuffer thread) -
org.apache.manifoldcf.crawlerthreads - Document stuffer thread woke up
