The "starting up" state means that MCF is resetting all documents to be PendingPurgatory, and setting other document states and statuses. I have seen situations where this takes longer than some configured Postgresql timeout and then the initialization does not complete, and is rolled back. I didn't think Postgresql could fail silently in this way, but if it does you should see indication of an error in the postgresql logs.
The best way of figuring out what MCF is up to is getting a thread dump of the agents process. If you could get that and send it along, I can tell you more. Karl On Sun, Sep 20, 2015 at 6:39 AM, Mike Caceres <[email protected]> wrote: > Hi, > > I am running a job to ingest documents from Documentum. Things are > generally working, but lately the jobs seems to hang for hours in the > Starting Up status. > > The version of ManifoldCF is 2.1. > > Here is what I see in Postgres and a bit of the log file, which seems to > show a cycle over and over trying to pick up jobs, but not getting the ones > with status B. > > Is there anything I can do to make the job change state to Active? > > Thank you! > Mike > > > dbname=# select id, description, status, startmethod from jobs; > id | description | status | startmethod > ---------------+--------------------------+--------+------------- > 1442492810054 | full-crawl-production-06 | B | D > > dbname=# select j.description, j.status, jq.status, jq.seedingprocessid, > count (*) from jobqueue jq right join jobs j on j.id = jq.jobid group by > j.description, j.status, jq.status, jq.seedingprocessid order by > j.description, j.status, jq.status; > description | status | status | seedingprocessid | count > --------------------------+--------+--------+------------------+-------- > full-crawl-production-06 | B | P | A | 12795 > > > DEBUG 2015-09-20 10:07:07,780 (Delete startup thread) - > org.apache.manifoldcf.db - Requested query: [SELECT id FROM jobs WHERE > status=? FOR UPDATE] > DEBUG 2015-09-20 10:07:07,780 (Thread-31855) - org.apache.manifoldcf.db - > Actual query: [START TRANSACTION] > DEBUG 2015-09-20 10:07:07,780 (Thread-31855) - org.apache.manifoldcf.db - > Done actual query (0ms): [START TRANSACTION] > DEBUG 2015-09-20 10:07:07,781 (Thread-31856) - org.apache.manifoldcf.db - > Actual query: [SELECT id FROM jobs WHERE status=? FOR UPDATE] > DEBUG 2015-09-20 10:07:07,781 (Thread-31856) - org.apache.manifoldcf.db - > Parameter 0: 'E' > DEBUG 2015-09-20 10:07:07,781 (Thread-31856) - org.apache.manifoldcf.db - > Done actual query (0ms): [SELECT id FROM jobs WHERE status=? FOR UPDATE] > DEBUG 2015-09-20 10:07:07,781 (Delete startup thread) - > org.apache.manifoldcf.db - Committing transaction! > DEBUG 2015-09-20 10:07:07,781 (Thread-31857) - org.apache.manifoldcf.db - > Actual query: [COMMIT] > DEBUG 2015-09-20 10:07:07,781 (Thread-31857) - org.apache.manifoldcf.db - > Done actual query (0ms): [COMMIT] > DEBUG 2015-09-20 10:07:07,782 (Delete startup thread) - > org.apache.manifoldcf.db - Ending transaction > DEBUG 2015-09-20 10:07:07,860 (Job reset thread) - > org.apache.manifoldcf.db - Requested query: [SELECT id FROM jobs WHERE > (status=? OR status=? OR status=? OR status=? OR status=? OR status=? OR > status=? OR status=? OR status=? OR status=?)] > DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - > Actual query: [SELECT id FROM jobs WHERE (status=? OR status=? OR status=? > OR status=? OR status=? OR status=? OR status=? OR status=? OR status=? OR > status=?)] > DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - > Parameter 0: 'X' > DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - > Parameter 1: 'Y' > DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - > Parameter 2: 'M' > DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - > Parameter 3: 'v' > DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - > Parameter 4: 'F' > DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - > Parameter 5: 'f' > DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - > Parameter 6: 'H' > DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - > Parameter 7: 'h' > DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - > Parameter 8: 'G' > DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - > Parameter 9: 'g' > DEBUG 2015-09-20 10:07:07,860 (Thread-31858) - org.apache.manifoldcf.db - > Done actual query (0ms): [SELECT id FROM jobs WHERE (status=? OR status=? > OR status=? OR status=? OR status=? OR status=? OR status=? OR status=? OR > status=? OR status=?)] > DEBUG 2015-09-20 10:07:07,861 (Job reset thread) - > org.apache.manifoldcf.db - Requested query: [SELECT id FROM jobs WHERE > (status=? OR status=?)] > DEBUG 2015-09-20 10:07:07,861 (Thread-31859) - org.apache.manifoldcf.db - > Actual query: [SELECT id FROM jobs WHERE (status=? OR status=?)] > DEBUG 2015-09-20 10:07:07,861 (Thread-31859) - org.apache.manifoldcf.db - > Parameter 0: 'I' > DEBUG 2015-09-20 10:07:07,861 (Thread-31859) - org.apache.manifoldcf.db - > Parameter 1: 'i' > DEBUG 2015-09-20 10:07:07,861 (Thread-31859) - org.apache.manifoldcf.db - > Done actual query (0ms): [SELECT id FROM jobs WHERE (status=? OR status=?)] > DEBUG 2015-09-20 10:07:07,861 (Job reset thread) - > org.apache.manifoldcf.db - Requested query: [SELECT id FROM jobs WHERE > status=?] > DEBUG 2015-09-20 10:07:07,862 (Thread-31860) - org.apache.manifoldcf.db - > Actual query: [SELECT id FROM jobs WHERE status=?] > DEBUG 2015-09-20 10:07:07,862 (Thread-31860) - org.apache.manifoldcf.db - > Parameter 0: 'S' > DEBUG 2015-09-20 10:07:07,862 (Thread-31860) - org.apache.manifoldcf.db - > Done actual query (0ms): [SELECT id FROM jobs WHERE status=?] > DEBUG 2015-09-20 10:07:07,979 (Job notification thread) - > org.apache.manifoldcf.db - Beginning transaction of type 1 > DEBUG 2015-09-20 10:07:07,979 (Job notification thread) - > org.apache.manifoldcf.db - Requested query: [SELECT id,failtime,failcount > FROM jobs WHERE status=? FOR UPDATE] > DEBUG 2015-09-20 10:07:07,979 (Thread-31861) - org.apache.manifoldcf.db - > Actual query: [START TRANSACTION] > DEBUG 2015-09-20 10:07:07,980 (Thread-31861) - org.apache.manifoldcf.db - > Done actual query (1ms): [START TRANSACTION] > DEBUG 2015-09-20 10:07:07,980 (Thread-31862) - org.apache.manifoldcf.db - > Actual query: [SELECT id,failtime,failcount FROM jobs WHERE status=? FOR > UPDATE] > DEBUG 2015-09-20 10:07:07,980 (Thread-31862) - org.apache.manifoldcf.db - > Parameter 0: 's' > DEBUG 2015-09-20 10:07:07,980 (Thread-31862) - org.apache.manifoldcf.db - > Done actual query (0ms): [SELECT id,failtime,failcount FROM jobs WHERE > status=? FOR UPDATE] > DEBUG 2015-09-20 10:07:07,980 (Job notification thread) - > org.apache.manifoldcf.db - Committing transaction! > DEBUG 2015-09-20 10:07:07,981 (Thread-31863) - org.apache.manifoldcf.db - > Actual query: [COMMIT] > DEBUG 2015-09-20 10:07:07,981 (Thread-31863) - org.apache.manifoldcf.db - > Done actual query (0ms): [COMMIT] > DEBUG 2015-09-20 10:07:07,981 (Job notification thread) - > org.apache.manifoldcf.db - Ending transaction > DEBUG 2015-09-20 10:07:07,981 (Job notification thread) - > org.apache.manifoldcf.db - Beginning transaction of type 1 > DEBUG 2015-09-20 10:07:07,981 (Job notification thread) - > org.apache.manifoldcf.db - Requested query: [SELECT id,failtime,failcount > FROM jobs WHERE status=? FOR UPDATE] > DEBUG 2015-09-20 10:07:07,981 (Thread-31864) - org.apache.manifoldcf.db - > Actual query: [START TRANSACTION] > DEBUG 2015-09-20 10:07:07,981 (Thread-31864) - org.apache.manifoldcf.db - > Done actual query (0ms): [START TRANSACTION] > DEBUG 2015-09-20 10:07:07,982 (Thread-31865) - org.apache.manifoldcf.db - > Actual query: [SELECT id,failtime,failcount FROM jobs WHERE status=? FOR > UPDATE] > DEBUG 2015-09-20 10:07:07,982 (Thread-31865) - org.apache.manifoldcf.db - > Parameter 0: 'd' > DEBUG 2015-09-20 10:07:07,982 (Thread-31865) - org.apache.manifoldcf.db - > Done actual query (0ms): [SELECT id,failtime,failcount FROM jobs WHERE > status=? FOR UPDATE] > DEBUG 2015-09-20 10:07:07,982 (Job notification thread) - > org.apache.manifoldcf.db - Committing transaction! > DEBUG 2015-09-20 10:07:07,982 (Thread-31866) - org.apache.manifoldcf.db - > Actual query: [COMMIT] > DEBUG 2015-09-20 10:07:07,982 (Thread-31866) - org.apache.manifoldcf.db - > Done actual query (0ms): [COMMIT] > DEBUG 2015-09-20 10:07:07,982 (Job notification thread) - > org.apache.manifoldcf.db - Ending transaction > DEBUG 2015-09-20 10:07:07,991 (Job start thread) - > org.apache.manifoldcf.crawlerthreads - Job start thread - checking for jobs > to start at 1442743627991 > DEBUG 2015-09-20 10:07:07,991 (Job start thread) - > org.apache.manifoldcf.db - Beginning transaction of type 1 > DEBUG 2015-09-20 10:07:07,991 (Job start thread) - > org.apache.manifoldcf.db - Requested query: [SELECT > id,lasttime,status,startmethod,connectionname FROM jobs WHERE (status=? OR > status=? OR status=? OR status=? OR status=?) AND startmethod!=? FOR UPDATE] > DEBUG 2015-09-20 10:07:07,991 (Thread-31867) - org.apache.manifoldcf.db - > Actual query: [START TRANSACTION] > DEBUG 2015-09-20 10:07:07,991 (Thread-31867) - org.apache.manifoldcf.db - > Done actual query (0ms): [START TRANSACTION] > DEBUG 2015-09-20 10:07:07,992 (Thread-31868) - org.apache.manifoldcf.db - > Actual query: [SELECT id,lasttime,status,startmethod,connectionname FROM > jobs WHERE (status=? OR status=? OR status=? OR status=? OR status=?) AND > startmethod!=? FOR UPDATE] > DEBUG 2015-09-20 10:07:07,992 (Thread-31868) - org.apache.manifoldcf.db - > Parameter 0: 'N' > DEBUG 2015-09-20 10:07:07,992 (Thread-31868) - org.apache.manifoldcf.db - > Parameter 1: 'W' > DEBUG 2015-09-20 10:07:07,992 (Thread-31868) - org.apache.manifoldcf.db - > Parameter 2: 'w' > DEBUG 2015-09-20 10:07:07,992 (Thread-31868) - org.apache.manifoldcf.db - > Parameter 3: 'Z' > DEBUG 2015-09-20 10:07:07,992 (Thread-31868) - org.apache.manifoldcf.db - > Parameter 4: 'z' > DEBUG 2015-09-20 10:07:07,992 (Thread-31868) - org.apache.manifoldcf.db - > Parameter 5: 'D' > DEBUG 2015-09-20 10:07:07,992 (Thread-31868) - org.apache.manifoldcf.db - > Done actual query (0ms): [SELECT > id,lasttime,status,startmethod,connectionname FROM jobs WHERE (status=? OR > status=? OR status=? OR status=? OR status=?) AND startmethod!=? FOR UPDATE] > DEBUG 2015-09-20 10:07:07,992 (Job start thread) - > org.apache.manifoldcf.db - Beginning transaction of type 1 > DEBUG 2015-09-20 10:07:07,992 (Job start thread) - > org.apache.manifoldcf.db - Ending transaction > DEBUG 2015-09-20 10:07:07,992 (Job start thread) - > org.apache.manifoldcf.db - Committing transaction! > DEBUG 2015-09-20 10:07:07,992 (Thread-31869) - org.apache.manifoldcf.db - > Actual query: [COMMIT] > DEBUG 2015-09-20 10:07:07,992 (Thread-31869) - org.apache.manifoldcf.db - > Done actual query (0ms): [COMMIT] > DEBUG 2015-09-20 10:07:07,993 (Job start thread) - > org.apache.manifoldcf.db - Ending transaction > DEBUG 2015-09-20 10:07:07,993 (Job start thread) - > org.apache.manifoldcf.db - Beginning transaction of type 1 > DEBUG 2015-09-20 10:07:07,993 (Job start thread) - > org.apache.manifoldcf.db - Requested query: [SELECT id,status FROM jobs > WHERE (status=? OR status=? OR status=? OR status=? OR status=? OR > status=?) AND windowend<? FOR UPDATE] > DEBUG 2015-09-20 10:07:07,993 (Thread-31870) - org.apache.manifoldcf.db - > Actual query: [START TRANSACTION] > DEBUG 2015-09-20 10:07:07,993 (Thread-31870) - org.apache.manifoldcf.db - > Done actual query (0ms): [START TRANSACTION] > DEBUG 2015-09-20 10:07:07,993 (Thread-31871) - org.apache.manifoldcf.db - > Actual query: [SELECT id,status FROM jobs WHERE (status=? OR status=? OR > status=? OR status=? OR status=? OR status=?) AND windowend<? FOR UPDATE] > DEBUG 2015-09-20 10:07:07,993 (Thread-31871) - org.apache.manifoldcf.db - > Parameter 0: 'A' > DEBUG 2015-09-20 10:07:07,993 (Thread-31871) - org.apache.manifoldcf.db - > Parameter 1: 'a' > DEBUG 2015-09-20 10:07:07,993 (Thread-31871) - org.apache.manifoldcf.db - > Parameter 2: 'R' > DEBUG 2015-09-20 10:07:07,993 (Thread-31871) - org.apache.manifoldcf.db - > Parameter 3: 'r' > DEBUG 2015-09-20 10:07:07,993 (Thread-31871) - org.apache.manifoldcf.db - > Parameter 4: 'P' > DEBUG 2015-09-20 10:07:07,993 (Thread-31871) - org.apache.manifoldcf.db - > Parameter 5: 'p' > DEBUG 2015-09-20 10:07:07,993 (Thread-31871) - org.apache.manifoldcf.db - > Parameter 6: '1442743627991' > DEBUG 2015-09-20 10:07:07,994 (Thread-31871) - org.apache.manifoldcf.db - > Done actual query (1ms): [SELECT id,status FROM jobs WHERE (status=? OR > status=? OR status=? OR status=? OR status=? OR status=?) AND windowend<? > FOR UPDATE] > DEBUG 2015-09-20 10:07:07,994 (Job start thread) - > org.apache.manifoldcf.db - Ending transaction > DEBUG 2015-09-20 10:07:07,994 (Job start thread) - > org.apache.manifoldcf.db - Committing transaction! > DEBUG 2015-09-20 10:07:07,994 (Thread-31872) - org.apache.manifoldcf.db - > Actual query: [COMMIT] > DEBUG 2015-09-20 10:07:07,994 (Thread-31872) - org.apache.manifoldcf.db - > Done actual query (0ms): [COMMIT] > DEBUG 2015-09-20 10:07:08,274 (Stuffer thread) - > org.apache.manifoldcf.crawlerthreads - Document stuffer thread woke up >
