Wondering if any other insights into async_job entries with NULL job_cmd and never have a job_complete_msid. Here is some detailed info:
MariaDB [cloud]> select id, job_cmd,job_status,last_updated,created,job_complete_msid from async_job order by job_complete_msid,last_updated,created; +--------+-----------------------------------------------------------------------+------------+---------------------+---------------------+-------------------+ | id | job_cmd | job_status | last_updated | created | job_complete_msid | +--------+-----------------------------------------------------------------------+------------+---------------------+---------------------+-------------------+ | 626231 | NULL | 0 | NULL | 2025-01-24 04:58:57 | NULL | | 626236 | NULL | 0 | NULL | 2025-01-24 04:58:57 | NULL | | 627366 | NULL | 0 | NULL | 2025-01-30 20:19:51 | NULL | | 629751 | NULL | 0 | NULL | 2025-02-06 04:58:08 | NULL | MariaDB [cloud]> select uuid from async_job where id=629751; +--------------------------------------+ | uuid | +--------------------------------------+ | 7981ecf8-4f65-419c-8f1d-0f8289b9065a | Output from queryAsyncJobResult: https://xxxx/client/api?command=queryAsyncJobResult&jobid=7981ecf8-4f65-419c-8f1d-0f8289b9065a <queryasyncjobresultresponse cloud-stack-version="4.15.2.0"> <accountid>275f3f54-e249-11e5-90e8-005056a324ff</accountid> <userid>275f4b7b-e249-11e5-90e8-005056a324ff</userid> <jobstatus>0</jobstatus> <jobprocstatus>0</jobprocstatus> <jobresultcode>0</jobresultcode> <created>2025-02-06T04:58:08+0000</created> <jobid>7981ecf8-4f65-419c-8f1d-0f8289b9065a</jobid> </queryasyncjobresultresponse> MariaDB [cloud]> select * from user where user.uuid ='275f4b7b-e249-11e5-90e8-005056a324ff'\G *************************** 1. row *************************** id: 1 uuid: 275f4b7b-e249-11e5-90e8-005056a324ff username: system password: 0.9057436724246887 account_id: 1 firstname: system lastname: cloud email: NULL state: enabled api_key: NULL secret_key: NULL created: 2016-03-04 20:39:12 removed: NULL timezone: NULL registration_token: NULL is_registered: 0 incorrect_login_attempts: 0 default: 1 source: UNKNOWN external_entity: NULL MariaDB [cloud]> select * from account where uuid='275f3f54-e249-11e5-90e8-005056a324ff'\G *************************** 1. row *************************** id: 1 account_name: system uuid: 275f3f54-e249-11e5-90e8-005056a324ff type: 1 role_id: 1 domain_id: 1 state: enabled removed: NULL cleanup_needed: 0 network_domain: NULL default_zone_id: NULL default: 1 Some excerpts from the mgmt. server log: 2025-02-06 04:58:08,831 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-2f094c04) (logid:9255bef7) Execute sync-queue item: SyncQueueItemVO {id:33711, queueId: 281451, contentType: AsyncJob, contentId: 629756, lastProcessMsid: 345050927939, lastprocessNumber: 6, lastProcessTime: Thu Feb 06 04:58:08 UTC 2025, created: Thu Feb 06 04:58:08 UTC 2025} 2025-02-06 04:58:08,832 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-2f094c04) (logid:9255bef7) Schedule queued job-629756 2025-02-06 04:58:08,839 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756) (logid:ca42fda8) Add job-629756 into job monitoring 2025-02-06 04:58:08,845 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756) (logid:7981ecf8) Executing AsyncJobVO {id:629756, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo: rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAQRXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345050908593, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Feb 06 04:58:08 UTC 2025, removed: null} 2025-02-06 04:58:08,848 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756) (logid:7981ecf8) Run VM work job: com.cloud.vm.VmWorkStop for VM 4165, job origin: 629751 2025-02-06 04:58:08,849 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Execute VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":1,"accountId":1,"vmId":4165,"handlerName":"VirtualMachineManagerImpl"} 2025-02-06 04:58:08,852 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Stopped called on VM[User|i-2806-4165-VM] but the state is Destroyed 2025-02-06 04:58:08,852 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Done executing VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":1,"accountId":1,"vmId":4165,"handlerName":"VirtualMachineManagerImpl"} 2025-02-06 04:58:08,852 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Complete async job-629756, jobStatus: SUCCEEDED, resultCode: 0, result: null 2025-02-06 04:58:08,853 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Publish async job-629756 complete on message bus 2025-02-06 04:58:08,853 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Wake up jobs related to job-629756 2025-02-06 04:58:08,853 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Update db status for job-629756 2025-02-06 04:58:08,854 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Wake up jobs joined with job-629756 and disjoin all subjobs created from job- 629756 2025-02-06 04:58:08,860 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756) (logid:7981ecf8) Done with run of VM work job: com.cloud.vm.VmWorkStop for VM 4165, job origin: 629751 2025-02-06 04:58:08,860 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756) (logid:7981ecf8) Done executing com.cloud.vm.VmWorkStop for job-629756 2025-02-06 04:58:08,861 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756) (logid:7981ecf8) Remove job-629756 from job monitoring -----Original Message----- From: Daan Hoogland <daan.hoogl...@gmail.com> Sent: Monday, August 12, 2024 4:39 AM To: users@cloudstack.apache.org Subject: [CAUTION: SUSPECT SENDER] Re: Async Jobs Warning! This message was sent from outside your organization and we were unable to verify the sender. Steve, Did you get to the bottom of this? On Tue, May 28, 2024 at 4:56 PM S.Fuller <steveful...@gmail.com> wrote: > > I have a question about asynchronous jobs. My understanding is that > any async jobs will be recorded in the async_jobs table in the > database. They should have a value in the job_complete_msid table when > they are done, and then be removed 24 hours later. > > Every once in a while, we will have a job that shows up in the table, > with a NULL value for job_complete_msid implying that the job is still > running. > However, when I see this, there's a null command for the job_cmd and > the last_updated value as well. > > Should I be concerned, and do you have any suggestions on how to track > down what might be causing these? I might not see one of these for 4 > or 5 months, so it's not a large volume. > > -- > Steve Fuller > steveful...@gmail.com -- Daan
