Arnold Galovics created FINERACT-2926:
-----------------------------------------
Summary: StuckJobListener waits forever when a COB partition was
killed
Key: FINERACT-2926
URL: https://issues.apache.org/jira/browse/FINERACT-2926
Project: Apache Fineract
Issue Type: Bug
Reporter: Arnold Galovics
Assignee: Arnold Galovics
🐛 Steps to Reproduce
1. Run Fineract 1.15.0 with batch manager and batch worker enabled in one
container, business date enabled, and 2 loans lagging behind the COB date.
2. Start Loan COB with POST /jobs/\{id}?command=executeJob.
3. Kill the container (docker kill) while the COB partition step is STARTED.
4. Start the container again and read the application log and the
batch_job_execution and batch_step_execution tables.
❌ Actual Result
• The job execution stays STARTED/UNKNOWN and the partition step stays STARTED
forever.
• StuckJobListener logs "Sleeping for a second to wait for the partitions to
complete for job <id>" in an endless loop and never restarts the job. Readiness
is not blocked.
• Boot resets job.currently_running, but the execution is never closed.
• Marking the partition FAILED does not help: the loop exits only on COMPLETED.
✅ Expected Result
• A partition step that has not been updated for longer than a timeout
(proposed: fineract.job.stuck-partition-timeout, default 10m) is marked FAILED,
so the job is restarted and re-runs only the unfinished partitions. COMPLETED
partitions are not re-run.
• No endless loop, and one stuck tenant does not block stuck-job recovery for
the tenants after it.
ℹ️ Notes
• Observed on Fineract 1.15.0, Docker, Postgres 16, single-instance topology.
Not tested: multi-instance topology.
• Workaround: set the STARTED step and job executions to ABANDONED and reboot.
ABANDONED only closes the execution; it cannot be restarted.
• is_suspended is deliberately not honoured: restarting a stuck job is crash
recovery, not a scheduled run.
🔧 Technical Notes
• StuckJobListener.java:17-45.
StuckJobExecutorServiceImpl#waitUntilAllPartitionsFinished loops on
JobExecutionRepository#getNotCompletedPartitionsCount, which counts every
non-partitioner step with STATUS <> COMPLETED, so FAILED, STOPPED and ABANDONED
worker steps also loop forever. The listener runs on an event thread and
handles tenants one after another.
• Fix: count only STARTING, STARTED and STOPPING steps; in each loop iteration
run an UPDATE setting such steps to FAILED (EXIT_CODE FAILED, END_TIME now)
where LAST_UPDATED is older than now minus the timeout. The existing
updateJobStatusToFailed and restart then re-run only FAILED partitions
(SimpleStepExecutionSplitter#shouldStart restarts only FAILED and STOPPED,
which is why ABANDONED is not the target state).
• A live worker updates LAST_UPDATED on every chunk commit, so it is not marked
dead. The timeout must exceed the slowest chunk. LAST_UPDATED is written in the
JVM time zone, so pods with mixed time zones would break the cutoff.
• Test: StuckJobExecutorServiceImplTest (Mockito): count 1 then 0 calls the
stale update with the right cutoff, then updateJobStatusToFailed, then restart;
count 0 goes straight to restart.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
