Piotr Wargulak created FINERACT-3004:
----------------------------------------
Summary: COB silently rolls back a whole chunk when one loan's
business step fails: loans stay locked with no error and the job reports success
Key: FINERACT-3004
URL: https://issues.apache.org/jira/browse/FINERACT-3004
Project: Apache Fineract
Issue Type: Bug
Reporter: Piotr Wargulak
When a business step throws for one loan during a scheduled Loan COB or Working
Capital Loan COB run, every loan of that chunk is left behind without any trace:
* the loans keep their {{LOAN_COB_CHUNK_PROCESSING}} lock with {{error}}
empty, so the lock is never overrulable;
* none of them gets {{last_closed_business_date}} moved, and none of the
chunk's work is kept, including the loans that had no problem;
* the step ends {{COMPLETED}} and the job run is recorded as {{{}success{}}}.
What happens, step by step (Spring Batch 6.0.5, {{{}ChunkOrientedStep{}}}):
# The item processor runs {{{}COBBusinessStepServiceImpl#run{}}}, which is
{{@Transactional}} and joins the chunk transaction. The step's exception leaves
that method, so the JPA transaction manager marks the chunk transaction
rollback-only ("Participating transaction failed - marking existing transaction
as rollback-only").
# The worker step is fault tolerant with an immediate retry policy, so
{{doProcess}} retries the item in the same transaction and persistence context.
Two outcomes:
** the retry succeeds (it can: the first attempt's changes are still in the
persistence context) and nothing is reported at all;
** the retries are used up: {{onProcessError}} is called, the item is skipped
({{{}doSkipInProcess{}}}), and the chunk goes on.
# After the chunk, {{doExecute}} finds
{{{}transactionStatus.isRollbackOnly(){}}}, rolls back and returns
({{{}ChunkOrientedStep:380-389{}}}). No exception, no listener, no scan; the
next chunk starts and the step completes.
A process skip therefore cannot work while processing writes in the chunk
transaction: the skipped item has already doomed the transaction of the items
processed with it.
In the retries-used-up case,
{{AbstractLoanItemListener#updateAccountLockWithError}} also failed for us with
{{UnexpectedRollbackException}} (Working Capital, on our build); we have not
established why its {{REQUIRES_NEW}} transaction is rollback-only.
*Steps to reproduce*
_(a GL closure on the COB date makes one loan's business step fail)_
Two attached integration tests, each with two loans of one chunk: one whose
step posts a journal entry on the COB date, and one with no accounting, which
posts nothing and so is not concerned by the closure.
* {{LoanCobBusinessStepFailureTest}} (Loan COB, based on
{{{}LoanAccrualTransactionOnChargeSubmittedDateTest{}}}): business date
2023-03-03 with {{charge-accrual-date}} = {{{}submitted-date{}}}; a
single-repayment loan with periodic accrual accounting and a penalty and a fee
charge, and a second loan on a product from the same template with accounting
none. Business date 2023-03-04, {{POST /glclosures}} for office 1 on
2023-03-03, run the Loan COB job.
{{LoanAccrualsProcessingServiceImpl#addPeriodicAccruals}} throws
{{{}JournalEntryInvalidException{}}}.
* {{FeignWorkingCapitalLoanCobBusinessStepFailureTest}} (Working Capital Loan
COB): a loan on an accrual ({{{}ACC_DEF_REV_AM{}}}) product with a discount and
a repayment on 2026-01-02, and a loan on a product without accounting. Business
date 2026-01-03, closure on 2026-01-02, run the Working Capital Loan COB job.
{{WorkingCapitalLoanDiscountFeeAmortizationServiceImpl#processDiscountFeeAmortization}}
throws {{{}JournalEntryInvalidException{}}}.
On develop 537e19aaa9 (PostgreSQL) both tests fail on the loan with nothing to
post: {{expected: <2023-03-03> but was: <null>}} and {{{}expected: <2026-01-02>
but was: <null>{}}}. After the runs both loans of each job keep
{{LOAN_COB_CHUNK_PROCESSING}} with no error, and both job runs are recorded as
{{{}success{}}}. The application log shows the exception, then "Transactional
code has requested rollback".
{*}Possible directions{*}:
* run each loan's business steps in their own transaction (a nested one, or
REQUIRES_NEW plus writing the loan in it), so one loan's failure does not doom
the chunk; this changes the semantics FINERACT-2684 settled;
* on a process failure, roll the chunk back and process it again without the
failed loan, recording the error on its lock in an independent transaction (as
Spring Batch 5's fault-tolerant chunk processor did for a process skip);
* at the very least, fail the step when a chunk is rolled back this way, so
the job does not report success while loans are left locked.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
