Hi all,
While testing the MariaDB 12 upgrade work, I ran into a behavioral
regression in Fineract integration tests that appears related to MariaDB
12’s transaction behavior.
What I observed
After rebasing onto the latest develop and running the MariaDB 12 CI/test
flow, the failures were not startup, JDBC authentication, Liquibase, or
Hibernate dialect issues. MariaDB and Fineract both started correctly. The
actual failures happened later in concurrency-sensitive integration tests,
with MariaDB returning errors of the form: *Record has changed since last
read*
I reproduced this in tests including:
-
FeignTrialBalanceSummaryReportTest.testExternalAssetOwnerEntriesAppearInReport
- ExternalAssetOwnerTransferCancelTest
-
InitiateExternalAssetOwnerTransferTest.saleTransferWithSameOwnerExternalIdInParallelShouldNotFail
- SavingsAccountTransactionTest.testDeadlockSavingsBatchTransactions
- SavingsAccountTransactionTest.testConcurrentSavingsBatchTransactions
The affected tables included:
- m_external_asset_owner_transfer
- m_savings_account
Important non-root-cause detail
The CI logs also showed:
Access denied for user 'runner'
That turned out to be only the readiness probe using mysqladmin ping without
credentials. The workflow continued into DB initialization immediately
afterwards, so that warning was noise, not the actual failure.
Working compatibility fix
I tested MariaDB 12.2 locally and found that explicitly setting:
innodb_snapshot_isolation=OFF restored the behavior those tests were
already passing against before the upgrade. With that setting applied, the
previously failed targeted MariaDB 12 integration tests passed locally.
I applied that compatibility setting consistently in the upgrade branch for:
- GitHub Actions MariaDB workflow
- Local Docker compose
- Kubernetes MariaDB deployment
- MariaDB Testcontainers setup used by tests
I want to open this discussion because this is not a minor toggle. This
variable is enabled by default in MariaDB 12, and turning it off should be
treated as a conscious compatibility decision, not as an invisible
implementation detail.
My current understanding is:
1.
For the MariaDB 12 upgrade PR, using innodb_snapshot_isolation=OFF is an
acceptable compatibility fix.
- It keeps the scope of the upgrade contained.
- It avoids mixing a database version bump with deeper
transaction/locking changes in application code.
2.
This should not be considered the long-term final state.
- Fineract already has optimistic locking / concurrency-sensitive flows.
- MariaDB 12’s default behavior is exposing places where that locking
behavior should be improved.
- We should raise a follow-up Jira to review and strengthen the
affected locking/transaction patterns so Fineract can move toward proper
support with innodb_snapshot_isolation=ON.
So the immediate proposal is:
- keep innodb_snapshot_isolation=OFF as the MariaDB 12 compatibility
setting for this upgrade
- document clearly why it is being turned off
- create a follow-up Jira for improving Fineract’s locking model and
concurrency handling under MariaDB 12 defaults
I would appreciate community input on whether this is the right short-term
path, and whether there are concerns with treating this as a compatibility
setting for the upgrade while follow-up locking work is done separately.
Ticket: https://issues.apache.org/jira/browse/FINERACT-2516
PR: https://github.com/apache/fineract/pull/5566
CC: @Victor Romero @Adam Monsen
Thanks,
Aira
