sb-abhish3k opened a new pull request, #14169:
URL: https://github.com/apache/cloudstack/pull/14169
### Description
We are observing high failures from `CheckedReservation` timeouts and
elsewhere seeing `MySQLTransactionRollbackException: Lock wait timeout
exceeded` from a similar bottleneck with quota transaction timeouts.
During VM creation 5-7 `CheckedReservation` are constructed, which acquire a
`GlobalLock` and go check resource limits and create/persist a reservation.
During the "check resource limits" portion, in
`ResourceLimitManager#checkResourceLimitWithTag` within the context of a DB
transaction which has InnoDB timeout implications, it attempts before
proceeding and releasing the `GlobalLock` to:
1. lock account and owner domain rows
2. check account limits
3. check domain limits
The issue with step 3 is that it:
a. skips the `ROOT` domain explicitly after locking it in step 1 and
b. short circuits and doesn't even do the validation if the domain limits
are unlimited (`-1`)
We are currently configuring limits at the account's domain level only, and
have intermediate domain levels mapping to availability zones.
Effectively what the current unchanged code means for us is if we have
thousands of concurrent requests to create a VM, even if they are segmented by
accounts/domains in different zone domain hierarchies, they will _all_ try and
wait on other processes to lock on the same rows for the ROOT domain and the
zone domains (configured with -1), never throwing an exception and generally
bypassing the checks altogether.
It seems like an inefficiency to lock rows that will never be checked.
Fixing it should decrease the time it takes to provision a VM and reduce
transaction load on the DB, especially during create surges. It seems like it
could also apply to other users of CloudStack upstream.
#### Changes
This PR eliminates unnecessary `CheckedReservation` contention by removing
quota limit checks for cases where a tagged resource limit is `-1` (unlimited)
on a parent domain.
Summary of changes:
- engine/schema/.../ResourceLimitDao.java — added
listDomainIdsWithFiniteLimit interface method.
- engine/schema/.../ResourceLimitDaoImpl.java — added two named
SearchBuilders (tag-EQ and tag-NULL variants, both with domainId IN + max NEQ)
and the implementation, matching the existing constructor-built style.
- server/.../ResourceLimitManagerImpl.java — lockAccountAndOwnerDomainRows
now delegates to a new protected listRowsToLockForLimitCheck helper that locks
the account row + only ancestor-domain rows with a finite explicit limit, with
fallbacks to listAllRowsToUpdate for finite global default and
tagged-account-row create-on-miss.
- server/.../ResourceLimitManagerImplTest.java — 5 new tests covering the
four scenarios from the plan plus the create-on-miss fallback.
-
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [ ] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to
-->
#### How did you try to break this feature and the system with this change?
<!-- see how your change affects other areas of the code, etc. -->
<!-- Please read the
[CONTRIBUTING](https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
document -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
