sivakumar-mahalingam opened a new issue, #4462:
URL: https://github.com/apache/amoro/issues/4462
### What happened?
The AMS overview page periodically refreshes its cached list of top tables.
This cache is shared between:
1. A background thread that refreshes the cache.
2. HTTP request threads that read the cache for the
`/api/ams/v1/overview/top` endpoint.
The cache is currently a regular `ArrayList`, which is not thread-safe.
During a refresh, AMS first removes every entry and then adds the updated
entries:
this.allTopTableItem.clear();
this.allTopTableItem.addAll(topTableItemMap.values());
These are two separate operations. An HTTP request can read the cache after
`clear()` has completed but before `addAll()` has completed. In that case, the
API may temporarily return an empty top-table list even though tables exist.
A request may also copy the list while it is being modified. Because the
read and write operations are not synchronized, this can produce an
inconsistent result or a runtime exception.
Expected behavior:
Every request should receive a complete cache snapshot—either the snapshot
from before the refresh or the new snapshot after the refresh.
Actual behavior:
Requests can observe the temporary or partially updated state of the cache
while a background refresh is running.
This issue is intermittent and depends on the timing of the background
refresh and incoming HTTP requests.
### Affects Versions
master
### What table formats are you seeing the problem on?
Mixed-Hive, Mixed-Iceberg, Paimon, Iceberg
### What engines are you seeing the problem on?
AMS
### How to reproduce
This is an intermittent concurrency issue.
1. Start AMS with several registered tables.
2. Configure a short overview cache refresh interval, for example:
```
overview-cache:
refresh-interval: 1s
```
3. Continuously send concurrent requests to:
```
GET /api/ams/v1/overview/top?order=asc&orderBy=tableSize&limit=10
```
4. Allow the scheduled OverviewManager refresh to run while the requests are
being processed.
5. Compare the returned top-table lists across requests.
Because refreshTableCache() calls clear() and addAll() separately on a
shared ArrayList, a request can run after clear() but before addAll(), or while
the ArrayList storage is being modified. The response can consequently contain
an empty or inconsistent top-table list even though registered tables exist.
The race can also be demonstrated with a concurrency test that repeatedly
invokes OverviewManager.refresh() in one thread while other threads call
getAllTopTableItem().
### Relevant log output
```shell
No deterministic log output is available because this is a timing-dependent
race. Depending on timing, the request may return an inconsistent result
without logging an exception.
```
### Anything else
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
