ktopcuoglu opened a new issue, #68508:
URL: https://github.com/apache/doris/issues/68508
### What's Wrong?
`S3ClientFactory::_cache` (`be/src/util/s3_util.h`) is a plain
`std::unordered_map<S3ClientConf, ..., S3ClientConfHash>` with no eviction,
TTL, or size cap:
```cpp
auto it = _cache.find(s3_conf);
if (it != _cache.end()) { ... }
...
auto [it, _] = _cache.emplace(s3_conf, std::move(obj_client));
```
(`s3_util.cpp`, `S3ClientFactory::create` — identical on `master` and
`branch-4.1`/4.1.4.)
`S3ClientConf`'s identity includes the **bucket**, so every distinct bucket
a BE process ever accesses creates and permanently retains a new cached client.
This is a real problem for any catalog where storage is bucket-per-table rather
than bucket-per-warehouse — e.g. AWS S3 Tables, which auto-provisions one
bucket per Iceberg table. A long-lived BE process that scans N distinct tables
accumulates N permanently-cached `Aws::S3::S3Client` objects, each with its own
libcurl connection pool (CA bundle parsed per pooled handle) and, when using
STS/WebIdentity-style credentials, an auxiliary STS credentials-provider
sub-client built via a bare default constructor — bypassing Doris's own
`caFile`/connection settings entirely.
Confirmed via jemalloc heap profiling (`prof_active`) that this is genuinely
**live** allocated memory (`jemalloc allocated`, not just retained/fragmented
`resident`), invisible to Doris's own `MemTracker`/workload-group accounting —
it shows up purely as `UntrackedMemory` while workload groups report ~0 the
whole time. One measured instance attributed roughly 700 MiB to these
allocation sites after a routine ETL window, in a steadily-climbing,
never-plateauing pattern that eventually leads to the process being killed
under memory pressure.
### How to Reproduce
1. Configure an external catalog (e.g. Iceberg REST) where table storage
maps to a distinct S3 bucket per table (S3 Tables does this automatically).
2. From a single long-lived BE process, scan many distinct tables over time,
no restarts between them.
3. Observe RSS / jemalloc `allocated` climbing roughly linearly with the
number of *distinct* buckets ever touched, never plateauing, independent of
query concurrency or workload-group memory settings.
4. Workload-group memory metrics stay near zero throughout — the growth is
invisible from Doris's own accounting.
### What You Expected
Some bound on `S3ClientFactory`'s cache — idle-TTL and/or LRU eviction,
similar to the bounded/expiring cache (`_azure_cache`, capacity 256, LRU +
expiry pruning) added for Azure clients in #68117. That PR explicitly left the
non-Azure `_cache` untouched.
### Version
Reproduced on the official 4.1.4 release (branch-4.1). Same unbounded
`_cache` design confirmed present on `master` by source inspection.
### Anything Else?
Happy to contribute a patch — fix direction: capacity-bounded LRU + idle-TTL
eviction on `_cache` (mirroring #68117's `_azure_cache`); share one
credentials-provider/STS client across cache entries that share the same
credential config instead of one per bucket; and separately, the
`maxConnections` fallback to 102400 in `_create_s3_client` when no value is
supplied looks worth revisiting too.
Related but distinct: #66300 (different trigger, same "unbounded S3
connection" smell); #66997/#65416/#67056 (separate BE thread-pool leak,
unrelated to this cache).
- [x] Yes, willing to submit a PR.
---
*This issue was drafted by an AI coding assistant (Claude) based on
source-code review and a heap-profiling investigation carried out by our
engineering team, and was reviewed and submitted by a human maintainer of this
report.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
