tracy1014-hub commented on issue #65416:
URL: https://github.com/apache/doris/issues/65416#issuecomment-4998805450
### Update: PR #65586 helps but doesn't fully fix rs_normal growth — real
root cause is `fs.s3a.connection.request.timeout=60s` (hadoop-aws SDK v2 timer)
@yiguolei @zhangstar333 following up on my paimon-prod (Paimon 1.4.1 on S3,
Doris 4.1.1-rc01) reproduction. Compiled PR #65586 locally as suggested and
tested it side by side with vanilla — TL;DR: **PR #65586 fixes the
S3AFileSystem cache layer (correctly and provably), but it does not stop
rs_normal from growing at ~80/min in our Paimon-only workload.** The real leak
source is one level deeper.
#### Test 1 — PR #65586 alone
Built patched `hadoop-deps.jar` + `doris-fe.jar` from `apache/doris#65586`
head (`453115a4`), overlaid on `apache/doris:be-4.1.1`. Rolled to one BE
(be-3). Observed for ~15h side-by-side against three unpatched BEs.
`jmap -histo:live` on the patched BE showed the FS layer is fixed:
| class | vanilla BE (rs≈27k) | PR #65586 BE (rs≈27k) |
|---|---|---|
| `org.apache.hadoop.fs.s3a.S3AFileSystem` | 5,000+ | **6** |
| `org.apache.hadoop.fs.s3a.S3AInstrumentation` | 5,000+ | **6** |
| `software.amazon.awssdk.services.s3.DefaultS3Client` | ~5,000 | **13** |
So `doris.fs.cache.key` is doing exactly what it should — FS instances are
being reused. **But `rs_normal` (i.e. `sdk-ScheduledExecutor-*` worker threads)
kept climbing at the same rate as the unpatched BEs, and hit the 25k+ freeze
threshold in ~4h.**
The real numbers on the patched BE:
| class | count |
|---|---|
| `java.util.concurrent.ScheduledThreadPoolExecutor` | **5,895** |
| `software.amazon.awssdk.utils.NamedThreadFactory` | **5,536** |
| `java.util.concurrent.ThreadPoolExecutor$Worker` | **29,473** |
| `java.lang.Thread` | **29,533** |
MAT `merge_shortest_paths` on those STPEs shows the GC root is each STPE's
own daemon worker Thread (`sdk-ScheduledExecutor-N-M`) parked on
`DelayedWorkQueue.take` — classic strong-ref cycle.
#### Test 2 — JFR ThreadStart sampling: where do new STPEs come from
60-second JFR trace on be-3 captured **190 new `sdk-ScheduledExecutor`
thread starts**. Every single stack was identical:
```
java.util.concurrent.ThreadPoolExecutor.addWorker
← ScheduledThreadPoolExecutor.ensurePrestart
← ScheduledThreadPoolExecutor.delayedExecute
← ScheduledThreadPoolExecutor.schedule
←
software.amazon.awssdk.core.internal.http.timers.TimerUtils.timeSyncTaskIfNeeded
← source
```
Looking at `TimerUtils.timeSyncTaskIfNeeded` (SDK v2):
```java
if (timeout <= 0) return NoOpTimeoutTracker.INSTANCE; // ← short-circuit
// otherwise schedule the timer, which addWorker's a new daemon thread
```
The `timeout` is `fs.s3a.connection.request.timeout` → `apiCallTimeout` /
`apiCallAttemptTimeout` on `ClientOverrideConfiguration`. **hadoop-aws 3.4.2
default is 60 seconds** (see `Constants.DEFAULT_REQUEST_TIMEOUT_DURATION` and
`AWSClientConfig.createApiConnectionSettings`). Because SDK v2 uses
`ScheduledThreadPoolExecutor.corePoolSize=1, allowCoreThreadTimeOut=false` for
its internal timer STPE, every `schedule()` call ensures a core worker is
prestarted; those workers park forever and are strongly reachable via the
STPE's `workQueue` array.
Since Paimon `HadoopFileIO` issues an S3 request per split per read op, and
the FS cache reuse (post-#65586) is by-bucket only, **each S3 request still
triggers one `sdk-ScheduledExecutor` daemon thread that never dies**. Under our
workload that's ~80/min per BE — 4h to freeze.
#### The one-line fix
Zero code change, just a catalog property so hadoop-aws sets
`apiCallTimeout=0` and `TimerUtils` takes the no-op path:
```sql
ALTER CATALOG paimon SET PROPERTIES ("fs.s3a.connection.request.timeout" =
"0");
-- restart BE pods so hadoop Configuration picks up the new value on next
S3AFileSystem.initialize
```
Verified on paimon-prod for the last 30+ minutes across all 4 BEs (all on
the base `apache/doris:be-4.1.1` image, no fscachefix):
- `ScheduledThreadPoolExecutor` count: **0** (was 5,895)
- `NamedThreadFactory` count: **5** stable, not growing (was 5,536 and
growing)
- `rs_normal` thread count: **48-68 stable per BE**, not growing (was 20-27k
and growing)
- `total_threads`: 2058-2078 (was 24-29k)
- RSS: 1.2-2.5 GB per BE (was 6-12 GB)
Side effect is minor: no more 60s API call timeout, but socket/connection
timeouts (`fs.s3a.connection.timeout`, socket read timeout) still apply, and
Doris' query-level timeout already covers the user-facing SLA.
#### Suggestions for 4.1.4
1. Merge PR #65586 (still needed for correct FS reuse and credential
isolation).
2. Additionally, either default `apiCallTimeout` to 0 for the Doris-owned S3
clients, **or** document the interaction so users of Paimon-on-S3 know to set
`fs.s3a.connection.request.timeout=0` in their catalog. Right now the default
is a functional footgun on any long-running Doris BE that reads S3.
3. Optionally file an upstream `HADOOP-*` ticket asking whether SDK v2's
per-request timeout timer should share a bounded executor instead of relying on
the caller to shutdown/reuse — the fact that a single `schedule()` call
permanently adds a core worker is surprising and undocumented.
Happy to attach the full jmap/jstack/MAT artifacts if useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
