sundapeng opened a new pull request, #10812:
URL: https://github.com/apache/paimon/pull/10812
### Purpose
Containers reading through paimon-jindo, such as compaction TaskManagers,
get OOMKilled while the Java heap is far from full. JindoSDK keeps a readahead
cache of tens of MiB per open read stream outside the JVM, and glibc keeps most
of it resident after the streams close. Nothing in the logs shows how many
streams were open or how much native memory Jindo held when the container died.
This PR adds an opt-in, JVM-wide reporter to paimon-jindo:
- `JindoIOStats` counts the streams opened through `JindoFileIO`: currently
open, peak since the previous line, opened, closed and bytes. A stream closed
twice is counted once.
- With `jindo.io-stats.interval` set (for example `1 min`), a daemon thread
logs one INFO line per interval while streams are open or have been opened or
closed since the previous line. The default is `0`, which disables the reporter.
- The line combines the stream counters with:
- JindoSDK's native metrics: prefetch cache size and capacity, open native
streams, prefetch fallbacks on insufficient memory.
- Process memory: the RSS split, heap, direct memory, metaspace, threads,
and whether jemalloc is loaded.
- The memory cgroup (v1 or v2): usage and limit, working set, peak, limit
hits and OOM kills.
- The thread samples the cgroup every second and logs the same line at WARN
when the working set first crosses 80%, 90% or 95% of the limit, resetting
below 70%. The last line before an OOM kill therefore shows what was open.
- Native metrics are read only through cached file systems, which are never
closed. A failure is logged at DEBUG and turns that section off; it never
affects reads or writes.
Example (JDK 11, 16 read streams in a 1100 MiB memory cgroup):
```
WARN JindoIOStats - Container memory working set is at 85% of the limit.
Jindo IO stats: input open=16 peak=16 opened=16 closed=0 read=384MiB; output
open=0 peak=0 opened=0 closed=0 written=0MiB; jindo
prefetchCache=346MiB/4423MiB readStreams=16 writeStreams=0 memInsufficient=0;
process rss=1003MiB anon=939MiB file=63MiB shmem=0MiB threads=42
heap=69MiB/494MiB/494MiB direct=0MiB metaspace=13MiB jemalloc=false; cgroup
usage=943MiB/1100MiB workingSet=943MiB peak=943MiB limitHits=0 oomKills=0
```
### Tests
- `JindoIOStatsTest` (8 tests) covers:
- the counters and peak, and stream counting through
`HadoopCompliantFileIO` with a mocked `JindoHadoopSystem` (including double
close and byte counts);
- activity detection after closes;
- parsing the native metrics JSON;
- cgroup v1, v2 and none;
- the alarm levels.
- All 41 tests in `paimon-jindo` pass, and so do checkstyle and spotless.
- I also ran it by hand on JDK 11 against OSS with jindosdk 6.10.2, in
memory-limited cgroups:
- **1100 MiB limit:** 16 read streams logged the 85% WARN above. After
they closed, `prefetchCache` was 0 MiB while anon stayed at 934 MiB, which is
glibc retention rather than Jindo.
- **960 MiB limit:** the 90% WARN, with `input open=16` and
`prefetchCache=308MiB`, was the last line before the kernel OOM-killed the JVM.
- Overhead: about 50 us per one-second sample, and about 0.2 ms per line
plus about 0.4 ms for the native metrics dump.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
