sundapeng opened a new pull request, #10580:
URL: https://github.com/apache/paimon/pull/10580
### Purpose
Only token refreshes of `RESTTokenFileIO` are logged today. A cache that
creates a client per token, a token that is refreshed on every access, or
streams that are never closed leave no trace until the process runs out of
memory or file descriptors. For example, a token storm on PyArrow S3 with
`ulimit -n 1024` hangs without a single warning.
This pull request adds three log lines, each with a bounded rate:
- **FileIO cache summary**, on a cache miss, at most every 10 minutes:
cached FileIOs and tokens, FileIOs created and evicted for size since the last
summary, and open file descriptors against the soft limit. It is a warning when
FileIOs were evicted for size or descriptors are above 80% of the limit, and
the first warning does not wait for the interval, because a storm exhausts
descriptors within a second. Counting descriptors costs about 0.4 ms at 1,000
of them, next to the token request and the new client of every miss. Cache hits
and idle processes log nothing.
- **Short-lived token warning**, at most every 10 minutes with a count, when
a refreshed token already falls within the refresh margin, so every access
refreshes it again.
- **Open Jindo streams**, when their number passes 256, 512, 1024 and so on,
so a stream leak logs O(log n) lines. Streams closed by garbage collection are
not reported: PyArrow drops the files it opens for dataset footers without
`close()`, so that number grows on healthy reads too.
### Tests
- `test_cache_misses_log_one_summary_per_interval`,
`test_short_lived_token_warns_once_per_interval` and
`JindoStreamStatsTest.test_logs_only_when_open_streams_pass_a_power_of_two`.
Removing any trigger or rate limit fails at least one of them. The two test
files pass on Python 3.11 and 3.13, and give the same results as master on
Python 3.6.
- pyjindo against a local mock OSS, with the real `refresh_token` and
pypaimon logs on:
- A 60 s stress run of 8 threads with a cache of 3 FileIOs: 5,766 log
lines, 6 of them new (the first summary, an eviction warning and the stream
counts from 256 to 2,048).
- The token storm on PyArrow S3 with `ulimit -n 1024` still hangs, but
first logs 3 new lines: the short-lived token warning, the first summary and a
warning with `543/1000 FileIOs ... open files 820/1024`.
- Exit, fault and refresh error scenarios behave as on master.
- flake8 with `dev/cfg.ini` passes.
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
