zclllyybb commented on issue #65524:
URL: https://github.com/apache/doris/issues/65524#issuecomment-4955343340
Breakwater-GitHub-Analysis-Slot: slot_84bc83b0a52a
This content is generated by AI for reference only.
Initial maintainer-facing analysis:
I would treat this as a valid FE lifecycle bug, not only a Ranger Admin-side
capacity problem. I checked the reported release commit
`5960d4cea0e6d2a08f62b45a07a662395d2bbcfc` / `4.1.0-rc03`, and the code path
matches the reported "+2 PolicyRefresher threads per successful checkpoint"
pattern.
Code evidence:
- `Checkpoint.doCheckpoint()` creates a checkpoint `Env` for replay/save,
destroys it after saving the image, then creates another checkpoint `Env` to
load and verify the generated image. In the 4.1.0 tag this is the
`Env.getCurrentEnv()` path around `Checkpoint.java` lines 156 and 182, with
`Env.destroyCheckpoint()` around lines 176 and 199.
- `Env.getCurrentEnv()` creates the checkpoint environment via
`EnvFactory.getInstance().createEnv(true)` when the current thread is the
checkpoint thread.
- The `Env` constructor always creates a new `AccessControllerManager(auth)`.
- `AccessControllerManager` loads the configured default access controller
during construction. With `access_controller_type=ranger-doris`,
`RangerDorisAccessControllerFactory.createAccessController()` returns `new
RangerDorisAccessController("doris")`.
- `RangerDorisAccessController` constructs a `RangerDorisPlugin`, and
`RangerDorisPlugin` calls `RangerBasePlugin.init()` in its constructor. That is
the path that starts the Ranger policy refresher.
- `Env.destroyCheckpoint()` only clears the static `CHECKPOINT` reference.
In this branch, `AccessControllerManager` / `CatalogAccessController` do not
expose a close lifecycle, and the Ranger controller path does not call
`RangerBasePlugin.cleanup()`.
So one successful checkpoint can create two temporary Ranger Doris plugins,
and neither is explicitly cleaned up. That makes the reported count formula:
```text
1 main FE plugin + 2 * successful checkpoint count
```
consistent with the current code.
I also checked refreshed `upstream/branch-4.1` and `upstream/master`; both
still show `RangerDorisAccessControllerFactory` returning `new
RangerDorisAccessController("doris")`, and the checkpoint cleanup shape is
still the same. The older public singleton fixes referenced by the issue do not
appear to be ancestors of `4.1.0`, `branch-4.1`, or `master` in the refreshed
local repository state I inspected.
Suggested next steps:
1. Fix this on `master` and `branch-4.1`, then backport to affected 4.1.x
release lines.
2. Prefer preventing checkpoint `Env` construction from creating independent
long-lived Ranger Doris plugins. Restoring a process-wide singleton for the
internal Doris Ranger controller/plugin is the narrowest direction and matches
the previous fix intent.
3. If maintainers prefer a general lifecycle fix instead, add an explicit
close path from `Env.destroyCheckpoint()` down through
`AccessControllerManager` / access controllers and call Ranger plugin cleanup
for checkpoint-owned plugins. That path needs care because the serving FE
plugin must not be cleaned by temporary checkpoint `Env` teardown.
4. Add a regression test around checkpoint `Env` creation/destruction with
`access_controller_type=ranger-doris`. The test should assert that one
successful checkpoint does not leave two additional live Ranger plugin
refresher instances, or that checkpoint-owned Ranger plugins are explicitly
cleaned.
5. Audit similar access-controller factories for background-thread
ownership, especially any Ranger-backed controller that can be instantiated
while replaying or verifying checkpoint metadata.
Operational mitigation until a fix is available:
- Restarting the affected FE clears the accumulated leaked refresher threads
in that FE process.
- Increasing `cloud_checkpoint_image_stale_threshold_seconds` can reduce the
cloud stale-image checkpoint leak rate, but it only slows the leak and changes
image freshness behavior.
- Increasing the Ranger policy polling interval can reduce Ranger Admin
request pressure, but it does not fix the leaked FE threads.
Additional information that would be useful, but is not required to accept
the bug:
- A thread dump before and after two consecutive successful checkpoints from
the same FE process.
- FE logs covering several consecutive checkpoints, including both
checkpoint trigger lines and `Created PolicyRefresher Thread(...)` lines.
- The effective Ranger polling interval and Ranger Admin endpoint count, to
quantify operational impact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
