RoggeDeior opened a new issue, #65524: URL: https://github.com/apache/doris/issues/65524
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version ```text DORIS_BUILD_VERSION=doris-4.1.0-rc03 DORIS_BUILD_HASH=5960d4cea0e6d2a08f62b45a07a662395d2bbcfc Java=17.0.2 Deployment=Kubernetes Mode=Cloud / compute-storage decoupled Authorization=Apache Ranger Ranger version=2.7.0 access_controller_type=ranger-doris ``` The build hash is the same commit used by the Doris 4.1.0 release tag. ### What's Wrong? I searched existing issues and found: * https://github.com/apache/doris/issues/45641 * https://github.com/apache/doris/pull/45645 PR #45645 previously made `RangerDorisAccessController` a singleton. However, this fix is no longer effective in Doris 4.1.0 because the new `RangerDorisAccessControllerFactory` directly creates a new controller for every checkpoint Env. Every successful metadata checkpoint creates two new Ranger `PolicyRefresher` threads. These threads are not stopped when the temporary checkpoint Env is destroyed. The number of threads therefore increases continuously: ```text PolicyRefresher count = 1 normal refresher created by main + checkpoint count × 2 ``` In our test environment: ```text Journal-triggered checkpoints: 3 Cloud stale-image checkpoints: 135 Total checkpoints: 138 PolicyRefresher created by main: 1 PolicyRefresher created by leaderCheckpointer: 276 Total PolicyRefresher threads: 277 ``` The numbers match exactly: ```text 1 + 138 × 2 = 277 ``` Thread creator distribution from the FE logs: ```text Count Thread ----- ------------------ 276 leaderCheckpointer 1 main ``` The creation rate also matches the default cloud checkpoint interval. With: ```text cloud_checkpoint_image_stale_threshold_seconds = 3600 ``` one cloud checkpoint is triggered every hour, and two new PolicyRefresher threads are leaked per checkpoint, resulting in approximately 48 additional refresher threads per day. ### Evidence from FE Logs A checkpoint directly creates a new Ranger plugin and PolicyRefresher: ```text RuntimeLogger 2026-07-07 11:10:57,122 INFO (leaderCheckpointer|143) [Checkpoint.doCheckpoint():122] Trigger checkpoint since last checkpoint journal id: 5117761 is less than current finalized journal id: 5162245 RuntimeLogger 2026-07-07 11:10:57,140 INFO (leaderCheckpointer|143) [Checkpoint.doCheckpoint():155] begin to generate new image: image.5162245 RuntimeLogger 2026-07-07 11:10:57,151 INFO (leaderCheckpointer|143) [AccessControllerManager.loadAccessControllerPlugins():102] Found Authentication Plugin Factories: ranger-doris from class path. RuntimeLogger 2026-07-07 11:10:57,159 INFO (leaderCheckpointer|143) [RangerBasePlugin.init():308] Created PolicyRefresher Thread(PolicyRefresher(serviceName=doris)-377) ``` Every checkpoint leaves two additional PolicyRefresher threads running permanently. In our production environment, the FE accumulated: ```text 1321 PolicyRefresher threads ``` At the default 30-second Ranger polling interval, this generated approximately: ```text 1321 refreshers × 2 Ranger endpoints / 30 seconds ≈ 88 requests per second ``` The affected endpoints included: ```text /service/plugins/policies/download/doris /service/roles/download/doris ``` This eventually caused Ranger Admin to accumulate millions of asynchronous plugin-info update tasks: ```text RangerTransactionService: scheduled = 20,751,852 pending = 5,334,642 ``` Ranger Admin then experienced: ```text Old generation usage: approximately 96% Full GC count: 5197 Full GC time: approximately 18 hours java.lang.OutOfMemoryError: GC overhead limit exceeded ``` Tomcat Acceptor/Poller threads eventually disappeared, while port 6080 remained in LISTEN state with a full TCP accept backlog, making Ranger Admin appear running but inaccessible. ### What You Expected? Only one Ranger PolicyRefresher should exist per Doris FE process. Creating or destroying a temporary checkpoint Env must not leave any long-running Ranger background threads. ### How to Reproduce? 1. Deploy Doris 4.1.0 in Cloud Mode. 2. Configure Ranger authorization: ```properties access_controller_type = ranger-doris ``` 3. Keep the default configuration: ```properties cloud_checkpoint_image_stale_threshold_seconds = 3600 ``` 4. Start FE and wait for several cloud metadata checkpoints. 5. Count Ranger PolicyRefresher threads: ```bash jcmd <FE_PID> Thread.print \ | grep -c '^"PolicyRefresher(serviceName=doris)' ``` 6. Count checkpoint events: ```bash grep -R -h \ 'Trigger checkpoint since' \ /opt/apache-doris/fe/log | wc -l grep -R -h \ 'Trigger checkpoint in cloud mode because latest image is expired' \ /opt/apache-doris/fe/log | wc -l ``` 7. Count PolicyRefresher creation records: ```bash grep -R -h \ 'Created PolicyRefresher Thread(PolicyRefresher(serviceName=doris)' \ /opt/apache-doris/fe/log | wc -l ``` The PolicyRefresher count grows by two after every successful checkpoint. ### Root Cause Analysis The observed call chain is: ```text Checkpoint.doCheckpoint() -> Env.getCurrentEnv() -> EnvFactory.createEnv(true) -> new checkpoint Env -> new AccessControllerManager -> RangerDorisAccessControllerFactory.createAccessController() -> new RangerDorisAccessController("doris") -> RangerBasePlugin.init() -> new PolicyRefresher ``` `Checkpoint.doCheckpoint()` obtains and destroys a checkpoint Env twice during one successful checkpoint. The relevant `Env.getCurrentEnv()` bytecode is equivalent to: ```java if (isCheckpointThread()) { if (CHECKPOINT == null) { CHECKPOINT = EnvFactory.getInstance().createEnv(true); } return CHECKPOINT; } ``` However, `Env.destroyCheckpoint()` only clears the static reference: ```java public static void destroyCheckpoint() { if (CHECKPOINT != null) { CHECKPOINT = null; } } ``` It does not close `AccessControllerManager`, clean up the Ranger plugin, or stop PolicyRefresher. The Doris 4.1.0 factory directly creates a new controller: https://github.com/apache/doris/blob/4.1.0/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/RangerDorisAccessControllerFactory.java ```java @Override public RangerDorisAccessController createAccessController( Map<String, String> prop) { return new RangerDorisAccessController("doris"); } ``` `RangerDorisAccessController` in Doris 4.1.0 also no longer contains the singleton `getInstance()` implementation introduced by PR #45645. ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
