singer-bin opened a new pull request, #8626: URL: https://github.com/apache/hadoop/pull/8626
### Description of PR JIRA: [HDFS-17953](https://issues.apache.org/jira/browse/HDFS-17953) #### Problem When a DataNode is configured with HA NameNodes (`dfs.ha.namenodes.<nsId>`), but one of the configured NameNode IDs has no (or a wrong) RPC address — or the NameNode is permanently unreachable — the `IncrementalBlockReportManager.pendingIBRs` map for that actor grows without bound and eventually causes the DataNode to OOM. In `BPOfferService#notifyNamenodeBlock`, every block receive/delete event is added to the IBR queue of *every* `BPServiceActor`, including the actor for the unreachable NameNode. In `IncrementalBlockReportManager#sendIBRs`, on send failure `putMissing()` puts all blocks back into the queue. When the NameNode is permanently unreachable the send always fails, new events keep arriving, and the queue grows indefinitely — there is no size limit, no TTL and no eviction. The only existing protection (HDFS-9917) calls `clearIBRs()` during `reRegister()` for STANDBY/OBSERVER NNs, but `reRegister()` is never triggered when the NN is completely unreachable (no `DNA_REGISTER` command can be received). A production heap dump showed ~223M `ReceivedDeletedBlockInfo` instances occupying ~30 GB, caused by a stale `nn3` whose RPC address was never configured (it fell back to the logical `nameservice:8020` which never resolves). #### Fix 1. Add a configurable cap `dfs.datanode.ibr.max.pending.size` (default 1,000,000). When exceeded, the pending IBR queue is cleared with a warning. 2. Add a staleness guard `dfs.datanode.ibr.max.stale.interval.ms` (default 30 min): if no successful IBR send happens within this window while entries are pending, the queue is cleared. 3. After clearing (overflow or staleness), a Full Block Report is scheduled so the NameNode gets a complete, consistent view once reachable again. This is safe: the FBR is the ultimate consistency guarantee; IBRs are only an optimization for incremental updates between FBRs. Dropping queued IBRs for an unreachable NN cannot cause inconsistency because a fresh FBR is sent on reconnect/re-register. Both protections can be disabled by setting the corresponding value to `0`. #### How was this patch tested? New unit tests in `TestIncrementalBlockReportManager`: - `testIBRQueueSizeLimit` — queue is capped and cleared on overflow. - `testIBRQueueStaleInterval` — queue is cleared after the stale interval. - `testIBRQueueNoLimit` — cap disabled (0) keeps old behavior. - `testIBRDeduplication` — per-block dedup still works. - `testTotalPendingIBRSizeMultipleStorages` — size counting across storages. #### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [x] Object storage: N/A - [x] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? N/A -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
