Hi all, I'd like to start a discussion on what changes are needed to make the authority (permission) module work correctly under high-availability scenarios — specifically when a DataNode is temporarily unreachable or network partitioned from the ConfigNode. Background When an authority operation (e.g., CREATE USER, GRANT, REVOKE) executes on the ConfigNode, the change is committed via Raft consensus. After that, the ConfigNode must broadcast a cache-invalidation request to every DataNode so that stale permission data is evicted from local caches. If a DataNode does not acknowledge this invalidation, it may continue authorizing a privilege that has already been revoked — which is a correctness issue. Currently, this broadcast is done via synchronous per-DataNode RPC calls inside AuthOperationProcedure. There are two concerns: 1. Blocking retry at the ConfigNode: If one DataNode is unreachable, the procedure holds the calling thread in a per-node retry loop with a timeout threshold. This is inconsistent with how schema-cache invalidation works — the ClusterCachePropagator + MetadataBroadcastVerdict infrastructure already exists for table/tree schema DDL and handles the PROCEED / WAIT / FAIL decision uniformly. 2. Independent staleness-detection at the DataNode: The DataNode-side ClusterAuthorityFetcher uses its own heartbeat-based mechanism (refreshToken() / checkCacheAvailable()) to decide when to discard cached permission data. This means the authority module operates outside the MetadataLeaseManager fence-detection framework. The schema modules (DataNodeTableCache, TreeDeviceSchemaCache) already use failIfMetadataLeaseFenced() with a fail-closed policy; the authority module should do the same. Proposed changes ConfigNode side: - Replace the synchronous per-DataNode RPC loop in AuthOperationProcedure with ClusterCachePropagator.propagate(), reusing the same verdict logic that schema DDL already uses. This makes timeout, retry, and self-fence detection consistent across all cache-invalidation broadcasts. A new CnToDnAsyncRequestType.INVALIDATE_PERMISSION_CACHE would be added to the async request manager. DataNode side: - Remove the independent heartbeat-based staleness detection from ClusterAuthorityFetcher (refreshToken(), checkCacheAvailable(), the cacheOutDate / heartBeatTimeStamp fields). Every authority-check entry point should call MetadataLeaseManager.failIfMetadataLeaseFenced() with RETRY_UNTIL_SUCCESS policy, so the authority path fails closed when the metadata lease is fenced — exactly as DataNodeTableCache does. - Add AuthorityChecker::invalidateAllCache to the MetadataLeaseManager.defaultClearCacheList(), so when the lease is fenced and subsequently recovered, the permission cache is dropped and will be lazily re-fetched from the ConfigNode on the next permission check. Open questions - The table schema cache (DataNodeTableCache) actively re-pulls all metadata from the ConfigNode after lease recovery via reloadTableCacheAfterLeaseRecovery(). Should the authority cache do the same, or is lazy re-fetch on the next permission check sufficient? Given that authority data is purely lazy-loaded at DataNode startup as well, maintaining the same behavior post-recovery seems reasonable, but I'd like to hear opinions. - The dataNodesToInvalid field in AuthOperationProcedure was used to track per-DataNode timeout state. If we switch to ClusterCachePropagator, this field becomes unused. Should it be removed immediately (with forward/backward compatibility handled via serialization versioning), or should it be kept for a release cycle? I'm interested in hearing whether this direction makes sense and whether there are scenarios or edge cases I've missed. Thanks, Yaobin Chen --- 大家好， 想发起一个讨论，关于权限模块在高可用场景下（特别是某个 DataNode 暂时不可达或与 ConfigNode 发生网络分区时）需要做哪些改造。 背景 当权限操作（如 CREATE USER、GRANT、REVOKE）在 ConfigNode 上执行时，变更通过 Raft 共识提交。之后 ConfigNode 需要广播缓存失效请求到每个 DataNode，使本地的过期权限数据被淘汰。如果某个 DataNode 没有确认这个失效请求，它可能继续授权已被撤销的权限——这是一个正确性问题。 目前，这个广播是在 AuthOperationProcedure 中通过同步的逐 DataNode RPC 调用来完成的。有两个问题： 1. ConfigNode 侧的阻塞重试：如果某个 DataNode 不可达，procedure 会在逐节点的重试循环中阻塞调用线程。这与 schema 缓存失效的工作方式不一致——ClusterCachePropagator + MetadataBroadcastVerdict 基础设施已经用于 table/tree schema DDL，统一处理 PROCEED / WAIT / FAIL 的判定。 2. DataNode 侧独立的过期检测：ClusterAuthorityFetcher 使用自己独立的心跳机制（refreshToken() / checkCacheAvailable()）来决定何时丢弃缓存的权限数据。这意味着权限模块运行在 MetadataLeaseManager 的隔离检测框架之外。Schema 模块（DataNodeTableCache、TreeDeviceSchemaCache）已经在使用 failIfMetadataLeaseFenced() 配合 fail-closed 策略；权限模块应该做同样的事。 建议的改造 ConfigNode 侧： - 将 AuthOperationProcedure 中的同步逐节点 RPC 循环替换为 ClusterCachePropagator.propagate()，复用与 schema DDL 相同的判定逻辑。这使得超时、重试、自隔离检测在所有的缓存失效广播中保持一致。需要新增 CnToDnAsyncRequestType.INVALIDATE_PERMISSION_CACHE 到异步请求管理器。 DataNode 侧： - 从 ClusterAuthorityFetcher 中移除独立的心跳过期检测（refreshToken()、checkCacheAvailable() 以及 cacheOutDate / heartBeatTimeStamp 字段）。所有权限检查入口统一调用 MetadataLeaseManager.failIfMetadataLeaseFenced() 并传入 RETRY_UNTIL_SUCCESS 策略，使得权限路径在元数据租约被隔离时 fail closed——与 DataNodeTableCache 行为一致。 - 将 AuthorityChecker::invalidateAllCache 添加到 MetadataLeaseManager.defaultClearCacheList() 中，这样当租约被隔离并随后恢复时，权限缓存被丢弃，在下次权限检查时从 ConfigNode 懒加载重新获取。 需要讨论的问题 - 表 schema 缓存（DataNodeTableCache）在租约恢复后会通过 reloadTableCacheAfterLeaseRecovery() 主动从 ConfigNode 全量拉取元数据。权限缓存是否也需要类似操作，还是恢复后惰性拉取就足够了？考虑到权限数据在 DataNode 启动时也是纯惰性拉取的，恢复后保持同样行为似乎是合理的，但想听听大家的意见。 - AuthOperationProcedure 中的 dataNodesToInvalid 字段用于追踪逐节点的超时状态。如果切换到 ClusterCachePropagator，这个字段就不再使用了。是直接移除（通过序列化版本管理处理前后兼容），还是保留一个发布周期的兼容？
