Hi everyone! This is a bug fix for data inconsistency that can occur when metadata lease fencing intersects with consensus-based writes. Background: Metadata Lease Mechanism: Each DataNode holds a "metadata lease" granted by the ConfigNode. The ConfigNode periodically pushes heartbeat messages to renew this lease. While the heartbeat is received within the configured metadata_lease_fence_ms threshold, the DataNode trusts its cached metadata (table schemas, tree schemas, partitions, etc.) and uses them for local write execution. If the heartbeat does not arrive before the threshold expires, the lease is "fenced" — the DataNode stops trusting its caches to prevent serving stale metadata and generating dirty data.
Trigger Scenarios Lease fencing can occur under several realistic conditions: 1. Network partition between DataNode and ConfigNode: The DataNode remains connected to other DataNodes (so consensus logs continue to arrive), but cannot reach the ConfigNode to renew its lease. 2. ConfigNode leader election or restart: A brief interruption in heartbeat delivery can cause lease expiry even though the DataNode is otherwise healthy. 3. GC pause or resource contention on the heartbeat processing thread: The scheduled lease check may detect an expired lease even though the ConfigNode is still sending heartbeats. When any of these occurs while a consensus write is being applied, the request can fail with a MetadataLeaseFencedException. Before this fix, the write path did not handle MetadataLeaseFencedException properly across several layers: - At the storage engine layer, a batch write (e.g. InsertTabletNode) is split into multiple time-partition fragments. Some fragments could succeed (written into MemTable and WAL) before the lease expired, while subsequent fragments would fail. The successful fragments were not rolled back, leaving the replica in a partially-written state. - The consensus state machine execution layer caught specific checked exceptions but did not catch MetadataLeaseFencedException. The exception escaped as a generic runtime error without a proper status code. - The state machine's retry logic did not include lease fencing, so even transient lease expiry could not be recovered by retrying. - The retry mechanism in the session and coordinator layers had no awareness of this error code, so clients could not retry on a different (non-fenced) node. Fix: The fix addresses the issue at each layer: - MetadataLeaseFencedException is now parameterized with a LeaseFencedRetryPolicy, producing two distinct status codes: METADATA_LEASE_FENCED for non-retryable paths (e.g., query-time cache access) and METADATA_LEASE_FENCED_RETRY_REQUIRED for the write path where retry is appropriate. - The consensus state machine (DataRegionStateMachine) now retries writes that fail with this code, up to a configurable maximum. - The Ratis-based state machine proxy (ApplicationStateMachineProxy) now has proper infinite-retry semantics for recoverable errors, instead of a broken do-while loop that never actually retried. - The session and coordinator retry mechanisms recognize the new status code, enabling clients to transparently retry on a different healthy DataNode. All changes are on the write path only. Non-write operations (queries, cache lookups) remain fail-closed: they reject the operation immediately with METADATA_LEASE_FENCED without retry, which is the safe default. Feedback and review welcome. --- 中文 主题：[讨论] 修复元数据租约过期导致共识写入部分成功与数据不一致 本修复针对元数据租约（metadata lease）与共识写入的交叉场景下可能出现的数据不一致问题。 背景：元数据租约机制 每个 DataNode 持有一个由 ConfigNode 授予的元数据租约。ConfigNode 通过心跳消息定期续约。在 metadata_lease_fence_ms 配置的阈值时间内收到心跳时，DataNode 信任其本地缓存的元数据（表结构、树形结构、分区信息等）并用于本地写入执行。若心 未在阈值内到达，租约即被"隔离"（fenced）—— DataNode 停止信任缓存，防止基于过期元数据产生脏数据。 触发场景 租约过期可能在以下实际情况中发生： 1. DataNode 与 ConfigNode 之间网络分区：DataNode 仍连接其他 DataNode（共识日志正常到达），但无法连接 ConfigNode 续约。 2. ConfigNode 主节点选举或重启：心跳发送短暂中断，即使 DataNode 本身健康也会导致租约过期。 3. GC 停顿或心跳处理线程资源争用：定时租约检查发现过期，但 ConfigNode 实际仍在正常发送心跳。 以上任一场景与共识写入并发时，请求即会因 MetadataLeaseFencedException 失败。 问题描述 修复前，写入路径在多个层面未正确处理此异常： - 存储引擎层：批量写入（如 InsertTabletNode）按时间分区拆分为多个 fragment。租约过期前，部分 fragment 可能已成功写入 MemTable 和 WAL，后续 fragment 却因租约检查失败而中断。已成功的 fragment 不会被回滚，导致副本处于部分写入状态。 - 共识状态机执行层：相关访问者类仅捕获特定的受检异常，未捕获MetadataLeaseFencedException。异常以通用运行时错误形式逃逸，未返回明确的状态码。 - 状态机的重试逻辑不包含租约过期错误码，即使短暂的租约失效也不可恢复。 - Session 与 Coordinator 层的重试机制无法识别此错误码，客户端无法换到其他未隔离的节点重试。 修复方案 修复在以下各层展开： - MetadataLeaseFencedException 增加重试策略参数，映射为两个独立状态码：METADATA_LEASE_FENCED（不重试路径，如查询时缓存访问）和METADATA_LEASE_FENCED_RETRY_REQUIRED（写入路径，适合重试）。 - 共识状态机（DataRegionStateMachine）对写入失败且状态码为可重试时，进行有限次重试。 - Ratis 共识层的状态机代理（ApplicationStateMachineProxy）修复了原有失效的 do-while循环，对可恢复异常实现真正的重试语义。 - Session 和 Coordinator 重试机制识别新状态码，客户端可透明切换到其他健康 DataNode 重试。 Best wishes Yaobin Chen
