Hi all,
Background. In the table model, every DataNode caches table definitions (DataNodeTableCache) and the ConfigNode keeps them up to date by pushing update requests; a DataNode that cannot reach the ConfigNode relies on a metadata lease and re-syncs its whole table cache after the lease recovers. A rename runs as a multi-phase procedure: the new name is pre-released to every DataNode, then the rename is committed in the ConfigNode schema, and finally the old name is commit-released to the DataNodes. The target set of the pre-release is computed once, when that broadcast starts, and is reused for the whole retry loop, whereas the target set of the commit-release is computed when that broadcast starts. Scenarios we considered: a DataNode is started or restarted while a rename is in its pre-release phase, so it registers after the pre-release target set was already computed but before the commit-release is sent; a rolling restart or a scale-out during the same window; a DataNode that was temporarily unreachable but not yet provably fenced when the pre-release went out. Problem. A DataNode in the first scenario never applies the pre-release, so it holds no entry for the new name, while its activation snapshot still contains the table under the old name, because the ConfigNode commits the rename only after the pre-release phase. The commit-release request carries the two names but not the table definition, so it cannot fill that gap either. That DataNode therefore keeps serving the old name as a usable table and answers TABLE_NOT_EXISTS for the new name, while the ConfigNode reports the DDL as successful. The state does not converge on its own; it is corrected only by a metadata lease recovery (full re-sync) or by a DataNode restart. Proposed fix, in two layers. On the DataNode side, when a commit-release arrives for a table that has no pre-released entry and the request carries the old name, the cache is repaired: the old name is dropped and the new name is left as a placeholder that makes the next lookup fetch the committed definition from the ConfigNode. On the ConfigNode side, the commit-release of a rename is sent through the same retry mechanism as the pre-release, so it is retried until every DataNode acknowledges or is provably self-fenced instead of being a single best-effort broadcast. Impact and scope. Only rename procedures are affected; the other table DDL procedures keep their current single-shot commit-release, because they do not change a table name. A DataNode that still does not acknowledge after the retries is logged rather than treated as a failure, since the rename is already committed and cannot be rolled back, and it re-syncs on lease recovery. The repair is lazy by design: the new name reaches a DataNode on its next lookup, not at commit time. Read and write semantics on the DataNode are unchanged. pr: https://github.com/apache/iotdb/pull/18800 ----------------- Hi, 背景 表模型下每个 DataNode 都缓存表定义(DataNodeTableCache),ConfigNode 通过推送更新请求保持它们一致;连不上 ConfigNode 的 DataNode 依赖 metadata lease,在租约恢复后重新同步整份表缓存。改名是一个多阶段 procedure:先把新名字 pre-release 给所有 DataNode,再在 ConfigNode 的元数据里提交改名,最后把老名字 commit-release 给 DataNode。pre-release 的目标集合是广播开始时算一次、整个重试循环复用;commit-release 的目标集合是那次广播开始时才算。 我们考虑过的触发场景:某台 DataNode 在改名的 pre-release 阶段启动或被重启,于是它在 pre-release 目标集合算完之后、commit-release 发出之前完成注册;同一窗口内做滚动重启或扩容;pre-release 发出时有数据节点暂时不可达、但还没被判定为已隔离。 问题 第一种场景下的 DataNode 从未应用 pre-release,因此没有新名字的任何条目;而它的激活快照里还留着老名字的表,因为 ConfigNode 要到 pre-release 阶段之后才提交改名。commit-release 请求只带两个名字、不带表定义,所以也补不上这个缺口。结果是这台 DataNode 仍把老名字当成可用表对外服务,访问新名字则返回 TABLE_NOT_EXISTS,而 ConfigNode 那边把这个 DDL 记为成功。这个状态不会自行收敛,只能靠 metadata lease 恢复(整份重新同步)或重启 DataNode 来纠正。 修复方案 DataNode 侧:当 commit-release 到达、本地没有对应的 pre-release 条目、且请求里带了老名字时,修复缓存——把老名字删掉,并给新名字留一个占位,使下一次查找去 ConfigNode 拉取已提交的定义。ConfigNode 侧:改名的 commit-release 改用与 pre-release 相同的重试机制发送,重试到每个 DataNode 都确认或可被证明已自隔离为止,而不是只尽力发一次。 影响范围。只有改名类 procedure 受影响;其它表 DDL 保持现有的单发 commit-release,因为它们不改表名。重试之后仍不确认的 DataNode 只记日志、不算失败,因为改名已经提交且无法回滚,它会在租约恢复时重新同步。修复是惰性的:新名字在下次查找时才到这台 DataNode,而不是在提交时。DataNode 上的读写语义不变。 pr: https://github.com/apache/iotdb/pull/18800 Best wishes Yaobin Chen
