Hi all,

Background. 
In the table model, every DataNode caches table definitions 
(DataNodeTableCache) and the ConfigNode keeps them up to date by pushing update 
requests; a DataNode that cannot reach the ConfigNode relies on a metadata 
lease and re-syncs its whole table cache after the lease recovers. A rename 
runs as a multi-phase procedure: the new name is pre-released to every 
DataNode, then the rename is committed in the ConfigNode schema, and finally 
the old name is commit-released to the DataNodes. The target set of the 
pre-release is computed once, when that broadcast starts, and is reused for the 
whole retry loop, whereas the target set of the commit-release is computed when 
that broadcast starts.


Scenarios we considered: a DataNode is started or restarted while a rename is 
in its pre-release phase, so it registers after the pre-release target set was 
already computed but before the commit-release is sent; a rolling restart or a 
scale-out during the same window; a DataNode that was temporarily unreachable 
but not yet provably fenced when the pre-release went out.


Problem. 
A DataNode in the first scenario never applies the pre-release, so it holds no 
entry for the new name, while its activation snapshot still contains the table 
under the old name, because the ConfigNode commits the rename only after the 
pre-release phase. The commit-release request carries the two names but not the 
table definition, so it cannot fill that gap either. That DataNode therefore 
keeps serving the old name as a usable table and answers TABLE_NOT_EXISTS for 
the new name, while the ConfigNode reports the DDL as successful. The state 
does not converge on its own; it is corrected only by a metadata lease recovery 
(full re-sync) or by a DataNode restart.


Proposed fix, in two layers. On the DataNode side, when a commit-release 
arrives for a table that has no pre-released entry and the request carries the 
old name, the cache is repaired: the old name is dropped and the new name is 
left as a placeholder that makes the next lookup fetch the committed definition 
from the ConfigNode. On the ConfigNode side, the commit-release of a rename is 
sent through the same retry mechanism as the pre-release, so it is retried 
until every DataNode acknowledges or is provably self-fenced instead of being a 
single best-effort broadcast.


Impact and scope. Only rename procedures are affected; the other table DDL 
procedures keep their current single-shot commit-release, because they do not 
change a table name. A DataNode that still does not acknowledge after the 
retries is logged rather than treated as a failure, since the rename is already 
committed and cannot be rolled back, and it re-syncs on lease recovery. The 
repair is lazy by design: the new name reaches a DataNode on its next lookup, 
not at commit time. Read and write semantics on the DataNode are unchanged.


pr: https://github.com/apache/iotdb/pull/18800
-----------------


Hi,


背景
表模型下每个 DataNode 都缓存表定义(DataNodeTableCache),ConfigNode 通过推送更新请求保持它们一致;连不上 
ConfigNode 的 DataNode 依赖 metadata lease,在租约恢复后重新同步整份表缓存。改名是一个多阶段 
procedure:先把新名字 pre-release 给所有 DataNode,再在 ConfigNode 的元数据里提交改名,最后把老名字 
commit-release 给 DataNode。pre-release 的目标集合是广播开始时算一次、整个重试循环复用;commit-release 
的目标集合是那次广播开始时才算。


我们考虑过的触发场景:某台 DataNode 在改名的 pre-release 阶段启动或被重启,于是它在 pre-release 
目标集合算完之后、commit-release 发出之前完成注册;同一窗口内做滚动重启或扩容;pre-release 
发出时有数据节点暂时不可达、但还没被判定为已隔离。


问题
第一种场景下的 DataNode 从未应用 pre-release,因此没有新名字的任何条目;而它的激活快照里还留着老名字的表,因为 ConfigNode 
要到 pre-release 阶段之后才提交改名。commit-release 请求只带两个名字、不带表定义,所以也补不上这个缺口。结果是这台 
DataNode 仍把老名字当成可用表对外服务,访问新名字则返回 TABLE_NOT_EXISTS,而 ConfigNode 那边把这个 DDL 
记为成功。这个状态不会自行收敛,只能靠 metadata lease 恢复(整份重新同步)或重启 DataNode 来纠正。


修复方案
DataNode 侧:当 commit-release 到达、本地没有对应的 pre-release 
条目、且请求里带了老名字时,修复缓存——把老名字删掉,并给新名字留一个占位,使下一次查找去 ConfigNode 拉取已提交的定义。ConfigNode 
侧:改名的 commit-release 改用与 pre-release 相同的重试机制发送,重试到每个 DataNode 
都确认或可被证明已自隔离为止,而不是只尽力发一次。


影响范围。只有改名类 procedure 受影响;其它表 DDL 保持现有的单发 commit-release,因为它们不改表名。重试之后仍不确认的 
DataNode 只记日志、不算失败,因为改名已经提交且无法回滚,它会在租约恢复时重新同步。修复是惰性的:新名字在下次查找时才到这台 
DataNode,而不是在提交时。DataNode 上的读写语义不变。
pr: https://github.com/apache/iotdb/pull/18800


Best wishes
Yaobin Chen

Reply via email to