Hi all, I'd like to share a bug I recently found and the fix I am working on in the following PR: - master: https://github.com/apache/iotdb/pull/19107 When a UDF is registered or replaced while queries are still running, the UDF class loader that a running query depends on can be deprecated and closed before the query finishes. Table-model UDFs that lazily load helper classes during analysis or execution may then hit classloading failures such as ClassNotFoundException or NoClassDefFoundError. Fragments deserialized on remote DataNodes through RPC or consensus can suffer from the same problem, because they previously had no query-scope pin at all. The root cause is that the pin was placed on the wrong class loader: - A query pinned the currently active class loader at scope entry, not the exact loader that defined the reflected UDF class. - Registering a UDF creates a new loader generation, reloads all registered classes into it and deprecates the old loader, so a running query could pin an unrelated loader and still miss the class it actually uses. - Pinning only covered transformer initialization; lazy class loading during analysis/execution and fragment deserialization/planning were not covered. - UDF type classification for SHOW FUNCTIONS / information_schema.functions relied on reflection that required a query scope, so external table scalar UDFs could be reported as UNKNOWN. The fix I am working on pins functionClass.getClassLoader() returned by UDFManagementService.reflect() and keeps it pinned for the lifetime of the owning query or fragment scope: - Add QueryScope handles with active-action tracking and deferred close, and bind them across analysis, planning and scheduling. - Pin inside reflect() under the UDF registry read lock, and reject any attempt to bind a finalized scope. - Bind fragment scopes around LocalExecutionPlanner.plan() and transfer ownership into FragmentInstanceContext; apply the same handling to RPC and consensus deserialization. - Remove TransformOperator-local class loader pin management. - Classify UDF types from the registered Class hierarchy / UDFInformation without instantiating the function. Main changes include: - UDFClassLoaderManager: QueryScope, withQueryScope, pinClassLoaderForCurrentQuery, atomic catalog publication. - UDFManagementService: read/write lock, pinning in reflect(), non-instantiating type check. - UDFTable: immutable snapshots for the class catalog. - FragmentInstance / FragmentInstanceContext / LocalExecutionPlanner / QueryExecution and the RPC & consensus paths: scope propagation and cleanup. - Regression tests for scope finalization, retries, lazy loading and fragment scope transfer. This work is still in progress. Local validation so far is limited to static inspection and git diff --check; Maven tests have not been run yet. Feedback and suggestions are welcome.
大家好， 我想分享一个最近发现的问题，以及我正在以下 PR 中尝试的修复： - master：https://github.com/apache/iotdb/pull/19107 当 UDF 在查询仍在执行时被注册或替换，运行中查询所依赖的 UDF classloader 可能在查询结束前就被标记废弃并关闭。对于会在分析或执行阶段延迟加载辅助类的表模型 UDF，此时可能出现 ClassNotFoundException、NoClassDefFoundError 等类加载失败。通过 RPC 或共识在远端 DataNode 上反序列化的 Fragment 也可能遇到同样的问题，因为它们之前完全没有 query scope 的 pin。 根本原因是 pin 打在了错误的 classloader 上： - 查询在进入作用域时 pin 的是“当前活跃”的 classloader，而不是真正定义被反射 UDF 类的那个 loader。 - 注册 UDF 会创建新一代 classloader、把所有已注册类重新加载到其中并废弃旧 loader，因此运行中的查询可能 pin 了无关的 loader，却仍然漏掉了它实际使用的类。 - pin 只覆盖了 transformer 初始化阶段，分析/执行期间的延迟类加载以及 Fragment 反序列化和规划都没有覆盖。 - SHOW FUNCTIONS / information_schema.functions 的 UDF 类型判断依赖需要 query scope 的反射，导致外部表标量函数可能被识别为 UNKNOWN。 我目前正在做的修复是：pin 住 UDFManagementService.reflect() 返回的 functionClass.getClassLoader()，并让它在所属查询或 Fragment 作用域的生命周期内保持有效： - 新增带 active-action 计数和延迟关闭的 QueryScope 句柄，并在分析、规划、调度各阶段绑定。 - 在 UDF 注册表读锁内于 reflect() 中完成 pin，并拒绝绑定已 finalize 的 scope。 - 在 LocalExecutionPlanner.plan() 外围绑定 Fragment scope，把所有权移交给 FragmentInstanceContext；RPC 和共识反序列化路径做同样处理。 - 移除 TransformOperator 内部临时的 classloader pin 管理。 - 通过已注册的 Class 层级 / UDFInformation 判断 UDF 类型，不再实例化 UDF。 主要修改包括： - UDFClassLoaderManager：QueryScope、withQueryScope、pinClassLoaderForCurrentQuery、类目录的原子发布。 - UDFManagementService：读写锁、reflect() 中的 pin、不实例化的类型检查。 - UDFTable：类目录改为不可变快照。 - FragmentInstance / FragmentInstanceContext / LocalExecutionPlanner / QueryExecution 以及 RPC 与共识路径：作用域传递与清理。 - 针对 scope finalize、重试、懒加载和 Fragment scope 转移的回归测试。 目前这个修复仍在进行中，本地只做了静态检查和 git diff --check，还没有运行 Maven 测试。欢迎大家提出反馈和建议。 Best regards, Wenwei Shu
