JackieTien97 commented on code in PR #19107:
URL: https://github.com/apache/iotdb/pull/19107#discussion_r4464810919


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceManager.java:
##########
@@ -179,12 +181,29 @@ public FragmentInstanceInfo execDataQueryFragmentInstance(
                 context.setHighestPriority(instance.isHighestPriority());
 
                 try {
-                  List<PipelineDriverFactory> driverFactories =
-                      planner.plan(
-                          instance.getFragment().getPlanNodeTree(),
-                          instance.getFragment().getTypeProvider(),
-                          context,
-                          dataNodeQueryContexts[0]);
+                  List<PipelineDriverFactory> driverFactories;
+                  if (instance.getSessionInfo() != null
+                      && instance.getSessionInfo().getSqlDialect() == 
SqlDialect.TABLE) {
+                    UDFManagementService.getInstance().acquireReadLock();

Review Comment:
   This blocks on the UDF read lock **inside** the 
`instanceExecution.computeIfAbsent(...)` mapping function, i.e. while holding 
that CHM bin. Together with the read lock that `QueryExecution#analyze` now 
holds for the entire table analysis, I think this can deadlock:
   
   1. Thread A analyzes a table-model insert with new devices. It holds the 
read lock (`QueryExecution` L344). `TableDeviceSchemaValidator` → 
`TableDeviceSchemaFetcher#fetchMissingDeviceSchemaForDataInsertion` runs a 
nested `FetchDevice` query with timeout `Long.MAX_VALUE`, whose local schema FI 
goes through `execSchemaQueryFragmentInstance` → 
`instanceExecution.computeIfAbsent(...)`.
   2. A `CREATE FUNCTION` arrives, and `register()` → `acquireWriteLock()` 
waits for A.
   3. Thread T dispatches a table data FI, enters this `computeIfAbsent` lambda 
(holding bin B), and blocks here. The non-fair `ReentrantReadWriteLock` blocks 
new readers once a writer is queued.
   4. If A's schema FI lands in bin B, A blocks on that bin.
   
   That makes the cycle A → T → writer → A. None of `lock()`, `synchronized`, 
or the nested fetch time out, so it never resolves. Every later table analysis 
or FI planning on that DataNode (and tree UDF queries via `TransformOperator`) 
then queues behind the writer, and the ConfigNode keeps holding its `udfInfo` 
lock until its DataNode requests time out. The more table FIs arrive while the 
writer is queued, the more bins are held, so a collision gets more likely under 
load.
   
   Could we take the read lock before entering `computeIfAbsent` instead of 
blocking inside it? It would also help not to hold the read lock across the 
whole analysis (see the comment on `QueryExecution` L344).



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/QueryExecution.java:
##########
@@ -164,6 +168,7 @@ public QueryExecution(IPlanner planner, MPPQueryContext 
context, ExecutorService
             // preventing a premature close raced with late-scheduled 
FragmentInstances.
             context.releaseExternalTsFileQueryResources();
             this.stop(cause);
+            finalizeTableUDFClassLoader();

Review Comment:
   For non-query statements this listener is the only place that releases the 
analysis pin. `releaseResource()` is reached through 
`Coordinator#cleanupQueryExecution`, but `Coordinator#execution` only puts 
`isQuery()` executions into `queryExecutionMap`. So a non-query table statement 
that never reaches a terminal state leaks its pin:
   
   - `LOAD '/x.tsfile' WITH ('async'='true')` in the table model: 
`visitLoadTsFile` sets `QueryType.OTHER`, and a successful `doAsyncLoad` sets 
`finishQueryAfterAnalyze` without a fail status. `startInternal()` then calls 
`constructResultForMemorySource()` and `transitionToRunning()` and returns, so 
the state stays `RUNNING` forever.
   - `UPDATE t SET a = 'v' WHERE <predicate that matches no device>`: 
`QueryType.WRITE`, and `parseWhere` returns false so the analysis is marked 
finished.
   - Any non-query statement whose `start()` throws from 
`doLogicalPlan`/`doDistributedPlan`/`schedule()`.
   
   Each of these leaves an entry in `queryIdToUDFClassLoaderMap` and a 
permanent `activeQueriesCount` on the loader. After the next `CREATE FUNCTION`, 
the deprecated loader is never closed: jar handles and metaspace leak, and on 
Windows the jar can't be deleted. Could the pin be released at the end of 
`start()` for non-query statements, or could only `QueryType.READ` statements 
be pinned?



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/QueryExecution.java:
##########
@@ -332,7 +337,24 @@ private void constructResultForMemorySource() {
 
   // Analyze the statement in QueryContext. Generate the analysis this query 
need
   private IAnalysis analyze(MPPQueryContext context) {
-    return planner.analyze(context);
+    if (!(planner instanceof TableModelPlanner)) {
+      return planner.analyze(context);
+    }
+    String queryId = context.getQueryId().getId();
+    UDFManagementService.getInstance().acquireReadLock();
+    try {
+      synchronized (tableUdfClassLoaderLock) {
+        UDFClassLoaderManager.getInstance().initializeUDFQuery(queryId);

Review Comment:
   This pins `activeClassLoader`, which is not necessarily the loader that 
defined the table UDF classes stored in `UDFTable`. 
`UDFManagementService#doRegister` rotates the global active loader on every 
registration, but `updateAllRegisteredClasses(model, ...)` only reloads classes 
of the model being registered. The previous loader is deprecated and, if 
nothing pins it, closed right away in `markAsDeprecated()`.
   
   Example:
   
   1. `CREATE FUNCTION my_scalar` (table): the active loader becomes L1, and 
table classes are loaded by L1.
   2. `CREATE FUNCTION my_udtf` (tree): the active loader becomes L2, and only 
tree classes are reloaded. L1 is deprecated and closed immediately.
   3. `SELECT my_scalar(s1) FROM t` pins L2, here and in 
`FragmentInstanceContext#initializeTableModelUDFQuery`, but `my_scalar`'s class 
still comes from the closed L1. The first lazy load of a helper class, 
resource, or `ServiceLoader` entry fails with 
`NoClassDefFoundError`/`ClassNotFoundException`.
   
   The same happens after a DataNode restart when both models have UDFs. 
`prepareUDFResources()` calls `doRegister` once per UDF, rotating each time, 
and `UDFClassLoaderManager.start()` then replaces the active loader again. So 
the failure this PR targets can still happen in mixed tree/table deployments. 
Reloading all models' classes on rotation, or pinning `clazz.getClassLoader()` 
instead of the active loader, would close this gap.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceManager.java:
##########
@@ -179,12 +181,29 @@ public FragmentInstanceInfo execDataQueryFragmentInstance(
                 context.setHighestPriority(instance.isHighestPriority());
 
                 try {
-                  List<PipelineDriverFactory> driverFactories =
-                      planner.plan(
-                          instance.getFragment().getPlanNodeTree(),
-                          instance.getFragment().getTypeProvider(),
-                          context,
-                          dataNodeQueryContexts[0]);
+                  List<PipelineDriverFactory> driverFactories;
+                  if (instance.getSessionInfo() != null
+                      && instance.getSessionInfo().getSqlDialect() == 
SqlDialect.TABLE) {

Review Comment:
   Only the data-query path is covered. Table-model schema FIs also instantiate 
scalar UDFs, but they go through `execSchemaQueryFragmentInstance` with neither 
the lock nor a pin:
   
   - `DataNodeTableOperatorGenerator#visitTableDeviceQueryCount` builds a 
`DevicePredicateFilter` with `ColumnTransformerBuilder` for 
`tagFuzzyPredicate`, e.g. `COUNT DEVICES FROM t WHERE my_udf(tag1) = 'x'`.
   - The schema-region attribute update does the same: 
`SchemaRegionMemoryImpl#updateTableDeviceAttribute` → 
`constructDevicePredicateUpdater`, e.g. `UPDATE t SET attr = my_udf(attr)`.
   
   On the DataNode that hosts the schema region, nothing pins the defining 
loader there, and `reflect()` can still race with `deregister()`.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/QueryExecution.java:
##########
@@ -332,7 +337,24 @@ private void constructResultForMemorySource() {
 
   // Analyze the statement in QueryContext. Generate the analysis this query 
need
   private IAnalysis analyze(MPPQueryContext context) {
-    return planner.analyze(context);
+    if (!(planner instanceof TableModelPlanner)) {
+      return planner.analyze(context);
+    }
+    String queryId = context.getQueryId().getId();
+    UDFManagementService.getInstance().acquireReadLock();

Review Comment:
   This holds the UDF read lock for the whole `planner.analyze(context)`, and 
`TableModelPlanner` analysis covers every table-model statement, including 
inserts, `LOAD`, `DELETE`, and `UPDATE`. That analysis can block for a long 
time:
   
   - the insert-time `FetchDevice` uses `Long.MAX_VALUE` as its timeout;
   - schema auto-creation goes through the ConfigNode;
   - `LOAD` analyzes whole TsFiles.
   
   Once `register()`/`deregister()` is queued for the write lock, the non-fair 
`ReentrantReadWriteLock` blocks every new reader.
   
   So one `CREATE FUNCTION` issued while a slow `LOAD` or insert is in analysis 
blocks all new table-model inserts and queries on that DataNode until that 
analysis finishes. That covers this line and FI planning in 
`FragmentInstanceManager`. Meanwhile `UDFManager#createFunction` on the 
ConfigNode keeps holding the `udfInfo` lock while it waits, which blocks other 
`CREATE/DROP FUNCTION` cluster-wide. If the ConfigNode-side request times out, 
the DataNode may still finish the registration afterwards. Before this PR, 
table writes were not affected by UDF DDL at all.
   
   Could the lock be narrowed to resolving UDF classes and taking the pin? Or 
could it be removed from the query path with an atomic `initializeUDFQuery` 
(see the review summary)?



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceManager.java:
##########
@@ -179,12 +181,29 @@ public FragmentInstanceInfo execDataQueryFragmentInstance(
                 context.setHighestPriority(instance.isHighestPriority());
 
                 try {
-                  List<PipelineDriverFactory> driverFactories =
-                      planner.plan(
-                          instance.getFragment().getPlanNodeTree(),
-                          instance.getFragment().getTypeProvider(),
-                          context,
-                          dataNodeQueryContexts[0]);
+                  List<PipelineDriverFactory> driverFactories;
+                  if (instance.getSessionInfo() != null
+                      && instance.getSessionInfo().getSqlDialect() == 
SqlDialect.TABLE) {
+                    UDFManagementService.getInstance().acquireReadLock();
+                    try {
+                      context.initializeTableModelUDFQuery();

Review Comment:
   For remote FIs, user table-function code has already run before this pin is 
taken. `DataNodeInternalRPCServiceImpl#sendFragmentInstance` → 
`FragmentInstance.deserializeFrom` → `TableFunctionProcessorNode.deserialize` 
calls `TableFunctionFactory.getTableFunction(name).createTableFunctionHandle()` 
and `handle.deserialize(bytes)`. If a `CREATE FUNCTION` rotates the loader 
between deserialization and this line, the handle class belongs to the old, 
possibly already closed loader, while this FI pins the new active one. The 
window is narrow, but it is the same kind of race this PR is trying to close.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/QueryExecution.java:
##########
@@ -466,6 +489,15 @@ private void releaseResource(Throwable t) {
     }
   }
 
+  private void finalizeTableUDFClassLoader() {
+    if (!(planner instanceof TableModelPlanner)) {
+      return;
+    }
+    synchronized (tableUdfClassLoaderLock) {

Review Comment:
   `analyze()` holds `tableUdfClassLoaderLock` for the entire 
`planner.analyze(context)`, and this method needs the same monitor. Suppose the 
query is killed or fails while `retry()` is re-analyzing. The state-change 
listener runs on the shared fixed-size `MPP-Coordinator-Executor` pool 
(`coordinator_read_executor_size`: 20 by default, 2 in the Edge config). It 
reaches `releaseResource()` → here and blocks for the rest of that analysis, 
which can be unbounded for an insert waiting on `FetchDevice`. That delays 
state notifications for other queries.
   
   Also, as soon as the listener gets the monitor it drops the pin, while 
`start()` keeps planning; CTE materialization and scalar subqueries can still 
instantiate UDFs at that point. Could the monitor guard only the 
initialize/finalize bookkeeping (e.g. a pinned flag) instead of the whole 
analysis?



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceManager.java:
##########
@@ -179,12 +181,29 @@ public FragmentInstanceInfo execDataQueryFragmentInstance(
                 context.setHighestPriority(instance.isHighestPriority());
 
                 try {
-                  List<PipelineDriverFactory> driverFactories =
-                      planner.plan(
-                          instance.getFragment().getPlanNodeTree(),
-                          instance.getFragment().getTypeProvider(),
-                          context,
-                          dataNodeQueryContexts[0]);
+                  List<PipelineDriverFactory> driverFactories;
+                  if (instance.getSessionInfo() != null
+                      && instance.getSessionInfo().getSqlDialect() == 
SqlDialect.TABLE) {
+                    UDFManagementService.getInstance().acquireReadLock();
+                    try {
+                      context.initializeTableModelUDFQuery();
+                      driverFactories =
+                          planner.plan(
+                              instance.getFragment().getPlanNodeTree(),
+                              instance.getFragment().getTypeProvider(),
+                              context,
+                              dataNodeQueryContexts[0]);
+                    } finally {
+                      UDFManagementService.getInstance().releaseReadLock();
+                    }
+                  } else {
+                    driverFactories =

Review Comment:
   Nit: the two branches duplicate the `planner.plan(...)` call, and the 
dialect check repeats the one inside 
`FragmentInstanceContext#initializeTableModelUDFQuery()`. Two options would 
keep the paths from drifting apart:
   
   - a boolean flag around a single `planner.plan` call;
   - a small helper that does "read lock → pin → run", shared with 
`QueryExecution#analyze`.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/QueryExecution.java:
##########
@@ -332,7 +337,24 @@ private void constructResultForMemorySource() {
 
   // Analyze the statement in QueryContext. Generate the analysis this query 
need
   private IAnalysis analyze(MPPQueryContext context) {
-    return planner.analyze(context);
+    if (!(planner instanceof TableModelPlanner)) {

Review Comment:
   Minor/perf: this condition applies to every table-model statement. High-rate 
`insertTablet`/`insertRecords`, `LOAD`, and `DELETE`, which never resolve UDFs, 
now pay for:
   
   - the global read lock and a `synchronized` block;
   - a put/remove on `queryIdToUDFClassLoaderMap`;
   - an increment/decrement on the single `AtomicLong` of the active loader.
   
   All writer threads contend on those cache lines. Could pinning be limited to 
statements that can reference functions (e.g. `QueryType.READ` and `UPDATE`), 
or done lazily on the first UDF resolution?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to