JackieTien97 commented on code in PR #19107:
URL: https://github.com/apache/iotdb/pull/19107#discussion_r4464810919
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceManager.java:
##########
@@ -179,12 +181,29 @@ public FragmentInstanceInfo execDataQueryFragmentInstance(
context.setHighestPriority(instance.isHighestPriority());
try {
- List<PipelineDriverFactory> driverFactories =
- planner.plan(
- instance.getFragment().getPlanNodeTree(),
- instance.getFragment().getTypeProvider(),
- context,
- dataNodeQueryContexts[0]);
+ List<PipelineDriverFactory> driverFactories;
+ if (instance.getSessionInfo() != null
+ && instance.getSessionInfo().getSqlDialect() ==
SqlDialect.TABLE) {
+ UDFManagementService.getInstance().acquireReadLock();
Review Comment:
This blocks on the UDF read lock **inside** the
`instanceExecution.computeIfAbsent(...)` mapping function, i.e. while holding
that CHM bin. Together with the read lock that `QueryExecution#analyze` now
holds for the entire table analysis, I think this can deadlock:
1. Thread A analyzes a table-model insert with new devices. It holds the
read lock (`QueryExecution` L344). `TableDeviceSchemaValidator` →
`TableDeviceSchemaFetcher#fetchMissingDeviceSchemaForDataInsertion` runs a
nested `FetchDevice` query with timeout `Long.MAX_VALUE`, whose local schema FI
goes through `execSchemaQueryFragmentInstance` →
`instanceExecution.computeIfAbsent(...)`.
2. A `CREATE FUNCTION` arrives, and `register()` → `acquireWriteLock()`
waits for A.
3. Thread T dispatches a table data FI, enters this `computeIfAbsent` lambda
(holding bin B), and blocks here. The non-fair `ReentrantReadWriteLock` blocks
new readers once a writer is queued.
4. If A's schema FI lands in bin B, A blocks on that bin.
That makes the cycle A → T → writer → A. None of `lock()`, `synchronized`,
or the nested fetch time out, so it never resolves. Every later table analysis
or FI planning on that DataNode (and tree UDF queries via `TransformOperator`)
then queues behind the writer, and the ConfigNode keeps holding its `udfInfo`
lock until its DataNode requests time out. The more table FIs arrive while the
writer is queued, the more bins are held, so a collision gets more likely under
load.
Could we take the read lock before entering `computeIfAbsent` instead of
blocking inside it? It would also help not to hold the read lock across the
whole analysis (see the comment on `QueryExecution` L344).
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/QueryExecution.java:
##########
@@ -164,6 +168,7 @@ public QueryExecution(IPlanner planner, MPPQueryContext
context, ExecutorService
// preventing a premature close raced with late-scheduled
FragmentInstances.
context.releaseExternalTsFileQueryResources();
this.stop(cause);
+ finalizeTableUDFClassLoader();
Review Comment:
For non-query statements this listener is the only place that releases the
analysis pin. `releaseResource()` is reached through
`Coordinator#cleanupQueryExecution`, but `Coordinator#execution` only puts
`isQuery()` executions into `queryExecutionMap`. So a non-query table statement
that never reaches a terminal state leaks its pin:
- `LOAD '/x.tsfile' WITH ('async'='true')` in the table model:
`visitLoadTsFile` sets `QueryType.OTHER`, and a successful `doAsyncLoad` sets
`finishQueryAfterAnalyze` without a fail status. `startInternal()` then calls
`constructResultForMemorySource()` and `transitionToRunning()` and returns, so
the state stays `RUNNING` forever.
- `UPDATE t SET a = 'v' WHERE <predicate that matches no device>`:
`QueryType.WRITE`, and `parseWhere` returns false so the analysis is marked
finished.
- Any non-query statement whose `start()` throws from
`doLogicalPlan`/`doDistributedPlan`/`schedule()`.
Each of these leaves an entry in `queryIdToUDFClassLoaderMap` and a
permanent `activeQueriesCount` on the loader. After the next `CREATE FUNCTION`,
the deprecated loader is never closed: jar handles and metaspace leak, and on
Windows the jar can't be deleted. Could the pin be released at the end of
`start()` for non-query statements, or could only `QueryType.READ` statements
be pinned?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/QueryExecution.java:
##########
@@ -332,7 +337,24 @@ private void constructResultForMemorySource() {
// Analyze the statement in QueryContext. Generate the analysis this query
need
private IAnalysis analyze(MPPQueryContext context) {
- return planner.analyze(context);
+ if (!(planner instanceof TableModelPlanner)) {
+ return planner.analyze(context);
+ }
+ String queryId = context.getQueryId().getId();
+ UDFManagementService.getInstance().acquireReadLock();
+ try {
+ synchronized (tableUdfClassLoaderLock) {
+ UDFClassLoaderManager.getInstance().initializeUDFQuery(queryId);
Review Comment:
This pins `activeClassLoader`, which is not necessarily the loader that
defined the table UDF classes stored in `UDFTable`.
`UDFManagementService#doRegister` rotates the global active loader on every
registration, but `updateAllRegisteredClasses(model, ...)` only reloads classes
of the model being registered. The previous loader is deprecated and, if
nothing pins it, closed right away in `markAsDeprecated()`.
Example:
1. `CREATE FUNCTION my_scalar` (table): the active loader becomes L1, and
table classes are loaded by L1.
2. `CREATE FUNCTION my_udtf` (tree): the active loader becomes L2, and only
tree classes are reloaded. L1 is deprecated and closed immediately.
3. `SELECT my_scalar(s1) FROM t` pins L2, here and in
`FragmentInstanceContext#initializeTableModelUDFQuery`, but `my_scalar`'s class
still comes from the closed L1. The first lazy load of a helper class,
resource, or `ServiceLoader` entry fails with
`NoClassDefFoundError`/`ClassNotFoundException`.
The same happens after a DataNode restart when both models have UDFs.
`prepareUDFResources()` calls `doRegister` once per UDF, rotating each time,
and `UDFClassLoaderManager.start()` then replaces the active loader again. So
the failure this PR targets can still happen in mixed tree/table deployments.
Reloading all models' classes on rotation, or pinning `clazz.getClassLoader()`
instead of the active loader, would close this gap.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceManager.java:
##########
@@ -179,12 +181,29 @@ public FragmentInstanceInfo execDataQueryFragmentInstance(
context.setHighestPriority(instance.isHighestPriority());
try {
- List<PipelineDriverFactory> driverFactories =
- planner.plan(
- instance.getFragment().getPlanNodeTree(),
- instance.getFragment().getTypeProvider(),
- context,
- dataNodeQueryContexts[0]);
+ List<PipelineDriverFactory> driverFactories;
+ if (instance.getSessionInfo() != null
+ && instance.getSessionInfo().getSqlDialect() ==
SqlDialect.TABLE) {
Review Comment:
Only the data-query path is covered. Table-model schema FIs also instantiate
scalar UDFs, but they go through `execSchemaQueryFragmentInstance` with neither
the lock nor a pin:
- `DataNodeTableOperatorGenerator#visitTableDeviceQueryCount` builds a
`DevicePredicateFilter` with `ColumnTransformerBuilder` for
`tagFuzzyPredicate`, e.g. `COUNT DEVICES FROM t WHERE my_udf(tag1) = 'x'`.
- The schema-region attribute update does the same:
`SchemaRegionMemoryImpl#updateTableDeviceAttribute` →
`constructDevicePredicateUpdater`, e.g. `UPDATE t SET attr = my_udf(attr)`.
On the DataNode that hosts the schema region, nothing pins the defining
loader there, and `reflect()` can still race with `deregister()`.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/QueryExecution.java:
##########
@@ -332,7 +337,24 @@ private void constructResultForMemorySource() {
// Analyze the statement in QueryContext. Generate the analysis this query
need
private IAnalysis analyze(MPPQueryContext context) {
- return planner.analyze(context);
+ if (!(planner instanceof TableModelPlanner)) {
+ return planner.analyze(context);
+ }
+ String queryId = context.getQueryId().getId();
+ UDFManagementService.getInstance().acquireReadLock();
Review Comment:
This holds the UDF read lock for the whole `planner.analyze(context)`, and
`TableModelPlanner` analysis covers every table-model statement, including
inserts, `LOAD`, `DELETE`, and `UPDATE`. That analysis can block for a long
time:
- the insert-time `FetchDevice` uses `Long.MAX_VALUE` as its timeout;
- schema auto-creation goes through the ConfigNode;
- `LOAD` analyzes whole TsFiles.
Once `register()`/`deregister()` is queued for the write lock, the non-fair
`ReentrantReadWriteLock` blocks every new reader.
So one `CREATE FUNCTION` issued while a slow `LOAD` or insert is in analysis
blocks all new table-model inserts and queries on that DataNode until that
analysis finishes. That covers this line and FI planning in
`FragmentInstanceManager`. Meanwhile `UDFManager#createFunction` on the
ConfigNode keeps holding the `udfInfo` lock while it waits, which blocks other
`CREATE/DROP FUNCTION` cluster-wide. If the ConfigNode-side request times out,
the DataNode may still finish the registration afterwards. Before this PR,
table writes were not affected by UDF DDL at all.
Could the lock be narrowed to resolving UDF classes and taking the pin? Or
could it be removed from the query path with an atomic `initializeUDFQuery`
(see the review summary)?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceManager.java:
##########
@@ -179,12 +181,29 @@ public FragmentInstanceInfo execDataQueryFragmentInstance(
context.setHighestPriority(instance.isHighestPriority());
try {
- List<PipelineDriverFactory> driverFactories =
- planner.plan(
- instance.getFragment().getPlanNodeTree(),
- instance.getFragment().getTypeProvider(),
- context,
- dataNodeQueryContexts[0]);
+ List<PipelineDriverFactory> driverFactories;
+ if (instance.getSessionInfo() != null
+ && instance.getSessionInfo().getSqlDialect() ==
SqlDialect.TABLE) {
+ UDFManagementService.getInstance().acquireReadLock();
+ try {
+ context.initializeTableModelUDFQuery();
Review Comment:
For remote FIs, user table-function code has already run before this pin is
taken. `DataNodeInternalRPCServiceImpl#sendFragmentInstance` →
`FragmentInstance.deserializeFrom` → `TableFunctionProcessorNode.deserialize`
calls `TableFunctionFactory.getTableFunction(name).createTableFunctionHandle()`
and `handle.deserialize(bytes)`. If a `CREATE FUNCTION` rotates the loader
between deserialization and this line, the handle class belongs to the old,
possibly already closed loader, while this FI pins the new active one. The
window is narrow, but it is the same kind of race this PR is trying to close.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/QueryExecution.java:
##########
@@ -466,6 +489,15 @@ private void releaseResource(Throwable t) {
}
}
+ private void finalizeTableUDFClassLoader() {
+ if (!(planner instanceof TableModelPlanner)) {
+ return;
+ }
+ synchronized (tableUdfClassLoaderLock) {
Review Comment:
`analyze()` holds `tableUdfClassLoaderLock` for the entire
`planner.analyze(context)`, and this method needs the same monitor. Suppose the
query is killed or fails while `retry()` is re-analyzing. The state-change
listener runs on the shared fixed-size `MPP-Coordinator-Executor` pool
(`coordinator_read_executor_size`: 20 by default, 2 in the Edge config). It
reaches `releaseResource()` → here and blocks for the rest of that analysis,
which can be unbounded for an insert waiting on `FetchDevice`. That delays
state notifications for other queries.
Also, as soon as the listener gets the monitor it drops the pin, while
`start()` keeps planning; CTE materialization and scalar subqueries can still
instantiate UDFs at that point. Could the monitor guard only the
initialize/finalize bookkeeping (e.g. a pinned flag) instead of the whole
analysis?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceManager.java:
##########
@@ -179,12 +181,29 @@ public FragmentInstanceInfo execDataQueryFragmentInstance(
context.setHighestPriority(instance.isHighestPriority());
try {
- List<PipelineDriverFactory> driverFactories =
- planner.plan(
- instance.getFragment().getPlanNodeTree(),
- instance.getFragment().getTypeProvider(),
- context,
- dataNodeQueryContexts[0]);
+ List<PipelineDriverFactory> driverFactories;
+ if (instance.getSessionInfo() != null
+ && instance.getSessionInfo().getSqlDialect() ==
SqlDialect.TABLE) {
+ UDFManagementService.getInstance().acquireReadLock();
+ try {
+ context.initializeTableModelUDFQuery();
+ driverFactories =
+ planner.plan(
+ instance.getFragment().getPlanNodeTree(),
+ instance.getFragment().getTypeProvider(),
+ context,
+ dataNodeQueryContexts[0]);
+ } finally {
+ UDFManagementService.getInstance().releaseReadLock();
+ }
+ } else {
+ driverFactories =
Review Comment:
Nit: the two branches duplicate the `planner.plan(...)` call, and the
dialect check repeats the one inside
`FragmentInstanceContext#initializeTableModelUDFQuery()`. Two options would
keep the paths from drifting apart:
- a boolean flag around a single `planner.plan` call;
- a small helper that does "read lock → pin → run", shared with
`QueryExecution#analyze`.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/QueryExecution.java:
##########
@@ -332,7 +337,24 @@ private void constructResultForMemorySource() {
// Analyze the statement in QueryContext. Generate the analysis this query
need
private IAnalysis analyze(MPPQueryContext context) {
- return planner.analyze(context);
+ if (!(planner instanceof TableModelPlanner)) {
Review Comment:
Minor/perf: this condition applies to every table-model statement. High-rate
`insertTablet`/`insertRecords`, `LOAD`, and `DELETE`, which never resolve UDFs,
now pay for:
- the global read lock and a `synchronized` block;
- a put/remove on `queryIdToUDFClassLoaderMap`;
- an increment/decrement on the single `AtomicLong` of the active loader.
All writer threads contend on those cache lines. Could pinning be limited to
statements that can reference functions (e.g. `QueryType.READ` and `UPDATE`),
or done lazily on the first UDF resolution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]