924060929 commented on code in PR #69255:
URL: https://github.com/apache/doris/pull/69255#discussion_r4638337548
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java:
##########
@@ -935,6 +937,24 @@ public synchronized List<Function> getFunctions() {
return FunctionUtil.getFunctions(name2Function);
}
+ synchronized void registerFunctionsById(ConcurrentMap<Long, Function>
functionsById) {
+ id2Function = functionsById;
+ for (Function function : getFunctions()) {
+ id2Function.put(function.getId(), function);
+ }
+ }
+
+ public synchronized void unregisterFunctionsById() {
Review Comment:
Two different objects are mutated here, and the name hides it.
`id2Function` on this class is only a handle to `Env.id2Function`
(`Env.java:401`, `final`, process lifetime); the map has several holders - Env
itself, every `Database`, and `GlobalFunctionMgr`. So:
- `id2Function.remove(...)` mutates **Env's shared map**. This is the only
thing that makes `Env.getFunctionById()` stop returning these ids, i.e. what
drives the report-based `CLEAN_UDF_CACHE` reconciliation
(`ReportHandler.java:415`).
- `id2Function = null` mutates **only this object's field**. It stops
`FunctionUtil.addFunctionImpl()` from registering new functions
(`FunctionUtil.java:174`); it removes nothing from the shared map.
Both are needed - I traced the FORCE DROP path, where this runs at drop time
and again from `CatalogRecycleBin.eraseDatabase()`, so the early return is
required - but the order is forced (the field is the only handle to the map)
and bundling them makes the method read as "just null the field". A reader who
drops the loop silently breaks the compensation path: if the drop-time cleanup
never reached a backend (BE down, task expired, FE restarted after the
recycle-bin entry was erased), FE would still "know" the id and never ask the
BE to release that function's process pool and package files.
On the name: `...ById` takes no id argument, and "unregister this database's
functions" also reads as "delete them", which would be wrong - the `Function`
objects must survive for `RECOVER DATABASE`.
Suggestions, in increasing order of change:
1. Keep one method, but name it for what it does and document both effects,
e.g. `detachFunctionIndex()` (same for `GlobalFunctionMgr`,
`detachFunctionIndexes` for the bulk `CatalogRecycleBin` variant), with
Javadoc: "removes this database's functions from Env's shared id index and
stops registering new ones; function objects are kept for RECOVER DATABASE;
safe to call twice".
2. Split the two responsibilities: `Env.detachDbFunctions(Database db)`
(mutates the shared index) + `Database.markFunctionsDetached()` (mutates the
flag). That removes the ordering constraint and the "nullable field as detached
sentinel" pattern relied on by
`FunctionUtil.addFunctionImpl()`/`dropFunctionImpl()`.
3. Or key the index by database (`Map<dbId, Map<funcId, Function>>`):
unbinding becomes a single `remove(dbId)`.
Also `remove(function.getId(), function)` uses the conditional form; ids
come from `Env.getNextId()` and are never reused, so `remove(id)` is equivalent
- if the CAS semantics are intentional, a comment would help.
##########
fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java:
##########
@@ -389,6 +402,23 @@ protected void exec() {
}
}
+ static void pythonUdfReport(long backendId, Set<Long> functionIds) {
+ if (functionIds.isEmpty()) {
+ return;
+ }
+ // A queued report can outlive removal of its backend.
+ if (Env.getCurrentSystemInfo().getBackend(backendId) == null) {
+ return;
+ }
+ List<CleanUDFCacheTask> cleanups = new ArrayList<>();
+ for (long functionId : functionIds) {
+ if (Env.getCurrentEnv().getFunctionById(functionId) == null) {
Review Comment:
This line makes `Env.id2Function` the authority for a **destructive**
action: a miss here means the BE drops both the function's process pool
(`PythonServerManager::drop_function`) and its package cache
(`UserFunctionCache::drop_function_cache`; files are deleted when the last
reference is released). The only recovery is re-downloading from the function's
`location` on the next query, so if that artifact is gone (`file://` deleted,
URL expired) the function is permanently broken.
I could not find a way to get a stale miss on a live function today - ids
are added to the index before the name is published
(`FunctionUtil.addFunctionImpl`), and I checked every database
insertion/removal path plus the image/journal replay paths. The risk is
structural rather than an existing bug: the index is a second, hand-maintained
view of the catalog, keyed through a nullable field that doubles as the
"detached" flag, and this PR already had to fix one path for it
(`InternalCatalog.loadDb` now goes through `unprotectCreateDb`). Any future
path that inserts a database or a function without going through those two
chokepoints turns this line from "leak" into "delete a live function's files".
Given how cheap it is, a confirmation window would bound the blast radius:
keep `(backendId, functionId) -> firstSeen` and only enqueue the cleanup once
the id has been missing for e.g. 10-30 minutes (much longer than a journal
replay lag, far shorter than the 6h report cadence); or make the BE stage the
deletion (mark, then delete only if not re-created within N minutes).
Alternatively make the invariant explicit: `addFunctionImpl` should fail loudly
when the index is not bound and the database is still in the catalog, instead
of silently skipping the registration.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java:
##########
@@ -537,6 +539,11 @@ public void dropDb(String dbName, boolean ifExists,
boolean force) throws DdlExc
Env.getCurrentRecycleBin().recycleDatabase(db, tableNames,
tableIds, false, force, 0);
recycleTime =
Env.getCurrentRecycleBin().getRecycleTimeById(db.getId());
+ // Recoverable databases keep function IDs until permanent
erasure.
+ if (force) {
+ functionSignatures =
FunctionUtil.collectPythonFunctions(db);
Review Comment:
This is the FORCE DROP branch, and the same functions are cleaned up twice:
here (after the catalog lock is released - good), and again when the database
is finally erased from the recycle bin.
`CatalogRecycleBin.eraseDatabase(Database)` (`CatalogRecycleBin.java:458-461`)
calls `unregisterFunctionsById()` + `cleanUdfCache(collectPythonFunctions(db))`
on the same object.
For a force drop the entry is written with `recycleTime = 0`, so it is
erased by the next recycle-bin scan, i.e. seconds later. While the first batch
is still queued, `AgentTaskQueue.addTask()` de-dupes on `signature ==
functionId` and nothing extra is sent; once it has been acknowledged, a second
full round of `CLEAN_UDF_CACHE` RPCs goes out. It is idempotent on the BE, so
this is wasted work rather than a correctness issue - but
`DropDbTest.testForceDropDbSubmitsFunctionCacheCleanup` only asserts the
drop-time submission, so the duplication is not covered either. Consider
letting the recycle-bin erasure own the cleanup in all cases, or tracking which
ids were already submitted.
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -7792,6 +7804,14 @@ public GlobalFunctionMgr getGlobalFunctionMgr() {
return globalFunctionMgr;
}
+ public Function getFunctionById(long functionId) {
+ return id2Function.get(functionId);
+ }
+
+ public void registerDbFunctionsById(Database db) {
Review Comment:
Binding goes through `Env` (`registerDbFunctionsById`, plus the constructor
/ `loadCatalog` / `loadRecycleBin` / `loadGlobalFunction` hooks), but unbinding
has no counterpart here: `Database.unregisterFunctionsById()` is called
directly from `InternalCatalog.dropDb()`, `InternalCatalog.replayDropDb()`,
`CatalogRecycleBin.eraseDatabase(Database)` and
`CatalogRecycleBin.replayEraseDatabase()`. Adding
`unregisterDbFunctionsById(Database db)` and routing those four call sites
through it would put both directions of this index's lifecycle in one class,
and makes the "only Env mutates the shared map" rule visible in one place.
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/GlobalFunctionMgr.java:
##########
@@ -45,6 +45,21 @@ public class GlobalFunctionMgr extends MetaObject implements
GsonPostProcessable
// user define function
@SerializedName(value = "name2Function")
private ConcurrentMap<String, ImmutableList<Function>> name2Function =
Maps.newConcurrentMap();
+ private transient ConcurrentMap<Long, Function> id2Function;
+
+ synchronized void registerFunctionsById(ConcurrentMap<Long, Function>
functionsById) {
+ id2Function = functionsById;
+ for (Function function : getFunctions()) {
+ id2Function.put(function.getId(), function);
+ }
+ }
+
+ synchronized void unregisterFunctionsById() {
+ for (Function function : getFunctions()) {
+ id2Function.remove(function.getId(), function);
Review Comment:
Asymmetric with `Database.unregisterFunctionsById()` (`Database.java:949`),
which returns early when `id2Function == null`; the null check is missing here.
Not reachable today (the only instance is created and bound in the `Env`
constructor, and `Env.loadGlobalFunction()` re-binds after replacing), but
`Env.loadGlobalFunction()` does call this on the instance being replaced, so
any `GlobalFunctionMgr` constructed outside `Env` (tests, future code) would
NPE. Either guard it the same way, or drop the guard on both sides and document
the "always bound" invariant.
##########
fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java:
##########
@@ -199,6 +199,9 @@ public TMasterResult finishTaskInternal(TFinishTaskRequest
request) {
case DROP:
finishDropReplica(task);
break;
+ case CLEAN_UDF_CACHE:
Review Comment:
For a best-effort task like this, a finish report can legitimately find no
queued task: the entry expires after `CleanUDFCacheTask.RETENTION_MS` (5 min)
or is dropped when the backend is removed, while the BE may still send
`finish_task` later. In that case the code above returns `CANCELLED` and logs
`cannot find task. type: CLEAN_UDF_CACHE ...`. `DROP`, `RELEASE_SNAPSHOT` and
`CLEAR_TRANSACTION_TASK` are already tolerated there; `CLEAN_UDF_CACHE` looks
like it belongs in the same set, to avoid spurious warnings (the BE does not
retry on a non-OK result, it only logs).
##########
fe/fe-core/src/main/java/org/apache/doris/task/CleanUDFCacheTask.java:
##########
@@ -23,18 +23,30 @@
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+import java.util.concurrent.TimeUnit;
public class CleanUDFCacheTask extends AgentTask {
+ private static final long RETENTION_MS = TimeUnit.MINUTES.toMillis(5);
private static final Logger LOG =
LogManager.getLogger(CleanUDFCacheTask.class);
private String functionSignature;
private long functionId;
public CleanUDFCacheTask(long backendId, String signature, long
functionId) {
- super(null, backendId, TTaskType.CLEAN_UDF_CACHE, -1, -1, -1, -1, -1,
-1, -1);
+ this(backendId, signature, functionId, false);
+ }
+
+ public CleanUDFCacheTask(long backendId, String signature, long
functionId, boolean retry) {
Review Comment:
The 3-arg constructor (and therefore `retry == false`) has no remaining
callers - all production and test call sites use the 4-arg form. Worth deleting
it, because the flag currently controls three things that the name does not
suggest:
- whether the FE tracks the task at all (`AgentTaskQueue.addTask` is keyed
by `signature`, so `retry` chooses between `signature = functionId` - tracked,
de-duped, resendable, expirable - and `signature = -1`, where all such tasks
collapse onto one queue entry);
- whether the BE sends a finish report (`clean_udf_cache_callback()` only
calls `finish_task()` when `req.signature != -1`);
- and therefore whether the entry can ever leave `AgentTaskQueue` other than
by the 5-minute expiry.
If the fire-and-forget form is kept for compatibility, consider expressing
it in the name (`CleanUDFCacheTask.untracked(...)` / `tracked(...)`) or
splitting the type, so a future caller cannot accidentally create a task that
is never acknowledged.
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionUtil.java:
##########
@@ -48,6 +53,33 @@
public class FunctionUtil {
private static final Logger LOG = LogManager.getLogger(FunctionUtil.class);
+ public static void cleanUdfCache(Map<Long, String> functionSignatures) {
Review Comment:
Two things:
1. Layering: `FunctionUtil` (catalog) now constructs `CleanUDFCacheTask` and
drives `AgentTaskExecutor`, so the catalog package depends on the task package.
`cleanUdfCache` and `submitUdfCacheCleanupTasks` are also two public entry
points with overlapping meaning (one builds + sends, one only sends). Moving
construction/submission into a small helper in `org.apache.doris.task` and
leaving `FunctionUtil` with `collectPythonFunctions` would keep the dependency
one-way and make the split obvious.
2. Call sites: `CatalogRecycleBin.eraseDatabase(Database)` calls this while
holding the recycle-bin write lock (`eraseDatabase(currentTimeMs, keepNum)` /
`eraseDatabaseWithSameName` / `eraseDatabaseInstantly`).
`AgentTaskExecutor.submit()` is non-blocking (cached pool +
`RejectedExecutionException` handling), so no deadlock - but building
`#backends x #functions` task objects under that lock is avoidable; the set
could be submitted after unlocking, like `InternalCatalog.dropDb()` does.
`collectPythonFunctions()` uses `Collectors.toMap(Function::getId, ...)`,
which throws on duplicate keys - fine as long as function ids stay globally
unique (they do, `Env.getNextId()`), just flagging the implicit assumption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]