924060929 commented on code in PR #69255:
URL: https://github.com/apache/doris/pull/69255#discussion_r4638337548


##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java:
##########
@@ -935,6 +937,24 @@ public synchronized List<Function> getFunctions() {
         return FunctionUtil.getFunctions(name2Function);
     }
 
+    synchronized void registerFunctionsById(ConcurrentMap<Long, Function> 
functionsById) {
+        id2Function = functionsById;
+        for (Function function : getFunctions()) {
+            id2Function.put(function.getId(), function);
+        }
+    }
+
+    public synchronized void unregisterFunctionsById() {

Review Comment:
   Two different objects are mutated here, and the name hides it.
   
   `id2Function` on this class is only a handle to `Env.id2Function` 
(`Env.java:401`, `final`, process lifetime); the map has several holders - Env 
itself, every `Database`, and `GlobalFunctionMgr`. So:
   
   - `id2Function.remove(...)` mutates **Env's shared map**. This is the only 
thing that makes `Env.getFunctionById()` stop returning these ids, i.e. what 
drives the report-based `CLEAN_UDF_CACHE` reconciliation 
(`ReportHandler.java:415`).
   - `id2Function = null` mutates **only this object's field**. It stops 
`FunctionUtil.addFunctionImpl()` from registering new functions 
(`FunctionUtil.java:174`); it removes nothing from the shared map.
   
   Both are needed - I traced the FORCE DROP path, where this runs at drop time 
and again from `CatalogRecycleBin.eraseDatabase()`, so the early return is 
required - but the order is forced (the field is the only handle to the map) 
and bundling them makes the method read as "just null the field". A reader who 
drops the loop silently breaks the compensation path: if the drop-time cleanup 
never reached a backend (BE down, task expired, FE restarted after the 
recycle-bin entry was erased), FE would still "know" the id and never ask the 
BE to release that function's process pool and package files.
   
   On the name: `...ById` takes no id argument, and "unregister this database's 
functions" also reads as "delete them", which would be wrong - the `Function` 
objects must survive for `RECOVER DATABASE`.
   
   Suggestions, in increasing order of change:
   
   1. Keep one method, but name it for what it does and document both effects, 
e.g. `detachFunctionIndex()` (same for `GlobalFunctionMgr`, 
`detachFunctionIndexes` for the bulk `CatalogRecycleBin` variant), with 
Javadoc: "removes this database's functions from Env's shared id index and 
stops registering new ones; function objects are kept for RECOVER DATABASE; 
safe to call twice".
   2. Split the two responsibilities: `Env.detachDbFunctions(Database db)` 
(mutates the shared index) + `Database.markFunctionsDetached()` (mutates the 
flag). That removes the ordering constraint and the "nullable field as detached 
sentinel" pattern relied on by 
`FunctionUtil.addFunctionImpl()`/`dropFunctionImpl()`.
   3. Or key the index by database (`Map<dbId, Map<funcId, Function>>`): 
unbinding becomes a single `remove(dbId)`.
   
   Also `remove(function.getId(), function)` uses the conditional form; ids 
come from `Env.getNextId()` and are never reused, so `remove(id)` is equivalent 
- if the CAS semantics are intentional, a comment would help.



##########
fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java:
##########
@@ -389,6 +402,23 @@ protected void exec() {
         }
     }
 
+    static void pythonUdfReport(long backendId, Set<Long> functionIds) {
+        if (functionIds.isEmpty()) {
+            return;
+        }
+        // A queued report can outlive removal of its backend.
+        if (Env.getCurrentSystemInfo().getBackend(backendId) == null) {
+            return;
+        }
+        List<CleanUDFCacheTask> cleanups = new ArrayList<>();
+        for (long functionId : functionIds) {
+            if (Env.getCurrentEnv().getFunctionById(functionId) == null) {

Review Comment:
   This line makes `Env.id2Function` the authority for a **destructive** 
action: a miss here means the BE drops both the function's process pool 
(`PythonServerManager::drop_function`) and its package cache 
(`UserFunctionCache::drop_function_cache`; files are deleted when the last 
reference is released). The only recovery is re-downloading from the function's 
`location` on the next query, so if that artifact is gone (`file://` deleted, 
URL expired) the function is permanently broken.
   
   I could not find a way to get a stale miss on a live function today - ids 
are added to the index before the name is published 
(`FunctionUtil.addFunctionImpl`), and I checked every database 
insertion/removal path plus the image/journal replay paths. The risk is 
structural rather than an existing bug: the index is a second, hand-maintained 
view of the catalog, keyed through a nullable field that doubles as the 
"detached" flag, and this PR already had to fix one path for it 
(`InternalCatalog.loadDb` now goes through `unprotectCreateDb`). Any future 
path that inserts a database or a function without going through those two 
chokepoints turns this line from "leak" into "delete a live function's files".
   
   Given how cheap it is, a confirmation window would bound the blast radius: 
keep `(backendId, functionId) -> firstSeen` and only enqueue the cleanup once 
the id has been missing for e.g. 10-30 minutes (much longer than a journal 
replay lag, far shorter than the 6h report cadence); or make the BE stage the 
deletion (mark, then delete only if not re-created within N minutes). 
Alternatively make the invariant explicit: `addFunctionImpl` should fail loudly 
when the index is not bound and the database is still in the catalog, instead 
of silently skipping the registration.



##########
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java:
##########
@@ -537,6 +539,11 @@ public void dropDb(String dbName, boolean ifExists, 
boolean force) throws DdlExc
 
                 Env.getCurrentRecycleBin().recycleDatabase(db, tableNames, 
tableIds, false, force, 0);
                 recycleTime = 
Env.getCurrentRecycleBin().getRecycleTimeById(db.getId());
+                // Recoverable databases keep function IDs until permanent 
erasure.
+                if (force) {
+                    functionSignatures = 
FunctionUtil.collectPythonFunctions(db);

Review Comment:
   This is the FORCE DROP branch, and the same functions are cleaned up twice: 
here (after the catalog lock is released - good), and again when the database 
is finally erased from the recycle bin. 
`CatalogRecycleBin.eraseDatabase(Database)` (`CatalogRecycleBin.java:458-461`) 
calls `unregisterFunctionsById()` + `cleanUdfCache(collectPythonFunctions(db))` 
on the same object.
   
   For a force drop the entry is written with `recycleTime = 0`, so it is 
erased by the next recycle-bin scan, i.e. seconds later. While the first batch 
is still queued, `AgentTaskQueue.addTask()` de-dupes on `signature == 
functionId` and nothing extra is sent; once it has been acknowledged, a second 
full round of `CLEAN_UDF_CACHE` RPCs goes out. It is idempotent on the BE, so 
this is wasted work rather than a correctness issue - but 
`DropDbTest.testForceDropDbSubmitsFunctionCacheCleanup` only asserts the 
drop-time submission, so the duplication is not covered either. Consider 
letting the recycle-bin erasure own the cleanup in all cases, or tracking which 
ids were already submitted.



##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -7792,6 +7804,14 @@ public GlobalFunctionMgr getGlobalFunctionMgr() {
         return globalFunctionMgr;
     }
 
+    public Function getFunctionById(long functionId) {
+        return id2Function.get(functionId);
+    }
+
+    public void registerDbFunctionsById(Database db) {

Review Comment:
   Binding goes through `Env` (`registerDbFunctionsById`, plus the constructor 
/ `loadCatalog` / `loadRecycleBin` / `loadGlobalFunction` hooks), but unbinding 
has no counterpart here: `Database.unregisterFunctionsById()` is called 
directly from `InternalCatalog.dropDb()`, `InternalCatalog.replayDropDb()`, 
`CatalogRecycleBin.eraseDatabase(Database)` and 
`CatalogRecycleBin.replayEraseDatabase()`. Adding 
`unregisterDbFunctionsById(Database db)` and routing those four call sites 
through it would put both directions of this index's lifecycle in one class, 
and makes the "only Env mutates the shared map" rule visible in one place.



##########
fe/fe-core/src/main/java/org/apache/doris/catalog/GlobalFunctionMgr.java:
##########
@@ -45,6 +45,21 @@ public class GlobalFunctionMgr extends MetaObject implements 
GsonPostProcessable
     // user define function
     @SerializedName(value = "name2Function")
     private ConcurrentMap<String, ImmutableList<Function>> name2Function = 
Maps.newConcurrentMap();
+    private transient ConcurrentMap<Long, Function> id2Function;
+
+    synchronized void registerFunctionsById(ConcurrentMap<Long, Function> 
functionsById) {
+        id2Function = functionsById;
+        for (Function function : getFunctions()) {
+            id2Function.put(function.getId(), function);
+        }
+    }
+
+    synchronized void unregisterFunctionsById() {
+        for (Function function : getFunctions()) {
+            id2Function.remove(function.getId(), function);

Review Comment:
   Asymmetric with `Database.unregisterFunctionsById()` (`Database.java:949`), 
which returns early when `id2Function == null`; the null check is missing here. 
Not reachable today (the only instance is created and bound in the `Env` 
constructor, and `Env.loadGlobalFunction()` re-binds after replacing), but 
`Env.loadGlobalFunction()` does call this on the instance being replaced, so 
any `GlobalFunctionMgr` constructed outside `Env` (tests, future code) would 
NPE. Either guard it the same way, or drop the guard on both sides and document 
the "always bound" invariant.



##########
fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java:
##########
@@ -199,6 +199,9 @@ public TMasterResult finishTaskInternal(TFinishTaskRequest 
request) {
                 case DROP:
                     finishDropReplica(task);
                     break;
+                case CLEAN_UDF_CACHE:

Review Comment:
   For a best-effort task like this, a finish report can legitimately find no 
queued task: the entry expires after `CleanUDFCacheTask.RETENTION_MS` (5 min) 
or is dropped when the backend is removed, while the BE may still send 
`finish_task` later. In that case the code above returns `CANCELLED` and logs 
`cannot find task. type: CLEAN_UDF_CACHE ...`. `DROP`, `RELEASE_SNAPSHOT` and 
`CLEAR_TRANSACTION_TASK` are already tolerated there; `CLEAN_UDF_CACHE` looks 
like it belongs in the same set, to avoid spurious warnings (the BE does not 
retry on a non-OK result, it only logs).



##########
fe/fe-core/src/main/java/org/apache/doris/task/CleanUDFCacheTask.java:
##########
@@ -23,18 +23,30 @@
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 
+import java.util.concurrent.TimeUnit;
 
 public class CleanUDFCacheTask extends AgentTask {
+    private static final long RETENTION_MS = TimeUnit.MINUTES.toMillis(5);
     private static final Logger LOG = 
LogManager.getLogger(CleanUDFCacheTask.class);
     private String functionSignature;
     private long functionId;
 
     public CleanUDFCacheTask(long backendId, String signature, long 
functionId) {
-        super(null, backendId, TTaskType.CLEAN_UDF_CACHE, -1, -1, -1, -1, -1, 
-1, -1);
+        this(backendId, signature, functionId, false);
+    }
+
+    public CleanUDFCacheTask(long backendId, String signature, long 
functionId, boolean retry) {

Review Comment:
   The 3-arg constructor (and therefore `retry == false`) has no remaining 
callers - all production and test call sites use the 4-arg form. Worth deleting 
it, because the flag currently controls three things that the name does not 
suggest:
   
   - whether the FE tracks the task at all (`AgentTaskQueue.addTask` is keyed 
by `signature`, so `retry` chooses between `signature = functionId` - tracked, 
de-duped, resendable, expirable - and `signature = -1`, where all such tasks 
collapse onto one queue entry);
   - whether the BE sends a finish report (`clean_udf_cache_callback()` only 
calls `finish_task()` when `req.signature != -1`);
   - and therefore whether the entry can ever leave `AgentTaskQueue` other than 
by the 5-minute expiry.
   
   If the fire-and-forget form is kept for compatibility, consider expressing 
it in the name (`CleanUDFCacheTask.untracked(...)` / `tracked(...)`) or 
splitting the type, so a future caller cannot accidentally create a task that 
is never acknowledged.



##########
fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionUtil.java:
##########
@@ -48,6 +53,33 @@
 public class FunctionUtil {
     private static final Logger LOG = LogManager.getLogger(FunctionUtil.class);
 
+    public static void cleanUdfCache(Map<Long, String> functionSignatures) {

Review Comment:
   Two things:
   
   1. Layering: `FunctionUtil` (catalog) now constructs `CleanUDFCacheTask` and 
drives `AgentTaskExecutor`, so the catalog package depends on the task package. 
`cleanUdfCache` and `submitUdfCacheCleanupTasks` are also two public entry 
points with overlapping meaning (one builds + sends, one only sends). Moving 
construction/submission into a small helper in `org.apache.doris.task` and 
leaving `FunctionUtil` with `collectPythonFunctions` would keep the dependency 
one-way and make the split obvious.
   2. Call sites: `CatalogRecycleBin.eraseDatabase(Database)` calls this while 
holding the recycle-bin write lock (`eraseDatabase(currentTimeMs, keepNum)` / 
`eraseDatabaseWithSameName` / `eraseDatabaseInstantly`). 
`AgentTaskExecutor.submit()` is non-blocking (cached pool + 
`RejectedExecutionException` handling), so no deadlock - but building 
`#backends x #functions` task objects under that lock is avoidable; the set 
could be submitted after unlocking, like `InternalCatalog.dropDb()` does.
   
   `collectPythonFunctions()` uses `Collectors.toMap(Function::getId, ...)`, 
which throws on duplicate keys - fine as long as function ids stay globally 
unique (they do, `Env.getNextId()`), just flagging the implicit assumption.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to