yujun777 opened a new pull request, #70466: URL: https://github.com/apache/doris/pull/70466
### Background ANALYZE collects column statistics in tasks and writes buffered results to the internal statistics table. Live jobs retain their tasks and SQL executors, while separate bounded maps retain job/task history. ### What problem does this PR solve? Issue Number: None Related PR: #66488 (shared partition-statistics map cleanup) PR Type: Bugfix Problem Summary: A failed statistics INSERT can leave previously collected tasks in RUNNING state and retain their job, executor, and query plans indefinitely. The watchdog treats execution failures as cancellations, while the worker skips failure handling once `killed` is set; timeout and explicit cancellation also have incomplete job-level cleanup. This change makes job termination idempotent, unregisters both live-job maps, and releases SQL executors after execution and audit logging finish. ### Details - **Change map:** Production changes cover job/batch lifetime, terminal status serialization, cancellation routing, and executor references. The tests cover failure, timeout, history eviction, late callbacks, and concurrent flush/cancel behavior. - **Frequency:** Repeated statistics-write failures can accumulate live jobs; occurrence depends on the worker/watchdog interleaving. Timeout and user cancellation are additional terminal paths. - **Root cause:** Partial cleanup relies on all task status callbacks reaching a terminal state. The watchdog catches `ExecutionException` as though it were a timeout, and the worker's `killed` guard can suppress the job failure callback. Completed INSERT executors also remain referenced. - **Why existing tests missed it:** Job tests mocked the flush and did not verify retained executors, both live registries, or a delayed completion racing cancellation. - **Fix:** Serialize batch ownership under the job monitor but execute SQL outside it, so cancellation can interrupt a flush. Use one terminal decision for success/failure/cancel; preserve the cancellation handle until the SQL exits, reject late buffer/partition-map writes, and prevent terminal task status from being overwritten. History eviction no longer prevents live-job status updates or cleanup. - **Other paths and branches:** Automatic, asynchronous manual, synchronous, timeout, and KILL ANALYZE paths are covered. Successful synchronous analyze retains the partition metadata until its caller updates table statistics. This PR targets master. ### Testing - `bash run-fe-ut.sh --run 'org.apache.doris.statistics.analysis.*Test'`: 110 tests passed, 0 failures/errors/skips. - `mvn checkstyle:check -pl fe-core`: 0 violations. - Local review completed: no outstanding findings. Verified terminal cleanup, synchronous metadata consumption, history eviction, final small batches, and flush/cancellation interleavings. ### Release note FE releases execution state from completed, failed, or cancelled ANALYZE jobs, preventing heap growth from retained statistics INSERT plans. ### Check List (For Author) - Behavior changed: Yes (failed and cancelled ANALYZE jobs reliably terminate and release execution state). - Does this need documentation: No. - Pre-submission status: Ready: package tests, Checkstyle, and local review passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
