linrrzqqq opened a new pull request, #69255: URL: https://github.com/apache/doris/pull/69255
### What problem does this PR solve? Issue Number: None doc: https://github.com/apache/doris-website/pull/4517 Problem Summary: Refactor Python UDF process scheduling to maintain an independent process pool for each function id, simplifying process allocation, package isolation and lifecycle management for Python UDF, UDAF and UDTF. As part of the refactoring, add two function properties: - `pool_size`: maximum number of Python processes per function on each BE, defaulting to `3`. - `auto_suspend`: idle timeout in seconds, defaulting to `14400`(4h) to keep pools resident. Nonnegative values allow idle pools to be terminated during the 30-second monitor scan and recreated on the next invocation. Both properties are persisted in function metadata and currently supported only for Python functions. Adapt cleanup to the new pool ownership: - DROP FUNCTION releases the corresponding pool while existing clients retain the processes and package files they still use. - Processes retained by active clients remain included in memory accounting. - Permanently removed databases trigger cleanup of their Python function pools. - Deleted function ids cannot recreate package-cache entries that could delete files still used by active queries. ### Release note Python functions use independent process pools. The pool_size and auto_suspend properties control pool capacity and idle reclamation. ### Check List (For Author) - Test: Full BE/FE build; 83 BE unit tests; 46 FE unit tests; clang-tidy and header hygiene; cluster startup and basic query verification. Python regression suites were exercised during implementation. - Behavior changed: Yes, Python process allocation and lifecycle management are scoped by function id, with configurable pool size and idle reclamation. - Does this need documentation: Yes, English and Chinese documentation changes are prepared separately in doris-website. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
