linrrzqqq opened a new pull request, #69255:
URL: https://github.com/apache/doris/pull/69255
### What problem does this PR solve?
Issue Number: None
Problem Summary:
Refactor Python UDF process scheduling to maintain an independent process
pool for each function id, simplifying process allocation, package isolation
and lifecycle management for Python UDF, UDAF and UDTF.
As part of the refactoring, add two function properties:
- `pool_size`: maximum number of Python processes per function on each BE,
defaulting to `3`.
- `auto_suspend`: idle timeout in seconds, defaulting to `-1` to keep pools
resident. Nonnegative values allow idle pools to be terminated during the
30-second monitor scan and recreated on the next invocation.
Both properties are persisted in function metadata and currently supported
only for Python functions.
Adapt cleanup to the new pool ownership:
- DROP FUNCTION releases the corresponding pool while existing clients
retain the processes and package files they still use.
- Processes retained by active clients remain included in memory accounting.
- Permanently removed databases trigger cleanup of their Python function
pools.
- Deleted function ids cannot recreate package-cache entries that could
delete files still used by active queries.
### Release note
Python functions use independent process pools. The pool_size and
auto_suspend properties control pool capacity and idle reclamation.
### Check List (For Author)
- Test: Full BE/FE build; 83 BE unit tests; 46 FE unit tests; clang-tidy and
header hygiene; cluster startup and basic query verification. Python regression
suites were exercised during implementation.
- Behavior changed: Yes, Python process allocation and lifecycle management
are scoped by function id, with configurable pool size and idle reclamation.
- Does this need documentation: Yes, English and Chinese documentation
changes are prepared separately in doris-website.
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
