BugsEverywhere opened a new pull request, #29470: URL: https://github.com/apache/flink/pull/29470
## What is the purpose of the change Implement [FLINK-40471](https://issues.apache.org/jira/browse/FLINK-40471), as specified in FLIP-591. DataFrame scalar UDFs can configure their Python stage's parallelism with `concurrency` and pandas/Arrow batch sizes with `batch_size`, independently of other UDFs. Omitting either option preserves the existing behavior. ## Brief change log - Validate the optional positive integer parameters and reject `batch_size` for general UDFs. - Carry execution settings into Java Python functions, split UDFs with different settings into separate stages, and apply each stage's parallelism and batch-size overrides. - Document the parameters and add declaration, planner, and execution tests. ## Verifying this change - All 28 DataFrame UDF tests pass, covering validation, defaults, streaming/batch plans with separate and nested UDFs, and actual pandas/Arrow batch sizes in the same pipeline. - Existing Python function-kind splitting and nested CSE Java tests pass (33 tests). - Maven reactor verification on Java 21 completed across all modules over resumed runs. - Spotless, Checkstyle, Python type checking, and changed-file Flake8 pass. - Full Sphinx documentation build passes with warnings treated as errors. ## Does this pull request potentially affect one of the following parts: - Dependencies: no. - Public API: yes; optional DataFrame `udf()` parameters specified in FLIP-591. - Serializers: no state serializer changes. - Runtime per-record code paths: no; existing operators are configured during planning. - Deployment or recovery: no changes to checkpointing or cluster deployment; explicitly configured Python stages use the requested parallelism. - S3 file system connector: no. ## Documentation - New feature: yes. - Documented in the Python API docstring and DataFrame UDF Sphinx reference. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
