auroflow opened a new pull request, #29210: URL: https://github.com/apache/flink/pull/29210
## What is the purpose of the change Implement [FLINK-40472](https://issues.apache.org/jira/browse/FLINK-40472) by adding synchronous Arrow vectorized scalar UDFs to the PyFlink DataFrame API. UDFs receive Arrow arrays and can apply `pyarrow.compute` operations directly to column batches. Arrow mode can be inferred from Arrow container annotations or selected with `func_type="arrow"`. The Table API supports explicit Arrow mode. ## Brief change log - Add Arrow UDF declaration and annotation inference, and document the new execution mode. - Distinguish general, pandas and Arrow functions in Python calculation splitting and map merging. - Extend the existing Arrow operator, worker and coder protocol to carry native Arrow batches while retaining pandas defaults. - Support nested and whole-row arguments, normalize chunked results, and validate result containers, row counts and terminal schemas. - Share Arrow schema mapping, preserve nested nullability and PyArrow 5 compatibility, reduce temporary validation allocations, and reset map child writers between batches. ## Verifying this change This change extends existing declaration, worker, coder, planner and Arrow reader/writer tests, plus the combined DataFrame UDF integration test. Tests cover observable values, schemas, execution plans and error behavior. - Full Python suite: **1,284 passed, 61 skipped, 528 subtests passed**. - Focused pure-Python and compiled coder checks passed on PyArrow 5, 18 and 23. - Targeted Java planner, protocol, Arrow operator and reader/writer tests passed, including `PythonCalcSplitFunctionKindRuleTest`, `ProtoUtilsTest` and `ArrowReaderWriterTest`. - Java 17 reactor build passed with `./mvnw clean install -DskipTests -Dfast -Pskip-webui-build -T4`. The full Maven test suite was not run. - Spotless, flake8 and mypy checks passed, including the pre-push checks. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **yes** — extends the Python DataFrame and Table UDF APIs. - The serializers: **yes** — Python Arrow coders, Arrow writers and additive worker-protocol metadata; managed-state serializers are unchanged. - The runtime per-record code paths (performance sensitive): **yes** — Python scalar execution and Arrow batch serialization. - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: **no** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **yes** - If yes, how is the feature documented? **docs** — DataFrame UDF reference documentation and Python API docstrings. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
