yiyutian1 opened a new pull request, #29939:
URL: https://github.com/apache/flink/pull/29939
## What is the purpose of the change
In batch mode, an `AsyncScalarFunction` call is planned as a regular
`BatchPhysicalCalc` and fails in calc codegen with "Only scalar functions can
be used in a projection or filter operation". This PR plans it as an async calc
on `AsyncWaitOperator`, mirroring the streaming path.
## Brief change log
- Add `BatchPhysicalAsyncCalcRule`, `BatchPhysicalAsyncCalc` and exec node
`BatchExecAsyncCalc` (`batch-exec-async-calc`, version 1), reusing
`CommonExecAsyncCalc`. `BatchPhysicalCalcRule` skips calcs with async calls.
- The async calc split rules already run in batch. Add
`AsyncCorrelateSplitRule.CORRELATE_SPLIT_ASYNC_SCALAR` so async calls in
correlate arguments are split out too.
- Keep `BatchExecAsyncCalc` out of multiple-input operators, and let it
propagate distribution in `ForwardHashExchangeProcessor` like `BatchExecCalc`.
- Mark the `table.exec.async-scalar.*` options as batch and streaming, and
document in `udfs.md` that a retry still pending at end of input gets one more
attempt in batch.
## Verifying this change
This change added tests and can be verified as follows:
- `AsyncCalcTest` (batch plan tests), including mixed Python and Java
async calls, and `BatchPhysicalAsyncCalcRuleTest`.
- `AsyncCalcBatchRestoreTest`, reusing the streaming
`AsyncCalcTestPrograms`.
- `AsyncCalcITCase` (batch): end-to-end results, output order, async calls
in filters and correlate arguments, aggregation on top of an async calc, and
retry at end of input.
- `ForwardHashExchangeTest` case for an async calc between a hash join and
a hash aggregate.
- Existing batch plan tests are unchanged.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no (only the documented exec mode of existing options)
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]