yiyutian1 opened a new pull request, #29980:
URL: https://github.com/apache/flink/pull/29980
## What is the purpose of the change
In batch mode, a query calling an `AsyncTableFunction` through `LATERAL`
fails with "Cannot generate a valid execution plan for the given query:
Unsupported function for scan:ASYNC_TABLE". This PR plans it as an async
correlate on `AsyncWaitOperator`, sharing the planner logic with the streaming
async correlate.
This is the async table function counterpart of #29939 (FLINK-41482, async
scalar functions in batch). The two PRs are independent and can be merged in
either order.
## Notes to reviewers
The PR has two commits: the first contains the actual logic and docs, the
second only tests.
## Brief change log
- Add `BatchPhysicalAsyncCorrelateRule`, `BatchPhysicalAsyncCorrelate` and
exec node `BatchExecAsyncCorrelate` (`batch-exec-async-correlate`, version 1),
reusing `CommonExecAsyncCorrelate`. `BatchPhysicalCorrelateRule` skips async
table functions, and `BatchPhysicalConstantTableFunctionScanRule` handles them
like its streaming counterpart.
- Extract `CommonPhysicalAsyncCorrelateRule` from
`StreamPhysicalAsyncCorrelateRule`, and move the projection and condition check
into `AsyncTableUtil#validateAsyncCorrelate`, so stream and batch share them.
The streaming behaviour doesn't change.
- Add `AsyncCorrelateSplitRule.CORRELATE_SPLIT_ASYNC_TABLE` and
`FlinkLogicalCalcRemoveRule` to the batch logical rewrite, matching the
streaming rule set.
- Keep `BatchExecAsyncCorrelate` out of multiple-input operators, since
`TableOperatorWrapper` doesn't set the mailbox executor that
`AsyncWaitOperatorFactory` needs. Let it propagate distribution in
`ForwardHashExchangeProcessor` like `BatchExecCorrelate`.
- Mark the `table.exec.async-table.*` options as batch and streaming, and
document that a retry still pending when a bounded input ends gets one more
attempt.
## Verifying this change
This change added tests and can be verified as follows:
- `AsyncCorrelateTest` (batch plan tests), including left join, scalar
arguments, a constant async table function, the unsupported projection and
condition cases, and multiple-input handling.
- `AsyncCorrelateBatchRestoreTest`, using `AsyncCorrelateTestPrograms`,
which moves to `exec/common`.
- `AsyncCorrelateBatchSemanticTests`: calls still in flight when the input
ends, a scalar function in the async table function's argument, and an async
correlate between an aggregation and a join (this one fails without the
multiple-input exclusion).
- `ForwardHashExchangeTest` case for an async correlate between a hash
join and a hash aggregate.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no (only the documented exec mode of existing options)
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]