[
https://issues.apache.org/jira/browse/FLINK-41482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiyu Tian updated FLINK-41482:
------------------------------
Description:
Async functions are only supported in streaming. FLIP-400 (AsyncScalarFunction)
and FLIP-498 (AsyncTableFunction) scoped batch out.
In batch mode today, an AsyncScalarFunction call is planned by
BatchPhysicalCalcRule as a regular BatchPhysicalCalc and fails in calc codegen
with "Only scalar functions can be used in a projection or filter operation".
The async table function counterpart is FLINK-37927.
Proposal: mirror the streaming path in the batch planner.
* The async calc split rules already run in sync UDFs).
AddAsyncCorrelateSplitRule.CORRELATE_SPLIT_ASYNC_SCALAR so async calls in
correlate arguments are split out too.
* Add BatchPhysicalAsyncCalcRule / BatchPhyse BatchExecAsyncCalc,
reusingCommonExecAsyncCalc, the async codegen and AsyncWaitOperator.
* Keep the existing table.exec.async-scalar.* options (max concurrent
operations, timeout, retry). Output mode stays ORDERED.
* Batch input ends with DRAIN, so in-flight calls complete before the task
finishes.
* Keep async exec nodes out of batch multiple-input operators, since the
multiple-input operator doesn't set the
mailbox executor that AsyncWaitOperatorFacto
This only turns a query that fails today into one that runs. Existing batch and
streaming plans don't change.
Known limitation: at end of input, AsyncWaitg retry one immediate attempt and
thendisables retries. In batch, a call that fails more than once fails the job
regardless of max-attempts. This PR documents it, and fixing it needs an
operator change, which can be a follow-up.
Both FLIPs scoped batch out, so I'm happy to follow up with a dev@ discussion
or a FLIP if the community thinks it's needed.
was:
Async functions are only supported in streaming. FLIP-400 (AsyncScalarFunction)
and FLIP-498 (AsyncTableFunction) scoped batch out.
In batch mode today, an AsyncScalarFunction call is planned by
BatchPhysicalCalcRule as a regular BatchPhysicalCalc and fails in calc codegen
with "Only scalar functions can be used in a projection or filter operation".
The async table function counterpart is FLINK-37927.
Proposal: mirror the streaming path in the batch planner.
* Run the async calc split rules in the batch logical rewrite phase.
* Add BatchPhysicalAsyncCalcRule / BatchPhysicalAsyncCalc and exec node
BatchExecAsyncCalc, reusing CommonExecAsyncCalc, the async codegen and
AsyncWaitOperator.
* Keep the existing table.exec.async-scalar.* options (buffer capacity,
timeout, retry). Output mode stays ORDERED.
* Batch input ends with DRAIN, so in-flight calls complete before the task
finishes.
* Keep async exec nodes out of batch multiple-input operators, since
AsyncWaitOperatorFactory needs the task mailbox executor.
This only turns a query that fails today into one that runs. Existing batch and
streaming plans don't change.
Both FLIPs scoped batch out, so I'm happy to follow up with a dev@ discussion
or a FLIP if the community thinks it's needed.
> Support AsyncScalarFunction in batch mode
> -----------------------------------------
>
> Key: FLINK-41482
> URL: https://issues.apache.org/jira/browse/FLINK-41482
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
> Affects Versions: 2.5.0
> Reporter: Yiyu Tian
> Priority: Major
>
> Async functions are only supported in streaming. FLIP-400
> (AsyncScalarFunction) and FLIP-498 (AsyncTableFunction) scoped batch out.
> In batch mode today, an AsyncScalarFunction call is planned by
> BatchPhysicalCalcRule as a regular BatchPhysicalCalc and fails in calc
> codegen with "Only scalar functions can be used in a projection or filter
> operation". The async table function counterpart is FLINK-37927.
> Proposal: mirror the streaming path in the batch planner.
> * The async calc split rules already run in sync UDFs).
> AddAsyncCorrelateSplitRule.CORRELATE_SPLIT_ASYNC_SCALAR so async calls in
> correlate arguments are split out too.
> * Add BatchPhysicalAsyncCalcRule / BatchPhyse BatchExecAsyncCalc,
> reusingCommonExecAsyncCalc, the async codegen and AsyncWaitOperator.
> * Keep the existing table.exec.async-scalar.* options (max concurrent
> operations, timeout, retry). Output mode stays ORDERED.
> * Batch input ends with DRAIN, so in-flight calls complete before the task
> finishes.
> * Keep async exec nodes out of batch multiple-input operators, since the
> multiple-input operator doesn't set the
> mailbox executor that AsyncWaitOperatorFacto
> This only turns a query that fails today into one that runs. Existing batch
> and streaming plans don't change.
> Known limitation: at end of input, AsyncWaitg retry one immediate attempt and
> thendisables retries. In batch, a call that fails more than once fails the
> job regardless of max-attempts. This PR documents it, and fixing it needs an
> operator change, which can be a follow-up.
> Both FLIPs scoped batch out, so I'm happy to follow up with a dev@ discussion
> or a FLIP if the community thinks it's needed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)