[ 
https://issues.apache.org/jira/browse/FLINK-41482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yiyu Tian updated FLINK-41482:
------------------------------
    Description: 
Async functions are only supported in streaming. FLIP-400 (AsyncScalarFunction) 
and FLIP-498 (AsyncTableFunction) scoped batch out.

In batch mode today, an AsyncScalarFunction call is planned by 
BatchPhysicalCalcRule as a regular BatchPhysicalCalc and fails in calc codegen 
with "Only scalar functions can be used in a projection or filter operation". 
The async table function counterpart is FLINK-37927.

Proposal: mirror the streaming path in the batch planner.
* The async calc split rules already run in sync UDFs). 
AddAsyncCorrelateSplitRule.CORRELATE_SPLIT_ASYNC_SCALAR so async calls in 
correlate arguments are split out too.
* Add BatchPhysicalAsyncCalcRule / BatchPhyse BatchExecAsyncCalc, 
reusingCommonExecAsyncCalc, the async codegen and AsyncWaitOperator.
* Keep the existing table.exec.async-scalar.* options (max concurrent 
operations, timeout, retry). Output mode stays ORDERED.
* Batch input ends with DRAIN, so in-flight calls complete before the task 
finishes.
* Keep async exec nodes out of batch multiple-input operators, since the 
multiple-input operator doesn't set the
mailbox executor that AsyncWaitOperatorFacto

This only turns a query that fails today into one that runs. Existing batch and 
streaming plans don't change.

Known limitation: at end of input, AsyncWaitg retry one immediate attempt and 
thendisables retries. In batch, a call that fails more than once fails the job 
regardless of max-attempts. This PR documents it, and fixing it needs an 
operator change, which can be a follow-up.

Both FLIPs scoped batch out, so I'm happy to follow up with a dev@ discussion 
or a FLIP if the community thinks it's needed.

  was:
Async functions are only supported in streaming. FLIP-400 (AsyncScalarFunction) 
and FLIP-498 (AsyncTableFunction) scoped batch out.

In batch mode today, an AsyncScalarFunction call is planned by 
BatchPhysicalCalcRule as a regular BatchPhysicalCalc and fails in calc codegen 
with "Only scalar functions can be used in a projection or filter operation". 
The async table function counterpart is FLINK-37927.

Proposal: mirror the streaming path in the batch planner.
* Run the async calc split rules in the batch logical rewrite phase.
* Add BatchPhysicalAsyncCalcRule / BatchPhysicalAsyncCalc and exec node 
BatchExecAsyncCalc, reusing CommonExecAsyncCalc, the async codegen and 
AsyncWaitOperator.
* Keep the existing table.exec.async-scalar.* options (buffer capacity, 
timeout, retry). Output mode stays ORDERED.
* Batch input ends with DRAIN, so in-flight calls complete before the task 
finishes.
* Keep async exec nodes out of batch multiple-input operators, since 
AsyncWaitOperatorFactory needs the task mailbox executor.

This only turns a query that fails today into one that runs. Existing batch and 
streaming plans don't change.

Both FLIPs scoped batch out, so I'm happy to follow up with a dev@ discussion 
or a FLIP if the community thinks it's needed.


> Support AsyncScalarFunction in batch mode
> -----------------------------------------
>
>                 Key: FLINK-41482
>                 URL: https://issues.apache.org/jira/browse/FLINK-41482
>             Project: Flink
>          Issue Type: Improvement
>          Components: Table SQL / Planner
>    Affects Versions: 2.5.0
>            Reporter: Yiyu Tian
>            Priority: Major
>
> Async functions are only supported in streaming. FLIP-400 
> (AsyncScalarFunction) and FLIP-498 (AsyncTableFunction) scoped batch out.
> In batch mode today, an AsyncScalarFunction call is planned by 
> BatchPhysicalCalcRule as a regular BatchPhysicalCalc and fails in calc 
> codegen with "Only scalar functions can be used in a projection or filter 
> operation". The async table function counterpart is FLINK-37927.
> Proposal: mirror the streaming path in the batch planner.
> * The async calc split rules already run in sync UDFs). 
> AddAsyncCorrelateSplitRule.CORRELATE_SPLIT_ASYNC_SCALAR so async calls in 
> correlate arguments are split out too.
> * Add BatchPhysicalAsyncCalcRule / BatchPhyse BatchExecAsyncCalc, 
> reusingCommonExecAsyncCalc, the async codegen and AsyncWaitOperator.
> * Keep the existing table.exec.async-scalar.* options (max concurrent 
> operations, timeout, retry). Output mode stays ORDERED.
> * Batch input ends with DRAIN, so in-flight calls complete before the task 
> finishes.
> * Keep async exec nodes out of batch multiple-input operators, since the 
> multiple-input operator doesn't set the
> mailbox executor that AsyncWaitOperatorFacto
> This only turns a query that fails today into one that runs. Existing batch 
> and streaming plans don't change.
> Known limitation: at end of input, AsyncWaitg retry one immediate attempt and 
> thendisables retries. In batch, a call that fails more than once fails the 
> job regardless of max-attempts. This PR documents it, and fixing it needs an 
> operator change, which can be a follow-up.
> Both FLIPs scoped batch out, so I'm happy to follow up with a dev@ discussion 
> or a FLIP if the community thinks it's needed.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to