[
https://issues.apache.org/jira/browse/FLINK-41310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bonnie Varghese updated FLINK-41310:
------------------------------------
Description:
{\{BatchExecAdaptiveJoin}} computes the join's managed memory budget at
translation time from the node's config, but builds the operator at runtime,
where it reads \{{table.exec.resource.external-buffer-memory}} from the job
config. The buffer is a share of that budget, so both values should come from
the same config.
{\{BatchExecHashJoin}} and \{{BatchExecSortMergeJoin}} read both values at
translation time. The adaptive join only decides the join strategy at runtime;
its memory budget is already fixed before it runs.
This should be fixed after FLINK-41263 is merged.
was:
BatchExecAdaptiveJoin computes the join's managed memory from the compiled
plan's config. It builds the actual join operator at runtime, though, and that
operator reads
`table.exec.resource.external-buffer-memory` from the config of the session
that executes the plan.
If the two sessions use different values, the buffer no longer fits the budget.
There are two problems associated with this:
1. A large enough value fails the job with "Memory size is too small".
2. A smaller difference just takes memory from the sorters without any error.
BatchExecHashJoin and BatchExecSortMergeJoin aren't affected, because they read
both values from the plan. This should be fixed after FLINK-41263 is merged.
> BatchExecAdaptiveJoin should take external-buffer-memory from the compiled
> plan
> -------------------------------------------------------------------------------
>
> Key: FLINK-41310
> URL: https://issues.apache.org/jira/browse/FLINK-41310
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Affects Versions: 2.5.0
> Reporter: Bonnie Varghese
> Assignee: Bonnie Varghese
> Priority: Major
> Fix For: 2.5.0
>
>
> {\{BatchExecAdaptiveJoin}} computes the join's managed memory budget at
> translation time from the node's config, but builds the operator at runtime,
> where it reads \{{table.exec.resource.external-buffer-memory}} from the job
> config. The buffer is a share of that budget, so both values should come from
> the same config.
> {\{BatchExecHashJoin}} and \{{BatchExecSortMergeJoin}} read both values at
> translation time. The adaptive join only decides the join strategy at
> runtime; its memory budget is already fixed before it runs.
> This should be fixed after FLINK-41263 is merged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
