[
https://issues.apache.org/jira/browse/FLINK-41310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bonnie Varghese updated FLINK-41310:
------------------------------------
Description:
BatchExecAdaptiveJoin computes the join's managed memory from the compiled
plan's config. It builds the actual join operator at runtime, though, and that
operator reads \\{{table.exec.resource.external-buffer-memory}} from the config
of the session that executes the plan.
If the two sessions use different values, the buffer no longer fits the budget.
There are two problems associated with this:
1. A large enough value fails the job with "Memory size is too small".
2. A smaller difference just takes memory from the sorters without any error.
BatchExecHashJoin and BatchExecSortMergeJoin aren't affected, because they read
both values from the plan. This should be fixed after FLINK-41263 is merged.
was:
BatchExecAdaptiveJoin computes the join's managed memory from the compiled
plan's config. It builds the actual join operator at runtime, though, and that
operator reads `table.exec.resource.external-buffer-memory` from the config of
the session that executes the plan.
If the two sessions use different values, the buffer no longer fits the budget.
There are two problems associated with this:
1. A large enough value fails the job with "Memory size is too small".
2. A smaller difference just takes memory from the sorters without any error.
BatchExecHashJoin and BatchExecSortMergeJoin aren't affected, because they read
both values from the plan. This should be fixed after FLINK-41263 is merged.
> BatchExecAdaptiveJoin sizes the external buffer from the runtime config
> instead of the compiled plan
> -----------------------------------------------------------------------------------------------------
>
> Key: FLINK-41310
> URL: https://issues.apache.org/jira/browse/FLINK-41310
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Affects Versions: 2.5.0
> Reporter: Bonnie Varghese
> Assignee: Bonnie Varghese
> Priority: Major
> Fix For: 2.5.0
>
>
> BatchExecAdaptiveJoin computes the join's managed memory from the compiled
> plan's config. It builds the actual join operator at runtime, though, and
> that operator reads \\{{table.exec.resource.external-buffer-memory}} from the
> config of the session that executes the plan.
> If the two sessions use different values, the buffer no longer fits the
> budget. There are two problems associated with this:
> 1. A large enough value fails the job with "Memory size is too small".
> 2. A smaller difference just takes memory from the sorters without any error.
> BatchExecHashJoin and BatchExecSortMergeJoin aren't affected, because they
> read both values from the plan. This should be fixed after FLINK-41263 is
> merged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
