[
https://issues.apache.org/jira/browse/FLINK-18019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18019:
----------------------------
Description:
Currently If the following configuration is configured in flink-conf.yaml:
{code:java}
state.backend: rocksdb
state.checkpoints.dir: file:///tmp/flink-checkpoints
{code}
and the following configuration is configured via TableConfig:
{code:java}
tableConfig.getConfiguration().setString("state.backend.rocksdb.memory.fixed-per-slot",
"200MB")
tableConfig.getConfiguration().setString("taskmanager.memory.task.off-heap.size",
"200MB")
{code}
Then users submit the job via CliFrontend, the configuration set via
TableConfig will not take effect.
Intuitively, it should be that user specified configuration via TableConfig(has
higher priority) and the configuration specified via flink-conf.yaml together
determines the configuration of a job. However, it doesn't hold in all cases.
The root cause is that only the configuration specified in TableConfig in
passed to *StreamExecutionEnvironment* during translate to plan. For the above
case, as *state.backend* is not specified in TableConfig and so the
configuration *state.backend.rocksdb.memory.fixed-per-slot* will also not take
effect. Please note that in this case, the state backend will be RocksDB
without the configuration *state.backend.rocksdb.memory.fixed-per-slot* and
*taskmanager.memory.task.off-heap.size.*
was:
Currently If the following configuration is configured in flink-conf.yaml:
{code:java}
state.backend: rocksdb
state.checkpoints.dir: file:///tmp/flink-checkpoints
{code}
and the following configuration is configured via TableConfig:
{code:java}
tableConfig.getConfiguration().setString("state.backend.rocksdb.memory.fixed-per-slot",
"200MB")
tableConfig.getConfiguration().setString("taskmanager.memory.task.off-heap.size",
"200MB")
{code}
Then users submit the job via CliFrontend, the configuration set via
TableConfig will not take effect.
Intuitively, it should be that user specified configuration via TableConfig(has
higher priority) and the configuration specified via flink-conf.yaml together
determines the configuration of a job. However, it doesn't hold in all cases.
The root cause is that only the configuration specified in TableConfig in
passed to *StreamExecutionEnvironment*. For the above case, as *state.backend*
is not specified in TableConfig, the configuration
*state.backend.rocksdb.memory.fixed-per-slot* will also not take effect.
> The configuration specified in TableConfig may not take effect in certain
> cases
> -------------------------------------------------------------------------------
>
> Key: FLINK-18019
> URL: https://issues.apache.org/jira/browse/FLINK-18019
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Legacy Planner, Table SQL / Planner
> Affects Versions: 1.10.0, 1.11.0
> Reporter: Dian Fu
> Priority: Major
>
> Currently If the following configuration is configured in flink-conf.yaml:
> {code:java}
> state.backend: rocksdb
> state.checkpoints.dir: file:///tmp/flink-checkpoints
> {code}
> and the following configuration is configured via TableConfig:
> {code:java}
> tableConfig.getConfiguration().setString("state.backend.rocksdb.memory.fixed-per-slot",
> "200MB")
> tableConfig.getConfiguration().setString("taskmanager.memory.task.off-heap.size",
> "200MB")
> {code}
> Then users submit the job via CliFrontend, the configuration set via
> TableConfig will not take effect.
> Intuitively, it should be that user specified configuration via
> TableConfig(has higher priority) and the configuration specified via
> flink-conf.yaml together determines the configuration of a job. However, it
> doesn't hold in all cases.
> The root cause is that only the configuration specified in TableConfig in
> passed to *StreamExecutionEnvironment* during translate to plan. For the
> above case, as *state.backend* is not specified in TableConfig and so the
> configuration *state.backend.rocksdb.memory.fixed-per-slot* will also not
> take effect. Please note that in this case, the state backend will be RocksDB
> without the configuration *state.backend.rocksdb.memory.fixed-per-slot* and
> *taskmanager.memory.task.off-heap.size.*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
