Hi flink fans,
I'm hoping for an easy solution. I'm trying to upgrade my 9.3 cluster to
flink 10.1, but i'm running into memory configuration errors.
Such as:
*Caused by: org.apache.flink.configuration.IllegalConfigurationException:
The network memory min (64 mb) and max (1 gb) mismatch, the network memory
has to be resolved and set to a fixed value before task executor starts*
*Caused by: org.apache.flink.configuration.IllegalConfigurationException:
The required configuration option Key: 'taskmanager.cpu.cores' , default:
null (fallback keys: []) is not set*
I was able to fix a cascade of errors by explicitly setting these values:
taskmanager.memory.managed.size: {{
.Values.analytics.flink.taskManagerManagedSize }}
taskmanager.memory.task.heap.size: {{
.Values.analytics.flink.taskManagerHeapSize }}
taskmanager.memory.jvm-metaspace.size: 500m
taskmanager.cpu.cores: 4
So, the documentation implies that flink will default many of these values,
however my 101. cluster doesn't seem to be doing this. 9.3, worked great!
Do I really have to set all the memory (even network) values? If not, what
am I missing?
If i do have to set all the memory parameters, how do I resolve "The
network memory min (64 mb) and max (1 gb) mismatch"?
My cluster runs standalone jobs on kube
flnk-config.yaml:
state.backend: rocksdb
state.backend.incremental: true
state.checkpoints.num-retained: 1
taskmanager.memory.managed.size: {{
.Values.analytics.flink.taskManagerManagedSize }}
taskmanager.memory.task.heap.size: {{
.Values.analytics.flink.taskManagerHeapSize }}
taskmanager.memory.jvm-metaspace.size: 500m
taskmanager.cpu.cores: 4
taskmanager.numberOfTaskSlots: {{
.Values.analytics.task.numberOfTaskSlots }}
parallelism.default: {{ .Values.analytics.flink.parallelism }}
JobManger:
command: ["/opt/flink/bin/standalone-job.sh"]
args: ["start-foreground", "-j={{ .Values.analytics.flinkRunnable
}}", ...
TakManager
command: ["/opt/flink/bin/taskmanager.sh"]
args: [
"start-foreground",
"-Djobmanager.rpc.address=localhost",
"-Dmetrics.reporter.prom.port=9430"]