Hi all, I'm wanting to run quite a number of PyFlink jobs on Kubernetes, where the amount of state and number of events being processed is small and therefore I'd like to use as little memory in my clusters as possible so I can bin pack most efficiently. I'm running a Flink cluster per job. I'm currently trying to see how small I can make the memory settings.
I set taskmanager.memory.process.size: 512mb in my Flink config. The container is being started with requested memory of 512Mi. When my TaskManager starts up, I get an error message: IllegalConfigurationException: Sum of configured Framework Heap Memory (128mb), Framework Off-Heap Memory (128mb), Task Off-Heap Memory (0 bytes), Managed Memory (25.6mb) and Network Memory (64mb) exceed configured Total Flink Memory (64mb). My understanding of the docs (https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/mem_setup_tm/) is that I should just need to set taskmanager.memory.process.size. Where is the 64mb figure coming from which makes up the Total Flink Memory? How do I change this? Many thanks, John
