Jingsong Lee created FLINK-15172: ------------------------------------ Summary: optimize the operator algorithm to lazily allocate memory Key: FLINK-15172 URL: https://issues.apache.org/jira/browse/FLINK-15172 Project: Flink Issue Type: Sub-task Components: Table SQL / Runtime Reporter: Jingsong Lee Fix For: 1.11.0
Now after FLINK-14063 , operators will get all manage memory of TaskManager, The cost of over allocate memory is very high, lead to performance regression of small batch sql jobs: * Allocate memory will have the cost of memory management algorithm. * Allocate memory will have the cost of memory initialization, will set all memory to zero. And this initialization will require the operating system to actually allocate physical memory. * Over allocate memory will squash the file cache too. We can optimize the operator algorithm, apply lazy allocation, and avoid meaningless memory allocation. -- This message was sent by Atlassian Jira (v8.3.4#803005)