JingsongLi opened a new pull request #10797: [FLINK-15172][table-blink] Optimize the operator algorithm to lazily allocate memory URL: https://github.com/apache/flink/pull/10797 ## What is the purpose of the change Now after FLINK-14063 , operators will get all manage memory of TaskManager, The cost of over allocate memory is very high, lead to performance regression of small batch sql jobs: Allocate memory will have the cost of memory management algorithm. Allocate memory will have the cost of memory initialization, will set all memory to zero. And this initialization will require the operating system to actually allocate physical memory. Over allocate memory will squash the file cache too. We can optimize the operator algorithm, apply lazy allocation, and avoid meaningless memory allocation. ## Brief change log - Introduce LazyMemorySegmentPool - Improve operators to lazily allocate memory ## Verifying this change This change is already covered by existing tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): (no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no - The S3 file system connector: (no ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? JavaDocs
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
