[
https://issues.apache.org/jira/browse/FLINK-15103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992352#comment-16992352
]
Jingsong Lee edited comment on FLINK-15103 at 12/10/19 9:57 AM:
----------------------------------------------------------------
[~xintongsong] , You reminded me to test small batch sql jobs. We also
encounter the cost of over allocate memory, that lead to performance regression
of small batch sql jobs after FLINK-14063 .
* Allocate memory will have the cost of memory management algorithm.
* Allocate memory will have the cost of memory initialization, will set all
memory to zero. And this initialization will require the operating system to
actually allocate physical memory.
* Over allocate memory will squash the file cache too.
We can optimize the operator algorithm, apply lazy allocation, and avoid
meaningless memory allocation. Created FLINK-15172 to continue to do benchmark
and improve.
was (Author: lzljs3620320):
[~xintongsong] , You remain me to test small batch sql jobs. We also encounter
the cost of over allocate memory, that lead to performance regression of small
batch sql jobs after FLINK-14063 .
* Allocate memory will have the cost of memory management algorithm.
* Allocate memory will have the cost of memory initialization, will set all
memory to zero. And this initialization will require the operating system to
actually allocate physical memory.
* Over allocate memory will squash the file cache too.
We can optimize the operator algorithm, apply lazy allocation, and avoid
meaningless memory allocation. Created FLINK-15172 to continue to do benchmark
and improve.
> Performance regression on 3.12.2019 in various benchmarks
> ---------------------------------------------------------
>
> Key: FLINK-15103
> URL: https://issues.apache.org/jira/browse/FLINK-15103
> Project: Flink
> Issue Type: Bug
> Components: Benchmarks
> Reporter: Piotr Nowojski
> Priority: Blocker
> Fix For: 1.10.0
>
>
> Various benchmarks show a performance regression that happened on December
> 3rd:
> [arrayKeyBy (probably the most easily
> visible)|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=arrayKeyBy&env=2&revs=200&equid=off&quarts=on&extr=on]
>
> [tupleKeyBy|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=tupleKeyBy&env=2&revs=200&equid=off&quarts=on&extr=on]
>
> [twoInputMapSink|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=twoInputMapSink&env=2&revs=200&equid=off&quarts=on&extr=on]
> [globalWindow (small
> one)|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=globalWindow&env=2&revs=200&equid=off&quarts=on&extr=on]
> and possible others.
> Probably somewhere between those commits: -8403fd4- 2d67ee0..60b3f2f
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
