[
https://issues.apache.org/jira/browse/FLINK-15103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16990039#comment-16990039
]
Xintong Song edited comment on FLINK-15103 at 12/6/19 6:10 PM:
---------------------------------------------------------------
bq. I was doing the git bisect locally and I'm almost sure that this commit
4b8ed643a4d85c9440a8adbc0798b8a4bbd9520b is the first one with the regression
This is exactly the commit I mentioned that decreases heap size, by making
managed memory always off-heap.
This behavior change is first discussed and agreed offline by [~sewen],
[~ykt836], [~liyu], [~zhuzh] and me, then posted and discussed online in this
[thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Make-Managed-Memory-always-off-heap-Adjustment-to-FLIP-49-td35365.html#none].
The purpose of this change is to trade off between out of box experiences of
streaming with heap state backend, streaming with rocks state backend, and
batch.
I'm not sure what is the assumption / precondition for the performance
benchmarks. Do we assume always use the default out of box resource
configurations? Or we assume memory configurations are well tuned (of course
the total memory should stay unchanged) to fit the use case? If it's the
latter, we can simply set the managed memory size / fraction to 0 for these
cases (as long as they are streaming jobs that use heap state backend) to have
similar heap size as before.
was (Author: xintongsong):
bq. I was doing the git bisect locally and I'm almost sure that this commit
4b8ed643a4d85c9440a8adbc0798b8a4bbd9520b is the first one with the regression
This is exactly the commit I mentioned that decreases heap size, by making
managed memory always off-heap.
This behavior change is first discussed and agreed offline by [~sewen],
[~ykt836], [~liyu], [~zhuzh] and me, then posted and discussed online in this
[thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Make-Managed-Memory-always-off-heap-Adjustment-to-FLIP-49-td35365.html#none].
The purpose of this change is to trade off between out of box experience for
streaming with heap state backend, streaming with rocks state backend, and
batch.
I'm not sure what is the assumption / precondition for the performance
benchmarks. Do we assume always use the default out of box resource
configurations? Or we assume memory configurations are well tuned (of course
the total memory should stay unchanged) to fit the use case? If it's the
latter, we can simply set the managed memory size / fraction to 0 for these
cases (as long as they are streaming jobs that use heap state backend) to have
similar heap size as before.
> Performance regression on 3.12.2019 in various benchmarks
> ---------------------------------------------------------
>
> Key: FLINK-15103
> URL: https://issues.apache.org/jira/browse/FLINK-15103
> Project: Flink
> Issue Type: Bug
> Components: Benchmarks
> Reporter: Piotr Nowojski
> Priority: Blocker
> Fix For: 1.10.0
>
>
> Various benchmarks show a performance regression that happened on December
> 3rd:
> [arrayKeyBy (probably the most easily
> visible)|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=arrayKeyBy&env=2&revs=200&equid=off&quarts=on&extr=on]
>
> [tupleKeyBy|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=tupleKeyBy&env=2&revs=200&equid=off&quarts=on&extr=on]
>
> [twoInputMapSink|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=twoInputMapSink&env=2&revs=200&equid=off&quarts=on&extr=on]
> [globalWindow (small
> one)|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=globalWindow&env=2&revs=200&equid=off&quarts=on&extr=on]
> and possible others.
> Probably somewhere between those commits: -8403fd4- 2d67ee0..60b3f2f
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
