[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844903#comment-17844903 ] Rui Fan commented on FLINK-35040: - I attached 3 flamegraphs here, these flamegraphs are generated on flink benchmark server, many thanks to [~Zakelly] for the help. >From these flamegraphs, we cannot find any org.apache.commons.io code is >called (I search commons in flamegraph). But the performance regresses after >commons.io is upgraded. It's a little strange. cc [~Weijie Guo] [~rmetzger] [~uce] > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Labels: pull-request-available > Attachments: image-2024-04-08-10-51-07-403.png, > image-2024-04-11-12-53-53-353.png, result1.html, result2.html, result3.html, > screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837041#comment-17837041 ] Rui Fan commented on FLINK-35040: - Thanks [~slfan1989] and [~rmetzger] for the comment! I didn't find any related issue from commons-io JIRA[1]. Also, I ran benchmark on my Mac with jdk11, and try to analyze why code path causes this regression. I use async-profiler wall mode to analyze the benchmark, and didn't find any code from commons.io package. Do you have any idea to troubleshooting? [1]https://issues.apache.org/jira/browse/IO-855?jql=project%20%3D%20IO%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened) > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Labels: pull-request-available > Attachments: image-2024-04-08-10-51-07-403.png, > image-2024-04-11-12-53-53-353.png, screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836843#comment-17836843 ] Robert Metzger commented on FLINK-35040: Do we know what causes the performance degradation with commons-io? Maybe there's a ticket in the commons-io project that helps us understand what is going on? If not, it might make sense to report to commons-io, so that they are aware that they have a performance degradation. > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Labels: pull-request-available > Attachments: image-2024-04-08-10-51-07-403.png, > image-2024-04-11-12-53-53-353.png, screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836099#comment-17836099 ] Shilun Fan commented on FLINK-35040: [~fanrui] During the compilation process, we found that commons-compress requires a higher version of commons-io, otherwise there will be a class not found exception. > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Labels: pull-request-available > Attachments: image-2024-04-08-10-51-07-403.png, > image-2024-04-11-12-53-53-353.png, screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[
https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836091#comment-17836091
]
Rui Fan commented on FLINK-35040:
-
Hi [~slfan1989] , thanks for your quick feedback!
FLINK-34955 wants to fix CVE issues of {{{}common-compress{}}}, but it upgrades
the {{commons-io}} together. I try to revert {{commons-io}} to 2.11.0, and the
performance is recovered.
My question is why do you upgrade the commons-io in FLINK-34955, and I didn't
see any vulnerabilities for commons-io. Could I revert {{commons-io}} to 2.11.0?
> The performance of serializerHeavyString regresses since April 3
>
>
> Key: FLINK-35040
> URL: https://issues.apache.org/jira/browse/FLINK-35040
> Project: Flink
> Issue Type: Bug
> Components: Benchmarks
>Affects Versions: 1.20.0
>Reporter: Rui Fan
>Assignee: Rui Fan
>Priority: Blocker
> Labels: pull-request-available
> Attachments: image-2024-04-08-10-51-07-403.png,
> image-2024-04-11-12-53-53-353.png, screenshot-1.png
>
>
> The performance of serializerHeavyString regresses since April 3, and had not
> yet recovered on April 8th.
> It seems Java 11 regresses, and Java 8 and Java 17 are fine.
> http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836004#comment-17836004 ] Shilun Fan commented on FLINK-35040: [~fanrui] I received your message, and I apologize for any inconvenience caused. The reason for upgrading commons-compress to version 1.26.0 is indeed due to the known CVE issues in commons-compress 1.24.0. We can refer to the following link: [https://mvnrepository.com/artifact/org.apache.commons/commons-compress/1.24.0] Direct vulnerabilities: CVE-2024-26308 CVE-2024-25710 Addressing the CVE issue is indeed necessary to prevent vulnerabilities in our system. I suggest we consider upgrading to version 1.26.1 to address this problem. Reverting to 1.24.0 might only serve as a temporary solution. > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Attachments: image-2024-04-08-10-51-07-403.png, > image-2024-04-11-12-53-53-353.png, screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836005#comment-17836005 ] Shilun Fan commented on FLINK-35040: [~fanrui] [~Weijie Guo] How is serializerHeavyString tested? I haven't seen any testing scripts here. Could you provide some hints? > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Attachments: image-2024-04-08-10-51-07-403.png, > image-2024-04-11-12-53-53-353.png, screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[
https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836000#comment-17836000
]
Rui Fan commented on FLINK-35040:
-
{quote}Can this result be consistently reproduced?
{quote}
Yes, it can be reproduced in flink benchmark server.
{quote}This link require username and password. Would you mind sharing the
screenshot :)
{quote}
Following is the result, and the corresponding code branch is :
[https://github.com/1996fanrui/flink/commits/revert-163b9cca6d2ccac0ff89dd985e3232667ddfb14f/]
The branch is based on flink master branch, and revert the
163b9cca6d2ccac0ff89dd985e3232667ddfb14f.
!image-2024-04-11-12-53-53-353.png!
> The performance of serializerHeavyString regresses since April 3
>
>
> Key: FLINK-35040
> URL: https://issues.apache.org/jira/browse/FLINK-35040
> Project: Flink
> Issue Type: Bug
> Components: Benchmarks
>Affects Versions: 1.20.0
>Reporter: Rui Fan
>Assignee: Rui Fan
>Priority: Blocker
> Attachments: image-2024-04-08-10-51-07-403.png,
> image-2024-04-11-12-53-53-353.png, screenshot-1.png
>
>
> The performance of serializerHeavyString regresses since April 3, and had not
> yet recovered on April 8th.
> It seems Java 11 regresses, and Java 8 and Java 17 are fine.
> http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836001#comment-17836001 ] Rui Fan commented on FLINK-35040: - Hi [~slfan1989] , would you mind helping take a look as well? thanks in advanced. > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Attachments: image-2024-04-08-10-51-07-403.png, > image-2024-04-11-12-53-53-353.png, screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835990#comment-17835990 ] Weijie Guo commented on FLINK-35040: > http://jenkins.flink-speed.xyz/job/flink-benchmark-request/71/ This link require username and password :) > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Attachments: image-2024-04-08-10-51-07-403.png, screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835959#comment-17835959 ] Rui Fan commented on FLINK-35040: - Also, I revert [163b9cca6d2ccac0ff89dd985e3232667ddfb14f|https://github.com/apache/flink/commit/163b9cca6d2ccac0ff89dd985e3232667ddfb14f] in the latest master branch, and the benchmark result is recovered to 155.761059 ops/ms. See: http://jenkins.flink-speed.xyz/job/flink-benchmark-request/71/ > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Attachments: image-2024-04-08-10-51-07-403.png, screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[ https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835957#comment-17835957 ] Weijie Guo commented on FLINK-35040: > The performance is regressed with commit > 163b9cca6d2ccac0ff89dd985e3232667ddfb14f, and the performance is fine without > it. Can this result be consistently reproduced? > The performance of serializerHeavyString regresses since April 3 > > > Key: FLINK-35040 > URL: https://issues.apache.org/jira/browse/FLINK-35040 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.20.0 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Blocker > Attachments: image-2024-04-08-10-51-07-403.png, screenshot-1.png > > > The performance of serializerHeavyString regresses since April 3, and had not > yet recovered on April 8th. > It seems Java 11 regresses, and Java 8 and Java 17 are fine. > http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200 > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-35040) The performance of serializerHeavyString regresses since April 3
[
https://issues.apache.org/jira/browse/FLINK-35040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17835662#comment-17835662
]
Rui Fan commented on FLINK-35040:
-
After running a series of benchmarks, I found FLINK-34955 causes the
performance regression.
The commit of FLINK-34955 is :
[163b9cca6d2ccac0ff89dd985e3232667ddfb14f|https://github.com/apache/flink/commit/163b9cca6d2ccac0ff89dd985e3232667ddfb14f]
, and its previous commit is
[8a18b119c958568d58f346c0e0868784a0ab9653|https://github.com/apache/flink/commit/8a18b119c958568d58f346c0e0868784a0ab9653].
I ran serializerHeavyString in the flink community benchmark jenkins for these
2 commits, the following is the result:
{code:java}
163b9cca6d2ccac0ff89dd985e3232667ddfb14f:
The First round: 129.695712 ops/ms , link:
http://jenkins.flink-speed.xyz/job/flink-benchmark-request/65/
The Second round: 134.81 ops/ms , link:
http://jenkins.flink-speed.xyz/job/flink-benchmark-request/67/
"org.apache.flink.benchmark.SerializationFrameworkMiniBenchmarks.serializerHeavyString","thrpt",1,30,129.695712,4.497077,"ops/ms"
"org.apache.flink.benchmark.SerializationFrameworkMiniBenchmarks.serializerHeavyString","thrpt",1,30,134.81,0.685663,"ops/ms"
8a18b119c958568d58f346c0e0868784a0ab9653:
The First round: 156.328850 ops/ms , link:
http://jenkins.flink-speed.xyz/job/flink-benchmark-request/66/
The Second round: 154.229828 ops/ms , link:
http://jenkins.flink-speed.xyz/job/flink-benchmark-request/68/
"org.apache.flink.benchmark.SerializationFrameworkMiniBenchmarks.serializerHeavyString","thrpt",1,30,156.328850,1.880390,"ops/ms"
"org.apache.flink.benchmark.SerializationFrameworkMiniBenchmarks.serializerHeavyString","thrpt",1,30,154.229828,3.453092,"ops/ms"
{code}
The performance is regressed with commit
163b9cca6d2ccac0ff89dd985e3232667ddfb14f, and the performance is fine without
it. So I think FLINK-34955 causes the performance regression.
The jdk8 and jdk17 are fine, and my test server with jdk11 doesn't have any
regression. So I'm not sure if it's an issue.
> The performance of serializerHeavyString regresses since April 3
>
>
> Key: FLINK-35040
> URL: https://issues.apache.org/jira/browse/FLINK-35040
> Project: Flink
> Issue Type: Bug
> Components: Benchmarks
>Affects Versions: 1.20.0
>Reporter: Rui Fan
>Assignee: Rui Fan
>Priority: Blocker
> Attachments: image-2024-04-08-10-51-07-403.png, screenshot-1.png
>
>
> The performance of serializerHeavyString regresses since April 3, and had not
> yet recovered on April 8th.
> It seems Java 11 regresses, and Java 8 and Java 17 are fine.
> http://flink-speed.xyz/timeline/#/?exe=1,6,12=serializerHeavyString=on=on=off=3=200
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
