[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510872#comment-16510872 ] swy commented on FLINK-9506: no problem an email also sent to ML. > Flink ReducingState.add causing more

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510870#comment-16510870 ] Stefan Richter commented on FLINK-9506: --- [~yow] I would suggest that you discuss it on the user

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510864#comment-16510864 ] swy commented on FLINK-9506: Hi [~sihuazhou] I just sent an email to your personal mail box. Let's continue

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510828#comment-16510828 ] Sihua Zhou commented on FLINK-9506: --- Hi [~yow], I didn't see the email you sent yet, but I just had a

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510789#comment-16510789 ] swy commented on FLINK-9506: Hi [~sihuazhou] an email is sent to "u...@flink.apache.org" with title "Flink

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510757#comment-16510757 ] Sihua Zhou commented on FLINK-9506: --- Hi, I think you could send your question to the

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510752#comment-16510752 ] swy commented on FLINK-9506: Alright i am ok to close the ticket. Do you mind to share me the information on

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510750#comment-16510750 ] Sihua Zhou commented on FLINK-9506: --- I'm + 1 to close this ticket and move to ML, we can definitely

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510747#comment-16510747 ] Stefan Richter commented on FLINK-9506: --- Hi, I think this discussion has no connection with this

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510746#comment-16510746 ] swy commented on FLINK-9506: Hi [~sihuazhou] it surprise me as well. Please advice if something wrong in the

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510716#comment-16510716 ] Sihua Zhou commented on FLINK-9506: --- Hi [~yow] "If we further comment out recordStore.add() then

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510715#comment-16510715 ] swy commented on FLINK-9506: Hi [~sihuazhou] please download source from

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510704#comment-16510704 ] swy commented on FLINK-9506: Hi [~sihuazhou] the picture is captured without any line being commented-out.

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510649#comment-16510649 ] Sihua Zhou commented on FLINK-9506: --- Additional, do you make sure that you are using the RocksDb

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510647#comment-16510647 ] Sihua Zhou commented on FLINK-9506: --- Hi [~yow] in the input_stop_when_timer_run.png, does the yellow

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510632#comment-16510632 ] swy commented on FLINK-9506: Hi [~sihuazhou] to your questions, 1. Checkpoint is disable at the moment. But

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510580#comment-16510580 ] Sihua Zhou commented on FLINK-9506: --- Hi [~yow] I don't think this a limitation in Flink, we have more

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510575#comment-16510575 ] swy commented on FLINK-9506: Hi [~sihuazhou] you are right. Fluctuation only happened when "keyby" with

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509818#comment-16509818 ] Sihua Zhou commented on FLINK-9506: --- Hi [~yow] What do you means by the empty ProcessAggregation? Did

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509808#comment-16509808 ] swy commented on FLINK-9506: Hi [~sihuazhou] I hope to close the ticket too but the problem still persists

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507815#comment-16507815 ] Sihua Zhou commented on FLINK-9506: --- Hi [~yow] I think we can close this ticket now, do you agree?

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504820#comment-16504820 ] swy commented on FLINK-9506: thank for response [~sihuazhou], the key length is around 50 chars. We will

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504729#comment-16504729 ] Sihua Zhou commented on FLINK-9506: --- Hi [~yow] From the top of my head, I list answers here: - >> 1.

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504675#comment-16504675 ] swy commented on FLINK-9506: [~sihuazhou] your idea is brilliant, but the first test result is not show too

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503505#comment-16503505 ] Sihua Zhou commented on FLINK-9506: --- [~yow] Maybe there is one more optimization that could have a try,

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503426#comment-16503426 ] swy commented on FLINK-9506: [~kkrugler] please don't close the ticket yet. Because the performance

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503401#comment-16503401 ] Ken Krugler commented on FLINK-9506: @swy - the questions about setting up RocksDB, and your

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503284#comment-16503284 ] swy commented on FLINK-9506: [~srichter] Thanks for tips, after implement Rocksdb the performance seems much

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500211#comment-16500211 ] swy commented on FLINK-9506: [~srichter] Really appreciated your help. To your points, *[First, when using

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500165#comment-16500165 ] Stefan Richter commented on FLINK-9506: --- [~yow] I had another look at your code and can point out a

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499972#comment-16499972 ] swy commented on FLINK-9506: [~srichter] thanks for good explanation. We have no choice but need to store

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499947#comment-16499947 ] Stefan Richter commented on FLINK-9506: --- >From what I can see, the problem is purely related to

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499873#comment-16499873 ] Sihua Zhou commented on FLINK-9506: --- [~yow] I think RocksDB backend could give a more stable

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499869#comment-16499869 ] swy commented on FLINK-9506: Any workaround would be appreciated, do you think using rocksdb help in such

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499867#comment-16499867 ] Sihua Zhou commented on FLINK-9506: --- Thanks for trying it out. Then I think the performance drop and

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499859#comment-16499859 ] swy commented on FLINK-9506: [~sihuazhou] Your tricks quite promising as the performance has been improved

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499672#comment-16499672 ] Sihua Zhou commented on FLINK-9506: --- Hi [~yow] could you please just replace the getKey() as follow and

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499667#comment-16499667 ] swy commented on FLINK-9506: Thanks for pointing out, this is a mistake in the sample. In real application

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499560#comment-16499560 ] Ken Krugler commented on FLINK-9506: Why are you creating a new {{AggregrationFunction}} every time

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499445#comment-16499445 ] swy commented on FLINK-9506: what we want to know is: Is this something expected or something wrong in our

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499412#comment-16499412 ] swy commented on FLINK-9506: Hi [~sihuazhou] the keyby is comes from one of the private member in POJO.

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499395#comment-16499395 ] Sihua Zhou commented on FLINK-9506: --- Hi [~yow] could you please give some information of the `keyBy()`?

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499393#comment-16499393 ] swy commented on FLINK-9506: Hi Fabian, 1. Yes. 2. Yes. The state is suppose to store in Task Manager's

[jira] [Commented] (FLINK-9506) Flink ReducingState.add causing more than 100% performance drop

[ https://issues.apache.org/jira/browse/FLINK-9506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499356#comment-16499356 ] Fabian Hueske commented on FLINK-9506: -- Thanks for opening this issue. I have a few questions: 1)