[jira] [Commented] (FLINK-20044) Disposal of RocksDB could last forever

2021-11-01 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436634#comment-17436634 ] Jiayi Liao commented on FLINK-20044: [~liyu] We haven't upgrade our Flink version recently. But I

[jira] [Issue Comment Deleted] (FLINK-22672) Some enhancements for pluggable shuffle service framework

2021-09-06 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-22672: --- Comment: was deleted (was: Hi [~maguowei], [~jinxing6...@126.com], [~kevin.cyj], Currently we're

[jira] [Updated] (FLINK-24148) Add bloom filter policy option in RocksDBConfiguredOptions

2021-09-03 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-24148: --- Description: Bloom filter can efficiently enhance the read on RocksDB, especially for the reading

[jira] [Commented] (FLINK-24148) Add bloom filter policy option in RocksDBConfiguredOptions

2021-09-03 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17409412#comment-17409412 ] Jiayi Liao commented on FLINK-24148: [~yunta] Hi, could you have a look on this ? I can prepare a PR

[jira] [Created] (FLINK-24148) Add bloom filter policy option in RocksDBConfiguredOptions

2021-09-03 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-24148: -- Summary: Add bloom filter policy option in RocksDBConfiguredOptions Key: FLINK-24148 URL: https://issues.apache.org/jira/browse/FLINK-24148 Project: Flink Issue

[jira] [Commented] (FLINK-23180) Initialize checkpoint location lazily in DataStream Batch Jobs

2021-09-03 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17409403#comment-17409403 ] Jiayi Liao commented on FLINK-23180: [~pnowojski] I can work on this. Okay, we can skip the

[jira] [Updated] (FLINK-20684) Add compression type option in RocksDBConfigurableOptions

2021-09-03 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-20684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-20684: --- Labels: (was: auto-deprioritized-major) > Add compression type option in

[jira] [Commented] (FLINK-23180) Initialize checkpoint location lazily in DataStream Batch Jobs

2021-09-03 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17409368#comment-17409368 ] Jiayi Liao commented on FLINK-23180: [~yunta][~pnowojski] I think this might be a very common

[jira] [Updated] (FLINK-23180) Initialize checkpoint location lazily in DataStream Batch Jobs

2021-09-03 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-23180: --- Description: Currently batch jobs will initialize checkpoint location eagerly when

[jira] [Commented] (FLINK-24120) The memory will rise a little after each savepoint ends

2021-09-02 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17409257#comment-17409257 ] Jiayi Liao commented on FLINK-24120: [~trohrmann] Okay, will add this today. > The memory will

[jira] [Commented] (FLINK-24120) The memory will rise a little after each savepoint ends

2021-09-01 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408497#comment-17408497 ] Jiayi Liao commented on FLINK-24120: Try add an environment variable for taskmanagers:

[jira] [Commented] (FLINK-22672) Some enhancements for pluggable shuffle service framework

2021-09-01 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407870#comment-17407870 ] Jiayi Liao commented on FLINK-22672: Hi [~maguowei], [~jinxing6...@126.com], [~kevin.cyj],

[jira] [Comment Edited] (FLINK-22672) Some enhancements for pluggable shuffle service framework

2021-09-01 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407870#comment-17407870 ] Jiayi Liao edited comment on FLINK-22672 at 9/1/21, 6:40 AM: - Hi

[jira] [Created] (FLINK-23690) Processing timers can be triggered more efficiently

2021-08-09 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-23690: -- Summary: Processing timers can be triggered more efficiently Key: FLINK-23690 URL: https://issues.apache.org/jira/browse/FLINK-23690 Project: Flink Issue Type:

[jira] [Updated] (FLINK-23689) TestProcessingTimeService.setCurrentTime(long) not delay the firing timer by 1ms delay

2021-08-09 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-23689: --- Description: FLINK-9857 enabled {{SystemProcessingTimeService}} to fire processing timers by 1ms

[jira] [Created] (FLINK-23689) TestProcessingTimeService.setCurrentTime(long) not delay the firing timer by 1ms delay

2021-08-09 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-23689: -- Summary: TestProcessingTimeService.setCurrentTime(long) not delay the firing timer by 1ms delay Key: FLINK-23689 URL: https://issues.apache.org/jira/browse/FLINK-23689

[jira] [Commented] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-08-04 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17393563#comment-17393563 ] Jiayi Liao commented on FLINK-23208: [~pnowojski] Yes. It's in-progress, I'm pretty busy these days

[jira] [Comment Edited] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-21 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385255#comment-17385255 ] Jiayi Liao edited comment on FLINK-23208 at 7/22/21, 5:35 AM: -- [~pnowojski]

[jira] [Commented] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-21 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17385255#comment-17385255 ] Jiayi Liao commented on FLINK-23208: [~pnowojski] Before the change, we need to fix the bug in

[jira] [Commented] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-14 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380948#comment-17380948 ] Jiayi Liao commented on FLINK-23208: [~pnowojski] I can work on this. Should the benchmark be

[jira] [Commented] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-14 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380670#comment-17380670 ] Jiayi Liao commented on FLINK-23208: [~pnowojski] You mean something like this ? {code:java}

[jira] [Commented] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-13 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379991#comment-17379991 ] Jiayi Liao commented on FLINK-23208: [~pnowojski] Exactly! > the first/next trigger call will be

[jira] [Comment Edited] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-13 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379901#comment-17379901 ] Jiayi Liao edited comment on FLINK-23208 at 7/13/21, 1:58 PM: -- [~pnowojski]

[jira] [Commented] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-13 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379901#comment-17379901 ] Jiayi Liao commented on FLINK-23208: [~pnowojski] >Why registering and firing timers is

[jira] [Comment Edited] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-13 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379810#comment-17379810 ] Jiayi Liao edited comment on FLINK-23208 at 7/13/21, 10:31 AM: ---

[jira] [Commented] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-13 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379810#comment-17379810 ] Jiayi Liao commented on FLINK-23208: [~pnowojski], sorry for making this unclear, it's really a bit

[jira] [Commented] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-12 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379237#comment-17379237 ] Jiayi Liao commented on FLINK-23208: [~pnowojski] Yes. I've read FLINK-9857 before, IIUC FLINK-9857

[jira] [Updated] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-04 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-23208: --- Labels: critical (was: ) > Late processing timers need to wait 1ms at least to be fired >

[jira] [Updated] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-04 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-23208: --- Affects Version/s: 1.14.0 1.11.3 1.13.0

[jira] [Updated] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-01 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-23208: --- Description: The problem is from the codes below: {code:java} public static long

[jira] [Updated] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-01 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-23208: --- Attachment: screenshot-1.png > Late processing timers need to wait 1ms at least to be fired >

[jira] [Updated] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-01 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-23208: --- Description: The problem is from the codes below: {code:java} public static long

[jira] [Updated] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-01 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-23208: --- Description: The problem is from the codes below: {code:java} public static long

[jira] [Updated] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-01 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-23208: --- Description: The problem is from the codes below: {code:java} public static long

[jira] [Updated] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-01 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-23208: --- Description: The problem is from the codes below: ``` public static long

[jira] [Created] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-01 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-23208: -- Summary: Late processing timers need to wait 1ms at least to be fired Key: FLINK-23208 URL: https://issues.apache.org/jira/browse/FLINK-23208 Project: Flink

[jira] [Updated] (FLINK-23208) Late processing timers need to wait 1ms at least to be fired

2021-07-01 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-23208: --- Priority: Critical (was: Major) > Late processing timers need to wait 1ms at least to be fired >

[jira] [Created] (FLINK-23180) Initialize checkpoint location lazily in DataStream Batch Jobs

2021-06-29 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-23180: -- Summary: Initialize checkpoint location lazily in DataStream Batch Jobs Key: FLINK-23180 URL: https://issues.apache.org/jira/browse/FLINK-23180 Project: Flink

[jira] [Commented] (FLINK-22887) Backlog based optimizations for RebalancePartitioner and RescalePartitioner

2021-06-24 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368803#comment-17368803 ] Jiayi Liao commented on FLINK-22887: [~akalashnikov] I've implemented this optimization in our inner

[jira] [Commented] (FLINK-22887) Backlog based optimizations for RebalancePartitioner and RescalePartitioner

2021-06-24 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368697#comment-17368697 ] Jiayi Liao commented on FLINK-22887: [~pnowojski] In my implementation, the optimized partitioner

[jira] [Commented] (FLINK-22887) Backlog based optimizations for RebalancePartitioner and RescalePartitioner

2021-06-21 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17366931#comment-17366931 ] Jiayi Liao commented on FLINK-22887: [~pnowojski][~akalashnikov] Thanks for the feedback. My idea

[jira] [Commented] (FLINK-23003) Resource leak in RocksIncrementalSnapshotStrategy

2021-06-15 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-23003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364018#comment-17364018 ] Jiayi Liao commented on FLINK-23003: [~Yanfei Lei] This looks like a duplicate of

[jira] [Commented] (FLINK-22886) Thread leak in RocksDBStateUploader

2021-06-06 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358338#comment-17358338 ] Jiayi Liao commented on FLINK-22886: [~yunta] Hi Yun, [~mayuehappy] is solving this issue in our

[jira] [Updated] (FLINK-22886) Thread leak in RocksDBStateUploader

2021-06-06 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-22886: --- Labels: critical (was: ) > Thread leak in RocksDBStateUploader >

[jira] [Created] (FLINK-22887) Backlog based optimizations for RebalancePartitioner and RescalePartitioner

2021-06-05 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-22887: -- Summary: Backlog based optimizations for RebalancePartitioner and RescalePartitioner Key: FLINK-22887 URL: https://issues.apache.org/jira/browse/FLINK-22887 Project:

[jira] [Updated] (FLINK-22886) Thread leak in RocksDBStateUploader

2021-06-05 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-22886: --- Description: {{ExecutorService}} in {{RocksDBStateUploader}} is not shut down, which may leak

[jira] [Updated] (FLINK-22886) Thread leak in RocksDBStateUploader

2021-06-05 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-22886: --- Description: {{ExecutorService}} in {{RocksDBStateUploader}} is not shut down, which may leak

[jira] [Updated] (FLINK-22886) Thread leak in RocksDBStateUploader

2021-06-05 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-22886: --- Attachment: image-2021-06-06-13-46-34-604.png > Thread leak in RocksDBStateUploader >

[jira] [Created] (FLINK-22886) Thread leak in RocksDBStateUploader

2021-06-05 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-22886: -- Summary: Thread leak in RocksDBStateUploader Key: FLINK-22886 URL: https://issues.apache.org/jira/browse/FLINK-22886 Project: Flink Issue Type: Bug

[jira] [Commented] (FLINK-22805) Dynamic configuration of Flink checkpoint interval

2021-05-30 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354188#comment-17354188 ] Jiayi Liao commented on FLINK-22805: This is a good point. But I think the root problem is that, the

[jira] [Commented] (FLINK-10644) Batch Job: Speculative execution

2021-04-29 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17337040#comment-17337040 ] Jiayi Liao commented on FLINK-10644: Yes. We also implement our own speculation in ByteDance. This

[jira] [Commented] (FLINK-15160) Clean up is not applied if there are no incoming events for a key.

2021-04-22 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17328809#comment-17328809 ] Jiayi Liao commented on FLINK-15160: We've met the problem recently. To solve this, we create a new

[jira] [Updated] (FLINK-19016) Checksum mismatch when restore from RocksDB

2021-04-22 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-19016: --- Labels: (was: stale-major) > Checksum mismatch when restore from RocksDB >

[jira] [Closed] (FLINK-18203) Reduce objects usage in redistributing union states

2021-04-22 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-18203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao closed FLINK-18203. -- Resolution: Won't Fix Close this because union state is going to be deprecated in the future. >

[jira] [Closed] (FLINK-15923) Remove DISCARDED in TaskAcknowledgeResult

2021-04-22 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao closed FLINK-15923. -- Resolution: Won't Fix > Remove DISCARDED in TaskAcknowledgeResult >

[jira] [Commented] (FLINK-18572) Flink web UI doesn't display tolerable-failed-checkpoints

2021-04-22 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-18572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17328188#comment-17328188 ] Jiayi Liao commented on FLINK-18572: +1 on this feature. > Flink web UI doesn't display

[jira] [Closed] (FLINK-17320) Java8 lambda expression cannot be serialized.

2021-04-22 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-17320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao closed FLINK-17320. -- Resolution: Won't Fix Close this since this is not a Flink related problem. > Java8 lambda

[jira] [Commented] (FLINK-19016) Checksum mismatch when restore from RocksDB

2021-04-22 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17328078#comment-17328078 ] Jiayi Liao commented on FLINK-19016: A quick update. We've met this issue again after turn

[jira] [Updated] (FLINK-19016) Checksum mismatch when restore from RocksDB

2021-04-22 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-19016: --- Affects Version/s: 1.13.1 1.13.0 1.12.2 > Checksum

[jira] [Closed] (FLINK-19011) Parallelize the restore operation in OperatorStateBackend

2021-04-22 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao closed FLINK-19011. -- Resolution: Won't Fix > Parallelize the restore operation in OperatorStateBackend >

[jira] [Commented] (FLINK-13247) Implement external shuffle service for YARN

2021-04-20 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325805#comment-17325805 ] Jiayi Liao commented on FLINK-13247: [~trohrmann][~zjwang] It's okay for me. I'll draft a desgin

[jira] [Commented] (FLINK-5601) Window operator does not checkpoint watermarks

2021-04-19 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324800#comment-17324800 ] Jiayi Liao commented on FLINK-5601: --- [~liyu] Yes. My idea is to add watermark as state into

[jira] [Commented] (FLINK-13247) Implement external shuffle service for YARN

2021-04-17 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324189#comment-17324189 ] Jiayi Liao commented on FLINK-13247: Actually we've already implemented Yarn shuffle service in our

[jira] [Commented] (FLINK-16193) Improve error messaging when a key is assigned to the wrong key group range

2021-04-16 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17323641#comment-17323641 ] Jiayi Liao commented on FLINK-16193: Maybe I'm a bit late for this discussion... I've met this issue

[jira] [Updated] (FLINK-21990) SourceStreamTask will always hang if the CheckpointedFunction#snapshotState throws an exception.

2021-03-28 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-21990: --- Priority: Critical (was: Major) > SourceStreamTask will always hang if the

[jira] [Commented] (FLINK-21990) SourceStreamTask will always hang if the CheckpointedFunction#snapshotState throws an exception.

2021-03-28 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17310341#comment-17310341 ] Jiayi Liao commented on FLINK-21990: [~pnowojski] We've encountered the problem when upgrading Flink

[jira] [Commented] (FLINK-21413) TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend

2021-02-22 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288266#comment-17288266 ] Jiayi Liao commented on FLINK-21413: [~liyu] Yes. I'll submit a PR soon. > TtlMapState and

[jira] [Commented] (FLINK-21413) TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend

2021-02-21 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288147#comment-17288147 ] Jiayi Liao commented on FLINK-21413: [~liyu] Oops... I didn't notice that the #remove(UK) method

[jira] [Commented] (FLINK-21413) TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend

2021-02-20 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287600#comment-17287600 ] Jiayi Liao commented on FLINK-21413: Sure. I'm still thinking about the solution. I've come up with

[jira] [Updated] (FLINK-21413) TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend

2021-02-18 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-21413: --- Description: Take the #TtlMapState as an example,   {code:java} public Map>

[jira] [Updated] (FLINK-21413) TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend

2021-02-18 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-21413: --- Attachment: image-2021-02-19-11-13-58-672.png > TtlMapState and TtlListState cannot be clean

[jira] [Created] (FLINK-21413) TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend

2021-02-18 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-21413: -- Summary: TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend Key: FLINK-21413 URL: https://issues.apache.org/jira/browse/FLINK-21413

[jira] [Commented] (FLINK-21321) Change RocksDB incremental checkpoint re-scaling to use deleteRange

2021-02-08 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281075#comment-17281075 ] Jiayi Liao commented on FLINK-21321: [~legojoey17] We've met the same problem recently. The

[jira] [Updated] (FLINK-20684) Add compression type option in RocksDBConfigurableOptions

2020-12-20 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-20684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-20684: --- Description: Currently RocksDB uses snappy compression by default, but this could be a huge cost

[jira] [Created] (FLINK-20684) Add compression type option in RocksDBConfigurableOptions

2020-12-20 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-20684: -- Summary: Add compression type option in RocksDBConfigurableOptions Key: FLINK-20684 URL: https://issues.apache.org/jira/browse/FLINK-20684 Project: Flink Issue

[jira] [Closed] (FLINK-19150) Behaviour change after migration from 1.9 to 1.11

2020-12-17 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao closed FLINK-19150. -- Resolution: Invalid > Behaviour change after migration from 1.9 to 1.11 >

[jira] [Updated] (FLINK-16597) Handle exception thrown from close method in OutputFormat

2020-12-11 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-16597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-16597: --- Priority: Minor (was: Major) > Handle exception thrown from close method in OutputFormat >

[jira] [Commented] (FLINK-17571) A better way to show the files used in currently checkpoints

2020-12-07 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-17571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245142#comment-17245142 ] Jiayi Liao commented on FLINK-17571: [~klion26] Is it necessary to list the checkpoint file in JM by

[jira] [Commented] (FLINK-19970) State leak in CEP Operators (expired events/keys not removed from state)

2020-12-02 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242914#comment-17242914 ] Jiayi Liao commented on FLINK-19970: [~dwysakowicz] Does this bug affect all the patterns? > State

[jira] [Commented] (FLINK-19970) State leak in CEP Operators (expired events/keys not removed from state)

2020-12-01 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241426#comment-17241426 ] Jiayi Liao commented on FLINK-19970: [~Jamalarm]  Do you figure out the root cause of the state

[jira] [Closed] (FLINK-20269) The flush happens too frequent in SavepoinV2Serializer

2020-11-22 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-20269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao closed FLINK-20269. -- Resolution: Duplicate > The flush happens too frequent in SavepoinV2Serializer >

[jira] [Commented] (FLINK-20269) The flush happens too frequent in SavepoinV2Serializer

2020-11-22 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-20269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236888#comment-17236888 ] Jiayi Liao commented on FLINK-20269: [~liyu] Thanks for the remind. I'll close this due to

[jira] [Updated] (FLINK-20269) The flush happens too frequent in SavepoinV2Serializer

2020-11-21 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-20269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-20269: --- Description: The reason I notice this is, I find that the metadata's persistence can be very slow 

[jira] [Created] (FLINK-20269) The flush happens too frequent in SavepoinV2Serializer

2020-11-21 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-20269: -- Summary: The flush happens too frequent in SavepoinV2Serializer Key: FLINK-20269 URL: https://issues.apache.org/jira/browse/FLINK-20269 Project: Flink Issue

[jira] [Closed] (FLINK-6984) Do not recover CompletedCheckpointStore on every restore

2020-11-18 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao closed FLINK-6984. - Resolution: Duplicate Resoloved in FLINK-19596. > Do not recover CompletedCheckpointStore on every

[jira] [Updated] (FLINK-6984) Do not recover CompletedCheckpointStore on every restore

2020-11-18 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-6984: -- Fix Version/s: 1.12.0 > Do not recover CompletedCheckpointStore on every restore >

[jira] [Commented] (FLINK-16753) Exception from AsyncCheckpointRunnable should be wrapped in CheckpointException

2020-11-13 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-16753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231938#comment-17231938 ] Jiayi Liao commented on FLINK-16753: [~zjwang] [~NicoK] The change is simple. I've submitted two

[jira] [Commented] (FLINK-20099) HeapStateBackend checkpoint error hidden under cryptic message

2020-11-12 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-20099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17230469#comment-17230469 ] Jiayi Liao commented on FLINK-20099: Is this a duplicate issue of 

[jira] [Created] (FLINK-20087) CheckpointCoordinator waits until all tasks finish initialization of states to trigger checkpoint

2020-11-11 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-20087: -- Summary: CheckpointCoordinator waits until all tasks finish initialization of states to trigger checkpoint Key: FLINK-20087 URL: https://issues.apache.org/jira/browse/FLINK-20087

[jira] [Commented] (FLINK-20044) Disposal of RocksDB could last forever

2020-11-08 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17228315#comment-17228315 ] Jiayi Liao commented on FLINK-20044: [~sewen] I guess this is not a cancellation situation here

[jira] [Created] (FLINK-20044) Disposal of RocksDB could last forever

2020-11-07 Thread Jiayi Liao (Jira)
Jiayi Liao created FLINK-20044: -- Summary: Disposal of RocksDB could last forever Key: FLINK-20044 URL: https://issues.apache.org/jira/browse/FLINK-20044 Project: Flink Issue Type: Bug

[jira] [Commented] (FLINK-19778) Failed job reinitiated with wrong checkpoint after a ZK reconnection

2020-10-23 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219520#comment-17219520 ] Jiayi Liao commented on FLINK-19778: In the JM log, the point is why zk store cannot read

[jira] [Closed] (FLINK-19596) Do not recover CompletedCheckpointStore on each failover

2020-10-22 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao closed FLINK-19596. -- Resolution: Fixed Fixed in https://issues.apache.org/jira/browse/FLINK-19401 > Do not recover

[jira] [Commented] (FLINK-19594) SubTasks start index don't unified and may confuse users

2020-10-20 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217372#comment-17217372 ] Jiayi Liao commented on FLINK-19594: Link an duplicated issue reported earlier. 

[jira] [Commented] (FLINK-16051) Subtask ID in Overview-Subtasks should start from 1

2020-10-20 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-16051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217369#comment-17217369 ] Jiayi Liao commented on FLINK-16051: [~zjwang] This is reported by a user recently, I've linked the

[jira] [Closed] (FLINK-10284) TumblingEventTimeWindows's offset should can be less than zero.

2020-10-20 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-10284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao closed FLINK-10284. -- Resolution: Fixed Close this since this is already fixed in another JIRA issue. >

[jira] [Commented] (FLINK-19596) Do not recover CompletedCheckpointStore on each failover

2020-10-19 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216590#comment-17216590 ] Jiayi Liao commented on FLINK-19596: [~roman_khachatryan] In the losing-regaining leadership case,

[jira] [Commented] (FLINK-19596) Do not recover CompletedCheckpointStore on each failover

2020-10-19 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216541#comment-17216541 ] Jiayi Liao commented on FLINK-19596: [~yunta] I didn't know there's already a JIRA before. Thanks

[jira] [Comment Edited] (FLINK-5463) RocksDB.disposeInternal does not react to interrupts, blocks task cancellation

2020-10-15 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17214604#comment-17214604 ] Jiayi Liao edited comment on FLINK-5463 at 10/15/20, 10:51 AM: --- Reopen this

[jira] [Updated] (FLINK-5463) RocksDB.disposeInternal does not react to interrupts, blocks task cancellation

2020-10-15 Thread Jiayi Liao (Jira)
[ https://issues.apache.org/jira/browse/FLINK-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-5463: -- Affects Version/s: 1.9.0 > RocksDB.disposeInternal does not react to interrupts, blocks task

  1   2   3   4   5   >