[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-08-08 Thread Yue Ma (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752055#comment-17752055 ] Yue Ma commented on FLINK-31238: [~pnowojski] [~srichter] [~yunta] [~masteryhx]  Sorry for the late

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-08-02 Thread Hangxiang Yu (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750127#comment-17750127 ] Hangxiang Yu commented on FLINK-31238: -- [~mayuehappy] Thanks a lot for your efforts! If the pr/FLIP

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-07-22 Thread Yue Ma (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17745875#comment-17745875 ] Yue Ma commented on FLINK-31238: [~yunta]  ok , sorry for the delay:( > Use IngestDB to speed up

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-07-22 Thread Yue Ma (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17745874#comment-17745874 ] Yue Ma commented on FLINK-31238: [~pnowojski] Sorry for missing your comment. The newly added api in

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-07-17 Thread Yun Tang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743700#comment-17743700 ] Yun Tang commented on FLINK-31238: -- Since the feature freeze date for flink-1.18 is July 24th, and no

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-07-07 Thread Piotr Nowojski (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17740998#comment-17740998 ] Piotr Nowojski commented on FLINK-31238: [~mayuehappy] thanks for your work on this! Really nice

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-06-25 Thread Yue Ma (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736879#comment-17736879 ] Yue Ma commented on FLINK-31238: [~srichter]  Thanks very much for the reply, I hope to continue this

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-06-19 Thread Stefan Richter (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17734183#comment-17734183 ] Stefan Richter commented on FLINK-31238: [~mayuehappy] Thanks for working on this topic! I saw

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-03-29 Thread Yue Ma (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706254#comment-17706254 ] Yue Ma commented on FLINK-31238: [~yunta] Thanks for the reply, I have written the detailed design plan

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-03-10 Thread Yun Tang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17699160#comment-17699160 ] Yun Tang commented on FLINK-31238: -- [~mayuehappy] Thanks for the reply. Considering the efforts to

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-03-08 Thread Yue Ma (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698227#comment-17698227 ] Yue Ma commented on FLINK-31238: [~yunta] Thanks for replying ~ Yes, in the current implementation both

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-03-08 Thread Yue Ma (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17698215#comment-17698215 ] Yue Ma commented on FLINK-31238: [~masteryhx]  Thanks for the reply, I understand your concerns. It's a

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-03-08 Thread Yun Tang (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17697815#comment-17697815 ] Yun Tang commented on FLINK-31238: -- [~mayuehappy] Thanks for the great work! If my understanding is

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-03-08 Thread Hangxiang Yu (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17697787#comment-17697787 ] Hangxiang Yu commented on FLINK-31238: -- [~mayuehappy] , Thanks for your detailed explanation.

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-03-06 Thread Yue Ma (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17697239#comment-17697239 ] Yue Ma commented on FLINK-31238: [~yunta] [~klion26] [~ym] [~roman]  Can you guys take a look at the

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-02-28 Thread Yue Ma (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694835#comment-17694835 ] Yue Ma commented on FLINK-31238: [~masteryhx] Thanks for reply. 1. In our current implementation,

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-02-28 Thread Hangxiang Yu (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694813#comment-17694813 ] Hangxiang Yu commented on FLINK-31238: -- Thank you for sharing the design. The duration of rescaling

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-02-28 Thread Yue Ma (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694475#comment-17694475 ] Yue Ma commented on FLINK-31238: 1. IngestExternalFile will use global_seq_no, but when we rewrite, the

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-02-28 Thread Yanfei Lei (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694438#comment-17694438 ] Yanfei Lei commented on FLINK-31238: Thanks for your sharing, this drastically reduces the number of

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-02-27 Thread Yue Ma (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17693994#comment-17693994 ] Yue Ma commented on FLINK-31238: [~Yanfei Lei] thanks a lot for reply. Let me give an example to

[jira] [Commented] (FLINK-31238) Use IngestDB to speed up Rocksdb rescaling recovery

2023-02-27 Thread Yanfei Lei (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17693957#comment-17693957 ] Yanfei Lei commented on FLINK-31238: [~mayuehappy] Great work!  I also tried to use ingest() to