[jira] [Commented] (FLINK-33798) Automatically clean up rocksdb logs when the task failover.

2024-03-13 Thread Ming Li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-33798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825913#comment-17825913 ] Ming Li commented on FLINK-33798: - [~fanrui], good idea, I will back port to 1.17 , 1.18 and 1.19. >

[jira] [Commented] (FLINK-32181) Drop support for Maven 3.2.5

2023-12-12 Thread Ming Li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-32181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17796044#comment-17796044 ] Ming Li commented on FLINK-32181: - I also encountered the same problem. Is it necessary to forcefully

[jira] [Created] (FLINK-33798) Automatically clean up rocksdb logs when the task failover.

2023-12-11 Thread Ming Li (Jira)
Ming Li created FLINK-33798: --- Summary: Automatically clean up rocksdb logs when the task failover. Key: FLINK-33798 URL: https://issues.apache.org/jira/browse/FLINK-33798 Project: Flink Issue

[jira] [Commented] (FLINK-32698) Add getCheckpointOptions interface in ManagedSnapshotContext

2023-07-27 Thread Ming Li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-32698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17747789#comment-17747789 ] Ming Li commented on FLINK-32698: - [~zjureel] [~zhuzh] [~fanrui]  Thanks for your suggestions, I will

[jira] [Updated] (FLINK-32698) Add getCheckpointOptions interface in ManagedSnapshotContext

2023-07-26 Thread Ming Li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-32698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Li updated FLINK-32698: Description: Currently only {{checkpointID}} and {{checkpointTimestamp}} information are provided in

[jira] [Created] (FLINK-32698) Add getCheckpointOptions interface in ManagedSnapshotContext

2023-07-26 Thread Ming Li (Jira)
Ming Li created FLINK-32698: --- Summary: Add getCheckpointOptions interface in ManagedSnapshotContext Key: FLINK-32698 URL: https://issues.apache.org/jira/browse/FLINK-32698 Project: Flink Issue

[jira] [Created] (FLINK-32655) RecreateOnResetOperatorCoordinator did not forward notifyCheckpointAborted to the real OperatorCoordinator

2023-07-24 Thread Ming Li (Jira)
Ming Li created FLINK-32655: --- Summary: RecreateOnResetOperatorCoordinator did not forward notifyCheckpointAborted to the real OperatorCoordinator Key: FLINK-32655 URL: https://issues.apache.org/jira/browse/FLINK-32655

[jira] [Created] (FLINK-32001) SupportsRowLevelUpdate does not support returning only a part of the columns.

2023-05-04 Thread Ming Li (Jira)
Ming Li created FLINK-32001: --- Summary: SupportsRowLevelUpdate does not support returning only a part of the columns. Key: FLINK-32001 URL: https://issues.apache.org/jira/browse/FLINK-32001 Project: Flink

[jira] [Commented] (FLINK-26465) Optimize SortMergeReader: use loser tree to reduce comparisons

2023-03-17 Thread Ming Li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-26465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17701531#comment-17701531 ] Ming Li commented on FLINK-26465: - Hi, [~lzljs3620320] . Is this issue still going on? If there is no

[jira] [Created] (FLINK-31474) [Flink] Add failure information for out-of-order checkpoints

2023-03-15 Thread Ming Li (Jira)
Ming Li created FLINK-31474: --- Summary: [Flink] Add failure information for out-of-order checkpoints Key: FLINK-31474 URL: https://issues.apache.org/jira/browse/FLINK-31474 Project: Flink Issue

[jira] [Commented] (FLINK-31462) [Flink] Supports full calculation from specified snapshots in streaming mode

2023-03-15 Thread Ming Li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17700658#comment-17700658 ] Ming Li commented on FLINK-31462: - Hi, [~lzljs3620320] , For example, I consume two tables A and B. The

[jira] [Created] (FLINK-31465) [Flink] Fix shortcode errors in docs

2023-03-15 Thread Ming Li (Jira)
Ming Li created FLINK-31465: --- Summary: [Flink] Fix shortcode errors in docs Key: FLINK-31465 URL: https://issues.apache.org/jira/browse/FLINK-31465 Project: Flink Issue Type: Improvement

[jira] [Updated] (FLINK-31462) [Flink] Supports full calculation from specified snapshots in streaming mode

2023-03-15 Thread Ming Li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Li updated FLINK-31462: Description: Currently, the table store provides a startup mode for incremental consumption from a

[jira] [Updated] (FLINK-31462) [Flink] Supports full calculation from specified snapshots in streaming mode

2023-03-15 Thread Ming Li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Li updated FLINK-31462: Summary: [Flink] Supports full calculation from specified snapshots in streaming mode (was: [Flink]

[jira] [Created] (FLINK-31462) [Flink] Supports full calculation from the specified snapshot

2023-03-15 Thread Ming Li (Jira)
Ming Li created FLINK-31462: --- Summary: [Flink] Supports full calculation from the specified snapshot Key: FLINK-31462 URL: https://issues.apache.org/jira/browse/FLINK-31462 Project: Flink Issue

[jira] [Closed] (FLINK-31364) [Flink] add metrics for TableStore

2023-03-08 Thread Ming Li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Li closed FLINK-31364. --- Resolution: Duplicate > [Flink] add metrics for TableStore > -- > >

[jira] [Created] (FLINK-31364) [Flink] add metrics for TableStore

2023-03-07 Thread Ming Li (Jira)
Ming Li created FLINK-31364: --- Summary: [Flink] add metrics for TableStore Key: FLINK-31364 URL: https://issues.apache.org/jira/browse/FLINK-31364 Project: Flink Issue Type: Improvement

[jira] [Commented] (FLINK-31294) CommitterOperator forgot to close Committer when closing.

2023-03-01 Thread Ming Li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695457#comment-17695457 ] Ming Li commented on FLINK-31294: - [~lzljs3620320], hi, please help take a look at this issue if you

[jira] [Created] (FLINK-31294) CommitterOperator forgot to close Committer when closing.

2023-03-01 Thread ming li (Jira)
ming li created FLINK-31294: --- Summary: CommitterOperator forgot to close Committer when closing. Key: FLINK-31294 URL: https://issues.apache.org/jira/browse/FLINK-31294 Project: Flink Issue Type:

[jira] [Commented] (FLINK-31266) dashboard info error (received and send alway show 0 when having data)

2023-02-28 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694909#comment-17694909 ] ming li commented on FLINK-31266: - hi, [~Weijie Guo], [~Zhanghao Chen]  It is indeed confusing for new

[jira] [Commented] (FLINK-30985) [Flink][Table Store] Change the Splits allocation algorithm of ContinuousFileSplitEnumerator in TableStore to a fair algorithm.

2023-02-10 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17687053#comment-17687053 ] ming li commented on FLINK-30985: - [~lzljs3620320] The {{RequestSplit}} event may not have occurred when

[jira] [Commented] (FLINK-31008) [Flink][Table Store] The Split allocation of the same bucket in ContinuousFileSplitEnumerator may be out of order

2023-02-10 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17687046#comment-17687046 ] ming li commented on FLINK-31008: - [~lzljs3620320] I have created a pull request, please review it if

[jira] [Commented] (FLINK-30985) [Flink][Table Store] Change the Splits allocation algorithm of ContinuousFileSplitEnumerator in TableStore to a fair algorithm.

2023-02-10 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17687041#comment-17687041 ] ming li commented on FLINK-30985: - [~lzljs3620320] Consider extreme cases, for example, I only have one

[jira] [Commented] (FLINK-30985) [Flink][Table Store] Change the Splits allocation algorithm of ContinuousFileSplitEnumerator in TableStore to a fair algorithm.

2023-02-10 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686991#comment-17686991 ] ming li commented on FLINK-30985: - [~lzljs3620320] This is somewhat different from

[jira] [Commented] (FLINK-31008) [Flink][Table Store] The Split allocation of the same bucket in ContinuousFileSplitEnumerator may be out of order

2023-02-09 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686921#comment-17686921 ] ming li commented on FLINK-31008: - [~lzljs3620320] Yes, thanks a lot, I've been doing some related work

[jira] [Commented] (FLINK-31008) [Flink][Table Store] The Split allocation of the same bucket in ContinuousFileSplitEnumerator may be out of order

2023-02-09 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-31008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686912#comment-17686912 ] ming li commented on FLINK-31008: - hi, [~lzljs3620320], if you have time, please help to take a look at

[jira] [Created] (FLINK-31008) [Flink][Table Store] The Split allocation of the same bucket in ContinuousFileSplitEnumerator may be out of order

2023-02-09 Thread ming li (Jira)
ming li created FLINK-31008: --- Summary: [Flink][Table Store] The Split allocation of the same bucket in ContinuousFileSplitEnumerator may be out of order Key: FLINK-31008 URL:

[jira] [Updated] (FLINK-30985) [Flink][Table Store] Change the Splits allocation algorithm of ContinuousFileSplitEnumerator in TableStore to a fair algorithm.

2023-02-09 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-30985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming li updated FLINK-30985: Summary: [Flink][Table Store] Change the Splits allocation algorithm of ContinuousFileSplitEnumerator in

[jira] [Created] (FLINK-30985) [Flink][table-store] Change the Splits allocation algorithm of ContinuousFileSplitEnumerator in TableStore to a fair algorithm.

2023-02-09 Thread ming li (Jira)
ming li created FLINK-30985: --- Summary: [Flink][table-store] Change the Splits allocation algorithm of ContinuousFileSplitEnumerator in TableStore to a fair algorithm. Key: FLINK-30985 URL:

[jira] [Commented] (FLINK-30251) Move the IO with DFS during abort checkpoint to an asynchronous thread.

2022-12-09 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-30251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17645200#comment-17645200 ] ming li commented on FLINK-30251: - [~gaoyunhaii] thanks for your reply. We have tried setting timeout

[jira] [Commented] (FLINK-30251) Move the IO with DFS during abort checkpoint to an asynchronous thread.

2022-12-06 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-30251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17643730#comment-17643730 ] ming li commented on FLINK-30251: - hi, [~Yanfei Lei]. I re-read FLINK-28984, it seems that it is a

[jira] [Commented] (FLINK-30251) Move the IO with DFS during abort checkpoint to an asynchronous thread.

2022-11-30 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-30251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17641685#comment-17641685 ] ming li commented on FLINK-30251: - hi, [~Yanfei Lei] , I made a simple implementation, can you help to

[jira] [Commented] (FLINK-27681) Improve the availability of Flink when the RocksDB file is corrupted.

2022-11-30 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-27681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17641648#comment-17641648 ] ming li commented on FLINK-27681: - [~Yanfei Lei] thank you for your reply. In our production, the

[jira] [Updated] (FLINK-30251) Move the IO with DFS during abort checkpoint to an asynchronous thread.

2022-11-30 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-30251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming li updated FLINK-30251: Description: Currently when the {{checkpoint}} fails, we process the abort message in the Task's

[jira] [Updated] (FLINK-30251) Move the IO with DFS during abort checkpoint to an asynchronous thread.

2022-11-30 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-30251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming li updated FLINK-30251: Description: Currently when the {{checkpoint}} fails, we process the abort message in the Task's

[jira] [Updated] (FLINK-30251) Move the IO with DFS during abort checkpoint to an asynchronous thread.

2022-11-30 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-30251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming li updated FLINK-30251: Description: Currently when the {{checkpoint}} fails, we process the abort message in the Task's

[jira] [Created] (FLINK-30251) Move the IO with DFS during abort checkpoint to an asynchronous thread.

2022-11-30 Thread ming li (Jira)
ming li created FLINK-30251: --- Summary: Move the IO with DFS during abort checkpoint to an asynchronous thread. Key: FLINK-30251 URL: https://issues.apache.org/jira/browse/FLINK-30251 Project: Flink

[jira] [Commented] (FLINK-28390) Allows RocksDB to configure FIFO Compaction to reduce CPU overhead.

2022-07-24 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-28390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17570607#comment-17570607 ] ming li commented on FLINK-28390: - [~yunta] ok thanks, I got it. This commit has fixed the problem that

[jira] [Commented] (FLINK-28390) Allows RocksDB to configure FIFO Compaction to reduce CPU overhead.

2022-07-21 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-28390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569799#comment-17569799 ] ming li commented on FLINK-28390: - [~yunta] sorry, maybe I didn't describe clearly. In Flink we only

[jira] [Commented] (FLINK-28390) Allows RocksDB to configure FIFO Compaction to reduce CPU overhead.

2022-07-11 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-28390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17565029#comment-17565029 ] ming li commented on FLINK-28390: - [~yunta] the {{RocksDB}} version used by Flink already contains the

[jira] [Commented] (FLINK-28390) Allows RocksDB to configure FIFO Compaction to reduce CPU overhead.

2022-07-11 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-28390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564959#comment-17564959 ] ming li commented on FLINK-28390: - [~martijnvisser] we generally set the TTL for the state, assuming

[jira] [Commented] (FLINK-28010) Use deleteRange to optimize the clear operation of RocksDBMapState.

2022-07-11 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-28010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564905#comment-17564905 ] ming li commented on FLINK-28010: - [~yunta], can you assign this to me and help review the code? > Use

[jira] [Comment Edited] (FLINK-28473) JobManager restart/failover doesn't trigger local recovery on TaskManagers

2022-07-11 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-28473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564903#comment-17564903 ] ming li edited comment on FLINK-28473 at 7/11/22 9:51 AM: -- [~chesnay], at

[jira] [Comment Edited] (FLINK-28473) JobManager restart/failover doesn't trigger local recovery on TaskManagers

2022-07-11 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-28473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564903#comment-17564903 ] ming li edited comment on FLINK-28473 at 7/11/22 9:50 AM: -- [~chesnay], at

[jira] [Commented] (FLINK-28473) JobManager restart/failover doesn't trigger local recovery on TaskManagers

2022-07-11 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-28473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564903#comment-17564903 ] ming li commented on FLINK-28473: - [~chesnay], at present, after JM failover, all tasks must be

[jira] [Commented] (FLINK-28390) Allows RocksDB to configure FIFO Compaction to reduce CPU overhead.

2022-07-11 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-28390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564890#comment-17564890 ] ming li commented on FLINK-28390: - Hi, [~masteryhx], [~Zhanghao Chen] Yes, although we currently have

[jira] [Commented] (FLINK-28473) JobManager restart/failover doesn't trigger local recovery on TaskManagers

2022-07-11 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-28473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564875#comment-17564875 ] ming li commented on FLINK-28473: - Hi,[~lkokhreidze] I think after JM failover, it may be difficult for

[jira] [Commented] (FLINK-28010) Use deleteRange to optimize the clear operation of RocksDBMapState.

2022-07-06 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-28010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17563041#comment-17563041 ] ming li commented on FLINK-28010: - [~yunta], maybe we just need to add 1 to the prefix bytes? > Use

[jira] [Created] (FLINK-28390) Allows RocksDB to configure FIFO Compaction to reduce CPU overhead.

2022-07-05 Thread ming li (Jira)
ming li created FLINK-28390: --- Summary: Allows RocksDB to configure FIFO Compaction to reduce CPU overhead. Key: FLINK-28390 URL: https://issues.apache.org/jira/browse/FLINK-28390 Project: Flink

[jira] [Commented] (FLINK-28010) Use deleteRange to optimize the clear operation of RocksDBMapState.

2022-07-05 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-28010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17562403#comment-17562403 ] ming li commented on FLINK-28010: - hi, [~yunta], do you think this optimization is feasible? > Use

[jira] [Commented] (FLINK-28010) Use deleteRange to optimize the clear operation of RocksDBMapState.

2022-07-01 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-28010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17561322#comment-17561322 ] ming li commented on FLINK-28010: - [~Yanfei Lei] thank you for your reply, I think this can save us a

[jira] [Created] (FLINK-28010) Use deleteRange to optimize the clear operation of RocksDBMapState.

2022-06-11 Thread ming li (Jira)
ming li created FLINK-28010: --- Summary: Use deleteRange to optimize the clear operation of RocksDBMapState. Key: FLINK-28010 URL: https://issues.apache.org/jira/browse/FLINK-28010 Project: Flink

[jira] [Commented] (FLINK-20681) Support specifying the hdfs path when ship archives or files

2022-05-27 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17542901#comment-17542901 ] ming li commented on FLINK-20681: - hi, [~RocMarshal], what is the current progress of this issue? We

[jira] [Created] (FLINK-27681) Improve the availability of Flink when the RocksDB file is corrupted.

2022-05-18 Thread ming li (Jira)
ming li created FLINK-27681: --- Summary: Improve the availability of Flink when the RocksDB file is corrupted. Key: FLINK-27681 URL: https://issues.apache.org/jira/browse/FLINK-27681 Project: Flink

[jira] [Comment Edited] (FLINK-24815) Reduce the cpu cost of calculating stateSize during state allocation

2021-11-16 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444925#comment-17444925 ] ming li edited comment on FLINK-24815 at 11/17/21, 3:43 AM: [~pnowojski],

[jira] [Commented] (FLINK-24815) Reduce the cpu cost of calculating stateSize during state allocation

2021-11-16 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444925#comment-17444925 ] ming li commented on FLINK-24815: - [~pnowojski], um. . . you are right.  Although the size of the

[jira] [Comment Edited] (FLINK-24815) Reduce the cpu cost of calculating stateSize during state allocation

2021-11-14 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443533#comment-17443533 ] ming li edited comment on FLINK-24815 at 11/15/21, 3:54 AM: Hi, [~pnowojski]

[jira] [Commented] (FLINK-24815) Reduce the cpu cost of calculating stateSize during state allocation

2021-11-14 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443533#comment-17443533 ] ming li commented on FLINK-24815: - Hi, [~pnowojski] , In{{{} TaskStateAssignment#getSubtaskState{}}}, we

[jira] [Created] (FLINK-24815) Reduce the cpu cost of calculating stateSize during state allocation

2021-11-07 Thread ming li (Jira)
ming li created FLINK-24815: --- Summary: Reduce the cpu cost of calculating stateSize during state allocation Key: FLINK-24815 URL: https://issues.apache.org/jira/browse/FLINK-24815 Project: Flink

[jira] [Commented] (FLINK-21543) when using FIFO compaction, I found sst being deleted on the first checkpoint

2021-11-02 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437186#comment-17437186 ] ming li commented on FLINK-21543: - Hi, [~yunta], can you describe in detail the problem of data loss

[jira] [Commented] (FLINK-24086) Do not re-register SharedStateRegistry to reduce the recovery time of the job

2021-09-15 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415847#comment-17415847 ] ming li commented on FLINK-24086: - Hi, [~dwysakowicz], I am sorry for not being able to update in time,

[jira] [Commented] (FLINK-24086) Do not re-register SharedStateRegistry to reduce the recovery time of the job

2021-09-10 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17413232#comment-17413232 ] ming li commented on FLINK-24086: - Hi, [~dwysakowicz], Here I have a question: {quote}it reuses old 

[jira] [Commented] (FLINK-24086) Do not re-register SharedStateRegistry to reduce the recovery time of the job

2021-09-06 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17410902#comment-17410902 ] ming li commented on FLINK-24086: - [~pnowojski], thanks for your reply, I will try to make changes on

[jira] [Comment Edited] (FLINK-24086) Do not re-register SharedStateRegistry to reduce the recovery time of the job

2021-09-01 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408273#comment-17408273 ] ming li edited comment on FLINK-24086 at 9/1/21, 4:49 PM: -- {quote}We

[jira] [Comment Edited] (FLINK-24086) Do not re-register SharedStateRegistry to reduce the recovery time of the job

2021-09-01 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408273#comment-17408273 ] ming li edited comment on FLINK-24086 at 9/1/21, 4:48 PM: -- {quote}We

[jira] [Comment Edited] (FLINK-24086) Do not re-register SharedStateRegistry to reduce the recovery time of the job

2021-09-01 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408273#comment-17408273 ] ming li edited comment on FLINK-24086 at 9/1/21, 4:48 PM: -- {quote}We

[jira] [Commented] (FLINK-24086) Do not re-register SharedStateRegistry to reduce the recovery time of the job

2021-09-01 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408273#comment-17408273 ] ming li commented on FLINK-24086: - ??We implemented a new failover strategy (by discarding some data to

[jira] [Commented] (FLINK-24086) Do not re-register SharedStateRegistry to reduce the recovery time of the job

2021-09-01 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408138#comment-17408138 ] ming li commented on FLINK-24086: -   Hi, [~pnowojski], We implemented a new failover strategy (by

[jira] [Commented] (FLINK-24086) Do not re-register SharedStateRegistry to reduce the recovery time of the job

2021-09-01 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408017#comment-17408017 ] ming li commented on FLINK-24086: - Hi, [~pnowojski], Can you help to take a look at this ticket ? :) >

[jira] [Updated] (FLINK-24086) Do not re-register SharedStateRegistry to reduce the recovery time of the job

2021-08-31 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming li updated FLINK-24086: Description: At present, we only recover the {{CompletedCheckpointStore}} when the {{JobManager}}

[jira] [Created] (FLINK-24086) Do not re-register SharedStateRegistry to reduce the recovery time of the job

2021-08-31 Thread ming li (Jira)
ming li created FLINK-24086: --- Summary: Do not re-register SharedStateRegistry to reduce the recovery time of the job Key: FLINK-24086 URL: https://issues.apache.org/jira/browse/FLINK-24086 Project: Flink

[jira] [Commented] (FLINK-22483) Recover checkpoints when JobMaster gains leadership

2021-08-31 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407365#comment-17407365 ] ming li commented on FLINK-22483: - Hi, [~trohrmann]. Thanks for your reply, I will create a separate

[jira] [Comment Edited] (FLINK-22483) Recover checkpoints when JobMaster gains leadership

2021-08-30 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407055#comment-17407055 ] ming li edited comment on FLINK-22483 at 8/31/21, 3:46 AM: --- Hi, [~trohrmann],

[jira] [Commented] (FLINK-22483) Recover checkpoints when JobMaster gains leadership

2021-08-30 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407055#comment-17407055 ] ming li commented on FLINK-22483: - Hi, [~trohrmann], thanks for your reply. It seems that more of the

[jira] [Comment Edited] (FLINK-22483) Recover checkpoints when JobMaster gains leadership

2021-08-30 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17406647#comment-17406647 ] ming li edited comment on FLINK-22483 at 8/30/21, 9:25 AM: --- Hi, [~dmvk],

[jira] [Commented] (FLINK-22483) Recover checkpoints when JobMaster gains leadership

2021-08-30 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-22483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17406647#comment-17406647 ] ming li commented on FLINK-22483: - Hi, [~dmvk], [~edu05]. Could you please help answer one of my

[jira] [Updated] (FLINK-21990) SourceStreamTask will always hang if the CheckpointedFunction#snapshotState throws an exception.

2021-03-28 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming li updated FLINK-21990: Affects Version/s: 1.12.0 > SourceStreamTask will always hang if the CheckpointedFunction#snapshotState

[jira] [Updated] (FLINK-21990) SourceStreamTask will always hang if the CheckpointedFunction#snapshotState throws an exception.

2021-03-28 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming li updated FLINK-21990: Description: If the source in {{SourceStreamTask}} implements {{CheckpointedFunction}} and an exception

[jira] [Updated] (FLINK-21990) SourceStreamTask will always hang if the CheckpointedFunction#snapshotState throws an exception.

2021-03-28 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming li updated FLINK-21990: Description: If the source in {{SourceStreamTask}} implements {{CheckpointedFunction}} and an exception

[jira] [Updated] (FLINK-21990) SourceStreamTask will always hang if the CheckpointedFunction#snapshotState throws an exception.

2021-03-28 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming li updated FLINK-21990: Description: If the source in SourceStreamTask implements {{CheckpointedFunction}} and an exception is

[jira] [Updated] (FLINK-21990) SourceStreamTask will always hang if the CheckpointedFunction#snapshotState throws an exception.

2021-03-28 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming li updated FLINK-21990: Summary: SourceStreamTask will always hang if the CheckpointedFunction#snapshotState throws an exception.

[jira] [Updated] (FLINK-21990) Double check Task status when perform checkpoint.

2021-03-26 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-21990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming li updated FLINK-21990: Description: We need to double check Task status when making Checkpoint. Otherwise, after a Task failed,

[jira] [Created] (FLINK-21990) Double check Task status when perform checkpoint.

2021-03-26 Thread ming li (Jira)
ming li created FLINK-21990: --- Summary: Double check Task status when perform checkpoint. Key: FLINK-21990 URL: https://issues.apache.org/jira/browse/FLINK-21990 Project: Flink Issue Type: Bug

[jira] [Commented] (FLINK-8790) Improve performance for recovery from incremental checkpoint

2021-02-09 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-8790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281761#comment-17281761 ] ming li commented on FLINK-8790: Hi, [~legojoey17], Great Job! We are also exploring how to optimize the

[jira] [Commented] (FLINK-8790) Improve performance for recovery from incremental checkpoint

2020-12-28 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-8790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255484#comment-17255484 ] ming li commented on FLINK-8790: Hi, [~sihuazhou]. At present, we use the incremental state of rocksdb to

[jira] [Commented] (FLINK-20554) The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly displayed on the Overview page of the UI

2020-12-09 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-20554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247000#comment-17247000 ] ming li commented on FLINK-20554: - Hi, [~yunta], What do you think? > The Checkpointed Data Size of the

[jira] [Updated] (FLINK-20554) The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly displayed on the Overview page of the UI

2020-12-09 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-20554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming li updated FLINK-20554: Description: The {{Checkpointed Data Size}} of the {{Latest Completed Checkpoint}} always shows '-' in

[jira] [Created] (FLINK-20554) The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly displayed on the Overview page of the UI

2020-12-09 Thread ming li (Jira)
ming li created FLINK-20554: --- Summary: The Checkpointed Data Size of the Latest Completed Checkpoint is incorrectly displayed on the Overview page of the UI Key: FLINK-20554 URL:

[jira] [Commented] (FLINK-19006) project transformation does not support conversion to Tuple25 type

2020-08-26 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185184#comment-17185184 ] ming li commented on FLINK-19006: - Hi,[~ting.lt]. I am not sure who is responsible for this module,

[jira] [Comment Edited] (FLINK-19006) project transformation does not support conversion to Tuple25 type

2020-08-26 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-19006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185184#comment-17185184 ] ming li edited comment on FLINK-19006 at 8/26/20, 1:22 PM: --- Hi,[~ting.lt]. I

[jira] [Created] (FLINK-19006) project transformation does not support conversion to Tuple25 type

2020-08-20 Thread ming li (Jira)
ming li created FLINK-19006: --- Summary: project transformation does not support conversion to Tuple25 type Key: FLINK-19006 URL: https://issues.apache.org/jira/browse/FLINK-19006 Project: Flink

[jira] [Commented] (FLINK-18808) Task-level numRecordsOut metric may be underestimated

2020-08-14 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177788#comment-17177788 ] ming li commented on FLINK-18808: - Okay, I will fix it as soon as possible. Thank you for your reply. >

[jira] [Commented] (FLINK-18808) Task-level numRecordsOut metric may be underestimated

2020-08-14 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1712#comment-1712 ] ming li commented on FLINK-18808: - {quote}{{And as an optimisation in {{AbstractStreamOperator}}, we

[jira] [Comment Edited] (FLINK-18808) Task-level numRecordsOut metric may be underestimated

2020-08-14 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177743#comment-17177743 ] ming li edited comment on FLINK-18808 at 8/14/20, 12:38 PM: {quote}I would

[jira] [Commented] (FLINK-18808) Task-level numRecordsOut metric may be underestimated

2020-08-14 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177743#comment-17177743 ] ming li commented on FLINK-18808: - {quote}I would be afraid of this check adding too much computation

[jira] [Comment Edited] (FLINK-18808) Task-level numRecordsOut metric may be underestimated

2020-08-14 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177624#comment-17177624 ] ming li edited comment on FLINK-18808 at 8/14/20, 9:14 AM: --- My proposal is to

[jira] [Commented] (FLINK-18808) Task-level numRecordsOut metric may be underestimated

2020-08-14 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177624#comment-17177624 ] ming li commented on FLINK-18808: - My proposal is to group {{Output}} according to {{OutputTag}} in

[jira] [Commented] (FLINK-18808) Task-level numRecordsOut metric may be underestimated

2020-08-13 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177475#comment-17177475 ] ming li commented on FLINK-18808: - Hi, [~pnowojski]. Sorry, I'm still a little confused. In fact, in

[jira] [Commented] (FLINK-18808) Task-level numRecordsOut metric may be underestimated

2020-08-13 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176948#comment-17176948 ] ming li commented on FLINK-18808: - I understand what you mean. You may think that a record will only be

[jira] [Updated] (FLINK-18808) Task-level numRecordsOut metric may be underestimated

2020-08-13 Thread ming li (Jira)
[ https://issues.apache.org/jira/browse/FLINK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ming li updated FLINK-18808: Attachment: image-2020-08-13-18-36-13-282.png > Task-level numRecordsOut metric may be underestimated >

  1   2   >