mayuehappy commented on code in PR #24031:
URL: https://github.com/apache/flink/pull/24031#discussion_r1481169952
##########
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/restore/RocksDBIncrementalRestoreOperation.java:
##########
@@ -539,13 +552,14 @@ private void rescaleClipIngestDB(
}
// If there are temporary instances for which we detected overlap
in the buffer or
- // if we have remaining handles to restore, we will insert by copy
with from temporary
- // instances to base DB.
+ // if we have remaining unopened handles to restore, we will
insert by copy via
+ // temporary instances to base DB.
Review Comment:
In the old code, we called `choseTheBestStateHandleForInitial` to choose the
best state handle to init the initial db .Because if we use TheBestHandle as a
temporary DB instead of the main DB, we may need to write a lot of data when
copying. Can we maintain this logic int the new code ?
When choosing db to export, we prioritize `TheBestHandle` to ensure that we
will not copy TheBestHandle during the subsequent copying phase. Can this
ensure that there is no regression compared to the old code
##########
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/restore/RocksDBIncrementalRestoreOperation.java:
##########
@@ -488,6 +495,8 @@ private void rescaleClipIngestDB(
List<ColumnFamilyHandle> tmpColumnFamilyHandles =
tmpRestoreDBInfo.columnFamilyHandles;
+ // Check if the data in all SST files referenced in the handle
is within the
+ // proclaimed key-groups range of the handle.
if (RocksDBIncrementalCheckpointUtils.isSstDataInKeyGroupRange(
Review Comment:
Here we need to check
Is key in proclaimed range or is there overlap between the checking
stateHandles?
For example, the range of proclaimed is [1,5] [6,10] but the actual range is
[1,7] [8,9]. Should it be possible to export in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
