danny0405 commented on code in PR #18018:
URL: https://github.com/apache/hudi/pull/18018#discussion_r2744619774
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/index/RecordIndexCache.java:
##########
@@ -44,17 +44,33 @@
* <p>
* todo: use map backed by flink managed memory.
*/
+@Slf4j
public class RecordIndexCache implements Closeable {
@VisibleForTesting
@Getter
private final TreeMap<Long, ExternalSpillableMap<String,
HoodieRecordGlobalLocation>> caches;
private final HoodieWriteConfig writeConfig;
private final long maxCacheSizeInBytes;
+ // the checkpoint id corresponding to the minimum inflight hoodie instant.
+ private long ckpIdForMinInflightInstant;
+ private long recordCnt = 0;
+
+ /**
+ * The empirical value for the maximum number of pending instants, i.e.,
checkpoint of flink is success, while
+ * the hoodie instant is not committed.
+ */
+ private static final int CHECKPOINT_CACHES_MAX_NUM = 3;
Review Comment:
why 3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
