hudi-agent commented on code in PR #19952:
URL: https://github.com/apache/hudi/pull/19952#discussion_r4018496953
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/index/IndexBackendFactory.java:
##########
@@ -75,38 +66,12 @@ public static GlobalIndexBackend create(
if (conf.get(FlinkOptions.INDEX_BOOTSTRAP_ENABLED)) {
return new
RocksDBIndexBackend(conf.get(FlinkOptions.INDEX_BOOTSTRAP_ROCKSDB_PATH),
OptionsResolver.isPartitionedTable(conf));
} else {
- ListState<JobID> jobIdState =
context.getOperatorStateStore().getListState(
- new ListStateDescriptor<>(
- "bucket-assign-job-id-state",
- TypeInformation.of(JobID.class)
- ));
- long initCheckpointId = -1;
- if (context.isRestored()) {
- int attemptId =
RuntimeContextUtils.getAttemptNumber(runtimeContext);
- initCheckpointId = initCheckpointId(attemptId, jobIdState,
context.getRestoredCheckpointId().orElse(-1L), runtimeContext);
- }
- // set the jobId state with current job id.
- jobIdState.clear();
- jobIdState.add(RuntimeContextUtils.getJobId(runtimeContext));
+ // Match the writer's checkpoint ID so uncommitted index entries
remain protected from eviction.
+ long initCheckpointId = context.isRestored() ?
context.getRestoredCheckpointId().orElse(-1L) : -1L;
Review Comment:
🤖 One knock-on for the follow-up PR: while the scale-up bootstrap buffer
stays incomplete, `requestInflightInstants()` keeps returning its (older) key,
so `markAsEvictable` is pinned there and `cleanIfNecessary` can never release
any newer cache generation — the RLI cache just accumulates one generation per
checkpoint. Is it worth the follow-up also bounding that (e.g. excluding stuck
bootstrap buffers from the inflight set once they're deferred), or is the plan
that completing bootstrap after scale-up makes it moot?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
