hudi-agent commented on code in PR #19840:
URL: https://github.com/apache/hudi/pull/19840#discussion_r3931352380
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/DynamicBucketAssignFunction.java:
##########
@@ -143,6 +148,25 @@ public void processElement(HoodieFlinkInternalRow record,
Context ctx, Collector
out.collect(record);
}
+ /**
+ * Processes an index record that carries a pre-computed file group mapping
for a record key.
+ * The index record is used for time-bounded bootstrap of the partitioned
record level index,
+ * where the file group assignment is already known from the metadata table.
+ *
+ * <p>This method updates the partitioned index backend with the file group
assignment
+ * and registers the file group as an update in the bucket assigner so that
the file group
+ * is recognized as an existing one.
+ *
+ * @param record the index record carrying the partition path, record key,
and file id
+ */
+ protected void processIndexRecord(HoodieFlinkInternalRow record) {
+ String partitionPath = record.getPartitionPath();
+ String recordKey = record.getRecordKey();
+ String fileId = record.getFileId();
+ indexBackend.update(partitionPath, recordKey, fileId);
+ bucketAssigner.addUpdate(partitionPath, fileId);
Review Comment:
🤖 `indexBackend.update(...)` here routes through
`RecordLevelIndexBackend.update` -> `getOrBootstrapPartition`, which on the
first key for a partition lazily scans and loads the *entire* partition's RLI
from the metadata table (not just the time-windowed subset). Since the
time-bounded bootstrap operator is pushing these keys precisely to avoid a full
load, does this first `update()` end up triggering the very full-partition MDT
bootstrap it's trying to replace? @danny0405 could you confirm this is intended
and not defeating the time-bounded memory savings?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
