cshuo commented on code in PR #19837:
URL: https://github.com/apache/hudi/pull/19837#discussion_r3930572690
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/DynamicBucketAssignFunction.java:
##########
@@ -143,6 +153,15 @@ public void processElement(HoodieFlinkInternalRow record,
Context ctx, Collector
out.collect(record);
}
+ /**
+ * Applies a preloaded index record emitted by a bootstrap operator directly
to the partitioned
+ * index backend, since its {@code recordKey -> fileGroupId} mapping is
already known and does not
+ * need bucket assignment. The record carries no row data, so it is never
emitted downstream.
+ */
+ private void processIndexRecord(HoodieFlinkInternalRow record) {
Review Comment:
`update()` calls `getOrBootstrapPartition()`, triggering a full partition
scan on the first preloaded record in each downstream task. Could we add `void
bootstrap(String partitionPath, String recordKey, String fileId)` to
`PartitionedIndexBackend`? It would create the cache if absent and call
`cache.bootstrapRecordKey(recordKey, fileId)` directly, avoiding both lazy
bootstrap and marking the mapping as a normal data update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
