yihua commented on PR #19802: URL: https://github.com/apache/hudi/pull/19802#issuecomment-5946183886
The vector index lookup works with global RLI. On the partitioned RLI lookup: this is not a bug in the existing path and is unrelated to #19801. `readRecordIndexLocationsWithKeys(keys, partition)` has an implicit precondition that all keys in a batch hash to the same RLI shard, and every current caller (Spark write/read, Flink read) groups the lookup keys by partition AND shard before calling. The vector lookup caller does not follow this contract - it invokes the method after grouping the keys by data table partition only, without shard ID within the partition, which breaks on partitioned RLI with more than one shard per data table partition. The fix to make vector index work with partitioned RLI belongs in the vector arbiter (`VectorIndexArbiter`) as part of #20107, grouping finalist keys by mapRecordKeyToFileGroupIndex within each partition, the way the Spark/Flink read path does. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
