nsivabalan commented on issue #16913: URL: https://github.com/apache/hudi/issues/16913#issuecomment-4800678406
This appears to be addressed on current `master` as of HUDI-9505 / PR #13489 (commit `7deaa18c2417`), which introduced index format versioning (V2) for the secondary index. With SI V2: - **Writes** hash by the secondary-key portion only. `MetadataPartitionType.SECONDARY_INDEX.getFileGroupMappingFunction(indexVersion)` returns `HoodieTableMetadataUtil.getSecondaryKeyToFileGroupMappingFunction(true)` (see `HoodieTableMetadataUtil.java` ~L1423-1431 and `HoodieBackedTableMetadataWriter#getRecordTagger` ~L2044-2051), which extracts the unescaped secondary-key prefix from `secKey$recKey` before hashing. - **Reads** in `HoodieBackedTableMetadata#lookupIndexRecords` (~L295-303) hash the incoming secondary key directly via `mapRecordKeyToFileGroupIndex` and read a single file group instead of scanning all of them. The payload record key on disk is still `secKey$recKey`, but the file-group placement is determined by the secondary key alone, enabling hash-based join/lookup on SI as proposed. Safe to close if the V2 path is what was intended; please reopen if there is a remaining concern with V1 SI tables (which still hash by the full composite key). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
