voonhous commented on PR #18359: URL: https://github.com/apache/hudi/pull/18359#issuecomment-4863074519
I was trying to see how one would implement this feature and realised that the SI + RLI design will not work without significant changes to SI restrictions. `HoodieBackedTableMetadataWriter.updateSecondaryIndexIfPresent` **throws** `HoodieIndexException` for any `isInsertOverwriteOrDeletePartition()` operation when a secondary index exists. Enabling the index the SI design requires therefore breaks `insert_overwrite`, `insert_overwrite_table`, and `delete_partition` table-wide -- operations the SI design's own `preCommit` coverage table claims to support (C7, R7). In addition: - SI creation hard-requires the record index (`HoodieIndexUtils`), so the SI design's primary path needs two MDT index partitions maintained on every commit. - SI maintenance re-reads the **previous merged file slice** of every touched file group on every commit to compute old-value tombstones (`SecondaryIndexRecordGenerationUtils.convertWriteStatsToSecondaryIndexRecords`. This is a permanent write-path tax on all writers, absent from the SI design's cost model. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
