deepakpanda93 commented on issue #10407: URL: https://github.com/apache/hudi/issues/10407#issuecomment-4845958743
The root cause is a contradictory write config: INDEX_TYPE=SIMPLE was set together with the bucket-index storage layout (LAYOUT_PARTITIONER_CLASS_NAME=SparkBucketIndexPartitioner, BUCKET_INDEX_NUM_BUCKETS, etc.). Hudi only applies the bucket storage layout when INDEX_TYPE=BUCKET (see HoodieLayoutConfig), so forcing the bucket partitioner under a SIMPLE index is inconsistent. As @beyond1920 noted, that's why some commits produced bucket-prefixed file ids (00000018…) and others UUID-style (5f1c45ef…), scattering the same record key across different file groups → duplicates; and switching to INDEX_TYPE=BUCKET then fails with NumberFormatException because the existing UUID file groups don't carry the 8-char bucket-id prefix. To resolve: - Pick one index strategy and keep it consistent. For a non-bucket table, use SIMPLE/BLOOM/RECORD_INDEX and do not set SparkBucketIndexPartitioner or BUCKET_INDEX_*. For bucket indexing, set INDEX_TYPE=BUCKET from table creation. - Existing mismatched file groups won't reconcile by config change alone — they need a one-time rewrite (e.g., re-bootstrap the table) to consolidate, then a dedup pass. - On a recent 1.x release, the writer infers/reuses the key-gen and index config from the table's hoodie.properties (HUDI-5929, HUDI-5665), which reduces this kind of cross-run mismatch. If you can reproduce duplicates on 1.x with a single, consistent index configuration, please share the write configs and the file-group id format before/after. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
