SreeramGarlapati commented on PR #2509: URL: https://github.com/apache/iceberg-rust/pull/2509#issuecomment-4746076058
Changed direction after checking the Java reference: Iceberg Java doesn't error on a repeated path within a batch, it de-duplicates by file location (`DataFileSet`, keyed on `file.location()`). So rather than fail the commit, this now drops the duplicate and keeps the first occurrence. More forgiving for retry / file-discovery paths that can legitimately hand the same file twice, and it still removes the double-count corruption #2507 is about. @blackmwk the logic now lives in `FastAppendAction` as you suggested, and isn't gated on `check_duplicate` (your append.rs:88 note + @viirya) since it's pure in-memory work. @viirya MergeAppend / overwrite: keeping that as a separate follow-up, out of scope here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
