szehon-ho opened a new pull request #2779: URL: https://github.com/apache/iceberg/pull/2779
This is a fix for problem discussed in : https://github.com/apache/iceberg/issues/2735. The change looks big, but can be summarised: * Add check_duplicate_files flag in "Add Files Procedure", true by default (can be switched off by a flag) * Add checkDuplicateFiles flag in SparkTableUtil.importSparkPartitions * Implement it by loading the metadata table "files" and joining with the files that will be imported (in a Spark job) * Refactoring metadata table methods BaseSparkActions => SparkTableUtil so both can load the metadata table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
