MarigWeizhi commented on issue #16418: URL: https://github.com/apache/iceberg/issues/16418#issuecomment-4886346249
> Thinking more about this, there is nothing incremental going on here. we are always overwriting. There is no checkpointing and resumable progress. So the only issue here is we are avoiding FileAlreadyExistsException in case staging location is reused. > > I am not sure if it is worth designing a solution which is truly incremental. @mukund-thakur Sorry for the late reply. I believe distcp currently supports skipping existing data files. For my use case, performing a full overwrite of the table path and then re-running distcp is sufficient to meet the requirements for incremental migration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
