RussellSpitzer commented on issue #7657: URL: https://github.com/apache/iceberg/issues/7657#issuecomment-1554603272
The `rewrite-datafiles` is almost certainly a red herring. It shouldn't have any effect on the table if it failed since it is transactional like all Iceberg operations. It may be that this is coincidental or perhaps the system has just run out of some resource like disk space. > Spark will attempt to load and sort the entire table before inserting the new batch even though we are partitioning by loaded_at which should never need more than one partition. This is quite confusing to me. An "append" should never require reading anything from the target table. The source though will have a shuffle applied to it based on the `write.distribution-mode` of the table. We recently did change the default to hash which would request that the incoming data be shuffled by the partition of the destination table. If you are confident it does not need to be shuffled you can always set the mode to `none`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
