peach12345 commented on issue #14345: URL: https://github.com/apache/iceberg/issues/14345#issuecomment-3424890185
Hi @damianm-dev We are currently using the Iceberg 1.10 rewrite job. It appears that the job supports multi-level partition structures and compacts Parquet files within those folders/partitions. To delete outdated Parquet files, we use the deleteOrphanFiles job. We are currently using the following settings: scheduleOnInterval(10 minutes) minInputFiles(2) maxFileGroupSizeBytes(1GB) For deleteOrphanFiles: deleteBatchSize(100) scheduleOnInterval(10 mins) minAge(1 hour) For ExpireSnapshots: deleteBatchSize(100) scheduleOnInterval(10 mins) maxSnapshotAge(1 hour) We tried running the rewrite job without setting minInputFiles, but in that case the job never completed and threw exceptions like: "buffer pool is already destroyed." However, the ExpireSnapshots job doesn't seem to be working as expected — it’s not deleting old snapshot metadata files. If anyone has experience with the ExpireSnapshots job or has encountered similar issues, we’d appreciate any insights! :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
