allychen111 opened a new issue, #17456: URL: https://github.com/apache/iceberg/issues/17456
### Apache Iceberg version 1.9.2 ### Query engine Trino ### Please describe the bug 🐞 Hi Iceberg community, I am reporting a production-like failure pattern and asking for best-practice guidance on snapshot expiration policy and operational guardrails. Summary We observed channel signoff failures that surface as orchestration timeout, but the underlying data-plane error is missing Puffin delete files. The signoff DAG fails because resolver tasks fail during table reads with NotFoundException on deletes.puffin. Failure path observed Channel signoff DAG triggers fleet restart and waits for completion. Fleet control depends on resolver tasks to complete successfully. Resolver Spark read fails with NotFoundException for deletes.puffin. Fleet does not converge, signoff wait times out, signoff fails. Important context This is not a resolver intentionally reading an old snapshot by design. Resolver reads current table state in normal operation. We also run snapshot expiration maintenance using: retention window = 14 days retainLast = 4000 snapshots There are multiple namespaces/workflows operating on shared physical table storage locations (for example, maintenance in one namespace and signoff/read workloads in another), which can overlap in time. What we are trying to understand Even with 14 days and retainLast 4000, we still see missing Puffin file references at read time. We need guidance on whether this is primarily a policy sizing issue, an execution-order/concurrency issue, or a known caveat around maintenance in multi-workload environments. Retention window vs retainLast interpretation we are using retention window protects snapshots newer than X days. retainLast protects the newest N snapshots regardless of age. Effective protection is whichever keeps more history for a given table churn rate. However, this alone does not appear sufficient to prevent missing Puffin failures when maintenance and readers overlap or when multiple actors can affect the same storage paths. Questions for maintainers What is the recommended policy framework for delete-vector tables to avoid this failure mode? How should retention window be sized relative to pipeline lag, retries, and restart windows? How should retainLast be sized relative to snapshot churn rate? Is it recommended to enforce strict maintenance windows with no active readers/writers? In shared-storage or multi-namespace environments, is a single-owner maintenance model strongly recommended? Are there known pitfalls combining expireSnapshots and orphan-file cleanup that can lead to missing Puffin files for active reads? Are there diagnostics or invariant checks we should run before and after expiration to catch unsafe conditions early? Operational policy we are considering Increase retention and retainLast beyond current 14 days and 4000 snapshots. Run expiration only in tightly controlled windows, gated by workload health. Prevent any out-of-band physical file deletion paths. Centralize maintenance ownership for each physical table location. Add preflight and postflight read validation around maintenance runs. If there is existing documentation or prior issue discussion for this exact Puffin missing-file pattern, please point us to it. We are happy to help improve docs once the recommended guardrails are clear. Thanks. ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
