Hi Steven, There are constraints about Spark cherrypick_snapshot and publish_changes procedure https://iceberg.apache.org/docs/nightly/spark-procedures/#output_3 * for cherrypick_snapshot operation, it says "Only append and dynamic overwrite snapshots can be cherry-picked.". * for publish_changes operation, it says "Only append and dynamic overwrite snapshots can be successfully published."
So I was hoping we can have support for cherrypick static overwrite snapshot capabilities. Sorry that I made you confused. On Thu, May 30, 2024 at 1:08 PM Steven Wu <stevenz...@gmail.com> wrote: > Pucheng, > > I am not sure about others. At least I had some hard time understanding > what the problem/proposal is. What is "cherrypick static partition > overwrite"? > > Thanks, > Steven > > On Thu, May 30, 2024 at 11:59 AM Pucheng Yang <py...@pinterest.com.invalid> > wrote: > >> Hi community, >> >> I would like to follow up on this proposal and would like to check if >> anyone has concerns about the proposed implementation from a high level >> perspective? >> >> Thank you very much >> >> Best, >> Pucheng >> >> On Tue, May 28, 2024 at 9:02 PM Pucheng Yang <py...@pinterest.com> wrote: >> >>> Hi community, >>> >>> My client is looking for the support of cherrypick static >>> partition overwrite. Based on my understanding, the reason we can not do it >>> is because we do not preserve static overwrite filters. >>> >>> I would like to make a proposal to support cherrypick static overwrite: >>> 1. We will allow users to opt in preserving static overwrite filters >>> during snapshot summary >>> 2. When the snapshot is being cherry picked, we will first delete from >>> the current snapshot using the static overwrite fitler preserved in the >>> snapshot, then we will redo the add files operation. >>> >>> Please share if you have any thoughts or comments on this, if we agree, >>> I can open up a PR for this. >>> >>> Best, >>> Pucheng >>> >>
