I think I'm in favor of this but I would like some way of knowing whether or not a snapshot was produced while preserving row_ids or not. Just so we can make it clear on read what the row-lineage behavior of the writer was without knowing what system wrote the data.
On Thu, Mar 20, 2025 at 10:43 AM Ryan Blue <rdb...@gmail.com> wrote: > +1 for the PR and always having the lineage metadata. > > I think that is going to make the feature much more reliable. We don't > gain anything from allowing the feature to be turned off for compatibility, > when we have reasonable ways to interpret data written by any engine. > > Ryan > > On Wed, Mar 19, 2025 at 12:37 PM Daniel Weeks <dwe...@apache.org> wrote: > >> Hey everyone, >> >> When Row lineage was originally introduced, it was believed to be >> incompatible with equality deletes and we initially added lineage as a >> feature that could be turned on. Now that these features can co-exist >> <https://lists.apache.org/thread/vhl7p72433m904y115hmh7vnnbjdz4xn>, we >> would like to require lineage for v3 as there are benefits for feature >> enablement and adoption. >> >> We discussed this topic >> <https://www.youtube.com/watch?v=9BBZKTfcU0s&t=1475s> in the community >> sync, but I would like to raise this here on the dev list to have a broader >> discussion around the spec changes proposed in this PR >> <https://github.com/apache/iceberg/pull/12580>. >> >> The main points of discussion were around what implications this >> requirement has for writers, especially where tracking changes may be >> difficult/expensive. The proposal is to address this in the spec by >> clarifying the semantics of treating changes as upserts vs. delete/add is >> an engine implementation decision. The update to the spec would state that >> engines *should* track row ids through modification, but depending on >> the engine or the viability of tracking these changes, an engine may choose >> to model updates as deletes/adds. >> >> I'd love to get everyones thoughts/feedback on this proposal since we >> still have the opportunity to change this for v3, >> -Dan >> >
