Steven, that may be a good point to add to ensure the metadata is properly maintained. If I remember correctly, the Spark implementation already drops old DVs in DELETE/UPDATE/MERGE but the data compaction wasn't doing it originally. I wonder if we fixed it. Eduard may know more.
- Anton ср, 7 трав. 2025 р. о 16:29 Steven Wu <stevenz...@gmail.com> пише: > For the delete vection change, should we add the following > constraint/requirement for the write path in the spec? I don't know if this > is already the behavior of the Spark implementation. > > "if a data file is removed from the table, the corresponding DV reference > must also be removed from delete manifest file" > > This constraint is to guarantee no orphaned DVs in the table state. It > will be cheaper to calculate *accurate* table row count. Just iterate > through the manifest files (data and delete) using add and subtraction > calculations. There is no need to validate DVs if the referenced data files > are still part of the table, which can be a little more expensive. > > > > On Tue, May 6, 2025 at 9:18 AM Manu Zhang <owenzhang1...@gmail.com> wrote: > >> Thanks for clarification Ryan. >> >> I'm aware of the major changes, but I find it hard to go through all the >> related descriptions which are scattered all over the place. >> >> Manu >> >> On Tue, May 6, 2025 at 11:24 PM Ryan Blue <rdb...@gmail.com> wrote: >> >>> Manu, >>> >>> We aren't currently voting. We are discussing any outstanding items to >>> address before we close v3 to further changes and adopt the existing v3 >>> changes. Right now, the open item is to clarify NaN behavior in geometry >>> and geography, PR #12956 <https://github.com/apache/iceberg/pull/12956>. >>> >>> Thanks for noting that the row lineage changes should be added to the >>> appendix, I'll open a PR to add it. That appendix is an area to highlight >>> things that have changed across versions, but an omission does not alter >>> the requirements elsewhere the spec. The changes we are discussing are the >>> things that are noted as part of v3 in the spec. The major additions are >>> new types, DVs, and row lineage. >>> >>> Ryan >>> >>> On Tue, May 6, 2025 at 3:32 AM Manu Zhang <owenzhang1...@gmail.com> >>> wrote: >>> >>>> I'm wondering what changes we are voting for here. Is it everything >>>> related to >>>> https://iceberg.apache.org/spec/#version-3-extended-types-and-capabilities >>>> from >>>> the table spec? >>>> How about changes to other specs? >>>> >>>> Do we summarize all the changes in >>>> https://iceberg.apache.org/spec/#appendix-e-format-version-changes? It >>>> looks row lineage is missing here. >>>> >>>> Thanks, >>>> Manu >>>> >>>> On Tue, May 6, 2025 at 12:09 PM Anton Okolnychyi <aokolnyc...@gmail.com> >>>> wrote: >>>> >>>>> DVs in Spark seem to behave reasonably, serving as a reference >>>>> implementation of the V3 spec. There are areas for optimization/refinement >>>>> but nothing was observed that requires changing the spec. I would also >>>>> like >>>>> to add the notion of content overhead/metadata (for Puffin/Parquet >>>>> footers) >>>>> to manifests to optimize DVs maintenance. That said, it is optional >>>>> information and can be added after finalizing V3. >>>>> >>>>> - Anton >>>>> >>>>> пт, 2 трав. 2025 р. о 23:23 Jean-Baptiste Onofré <j...@nanthrax.net> >>>>> пише: >>>>> >>>>>> Hi Ryan >>>>>> >>>>>> All good for the spec. The idea for release is just a help to "double >>>>>> check" the spec is good (we already saw some slightly changes on the >>>>>> spec while working on release). I think we can be "confident" that we >>>>>> won't have unexpected change. >>>>>> >>>>>> Thanks ! >>>>>> Regards >>>>>> JB >>>>>> >>>>>> On Thu, May 1, 2025 at 7:04 PM Ryan Blue <rdb...@gmail.com> wrote: >>>>>> > >>>>>> > Thanks, everyone! Looks like there are a few points to discuss. >>>>>> > >>>>>> > [JB] Maybe a release with the core updated before announcing spec >>>>>> v3 officially would be a good idea ? >>>>>> > [Manu] Agree with Russell and JB that we make a “RC” release for V3 >>>>>> spec to test implementations, compatibility, etc before finalizing it. >>>>>> > >>>>>> > As Fokko noted, we are currently concerned about the spec and not >>>>>> implementations. The reason is that implementation work before the spec >>>>>> is >>>>>> finalized is to reduce risk and build confidence that the spec is >>>>>> complete >>>>>> and correct. Once that’s done, it is important to finalize the changes. >>>>>> If >>>>>> we don’t finalize the changes, then implementations don’t know how/what >>>>>> build and cannot plan when they will fully support v3 — because it could >>>>>> change. Most of the work in other implementations will take place after >>>>>> the >>>>>> spec is adopted. >>>>>> > >>>>>> > Our process for building confidence in new spec versions is to >>>>>> update the spec with pending changes, implement them to validate (and >>>>>> clarify or adjust as needed), and vote to adopt the new version as a >>>>>> confirmation that we agree that the spec changes are reasonable and >>>>>> correct. >>>>>> > >>>>>> > We’ve already voted to accept the pending v3 changes into the spec, >>>>>> so the changes have already been in a candidate state for quite some time >>>>>> to work on implementations. Now we’re at the point where we’ve >>>>>> implemented >>>>>> the features and, in my opinion, have demonstrated the spec changes are >>>>>> correct and complete. >>>>>> > >>>>>> > To that end, the question I’m raising in this thread is “what areas >>>>>> and features need further validation?” >>>>>> > >>>>>> > I appreciate the ideas here — releasing will assist other >>>>>> implementations — but I don’t think that changes the question for this >>>>>> thread. The aim is to identify specific risks and blockers that we need >>>>>> to >>>>>> tackle before adopting the changes. >>>>>> > >>>>>> > [Russell] We should probably come to a resolution on the compressed >>>>>> metadata.json name as well, although that’s mostly retroactive. V3 would >>>>>> be >>>>>> the place where we could officially change the naming convention. >>>>>> > >>>>>> > I don’t think that this affects v3, but we should agree before >>>>>> moving on. The only part of the spec that would depend on this is the >>>>>> paths >>>>>> used by file system tables and that strategy is deprecated. We should >>>>>> only >>>>>> document for clarify (we can’t change it) and I think we can do that any >>>>>> time. >>>>>> > >>>>>> > For the conventions used in catalog tables, I don’t think that we >>>>>> want to have requirements in the spec for file naming. We’ve avoided that >>>>>> in the past and it isn’t needed. It’s nice to have a convention in >>>>>> implementation notes, but there are other ways to handle this like magic >>>>>> bytes and catalog tracking. >>>>>> > >>>>>> > [Gang] it is implicit and obvious that only bucket transform can >>>>>> apply to multi-arg transform, it is still unclear the order of source >>>>>> columns and algorithm to use to calculate the bucket value >>>>>> > >>>>>> > I think there is some confusion here, but Fokko may have already >>>>>> cleared it up. >>>>>> > >>>>>> > Right now, there are no multi-argument transforms in the spec. We >>>>>> have discussed adding a multi-argument bucket function, but there is not >>>>>> currently one in the spec. In order to minimize changes required for v3, >>>>>> we >>>>>> opted to update the spec to allow adding new transforms in a >>>>>> forward-compatible way between major spec versions (implementations must >>>>>> ignore unknown transforms). >>>>>> > >>>>>> > [Jia] We’re currently addressing the handling of null/NaN values >>>>>> for X, Y, Z, and M coordinates in the Parquet format repository >>>>>> > >>>>>> > I agree that this is a good thing to clarify. We currently state >>>>>> that the ranges are [-180, 180] and [-90, 90] for geography, but we >>>>>> should >>>>>> state how points with NaN values are handled. >>>>>> > >>>>>> > >>>>>> > On Wed, Apr 30, 2025 at 12:27 PM Szehon Ho <szehon.apa...@gmail.com> >>>>>> wrote: >>>>>> >> >>>>>> >> Hi Jia >>>>>> >> >>>>>> >> I feel it would be nice to get that Parquet spec clarificiation >>>>>> https://github.com/apache/parquet-format/pull/494 into Iceberg V3 >>>>>> spec as well, once we finalize that. >>>>>> >> >>>>>> >> Thanks >>>>>> >> Szehon >>>>>> >> >>>>>> >> On Tue, Apr 29, 2025 at 10:55 PM Jia Yu <ji...@apache.org> wrote: >>>>>> >>> >>>>>> >>> Hi Szehon, >>>>>> >>> >>>>>> >>> Thanks for clarifying it. >>>>>> >>> >>>>>> >>> We’re currently addressing the handling of null/NaN values for X, >>>>>> Y, Z, and M coordinates in the Parquet format repository. We’ve already >>>>>> concluded that the spec of Parquet (same on the Iceberg side I believe) >>>>>> only needs additional clarification to guide expected behavior: >>>>>> https://github.com/apache/parquet-format/pull/494 >>>>>> >>> >>>>>> >>> BTW the Parquet Geo C++ PR has been merged today: >>>>>> https://github.com/apache/arrow/pull/45459 I believe the Parquet >>>>>> Geo Java PR is also very close. >>>>>> >>> >>>>>> >>> Thanks, >>>>>> >>> Jia >>>>>> >>> >>>>>> >>> On Tue, Apr 29, 2025 at 10:48 PM Fokko Driesprong < >>>>>> fo...@apache.org> wrote: >>>>>> >>>> >>>>>> >>>> Hey Ryan, >>>>>> >>>> >>>>>> >>>> Thanks for raising this, and I'm very excited to see V3 being >>>>>> finalized! >>>>>> >>>> >>>>>> >>>>> The v3 spec for multi-arg transform only advises to use >>>>>> `source-ids` instead of `source-id`. Although it is implicit and obvious >>>>>> that only bucket transform can apply to multi-arg transform, it is still >>>>>> unclear the order of source columns and algorithm to use to calculate the >>>>>> bucket value. >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> V3 now uses source IDs when there are multiple arguments and >>>>>> source IDs when there is just one. PR can be found here. This makes the >>>>>> serialization deterministic without knowing the format-version, >>>>>> simplifying >>>>>> the readers/writers. After some discussion on the PR, we've decided to >>>>>> leave out the multi-arg bucket transform so the V3 spec can be finalized. >>>>>> So V3 only contains the scaffolding for multi-arg transforms. >>>>>> >>>> >>>>>> >>>>> For Iceberg Geo, we are still waiting for the PR of geospatial >>>>>> bounds and geospatial predicate to be merged: >>>>>> https://github.com/apache/iceberg/pull/12667 >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> I think it is a good idea to distinguish between the spec and >>>>>> the actual code. If we all feel comfortable with the spec, I think we >>>>>> could >>>>>> finalize it. Being comfortable also means that we know that we have a >>>>>> working implementation, but I don't think we have to wrap up all the >>>>>> loose >>>>>> ends before voting on the spec. >>>>>> >>>> >>>>>> >>>> At the PyIceberg side, we're also working to catch up on the V3 >>>>>> capabilities. Having a Java release that exposes these capabilities >>>>>> helps, >>>>>> so we can do round-trip validation. >>>>>> >>>> >>>>>> >>>> Kind regards, >>>>>> >>>> Fokko >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> Op wo 30 apr 2025 om 07:26 schreef Jia Yu <ji...@apache.org>: >>>>>> >>>>> >>>>>> >>>>> Hi folks, >>>>>> >>>>> >>>>>> >>>>> For Iceberg Geo, we are still waiting for the PR of geospatial >>>>>> bounds and geospatial predicate to be merged: >>>>>> https://github.com/apache/iceberg/pull/12667 >>>>>> >>>>> >>>>>> >>>>> Should a release with core updates include this PR? >>>>>> >>>>> >>>>>> >>>>> Thanks, >>>>>> >>>>> Jia >>>>>> >>>>> >>>>>> >>>>> On Tue, Apr 29, 2025 at 10:21 PM Manu Zhang < >>>>>> owenzhang1...@gmail.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> Agree with Russell and JB that we make a "RC" release for V3 >>>>>> spec to test implementations, compatibility, etc before finalizing it. >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Manu >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Apr 30, 2025 at 12:24 PM Jean-Baptiste Onofré < >>>>>> j...@nanthrax.net> wrote: >>>>>> >>>>>>> >>>>>> >>>>>>> Hi Ryan >>>>>> >>>>>>> >>>>>> >>>>>>> It sounds good. >>>>>> >>>>>>> >>>>>> >>>>>>> About multi-args transforms, with the clarification we did a >>>>>> couple of weeks ago, I think we are good. >>>>>> >>>>>>> Maybe a release with the core updated before announcing spec >>>>>> v3 officially would be a good idea ? >>>>>> >>>>>>> >>>>>> >>>>>>> Regards >>>>>> >>>>>>> JB >>>>>> >>>>>>> >>>>>> >>>>>>> Le mer. 30 avr. 2025 à 00:35, Ryan Blue <rdb...@gmail.com> a >>>>>> écrit : >>>>>> >>>>>>>> >>>>>> >>>>>>>> Hi everyone, >>>>>> >>>>>>>> >>>>>> >>>>>>>> I think we’ve reached the point where it’s time to finalize >>>>>> and adopt the changes for Iceberg v3. We’ve been working toward this for >>>>>> the last few months and have now implemented the v3 features in the Java >>>>>> library to reduce the risk of needing changes or hitting problems (row >>>>>> lineage support in Spark 3.5 just went in!). We’ve also incorporated some >>>>>> clarifications and minor changes back into the spec from what we’ve >>>>>> learned. >>>>>> >>>>>>>> >>>>>> >>>>>>>> At this point, I’m confident that the spec is reasonable and >>>>>> correct. Thank you to everyone working on these reference >>>>>> implementations! >>>>>> >>>>>>>> >>>>>> >>>>>>>> The next step is to discuss any outstanding items or >>>>>> concerns about moving forward, and then to have a vote thread to adopt >>>>>> the >>>>>> spec. I’ll start off with a couple of items: >>>>>> >>>>>>>> >>>>>> >>>>>>>> One potential concern is that the upstream Variant spec >>>>>> hasn’t yet been finalized by the Parquet community, but we’ve built a >>>>>> full, >>>>>> independent implementation in Iceberg to validate the spec. I think the >>>>>> Parquet community is primarily waiting on getting the PRs in to have a >>>>>> Java >>>>>> reference implementation, so the risk of changes to the Variant spec is >>>>>> small. >>>>>> >>>>>>>> >>>>>> >>>>>>>> There’s also an on-going vote to add encryption keys in >>>>>> support of full table encryption that I think we want to get in. >>>>>> >>>>>>>> >>>>>> >>>>>>>> Any other items we may want to clear up? >>>>>> >>>>>>>> >>>>>> >>>>>>>> Ryan >>>>>> >>>>>
