Hi Szehon, Thanks for clarifying it.
We’re currently addressing the handling of null/NaN values for X, Y, Z, and M coordinates in the Parquet format repository. We’ve already concluded that the spec of Parquet (same on the Iceberg side I believe) only needs additional clarification to guide expected behavior: https://github.com/apache/parquet-format/pull/494 BTW the Parquet Geo C++ PR has been merged today: https://github.com/apache/arrow/pull/45459 I believe the Parquet Geo Java PR is also very close. Thanks, Jia On Tue, Apr 29, 2025 at 10:48 PM Fokko Driesprong <fo...@apache.org> wrote: > Hey Ryan, > > Thanks for raising this, and I'm very excited to see V3 being finalized! > > The v3 spec for multi-arg transform only advises to use `source-ids` >> instead of `source-id`. Although it is implicit and obvious that only >> bucket transform can apply to multi-arg transform, it is still unclear the >> order of source columns and algorithm to use to calculate the bucket value. >> > > V3 now uses source IDs when there are multiple arguments and source IDs > when there is just one. PR can be found here > <https://github.com/apache/iceberg/pull/12644>. This makes the > serialization deterministic without knowing the format-version, simplifying > the readers/writers. After some discussion on the PR, we've decided to > leave out the multi-arg bucket transform so the V3 spec can be finalized. > So V3 only contains the scaffolding for multi-arg transforms. > > For Iceberg Geo, we are still waiting for the PR of geospatial bounds and >> geospatial predicate to be merged: >> https://github.com/apache/iceberg/pull/12667 > > > I think it is a good idea to distinguish between the spec and the actual > code. If we all feel comfortable with the spec, I think we could finalize > it. Being comfortable also means that we know that we have a working > implementation, but I don't think we have to wrap up all the loose ends > before voting on the spec. > > At the PyIceberg side, we're also working to catch up on the V3 > capabilities <https://github.com/apache/iceberg-python/issues/1818>. > Having a Java release that exposes these capabilities helps, so we can do > round-trip validation. > > Kind regards, > Fokko > > > Op wo 30 apr 2025 om 07:26 schreef Jia Yu <ji...@apache.org>: > >> Hi folks, >> >> For Iceberg Geo, we are still waiting for the PR of geospatial bounds and >> geospatial predicate to be merged: >> https://github.com/apache/iceberg/pull/12667 >> >> Should a release with core updates include this PR? >> >> Thanks, >> Jia >> >> On Tue, Apr 29, 2025 at 10:21 PM Manu Zhang <owenzhang1...@gmail.com> >> wrote: >> >>> Agree with Russell and JB that we make a "RC" release for V3 spec to >>> test implementations, compatibility, etc before finalizing it. >>> >>> Thanks, >>> Manu >>> >>> On Wed, Apr 30, 2025 at 12:24 PM Jean-Baptiste Onofré <j...@nanthrax.net> >>> wrote: >>> >>>> Hi Ryan >>>> >>>> It sounds good. >>>> >>>> About multi-args transforms, with the clarification we did a couple of >>>> weeks ago, I think we are good. >>>> Maybe a release with the core updated before announcing spec v3 >>>> officially would be a good idea ? >>>> >>>> Regards >>>> JB >>>> >>>> Le mer. 30 avr. 2025 à 00:35, Ryan Blue <rdb...@gmail.com> a écrit : >>>> >>>>> Hi everyone, >>>>> >>>>> I think we’ve reached the point where it’s time to finalize and adopt >>>>> the changes for Iceberg v3. We’ve been working toward this for the last >>>>> few >>>>> months and have now implemented the v3 features in the Java library to >>>>> reduce the risk of needing changes or hitting problems (row lineage >>>>> support >>>>> in Spark 3.5 just went in!). We’ve also incorporated some clarifications >>>>> and minor changes back into the spec from what we’ve learned. >>>>> >>>>> At this point, I’m confident that the spec is reasonable and correct. >>>>> Thank you to everyone working on these reference implementations! >>>>> >>>>> The next step is to discuss any outstanding items or concerns about >>>>> moving forward, and then to have a vote thread to adopt the spec. I’ll >>>>> start off with a couple of items: >>>>> >>>>> One potential concern is that the upstream Variant spec hasn’t yet >>>>> been finalized by the Parquet community, but we’ve built a full, >>>>> independent implementation in Iceberg to validate the spec. I think the >>>>> Parquet community is primarily waiting on getting the PRs in to have a >>>>> Java >>>>> reference implementation, so the risk of changes to the Variant spec is >>>>> small. >>>>> >>>>> There’s also an on-going vote to add encryption keys in support of >>>>> full table encryption that I think we want to get in. >>>>> >>>>> Any other items we may want to clear up? >>>>> >>>>> Ryan >>>>> >>>>
