tanmayrauth commented on PR #2444:
URL: https://github.com/apache/iceberg-go/pull/2444#issuecomment-6909970724
Thanks @laskoviymishka, all six threads are addressed in the latest push.
ComputeNDVs now returns NDVStats{Snapshot, Columns}, so the Puffin writer gets
the snapshot ID and sequence number without resolving it again; Snapshot is
nil only when the table has no snapshot, in which case the result is empty like
Java's EMPTY_RESULT instead of an NDV=0 sketch per column. Column names are
resolved against the snapshot schema first, and only the columns being
sketched go to WithSelectedFields, so nested and geo columns are no longer
read; deletes on unselected columns still apply. Geo columns are skipped by
default and rejected with ErrInvalidArgument when named, because Java's
Conversions serializes geometry and geography only as a GeospatialBound point,
so there's no shared single-value encoding for Go-only WKB sketches to match.
Since names are checked before the scan is built, every unknown name, "*"
included, now returns ErrInvalidArgument instead of some returning ErrInvalidS
chema from Schema.Select, which matches Java's checkArgument.
The matrix is in this PR: TestComputeNDVsTypes covers 14 primitive types,
each checked against a sketch built from Iceberg literals (pinned to the Java
fixtures), and TestComputeNDVsSnapshotContents covers several data files,
position deletes, deletion vectors, equality deletes, and an older snapshot
after a schema change, with each delete case checking that real delete files
are present. One thing it turned up: empty strings and empty binary values
aren't counted, because DataSketches ignores empty byte arrays, and Java's
string fixture behaves the same way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]