tanmayrauth commented on issue #1929:
URL: https://github.com/apache/iceberg-go/issues/1929#issuecomment-5457992182
Thanks @zeroshade @laskoviymishka, this is great feedback.
Good call on `datasketches-go`, I'd missed it, and it saves me hand-porting
the serialization, which was the scary part. Had a look and it has everything:
the update sketch for the compute pass, Union for merging across files, and
CompactSketch/Encoder/Decode for the bytes. Reassuringly it defaults to seed
9001, same as Java's DEFAULT_UPDATE_SEED, so cross-reads should Just Work. That
folds the first two PRs into one ("adopt the lib + prove parity")
The thing I want to nail down in that first PR is exactly which
serialization the theta-v1 blob uses. I'm assuming the plain uncompressed v3
form (Java's CompactSketch.toByteArray()), but I'll confirm it by
round-tripping against a blob Java actually wrote rather than guessing.
On testing: yep, cloning the DV setup: Java-authored bytes checked in as
testdata, unit tests with no JVM in CI. And agreed we should do both, so I'll
add an integration-tagged test against the Spark image too, so we catch
anything the static fixtures miss.
On where the vectors live: totally agree they belong in iceberg-verification
as a submodule down the line (the arrow-testing / parquet-testing setup is a
nice model). Since that repo isn't up yet I'll start them under table/testdata
so this isn't blocked, and move them over once it exists. One thing I'll make
sure of: I'll generate those vectors straight from iceberg-java's own write
path, so they're the same canonical bytes the verification repo would end up
hosting, not a separate set that could drift. Happy to seed the Theta/NDV
vectors there and line up the recipe (dataset, lgK, seed, serial version)
whenever it's ready, @nssalian, tell me if I can help get it going.
Going to start on PR 1. Thanks again for the quick turnaround!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
