zeroshade commented on issue #1929: URL: https://github.com/apache/iceberg-go/issues/1929#issuecomment-5455469878
> Theta sketch serialization: The blob must be byte-compatible with Apache DataSketches so Java/Spark can read it. I don't see a maintained Go port of the compact Theta format, so my lean is to implement the compact serialization in-tree (bounded, and spec'd by DataSketches) and validate byte-for-byte against Java-produced fixtures — the same approach the DV code uses against Java. Flagging in case there's a preferred dependency I should use instead. What about https://pkg.go.dev/github.com/apache/[email protected]/theta#CompactSketch? The plan looks good and I agree with @laskoviymishka that the PR breakown looks good. > I'd bake the Java-authored bytes into the repo and keep the tests Java-less, rather than standing up a JVM on the test path. That's already the pattern the DV code uses, and I think it's worth cloning that approach exactly: the Puffin fixtures live under table/dv/testdata, and the x-client tests read those bytes back through the reader with no Java at test time. I think we should do both. We have the `integration` tag for integration tests where we stand up a spark image and run queries hitting iceberg-java. We should have Java-authored bytes as testdata and keep the unit-tests Java-less, but we should also add to the integration tests. > Separately, cc @nssalian: this is exactly the x-client byte parity that apache/iceberg-verification is for (see: [VOTE](https://lists.apache.org/thread/98ntkfmtrqhfj5w54x27tnxfq14fftqb)). I think the Theta/NDV parity vectors are a strong early candidate to live in the verification repo. Precisely! I agree completely here. My preference would actually be to add the Java-authored bytes and integration test strategy into `apache/iceberg-verification` and then just add it as a submodule here to use (similar to how github.com/apache/arrow-go uses github.com/apache/arrow-testing and github.com/apache/parquet-testing submodules with test files) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
