leerho opened a new issue, #192: URL: https://github.com/apache/datasketches-go/issues/192
The repository currently commits **737 `.sk` files (about 32 MB)** under `serialization_test_data/`: roughly 10.7 MB each of C++-, Go- and Java-generated snapshots. Because Go modules are served from the repository contents, these files are also part of every released module version (v0.1.0, v0.2.0). The other DataSketches implementations keep snapshots out of their repositories and releases: - **datasketches-java** ignores `**/*.sk` and `serialization_test_data/`, excludes `serialization_test_data/**` from its source release, and fetches the C++ and Go snapshots from a pinned [datasketches-tck](https://github.com/apache/datasketches-tck) revision with `tools/download_serialization_test_data.sh`. - **datasketches-cpp** is doing the same (apache/datasketches-cpp#535). Proposal: 1. Remove the TCK-generated snapshots (`serialization_test_data/{cpp,go,java}_generated_files/`) from the repository. 2. Add `*.sk` and `serialization_test_data/` to `.gitignore`. The current `.gitignore` only has `.idea` and `coverage.out`. 3. Fetch the Java and C++ snapshots from datasketches-tck before running the cross-language tests, locally, in CI and in `build/Dockerfile`. A small script like Java's would do. Go's own generated files are written locally, as they are now. 4. Decide where the six hand-made fixtures at the top of `serialization_test_data/` should live. These are the `tdigest_ref_*` files and the legacy `theta_compact_*_from_java_v1/v2` images, which the TCK generators don't produce. They could become TCK "special" snapshots, or stay in the repository as a small, explicitly allowed set. This keeps the repository and its releases small, gives every implementation one source of truth for cross-language data, and matches the layout used across the project (see apache/datasketches-tck#11 on aligning the TCK layout too). If you're OK with this plan but pressed for time, let us know and we can prepare the PRs for these changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
