0lai0 opened a new pull request, #1422:
URL: https://github.com/apache/mahout/pull/1422
### Related Issues
<!-- Closes #123 -->
Closes #1342
### Changes
- [ ] Bug fix
- [ ] New feature
- [ ] Refactoring
- [ ] Documentation
- [x] Test
- [x] CI/CD pipeline
- [ ] Other
### Why
<!-- Why is this change needed? -->
The f32 Parquet pipeline (A1–A3) lacked dedicated end-to-end fidelity
coverage and throughput validation. Without tests, regressions in the f32
reader or dispatch path could go undetected, and without a benchmark, the
expected 25–35% throughput improvement over the f64 path for native f32 columns
remained unmeasured.
### How
<!-- What was done? -->
- `qdp-core/tests/parquet_f32_fidelity.rs`: End-to-end f32-vs-f64 fidelity
test for the Parquet pipeline.
- CPU smoke(asserts in CI): f32 and f64 readers return the same logical
values up to f64→f32 rounding.
- GPU fidelity (Linux + CUDA, self-skip on no-CUDA stub via kernel probe):
same deterministic data written as native `List<Float32>` / `List<Float64>`,
read through matching readers, encoded with f32/f64 kernels, compared via
`fidelity_cross_precision`. Thresholds match `gpu_fidelity.rs` (1e-3 at 8–16
qubits; amplitude f32 norm precision — the issue's "≥0.99999 where applicable"
does not apply here).
- `qdp-core/tests/common/mod.rs`: Shared `List<Float32>` / `List<Float64>`
Parquet writers for fidelity fixtures.
- `qdp-python/benchmark/benchmark_parquet_f32.py` + `README.md`: Optional
local/GPU throughput benchmark (vec/s + f32/f64 speedup). Default run generates
separate native f32 and f64 files so the f32 path exercises zero-copy native
read; `--parquet PATH` benchmarks a single user file at both dtypes. Not wired
into CI.
- `.github/workflows/python-testing.yml`: Run `parquet_f32` (existing #1340
reader acceptance) and `parquet_f32_fidelity` in the `rust-check` job under
`QDP_NO_CUDA=1`. CPU smoke asserts; GPU cases skip without a GPU. Scoped to
these two binaries — other `gpu_*.rs` tests lack probe-skip and would fail on
the stub.
## Checklist
- [x] Added or updated unit tests for all changes
- [x] Added or updated documentation for all changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
