ryankert01 commented on code in PR #1407:
URL: https://github.com/apache/mahout/pull/1407#discussion_r3448687153
##########
qdp/qdp-core/src/pipeline_runner.rs:
##########
@@ -389,28 +428,67 @@ fn path_extension_lower(path: &Path) -> Option<String> {
.map(|s| s.to_lowercase())
}
-/// Dispatches by path extension to the appropriate io reader. Returns (data,
num_samples, sample_size).
-/// Unsupported or missing extension returns Err with message listing
supported formats.
+/// f64→f32 narrowing cast: values outside f32 range silently become ±Inf.
+fn cast_f64_to_batch_data(
+ data: Vec<f64>,
+ n: usize,
+ s: usize,
+ dtype: Precision,
+ fmt: &str,
+) -> (BatchData, usize, usize) {
+ if matches!(dtype, Precision::Float32) {
+ log::warn!(
+ "{fmt} file loaded as f64, casting to f32: values outside f32
range become ±Inf."
+ );
+ (
+ BatchData::F32(data.iter().map(|&x| x as f32).collect()),
+ n,
+ s,
+ )
+ } else {
+ (BatchData::F64(data), n, s)
+ }
+}
+
fn read_file_by_extension(
path: &Path,
null_handling: NullHandling,
-) -> Result<(Vec<f64>, usize, usize)> {
+ dtype: Precision,
+) -> Result<(BatchData, usize, usize)> {
+ use crate::reader::DataReader;
let ext_lower = path_extension_lower(path);
let ext = ext_lower.as_deref();
match ext {
Some("parquet") => {
- use crate::reader::DataReader;
- let mut reader = crate::readers::ParquetReader::new(path, None,
null_handling)?;
- reader.read_batch()
+ if matches!(dtype, Precision::Float32) {
Review Comment:
[blocking] This f32 dispatch also applies when the pipeline encoding is
`basis`. Basis indices are semantically integers, and valid indices above
`2^24` are not exactly representable as `f32` (`16_777_217` becomes
`16_777_216`). Because the f32 basis encoder validates after narrowing, this
can silently encode the wrong state. Please keep basis file batches in
f64/integer form, or reject f32 basis indices when exactness is not guaranteed.
##########
qdp/qdp-core/src/pipeline_runner.rs:
##########
@@ -389,28 +428,67 @@ fn path_extension_lower(path: &Path) -> Option<String> {
.map(|s| s.to_lowercase())
}
-/// Dispatches by path extension to the appropriate io reader. Returns (data,
num_samples, sample_size).
-/// Unsupported or missing extension returns Err with message listing
supported formats.
+/// f64→f32 narrowing cast: values outside f32 range silently become ±Inf.
+fn cast_f64_to_batch_data(
+ data: Vec<f64>,
+ n: usize,
+ s: usize,
+ dtype: Precision,
+ fmt: &str,
+) -> (BatchData, usize, usize) {
+ if matches!(dtype, Precision::Float32) {
+ log::warn!(
+ "{fmt} file loaded as f64, casting to f32: values outside f32
range become ±Inf."
+ );
+ (
+ BatchData::F32(data.iter().map(|&x| x as f32).collect()),
+ n,
+ s,
+ )
+ } else {
+ (BatchData::F64(data), n, s)
+ }
+}
+
fn read_file_by_extension(
path: &Path,
null_handling: NullHandling,
-) -> Result<(Vec<f64>, usize, usize)> {
+ dtype: Precision,
+) -> Result<(BatchData, usize, usize)> {
+ use crate::reader::DataReader;
let ext_lower = path_extension_lower(path);
let ext = ext_lower.as_deref();
match ext {
Some("parquet") => {
- use crate::reader::DataReader;
- let mut reader = crate::readers::ParquetReader::new(path, None,
null_handling)?;
- reader.read_batch()
+ if matches!(dtype, Precision::Float32) {
Review Comment:
One concern here: this f32 path also covers `basis`, where the values are
indices rather than ordinary float features. A valid index like `16_777_217`
for 25 qubits is not exactly representable as `f32`, so it becomes
`16_777_216`; the basis f32 encoder then validates the rounded value and
encodes the wrong state. Can we keep basis file input in f64/integer form, or
reject f32 basis input once indices exceed the exact f32 range?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
