AndreaBozzo opened a new issue, #10577: URL: https://github.com/apache/arrow-rs/issues/10577
### Is your feature request related to a problem or challenge? `ReaderBuilder::with_truncated_rows(true)` repairs a short row by padding it, but there is no way for a caller to learn that it happened. For anything that reports on data quality, a repaired parse and a clean parse are different outcomes, and right now they are indistinguishable. The information is not merely unexposed — it is destroyed. Padding fills the offsets to produce **zero-length fields** ([`records.rs`, the `self.current_field < self.num_columns` branch](https://github.com/apache/arrow-rs/blob/main/arrow-csv/src/reader/records.rs)), and the null regex later turns those into nulls. By the time a `RecordBatch` exists, a padded field is byte-identical to a genuinely empty trailing field: ``` name,age,city Alice,25,Rome -> ["Alice", "25", "Rome"] Carol,35, -> ["Carol", "35", NULL] <- three fields, last one empty Bob,30 -> ["Bob", "30", NULL] <- two fields, padded ``` Full repro (arrow-csv 59.1.0, output above is real): ```rust use arrow_array::{Array, StringArray}; use arrow_csv::ReaderBuilder; use arrow_schema::{DataType, Field, Schema}; use std::io::Cursor; use std::sync::Arc; fn main() { let schema = Arc::new(Schema::new(vec![ Field::new("name", DataType::Utf8, true), Field::new("age", DataType::Utf8, true), Field::new("city", DataType::Utf8, true), ])); let csv = "name,age,city\nAlice,25,Rome\nCarol,35,\nBob,30\n"; let reader = ReaderBuilder::new(schema) .with_header(true) .with_truncated_rows(true) .build(Cursor::new(csv)) .unwrap(); for batch in reader { let batch = batch.unwrap(); for row in 0..batch.num_rows() { let cells: Vec<String> = (0..batch.num_columns()) .map(|c| { let a = batch.column(c).as_any().downcast_ref::<StringArray>().unwrap(); if a.is_null(row) { "NULL".into() } else { format!("{:?}", a.value(row)) } }) .collect(); println!("row {row}: [{}]", cells.join(", ")); } } // Nothing here, or anywhere on the reader, reports that row 2 was padded. } ``` ### Describe the solution you'd like `RecordDecoder` already knows exactly when it pads — it is the branch quoted above. A counter incremented there, plus an accessor on `Decoder` / `Reader` / `BufReader`, would be enough: ```rust /// The number of rows padded because they had fewer fields than the schema. /// /// Always 0 unless `with_truncated_rows(true)` was set. pub fn truncated_row_count(&self) -> usize ``` Purely additive, and it costs nothing when the feature is off. ### Describe alternatives you've considered - **Inferring it from nulls in the batch.** Not possible, per the repro: a padded field and an empty trailing field are the same null. - **Disabling the null regex** (as in #6874). Doesn't help — with the regex off both cases become `""` instead of `NULL`, still identical. - **Pre-scanning the file with the `csv` crate.** What we ended up doing. It works and is cheaper than expected (~3% of profiling wall time on a 218 MB file, since the analysis dominates the parse), but it is a second full read of the input purely to recover a number the decoder already had. ### Additional context We hit this in [dataprof](https://github.com/AndreaBozzo/dataprof), a data profiler whose contract is that a repaired scan must never be reported as a clean one. Our Arrow-backed CSV engine was the only path that could report a short-row file with a perfect consistency score ([#470](https://github.com/AndreaBozzo/dataprof/issues/470)). Related but separate: `with_truncated_rows` is asymmetric — it recovers rows with *too few* fields, but a row with *too many* is always an error, with no equivalent opt-in to drop the surplus. Handling both directions currently means pre-scanning for the widest record and padding the schema out to it, then projecting the extra columns away. Happy to open that separately if it is worth its own issue. I'm willing to put up a PR for the counter if the approach sounds right. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
