This is an automated email from the ASF dual-hosted git repository.
Jefffrey pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 46a16a5dd2 fix(csv): Infer leading-plus integers and decimals (#11105)
46a16a5dd2 is described below
commit 46a16a5dd2993f1edecfca5561cc9b9253167085
Author: Johan Vaz <[email protected]>
AuthorDate: Thu Sep 17 10:27:01 2026 +0800
fix(csv): Infer leading-plus integers and decimals (#11105)
# Which issue does this PR close?
Related to #6882 and the review of #10361. This is the separate
schema-inference change requested in that review; it does not implement
header detection or close #6882.
# Rationale for this change
CSV numeric parsers accept a leading plus sign, but schema inference
currently classifies `+1` and `+1.5` as UTF-8. A column containing these
values can therefore be inferred as text even when the same input
decodes with an explicit numeric schema.
# What changes are included in this PR?
Allow one optional `+` or `-` in the existing integer and
decimal/scientific-notation inference regexes. Preserve the existing
integer overflow fallback and rejection of malformed signs. No
header-detection API or other inference rules are changed.
# Are these changes tested?
The regression covers signed integer, decimal and scientific notation,
the signed i64 maximum, overflow to UTF-8, and malformed signs. It also
decodes each input using the inferred schema and checks that every
sampled row is retained.
Validated on `984be43ee` against upstream `c60ac3fc6` with the
repository-pinned Rust 1.98.1:
- `cargo test -p arrow-csv --all-features`: 86 unit tests and 13
doctests passed.
- `cargo test -p arrow --features csv`: 159 unit/integration tests and
11 doctests passed.
- `cargo clippy -p arrow-csv --all-targets --all-features -- -D
warnings`: passed.
- `cargo fmt --all -- --check` and `git diff upstream/main --check`:
passed.
# Are there any user-facing changes?
Yes. CSV columns with supported leading-plus numeric values can now
infer as `Int64` or `Float64` instead of `Utf8`. Applications needing
text can continue to supply an explicit schema. Oversized integers
remain text.
# AI assistance disclosure
AI assistance was used for the extracted fix, regression tests, and this
description. The diff was inspected against current upstream and the
listed checks were run locally. Maintainer acceptance is pending.
---
arrow-csv/src/reader/mod.rs | 28 ++++++++++++++++++++++++++--
1 file changed, 26 insertions(+), 2 deletions(-)
diff --git a/arrow-csv/src/reader/mod.rs b/arrow-csv/src/reader/mod.rs
index 3b7cce2dc3..df4c450c14 100644
--- a/arrow-csv/src/reader/mod.rs
+++ b/arrow-csv/src/reader/mod.rs
@@ -186,8 +186,8 @@ use arrow_array::timezone::Tz;
static REGEX_SET: LazyLock<RegexSet> = LazyLock::new(|| {
RegexSet::new([
r"(?i)^(true)$|^(false)$(?-i)", //BOOLEAN
- r"^-?(\d+)$", //INTEGER
- r"^-?((\d*\.\d+|\d+\.\d*)([eE][-+]?\d+)?|\d+([eE][-+]?\d+))$",
//DECIMAL
+ r"^[+-]?(\d+)$", //INTEGER
+ r"^[+-]?((\d*\.\d+|\d+\.\d*)([eE][-+]?\d+)?|\d+([eE][-+]?\d+))$",
//DECIMAL
r"^\d{4}-\d\d-\d\d$", //DATE32
r"^\d{4}-\d\d-\d\d[T ]\d\d:\d\d:\d\d(?:[^\d\.].*)?$",
//Timestamp(Second)
r"^\d{4}-\d\d-\d\d[T ]\d\d:\d\d:\d\d\.\d{1,3}(?:[^\d].*)?$",
//Timestamp(Millisecond)
@@ -1388,6 +1388,30 @@ mod tests {
use arrow_array::cast::AsArray;
use arrow_cast::display::array_value_to_string;
+ #[test]
+ fn test_infer_schema_leading_plus_numbers() {
+ for (csv, expected_type) in [
+ ("+1\n2\n-3\n", DataType::Int64),
+ ("+1.5\n2.5\n-3.5\n", DataType::Float64),
+ ("+1e3\n+2.5e-2\n-3E+2\n", DataType::Float64),
+ ("+9223372036854775807\n0\n", DataType::Int64),
+ ("+9223372036854775808\n0\n", DataType::Utf8),
+ ("+-1\n2\n", DataType::Utf8),
+ ("+\n2\n", DataType::Utf8),
+ ] {
+ let (schema, records_read) = Format::default()
+ .infer_schema(Cursor::new(csv), None)
+ .unwrap();
+ assert_eq!(schema.field(0).data_type(), &expected_type, "CSV:
{csv:?}");
+ // Inferred numeric types must also be accepted by the CSV decoder.
+ let reader = ReaderBuilder::new(Arc::new(schema))
+ .build(Cursor::new(csv))
+ .unwrap();
+ let rows: usize = reader.map(|batch|
batch.unwrap().num_rows()).sum();
+ assert_eq!(rows, records_read, "CSV: {csv:?}");
+ }
+ }
+
#[test]
fn test_csv() {
let schema = Arc::new(Schema::new(vec