Jo2234 opened a new pull request, #10361: URL: https://github.com/apache/arrow-rs/pull/10361
# Which issue does this PR close? - Closes #6882. # Rationale for this change `Format` currently requires callers to set `with_header` explicitly, even though schema inference already samples CSV records and applies Arrow's type inference rules. This makes the default easy to get wrong for the common case where a CSV begins with textual column names. # What changes are included in this PR? - Add an opt-in `Format::infer_format` API that samples records and updates the format's header setting. - Classify the first record as a header only when at least one first-row value is text while the sampled values in that column resolve to a non-text, non-null Arrow type. This follows the type-consistency signal used by DuckDB's CSV sniffer while deliberately leaving all-string inputs ambiguous. - Reuse the existing CSV parser configuration and type inference, including delimiter, quote, escape, terminator, comment, null, and truncated-row settings. - Keep ambiguous inputs conservative: empty, single-record, and all-string samples remain headerless. - Document the API with a runnable example. The supplied reader is consumed for inference, like `infer_schema`; callers can rewind or create a new reader before parsing. # Are these changes tested? Yes. Added behavioral coverage for: - headers with typed data columns; - data without a header; - empty, single-record, and all-string ambiguity; - configured CSV delimiters; and - bounded sampling via `max_records`. Validation run locally: - `CARGO_BUILD_JOBS=2 cargo test -p arrow-csv --all-features` — 77 unit tests and 13 doctests passed. - `CARGO_BUILD_JOBS=2 cargo test -p arrow --features csv` — 157 unit/integration tests and 11 doctests passed. - `CARGO_BUILD_JOBS=2 cargo clippy -p arrow-csv --all-targets --all-features -- -D warnings` — passed. - `CARGO_BUILD_JOBS=2 RUSTDOCFLAGS='-D warnings' cargo doc -p arrow-csv --no-deps` — passed. - `cargo fmt --all -- --check` — passed. - `git diff --check` — passed. # Are there any user-facing changes? Yes. This adds a non-breaking public API. Existing defaults and parsing behavior are unchanged unless a caller explicitly invokes `infer_format`. # AI assistance disclosure AI assistance was used for implementation and test drafting. The resulting change was reviewed against the existing CSV parser and type-inference behavior, and the validation commands and results above were run on this branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
