peterolkhov opened a new pull request, #12144: URL: https://github.com/apache/arrow-rs/pull/12144
# Which issue does this PR close? No existing issue; found while checking CSV inference/read round trips. # Rationale for this change Values such as `١`, `１`, localized timestamps, and `falſe` match the Unicode type patterns, but the inferred decoders reject them. A schema inferred from a CSV can therefore fail to read that same CSV. # What changes are included in this PR? Use Utf8 for sampled non-ASCII values when inferring decoder types. Keep the existing lexical classifier for header detection so localized data does not change which row is treated as a header. Null filtering and sampling stay unchanged. # Are these changes tested? Added public inference-to-reader regressions for the affected literals, both header decisions, mixed typed columns, nulls, and sampling. The CSV unit tests and doctests pass locally; full workspace validation is pending fork CI. Local strict dependency lint hits existing diagnostics in `arrow-data`; CSV-only lint passes. # Are there any significant user-facing changes? No API changes. Assisted-by: OpenAI Codex (diagnosis, implementation, and tests). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
