kosiew opened a new pull request, #26179: URL: https://github.com/apache/datafusion/pull/26179
## Which issue does this PR close? - Part of #24612 ## Rationale for this change Deprecate `CsvSource::newlines_in_values()` as part of the CSV API cleanup while preserving compatibility and existing CSV serialization behavior. The getter simply returns `CsvOptions.newlines_in_values.unwrap_or(false)`. There is no replacement getter, so callers should retain their original `CsvOptions` when they need to inspect this setting. The public getter remains callable during the deprecation window, and CSV reading and repartitioning behavior are unchanged. ## What changes are included in this PR? - Mark `CsvSource::newlines_in_values()` as deprecated starting in 56.0.0, with an accurate migration note. - Replace the internal encoder's getter call with `self.options.newlines_in_values.unwrap_or(false)`, preserving the existing boolean mapping. - Extend `roundtrip_csv_scan_preserves_format_options` to cover `Some(true)`, `Some(false)`, and `None`. - Assert the protobuf `CsvScan.newlines_in_values` field after round-tripping the physical plan, while retaining all existing CSV format-option assertions. - Preserve malformed terminator validation for empty and two-byte terminators. - Document the deprecation and migration guidance in the 56.0.0 upgrade guide. No protobuf wire-format changes or new public APIs are introduced. ## Are these changes tested? Extended `roundtrip_csv_scan_preserves_format_options` to verify: - `Some(true)` serializes as `true`. - `Some(false)` serializes as `false`. - `None` serializes as `false`. - Existing CSV format options, including delimiter, quote, escape, comment, terminator, truncated rows, header, and GZIP compression, survive the round-trip. - Invalid zero-byte and two-byte terminators continue to produce decoding errors. The patch includes these regression tests, but does not establish whether they have been executed successfully. Recommended verification commands: ```bash cargo test -p datafusion-proto --test proto_integration roundtrip_csv_scan_preserves_format_options cargo test -p datafusion-datasource-csv cargo fmt --all cargo clippy --all-targets --all-features -- -D warnings ./ci/scripts/doc_prettier_check.sh --write --allow-dirty ``` ## Are there any user-facing changes? `CsvSource::newlines_in_values()` is deprecated starting in 56.0.0 but remains callable with unchanged behavior. Existing callers may receive deprecation warnings, potentially affecting downstream builds that treat warnings as errors. There is no replacement getter. Callers should retain the `CsvOptions` used to configure the source and read `options.newlines_in_values.unwrap_or(false)`. The 56.0.0 upgrade guide documents this migration. CSV parsing, repartitioning, and protobuf serialization behavior are unchanged. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
