buraksenn opened a new pull request, #24233:
URL: https://github.com/apache/datafusion/pull/24233
## Which issue does this PR close?
- Closes #24180.
## Rationale for this change
Physical-plan protobuf serialization does not preserve several CSV and JSON
scan options. Custom CSV terminators, JSON newline-delimited mode, and file
compression therefore revert to their defaults after a roundtrip, which can
cause the
decoded plan to read the file incorrectly.
## What changes are included in this PR?
- Add `terminator` to `CsvScanExecNode`.
- Add `newline_delimited` to `JsonScanExecNode`.
- Add `file_compression_type` to the shared `FileScanExecConf`.
- Serialize and restore these options in CSV and JSON scans.
## Are these changes tested?
Yes. Extended the CSV and JSON physical-plan roundtrip tests to cover
custom terminators, non-newline-delimited JSON, compression, and
backward-compatible defaults.
## Are there any user-facing changes?
CSV and JSON scans now preserve their format and compression options across
protobuf roundtrips. The protobuf changes are additive and backward compatible.
`JsonSource` also gains an `is_newline_delimited` getter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
