Gabriel39 opened a new pull request, #65501:
URL: https://github.com/apache/doris/pull/65501
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: FileScanner V2 used an independent CSV field-splitting
state machine that diverged from the line reader for bare quotes, escapes
outside enclosed fields, and configurations where escape equals enclose. It
also removed double quotes before nullable string conversion, causing quoted
null markers to be treated as NULL. This change reuses the line reader's
separator positions, preserves quoted-string provenance through null matching,
and adjusts separator positions when a UTF-8 BOM is removed.
### Release note
Fix CSV FileScanner V2 parsing for enclosed fields and quoted null literals.
### Check List (For Author)
- Test: Unit Test and Regression test
- `CsvV2ReaderTest`: 30 tests passed
- `test_local_tvf_csv_enclose_consistency`: passed
- Behavior changed: Yes. CSV V2 now matches the established enclosed-field
parser and preserves quoted null literals as strings.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
