Gabriel39 opened a new pull request, #64871:
URL: https://github.com/apache/doris/pull/64871
### What problem does this PR solve?
Issue Number: None
Related PR: #63781, #64671
Problem Summary: File scanner v2 did not carry the same fixes as the
existing file scanner path. Predicate rows filtered inside v2 file readers were
still reported through scanner load counters unless the scanner was a real load
source, and Hive TEXTFILE empty physical lines were still skipped unless
read_csv_empty_line_as_null was enabled. This change gates v2 load counter
reporting with the same FILE_STREAM exception used by FileScanner and adds a
delimited text hook so Hive Text v2 treats empty physical lines as records
while CSV keeps the old default behavior.
### Release note
Fix file scanner v2 load counter reporting and Hive TEXTFILE empty-line
handling.
### Check List (For Author)
- Test: Unit Test / Manual test
- Added TextV2ReaderTest coverage for Hive TEXTFILE empty line records,
single-column empty string fields, and COUNT pushdown.
- Ran git diff --check.
- Ran clang-format v16 through build-support/run_clang_format.py for
changed files.
- Attempted ./run-be-ut.sh --run
--filter='TextV2ReaderTest.*:FileScannerV2Test.*', but the local run was
blocked because the script needed to update/download datasketches-cpp and
network access was unavailable; no BE UT binary was already built.
- Attempted clang-tidy with the available compile_commands.json, but it
pointed at a stale /mnt/disk3/gabriel path; the project clang-tidy wrapper also
requires bash 4+ while only system bash is available.
- Behavior changed: Yes. File scanner v2 now matches v1 load counter gating
and Hive TEXTFILE empty-line semantics.
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
