Baymine opened a new pull request, #66122:
URL: https://github.com/apache/doris/pull/66122
### What problem does this PR solve?
Issue Number: close #66121
Problem Summary: A column-pruned Hive TEXTFILE query reads only the
columns listed in _col_idxs, but the field splitter still scanned every
separator of each line, including trailing fields that are never used.
For wide tables with many unused trailing columns this is wasted work
proportional to the unused tail length of every row.
This adds a split limit on the query path: after the reader resolves
_col_idxs it computes max_col_idx+1 and passes it to the splitter via
set_split_limit. The single-char, non-escape path of
HiveTextFieldSplitter stops as soon as it has emitted that many fields
(each field is still bounded by its own separator, so the prefix is
identical to a full split). The escape-configured path, the multi-char
(KMP) path, and EncloseCsvTextFieldSplitter ignore the limit and split
fully, so escaped separators and enclose quoting are unaffected.
### Release note
None
### Check List (For Author)
- Test: BE unit test (run-be-ut --filter="*HiveTextFieldSplitter*")
- Added 4 TEST_F cases covering early stop, limit at/above field
count, edge cases (empty input, consecutive separators, trailing
separator), and limit-ignored-when-escape-configured.
- All 9 tests in HiveTextFieldSplitterTest PASSED.
- Behavior changed: No (query results unchanged; only the number of
separator scans for the non-escape single-char path on
column-pruned queries).
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
