sollhui opened a new pull request, #68539:
URL: https://github.com/apache/doris/pull/68539
### What problem does this PR solve?
Related PR: #53374
Parallel JSON readers move a non-first split back by one byte and discard
its first partial line. If the split falls inside a multi-byte line delimiter,
the remaining delimiter bytes are not recognized and the following complete
JSON record is discarded as well. The load can succeed with missing rows and no
filtered rows.
Apply the delimiter-length lookbehind used by the CSV fix to both the legacy
JSON reader and the format-v2 JSON reader. Clamp the lookbehind at the
beginning of the file and extend the line-reader length by the actual
lookbehind so the original split end is preserved.
Add parameterized unit tests covering both readers, every byte boundary in a
small input, newline/CRLF/ASCII/UTF-8 delimiters, lookbehind longer than the
start offset, files with and without a trailing delimiter, and a 403-record
input split into four ranges. The tests compare the full sequence of distinct
IDs across multiple batches to detect both missing and duplicated records.
### Release note
Fix silent row loss when parallel JSON file scans split inside a
multi-character line delimiter.
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test (added, not executed)
- [ ] Manual test
- Validation: clang-format 16 on changed lines, build-header hygiene,
and `git diff --check` passed. Compilation and test execution were skipped at
the requester's instruction. An initial unit-test runner invocation stopped
during environment initialization, before compiling or running tests.
- Behavior changed:
- [x] Yes. JSON records following a split inside a multi-character
delimiter are retained.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
