ohye3166 opened a new issue, #70426:
URL: https://github.com/apache/doris/issues/70426
### Summary
On the legacy Hive JSON scan path, a malformed JSON line can make file
columns and partition columns have different row counts. Filtering then aborts
the BE in `ColumnString::filter`.
This is reproducible with synthetic data on the unmodified upstream commit
`7f5ba43de6c7f148e1dacdb7cb838394d2ec1537` (`doris-3.1.4-rc02`). The current
`branch-3.1` source still contains the same logic.
### Minimal input
Use an OpenX `JsonSerDe` table with nullable file columns `event string`,
`status int`, and a partition column `dt string`, with
`ignore.malformed.json=true`.
For example, a local file in partition `dt=2026-09-19` contains:
```json
{"event":"a","status":1}
{"event":"b","status":2
{"event":"c","status":3}
```
The original query selected `event` and `status`, grouped by them, and
filtered on the `dt` partition. The original data was LZO-compressed, but
compression is not required for this reproduction.
### Reproduction and stack
I used a local C++ harness that calls the production `NewJsonReader`, the
same nullable partition SerDe used by `VFileScanner`, and
`VExprContext::execute_conjuncts_and_filter_block`. No production data or
external Hive/HDFS service was involved.
For the missing-`}` input above, the unmodified reader produces:
```text
READ_ROWS=3
AFTER_READER event=2 status=2 dt=0
AFTER_PARTITION event=2 status=2 dt=3
column_string.cpp:316
Check failed: filter.size() == offsets.size() (2 vs. 3)
SIGABRT
```
The GDB stack includes:
```text
ColumnStr<unsigned int>::filter()
ColumnNullable::filter()
Block::filter_block_internal()
VExprContext::execute_conjuncts_and_filter_block()
```
### Root causes
There are three related row-accounting problems in the legacy reader:
1. For a missing closing brace, simdjson error handling rolls back the
partially written file columns, but does not mark the row empty.
`get_next_block()` still increments `read_rows`. The scanner subsequently
appends `read_rows` partition values, leaving the partition column one row
longer than the file columns.
2. For an invalid token that reaches `ignore_malformed_json_append_null()`,
that helper appends NULLs to the entire block, including the partition column.
Partition filling then appends the values again. A synthetic invalid-token
input similarly produces `filter.size()=3` versus `offsets.size()=4`.
3. The RapidJSON malformed-row branch appends a NULL row and then continues
to consume the next valid row in the same callback. Two output rows are counted
once in `read_rows`, which also breaks partition filling and result correctness.
### Suggested fix
- Restrict malformed-row NULL insertion to the file slot descriptors, by
column name; never append to partition or generated columns.
- After simdjson error cleanup, append one file-column NULL row when
`openx_json_ignore_malformed` is enabled, in both the outer simple-JSON catch
and the write-columns catch. This preserves one output row per malformed
document while keeping partial writes rolled back.
- Return immediately after a RapidJSON malformed row so each output row
increments `read_rows` exactly once.
- Add regression coverage for normal rows, a missing closing brace, an
invalid token, and multiple malformed rows. Validate exact values, NULLs,
partition values, and every column's row count, not only that the BE stays
alive.
A local patch using this approach passed eight cases across simdjson and
RapidJSON. After the fix, the missing-`}` example returns `(a,1)`,
`(NULL,NULL)`, `(c,3)`, with `dt=2026-09-19` on all three rows and no abort.
### Version notes
`4.1.5-release` defaults to `FileScannerV2`; its
`format_v2/json/json_reader.cpp` already restores the original row count before
appending a malformed-row NULL. However, the legacy
`format/json/new_json_reader.cpp` in 4.1.5 and the 3.1 legacy path still
contain the problematic logic. Backporting the entire ScannerV2 stack would be
very large, so a small targeted fix for the legacy path would be preferable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
