Doris-Breakwater commented on issue #70426: URL: https://github.com/apache/doris/issues/70426#issuecomment-6760943548
Breakwater-GitHub-Analysis-Slot: slot_50eb16cb9328 **Initial assessment:** The legacy reader source supports the reported row-accounting defects. This merits prioritization as a BE availability and query-correctness bug in Hive/OpenX malformed-record handling. The `ColumnString::filter` failure is a downstream detection of inconsistent column lengths; removing its check would leave the underlying correctness problem unresolved. The issue currently has no labels or assignee; suggested routing is to the Hive/external JSON scanner maintainers. **Verified from source** I inspected the supplied commit `7f5ba43de6c7f148e1dacdb7cb838394d2ec1537`. Its legacy `new_json_reader.cpp` is identical to the file at the current `branch-3.1` head checked during this investigation, `a3e9f66a1106a4208e6855ffef9e0fb8b3c981ff`. 1. **simdjson rollback can leave a counted row without file values.** `get_next_block()` initializes `is_empty_row=false` and increments `read_rows` whenever the callback leaves it false. Both simple-JSON exception handlers call `_handle_simdjson_error()`, which removes partial writes, but neither consistently marks the record empty nor appends its replacement NULL row when scanning continues. This supports the reported overcount mechanism. See [reader accounting](https://github.com/apache/doris/blob/7f5ba43de6c7f148e1dacdb7cb838394d2ec1537/be/src/vec/exec/format/json/new_json_reader.cpp#L217-L246) and [rollback and both exception handlers](https://github.com/apache/doris/blob/7f5ba43de6c7f148e1dacdb7cb838394d2ec1537/be/src/vec/exec/format/json/new_json_reader.cpp#L1285-L1416). 2. **The malformed-NULL helper also writes scanner-owned columns.** `ignore_malformed_json_append_null()` iterates every column in the Block. `VFileScanner` subsequently fills partition columns using `read_rows`; its Nullable SerDe appends values rather than replacing existing entries. Thus a partition NULL inserted by the reader receives additional partition values later. See [helper](https://github.com/apache/doris/blob/7f5ba43de6c7f148e1dacdb7cb838394d2ec1537/be/src/vec/exec/format/json/new_json_reader.cpp#L413-L422), [scanner handoff](https://github.com/apache/doris/blob/7f5ba43de6c7f148e1dacdb7cb838394d2ec1537/be/src/vec/exec/scan/vfile_scanner.cpp#L458-L489), and [append semantics](https://github.com/apache/doris/blob/7f5ba43de6c7f148e1dacdb7cb838394d2ec1537/be/src/vec/data_types/serde/data_type_nullable_serde.cpp#L131-L153). 3. **RapidJSON can emit multiple records per counted callback.** In `_vhandle_simple_json()`, a tolerated parse error appends NULLs and then executes `continue`. The same callback can consume the following valid record before returning, while its caller increments `read_rows` only once. See [RapidJSON malformed-record branch and loop](https://github.com/apache/doris/blob/7f5ba43de6c7f148e1dacdb7cb838394d2ec1537/be/src/vec/exec/format/json/new_json_reader.cpp#L504-L580). The normal FE OpenX setup selects line-oriented JSON with empty JSON paths and forwards `ignore.malformed.json`, making the simple-JSON path relevant to this table definition. See [HiveScanNode configuration](https://github.com/apache/doris/blob/7f5ba43de6c7f148e1dacdb7cb838394d2ec1537/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java#L549-L565). **Version boundary and validation limits** At `4.1.5-release` (commit `917b20be83878e98f94e3ebf7efccc80346e3e50`), the legacy simdjson path still has the all-column NULL helper and rollback/accounting pattern. The separate V2 reader [restores the pre-document row count before appending a tolerated NULL row](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/format_v2/json/json_reader.cpp#L1092-L1107), and [V2 is enabled by default for supported scans](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java#L1152-L1155). This is useful reference behavior, not evidence that every 4.1.5 query uses V2 or that the legacy path is fixed. I have not independently run the C++ harness, reproduced the SIGABRT, or validated the proposed patch. The exact column counts and eight passing cases remain reporter-provided results. No Doris code was changed. **Recommended next steps** - Prepare a focused legacy-reader fix: restrict malformed-row NULL insertion to the requested file slots, restore partial writes before inserting exactly one replacement row, and return immediately after RapidJSON emits that row. Preserve the invariant that the reported output-row count matches each populated file column before the scanner fills partitions/defaults. `block.rows()` alone is not a safe replacement for `read_rows`: the scanner explicitly notes that its first column may still be unfilled. - Attach the synthetic harness, build/run commands, and proposed patch or PR. Include how each parser was selected (`enable_simdjson_reader` and AVX2 build applicability), plus before/after output. These are the main missing artifacts for independent confirmation. - Add regression coverage for both 3.1 parser paths: valid records, missing closing brace, illegal token, consecutive malformed records, and malformed records at EOF/batch boundaries. Assert actual values, NULL rows, partition values, and equal column lengths; include reordered projections and file-column predicates as well as partition predicates. Retain coverage with malformed tolerance disabled and for existing load error handling. - For end-to-end confirmation, provide sanitized Hive DDL/SerDe properties, the exact SQL, and the selected scanner/parser path. Recheck the 4.1.5 legacy path separately from V2. The existing minimal input is sufficient to start; production data and a performance Profile are not needed for this correctness investigation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
