u70b3 opened a new pull request, #66514:
URL: https://github.com/apache/doris/pull/66514
### What problem does this PR solve?
Issue Number: close #61225
Related PR: cbfe3096dff (precedent: `has_children_column` guard for the
top-level parquet path), #65784
Problem Summary:
BE crashes while scanning Iceberg tables in the two modes reported by #61225:
1. **Pattern B – std::out_of_range / DCHECK abort.**
`StructNode::children_column_exists` does a bare `children.at()`. If the schema
info from FE is inconsistent with the scan projection, the whole BE process
dies (release: uncaught exception; debug: DCHECK abort). The top-level parquet
entry was fixed by cbfe3096dff, but sibling call sites were not:
- `OrcReader::_do_init_reader` top-level missing-column loop (the ORC
twin of cbfe3096dff)
- `IcebergParquetReader`/`IcebergOrcReader::on_before_init_reader`
classification loops (PARTITION_KEY + REGULAR branches)
- Nested struct field resolution: `StructColumnReader::read_column_data`
(parquet) and `OrcReader::_fill_doris_data_column` (orc)
- `IcebergPositionDeleteSysTableReader` (`$position_deletes` system table)
- Filter/optimization paths that iterate **all** tuple slots: page-index
stat func, min-max/bloom lambdas, bloom cache backfill, dict-filter fallback in
group reader, ORC pushdown type checks. These touch every slot in the tuple —
including synthetic slots such as TopN's GLOBAL_ROWID_COL that are never
registered in the schema tree — so they can crash even without any FE
misbehavior.
2. **Pattern A – SIGSEGV.** `ByteArrayDictDecoder::_decode_values` indexes
an empty `_dict_items` when data pages reference a dictionary that was never
decoded, dereferencing a null StringRef.
This PR applies defense-in-depth using the `has_children_column` API from
cbfe3096dff:
- **Query contract paths fail loudly** with `Status::InternalError` — a
contract violation is a bug and must not produce silent wrong results.
- **Optimization/filter paths silently skip the optimization** (return false
/ fall back to plain conjunct filtering) — no result correctness impact.
- **Pattern A is hardened, not root-caused**: empty dictionary + non-null
data now returns `Status::Corruption`. The issue provides no reproducer, so the
exact reason a dictionary page can go missing remains unproven; this converts
the process crash into a query error.
The schema mapping built by FE is designed as a superset of the BE scan
slots (top-level names = requested columns, row-lineage fields appended
explicitly, time-travel/TopN use the full schema, partition-evolution
PARTITION_KEY columns are present), so the new guards only fire on a genuine
contract violation and never change legitimate query behavior.
### Release note
Fix BE process crash (std::out_of_range / SIGSEGV) when scanning Iceberg
tables with inconsistent schema mapping - the query now fails with an error
instead.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Unit Test — new `HasChildrenColumnGuardsNestedStructField` (guard
fires for an unknown nested field) and `NestedStructFieldMissingInFileKeepsKey`
(legitimate schema evolution still fills defaults, so the guard never misfires)
in `table_schema_change_helper_test.cpp`; all 28 MockTableSchemaChangeHelper
tests pass locally. The existing external_table_p0/p2 Iceberg regression suites
cover behavior invariance; an end-to-end crash repro cannot be constructed in
SQL because it requires an FE/BE contract violation that a correct FE never
produces.
- Behavior changed:
- [x] No. (Legitimate query behavior is unchanged. On a schema-mapping
contract violation, the query now fails with InternalError instead of crashing
the BE process.)
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
