matanper opened a new pull request, #70729:
URL: https://github.com/apache/doris/pull/70729
### What problem does this PR solve?
Issue Number: close #69855
Related PR: None
Problem Summary:
A Stream Load with `format=arrow` and no `columns` header could not load
into a
table that has a column the Arrow stream does not carry. The usual case is a
nullable column added with `ALTER TABLE ... ADD COLUMN` after the sender was
built: every Arrow load from that sender failed until the sender was changed.
JSON loads instead load such a column as its default.
Reproduce:
1. `CREATE TABLE t (k INT, v STRING) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k)
BUCKETS 1 PROPERTIES ("replication_num" = "1");`
2. Stream Load an Arrow IPC stream with columns `k` and `v`
(`format=arrow`). It succeeds.
3. `ALTER TABLE t ADD COLUMN extra STRING NULL;`
4. Stream Load the same stream again.
Before: the load fails with
`Arrow stream requires exactly one column named 'extra'`.
After: the rows load with `extra` = NULL.
Root cause: the FE plans a source slot for every table column. The format_v2
`ArrowStreamReader::_open_next_stream` looked each requested slot up by name
in
the IPC schema and failed the load when the name was absent.
Fix: a requested column that the current IPC stream does not carry is now
treated as a missing field, in the same way the format_v2 JSON reader treats
a
field absent from a row. `get_block()` fills the column with a NULL
placeholder
and marks every row of it in `FileReader::_missing_fields`. Then the existing
`LoadTableReader::_materialize_missing_values` swaps in the slot's default
expression. The result:
- a nullable column without a default loads NULL;
- a column with a default value loads that default;
- a NOT NULL column without a default stays NULL, and the row is rejected as
`Column <name> is NULL but is not nullable`. That row counts toward
`max_filter_ratio`, so with the default ratio the load fails and the error
log
names the column. This is the same result as a JSON row missing that field.
Presence is decided per IPC stream, because one load body can concatenate
several streams with different schemas. A column holding an explicit NULL in
a
stream that carries it stays NULL; it is not replaced by the default. A
schema
with two fields of the same name still fails, now with
`Arrow stream has N columns named '<name>'`. Extra Arrow columns the load
does
not request were already ignored by name matching on master.
The format_v2 reader matches columns by name, so it never reaches the
out-of-range block position that the issue reports for branch-4.1's
positional
reader.
### Release note
Stream Load with `format=arrow` now loads a table column that is missing from
the Arrow stream as its default value (NULL for a nullable column), as JSON
loads do, instead of failing the load.
### Check List (For Author)
- Test: Unit Test / Regression test
- BE UT (`be/test/format_v2/table/load_reader_test.cpp`): replaced
`RejectsMissingRequestedColumn` with tests for a missing column with a
default, with an explicit NULL default, a NOT NULL column without a
default (rows filtered), concatenated streams with different schemas,
and
duplicate field names.
- Regression test
`load_p0/stream_load/test_arrow_stream_load_missing_columns`:
defaults, concatenated streams, NOT NULL without default, and the
ALTER TABLE ADD COLUMN reproduction. Its .out file was generated by the
regression script on an ASAN FE+BE cluster built from this branch.
- Ran on ASAN: all 80 tests of LoadArrowPipeTest, LoadJsonPipeTest and
LoadReaderTest pass; test_arrow_stream_load_missing_columns,
test_arrow_stream_load_uppercase_column and
test_arrow_stream_load_type_validation pass.
- Behavior changed: Yes. An Arrow Stream Load that used to fail on a table
column missing from the stream now loads that column's default.
- Does this need documentation: No
Performance: not a performance change. The per-block hot path adds one map
clear and one branch per requested column, so no benchmark was run.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]