matanper opened a new pull request, #69767:
URL: https://github.com/apache/doris/pull/69767
### What problem does this PR solve?
Issue Number: close #68890
Related PR: #68883
Problem Summary:
Stream Load with `format=arrow` could not load the Arrow forms clients
send for a VARIANT column. `DataTypeVariantV2SerDe::read_column_from_arrow`
accepted only the exact shape Doris writes for Arrow Flight,
`struct<metadata: binary, value: binary>` in that field order. It rejected
the other shapes:
- JSON text, one document per row (`utf8`, `large_utf8`, `binary`,
`large_binary`), which is how most clients send semi-structured events;
- `large_binary` children, or the two fields in the other order (pyarrow
builds the struct by name);
- a dictionary-encoded `metadata` child. A producer that encodes a record
batch at a time has one metadata blob per batch. As a plain binary child,
that blob is repeated in every row: about 1 KB per row for a batch mixing
~10 event sources and about 45 KB per row for one mixing all of them,
against ~1.3 KB of value bytes. As `dictionary<int32, binary>` with one
entry, it costs a few bytes per row.
On the reader side, FileScannerV2's Arrow stream reader normalizes every
column before the SerDe sees it, which decodes a dictionary into plain
binary. For dictionary metadata that copies the shared blob into every row,
and the SerDe then compares it row by row to find that all rows share it.
This PR:
1. Extends the VARIANT V2 Arrow reader to accept, with or without the
`arrow.parquet.variant` extension:
- JSON text in `utf8`/`large_utf8`/`binary`/`large_binary`, parsed with
the same options as `parse_to_variant`. A NULL row is JSON null.
- `struct<metadata, value>` in either field order with `binary` or
`large_binary` children. Rows go through
`ColumnVariantV2::EncodedRowsAppender`, which validates every row before
appending and keeps a batch that shares one metadata blob in the
single-metadata form. Before, each row was copied into a
`VariantBatchBuilder` and then copied again into the column.
- the same struct with `dictionary<integer, binary | large_binary>`
metadata. It maps directly onto `ColumnVariantV2::insert_encoded_rows`:
dictionary entries are the metadata blobs and indices are the metadata
ids. When every row uses one entry, only that entry is passed, so no
row's metadata is copied, hashed or compared.
Every shape runs `ValidateFull()` on the slice and Variant
metadata/payload
validation before the destination changes, so a bad batch appends
nothing. NULL struct rows become Variant null.
2. Adds `DataTypeVariantV2SerDe::is_arrow_input_type`. The FileScannerV2
Arrow stream reader skips `normalize_arrow_array` for a VARIANT column
whose Arrow type the SerDe reads directly, so dictionary metadata reaches
it encoded. `is_native_arrow_type`, which the Arrow writer uses to choose
its output shape, is unchanged.
Benchmark (`be/benchmark/variant/benchmark_variant_arrow_input.hpp`): one
4064-row batch of 24-field events (~1.3 KB of values per row), read into a
VARIANT column, RELEASE build, before = master, after = this PR.
| Input | Shared metadata | master | this PR |
|---|---|---|---|
| JSON text (`utf8`) | 1.7 KB | rejected | 12.3 ms |
| JSON text (`utf8`) | 64 KB | rejected | 14.9 ms |
| `struct<binary, binary>` | 1.7 KB | 39.0 ms | 6.3 ms (6.2x) |
| `struct<binary, binary>` | 64 KB | 531 ms | 52.9 ms (10x) |
| dictionary metadata, as the stream reader reads it | 1.7 KB | 41.3 ms
(decoded first) | 6.0 ms (6.9x) |
| dictionary metadata, as the stream reader reads it | 64 KB | 620 ms
(decoded first) | 6.2 ms (100x) |
Medians of 10 runs on one core of a c7i.8xlarge. On master, the
dictionary row is normalize_arrow_array followed by the SerDe, which is
what the stream reader did. In this PR the reader passes it through. The
plain struct with 64 KB metadata is still the slowest binary form, because
every row's copy of the metadata has to be compared to find that it is
shared; dictionary metadata avoids that.
### Release note
Stream Load with `format=arrow` can load VARIANT columns from JSON text
(`utf8`, `large_utf8`, `binary`, `large_binary`) or from Parquet Variant
binary `struct<metadata, value>` (with or without the `arrow.parquet.variant`
extension, `binary` or `large_binary` children, either field order, and the
`metadata` child optionally dictionary encoded).
### Check List (For Author)
- Test:
- Unit Test: `DataTypeVariantV2SerdeArrowInputTest` (12 new cases), with
the existing `DataTypeVariantV2Serde*` and `ArrowFlightVariantTest`
suites (59 tests, ASAN)
- Regression test: `load_p0/stream_load/test_arrow_stream_load_variant`,
run
against a RELEASE FE/BE built from this branch (`.out` generated by
the run)
- Benchmark: before/after numbers above
- Behavior changed: Yes. A VARIANT column in an Arrow Stream Load now
accepts the Arrow types above instead of failing. Text that is not valid
JSON fails the load, as other Arrow conversion errors do.
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
