raghav-reglobe commented on PR #63192:
URL: https://github.com/apache/doris/pull/63192#issuecomment-4968589256
Hi @eldenmoon — we've been running this PR's Iceberg VARIANT read path
against real workloads (Iceberg V3 tables whose VARIANT columns are written by
another engine, both shredded and unshredded parquet), and found one seam issue
you'd probably want to fix in the PR itself. Sharing the root cause since we've
already traced it through.
**Symptom**: element access on an Iceberg-read VARIANT silently returns NULL
for every key, while reading the whole value works:
```sql
SELECT CAST(v AS STRING) FROM iceberg_cat.db.t; -- {"imei":{"d":"865..."}}
✓
SELECT v['imei'] FROM iceberg_cat.db.t; -- NULL for every row ✗
```
No error anywhere — just NULLs, so it reads as missing data.
**Root cause**: a type seam between the reader and the element function.
- `be/src/format/parquet/parquet_variant_reader.cpp` decodes the variant
into a **scalar JSONB root** (`make_jsonb_field`, `TYPE_JSONB`) — no
subcolumns, no sparse/doc maps populated. (This is also why `CAST AS STRING`
works: it's plain root serialization.)
- `FunctionVariantElement::get_element_column`
(`be/src/exprs/function/function_variant_element.cpp`) has a scalar-root
fallback, but it's gated on `is_scalar_variant() && is_string_type(root)` —
**STRING roots only**. A scalar **JSONB** root skips it and falls through to
the subcolumn/sparse extraction machinery, which is empty for externally-read
variants, so every lookup returns default/NULL.
**Fix we've verified locally**: a JSONB-root branch mirroring the existing
STRING-root branch — serialize each row via `JsonbToJson::to_json_string` and
reuse the same simdjson `extract_from_document` path, producing the same
STRING-rooted result shape (chained access like `v['a']['b']` then works
through the existing branch on the second hop). Happy to send it as a PR
against your branch if useful — it's ~40 lines.
One forward-looking note: since the reader already handles the shredded
layout (`typed_value` groups in `schema_desc.cpp` /
`vparquet_column_reader.cpp`) but reassembles the full document per row, have
you considered path-level projection pushdown (materialize only the accessed
`typed_value.<path>` leaf + binary-residual merge for partially-shredded rows)?
That's where the big win is for wide shredded variants — we'd be glad to help
build that on top of this PR if you have a direction in mind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
