XiaoHongbo-Hope opened a new pull request, #10320:
URL: https://github.com/apache/paimon/pull/10320
## What changed
- Add `variant_select_fields(column, field_names)` for batch extraction of
literal top-level fields from VARIANT object columns. It returns Python
dictionaries without decoding unrelated fields, preserves each selected value's
encoded type (including mixed numeric types across rows), omits missing fields,
and distinguishes a VARIANT null from a SQL-null row.
- Share wide-object field lookup across paths in the existing typed
`variant_get(column, {path: arrow_type})` API. Keep its exact-type behavior and
fall back to the previous path for other shapes.
- Cover primitive and nested types, mixed metadata, null/missing values,
non-monotonic object offsets, malformed/truncated objects, and chunked arrays.
- Document the new API and its CPU-only scope in the PyPaimon data-types
guide.
Example:
```python
from pypaimon.data import variant_select_fields
rows = variant_select_fields(payload_column, ["state.x", "action.y"])
```
Field names are literal top-level names; dots in a name are not interpreted
as path separators. For typed nested paths, use the existing `variant_get` API.
## Measurements
Read-only VPC test of one fixed physical batch, with 2,128 VARIANT rows,
5,362 top-level fields per payload and 200 selected fields:
| Python decoding only | Wall time | CPU time |
| --- | ---: | ---: |
| Full `GenericVariant.to_python()` then select fields | 22.61 s | 22.61 s |
| `variant_select_fields` | 2.34 s | 2.34 s |
The selected output matched the full-decoding result row by row (9.66x for
this decoding stage). This is not an end-to-end sample-throughput claim; table
planning, Parquet reading, and downstream application work are outside these
timings. The benchmark used a candidate with the same extraction algorithm as
this PR; a final constant-time root-bounds check was added afterward.
For the existing typed API, 152 fields with a stable encoded numeric type
were selected from the same payload. On a bounded 100-row subset, `variant_get`
took 54.58 s before and 0.51 s after the shared lookup, with Arrow-equal
outputs. Fields with varying encoded types were excluded because the existing
typed API intentionally does not cast. A separate 100-row synthetic wide-object
A/B (5,362 fields, 200 selected) measured 44.58 s before and 0.60 s after, with
equal outputs.
## Validation
- `pytest -q` on the VARIANT path, set, core, and Arrow row-group test
modules: 230 passed, 48 subtests passed.
- `flake8` on changed Python modules and `git diff --check`: passed.
This PR adds a PyPaimon API. Applications must opt into
`variant_select_fields`; it does not automatically change downstream SDK
behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
