manuzhang opened a new pull request, #7015:
URL: https://github.com/apache/datafusion-comet/pull/7015
## Which issue does this PR close?
No existing GitHub issue. This fixes unbounded recursion in native Parquet
Variant normalization.
## Rationale for this change
A deeply nested Parquet Variant value can drive recursive native validation
and reconstruction until the executor thread exhausts its stack. Native scans
now support at most 64 object/list containers along a value path and return
`MALFORMED_VARIANT` for deeper input. Typed containers and residual values
share the same budget.
## What changes are included in this PR?
- Check residual nesting iteratively with shallow Variant iteration before
recursive validation and unshredding.
- Bound storage-schema traversal before recursive casts and VariantArray
construction.
- Make object-key compatibility traversal iterative and validate the depth
before recursive re-encoding.
- Document the native nesting limit.
## How are these changes tested?
Focused native Variant tests on macOS ARM64 with Rust 1.95.0:
```sh
CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_BUILD_JOBS=4 \
cargo +1.95.0 test --locked --offline -p datafusion-comet --lib \
parquet::cast_column::variant::tests --no-default-features
```
Before rebasing onto current main: 21 passed; 0 failed; 1 existing benchmark
ignored. The rebase preserves the patch exactly; the three affected files and
dependency manifests were unchanged upstream. New regressions cover objects and
lists at the limit and one beyond, residual/typed/mixed paths, and 4,096-level
residuals rejected on a 2 MiB stack. Existing compatibility and buffer-reuse
tests also pass.
`cargo fmt --all -- --check` and `git diff --check` pass. Spark SQL coverage
is requested through the `run-spark-4.1-tests` CI label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
