ArnabKarmakar1108 opened a new issue, #17510:
URL: https://github.com/apache/iceberg/issues/17510
### Feature Request / Improvement
## Problem
`VariantTestUtil.createArray` pre-computes the data section size by summing
`sizeInBytes()` for each element before allocating the output buffer:
```java
// api/src/test/java/org/apache/iceberg/variants/VariantTestUtil.java:273
for (VariantValue value : values) {
// TODO: produce size for every variant without serializing
dataSize += value.sizeInBytes();
}
```
This works for pre-serialized values (`SerializedPrimitive`,
`SerializedShortString`, etc.) where `sizeInBytes()` is just
`buffer().remaining()`. It breaks down for nested or non-buffer-backed values —
e.g. a `SerializedArray` passed as an element may not reflect the size needed
when embedded in a parent array without already being fully serialized.
The same pattern exists in `createObject` (line 220) without a TODO, but
would benefit from the same fix.
## Proposed behavior
Provide a way to compute the serialized byte size of any `VariantValue`
without building the full parent buffer first. Options to consider:
1. Add a `sizeInBytes()` implementation (or helper) that recursively
computes size from type/metadata for all variant physical types
2. Add a `VariantTestUtil.sizeInBytes(VariantValue)` helper that handles
nested arrays/objects correctly
3. Refactor `createArray` / `createObject` to use it and remove the TODO
`createArray` should then allocate the correct buffer size for mixed and
nested element types without redundant serialization work.
## Scope
- **Module:** `api` (test utility — `VariantTestUtil.java`)
- **May touch:** `VariantValue` / serialized variant classes if a shared
size API is added
- **Tests:** `TestSerializedArray`, `TestSerializedObject`, and any tests
using `VariantTestUtil.createArray` with nested values
- **No production API changes** unless maintainers prefer a public
`sizeInBytes` improvement in core variant types
## Notes
- Test-only improvement unless core variant sizing is generalized.
- `createObject` uses the same `sizeInBytes()` loop — consider fixing both
together.
- Related tests already cover nested arrays
(`TestSerializedArray.testArrayOfMixedTypes`).
### Query engine
None
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with
guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
