osipovartem opened a new issue, #11169:
URL: https://github.com/apache/arrow-rs/issues/11169
### Describe the problem
`parquet-variant-json` currently converts every non-integer JSON number to
`Variant::Double`. As a result, fixed-point values lose both decimal type
semantics and potentially precision:
```text
1.23 -> Double (expected Decimal4(123, 2))
9999999999999999.99 -> Double (expected Decimal8(..., 2))
9999999999999999999 -> Double (expected Decimal16(..., 0))
```
The crate already contains ignored tests for Decimal4/8/16 covering these
cases. The original implementation discussion in #7783 also identified this
as follow-up work.
This matters for Parquet Variant interoperability. Engines such as Snowflake
distinguish fixed-point Variant numbers (`DECIMAL`) from scientific or
floating-point numbers (`DOUBLE`).
### Proposed semantics
Use the existing mapping described in #7783:
* integers fitting in `i64`: preserve the current smallest integer encoding;
* fixed-point values: use the smallest fitting `VariantDecimal4/8/16`;
* integers wider than `i64` and at most 38 digits: use `VariantDecimal16`;
* exponent notation or values outside Variant decimal range: use `Double`;
* apply the same rules recursively inside arrays and objects.
### Implementation constraint
Enabling serde_json's `arbitrary_precision` feature is not a suitable fix. It
is unified across the Arrow workspace and changes unrelated serde_json users;
this was already observed during #7783. It also retains the intermediate
`serde_json::Value` tree.
A performance-first implementation should consume exact numeric lexemes and
append directly to `VariantBuilderExt`, avoiding an intermediate JSON value
tree and avoiding a second per-value conversion pass. Arrow JSON's tape
parser
already retains number text in `TapeElement::Number`, but `TapeDecoder` is
intentionally private. Two possible directions are:
1. expose a narrow, non-public-implementation-specific JSON token callback
API
from `arrow-json`; or
2. add a focused streaming parser in `parquet-variant-json` that writes
directly to the Variant builder.
I can implement either direction. Maintainer guidance on the preferred API
boundary would avoid introducing a second JSON tokenizer or prematurely
exposing `TapeDecoder`.
### Validation
The existing ignored decimal tests can be enabled, with added nested
object/list cases. A benchmark should compare integer-only, mixed-object, and
decimal-heavy JSON against the current serde path to ensure ordinary inputs
do
not regress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
