kskalski opened a new issue, #4856:
URL: https://github.com/apache/arrow-rs/issues/4856
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
I'm processing jsons with some exceptionally large numbers provided as json
`number`. They exceed both `int64` and exactly represented `float64`.
Currently json inference fallback to `float64`, which causes reader to lose
exactness when parsing values.
**Describe the solution you'd like**
I think since Rust `arrow` now supports decimal data type, it could be used
also for json inference (parsing already works if I enforce a proper data type).
I'm not sure which precision/scale should be inferred in this case, possibly
inference should track necessary scale and assume `(max precision - necessary
scale, necessary scale)`.
**Describe alternatives you've considered**
One way to work around that would be to use strings, since
`json::ReaderBuilder` with `.with_coerce_primitive(true)` allows to parse
interchangebly strings and numbers.
This might actually still be a valid solution if we need to support numbers
that are beyond even `decimal256`.
**Additional context**
```rust
const DATA_BIG_NUM: &str = r#"{"x": 1, "y": "01"}
{"x": 2000000000, "y": "02"}
{"x": 102174585645880460651957762, "y": "03"}
{"x": 102174585645880460651957762102, "y": "04"}"#;
#[test]
fn test_json_infers_coerce() {
let mut cursor = std::io::Cursor::new(&DATA_BIG_NUM);
let schema = infer_json_schema_from_seekable(&mut cursor,
Some(3)).unwrap();
let mut json =
json::ReaderBuilder::new(schema.into()).build(cursor).unwrap();
let batch = json.next().unwrap().unwrap();
// Inference falls into float64 datatype, which is lossy as
demonstrated below
assert_eq!(DataType::Decimal128(30, 0),
*batch.schema().fields()[0].data_type());
assert_eq!(
102174585645880460651957762i128,
Decimal128Array::from(batch.column(0).to_data()).value(2)
);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
