kskalski commented on issue #4856: URL: https://github.com/apache/arrow-rs/issues/4856#issuecomment-1735438925
The [JSON spec](https://www.rfc-editor.org/rfc/rfc8259.html#section-6)(JSON is pretty vague on the semantics of "numbers", focusing on syntax and only suggesting an "interoperable" range of numbers, which follows `float64` implementation (`[-(2^53)+1, (2^53)-1]`). So technically "number" syntax is valid also for numbers without exact `f64` representation, it's just implied that many systems will fail to read them precisely. In the wild I think the common approach is: - for parsers to accept any numbers with conforming syntax, interpreting them with maximum available/configured/implemented precision, e.g. `serde_json` will automatically move from using `u64` to `f64`, but also allows compiling with `arbitrary_precision` feature, in which case values are stored in memory as strings and conversion to numeric types are done on the fly - for serializers to fallback into json strings when number to be serialized falls outside of "interoperable" range - previous point imposes additional nice-to-have feature on parsers to permit "numeric strings" when they expect numbers Clearly picking wider storage type than required is a performance penalty and possibly there should be a choice whether user wants to pay it. I would argue that whenever we detect data point that *requires* wider type for exact representation, the default behavior should be to preserve it or signal error, so for inference it means switching to wider type, e.g. `int32` -> `int64` -> `decimal128` -> `decimal256` -> possibly down to `string`. The latter is a proper choice when user doesn't want to lose precision + enables `coerce_primitive` flag in the reader. Additional questions arise here: * whether inference should perform a 100% precise detection of representable type boundaries, since that really requires more complex logic than parsing numbers to a known target representation. * if the inference logic should differ when max number of rows is specified or not, there is temptation that limited inference should be more lenient on picking a type, while unlimited inference could assume that it has seen all the data (so e.g. it could use `int32` instead of `int64` when possible) The current behaviour of lossy parsing to `f64`, which to a large degree is possible for almost any number, could be kept under a flag - essentially it implements a specific use-case where user prefers performance more than correctness. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
