cj-zhukov commented on PR #86:
URL: https://github.com/apache/datafusion-spark/pull/86#issuecomment-6857605957
> Thanks @cj-zhukov, the overall structure looks reasonable and null
handling (null struct → NULL, null fields omitted, null map values / array
elements kept), binary, NaN/Infinity and dates all match Spark. However,
several common scalar types fall through to the default arrow-json encoders and
produce output that differs from Spark's `JacksonGenerator`:
>
> 1. **Timestamps**: Spark uses `timestampFormat`
`yyyy-MM-dd'T'HH:mm:ss.SSSXXX` in the session time zone (and
`yyyy-MM-dd'T'HH:mm:ss[.SSS]` for TIMESTAMP_NTZ). E.g. `2024-01-01
01:02:03.123456` UTC → Spark `"2024-01-01T01:02:03.123Z"`, PR
`"2024-01-01T01:02:03.123456Z"`. The session time zone is not read from the
config.
>
> 2. **Float/double**: Spark uses Java
`Double.toString`/`Float.toString`: `1e-5` → `1.0E-5` (PR `0.00001`),
`123456789.0` → `1.23456789E8`, `1e10` → `1.0E10` (PR `1.0e10`).
>
> 3. **Decimal**: Spark uses `BigDecimal.toString`: `0` as
`decimal(38,10)` → `0E-10` (PR `0.0000000000`), `0.0000001` as `decimal(10,8)`
→ `1.0E-7`.
>
> 4. **Map keys**: Spark uses the internal value's `toString`, e.g. a
date key → `{"19723":1}` (PR `{"2024-01-01":1}`), double key `1e10` →
`"1.0E10"`.
>
> 5. **Intervals and other unmapped types** pass through silently
(`interval '1' day` → `"1 days"`, Spark `"INTERVAL '1' DAY"`). Types without a
faithful mapping should be rejected at planning time.
>
> 6. Minor: control-char escapes are lowercase (`\u001f`); Jackson emits
`\u001F`.
>
>
> Could you also:
>
> * add `crates/sqllogictest/test_files/json/to_json.slt` covering the
cases above, with expected values verified against Spark,
>
> * drop the public `IntoJsonStruct` trait / custom `expr_fn::to_json`
in favour of the crate's usual `export_functions!` pattern (`named_struct`
already covers building the struct), and the speculative flavor abstraction, to
keep the surface area down?
@andygrove thanks for the detailed review - much appreciated, especially the
concrete Spark expectations for each type.
I'll address all of it: the timestamp, float, decimal and map-key
formatting, planning-time rejection of unmapped types, the escape case, the
`to_json.slt` verified against Spark, and the simplification to the
`export_functions!` pattern without the extra trait and flavor abstraction.
I'll ping you once it's ready for another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
