andygrove commented on issue #95:
URL:
https://github.com/apache/datafusion-spark/issues/95#issuecomment-6855812328
### What can be made Spark-compatible
I probed PySpark 4.2.0 on Java 17.0.17, once with the JVM default time zone
set to UTC and once to `America/Los_Angeles`. The Spark session time zone was
UTC both times.
| Argument and conversion | Spark 4.2.0 result | Can datafusion-spark match
it? |
|---|---|---|
| Strings, integers, doubles, booleans with `%s`, `%d`, `%x`, `%o`, `%e`,
`%f`, `%g`, `%a`, `%b`, `%c` | `java.util.Formatter` rules with `Locale.US` |
Yes. `format_string.rs` already implements most of them. |
| `%h` of a string or number | `format_string('%h', 'test')` is `ec06e15a`,
and `%h` of `12345678901` is `dfdc1c37` | Yes. It's the Java `hashCode`:
Murmur3 for `UTF8String`, and `Long.hashCode` for a BIGINT. |
| DECIMAL with `%f`, `%e`, `%g` or `%a`, and DATE with `%t` | An error (`f
!= org.apache.spark.sql.types.Decimal`, `Y != java.lang.Integer`) | Yes,
easily. |
| `%s` of an array, map, struct or DATE | `[1,2]`, `keys: [a], values: [1]`,
`[1]`, `19716` | Yes. These are deterministic, though odd. |
| `%t` of a TIMESTAMP | `format_string('%tH:%tM', TIMESTAMP '2023-12-25
14:30:45', ...)` is `16:30` with the JVM in UTC and `09:30` in
`America/Los_Angeles` | **No.** The result depends on the executor JVM's
default time zone, not `spark.sql.session.timeZone`. |
| `%s` of BINARY | `[B@329658f8`, then `[B@7aef2416` on the next run |
**No.** It's a Java object identity hash. |
| `%h` of an array or struct | `c604` for `array(1, 2)` | Not sensibly. It
hashes Spark's internal `UnsafeArrayData` layout. |
| `%s` of some DOUBLEs | `9.999999999999999E22` for `1.0E23D` on Java 17 |
It depends on the JDK version. `Double.toString` changed in JDK 19
(JDK-4511638), which I believe prints `1.0E23`, but I only tested Java 17. |
### Suggestion
- Match Spark exactly for the deterministic cases, including the errors for
DECIMAL with `%f`/`%e`/`%g`/`%a` and DATE with `%t`.
- Raise an error for the cases that can't be reproduced (`%t` of a
TIMESTAMP, `%s` of BINARY, `%h` of nested types) rather than return a plausible
value. Document them under #55. Comet can then fall back to Spark for them.
- The 35 `format_string.slt` records that apply `%t` to a TIMESTAMP assert
behavior Spark does not have and cannot match reliably, so they should change
along with whichever behavior is chosen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
