andygrove opened a new pull request, #76: URL: https://github.com/apache/datafusion-spark/pull/76
## Which issue does this PR close? N/A. Follow-up to #54 (the PySpark runner skips any query that calls `arrow_cast`, because Spark has no such function). ## Rationale for this change Many `.slt` queries use `arrow_cast` only to pick a type that has a plain SQL name. Writing those as `::` casts makes the tests easier to read, and Spark can run them, so the PySpark runner checks these queries instead of skipping them. ## What changes are included in this PR? `arrow_cast(x, 'T')` becomes `x::SQLTYPE` in 49 files (411 lines) when DataFusion maps the SQL type to exactly the same Arrow type and Spark accepts the type name: | Arrow type | SQL type | |---|---| | `Int8` / `Int16` / `Int32` / `Int64` | `TINYINT` / `SMALLINT` / `INT` / `BIGINT` | | `Float32` / `Float64` | `FLOAT` / `DOUBLE` | | `Boolean`, `Binary`, `Date32` | `BOOLEAN`, `BINARY`, `DATE` | | `Utf8View` | `STRING` | | `Decimal128(p, s)`, p ≤ 38 | `DECIMAL(p, s)` | | `List(Int32)` | `ARRAY<INT>` | `::` binds tighter than unary minus, so negative literals and expressions are wrapped in parentheses, for example `(-128)::TINYINT`. Without them, `-128::TINYINT` casts `128` first and overflows. `arrow_cast` is kept where no SQL type gives the same Arrow type: - `Utf8`, because `STRING` and `VARCHAR` map to `Utf8View` by default - unsigned integers, `Float16`, `Large*`, `*View`, `Date64`, `Time32`, `Decimal32`/`Decimal64`/`Decimal256`, `FixedSize*`, `Dictionary`, `Struct` and `Map` - timestamps with a precision or a time zone, and `Time64` ## Are these changes tested? Only test files changed. All 245 sqllogictest files pass (`cargo test -p datafusion-spark-sqllogictest`). ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
