andygrove opened a new pull request, #2305: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2305
## Summary - Adds `SparkSqlDialect` (`src/dialect/spark.rs`) with support for the key Spark SQL syntax features needed to parse real-world Spark SQL workloads - Adds `tests/sqlparser_spark.rs` with 16 tests; the `test_comet_sql_files` test reads all 1,152 SQL statements from the [Apache DataFusion Comet](https://github.com/apache/datafusion-comet) test suite and verifies they parse cleanly ### New Dialect trait methods (all default `false`) | Method | Purpose | |---|---| | `supports_create_table_using` | `CREATE TABLE t (...) USING parquet` | | `supports_long_type_as_bigint` | `LONG` as alias for `BIGINT` | | `supports_map_literal_with_angle_brackets` | `MAP<K, V>` type syntax | ### Parser/AST changes - New `HiveIOFormat::Using { format: Ident }` variant — renders as `USING <format>`, distinct from Hive's `STORED AS <format>` - `parse_hive_formats` handles `USING <format>` when dialect opts in - `STRUCT` type parsing now uses `supports_struct_literal()` trait method instead of `dialect_is!(BigQueryDialect | DatabricksDialect | GenericDialect)` — cleaner and extensible - New `MAP<K, V>` angle-bracket parsing path in `parse_data_type_helper` ### SparkSqlDialect capabilities `USING <format>` in CREATE TABLE, lambda functions, `DIV` integer division, aggregate `FILTER`, `GROUP BY` expressions/modifiers, `SELECT * EXCEPT`, `STRUCT<>` and `MAP<>` types, `LONG` type alias, nested comments, `!` as NOT, CTE without AS, multi-column aliases, `IGNORE NULLS` in window functions (already supported by the generic parser). ## Test plan - [ ] `cargo test --all-features` passes (zero regressions across all existing dialect tests) - [ ] `cargo clippy --all-targets --all-features -- -D warnings` clean - [ ] `cargo fmt --all -- --check` clean - [ ] All 16 Spark-specific tests pass, including parsing all 1,152 statements from the Comet test suite 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
