LucaCappelletti94 opened a new issue, #2698: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/2698
Over the past couple of weeks I have been working on [dejadoc](https://github.com/LucaCappelletti94/cargo-dejadoc), which is a tool to find code duplications. It started out as a tool to find duplicated doctests, which is where it guarantees 100% accuracy, but then I just continued to have a few ideas and now it is a general code duplication tool specialized for Rust. TL;DR: it does reduction to canonical form followed by execution graph isomorphisms for each context block larger than a given size. I hope it is interesting, and will likely lead to some cleaup. Ciao, Luca ### Parser, dialects, and AST - [ ] Operator formatting arms in `OperatorClassItem::fmt` and `OperatorFamilyItem::fmt` (54 tokens, `5900b4f1`). [`src/ast/ddl.rs:5061`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/ast/ddl.rs#L5061-L5076) · [`src/ast/ddl.rs:5297`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/ast/ddl.rs#L5297-L5312) - [ ] Repeated blocks in `Expr::fmt` (33 tokens, `ea06ea64`). [`src/ast/mod.rs:2151`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/ast/mod.rs#L2151-L2159) · [`src/ast/mod.rs:2165`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/ast/mod.rs#L2165-L2173) - [ ] `PrimaryKeyConstraint::span` and `UniqueConstraint::span` (95 tokens, `17ba4875`). [`src/ast/table_constraints.rs:498`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/ast/table_constraints.rs#L498-L512) · [`src/ast/table_constraints.rs:572`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/ast/table_constraints.rs#L572-L586) - [ ] Alias keyword handling in `MsSqlDialect::is_select_item_alias` and `is_table_factor_alias` (37 tokens, `786b90d1`). [`src/dialect/mssql.rs:148`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/dialect/mssql.rs#L148-L165) · [`src/dialect/mssql.rs:182`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/dialect/mssql.rs#L182-L199) - [ ] `parse_infix` in `MySqlDialect` and `SparkSqlDialect` (48 tokens, `1d1e6c9e`). [`src/dialect/mysql.rs:103`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/dialect/mysql.rs#L103-L119) · [`src/dialect/spark.rs:141`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/dialect/spark.rs#L141-L156) - [ ] Blocks in Snowflake `parse_create_table` and `parse_create_database` (31 tokens, `131fed19`). [`src/dialect/snowflake.rs:1021`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/dialect/snowflake.rs#L1021-L1026) · [`src/dialect/snowflake.rs:1218`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/dialect/snowflake.rs#L1218-L1223) - [ ] Arms in `parse_expr_prefix_by_unreserved_word` (32 tokens, `ad7f100d`). [`src/parser/mod.rs:1695`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L1695-L1704) · [`src/parser/mod.rs:1706`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L1706-L1715) - [ ] Blocks in `parse_cache_table` (103 tokens, `54bd5405`). [`src/parser/mod.rs:5658`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L5658-L5689) · [`src/parser/mod.rs:5691`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L5691-L5722) - [ ] Blocks in `parse_function_arg` and `parse_aggregate_function_arg` (38 tokens, `78e10075`). [`src/parser/mod.rs:6391`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L6391-L6401) · [`src/parser/mod.rs:6455`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L6455-L6463) - [ ] Blocks in `parse_create_operator_class` and `parse_operator_family_add_operator` (39 tokens, `427630ff`). [`src/parser/mod.rs:7622`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L7622-L7632) · [`src/parser/mod.rs:11896`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L11896-L11905) - [ ] Blocks in `parse_create_table` and `parse_optimize_table` (30 tokens, `644cb5fa`). [`src/parser/mod.rs:8978`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L8978-L8983) · [`src/parser/mod.rs:20621`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L20621-L20626) - [ ] Blocks within `parse_optional_column_option_inner` (37 tokens, `9fb20c90`). [`src/parser/mod.rs:9752`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L9752-L9766) · [`src/parser/mod.rs:9788`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L9788-L9804) - [ ] Blocks in `parse_optional_column_option_inner` and `parse_optional_table_constraint` (60 tokens, `498e55f2`). [`src/parser/mod.rs:9812`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L9812-L9826) · [`src/parser/mod.rs:10289`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/src/parser/mod.rs#L10289-L10303) ### Tests Any consolidation should retain the dialect-specific cases and assertions. A lexical match alone does not justify deleting coverage. - [ ] `parse_join_constraint_unnest_alias` in BigQuery and PostgreSQL tests (68 tokens, `bc48d2a7`). [`tests/sqlparser_bigquery.rs:1786`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_bigquery.rs#L1786-L1813) · [`tests/sqlparser_postgres.rs:6070`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_postgres.rs#L6070-L6096) - [ ] `test_bigquery_trim` and `test_duckdb_trim` (85 tokens, `b4f1ce3a`). [`tests/sqlparser_bigquery.rs:2456`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_bigquery.rs#L2456-L2482) · [`tests/sqlparser_duckdb.rs:860`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_duckdb.rs#L860-L886) - [ ] `parse_delimited_identifiers` in ClickHouse and SQL Server tests (205 tokens, `d533ff35`). [`tests/sqlparser_clickhouse.rs:158`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_clickhouse.rs#L158-L221) · [`tests/sqlparser_mssql.rs:860`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_mssql.rs#L860-L923) - [ ] Arms in `parse_delimited_identifiers` across five dialect test files (57 tokens, `dd0c118a`). [`tests/sqlparser_clickhouse.rs:165`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_clickhouse.rs#L165-L181) · [`tests/sqlparser_mssql.rs:867`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_mssql.rs#L867-L883) · [`tests/sqlparser_postgres.rs:4825`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_postgres.rs#L4825-L4841) · [`tests/sqlparser_redshift.rs:110`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_redshift.rs#L110-L126) · [`tests/sqlparser_snowflake.rs:1415`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L1415-L1431) - [ ] Blocks in `parse_use` across six dialect test files (45 tokens, `4a50dddb`). [`tests/sqlparser_clickhouse.rs:1485`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_clickhouse.rs#L1485-L1503) · [`tests/sqlparser_databricks.rs:229`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_databricks.rs#L229-L247) · [`tests/sqlparser_duckdb.rs:817`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_duckdb.rs#L817-L835) · [`tests/sqlparser_mssql.rs:1767`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_mssql.rs#L1767-L1785) · [`tests/sqlparser_mysql.rs:604`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_mysql.rs#L604-L622) · [`tests/sqlparser_snowflake.rs:3292`](https: //github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L3292-L3310) - [ ] First matching arms in `parse_json_object` and `parse_mssql_json_array` (30 tokens, `6628f171`). [`tests/sqlparser_common.rs:1841`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_common.rs#L1841-L1852) · [`tests/sqlparser_mssql.rs:1155`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_mssql.rs#L1155-L1166) - [ ] Second matching arms in `parse_json_object` and `parse_mssql_json_array` (30 tokens, `7f486c49`). [`tests/sqlparser_common.rs:1857`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_common.rs#L1857-L1868) · [`tests/sqlparser_mssql.rs:1171`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_mssql.rs#L1171-L1182) - [ ] `assert_mysql_query_value` arms in string-unescaping tests (37 tokens, `dd3b9b9e`). [`tests/sqlparser_common.rs:11801`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_common.rs#L11801-L11812) · [`tests/sqlparser_common.rs:11864`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_common.rs#L11864-L11875) - [ ] First matching arms in `parse_create_table_like` and `parse_create_table_like_with_defaults` (37 tokens, `9b33762e`). [`tests/sqlparser_common.rs:19050`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_common.rs#L19050-L19062) · [`tests/sqlparser_postgres.rs:685`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_postgres.rs#L685-L697) - [ ] Second matching arms in `parse_create_table_like` and `parse_create_table_like_with_defaults` (37 tokens, `7eb8d06f`). [`tests/sqlparser_common.rs:19067`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_common.rs#L19067-L19079) · [`tests/sqlparser_postgres.rs:703`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_postgres.rs#L703-L715) - [ ] `test_select_wildcard_with_exclude` in DuckDB and Snowflake tests (106 tokens, `bc66e513`). [`tests/sqlparser_duckdb.rs:156`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_duckdb.rs#L156-L189) · [`tests/sqlparser_snowflake.rs:1506`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L1506-L1539) - [ ] `parse_drop_trigger` in SQL Server and MySQL tests (36 tokens, `885b1dde`). [`tests/sqlparser_mssql.rs:2579`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_mssql.rs#L2579-L2591) · [`tests/sqlparser_mysql.rs:4362`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_mysql.rs#L4362-L4374) - [ ] `parse_create_table_gencol` in MySQL and SQLite tests (45 tokens, `2f1129d7`). [`tests/sqlparser_mysql.rs:1403`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_mysql.rs#L1403-L1416) · [`tests/sqlparser_sqlite.rs:318`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_sqlite.rs#L318-L331) - [ ] Blocks in `parse_grant` and `parse_revoke` (30 tokens, `0a0f9e04`). [`tests/sqlparser_mysql.rs:4106`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_mysql.rs#L4106-L4111) · [`tests/sqlparser_mysql.rs:4149`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_mysql.rs#L4149-L4154) - [ ] Blocks in `parse_create_index` and `parse_create_index_with_predicate` (67 tokens, `ab693f4d`). [`tests/sqlparser_postgres.rs:3008`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_postgres.rs#L3008-L3021) · [`tests/sqlparser_postgres.rs:3440`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_postgres.rs#L3440-L3453) - [ ] Arms in `test_snowflake_create_table_with_aggregation_policy` (33 tokens, `57e152aa`). [`tests/sqlparser_snowflake.rs:212`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L212-L222) · [`tests/sqlparser_snowflake.rs:232`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L232-L242) - [ ] Arms in `test_snowflake_create_table_with_row_access_policy` (33 tokens, `f3df65a9`). [`tests/sqlparser_snowflake.rs:252`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L252-L262) · [`tests/sqlparser_snowflake.rs:274`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L274-L284) - [ ] Arms in `test_snowflake_create_table_with_tag` (42 tokens, `af56ee7c`). [`tests/sqlparser_snowflake.rs:320`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L320-L331) · [`tests/sqlparser_snowflake.rs:341`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L341-L352) - [ ] Blocks in `test_create_stage_with_file_format` and `test_copy_into_file_format` (72 tokens, `0a513c29`). [`tests/sqlparser_snowflake.rs:2123`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L2123-L2142) · [`tests/sqlparser_snowflake.rs:2636`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L2636-L2655) · [`tests/sqlparser_snowflake.rs:2678`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L2678-L2697) - [ ] Blocks in `test_create_stage_with_copy_options` and `test_copy_into_copy_options` (46 tokens, `37e25c15`). [`tests/sqlparser_snowflake.rs:2159`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L2159-L2170) · [`tests/sqlparser_snowflake.rs:2713`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L2713-L2724) - [ ] Arms in `test_copy_into_file_format` (78 tokens, `88464afb`). [`tests/sqlparser_snowflake.rs:2636`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L2636-L2655) · [`tests/sqlparser_snowflake.rs:2678`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L2678-L2697) - [ ] Blocks in `test_snowflake_stage_object_names_into_location` and `test_snowflake_stage_object_names_into_table` (37 tokens, `cfc10449`). [`tests/sqlparser_snowflake.rs:2754`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L2754-L2763) · [`tests/sqlparser_snowflake.rs:2777`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/tests/sqlparser_snowflake.rs#L2777-L2786) ### Benchmarks Review whether sharing setup would change what each benchmark measures. - [ ] Closures in `parse_table_factor_paren_chain` and `parse_function_arg_call_chain` (33 tokens, `3c68771c`). [`sqlparser_bench/benches/sqlparser_bench.rs:263`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/sqlparser_bench/benches/sqlparser_bench.rs#L263-L270) · [`sqlparser_bench/benches/sqlparser_bench.rs:287`](https://github.com/apache/datafusion-sqlparser-rs/blob/68e626d16b3c5d48cadbce01b3f0839d5f4db30b/sqlparser_bench/benches/sqlparser_bench.rs#L287-L294) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
