mosche opened a new issue, #24314: URL: https://github.com/apache/beam/issues/24314
### What happened? When looking at [TPC-DS query 83](https://github.com/apache/beam/blob/883a362c930aca4298551697d7aaacbe7b6602f1/sdks/java/testing/tpcds/src/main/resources/queries/query83.sql) I noticed that the generated `PTransform` is extremely complex and not optimized as it should be. One issue I observed is that we're exploding / denormalizing the `RelNode` graph during the conversion to `BeamRelNode`s. For instance, in the case of query 83 this is happening for this subquery. ``` select d_week_seq from date_dim where d_date in ('1998-01-02','1998-10-15','1998-11-10') ``` Initially the repeated reference was normalized into a single `RelNode`. Though, when applying the conversion rules we might end up creating [multiple copies](https://github.com/apache/calcite/blob/e37ecb63e0c59aec7a4ba79b4d3473dd1cd2cfed/core/src/main/java/org/apache/calcite/plan/volcano/RelSubset.java#L734-L739), leading to independent PTransforms of the same "SQL" for each occurrence. Below a visualization of the denormalized transform hierarchy for TPC-DS query 83, followed by a similar one using [a hack](https://github.com/mosche/beam/commit/874fa34e7ed8567678443775adf520d326a30243) to keep it in its normalized shape. ### "Denormalized" transform hierarchy  ### "Normalized" transform hierarchy  ### Issue Priority Priority: 2 ### Issue Component Component: dsl-sql -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
