wangyum opened a new issue, #12566: URL: https://github.com/apache/gluten/issues/12566
## Description `BatchScanExecTransformer.doCanonicalize()` does not normalize `keyGroupedPartitioning`, and `BatchScanExecTransformerBase.hashCode()` does not include it. This is inconsistent with `equals()`, which (via `super.equals()` → `AbstractBatchScanExec.equals()`) compares `spjParams` — which contains `keyGroupedPartitioning`. ## Impact 1. **Broken exchange reuse (AQE):** When two `BatchScanExecTransformer` instances are semantically identical but have different expression IDs in their `keyGroupedPartitioning` expressions (e.g. different `exprId` on `AttributeReference`), `doCanonicalize()` does not normalize the `keyGroupedPartitioning` field. The canonicalized plans are therefore not equal, and AQE cannot reuse the exchange — resulting in a **redundant shuffle**. 2. **equals/hashCode contract violation:** `equals()` returns `true` for two plans with the same `keyGroupedPartitioning`, but `hashCode()` doesn't include `keyGroupedPartitioning` in its computation. Two equal objects may have different hash codes, violating the contract. ## Comparison with vanilla Spark Vanilla Spark's `BatchScanExec.equals/hashCode` uses `spjParams` which wraps `keyGroupedPartitioning`, `commonPartitionValues`, `applyPartialClustering`, and `replicatePartitions`. So `equals` and `hashCode` are both (indirectly) consistent with respect to `keyGroupedPartitioning`. Gluten's `BatchScanExecTransformerBase.hashCode()` returns `Objects.hashCode(batch, runtimeFilters, pushDownFilters)` — missing `keyGroupedPartitioning` (and the other SPJ fields). This was likely an oversight when `pushDownFilters` was added. ## Proposed fix 1. Add `keyGroupedPartitioning` normalization to `doCanonicalize()` so canonicalized plans with different expression IDs are properly equalized. 2. Add `keyGroupedPartitioning` to `hashCode()` to maintain the equals/hashCode contract. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
