peter-toth commented on PR #58013: URL: https://github.com/apache/spark/pull/58013#issuecomment-5302409828
Thanks @szehon-ho! All three are in, plus the two variants you mentioned along the way. 1. Added `spark.sql.sources.v2.bucketing.partition.filter.enabled` with an inner join and the join keys covering all of the partition keys, so no projection is involved at all: `table1` is partitioned by `(store_id, dept_id)` and has one key `table2` doesn't, so the intersection ends up smaller than what the scan reports back. Worth having - on the unfixed code it fails the count check rather than the subset check: `Before: 2 partition values. After: 3 partition values`. I also added the outer-join shape you described, where the values become one side's: a `LEFT OUTER` join whose right side reports a key the left doesn't have. And the other path, `allowCompatibleTransforms`, with `bucket(4, ...)` against `bucket(2, ...)` - there the pushed values are reduced into the two-bucket space while the keys the scan reports stay in its own, so it fails the same way. All three fail on `branch-4.1` today and pass with the fix. 2. Added `Seq(1)` over `(int, string)` partition keys. One detail: it doesn't fail in the hash. `InterpretedHashFunction.hash` matches on the value rather than the declared type, so the int under a `StringType` just hashes as an int and gets through. It fails in the ordering instead, once the raw key's hash collides with a projected one: `java.lang.ClassCastException: class java.lang.Integer cannot be cast to class org.apache.spark.unsafe.types.UTF8String` at `BaseGenericInternalRow.getUTF8String`, from `InternalRowComparableWrapper.equals`. 3. Renamed both: "the join key is a trailing partition key" and "the join keys have a gap in the partition keys". One thing beyond your list: every test now asserts the dynamic filter is pushed to the specific scan the case needs it on, rather than to any scan - `spjParams.joinKeyPositions` where there is a projection, `spjParams.reducers` for the bucket case, and the table name otherwise. Without that they'd stay green if the filter ever moved to the other side, which can't reproduce. Since four of the six carry no projection, I also reworded the description to lead with the general trigger - the scan's `outputPartitioning` values not being a superset of the keys it reported itself - with the projection as the practically reachable case rather than the definition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
