peter-toth commented on PR #55885:
URL: https://github.com/apache/spark/pull/55885#issuecomment-5997876779
@metanil, #59165 is merged (master `05a20fb758b`, branch-4.x `d8d72d2d3b1`),
so you can rebase onto it now. A few notes, since this PR touches the same code:
- The guard sits in `KeyedShuffleSpec.isExpressionCompatible` now. A
transform whose argument is not a bare column pairs only with a transform over
the same argument shape (`argumentShape`). `canCreatePartitioning` refuses such
a layout. `createPartitioning` and the identity reduce rebuild a transform
through `rebuiltOver`, which asserts that the arguments are columns.
`TransformExpression.withReference` is gone.
- With that gate in place, the total `functionId` from my withdrawn finding
4 is safe again: `bucket(4, b + 1)` can no longer pair with `bucket(4, x)`,
whatever the identity says. Keeping the `Option` is fine too, it's your call.
- This PR puts the literal parameters into `children`, so two of the new
helpers need to skip them:
- `argumentsAreColumns` checks
`children.forall(_.isInstanceOf[Attribute])`. With a literal parameter, every
`bucket` and `truncate` would count as a transform of an expression, so no side
could be shuffled onto it any more.
- `rebuiltOver` replaces every child with the other side's key, a literal
parameter included. Your `rewriteColumnSlots` looks like the right replacement.
- Please keep `argumentsAreColumns` on bare `Attribute`s rather than
switching it to `isColumnRef`. A `GetStructField` argument must stay refused
there: rebuilding `bucket(4, s.a)` over the other side's key gives `bucket(4,
s)`, which is the `ClassCastException` #59165 fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
