peter-toth commented on PR #59760: URL: https://github.com/apache/spark/pull/59760#issuecomment-6397826478
@metanil, a proposal that follows from SPARK-60027, see [my comment on #55885](https://github.com/apache/spark/pull/55885#issuecomment-6397822070). A transform whose arguments differ from its function's `inputTypes()` is another case Spark cannot evaluate as is. The call gets the raw values, and it fails with a `ClassCastException` or returns wrong rows. It happens on the two paths `canEvaluate` guards. So `canEvaluate` could also ask that the arguments, the bucket count from `numBucketsOpt` included, match `inputTypes()` in number and type. That would fix SPARK-60027 here too, and keep it backportable with this PR. Two tests would follow: - `ShuffleSpecSuite`'s `TestBucketFunction`, `FakeBucket` and the local `bucketFn` declare no type for the bucket count, so the count check would refuse them. #55885 already gives them one. - `inheritedBucket` has the same shape, so the count check would refuse it before the inherited `produceResult` comes into play. An `IntegerType` for the count keeps it testing what its name says. The Iceberg `TINYINT`/`SMALLINT` point from that comment applies here too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
