metanil commented on PR #59760: URL: https://github.com/apache/spark/pull/59760#issuecomment-6433729381
> @metanil, a proposal that follows from [SPARK-60027](https://issues.apache.org/jira/browse/SPARK-60027), see [my comment on #55885](https://github.com/apache/spark/pull/55885#issuecomment-6397822070). A transform whose arguments differ from its function's `inputTypes()` is another case Spark cannot evaluate as is. The call gets the raw values, and it fails with a `ClassCastException` or returns wrong rows. It happens on the two paths `canEvaluate` guards. So `canEvaluate` could also ask that the arguments, the bucket count from `numBucketsOpt` included, match `inputTypes()` in number and type. That would fix [SPARK-60027](https://issues.apache.org/jira/browse/SPARK-60027) here too, and keep it backportable with this PR. > > Two tests would follow: > > * `ShuffleSpecSuite`'s `TestBucketFunction`, `FakeBucket` and the local `bucketFn` declare no type for the bucket count, so the count check would refuse them. [[SPARK-50593][SQL] SPJ: Support truncate transform via generalized ReducibleFunction API #55885](https://github.com/apache/spark/pull/55885) already gives them one. > * `inheritedBucket` has the same shape, so the count check would refuse it before the inherited `produceResult` comes into play. An `IntegerType` for the count keeps it testing what its name says. > > The Iceberg `TINYINT`/`SMALLINT` point from that comment applies here too. @peter-toth Thanks for looking into this. Coincidentally, I also stumbled into this issue, glad you created the JIRA. I really like your idea and agree that this PR could cover [SPARK-60027](https://issues.apache.org/jira/browse/SPARK-60027) as well. I'll definitely incorporate it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
