rluvaton commented on PR #4657:
URL:
https://github.com/apache/datafusion-comet/pull/4657#issuecomment-4728702032
> Thanks @rluvaton. I will take a look at this soon, but I am concerned that
this is more complex than the current approach.
I think this will:
## Reduce line count
you don't need to duplicate shims code when they are the same
## Make it easier to read
When you look at specific shim directory like `spark 3.5` you should also
pay attention to the shim directory `spark 3.x` instead of just looking at
the shim code and understand right away
## Adding shims from specific spark version and up
for example, `HashPartitioningLike` was added in spark 4 and backported to
spark `3.5` and spark `3.4` and released in spark `3.5.1`
in the current approach, how would you create a function that check if spark
partitioning is hash based if the `HashPartitioningLike` was **not** backported
to spark 3.4 **which you still have shims for**?
with this pr you could write:
```scala
@enableIfVer(spark = ">= 3.5.1")
def isHashPartitioningLike(partitioning: Partitioning): Boolean = {
import org.apache.spark.sql.catalyst.plans.physical.HashPartitioningLike
partitioning match {
case hashLike: HashPartitioningLike => true
case par => false
}
}
@enableIfVer(spark = "< 3.5.1")
def isHashPartitioningLike(partitioning: Partitioning): Boolean = {
import org.apache.spark.sql.catalyst.plans.physical.HashPartitioning
partitioning match {
case hash: HashPartitioning => true
case par => false
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
