akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688029062
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in
RDD.treeAggregate to executor
+
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
@venkata91 and I tried doing this and ran into a couple issues while
trying to write the overloaded function. Seems like the issue is scala while
resolving overloaded methods, it looks at the first argument and if the type is
same on both the methods then that becomes ambiguous. Here is the version of
the overloaded method we were trying with (along with several other variations
of this :) ):
```
def treeAggregate[U: ClassTag](zeroValue: U)(
seqOp: (U, T) => U,
combOp: (U, U) => U,
depth: Int = 2): U = withScope {
treeAggregate(zeroValue)(seqOp, combOp, depth, false)
}
```
any suggestions? @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
