Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/18645#discussion_r128891202
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskContextSuite.scala ---
@@ -54,7 +54,10 @@ class TaskContextSuite extends SparkFunSuite with
BeforeAndAfter with LocalSpark
val rdd = new RDD[String](sc, List()) {
override def getPartitions = Array[Partition](StubPartition(0))
override def compute(split: Partition, context: TaskContext) = {
- context.addTaskCompletionListener(context =>
TaskContextSuite.completed = true)
+ context.addTaskCompletionListener(new TaskCompletionListener {
--- End diff --
I think that you can avoid source incompatibilities for Scala users by
removing the overloads which accept Scala functions and then adding in a
package-level implicit conversion to convert from Scala functions back into our
own custom trait / interface.
The trickiness here is that you need to preserve binary compatibility on
Scala 2.10/2.11, so the removal of the overload needs to be done conditionally
so that it only occurs when building with Scala 2.12. Rather than having a
separate source tree for 2.12, I'd propose defining the removed overload in a
mixin trait which comes from a separate source file and then configure the
build to use different versions of that file for 2.10/2.11 and for 2.12.
For details on this proposal, see
https://docs.google.com/document/d/1P_wmH3U356f079AYgSsN53HKixuNdxSEvo8nw_tgLgM/edit,
a document that I wrote in March 2016 which explores these source
incompatibility difficulties.
Applying that idea here, the idea would be to remove the method
```
def addTaskCompletionListener(f: (TaskContext) => Unit)
```
and add a package-level implicit conversion from `TaskContext => Unit` to
`TaskCompletionListener`, but to do this only in the 2.12 source tree / shim.
This approach has some caveats and could potentially impact Java users who are
doing weird things (violating the goal that Java Spark code is source and
binary compatible with all Scala versions). See the linked doc for a full
discussion of this problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
