Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/18645
Looking at the source compatibility changes made here, I was a little
confused about why we didn't need to make many more changes. In principle, it
seemed like the ambiguous overload issue affecting that `foreachPartition` call
should have also impacted calls like `.filter(_ > 0)`, so I was originally
expecting that we'd have to update hundreds of call sites.
It turns out that the final Scala 2.12 release has a nice feature which
resolves this ambiguity in many cases. In the release notes at
https://www.scala-lang.org/news/2.12.0/#lambda-syntax-for-sam-types, there's an
example where even though there's a potentially ambiguous overload,
"overloading resolution selects the one with the `Function1` argument type".
This is explained in more detail at
https://www.scala-lang.org/news/2.12.0/#sam-conversion-in-overloading-resolution.
Quoting:
> In Scala 2.12, both methods are applicable, therefore [overloading
resolution](http://www.scala-lang.org/files/archive/spec/2.12/06-expressions.html#overloading-resolution)
needs to pick the most specific alternative. The specification for [type
compatibility](http://www.scala-lang.org/files/archive/spec/2.12/03-types.html#compatibility)
has been updated to consider SAM conversion, so that the first alternative is
more specific.
This explains why we didn't have to update all call sites. However, why did
the `.foreachPartition` example break? I played around with this in the Scastie
Scala paste bin and I think that I've spotted the problem: all of the cases
where we had ambiguity seem to be cases where we're operating on Datasets in a
generic way and have a type parameter representing the Dataset's type.
So, this means that
```
def f(ds: Dataset[Int]): Dataset[Int] = ds.filter(_ != null)
```
is unambiguous but
```
def f[T](ds: Dataset[T]): Dataset[T] = ds.filter(_ != null)
```
fails with a compiler error in Scala 2.12. You can try this out at
https://scastie.scala-lang.org/JoshRosen/eBcxUGdORsiOQNULMUnAsw.
I took a look at the sections of the language spec linked in the quote
above but it's not immediately clear to me whether this is a compiler bug or
expected behavior (I'll have to take some more time to try to understand the
spec).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
