Github user heathermiller commented on the pull request:
https://github.com/apache/spark/pull/3262#issuecomment-63276199
TLDR; this is a really clever PR! Though I think in a kind of contrived
way, it's not *technically* source compatible. I think this can easily be
remedied by a small refactoring though.
1. I think it's possible to break *source* compatibility as follows:
- some Spark application imports `org.apache.spark.rdd._`
- there already exists a user-defined method `rddToPairRDDFunctions`
- then the implicit def in the `rdd` package object is now shadowed,
and no longer eligible.
- since these methods are not implicit in `SparkContext` anymore, code
that calls a method provided by the target type of `rddToPairRDDFunctions` no
longer compiles.
Admittedly, this might be a contrived example, and maybe you don't care
about supporting this case.
2. I'm a bit surprised by the following scenario in:
https://github.com/apache/spark/pull/3262/files#diff-67682b28156f41b39d6a5e305541c084R37.
Here, we call `groupByKey` on an rdd, but the `rddToPairRDDFunctions` implicit
def must be called first. The question is why this implicit def is found when
it's in the rdd package object. It must be in the implicit scope of
`org.apache.spark.rdd.RDD[(Int, Int)]`. The implicit scope contains all
implicits in companion objects of types that are part of that type. Since the
rdd package object is not a companion object, it's a bit surprising that its
implicits nonetheless are in the implicit scope. (Of note, this behavior is not
speced) In the Scala stdlib, for example, there are no implicits in the `scala`
package object â the implicits are all in `Predef`.
I think both point 1 & 2 can be remedied by instead putting these implicits
in an RDD companion object (which currently doesn't seem to exist).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
