Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/369#issuecomment-40031343
How would you write the signature with the implicit Ordering parameter as
well?
I tried with the below definition and below test
```
def sortBy[K: ClassTag](
f: (T) â K,
ascending: Boolean = true,
numPartitions: Int = this.partitions.size)
(implicit ord: Ordering[K]): RDD[T] =
this.keyBy[K](f)
.sortByKey(ascending, numPartitions)
.values
```
```
test("sortByKey with explicit ordering") {
val data = sc.parallelize(Seq("Bob|Smith|50", "Jane|Smith|40",
"Thomas|Williams|30", "Karen|Williams|60"))
val ageOrdered = Array("Thomas|Williams|30", "Jane|Smith|40",
"Bob|Smith|50", "Karen|Williams|60")
// last name, then first name
val nameOrdered = Array("Bob|Smith|50", "Jane|Smith|40",
"Karen|Williams|60", "Thomas|Williams|30")
case class Person(first: String, last: String, age: Int)
def parse(s: String): Person = {
val split = s.split("\\|")
Person(split(0), split(1), split(2).toInt)
}
object AgeOrdering extends Ordering[Person] {
def compare(a:Person, b:Person) = a.age compare b.age
}
object NameOrdering extends Ordering[Person] {
def compare(a:Person, b:Person) =
implicitly[Ordering[Tuple2[String,String]]].compare((a.last,
a.first), (b.last, b.first))
}
assert(data.sortBy(parse, false, 2)(AgeOrdering) === ageOrdered)
assert(data.sortBy(parse, false, 2)(NameOrdering) === nameOrdered)
}
```
But got an error that I'd have to pass in the ClassTag implicitly as well:
```
[error]
/Users/aash/git/spark/core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala:555:
not enough arguments for method sortBy: (implicit evidence$5:
scala.reflect.ClassTag[Person], implicit ord:
Ordering[Person])org.apache.spark.rdd.RDD[String].
[error] Unspecified value parameter ord.
[error] assert(data.sortBy(parse, false, 2)(AgeOrdering) === ageOrdered)
[error] ^
[error]
/Users/aash/git/spark/core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala:556:
not enough arguments for method sortBy: (implicit evidence$5:
scala.reflect.ClassTag[Person], implicit ord:
Ordering[Person])org.apache.spark.rdd.RDD[String].
[error] Unspecified value parameter ord.
[error] assert(data.sortBy(parse, false, 2)(NameOrdering) ===
nameOrdered)
[error] ^
[error] two errors found
[error] (core/test:compile) Compilation failed
[error] Total time: 9 s, completed Apr 10, 2014 1:13:27 AM
aash@aash-mbp ~/git/spark$
```
Is there a way to set the ordering of the implicit parameters so that the
Ordering goes before the ClassTag ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
