Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/20481#discussion_r165512101
--- Diff: core/src/main/scala/org/apache/spark/status/KVUtils.scala ---
@@ -69,14 +69,17 @@ private[spark] object KVUtils extends Logging {
db
}
- /** Turns a KVStoreView into a Scala sequence, applying a filter. */
- def viewToSeq[T](
- view: KVStoreView[T],
- max: Int)
- (filter: T => Boolean): Seq[T] = {
+ /**
+ * Turns a KVStoreView into a Scala sequence, applying a filter, sorting
the sequence and
+ * selecting the first `max` values.
+ */
+ def viewToSeq[T, S: Ordering](
+ view: KVStoreView[T],
+ max: Int)
+ (filter: T => Boolean)(sorter: T => S): Seq[T] = {
val iter = view.closeableIterator()
try {
- iter.asScala.filter(filter).take(max).toList
+ iter.asScala.filter(filter).toList.sortBy(sorter).take(max)
--- End diff --
@vanzin Yeah, I understand the expensive sort. However, adding indices
needs more work. Do you have time to try it since I'm not familiar with LevelDB?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
