Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1687#issuecomment-50766537
@srowen This is great! I was about to add the same function. We only have
`RDD.top` in core, but if we have `PairRDDFunctions.topByKey`, we can make
recommendation on a small batch of users/products. Another possible
optimization is to use more partitions for the final user/product features.
Beside some inline comments, could you also set the storage level of the final
user/product features to `MEMORY_AND_DISK` in this PR? We saw use cases where
the features were kicked out of memory by later jobs on the same cluster.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
