Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4675#discussion_r25042474
--- Diff:
examples/src/main/scala/org/apache/spark/examples/ml/MovieLensALS.scala ---
@@ -157,17 +157,23 @@ object MovieLensALS {
println(s"Test RMSE = $rmse.")
// Inspect false positives.
- predictions.registerTempTable("prediction")
-
sc.textFile(params.movies).map(Movie.parseMovie).toDF().registerTempTable("movie")
- sqlContext.sql(
- """
- |SELECT userId, prediction.movieId, title, rating, prediction
- | FROM prediction JOIN movie ON prediction.movieId = movie.movieId
- | WHERE rating <= 1 AND prediction >= 4
- | LIMIT 100
- """.stripMargin)
- .collect()
- .foreach(println)
+ // Note: We reference columns in 2 ways:
+ // (1) predictions("movieId") lets us specify the movieId column in
the predictions
+ // DataFrame, rather than the movieId column in the movies
DataFrame.
+ // (2) $"userId" specifies the userId column in the predictions
DataFrame.
+ // We could also write predictions("userId") but do not have to
since
+ // the movies DataFrame does not have a column "userId."
+ val movies = sc.textFile(params.movies).map(Movie.parseMovie).toDF()
+ val falsePositives = predictions.join(movies)
+ .where(predictions("movieId") === movies("movieId"))
+ .where($"rating" <= 1).where($"prediction" >= 4)
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
