[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user yoelb commented on the issue: https://github.com/apache/spark/pull/19687 I have the same issue @ManchesterUnited16 , did you find a solution? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/19687 @ManchesterUnited16 Since I tested your codes and it works for me, I'm going to merge this PR and backport it to 2.2. If it's a real issue, feel free to open a ticket with steps to reproduce it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/19687 > can you show me you maven dependency when you ran the program,thank you very much! maven dependency is used only when I compiled the codes. I used the official Spark 2.2.0 to compile the codes, and built Spark based on this PR (commit hash: f88fb50aec853f45dfc213cbeb9851a270ba3cbe) and ran my problem using it. Could you show how did you compile the codes, how did you build Spark, and how did you run the codes? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user ManchesterUnited16 commented on the issue: https://github.com/apache/spark/pull/19687 can you show me you maven dependency when you ran the program,thank you very much! At 2017-11-09 13:37:46, "Shixiong Zhu" wrote: @ManchesterUnited16 I ran your codes and didn't see NotSerializableException. How did you patch Spark with my PR? â You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/19687 @ManchesterUnited16 I ran your codes and didn't see `NotSerializableException`. How did you patch Spark with my PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user ManchesterUnited16 commented on the issue: https://github.com/apache/spark/pull/19687 import org.apache.spark.ml.evaluation.RegressionEvaluator import org.apache.spark.ml.recommendation.{ALS, ALSModel} import org.apache.spark.ml.recommendation.ALS.Rating import org.apache.spark.rdd.RDD import org.apache.spark.sql.{DataFrame, Row} // $example off$ import org.apache.spark.sql.SparkSession object ALSExample { // $example on$ case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long) def parseRating(str: String): Rating = { val fields = str.split("::") assert(fields.size == 4) Rating(fields(0).toInt, fields(1).toInt, fields(2).toFloat,fields(3).toLong) } // $example off$ def main(args: Array[String]) { val spark = SparkSession .builder .master("local") .appName("ALSExample") .getOrCreate() import spark.implicits._ // $example on$ val ratings: DataFrame = spark.read.textFile("D:\\xcar\\Spark_MLib\\ml_2.11.1\\src\\data\\mllib\\als\\sample_movielens_ratings.txt") .map(parseRating) .toDF() val Array(training, test) = ratings.randomSplit(Array(0.8, 0.2)) // Build the recommendation model using ALS on the training data val als = new ALS() .setMaxIter(5) .setRegParam(0.01) .setUserCol("userId") .setItemCol("movieId") .setRatingCol("rating") val model: ALSModel = als.fit(training) // Evaluate the model by computing the RMSE on the test data // Note we set cold start strategy to 'drop' to ensure we don't get NaN evaluation metrics model.setColdStartStrategy("drop") val predictions: DataFrame = model.transform(test) val evaluator: RegressionEvaluator = new RegressionEvaluator() .setMetricName("rmse") .setLabelCol("rating") .setPredictionCol("prediction") val rmse = evaluator.evaluate(predictions) println(s"Root-mean-square error = $rmse") // Generate top 10 movie recommendations for each user val userRecs: DataFrame = model.recommendForAllUsers(10) // Generate top 10 user recommendations for each movie val movieRecs: DataFrame = model.recommendForAllItems(10) // Generate top 10 movie recommendations for a specified set of users //val users = ratings.select(als.getUserCol).distinct().limit(3) //val userSubsetRecs = model.recommendForUserSubset(users, 10) //// Generate top 10 user recommendations for a specified set of movies //val movies = ratings.select(als.getItemCol).distinct().limit(3) //val movieSubSetRecs = model.recommendForItemSubset(movies, 10) // $example off$ userRecs.show() //movieRecs.show() //userSubsetRecs.show() //movieSubSetRecs.show() spark.stop() } } this is my code ,and when I run the line"userRecs.show()",there are some error "java.io.NotSerializableException: scala.reflect.api.TypeTags$PredefTypeCreator" At 2017-11-09 05:29:55, "Shixiong Zhu" wrote: @zsxwing commented on this pull request. In sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala: > @@ -441,4 +443,28 @@ class ExpressionEncoderSuite extends PlanTest with AnalysisTest { } } } + + /** + * Verify the size of scala.reflect.runtime.JavaUniverse.undoLog before and after `func` to + * ensure we don't leak Scala reflection garbage. + * + * @see org.apache.spark.sql.catalyst.ScalaReflection.cleanUpReflectionObjects + */ + private def verifyNotLeakingReflectionObjects[T](func: => T): T = { +def undoLogSize: Int = { + import scala.reflect.runtime.{JavaUniverse, universe} No special reason. I changed to use the fully qualified class name now. â You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19687 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83608/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19687 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19687 **[Test build #83608 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83608/testReport)** for PR 19687 at commit [`f88fb50`](https://github.com/apache/spark/commit/f88fb50aec853f45dfc213cbeb9851a270ba3cbe). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/19687 > LGTM, is it targeted for branch 2.2 too? Yeah, I will backport this to 2.2 once this PR gets merged. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/19687 @ManchesterUnited16 `cleanUpReflectionObjects` introduces new closures but those are created in `object ScalaReflection` so should be safe. Did you backport my PR to an old Spark version? If so, you probably hit https://github.com/apache/spark/pull/17639. Anyway, I added some defensive codes in the test to make sure encoders are serializable. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19687 **[Test build #83608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83608/testReport)** for PR 19687 at commit [`f88fb50`](https://github.com/apache/spark/commit/f88fb50aec853f45dfc213cbeb9851a270ba3cbe). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/19687 > java.io.NotSerializableException: scala.reflect.api.TypeTags$PredefTypeCreator @ManchesterUnited16 Probably this change may capture some unnecessary objects. Could you provide your codes to help me investigate it? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19687 LGTM, is it targeted for branch 2.2 too? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user ManchesterUnited16 commented on the issue: https://github.com/apache/spark/pull/19687 java.io.NotSerializableException: scala.reflect.api.TypeTags$PredefTypeCreator --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user ManchesterUnited16 commented on the issue: https://github.com/apache/spark/pull/19687 there is something wrong in my program --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19687 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83571/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19687 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19687 **[Test build #83571 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83571/testReport)** for PR 19687 at commit [`c03811f`](https://github.com/apache/spark/commit/c03811ff006058987fa8d5fb9f7d097b9acc9ac5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19687 **[Test build #83571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83571/testReport)** for PR 19687 at commit [`c03811f`](https://github.com/apache/spark/commit/c03811ff006058987fa8d5fb9f7d097b9acc9ac5). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19687: [SPARK-19644][SQL]Clean up Scala reflection garbage afte...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/19687 cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org