[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-49193225 QA tests have started for PR 929. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16733/consoleFull ---

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-49206560 QA results for PR 929:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-47180387 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-47180380 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-47182436 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16135/ --- If your project

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-47182435 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-45067410 I don't think checkpointing every 3 iterations is a good choice. Could you try different values, e.g., 3 and 20, and check the running times? Also, it doesn't solve the

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user witgo commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-45069637 @mengxr By calling this method `RDD.checkpoint`, `ContextCleaner` can clean up the shuffle data, reduce disk usage. Just as described in the table below.

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user witgo commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-45070672 As for the above data. One iteration write `160G` shuffle data . Three iterations will have occupied `480G` hard disk --- If your project is set up for it, you can reply

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user witgo commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-45071297 @mengxr Since I only have three test server, I need more time to test your ideas. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-45073179 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-45073488 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-45076638 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: Improve ALS algorithm resource usage

GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/929 Improve ALS algorithm resource usage You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark improve_als Alternatively you can review and

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-44742009 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-44742011 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-44742189 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-44742193 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-44742668 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15317/ --- If your project

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-44742667 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-44742949 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15318/ --- If your project

[GitHub] spark pull request: Improve ALS algorithm resource usage

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/929#issuecomment-44742948 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.