Github user debasish83 commented on the pull request:
https://github.com/apache/spark/pull/3221#issuecomment-85161041
@mengxr I added the optimization for lower triangular matrix and now they
are very close...Let me know what do you think and if there are any other
tricks you would like me to try...Note that with these optimization,
QuadraticMinimizer with POSITIVE constraint will also run much faster
Breeze QuadraticMinimizer (default):
unset solver; ./bin/spark-submit --master
spark://tusca09lmlvt00c.uswin.ad.vzwcorp.com:7077 --class
org.apache.spark.examples.mllib.MovieLensALS --jars
~/.m2/repository/com/github/scopt/scopt_2.10/3.2.0/scopt_2.10-3.2.0.jar
--total-executor-cores 1
./examples/target/spark-examples_2.10-1.3.0-SNAPSHOT.jar --rank 50
--numIterations 2 ~/datasets/ml-1m/ratings.dat
Got 1000209 ratings from 6040 users on 3706 movies.
Training: 800670, test: 199539.
Quadratic minimization userConstraint SMOOTH productConstraint SMOOTH
Running Breeze QuadraticMinimizer for users with constraint SMOOTH
Running Breeze QuadraticMinimizer for items with constraint SMOOTH
Test RMSE = 2.4985081126233846.
15/03/23 12:26:55 INFO ALS: solveTime 205.379 ms
15/03/23 12:26:55 INFO ALS: solveTime 72.116 ms
15/03/23 12:26:56 INFO ALS: solveTime 74.034 ms
15/03/23 12:26:56 INFO ALS: solveTime 77.379 ms
15/03/23 12:26:57 INFO ALS: solveTime 36.532 ms
15/03/23 12:26:57 INFO ALS: solveTime 29.775 ms
15/03/23 12:26:58 INFO ALS: solveTime 48.925 ms
15/03/23 12:26:58 INFO ALS: solveTime 51.904 ms
15/03/23 12:26:59 INFO ALS: solveTime 30.882 ms
15/03/23 12:26:59 INFO ALS: solveTime 30.658 ms
ML CholeskySolver:
export solver=mllib; ./bin/spark-submit --master
spark://tusca09lmlvt00c.uswin.ad.vzwcorp.com:7077 --class
org.apache.spark.examples.mllib.MovieLensALS --jars
~/.m2/repository/com/github/scopt/scopt_2.10/3.2.0/scopt_2.10-3.2.0.jar
--total-executor-cores 1
./examples/target/spark-examples_2.10-1.3.0-SNAPSHOT.jar --rank 50
--numIterations 2 ~/datasets/ml-1m/ratings.dat
Got 1000209 ratings from 6040 users on 3706 movies.
Training: 800670, test: 199539.
Quadratic minimization userConstraint SMOOTH productConstraint SMOOTH
Test RMSE = 2.4985081126233846.
TUSCA09LMLVT00C:spark-qp-als v606014$ grep solveTime
./work/app-20150323122612-0002/0/stderr
15/03/23 12:26:20 INFO ALS: solveTime 102.243 ms
15/03/23 12:26:21 INFO ALS: solveTime 38.195 ms
15/03/23 12:26:21 INFO ALS: solveTime 60.583 ms
15/03/23 12:26:22 INFO ALS: solveTime 59.882 ms
15/03/23 12:26:22 INFO ALS: solveTime 36.59 ms
15/03/23 12:26:23 INFO ALS: solveTime 36.021 ms
15/03/23 12:26:23 INFO ALS: solveTime 59.271 ms
15/03/23 12:26:24 INFO ALS: solveTime 59.217 ms
15/03/23 12:26:24 INFO ALS: solveTime 36.344 ms
15/03/23 12:26:25 INFO ALS: solveTime 35.838 ms
I am running only 2 iterations but you can see in the tail the solvers run
at par...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
