Github user debasish83 commented on the pull request:
https://github.com/apache/spark/pull/5005#issuecomment-83045631
I updated with the @mengxr checkin that cleaned jblas dependencies from
NNLS....the results updated above is with rank = 10 I think...A more realistic
rank is 50...
breeze nnls (including iterator overhead):
15/03/18 09:21:45 INFO ALS: solveTime 177.048 ms
15/03/18 09:21:45 INFO ALS: solveTime 189.642 ms
15/03/18 09:21:45 INFO ALS: solveTime 171.894 ms
15/03/18 09:21:45 INFO ALS: solveTime 203.503 ms
15/03/18 09:21:45 INFO ALS: solveTime 185.352 ms
15/03/18 09:21:45 INFO ALS: solveTime 190.051 ms
15/03/18 09:21:45 INFO ALS: solveTime 181.634 ms
15/03/18 09:21:45 INFO ALS: solveTime 148.325 ms
15/03/18 09:21:46 INFO ALS: solveTime 59.523 ms
15/03/18 09:21:46 INFO ALS: solveTime 78.501 ms
15/03/18 09:21:46 INFO ALS: solveTime 66.649 ms
15/03/18 09:21:46 INFO ALS: solveTime 66.078 ms
15/03/18 09:21:46 INFO ALS: solveTime 75.827 ms
15/03/18 09:21:46 INFO ALS: solveTime 50.72 ms
15/03/18 09:21:46 INFO ALS: solveTime 65.047 ms
15/03/18 09:21:46 INFO ALS: solveTime 59.184 ms
15/03/18 09:21:46 INFO ALS: solveTime 26.837 ms
15/03/18 09:21:46 INFO ALS: solveTime 26.061 ms
15/03/18 09:21:46 INFO ALS: solveTime 36.14 ms
15/03/18 09:21:46 INFO ALS: solveTime 33.341 ms
15/03/18 09:21:46 INFO ALS: solveTime 29.399 ms
15/03/18 09:21:46 INFO ALS: solveTime 28.61 ms
15/03/18 09:21:46 INFO ALS: solveTime 30.074 ms
15/03/18 09:21:46 INFO ALS: solveTime 32.351 ms
15/03/18 09:21:46 INFO ALS: solveTime 56.625 ms
15/03/18 09:21:47 INFO ALS: solveTime 54.29 ms
15/03/18 09:21:47 INFO ALS: solveTime 49.305 ms
15/03/18 09:21:47 INFO ALS: solveTime 63.742 ms
15/03/18 09:21:47 INFO ALS: solveTime 67.626 ms
15/03/18 09:21:47 INFO ALS: solveTime 55.348 ms
15/03/18 09:21:47 INFO ALS: solveTime 54.473 ms
15/03/18 09:21:47 INFO ALS: solveTime 66.704 ms
15/03/18 09:21:47 INFO ALS: solveTime 29.57 ms
15/03/18 09:21:47 INFO ALS: solveTime 33.871 ms
15/03/18 09:21:47 INFO ALS: solveTime 33.329 ms
15/03/18 09:21:47 INFO ALS: solveTime 40.322 ms
15/03/18 09:21:47 INFO ALS: solveTime 35.376 ms
15/03/18 09:21:47 INFO ALS: solveTime 35.826 ms
15/03/18 09:21:47 INFO ALS: solveTime 38.973 ms
15/03/18 09:21:47 INFO ALS: solveTime 34.908 ms
breeze nnls inner solve (subtracting iterator time):
15/03/18 09:21:45 INFO ALS: innerTime 103.235 ms
15/03/18 09:21:45 INFO ALS: innerTime 115.842 ms
15/03/18 09:21:45 INFO ALS: innerTime 104.233 ms
15/03/18 09:21:45 INFO ALS: innerTime 112.84 ms
15/03/18 09:21:45 INFO ALS: innerTime 97.474 ms
15/03/18 09:21:45 INFO ALS: innerTime 91.188 ms
15/03/18 09:21:45 INFO ALS: innerTime 115.043 ms
15/03/18 09:21:45 INFO ALS: innerTime 96.443 ms
15/03/18 09:21:46 INFO ALS: innerTime 31.696 ms
15/03/18 09:21:46 INFO ALS: innerTime 40.446 ms
15/03/18 09:21:46 INFO ALS: innerTime 42.288 ms
15/03/18 09:21:46 INFO ALS: innerTime 34.147 ms
15/03/18 09:21:46 INFO ALS: innerTime 33.404 ms
15/03/18 09:21:46 INFO ALS: innerTime 28.02 ms
15/03/18 09:21:46 INFO ALS: innerTime 33.372 ms
15/03/18 09:21:46 INFO ALS: innerTime 38.531 ms
15/03/18 09:21:46 INFO ALS: innerTime 20.651 ms
15/03/18 09:21:46 INFO ALS: innerTime 19.721 ms
15/03/18 09:21:46 INFO ALS: innerTime 27.865 ms
15/03/18 09:21:46 INFO ALS: innerTime 21.655 ms
15/03/18 09:21:46 INFO ALS: innerTime 22.413 ms
15/03/18 09:21:46 INFO ALS: innerTime 19.793 ms
15/03/18 09:21:46 INFO ALS: innerTime 18.858 ms
15/03/18 09:21:46 INFO ALS: innerTime 24.089 ms
15/03/18 09:21:46 INFO ALS: innerTime 43.602 ms
15/03/18 09:21:47 INFO ALS: innerTime 44.16 ms
15/03/18 09:21:47 INFO ALS: innerTime 32.279 ms
15/03/18 09:21:47 INFO ALS: innerTime 53.446 ms
15/03/18 09:21:47 INFO ALS: innerTime 48.279 ms
15/03/18 09:21:47 INFO ALS: innerTime 44.157 ms
15/03/18 09:21:47 INFO ALS: innerTime 37.386 ms
15/03/18 09:21:47 INFO ALS: innerTime 56.11 ms
15/03/18 09:21:47 INFO ALS: innerTime 21.043 ms
15/03/18 09:21:47 INFO ALS: innerTime 25.038 ms
15/03/18 09:21:47 INFO ALS: innerTime 24.97 ms
15/03/18 09:21:47 INFO ALS: innerTime 27.335 ms
15/03/18 09:21:47 INFO ALS: innerTime 26.706 ms
15/03/18 09:21:47 INFO ALS: innerTime 27.418 ms
15/03/18 09:21:47 INFO ALS: innerTime 30.885 ms
15/03/18 09:21:47 INFO ALS: innerTime 19.971 ms
mllib nnls:
15/03/18 09:23:14 INFO ALS: solveTime 89.822 ms
15/03/18 09:23:14 INFO ALS: solveTime 107.005 ms
15/03/18 09:23:14 INFO ALS: solveTime 80.083 ms
15/03/18 09:23:14 INFO ALS: solveTime 106.405 ms
15/03/18 09:23:14 INFO ALS: solveTime 94.193 ms
15/03/18 09:23:14 INFO ALS: solveTime 102.188 ms
15/03/18 09:23:14 INFO ALS: solveTime 100.847 ms
15/03/18 09:23:14 INFO ALS: solveTime 75.717 ms
15/03/18 09:23:14 INFO ALS: solveTime 45.384 ms
15/03/18 09:23:14 INFO ALS: solveTime 45.79 ms
15/03/18 09:23:14 INFO ALS: solveTime 67.02 ms
15/03/18 09:23:14 INFO ALS: solveTime 51.034 ms
15/03/18 09:23:14 INFO ALS: solveTime 45.844 ms
15/03/18 09:23:15 INFO ALS: solveTime 46.944 ms
15/03/18 09:23:15 INFO ALS: solveTime 45.024 ms
15/03/18 09:23:15 INFO ALS: solveTime 55.672 ms
15/03/18 09:23:15 INFO ALS: solveTime 26.168 ms
15/03/18 09:23:15 INFO ALS: solveTime 27.104 ms
15/03/18 09:23:15 INFO ALS: solveTime 33.295 ms
15/03/18 09:23:15 INFO ALS: solveTime 28.376 ms
15/03/18 09:23:15 INFO ALS: solveTime 27.73 ms
15/03/18 09:23:15 INFO ALS: solveTime 27.867 ms
15/03/18 09:23:15 INFO ALS: solveTime 27.287 ms
15/03/18 09:23:15 INFO ALS: solveTime 24.709 ms
15/03/18 09:23:15 INFO ALS: solveTime 45.617 ms
15/03/18 09:23:15 INFO ALS: solveTime 47.411 ms
15/03/18 09:23:15 INFO ALS: solveTime 60.453 ms
15/03/18 09:23:15 INFO ALS: solveTime 50.855 ms
15/03/18 09:23:15 INFO ALS: solveTime 44.043 ms
15/03/18 09:23:15 INFO ALS: solveTime 51.125 ms
15/03/18 09:23:15 INFO ALS: solveTime 58.812 ms
15/03/18 09:23:15 INFO ALS: solveTime 49.631 ms
15/03/18 09:23:16 INFO ALS: solveTime 35.489 ms
15/03/18 09:23:16 INFO ALS: solveTime 31.978 ms
15/03/18 09:23:16 INFO ALS: solveTime 29.305 ms
15/03/18 09:23:16 INFO ALS: solveTime 28.568 ms
15/03/18 09:23:16 INFO ALS: solveTime 40.257 ms
15/03/18 09:23:16 INFO ALS: solveTime 24.955 ms
15/03/18 09:23:16 INFO ALS: solveTime 28.446 ms
15/03/18 09:23:16 INFO ALS: solveTime 35.841 ms
So at realistic ranks still there is an overhead from iterator but in
overall flow I don't know how much difference does this overhead causes...
@mengxr if you have pointers on how to optimize Breeze NNLS code further
please let me know...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
