Github user dlwh commented on the pull request:
https://github.com/apache/spark/pull/5005#issuecomment-83054484
That's big enough that it's worth fixing. Are there any confounds?
On Mar 18, 2015 9:27 AM, "Debasish Das" <[email protected]> wrote:
> I updated with the @mengxr <https://github.com/mengxr> checkin that
> cleaned jblas dependencies from NNLS....the results updated above is with
> rank = 10 I think...A more realistic rank is 50...
>
> breeze nnls (including iterator overhead):
>
> 15/03/18 09:21:45 INFO ALS: solveTime 177.048 ms
> 15/03/18 09:21:45 INFO ALS: solveTime 189.642 ms
> 15/03/18 09:21:45 INFO ALS: solveTime 171.894 ms
> 15/03/18 09:21:45 INFO ALS: solveTime 203.503 ms
> 15/03/18 09:21:45 INFO ALS: solveTime 185.352 ms
> 15/03/18 09:21:45 INFO ALS: solveTime 190.051 ms
> 15/03/18 09:21:45 INFO ALS: solveTime 181.634 ms
> 15/03/18 09:21:45 INFO ALS: solveTime 148.325 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 59.523 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 78.501 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 66.649 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 66.078 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 75.827 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 50.72 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 65.047 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 59.184 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 26.837 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 26.061 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 36.14 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 33.341 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 29.399 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 28.61 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 30.074 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 32.351 ms
> 15/03/18 09:21:46 INFO ALS: solveTime 56.625 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 54.29 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 49.305 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 63.742 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 67.626 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 55.348 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 54.473 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 66.704 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 29.57 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 33.871 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 33.329 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 40.322 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 35.376 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 35.826 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 38.973 ms
> 15/03/18 09:21:47 INFO ALS: solveTime 34.908 ms
>
> breeze nnls inner solve (subtracting iterator time):
>
> 15/03/18 09:21:45 INFO ALS: innerTime 103.235 ms
> 15/03/18 09:21:45 INFO ALS: innerTime 115.842 ms
> 15/03/18 09:21:45 INFO ALS: innerTime 104.233 ms
> 15/03/18 09:21:45 INFO ALS: innerTime 112.84 ms
> 15/03/18 09:21:45 INFO ALS: innerTime 97.474 ms
> 15/03/18 09:21:45 INFO ALS: innerTime 91.188 ms
> 15/03/18 09:21:45 INFO ALS: innerTime 115.043 ms
> 15/03/18 09:21:45 INFO ALS: innerTime 96.443 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 31.696 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 40.446 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 42.288 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 34.147 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 33.404 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 28.02 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 33.372 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 38.531 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 20.651 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 19.721 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 27.865 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 21.655 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 22.413 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 19.793 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 18.858 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 24.089 ms
> 15/03/18 09:21:46 INFO ALS: innerTime 43.602 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 44.16 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 32.279 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 53.446 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 48.279 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 44.157 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 37.386 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 56.11 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 21.043 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 25.038 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 24.97 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 27.335 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 26.706 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 27.418 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 30.885 ms
> 15/03/18 09:21:47 INFO ALS: innerTime 19.971 ms
>
> mllib nnls:
>
> 15/03/18 09:23:14 INFO ALS: solveTime 89.822 ms
> 15/03/18 09:23:14 INFO ALS: solveTime 107.005 ms
> 15/03/18 09:23:14 INFO ALS: solveTime 80.083 ms
> 15/03/18 09:23:14 INFO ALS: solveTime 106.405 ms
> 15/03/18 09:23:14 INFO ALS: solveTime 94.193 ms
> 15/03/18 09:23:14 INFO ALS: solveTime 102.188 ms
> 15/03/18 09:23:14 INFO ALS: solveTime 100.847 ms
> 15/03/18 09:23:14 INFO ALS: solveTime 75.717 ms
> 15/03/18 09:23:14 INFO ALS: solveTime 45.384 ms
> 15/03/18 09:23:14 INFO ALS: solveTime 45.79 ms
> 15/03/18 09:23:14 INFO ALS: solveTime 67.02 ms
> 15/03/18 09:23:14 INFO ALS: solveTime 51.034 ms
> 15/03/18 09:23:14 INFO ALS: solveTime 45.844 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 46.944 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 45.024 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 55.672 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 26.168 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 27.104 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 33.295 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 28.376 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 27.73 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 27.867 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 27.287 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 24.709 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 45.617 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 47.411 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 60.453 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 50.855 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 44.043 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 51.125 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 58.812 ms
> 15/03/18 09:23:15 INFO ALS: solveTime 49.631 ms
> 15/03/18 09:23:16 INFO ALS: solveTime 35.489 ms
> 15/03/18 09:23:16 INFO ALS: solveTime 31.978 ms
> 15/03/18 09:23:16 INFO ALS: solveTime 29.305 ms
> 15/03/18 09:23:16 INFO ALS: solveTime 28.568 ms
> 15/03/18 09:23:16 INFO ALS: solveTime 40.257 ms
> 15/03/18 09:23:16 INFO ALS: solveTime 24.955 ms
> 15/03/18 09:23:16 INFO ALS: solveTime 28.446 ms
> 15/03/18 09:23:16 INFO ALS: solveTime 35.841 ms
>
> So at realistic ranks still there is an overhead from iterator but in
> overall flow I don't know how much difference does this overhead causes...
>
> @mengxr <https://github.com/mengxr> if you have pointers on how to
> optimize Breeze NNLS code further please let me know...
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/5005#issuecomment-83045631>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
