Github user dlwh commented on the pull request:
https://github.com/apache/spark/pull/5005#issuecomment-83739506
I mean, the liblbfgs code does callbacks, not iterators. They are duals
w.r.t. who has control over what gets executed when. (Push iterates vs pull
iterates) In general, it's easy to turn pull into push, but difficult to
turn push into pull, without threading anyway. That's why I prefer
iterators.
It might be that intermediate states aren't worth it here, because the
optimizer is so fast you only ever want the answer, and not progress. The
reason I do it for the minimizers is that sometimes the objective takes a
long time to optimize, and I'd like to checkpoint, or something.
-- David
On Thu, Mar 19, 2015 at 12:08 PM, Debasish Das <[email protected]>
wrote:
> @mengxr <https://github.com/mengxr> it's interesting that the iterator
> overhead is high only at initial iterations and as JVM gets hotter the
> runtime gets at par with the while loop...The iterator pattern for
> optimizers (David told me about it before) comes from
> http://www.chokkan.org/software/liblbfgs/ and I feel is very useful...If
> there are scala specific optimization to make it faster we should think on
> it...
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/5005#issuecomment-83720384>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
