This if fine but why is Mahout integrating all of the engines, isn’t that the job of the engine people. Anand and Dmitriy have already said they are not interested in anything but the DSL so give it to any project that wants it and agree to support it with them. Be welcoming!
Committing this will affect every committer that plans to work with the DSL because it’s incomplete and changes to it will often break the build on two places. Also everything done outside of the Sore DSL (much of it is still engine dependent) will be difficult or impossible to make engine independent without a fair amount of work. To me this all argues for treating the core math as an engine independent module that we provide to other projects. This keeps the engine dependent work in the engine projects. Since this is a very special commit I propose a vote. This is one where a single committer should not have the power to put such a burden on all development. On Jul 11, 2014, at 9:12 PM, Ted Dunning <[email protected]> wrote: On Fri, Jul 11, 2014 at 2:00 PM, Dmitriy Lyubimov <[email protected]> wrote: > And most importantly, as i always was saying, the main benefit to me is not > that there's a CF algorithm in Mahout, but that i can write a custom one of > my own with less effort than just writing directly to any given engine api. > That part is very conceptual (drm api-dependent) and would be common > regardless of my deployment infrastructure. In other words, everyone can > just write their own co-occurrence analysis version hopefully easier than > just writing it directly for Spark or directly to H20 if he/she wanted to. > that's the real story (at least in my talk). > I agree with this 110%. In fact, I would argue that if all the engines worked just the same that there would be absolutely no reason to have more than one. It is precisely the differences that make having more than one important. The goal of the DSL is to minimize the gratuitous aspects of those differences so that you can relatively easily make use of the virtues of one engine or another.
