I think that the proposal under discussion involves adding a dependency on
a maven released h2o artifact plus a contribution of Mahout translation
layers. These layers would give a sub-class of Matrix (and Vector) which
allow direct control over life span across multiple jobs but would
otherwise behave like their in-memory counter-parts.
Well I suppose that means they have to live in some processes which are not
processes I already have. And they have to be managed. So this is not just
an in-core subsystem. Sounds like a new back to me.
In Hadoop, every iteration must be scheduled as a separate job, rereads
invariant data and materializes its result to hdfs. Therefore, iterative
programs on Hadoop are an order of magnitude slower than on systems that
have dedicated support for iterations.
Does h2o help here or would we need to incorporate another system for
such
tasks?
H2o helps here in a couple of different ways.
The first and foremost is that primitive operations are easy
Additionally, data elements can survive a single programs execution. This
means that programs can be executed one after another to get composite
effects. This is astonishingly fast ... more along the speeds one would
expect from a single processor program.
I think the problem here is that the authors keep comparing these
techniques to slowest model available which is Hadoop.
But this is exact execution model of Spark. You get stuff repeatedly
executed on in-memory partitions and get approximately the speed of
iterative speed execution. I won't describe it as astonishing, though,
because indeed it is as fast as you can get things done in memory, no
faster, no slower. That's for example the reason why my linalg optimizer is
not hesitating to compute exact matrix geometry lazily if not known, for
optimization purposes, because the answer will be back in between 40 to 200
ms, assuming adequate RAM allocation. I have been using these paradigms for
more than a year now. This is all good stuff. I would not use word
astonshing, but sensible, yes. Main concern is if programming model is
called to be sacrificed just to do sensible things here.
(2) Efficient join implementations
If we look at a lot of Mahout's algorithm implementations with a
database
hat on, than we see lots of handcoded joins in our codebase, because
Hadoop
does not bring join primitives. This has lots of drawbacks, e.g. it
complicates the codebase and leads to hardcoded join strategies that
bake
certain assumptions into the code (e.g. ALS uses a broadcast-join which
assumes that one side fits into memory on each machine, RecommenderJob
uses
a repartition-join which is scalable but very slow for small
inputs,...).
+1
I think that h2o provides this but do not know in detail how. I do know
that many of the algorithms already coded make use of matrix
multiplication
which is essentially a join operation.
Essentially a join? The spark module optimizer picks out of at least 3
implementations: zip+combine, block-wise cartesian and finally, yes,
join+combine. Depends on orientation and the earlier operators in pipeline.
That's exactly my point about flexibility of programming model from the
optimizer point of view.
Obviously, I'd love to get rid of handcoded joins and implement ML
algorithms (which is hard enough on its own). Other systems help with
this
already. Spark, for example offers broadcast and repartition-join
primitives, Stratosphere has a join primitive and an optimizer that
automatically decides which join strategy to use, as well as a highly
optimized hybrid hashjoin implementation that can gracefully go
out-of-core
under memory pressure.
When you get into the realm of things on this level of sophistication, I
think that you have found the boundary where alternative foundations like
Spark and Stratosphere are better than h2o. The novelty with h2o is the
hypothesis that a very large fraction of interesting ML algorithms can be
implemented without this power. So far, this seems correct.
Again, this is largely along the lines "let's make a library of few
hand-optimized things". Which is noble, but -- I would argue -- not
ambitious enough. Most of the distributed ML projects do just that. We
should perhaps think along the lines what could be differentiating factor
for us.
Not that we should not care about performance. It should be, of course,
*sensible*. (Our MR code base of course does not give us that, as u said,
jumping off MR wagon is not even a question).
If you can forgive me for drawing parallels here, it's a difference between
something like Weka and R. Collection vs. platform _and_ collection induced
by platform. Platform of course also positively feeds into the speed of
collection growth directly.
When i use R, i don't have code consisting of algorithms calls. That is,
yes, it is doing off-the shelf use now and then, but it is far from being
the only thing it is doing. 95% of the things is as simple feature
massaging. I place no value in R for providing GLM for me. Gosh, this
particular offering is virtually hanging from anywhere these days.
But i do place value into it for doing custom feature prep and for, for
example being able to get 100 grad students to try their own k-means
implementation in seconds.
Why?
There has been a lot of talk here about building community and
contributions etc. Platform is what builds it, most directly and amazingly.
I would go on a limb here and say that Spark and mlib are experiencing
explosive growth of contributions not because it can do things with
in-memory datasets (which is important, but like i said, is has been long
since viewed no more than just sensible), but because of clarity of its
programming model. I think we have seen a very solid evidence that clarity
and richness of programming model was the thing that attracts communities.
If we grade roughly (very roughly!) what we have today, I can easily argue
that the acceptance levels follow the programming model very closely. e.g.
if i try to sort project with distributed programming models by (my
subjectively percieved) popularity, from bottom to top :
********
Hadoop MapReduce -- ok i don't even know how to organize the critique here,
too long of a list, almost nobody (but Mahout) does these things this way
today. Certainly, none of my last 2 employers did.
hive -- SQL like with severly constrained general programming language
capabilities, not conducive to batches. Pretty much limits to ad-hoc
exploration.
Pig -- a bit better, can write batches, but extra functionality mixins
(UDFs) are still a royal pain
Cascading -- even easier, rich primitives, easy batches, some manual
optimization of physical plan elements. One of the big cons is the
limitation of a rigid dataset tuple structure,
FlumeJava (Crunch in apache world) -- even better, but java closures are
just plain ugly, zero "scriptability". Its community has been hurt a little
bit because of the fact that it was a bit late to the show compared to
others (e.g. cascading), but it leveled off quickly.
Scala bindings for Cascading (Scalding) and FlumeJava -- better, hell, well
better on the closure and FP front! But still not being native to scala
from get go creates some miniature problems there.
Spark -- i think is fair to say the current community "king" above those
all -- all the aforementioned platform model pains are eliminated, although
on performance side i think there're still some pockets for improvement on
cost-based optimization side of things.
Stratosphere might be more interesting in this department, but I am not
sure at this point if that necessarily will translate into performance
benefits for ML.
********
The first few things are using the same computing model underneath and
essentially are having roughly the same performance. Yet there's clear
variation in community and acceptance.
In ML world, we are seeing approximately the same thing. The clearer the
programming model and ease of integration in to the process, the wider the
acceptance. I probably can pretty successfully argue that current most
performant ML "thing" as it stands is GraphLab. And it is pretty
comprehensive in problem coverage (I think it does cover e.g. recommender
concerns greater than h2o and Mahout together, for example). But i can also
pretty successfully argue it is being rejected a lot of time for being just
a collection (which is, in addition, is hard to call from jvm, i.e.
integration again). It is actually so bad, that people in my company would
rather go back to 20 snow wired R servers than think of even entertaining
an architecture including GraphLab component. (Yes, variance of this sample
as high as it gets, just saying what i hear).
So as a general guideline to solve the current ills, it would stand to
reason to adopt platform priority and algorithm collection as a function of
such platform, rather than collection as a function of few dedicated
efforts. Yes -- it has to be *sensibly* performant -- but this does not
have to be mostly a concern of the code in this project directly. Rather,
it has to be a concern of the backs (i.e. dependencies) and our in-core
support.
Our pathological fear of being a performance scapegoat totally obscurs the
fact that performance is mostly a function of the back and that we were
riding on a wrong back for a long time. As long as we don't cling to a
particular back, it shouldn't be a problem. What one would rather accept:
being initially 5x slower than Graphlab (but on par with MLlib) but beat
these on community support, or being on par but anemic in community? If 02
platform feels the performance has been so important to sacrifice
programming model, why they feel the need to join an apache project? After
all, they have been an open project for a long time already and have built
their own community, big or small. Spark has just now become a top-level
apache project, and joined apache incubator mere 2 months ago and did not
have any trouble attracting community outside Apache at all. Stratosphere
is not even in Apache. Similarly, did it help Mahout to be in Apache to get
anywhere close in community measurement to these? So this totally refutes
the argument one has to be an Apache project to get its exclusive qualities
highlighted. Perhaps in the end it is more about the importance of the
qualities to the community and quality of contributions.
A lot of this platform and programming model priority is probably easier to
say than do, but some of linalg and data frame things are ridiculously easy
though in terms of amount of effort. If i could do linalg optmizer with
bindings for sparks with 2 nights a month, the same can be done for
multiple backs and data frames in a jiffy. Well, the back should have a
clear programming model of course as a prerequisite. Which brings us back
to the issue of richness of distributed primitives.
