On Sun, Apr 27, 2014 at 8:07 PM, Dmitriy Lyubimov <[email protected]> wrote:
> > > > On Sun, Apr 27, 2014 at 7:57 PM, Anand Avati <[email protected]> wrote: > >> Hi Ted, Dmitry, >> Background: I am exploring the feasibility of providing H2O distributed >> "backend" to the DSL. >> > > Very cool. that's actually was one of my initial proposals on how to > approach this. Got pushed back on this though. > We are exploring various means of integration. The Jira mentioned providing Matrix and Vector implementations as an initial exploration. That task by itself had a lot of value in terms of reconciling some ground level issues (build/mvn compatibility, highlighting some classloader related challenges etc. on the H2O side.) Plugging behind a common DSL makes sense, though there may be value in other points of integration too, to exploit H2O's strengths. > > >> At a high level it appears that implementing physical operators for >> DrmLike over H2O does not seem extremely challenging. All the operators in >> the DSL seem to have at least an approximate equivalent in H2O's own >> (R-like) DSL, and wiring one operator with another's implementation seems >> like a tractable problem. >> > > It should be tractable, sure, even for map reduce. The question is whether > there's enough diversity to do certain optimizations in a certain way. E.g. > if two matrices are identically partitioned, then do map-side zip instead > of actual parallel join etc. > > But it should be tractable, indeed. > Yes, H2O has ways to do such things - a single map/reduce task on two matrices "side by side" which are similarly partitioned (i.e, sharing the same VectorGroup in H2O terminology) > The reason I write, is to better understand the split between the Mahout >> DSL and Spark (both current and future). As of today, the DSL seems to be >> pretty tightly coupled with Spark. >> >> E.g: >> >> - DSSVD.scala imports o.a.spark.storage.StorageLevel >> > > This is a known thing, I think i noted it somewhere in jira. That, and rdd > property of CheckpointedDRM. This needs to be abstracted away. > > >> - drm.plan.CheckpointAction: the result of exec() and checkpoint() is >> DrmRddInput (instead of, say, DrmLike) >> > > CheckpointAction is part of physical layer. This is something that would > have to be completely re-written for a new engine. This is the "plugin" > api, but it is never user-facing (logical plan facing). > It somehow felt that the optimizer was logical-ish. Do you mean the optimizations in CheckpointAction are specific to Spark and cannot be inherited in general to other backends (not that I think that is wrong)? > >> Firstly, I don't think I am presenting some new revelation you guys don't >> already know - I'm sure you know that the logical vs physical "split" in >> the DSL is not absolute (yet). >> > > Aha. Exactly > > >> >> That being said, I would like to understand if there are plans, or >> efforts already underway to make the DSL (i.e how DSSVD would be written) >> and the logical layer (i.e drm.plan.* optimizer etc) more "pure" and move >> the Spark specific code entirely into the physical domain. I recall Dmitry >> mentioning that a new engine other than Spark was also being planned, >> therefore I deduce some thought for such "purification" has already been >> applied. >> > > Aha. The hope is for Stratosphere. But there are few items that need to be > done by Stratosphere folks before we can leverage it fully. Or, let's say, > leverage it much better than we otherwise could. Makes sense to wait a bit. > > >> >> It would be nice to see changes approximately like: >> >> Rename ./spark => ./dsl >> Rename ./spark/src/main/scala/org/apache/mahout/sparkbindings => >> ./dsl/src/main/scala/org/apache/mahout/dsl >> Rename ./spark/src/main/scala/org/apache/mahout/sparkbindings/blas => >> ./dsl/main/scala/org/apache/mahout/dsl/spark-backend >> > > i was thinking along the lines factoring out public traits and logical > operators (DRMLike etc.) out of spark module into independent module > without particular engine dependencies. Exactly. It just hasn't come to > that yet. > > >> along with appropriately renaming packages and imports, and confining >> references to RDD and SparkContext completely within spark-backend. >> >> I think such a clean split would be necessary to introduce more backend >> engines. If no efforts are already underway, I would be glad to take on the >> DSL "purification" task. >> > > i think you got very close to my thinking about further steps here. Like i > said, i was just idling in wait for something like Stratosphere to become > closer to our orbit. > OK, I think there is reasonable alignment on the goal. But you were not clear on whether you are going to be doing the purification split in the near future? or is that still an "unassigned task" which I can pick up? Avati
