in my humble opinion Spark should've supported linalg a-la [1] before it even started dumping methodologies into mllib.
[1] http://mahout.apache.org/users/sparkbindings/home.html On Wed, Jul 2, 2014 at 2:16 PM, Thunder Stumpges <thunder.stump...@gmail.com > wrote: > Thanks. I always hate having to do stuff like this. It seems like they > went a bit overboard with all the "private[mllib]" declarations... possibly > all in the name of "thou shalt not change your public API". If you don't > make your public API usable, we end up having to work around it anyway... > > Oh well. > > Thunder > > > > On Wed, Jul 2, 2014 at 2:05 PM, Koert Kuipers <ko...@tresata.com> wrote: > >> i did the second option: re-implemented .toBreeze as .breeze using pimp >> classes >> >> >> On Wed, Jul 2, 2014 at 5:00 PM, Thunder Stumpges < >> thunder.stump...@gmail.com> wrote: >> >>> I am upgrading from Spark 0.9.0 to 1.0 and I had a pretty good amount of >>> code working with internals of MLLib. One of the big changes was the move >>> from the old jblas.Matrix to the Vector/Matrix classes included in MLLib. >>> >>> However I don't see how we're supposed to use them for ANYTHING other >>> than a container for passing data to the included APIs... how do we do any >>> math on them? Looking at the internal code, there are quite a number of >>> private[mllib] declarations including access to the Breeze representations >>> of the classes. >>> >>> Was there a good reason this was not exposed? I could see maybe not >>> wanting to expose the 'toBreeze' function which would tie it to the breeze >>> implementation, however it would be nice to have the various mathematics >>> wrapped at least. >>> >>> Right now I see no way to code any vector/matrix math without moving my >>> code namespaces into org.apache.spark.mllib or duplicating the code in >>> 'toBreeze' in my own util functions. Not very appealing. >>> >>> What are others doing? >>> thanks, >>> Thunder >>> >>> >> >
