Re: Export BLAS module on Spark MLlib
I used reflection initially, but I found it's very slow especially in a tight loop. Maybe caching the reflection can help which I never try. Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 0xAF08DF8D On Mon, Nov 30, 2015 at 2:15 PM, Burak Yavuz wrote: > Or you could also use reflection like in this Spark Package: > https://github.com/brkyvz/lazy-linalg/blob/master/src/main/scala/com/brkyvz/spark/linalg/BLASUtils.scala > > Best, > Burak > > On Mon, Nov 30, 2015 at 12:48 PM, DB Tsai wrote: >> >> The workaround is have your code in the same package, or write some >> utility wrapper in the same package so you can use them in your code. >> Mostly we implement those BLAS for our own need, and we don't have >> general use-case in mind. As a result, if we open them up prematurely, >> it will add our api maintenance cost. Once it's getting mature, and >> people are asking for them, we will gradually make them public. >> >> Thanks. >> >> Sincerely, >> >> DB Tsai >> -- >> Web: https://www.dbtsai.com >> PGP Key ID: 0xAF08DF8D >> >> >> On Sat, Nov 28, 2015 at 5:20 AM, Sasaki Kai wrote: >> > Hello >> > >> > I'm developing a Spark package that manipulates Vector and Matrix for >> > machine learning. >> > This package uses mllib.linalg.Vector and mllib.linalg.Matrix in order >> > to >> > achieve compatible interface to mllib itself. But mllib.linalg.BLAS >> > module >> > is private inside spark package. We cannot use BLAS from spark package. >> > Due to this, there is no way to manipulate mllib.linalg.{Vector, Matrix} >> > from spark package side. >> > >> > Is there any reason why BLAS module is not set public? >> > If we cannot use BLAS, what is the reasonable option to manipulate >> > Vector >> > and Matrix from spark package? >> > >> > Regards >> > Kai Sasaki(@Lewuathe) >> > >> > >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Export BLAS module on Spark MLlib
Or you could also use reflection like in this Spark Package: https://github.com/brkyvz/lazy-linalg/blob/master/src/main/scala/com/brkyvz/spark/linalg/BLASUtils.scala Best, Burak On Mon, Nov 30, 2015 at 12:48 PM, DB Tsai wrote: > The workaround is have your code in the same package, or write some > utility wrapper in the same package so you can use them in your code. > Mostly we implement those BLAS for our own need, and we don't have > general use-case in mind. As a result, if we open them up prematurely, > it will add our api maintenance cost. Once it's getting mature, and > people are asking for them, we will gradually make them public. > > Thanks. > > Sincerely, > > DB Tsai > -- > Web: https://www.dbtsai.com > PGP Key ID: 0xAF08DF8D > > > On Sat, Nov 28, 2015 at 5:20 AM, Sasaki Kai wrote: > > Hello > > > > I'm developing a Spark package that manipulates Vector and Matrix for > > machine learning. > > This package uses mllib.linalg.Vector and mllib.linalg.Matrix in order to > > achieve compatible interface to mllib itself. But mllib.linalg.BLAS > module > > is private inside spark package. We cannot use BLAS from spark package. > > Due to this, there is no way to manipulate mllib.linalg.{Vector, Matrix} > > from spark package side. > > > > Is there any reason why BLAS module is not set public? > > If we cannot use BLAS, what is the reasonable option to manipulate Vector > > and Matrix from spark package? > > > > Regards > > Kai Sasaki(@Lewuathe) > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
Re: Export BLAS module on Spark MLlib
The workaround is have your code in the same package, or write some utility wrapper in the same package so you can use them in your code. Mostly we implement those BLAS for our own need, and we don't have general use-case in mind. As a result, if we open them up prematurely, it will add our api maintenance cost. Once it's getting mature, and people are asking for them, we will gradually make them public. Thanks. Sincerely, DB Tsai -- Web: https://www.dbtsai.com PGP Key ID: 0xAF08DF8D On Sat, Nov 28, 2015 at 5:20 AM, Sasaki Kai wrote: > Hello > > I'm developing a Spark package that manipulates Vector and Matrix for > machine learning. > This package uses mllib.linalg.Vector and mllib.linalg.Matrix in order to > achieve compatible interface to mllib itself. But mllib.linalg.BLAS module > is private inside spark package. We cannot use BLAS from spark package. > Due to this, there is no way to manipulate mllib.linalg.{Vector, Matrix} > from spark package side. > > Is there any reason why BLAS module is not set public? > If we cannot use BLAS, what is the reasonable option to manipulate Vector > and Matrix from spark package? > > Regards > Kai Sasaki(@Lewuathe) > > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Export BLAS module on Spark MLlib
Hello I'm developing a Spark package that manipulates Vector and Matrix for machine learning. This package uses mllib.linalg.Vector and mllib.linalg.Matrix in order to achieve compatible interface to mllib itself. But mllib.linalg.BLAS module is private inside spark package. We cannot use BLAS from spark package. Due to this, there is no way to manipulate mllib.linalg.{Vector, Matrix} from spark package side. Is there any reason why BLAS module is not set public? If we cannot use BLAS, what is the reasonable option to manipulate Vector and Matrix from spark package? Regards Kai Sasaki(@Lewuathe)