kiszk commented on a change in pull request #29139: URL: https://github.com/apache/spark/pull/29139#discussion_r456874200
########## File path: docs/ml-linalg-guide.md ########## @@ -0,0 +1,85 @@ +# Spark MLlib Linear Algebra Acceleration Guide + +## Introduction + +This guide provides necessary information to enable accelerated linear algebra processing for Spark MLlib. + +Spark MLlib defines Vector and Matrix as basic data types for machine learning algorithms. On top of them, [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) and [LAPACK](https://en.wikipedia.org/wiki/LAPACK) operations are implemented and supported by [netlib-java](https://github.com/fommil/netlib-Java).[^1] `netlib-java` can use optimized native linear algebra libraries (refered to as "native libraries" or "BLAS libraries" hereafter) for faster numerical processing. [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) and [OpenBLAS](http://www.openblas.net) are two most popular ones. + +However due to license restrictions, the official released Spark binaries by default doesn't contain native libraries support for `netlib-java`. + +The following sections describe how to enable `netlib-java` with native libraries support for Spark MLlib and how to install native libraries and configure them properly. + +[^1]: The algorithms may call Breeze and it will in turn call `netlib-java`. + +## Enable `netlib-java` with native library proxies + +`netlib-java` native libraries has a dependency on `libgfortran`. It requires GFORTRAN 1.4 or above. This can be obtained by installing `libgfortran` package. After installation, the following command can be used to verify if it is installed properly. Review comment: nit: `libraries` -> `library` or `has` -> `have` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
