Hi Amado, PredictionIO does not officially support Spark 2.0 yet, and I have not personally tried the latest version of MLlib, so I cannot comment on whether it has gotten better with the same number of iterations. My guess is that it would be similar if you don't specify a non-default JVM heap memory (default is half a GB).
If you are testing locally, you can use spark-submit bypass arguments (explained in "pio help"), e.g. pio train -- --driver-memory 2g Regards, Donald On Mon, Oct 31, 2016 at 1:58 PM, Amado Espinosa <[email protected]> wrote: > Hi, thanks you very much for your help. We recently increase our server > memory until 12GB but I still doing the predictionio setup. I realize that > your manual install reference recommends spark-1.5.1 and Hadoop 2.6 .. > there are newer versions of Spark and also a new 2.0 branch ... Can I > expect better results with this implementation?? I really appreciate any > comments about that. > > Do you have any reference or documentation to try Increase Spark driver > memory locally?? > > Best Regards, > > On Mon, Oct 31, 2016 at 12:51 PM, Donald Szeto <[email protected]> wrote: > >> Hi Amado, >> >> You have several options: >> >> 1. Train on a real Spark cluster with more executor memory. >> 2. Increase Spark driver memory if you want to test locally. >> 3. Reduce number of iterations in engine.json. >> >> Regards, >> Donald >> >> On Sun, Oct 30, 2016 at 7:42 AM, Amado Espinosa <[email protected] >> > wrote: >> >>> >>> Hi, I stuck trying to train the Recommendation Engine. The issue seems >>> to be related with >>> >>> [WARN] [BLAS] Failed to load implementation from: >>> com.github.fommil.netlib.NativeSystemBLAS >>> [WARN] [BLAS] Failed to load implementation from: >>> com.github.fommil.netlib.NativeRefBLAS >>> [WARN] [LAPACK] Failed to load implementation from: >>> com.github.fommil.netlib.NativeSystemLAPACK >>> [WARN] [LAPACK] Failed to load implementation from: >>> com.github.fommil.netlib.NativeRefLAPACK >>> >>> Warnings, I googled and I find some tips like >>> >>> To fix this, you need to download the source code, and build it yourself >>> passing the “-Pnetlib-lgpl” option to maven. >>> >>> I've did that using this command >>> >>> ./make-distribution.sh --name hadoop-2.6 --tgz -Phadoop-2.6 -Pyarn >>> -Pnetlib-lgpl -DskipTests >>> >>> but still I;m getting the same issue .. >>> I attached the log, do you have any guideline to avoid this issue? I'm >>> miss some steps during the installation/setup? >>> >>> Thanks for your help, >>> Best Regards. >>> >>> >> >
