libgfortran.x86_64 4.1.2-52.el5_8.1 comes with libgfortran.so.1 but not libgfortran.so.3. JBLAS requires the latter. If you have root access, you can try to install a newer version of libgfortran. Otherwise, maybe you can try Spark 1.3, which doesn't use JBLAS in ALS. -Xiangrui
On Tue, Mar 3, 2015 at 11:21 AM, Prashant Sharma <[email protected]> wrote: > Hi Folks, > > We are trying to run the following code from the spark shell in a CDH 5.3 > cluster running on RHEL 5.8. > > spark-shell --master yarn --deploy-mode client --num-executors 15 > --executor-cores 6 --executor-memory 12G > import org.apache.spark.mllib.recommendation.ALS > import org.apache.spark.mllib.recommendation.Rating > val users_item_score_clean = > sc.textFile("/tmp/spark_mllib_test").map(_.split(",")) > val ratings = users_item_score_clean.map(x=> Rating(x(0).toInt, x(1).toInt, > x(2).toDouble)) > val rank = 10 > val numIterations = 20 > val alpha = 1.0 > val lambda = 0.01 > val model = ALS.trainImplicit(ratings, rank, numIterations, lambda,alpha) > > > > We are getting the following error (detailed error is attached in > error.log): > > > -- org.jblas ERROR Couldn't load copied link file: > java.lang.UnsatisfiedLinkError: > /u08/hadoop/yarn/nm/usercache/sharma.p/appcache/application_1425015707226_0128/ > container_e12_1425015707226_0128_01_000010/tmp/jblas7605371780736016929libjblas_arch_flavor.so: > libgfortran.so.3: > cannot open shared object file: No such file or directory. > > On Linux 64bit, you need additional support libraries. > You need to install libgfortran3. > > For example for debian or Ubuntu, type "sudo apt-get install libgfortran3" > > For more information, see > https://github.com/mikiobraun/jblas/wiki/Missing-Libraries > 15/03/02 14:50:25 ERROR executor.Executor: Exception in task 22.0 in stage > 6.0 (TID 374) > java.lang.UnsatisfiedLinkError: org.jblas.NativeBlas.dposv(CII[DII[DII)I > at org.jblas.NativeBlas.dposv(Native Method) > at org.jblas.SimpleBlas.posv(SimpleBlas.java:369) > > > This exact code runs fine on another CDH 5.3 cluster which runs on RHEL 6.5. > > "libgfortran.so.3" is not present on the problematic cluster. > > [root@node04 ~]# find / -name libgfortran.so.3 2>/dev/null > > > I am able to find "libgfortran.so.3 " on the cluster where the above code > works: > > [root@workingclusternode04 ~]# find / -name libgfortran.so.3 2>/dev/null > /usr/lib64/libgfortran.so.3 > > > > The following output shows that the fortran packages are installed on both > the clusters: > > On the cluster where this is not working > > [root@node04 ~]# yum list | grep -i fortran > gcc-gfortran.x86_64 4.1.2-52.el5_8.1 installed > libgfortran.i386 4.1.2-52.el5_8.1 installed > libgfortran.x86_64 4.1.2-52.el5_8.1 installed > > > On the cluster where the spark job is this working > > [root@ workingclusternode04 ~]# yum list | grep -i fortran > Repository 'bda' is missing name in configuration, using id > compat-libgfortran-41.x86_64 4.1.2-39.el6 @bda > gcc-gfortran.x86_64 4.4.7-4.el6 @bda > libgfortran.x86_64 4.4.7-4.el6 @bda > > > Has anybody run into this? Any pointers are much appreciated. > > Regards, > Prashant > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
