Re: netlib-java cannot load native lib in Windows when using spark-submit
I did not build my own Spark. I got the binary version online. If it can load the native libs from IDE, it should also be able to load native when running with "--matter local". On Mon, 23 Mar 2015 07:15 Burak Yavuz wrote: > Did you build Spark with: -Pnetlib-lgpl? > > Ref: https://spark.apache.org/docs/latest/mllib-guide.html > > Burak > > On Sun, Mar 22, 2015 at 7:37 AM, Ted Yu wrote: > >> How about pointing LD_LIBRARY_PATH to native lib folder ? >> >> You need Spark 1.2.0 or higher for the above to work. See SPARK-1719 >> >> Cheers >> >> On Sun, Mar 22, 2015 at 4:02 AM, Xi Shen wrote: >> >>> Hi Ted, >>> >>> I have tried to invoke the command from both cygwin environment and >>> powershell environment. I still get the messages: >>> >>> 15/03/22 21:56:00 WARN netlib.BLAS: Failed to load implementation from: >>> com.github.fommil.netlib.NativeSystemBLAS >>> 15/03/22 21:56:00 WARN netlib.BLAS: Failed to load implementation from: >>> com.github.fommil.netlib.NativeRefBLAS >>> >>> From the Spark UI, I can see: >>> >>> spark.driver.extraLibrary c:\openblas >>> >>> >>> Thanks, >>> David >>> >>> >>> On Sun, Mar 22, 2015 at 11:45 AM Ted Yu wrote: >>> Can you try the --driver-library-path option ? spark-submit --driver-library-path /opt/hadoop/lib/native ... Cheers On Sat, Mar 21, 2015 at 4:58 PM, Xi Shen wrote: > Hi, > > I use the *OpenBLAS* DLL, and have configured my application to work > in IDE. When I start my Spark application from IntelliJ IDE, I can see in > the log that the native lib is loaded successfully. > > But if I use *spark-submit* to start my application, the native lib > still cannot be load. I saw the WARN message that it failed to load both > the native and native-ref library. I checked the *Environment* tab in > the Spark UI, and the *java.library.path* is set correctly. > > > Thanks, > > David > > > >> >
Re: netlib-java cannot load native lib in Windows when using spark-submit
Did you build Spark with: -Pnetlib-lgpl? Ref: https://spark.apache.org/docs/latest/mllib-guide.html Burak On Sun, Mar 22, 2015 at 7:37 AM, Ted Yu wrote: > How about pointing LD_LIBRARY_PATH to native lib folder ? > > You need Spark 1.2.0 or higher for the above to work. See SPARK-1719 > > Cheers > > On Sun, Mar 22, 2015 at 4:02 AM, Xi Shen wrote: > >> Hi Ted, >> >> I have tried to invoke the command from both cygwin environment and >> powershell environment. I still get the messages: >> >> 15/03/22 21:56:00 WARN netlib.BLAS: Failed to load implementation from: >> com.github.fommil.netlib.NativeSystemBLAS >> 15/03/22 21:56:00 WARN netlib.BLAS: Failed to load implementation from: >> com.github.fommil.netlib.NativeRefBLAS >> >> From the Spark UI, I can see: >> >> spark.driver.extraLibrary c:\openblas >> >> >> Thanks, >> David >> >> >> On Sun, Mar 22, 2015 at 11:45 AM Ted Yu wrote: >> >>> Can you try the --driver-library-path option ? >>> >>> spark-submit --driver-library-path /opt/hadoop/lib/native ... >>> >>> Cheers >>> >>> On Sat, Mar 21, 2015 at 4:58 PM, Xi Shen wrote: >>> Hi, I use the *OpenBLAS* DLL, and have configured my application to work in IDE. When I start my Spark application from IntelliJ IDE, I can see in the log that the native lib is loaded successfully. But if I use *spark-submit* to start my application, the native lib still cannot be load. I saw the WARN message that it failed to load both the native and native-ref library. I checked the *Environment* tab in the Spark UI, and the *java.library.path* is set correctly. Thanks, David >>> >
Re: netlib-java cannot load native lib in Windows when using spark-submit
How about pointing LD_LIBRARY_PATH to native lib folder ? You need Spark 1.2.0 or higher for the above to work. See SPARK-1719 Cheers On Sun, Mar 22, 2015 at 4:02 AM, Xi Shen wrote: > Hi Ted, > > I have tried to invoke the command from both cygwin environment and > powershell environment. I still get the messages: > > 15/03/22 21:56:00 WARN netlib.BLAS: Failed to load implementation from: > com.github.fommil.netlib.NativeSystemBLAS > 15/03/22 21:56:00 WARN netlib.BLAS: Failed to load implementation from: > com.github.fommil.netlib.NativeRefBLAS > > From the Spark UI, I can see: > > spark.driver.extraLibrary c:\openblas > > > Thanks, > David > > > On Sun, Mar 22, 2015 at 11:45 AM Ted Yu wrote: > >> Can you try the --driver-library-path option ? >> >> spark-submit --driver-library-path /opt/hadoop/lib/native ... >> >> Cheers >> >> On Sat, Mar 21, 2015 at 4:58 PM, Xi Shen wrote: >> >>> Hi, >>> >>> I use the *OpenBLAS* DLL, and have configured my application to work in >>> IDE. When I start my Spark application from IntelliJ IDE, I can see in the >>> log that the native lib is loaded successfully. >>> >>> But if I use *spark-submit* to start my application, the native lib >>> still cannot be load. I saw the WARN message that it failed to load both >>> the native and native-ref library. I checked the *Environment* tab in >>> the Spark UI, and the *java.library.path* is set correctly. >>> >>> >>> Thanks, >>> >>> David >>> >>> >>> >>
Re: netlib-java cannot load native lib in Windows when using spark-submit
Hi Ted, I have tried to invoke the command from both cygwin environment and powershell environment. I still get the messages: 15/03/22 21:56:00 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS 15/03/22 21:56:00 WARN netlib.BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS >From the Spark UI, I can see: spark.driver.extraLibrary c:\openblas Thanks, David On Sun, Mar 22, 2015 at 11:45 AM Ted Yu wrote: > Can you try the --driver-library-path option ? > > spark-submit --driver-library-path /opt/hadoop/lib/native ... > > Cheers > > On Sat, Mar 21, 2015 at 4:58 PM, Xi Shen wrote: > >> Hi, >> >> I use the *OpenBLAS* DLL, and have configured my application to work in >> IDE. When I start my Spark application from IntelliJ IDE, I can see in the >> log that the native lib is loaded successfully. >> >> But if I use *spark-submit* to start my application, the native lib >> still cannot be load. I saw the WARN message that it failed to load both >> the native and native-ref library. I checked the *Environment* tab in >> the Spark UI, and the *java.library.path* is set correctly. >> >> >> Thanks, >> >> David >> >> >> >
Re: netlib-java cannot load native lib in Windows when using spark-submit
Can you try the --driver-library-path option ? spark-submit --driver-library-path /opt/hadoop/lib/native ... Cheers On Sat, Mar 21, 2015 at 4:58 PM, Xi Shen wrote: > Hi, > > I use the *OpenBLAS* DLL, and have configured my application to work in > IDE. When I start my Spark application from IntelliJ IDE, I can see in the > log that the native lib is loaded successfully. > > But if I use *spark-submit* to start my application, the native lib still > cannot be load. I saw the WARN message that it failed to load both the > native and native-ref library. I checked the *Environment* tab in the > Spark UI, and the *java.library.path* is set correctly. > > > Thanks, > > David > > >
netlib-java cannot load native lib in Windows when using spark-submit
Hi, I use the *OpenBLAS* DLL, and have configured my application to work in IDE. When I start my Spark application from IntelliJ IDE, I can see in the log that the native lib is loaded successfully. But if I use *spark-submit* to start my application, the native lib still cannot be load. I saw the WARN message that it failed to load both the native and native-ref library. I checked the *Environment* tab in the Spark UI, and the *java.library.path* is set correctly. Thanks, David
