[
https://issues.apache.org/jira/browse/SYSTEMML-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797676#comment-16797676
]
Niketan Pansare commented on SYSTEMML-2514:
-------------------------------------------
I usually use SystemML with Spark. To verify whether Spark is correctly
installed, please follow the below steps:
1. Java
=======
The Java version should be > 1.8.
> java -version
Set JAVA_HOME environment variable and include %JAVA_HOME%\bin in the
environment variable PATH
> ls "%JAVA_HOME%"
2. Spark
========
Download and extract Spark from https://spark.apache.org/downloads.html,
> tar -xzf spark-2.1.0-bin-hadoop2.7.tgz
and set environment variable SPARK_HOME to point to the extracted directory.
Next step, install winutils:
- Download winutils.exe from
http://github.com/steveloughran/winutils/raw/master/hadoop-2.6.0/bin/winutils.exe
- Place it in c:\winutils\bin
- Set environment variable HADOOP_HOME to point to c:\winutils
- Add c:\winutils\bin to the environment variable PATH.
- Finally, modify permission of hive directory that will be used by spark
> winutils.exe chmod 777 /tmp/hive
Finally, check if Spark is correctly installed:
> %SPARK_HOME%\bin\spark-shell
> %SPARK_HOME%\bin\pyspark
> Hadoop gives java.lang.UnsatisfiedLinkError
> -------------------------------------------
>
> Key: SYSTEMML-2514
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2514
> Project: SystemML
> Issue Type: Sub-task
> Components: Build
> Reporter: Janardhan
> Priority: Minor
> Attachments: hadoop_on_win.pdf
>
>
>
> {code:java}
> java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;
> at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native
> Method)
> at
> org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:559)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:219)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
> at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
> at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
> at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
> at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:910)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:788)
> at org.apache.sysml.test.utils.TestUtils.writeTestMatrix(TestUtils.java:1458)
> at org.apache.sysml.test.utils.TestUtils.writeTestMatrix(TestUtils.java:1514)
> at
> org.apache.sysml.test.integration.AutomatedTestBase.writeInputMatrix(AutomatedTestBase.java:530)
> at
> org.apache.sysml.test.integration.AutomatedTestBase.writeInputMatrix(AutomatedTestBase.java:587)
> at
> org.apache.sysml.test.integration.functions.blocks.VariableTest.testVariablePassing1(VariableTest.java:80)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
