You don't actually need it per se - its just that some of the Spark libraries are referencing Hadoop libraries even if they ultimately do not call them. When I was doing some early builds of Spark on Windows, I admittedly had Hadoop on Windows running as well and had not run into this particular issue.
On Wed, Jul 2, 2014 at 12:04 PM, Kostiantyn Kudriavtsev < kudryavtsev.konstan...@gmail.com> wrote: > No, I don’t > > why do I need to have HDP installed? I don’t use Hadoop at all and I’d > like to read data from local filesystem > > On Jul 2, 2014, at 9:10 PM, Denny Lee <denny.g....@gmail.com> wrote: > > By any chance do you have HDP 2.1 installed? you may need to install the > utils and update the env variables per > http://stackoverflow.com/questions/18630019/running-apache-hadoop-2-1-0-on-windows > > > On Jul 2, 2014, at 10:20 AM, Konstantin Kudryavtsev < > kudryavtsev.konstan...@gmail.com> wrote: > > Hi Andrew, > > it's windows 7 and I doesn't set up any env variables here > > The full stack trace: > > 14/07/02 19:59:31 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 14/07/02 19:59:31 ERROR Shell: Failed to locate the winutils binary in the > hadoop binary path > java.io.IOException: Could not locate executable null\bin\winutils.exe in > the Hadoop binaries. > at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318) > at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333) > at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326) > at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76) > at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93) > at org.apache.hadoop.security.Groups.<init>(Groups.java:77) > at > org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240) > at > org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255) > at > org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:283) > at > org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:36) > at > org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:109) > at > org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala) > at org.apache.spark.SparkContext.<init>(SparkContext.scala:228) > at org.apache.spark.SparkContext.<init>(SparkContext.scala:97) > at my.example.EtlTest.testETL(IxtoolsDailyAggTest.scala:13) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:232) > at junit.framework.TestSuite.run(TestSuite.java:227) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:81) > at org.junit.runner.JUnitCore.run(JUnitCore.java:130) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74) > at > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120) > > > Thank you, > Konstantin Kudryavtsev > > > On Wed, Jul 2, 2014 at 8:15 PM, Andrew Or <and...@databricks.com> wrote: > >> Hi Konstatin, >> >> We use hadoop as a library in a few places in Spark. I wonder why the >> path includes "null" though. >> >> Could you provide the full stack trace? >> >> Andrew >> >> >> 2014-07-02 9:38 GMT-07:00 Konstantin Kudryavtsev < >> kudryavtsev.konstan...@gmail.com>: >> >> Hi all, >>> >>> I'm trying to run some transformation on *Spark*, it works fine on >>> cluster (YARN, linux machines). However, when I'm trying to run it on local >>> machine (*Windows 7*) under unit test, I got errors: >>> >>> java.io.IOException: Could not locate executable null\bin\winutils.exe in >>> the Hadoop binaries. >>> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318) >>> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333) >>> at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326) >>> at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76) >>> at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93) >>> >>> >>> My code is following: >>> >>> @Test >>> def testETL() = { >>> val conf = new SparkConf() >>> val sc = new SparkContext("local", "test", conf) >>> try { >>> val etl = new IxtoolsDailyAgg() // empty constructor >>> >>> val data = sc.parallelize(List("in1", "in2", "in3")) >>> >>> etl.etl(data) // rdd transformation, no access to SparkContext or >>> Hadoop >>> Assert.assertTrue(true) >>> } finally { >>> if(sc != null) >>> sc.stop() >>> } >>> } >>> >>> >>> Why is it trying to access hadoop at all? and how can I fix it? Thank >>> you in advance >>> >>> Thank you, >>> Konstantin Kudryavtsev >>> >> >> > >