[ https://issues.apache.org/jira/browse/SPARK-12190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-12190. ------------------------------- Resolution: Duplicate Please search JIRA first; plenty of comment about /tmp/hive > spark does not start cleanly windows 7 64 bit > --------------------------------------------- > > Key: SPARK-12190 > URL: https://issues.apache.org/jira/browse/SPARK-12190 > Project: Spark > Issue Type: Bug > Components: Windows > Affects Versions: 1.5.2 > Environment: windows 7 64 bit > PATH includes: > C:\Users\Stefan\spark-1.5.2-bin-hadoop2.6\bin > C:\ProgramData\Oracle\Java\javapath > C:\Users\Stefan\scala\bin > SYSTEM variables set are: > JAVA_HOME=C:\Program Files\Java\jre1.8.0_65 > HADOOP_HOME=C:\Users\Stefan\hadoop-2.6.0 > (where the bin\winutils resides) > Reporter: stefan > Labels: newbie > > See environment description above for all my PATH info and ENV variables. > Hadoop is not compiled, nor is a distributed storage set up, but the hadoop > binary with winutils. exe was downloaded from here: > https://www.barik.net/archive/2015/01/19/172716/ > and moved to the home directory > Spark was not built on this machine but rather the precompiled binary was > downloaded. > Java is this version: > java version "1.8.0_65" > Java(TM) SE Runtime Environment (build 1.8.0_65-b17) > Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode) > Spark-shell is invoked and the error is shown below: > C:\Users\Stefan\spark-1.5.2-bin-hadoop2.6\bin>spark-shell > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. > Using Spark's repl log4j profile: > org/apache/spark/log4j-defaults-repl.properties > To adjust logging level use sc.setLogLevel("INFO") > Welcome to > ____ __ > / __/__ ___ _____/ /__ > _\ \/ _ \/ _ `/ __/ '_/ > /___/ .__/\_,_/_/ /_/\_\ version 1.5.2 > /_/ > Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_65) > Type in expressions to have them evaluated. > Type :help for more information. > 15/12/07 21:18:40 WARN MetricsSystem: Using default name DAGScheduler for > source because spark.app.id is not set. > Spark context available as sc. > 15/12/07 21:18:42 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" > is already registered. Ensure you dont have multiple JAR versions of the same > plugin in the classpath. The URL > "file:/C:/Users/Stefan/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar" > is already registered, and you are trying to register an identical plugin > located at URL > "file:/C:/Users/Stefan/spark-1.5.2-bin-hadoop2.6/bin/../lib/datanucleus-rdbms-3.2.9.jar." > 15/12/07 21:18:42 WARN General: Plugin (Bundle) "org.datanucleus" is already > registered. Ensure you dont have multiple JAR versions of the same plugin in > the classpath. The URL > "file:/C:/Users/Stefan/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar" > is already registered, and you are trying to register an identical plugin > located at URL > "file:/C:/Users/Stefan/spark-1.5.2-bin-hadoop2.6/bin/../lib/datanucleus-core-3.2.10.jar." > 15/12/07 21:18:42 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is > already registered. Ensure you dont have multiple JAR versions of the same > plugin in the classpath. The URL > "file:/C:/Users/Stefan/spark-1.5.2-bin-hadoop2.6/bin/../lib/datanucleus-api-jdo-3.2.6.jar" > is already registered, and you are trying to register an identical plugin > located at URL > "file:/C:/Users/Stefan/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar." > 15/12/07 21:18:42 WARN Connection: BoneCP specified but not present in > CLASSPATH (or one of dependencies) > 15/12/07 21:18:42 WARN Connection: BoneCP specified but not present in > CLASSPATH (or one of dependencies) > 15/12/07 21:18:47 WARN ObjectStore: Version information not found in > metastore. hive.metastore.schema.verification is not enabled so recording the > schema version 1.2.0 > 15/12/07 21:18:47 WARN ObjectStore: Failed to get database default, returning > NoSuchObjectException > 15/12/07 21:18:47 WARN : Your hostname, BloomBear-SSD resolves to a > loopback/non-reachable address: fe80:0:0:0:2424:cdcb:ecc1:c9cb%eth6, but we > couldn't find any external IP address! > java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: > /tmp/hive on HDFS should be writable. Current permissions are: --------- > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) > at > org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:171) > at > org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:162) > at > org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:160) > at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:167) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown > Source) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown > Source) > at java.lang.reflect.Constructor.newInstance(Unknown Source) > at > org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028) > at $iwC$$iwC.<init>(<console>:9) > at $iwC.<init>(<console>:18) > at <init>(<console>:20) > at .<init>(<console>:24) > at .<clinit>(<console>) > at .<init>(<console>:7) > at .<clinit>(<console>) > at $print(<console>) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at > org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) > at > org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340) > at > org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) > at > org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) > at > org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) > at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) > at > org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:132) > at > org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124) > at > org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324) > at > org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124) > at > org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64) > at > org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974) > at > org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159) > at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64) > at > org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108) > at > org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64) > at > org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991) > at > org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) > at > org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) > at > scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) > at > org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) > at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) > at org.apache.spark.repl.Main$.main(Main.scala:31) > at org.apache.spark.repl.Main.main(Main.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on > HDFS should be writable. Current permissions are: --------- > at > org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:612) > at > org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554) > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508) > ... 56 more > <console>:10: error: not found: value sqlContext > import sqlContext.implicits._ > ^ > <console>:10: error: not found: value sqlContext > import sqlContext.sql > ^ > Please copy the above output to a text editor so it is more legible. > There are three files that SPARK complains about being registered in > duplicate. For example, this file exists: > > "file:/C:/Users/Stefan/spark-1.5.2-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar" > but it is the same as this file > "file:/C:/Users/Stefan/spark-1.5.2-bin-hadoop2.6/bin/../lib/datanucleus-rdbms-3.2.9.jar." > I gave up w scala and installed python. Remarkably, pyspark starts cleanly. > So that means it is a scala install problem? > I changed one of my PATH entries from > C:\Users\Stefan\scala\bin > to > C:\Users\Stefan\scala > made no difference > So I searched for "datanucleus-rdbms-3.2.9.jar" within all the files in the > scala directory. No hits. This text also does not exist in any of the files > in the hadoop or the spark directories > So I do not know why spark is complaining about that. > I actually do not have this package installed: > package org.datanucleus.store.rdbms; > I think maybe because I did not set up a distributed storage system for > hadoop?? > I think this error may be related as well: > BoneCP specified but not present in CLASSPATH (or one of dependencies) > And then What's this about? > The root scratch dir: /tmp/hive on HDFS should be writable. > I need some hints to clear this up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org