i know i just wanna to test the shell interpreter's function. and i am confused....
On Wed, Jul 22, 2015 at 9:02 PM, IT CTO <goi....@gmail.com> wrote: > Why would you want to run spark-shell from zeppelin? as I see it > spark-shell in a REPL tool for spark and zeppelin is a REPL tool for spark > (and others...) > Eran > > On Wed, Jul 22, 2015 at 3:47 PM 江之源 <jiangzhiy...@liulishuo.com> wrote: > >> ok it works! 3q >> And i have an another question about shell >> The shell could work when type some simple commands.but it couldn’t work >> when i wanna to start a spark shell or spark submit even the sudo command. >> I wanna know the shell interpreter’s limit. why it always call >> Process exited with an error: 1 (Exit value: 1) when i type sudo or >> spark-submit >> >> and when i type spark-shell it feedback: >> >> >> setting ulimit -m to 57671680 Spark assembly has been built with Hive, >> including Datanucleus jars on classpath SLF4J: Class path contains multiple >> SLF4J bindings. SLF4J: Found binding in >> [jar:file:/home/pipeline/zeppelin-manager/zeppelin-0.5.0-incubating-SNAPSHOT/interpreter/sh/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: Found binding in >> [jar:file:/home/pipeline/zeppelin-manager/zeppelin-0.5.0-incubating-SNAPSHOT/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: Found binding in >> [jar:file:/opt/simple-spark.1.3.0/lib/spark-assembly-1.3.0-hadoop2.0.0-mr1-cdh4.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >> explanation. SLF4J: Actual binding is of type >> [org.slf4j.impl.Log4jLoggerFactory] log4j:ERROR setFile(null,true) call >> failed. java.io.FileNotFoundException: (没有那个文件或目录) at >> java.io.FileOutputStream.open(Native Method) at >> java.io.FileOutputStream.<init>(FileOutputStream.java:221) at >> java.io.FileOutputStream.<init>(FileOutputStream.java:142) at >> org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at >> org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165) at >> org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223) >> at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307) >> at >> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172) >> at >> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104) >> at >> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842) >> at >> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) >> at >> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) >> at >> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) >> at >> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580) >> at >> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) >> at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at >> org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:64) at >> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:285) at >> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155) >> at >> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132) >> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685) at >> org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:78) >> at >> org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:1996) >> at >> org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:1996) >> at scala.Option.getOrElse(Option.scala:120) at >> org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:1996) at >> org.apache.spark.SecurityManager.<init>(SecurityManager.scala:207) at >> org.apache.spark.repl.SparkIMain.<init>(SparkIMain.scala:118) at >> org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.<init>(SparkILoop.scala:187) >> at org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:216) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:948) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944) >> at >> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) >> at >> org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944) >> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058) at >> org.apache.spark.repl.Main$.main(Main.scala:31) at >> org.apache.spark.repl.Main.main(Main.scala) at >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569) >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) at >> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) at >> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) log4j:ERROR >> Either File or DatePattern options are not set for appender [dailyfile]. >> Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ >> .__/\_,_/_/ /_/\_\ version 1.3.0 /_/ Using Scala version 2.10.4 (OpenJDK >> 64-Bit Server VM, Java 1.7.0_71) Type in expressions to have them >> evaluated. Type :help for more information. >> org.apache.spark.SparkException: Found both spark.executor.extraClassPath >> and SPARK_CLASSPATH. Use only the former. at >> org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$7.apply(SparkConf.scala:339) >> at >> org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$7.apply(SparkConf.scala:337) >> at scala.collection.immutable.List.foreach(List.scala:318) at >> org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:337) >> at >> org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:325) >> at scala.Option.foreach(Option.scala:236) at >> org.apache.spark.SparkConf.validateSettings(SparkConf.scala:325) at >> org.apache.spark.SparkContext.<init>(SparkContext.scala:197) at >> org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1016) >> at $iwC$$iwC.<init>(<console>:9) at $iwC.<init>(<console>:18) at >> <init>(<console>:20) at .<init>(<console>:24) at .<clinit>(<console>) at >> .<init>(<console>:7) at .<clinit>(<console>) at $print(<console>) at >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) at >> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) >> at >> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338) >> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) >> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at >> org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) at >> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856) at >> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901) >> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813) at >> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:123) >> at >> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:122) >> at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324) >> at >> org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:122) >> at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64) at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:973) >> at >> org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:157) >> at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64) at >> org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:106) >> at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:990) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944) >> at >> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944) >> at >> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) >> at >> org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944) >> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058) at >> org.apache.spark.repl.Main$.main(Main.scala:31) at >> org.apache.spark.repl.Main.main(Main.scala) at >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569) >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) at >> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) at >> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> java.lang.NullPointerException at >> org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:141) at >> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:49) at >> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at >> org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1027) at >> $iwC$$iwC.<init>(<console>:9) >> <console>:10: error: not found: value sqlContext import >> sqlContext.implicits._ ^ <console>:10: error: not found: value sqlContext >> import sqlContext.sql ^ scala> Stopping spark context. <console>:14: error: >> not found: value sc sc.stop() ^ >> >> On Jul 22, 2015, at 8:36 PM, Alexander Bezzubov <abezzu...@nflabs.com> >> wrote: >> >> Hi, >> >> those are very valid steps indeed and you do not need to build custom >> spark version. >> >> Hope this helps! >> >> On Tue, Jul 21, 2015 at 7:46 PM, 江之源 <jiangzhiy...@liulishuo.com> wrote: >> >> hi >> To install the zeppelin with z-manager is a helpless method, because i >> install the zeppelin in manually way and failed.I have tried many many >> times. >> My cluster is spark1.3.0 hadoop 2.0.0-cdh4.5.0,and the model is >> standalone. >> I will install the zeppelin manually right now so i wanna you check my >> steps: >> >> 1. git clone the repository from the github. >> 2. mvn clean package >> 3. mvn install -DskipTests -Dspark.version=1.3.0 >> -Dhadoop.version=2.0.0-cdh4.5.0 >> (did zeppelin support cdh4.5.0) >> Should i have to do the custom built spark >> like(-Dspark.version=1.1.0-Custom) >> 4.modify my master spark://...:7077 >> Is it completed? or i lost something please tell me. >> thanks >> jzy >> >> 在 2015年7月21日，下午5:48，Alexander Bezzubov <abezzu...@nflabs.com> 写道： >> >> Hi, >> >> thank you for your interest in the project! >> >> It seems like the best way to get Zeppelin up and running in your case >> would be to build it manually with relevant Spark\Hadoop options as >> described here >> http://zeppelin.incubator.apache.org/docs/install/install.html >> >> Please, let me know if that helps. >> >> -- >> BR, >> Alex >> >> On Tue, Jul 21, 2015 at 11:35 AM, 江之源 <jiangzhiy...@liulishuo.com> wrote: >> >> hi >> i installed zeppelin some time before, but it always failed in my server >> cluster. i found the z-management Occasionally. I installed and success in >> my server. But when i wanna to read in my HDFS file like: >> >> sc.textFile("hdfs://llscluster/tmp/jzyresult/part-04093").count() >> >> >> it throw the errors in my cluster:Job aborted due to stage failure: Task >> 15 >> in stage 6.0 failed 4 times, most recent failure: Lost task 15.3 in stage >> 6.0 (TID 386, lls7): java.io.EOFException >> >> when i modify it to the local model, it could read HDFS file successfully. >> My cluster is Spark1.3.0 Hadoop2.0.0-CDH4.5.0. but the install options >> just >> have Spark1.3.0 and Hadoop2.0.0-CDH-4.7.0. Is this the cause to read HDFS >> file failed? >> Look forward to your reply! >> THANK YOU! >> JZY >> >> >> >> >> -- >> -- >> Kind regards, >> Alexander. >> >> >> >> >> >> -- >> -- >> Kind regards, >> Alexander. >> >> >>
