Dear Hive community, I am new to hive, Spark and Hadoop and I am trying to setup a zeppelin notebook so I can submit spark jobs in yarn cluster mode (I am trying to make spark driver to run in one of my cluster nodes).
I followed the Zeppelin documentation to setup spark in yarn cluster mode but when I submit a job I get the following error message: 1. java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.1.3.1.0.0-78 2. at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174) 3. at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139) 4. at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100) 5. at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368) 6. at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105) 7. at java.lang.Class.forName0(Native Method) 8. at java.lang.Class.forName(Class.java:348) 9. at org.apache.spark.util.Utils$.classForName(Utils.scala:230) 10. at org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1063) 11. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 12. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 13. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 14. at java.lang.reflect.Method.invoke(Method.java:498) 15. at org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:230) 16. at org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:165) 17. at org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:87) 18. at org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:102) 19. at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62) 20. at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) 21. at org.apache.zeppelin.spark.PySparkInterpreter.getSparkInterpreter(PySparkInterpreter.java:664) 22. at org.apache.zeppelin.spark.PySparkInterpreter.createGatewayServerAndStartScript(PySparkInterpreter.java:260) 23. at org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:194) 24. at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) 25. at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:617) 26. at org.apache.zeppelin.scheduler.Job.run(Job.java:188) 27. at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140) 28. at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 29. at java.util.concurrent.FutureTask.run(FutureTask.java:266) 30. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) 31. at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) 32. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 33. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 34. at java.lang.Thread.run(Thread.java:745) Any advice? Manuel NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
