All, I noticed Hortonworks has published Spark 1.4.1 Preview packages. http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
It worked for me with HDP 2.2 Thanks Senthil On Wed, Sep 9, 2015 at 12:07 AM, Senthil <[email protected]> wrote: > Hi Christian > > Thank you so much. the build meets my requirements. However I have trouble > in running the spark job after installation. The jobs are failing with the > following error. I get the same error when I run SparkPI app or > sparkshell. Also I could not see the image illustration to modify yarn.site > (hdp.stack and hdp.version) . Could you tell us how to set hdp-stack. Any > help will be greatly appreciated. > > > Here is the error on console as well as job log > > Application application_1441682258490_0008 failed 2 times due to AM > Container for appattempt_1441682258490_0008_000002 exited with exitCode: 1 > For more detailed output, check application tracking page: > http://ip-10-0-3-206.us-west-2.compute.internal:8088/proxy/application_1441682258490_0008/Then, > click on links to logs of each attempt. > Diagnostics: Exception from container-launch. > Container id: container_1441682258490_0008_02_000001 > Exit code: 1 > Exception message: > /hadoop/yarn/local/usercache/root/appcache/application_1441682258490_0008/container_1441682258490_0008_02_000001/launch_container.sh: > line 27: > $PWD:$PWD/__hadoop_conf__:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: > bad substitution > Stack trace: ExitCodeException exitCode=1: > /hadoop/yarn/local/usercache/root/appcache/application_1441682258490_0008/container_1441682258490_0008_02_000001/launch_container.sh: > line 27: > $PWD:$PWD/__hadoop_conf__:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: > bad substitution > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Container exited with a non-zero exit code 1 > Failing this attempt. Failing the application. > > > > Regards, > > Senthil > > On Mon, Sep 7, 2015 at 10:42 PM, Christian Tzolov <[email protected]> > wrote: > >> Hi Senthil, >> >> I've build Spark 1.4 RPMs using BigTop. Here are the instructions: >> http://blog.tzolov.net/2015/07/how-to-install-spark-140-on-pivotalhd.html?view=sidebar >> >> it has been tested with HDP2.2, PHD3.0. Make sure to set the right >> stack.name and stack.versions! >> >> Cheers, >> Christian >> >> On 7 September 2015 at 11:47, Senthil <[email protected]> wrote: >> >>> >>> Has anyone built HDP compatible RPMs (CentOS6) for Spark 1.4? >>> >>> Using Ambari, I am trying to automate Hadoop cluster setup with Hadoop >>> 2.6, Hive 0.14 and Spark 1.4. Ambari supports HDP 2.2 and able to install >>> Hadoop 2.6 and Hive. I like to customize the Ambari stack to include Spark >>> 1.4 instead of 1.2 (bundled) . >>> >>> Regards, >>> >>> - Senthil >>> >> >> >> >> -- >> Christian Tzolov <http://www.linkedin.com/in/tzolov> | Solution >> Architect, EMEA Practice Team | Pivotal <http://pivotal.io/> >> [email protected]|+31610285517 >> > > > > -- > - Senthil > -- - Senthil
