Frans, SparkR runs with R 3.1+. If possible, latest verison of R is recommended.
From: Saisai Shao [mailto:sai.sai.s...@gmail.com] Sent: Thursday, October 22, 2015 11:17 AM To: Frans Thamura Cc: Ajay Chander; Doug Balog; user spark mailing list Subject: Re: Spark_1.5.1_on_HortonWorks SparkR is shipped with Hortonworks version of Spark 1.4.1, there's no difference compared to community version, you could refer to the docs of Apache Spark. It would be better to ask HDP related questions in ( http://hortonworks.com/community/forums/forum/spark/ ). Sorry for not so familiar with SparkR related things. Thanks Saisai On Thu, Oct 22, 2015 at 11:02 AM, Frans Thamura <fr...@meruvian.org<mailto:fr...@meruvian.org>> wrote: talking about spark in hdp Is there reference about Spark-R, and what version should we install in R? -- Frans Thamura (曽志胜) Java Champion Shadow Master and Lead Investor Meruvian. Integrated Hypermedia Java Solution Provider. Mobile: +628557888699<tel:%2B628557888699> Blog: http://blogs.mervpolis.com/roller/flatburger (id) FB: http://www.facebook.com/meruvian TW: http://www.twitter.com/meruvian / @meruvian Website: http://www.meruvian.org "We grow because we share the same belief." On Thu, Oct 22, 2015 at 8:56 AM, Saisai Shao <sai.sai.s...@gmail.com<mailto:sai.sai.s...@gmail.com>> wrote: > How you start history server, do you still use the history server of 1.3.1, > or you started the history server in 1.5.1? > > The Spark tarball you used is the community version, so Application > TimelineServer based history provider is not supported, you could comment > this configuration "spark.history.provider", so it will use default > FsHistoryProvider, or you could configure "spark.history.provider" to > "org.apache.spark.deploy.history.FsHistoryProvider". > > If you still want to use this ATS based history server, you have to wait for > the technical preview release of Hortonworks. > > Thanks > Saisai > > > On Thu, Oct 22, 2015 at 9:47 AM, Ajay Chander > <itsche...@gmail.com<mailto:itsche...@gmail.com>> wrote: >> >> Hi Sasai, >> >> Thanks for your time. I have followed your inputs and downloaded >> "spark-1.5.1-bin-hadoop2.6" on one of the node say node1. And when I did a >> pie test everything seems to be working fine, except that the spark-history >> -server running on this node1 has gone down. It was complaining about >> missing class: >> >> 15/10/21 16:41:28 INFO HistoryServer: Registered signal handlers for >> [TERM, HUP, INT] >> 15/10/21 16:41:28 WARN SparkConf: The configuration key >> 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 >> and and may be removed in the future. Please use the new key >> 'spark.yarn.am.waitTime' instead. >> 15/10/21 16:41:29 WARN NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> 15/10/21 16:41:29 INFO SecurityManager: Changing view acls to: root >> 15/10/21 16:41:29 INFO SecurityManager: Changing modify acls to: root >> 15/10/21 16:41:29 INFO SecurityManager: SecurityManager: authentication >> disabled; ui acls disabled; users with view permissions: Set(root); users >> with modify permissions: Set(root) >> Exception in thread "main" java.lang.ClassNotFoundException: >> org.apache.spark.deploy.yarn.history.YarnHistoryProvider >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:348) >> at org.apache.spark.util.Utils$.classForName(Utils.scala:173) >> at >> org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:231) >> at >> org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala) >> >> >> I went to the lib folder and noticed that >> "spark-assembly-1.5.1-hadoop2.6.0.jar" is missing that class. I was able to >> get the spark history server started with 1.3.1 but not 1.5.1. Any inputs on >> this? >> >> Really appreciate your help. Thanks >> >> Regards, >> Ajay >> >> >> >> On Wednesday, October 21, 2015, Saisai Shao >> <sai.sai.s...@gmail.com<mailto:sai.sai.s...@gmail.com>> >> wrote: >>> >>> Hi Ajay, >>> >>> You don't need to copy tarball to all the nodes, only one node you want >>> to run spark application is enough (mostly the master node), Yarn will help >>> to distribute the Spark dependencies. The link I mentioned before is the one >>> you could follow, please read my previous mail. >>> >>> Thanks >>> Saisai >>> >>> >>> >>> On Thu, Oct 22, 2015 at 1:56 AM, Ajay Chander >>> <itsche...@gmail.com<mailto:itsche...@gmail.com>> >>> wrote: >>>> >>>> Thanks for your kind inputs. Right now I am running spark-1.3.1 on >>>> YARN(4 node cluster) on a HortonWorks distribution. Now I want to upgrade >>>> spark-1.3.1 to spark-1.5.1. So at this point of time, do I have to manually >>>> go and copy spark-1.5.1 tarbal to all the nodes or is there any alternative >>>> so that I can get it upgraded through Ambari UI ? If possible can anyone >>>> point me to a documentation online? Thank you. >>>> >>>> Regards, >>>> Ajay >>>> >>>> >>>> On Wednesday, October 21, 2015, Saisai Shao >>>> <sai.sai.s...@gmail.com<mailto:sai.sai.s...@gmail.com>> >>>> wrote: >>>>> >>>>> Hi Frans, >>>>> >>>>> You could download Spark 1.5.1-hadoop 2.6 pre-built tarball and copy >>>>> into HDP 2.3 sandbox or master node. Then copy all the conf files from >>>>> /usr/hdp/current/spark-client/ to your <spark-1.5.1>/conf, or you could >>>>> refer to this tech preview ( >>>>> http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/ >>>>> ), in "installing chapter", step 4 ~ 8 is what you need to do. >>>>> >>>>> Thanks >>>>> Saisai >>>>> >>>>> On Wed, Oct 21, 2015 at 1:27 PM, Frans Thamura >>>>> <fr...@meruvian.org<mailto:fr...@meruvian.org>> >>>>> wrote: >>>>>> >>>>>> Doug >>>>>> >>>>>> is it possible to put in HDP 2.3? >>>>>> >>>>>> esp in Sandbox >>>>>> >>>>>> can share how do you install it? >>>>>> >>>>>> >>>>>> F >>>>>> -- >>>>>> Frans Thamura (曽志胜) >>>>>> Java Champion >>>>>> Shadow Master and Lead Investor >>>>>> Meruvian. >>>>>> Integrated Hypermedia Java Solution Provider. >>>>>> >>>>>> Mobile: +628557888699<tel:%2B628557888699> >>>>>> Blog: http://blogs.mervpolis.com/roller/flatburger (id) >>>>>> >>>>>> FB: http://www.facebook.com/meruvian >>>>>> TW: http://www.twitter.com/meruvian / @meruvian >>>>>> Website: http://www.meruvian.org >>>>>> >>>>>> "We grow because we share the same belief." >>>>>> >>>>>> >>>>>> On Wed, Oct 21, 2015 at 12:24 PM, Doug Balog >>>>>> <doug.sparku...@dugos.com<mailto:doug.sparku...@dugos.com>> wrote: >>>>>> > I have been running 1.5.1 with Hive in secure mode on HDP 2.2.4 >>>>>> > without any problems. >>>>>> > >>>>>> > Doug >>>>>> > >>>>>> >> On Oct 21, 2015, at 12:05 AM, Ajay Chander >>>>>> >> <itsche...@gmail.com<mailto:itsche...@gmail.com>> >>>>>> >> wrote: >>>>>> >> >>>>>> >> Hi Everyone, >>>>>> >> >>>>>> >> Any one has any idea if spark-1.5.1 is available as a service on >>>>>> >> HortonWorks ? I have spark-1.3.1 installed on the Cluster and it is a >>>>>> >> HortonWorks distribution. Now I want upgrade it to spark-1.5.1. >>>>>> >> Anyone here >>>>>> >> have any idea about it? Thank you in advance. >>>>>> >> >>>>>> >> Regards, >>>>>> >> Ajay >>>>>> > >>>>>> > >>>>>> > >>>>>> > --------------------------------------------------------------------- >>>>>> > To unsubscribe, e-mail: >>>>>> > user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> >>>>>> > For additional commands, e-mail: >>>>>> > user-h...@spark.apache.org<mailto:user-h...@spark.apache.org> >>>>>> > >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: >>>>>> user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> >>>>>> For additional commands, e-mail: >>>>>> user-h...@spark.apache.org<mailto:user-h...@spark.apache.org> >>>>>> >>>>> >>> >
