:~/Downloads/package/spark-1.6.1-bin-hadoop2.6$ bin/spark-shell --master yarn-client
16/03/30 03:24:43 DEBUG ipc.Client: IPC Client (111576772) connection to myhost/192.168.1.108:8032 from myhost sending #138 16/03/30 03:24:43 DEBUG ipc.Client: IPC Client (111576772) connection to myhost/192.168.1.108:8032 from myhost got value #138 16/03/30 03:24:43 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 3ms 16/03/30 03:24:44 DEBUG ipc.Client: IPC Client (111576772) connection to myhost/192.168.1.108:8032 from myhost sending #139 16/03/30 03:24:44 DEBUG ipc.Client: IPC Client (111576772) connection to myhost/192.168.1.108:8032 from myhost got value #139 16/03/30 03:24:44 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 4ms scala> scala> scala> scala> scala> 16/03/30 03:24:45 DEBUG ipc.Client: IPC Client (111576772) connection to myhost/192.168.1.108:8032 from myhost sending #140 16/03/30 03:24:45 DEBUG ipc.Client: IPC Client (111576772) connection to myhost/192.168.1.108:8032 from myhost got value #140 16/03/30 03:24:45 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 4ms 16/03/30 03:24:46 DEBUG ipc.Client: IPC Client (111576772) connection to myhost/192.168.1.108:8032 from myhost sending #141 16/03/30 03:24:46 DEBUG ipc.Client: IPC Client (111576772) connection to myhost/192.168.1.108:8032 from myhost got value #141 16/03/30 03:24:46 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 2ms 16/03/30 03:24:47 DEBUG ipc.Client: IPC Client (111576772) connection to myhost/192.168.1.108:8032 from myhost sending #142 16/03/30 03:24:47 DEBUG ipc.Client: IPC Client (111576772) connection to myhost/192.168.1.108:8032 from myhost got value #142 16/03/30 03:24:47 DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 4ms On Wed, Mar 30, 2016 at 3:26 AM, Vineet Mishra <clearmido...@gmail.com> wrote: > Looks like still the same while the other MR application is working fine, > > > > On Wed, Mar 30, 2016 at 3:15 AM, Alexander Pivovarov <apivova...@gmail.com > > wrote: > >> for small cluster set the following settings >> >> yarn-site.xml >> >> <property> >> <name>yarn.scheduler.minimum-allocation-mb</name> >> <value>32</value> >> </property> >> >> >> capacity-scheduler.xml >> >> <property> >> <name>yarn.scheduler.capacity.maximum-am-resource-percent</name> >> <value>0.5</value> >> <description> >> Maximum percent of resources in the cluster which can be used to run >> application masters i.e. controls number of concurrent running >> applications. >> </description> >> </property> >> >> >> Probably yarn can not allocate mem for AM container. dafault value is 0.1 >> and Spark AM need 896 MB (and 0.1 gives just 393 MB which is not enough) >> >> On Tue, Mar 29, 2016 at 2:35 PM, Vineet Mishra <clearmido...@gmail.com> >> wrote: >> >>> Yarn seems to be running fine, I have successful MR jobs completed on >>> the same, >>> >>> *Cluster Metrics* >>> *Apps Submitted Apps Pending Apps Running Apps Completed Containers >>> Running Memory Used Memory Total Memory Reserved VCores Used VCores Total >>> VCores Reserved Active Nodes Decommissioned Nodes Lost Nodes Unhealthy >>> Nodes Rebooted Nodes* >>> *1 0 0 1 0 0 B 8 GB 0 B 0 8 0 1 0 0 0 0* >>> *User Metrics for dr.who* >>> *Apps Submitted Apps Pending Apps Running Apps Completed Containers >>> Running Containers Pending Containers Reserved Memory Used Memory Pending >>> Memory Reserved VCores Used VCores Pending VCores Reserved* >>> *0 0 0 1 0 0 0 0 B 0 B 0 B 0 0 0* >>> *Show entriesSearch: * >>> *ID* >>> *User* >>> *Name* >>> *Application Type* >>> *Queue* >>> *StartTime* >>> *FinishTime* >>> *State* >>> *FinalStatus* >>> *Progress* >>> *Tracking UI* >>> *application_1459287061048_0001 myhost word count MAPREDUCE root.myhost >>> Tue, 29 Mar 2016 21:31:39 GMT Tue, 29 Mar 2016 21:31:59 GMT FINISHED >>> SUCCEEDED * >>> *History* >>> >>> On Wed, Mar 30, 2016 at 2:52 AM, Alexander Pivovarov < >>> apivova...@gmail.com> wrote: >>> >>>> check resource manager and node manager logs. >>>> Maybe you find smth explaining why 1 app is pending >>>> >>>> do you have any app run successfully? *Apps Completed is 0 on the UI* >>>> >>>> >>>> On Tue, Mar 29, 2016 at 2:13 PM, Vineet Mishra <clearmido...@gmail.com> >>>> wrote: >>>> >>>>> Hi Alex/Surendra, >>>>> >>>>> Hadoop is up and running fine and I am able to run example on the same. >>>>> >>>>> *Cluster Metrics* >>>>> *Apps Submitted Apps Pending Apps Running Apps Completed Containers >>>>> Running Memory Used Memory Total Memory Reserved VCores Used VCores Total >>>>> VCores Reserved Active Nodes Decommissioned Nodes Lost Nodes Unhealthy >>>>> Nodes Rebooted Nodes* >>>>> *1 1 0 0 0 0 B 3.93 GB 0 B 0 4 0 1 0 0 0 0* >>>>> *User Metrics for dr.who* >>>>> *Apps Submitted Apps Pending Apps Running Apps Completed Containers >>>>> Running Containers Pending Containers Reserved Memory Used Memory Pending >>>>> Memory Reserved VCores Used VCores Pending VCores Reserved* >>>>> *0 1 0 0 0 0 0 0 B 0 B 0 B 0 0 0* >>>>> >>>>> Any Other trace? >>>>> >>>>> On Wed, Mar 30, 2016 at 2:31 AM, Alexander Pivovarov < >>>>> apivova...@gmail.com> wrote: >>>>> >>>>>> check 8088 ui >>>>>> - how many cores and memory available >>>>>> - how many slaves are active >>>>>> >>>>>> run teragen or pi from hadoop examples to make sure that yarn works >>>>>> >>>>>> On Tue, Mar 29, 2016 at 1:25 PM, Surendra , Manchikanti < >>>>>> surendra.manchika...@gmail.com> wrote: >>>>>> >>>>>>> Hi Vineeth, >>>>>>> >>>>>>> Can you please check resource(RAM,Cores) availability in your local >>>>>>> cluster, And change accordingly. >>>>>>> >>>>>>> Regards, >>>>>>> Surendra M >>>>>>> >>>>>>> -- Surendra Manchikanti >>>>>>> >>>>>>> On Tue, Mar 29, 2016 at 1:15 PM, Vineet Mishra < >>>>>>> clearmido...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> While starting Spark on Yarn on local cluster(Single Node Hadoop >>>>>>>> 2.6 yarn) I am facing some issues. >>>>>>>> >>>>>>>> As I try to start the Spark Shell it keeps on iterating in a >>>>>>>> endless loop while initiating, >>>>>>>> >>>>>>>> *6/03/30 01:32:38 DEBUG ipc.Client: IPC Client (1782965120) >>>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> >>>>>>>> from >>>>>>>> myhost sending #11971* >>>>>>>> *16/03/30 01:32:38 DEBUG ipc.Client: IPC Client (1782965120) >>>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> >>>>>>>> from >>>>>>>> myhost got value #11971* >>>>>>>> *16/03/30 01:32:38 DEBUG ipc.ProtobufRpcEngine: Call: >>>>>>>> getApplicationReport took 1ms* >>>>>>>> *16/03/30 01:32:38 INFO yarn.Client: Application report for >>>>>>>> application_1459260674306_0003 (state: ACCEPTED)* >>>>>>>> *16/03/30 01:32:38 DEBUG yarn.Client: * >>>>>>>> * client token: N/A* >>>>>>>> * diagnostics: N/A* >>>>>>>> * ApplicationMaster host: N/A* >>>>>>>> * ApplicationMaster RPC port: -1* >>>>>>>> * queue: root.thequeue* >>>>>>>> * start time: 1459269797431* >>>>>>>> * final status: UNDEFINED* >>>>>>>> * tracking URL: >>>>>>>> http://myhost:8088/proxy/application_1459260674306_0003/ >>>>>>>> <http://myhost:8088/proxy/application_1459260674306_0003/>* >>>>>>>> * user: myhost* >>>>>>>> >>>>>>>> *16/03/30 01:45:07 DEBUG ipc.Client: IPC Client (101088744) >>>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> >>>>>>>> from >>>>>>>> myhost sending #338* >>>>>>>> *16/03/30 01:45:07 DEBUG ipc.Client: IPC Client (101088744) >>>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> >>>>>>>> from >>>>>>>> myhost got value #338* >>>>>>>> *16/03/30 01:45:07 DEBUG ipc.ProtobufRpcEngine: Call: >>>>>>>> getApplicationReport took 2ms* >>>>>>>> *16/03/30 01:45:08 DEBUG ipc.Client: IPC Client (101088744) >>>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> >>>>>>>> from >>>>>>>> myhost sending #339* >>>>>>>> *16/03/30 01:45:08 DEBUG ipc.Client: IPC Client (101088744) >>>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> >>>>>>>> from >>>>>>>> myhost got value #339* >>>>>>>> *16/03/30 01:45:08 DEBUG ipc.ProtobufRpcEngine: Call: >>>>>>>> getApplicationReport took 2ms* >>>>>>>> *16/03/30 01:45:09 DEBUG ipc.Client: IPC Client (101088744) >>>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> >>>>>>>> from >>>>>>>> myhost sending #340* >>>>>>>> *16/03/30 01:45:09 DEBUG ipc.Client: IPC Client (101088744) >>>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> >>>>>>>> from >>>>>>>> myhost got value #340* >>>>>>>> *16/03/30 01:45:09 DEBUG ipc.ProtobufRpcEngine: Call: >>>>>>>> getApplicationReport took 2ms* >>>>>>>> *16/03/30 01:45:10 DEBUG ipc.Client: IPC Client (101088744) >>>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> >>>>>>>> from >>>>>>>> myhost sending #341* >>>>>>>> *16/03/30 01:45:10 DEBUG ipc.Client: IPC Client (101088744) >>>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> >>>>>>>> from >>>>>>>> myhost got value #341* >>>>>>>> *16/03/30 01:45:10 DEBUG ipc.ProtobufRpcEngine: Call: >>>>>>>> getApplicationReport took 1ms* >>>>>>>> >>>>>>>> Any leads would be appreciated. >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
