for small cluster set the following settings
yarn-site.xml
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>32</value>
</property>
capacity-scheduler.xml
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.5</value>
<description>
Maximum percent of resources in the cluster which can be used to run
application masters i.e. controls number of concurrent running
applications.
</description>
</property>
Probably yarn can not allocate mem for AM container. dafault value is 0.1
and Spark AM need 896 MB (and 0.1 gives just 393 MB which is not enough)
On Tue, Mar 29, 2016 at 2:35 PM, Vineet Mishra <clearmido...@gmail.com>
wrote:
> Yarn seems to be running fine, I have successful MR jobs completed on the
> same,
>
> *Cluster Metrics*
> *Apps Submitted Apps Pending Apps Running Apps Completed Containers
> Running Memory Used Memory Total Memory Reserved VCores Used VCores Total
> VCores Reserved Active Nodes Decommissioned Nodes Lost Nodes Unhealthy
> Nodes Rebooted Nodes*
> *1 0 0 1 0 0 B 8 GB 0 B 0 8 0 1 0 0 0 0*
> *User Metrics for dr.who*
> *Apps Submitted Apps Pending Apps Running Apps Completed Containers
> Running Containers Pending Containers Reserved Memory Used Memory Pending
> Memory Reserved VCores Used VCores Pending VCores Reserved*
> *0 0 0 1 0 0 0 0 B 0 B 0 B 0 0 0*
> *Show entriesSearch: *
> *ID*
> *User*
> *Name*
> *Application Type*
> *Queue*
> *StartTime*
> *FinishTime*
> *State*
> *FinalStatus*
> *Progress*
> *Tracking UI*
> *application_1459287061048_0001 myhost word count MAPREDUCE root.myhost
> Tue, 29 Mar 2016 21:31:39 GMT Tue, 29 Mar 2016 21:31:59 GMT FINISHED
> SUCCEEDED *
> *History*
>
> On Wed, Mar 30, 2016 at 2:52 AM, Alexander Pivovarov <apivova...@gmail.com
> > wrote:
>
>> check resource manager and node manager logs.
>> Maybe you find smth explaining why 1 app is pending
>>
>> do you have any app run successfully? *Apps Completed is 0 on the UI*
>>
>>
>> On Tue, Mar 29, 2016 at 2:13 PM, Vineet Mishra <clearmido...@gmail.com>
>> wrote:
>>
>>> Hi Alex/Surendra,
>>>
>>> Hadoop is up and running fine and I am able to run example on the same.
>>>
>>> *Cluster Metrics*
>>> *Apps Submitted Apps Pending Apps Running Apps Completed Containers
>>> Running Memory Used Memory Total Memory Reserved VCores Used VCores Total
>>> VCores Reserved Active Nodes Decommissioned Nodes Lost Nodes Unhealthy
>>> Nodes Rebooted Nodes*
>>> *1 1 0 0 0 0 B 3.93 GB 0 B 0 4 0 1 0 0 0 0*
>>> *User Metrics for dr.who*
>>> *Apps Submitted Apps Pending Apps Running Apps Completed Containers
>>> Running Containers Pending Containers Reserved Memory Used Memory Pending
>>> Memory Reserved VCores Used VCores Pending VCores Reserved*
>>> *0 1 0 0 0 0 0 0 B 0 B 0 B 0 0 0*
>>>
>>> Any Other trace?
>>>
>>> On Wed, Mar 30, 2016 at 2:31 AM, Alexander Pivovarov <
>>> apivova...@gmail.com> wrote:
>>>
>>>> check 8088 ui
>>>> - how many cores and memory available
>>>> - how many slaves are active
>>>>
>>>> run teragen or pi from hadoop examples to make sure that yarn works
>>>>
>>>> On Tue, Mar 29, 2016 at 1:25 PM, Surendra , Manchikanti <
>>>> surendra.manchika...@gmail.com> wrote:
>>>>
>>>>> Hi Vineeth,
>>>>>
>>>>> Can you please check resource(RAM,Cores) availability in your local
>>>>> cluster, And change accordingly.
>>>>>
>>>>> Regards,
>>>>> Surendra M
>>>>>
>>>>> -- Surendra Manchikanti
>>>>>
>>>>> On Tue, Mar 29, 2016 at 1:15 PM, Vineet Mishra <clearmido...@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> While starting Spark on Yarn on local cluster(Single Node Hadoop 2.6
>>>>>> yarn) I am facing some issues.
>>>>>>
>>>>>> As I try to start the Spark Shell it keeps on iterating in a endless
>>>>>> loop while initiating,
>>>>>>
>>>>>> *6/03/30 01:32:38 DEBUG ipc.Client: IPC Client (1782965120)
>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> from
>>>>>> myhost sending #11971*
>>>>>> *16/03/30 01:32:38 DEBUG ipc.Client: IPC Client (1782965120)
>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> from
>>>>>> myhost got value #11971*
>>>>>> *16/03/30 01:32:38 DEBUG ipc.ProtobufRpcEngine: Call:
>>>>>> getApplicationReport took 1ms*
>>>>>> *16/03/30 01:32:38 INFO yarn.Client: Application report for
>>>>>> application_1459260674306_0003 (state: ACCEPTED)*
>>>>>> *16/03/30 01:32:38 DEBUG yarn.Client: *
>>>>>> * client token: N/A*
>>>>>> * diagnostics: N/A*
>>>>>> * ApplicationMaster host: N/A*
>>>>>> * ApplicationMaster RPC port: -1*
>>>>>> * queue: root.thequeue*
>>>>>> * start time: 1459269797431*
>>>>>> * final status: UNDEFINED*
>>>>>> * tracking URL:
>>>>>> http://myhost:8088/proxy/application_1459260674306_0003/
>>>>>> <http://myhost:8088/proxy/application_1459260674306_0003/>*
>>>>>> * user: myhost*
>>>>>>
>>>>>> *16/03/30 01:45:07 DEBUG ipc.Client: IPC Client (101088744)
>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> from
>>>>>> myhost sending #338*
>>>>>> *16/03/30 01:45:07 DEBUG ipc.Client: IPC Client (101088744)
>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> from
>>>>>> myhost got value #338*
>>>>>> *16/03/30 01:45:07 DEBUG ipc.ProtobufRpcEngine: Call:
>>>>>> getApplicationReport took 2ms*
>>>>>> *16/03/30 01:45:08 DEBUG ipc.Client: IPC Client (101088744)
>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> from
>>>>>> myhost sending #339*
>>>>>> *16/03/30 01:45:08 DEBUG ipc.Client: IPC Client (101088744)
>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> from
>>>>>> myhost got value #339*
>>>>>> *16/03/30 01:45:08 DEBUG ipc.ProtobufRpcEngine: Call:
>>>>>> getApplicationReport took 2ms*
>>>>>> *16/03/30 01:45:09 DEBUG ipc.Client: IPC Client (101088744)
>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> from
>>>>>> myhost sending #340*
>>>>>> *16/03/30 01:45:09 DEBUG ipc.Client: IPC Client (101088744)
>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> from
>>>>>> myhost got value #340*
>>>>>> *16/03/30 01:45:09 DEBUG ipc.ProtobufRpcEngine: Call:
>>>>>> getApplicationReport took 2ms*
>>>>>> *16/03/30 01:45:10 DEBUG ipc.Client: IPC Client (101088744)
>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> from
>>>>>> myhost sending #341*
>>>>>> *16/03/30 01:45:10 DEBUG ipc.Client: IPC Client (101088744)
>>>>>> connection to myhost/192.168.1.108:8032 <http://192.168.1.108:8032> from
>>>>>> myhost got value #341*
>>>>>> *16/03/30 01:45:10 DEBUG ipc.ProtobufRpcEngine: Call:
>>>>>> getApplicationReport took 1ms*
>>>>>>
>>>>>> Any leads would be appreciated.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
