Re: How to run spark shell using YARN

2018-03-14 Thread kant kodali 
Do I need to set SPARK_DIST_CLASSPATH or SPARK_CLASSPATH ? The latest
version of spark (2.3) only has SPARK_CLASSPATH.

On Wed, Mar 14, 2018 at 11:37 AM, kant kodali <kanth...@gmail.com> wrote:

> Hi,
>
> I am not using emr. And yes I restarted several times.
>
> On Wed, Mar 14, 2018 at 6:35 AM, Anthony, Olufemi <
> olufemi.anth...@capitalone.com> wrote:
>
>> After you updated your yarn-site.xml  file, did you restart the YARN
>> resource manager ?
>>
>>
>>
>> https://aws.amazon.com/premiumsupport/knowledge-center/
>> restart-service-emr/
>>
>>
>>
>> Femi
>>
>>
>>
>> *From: *kant kodali <kanth...@gmail.com>
>> *Date: *Wednesday, March 14, 2018 at 6:16 AM
>> *To: *Femi Anthony <femib...@gmail.com>
>> *Cc: *vermanurag <anurag.ve...@fnmathlogic.com>, "user @spark" <
>> user@spark.apache.org>
>> *Subject: *Re: How to run spark shell using YARN
>>
>>
>>
>> 16GB RAM.  AWS m4.xlarge. It's a three node cluster and I only have YARN
>> and  HDFS running. Resources are barely used however I believe there is
>> something in my config that is preventing YARN to see that I have good
>> amount of resources I think (thats my guess I never worked with YARN
>> before). My mapred-site.xml is empty. Do I even need this? if so, what
>> should I set it to?
>>
>>
>>
>> On Wed, Mar 14, 2018 at 2:46 AM, Femi Anthony <femib...@gmail.com> wrote:
>>
>> What's the hardware configuration of the box you're running on i.e. how
>> much memory does it have ?
>>
>>
>>
>> Femi
>>
>>
>>
>> On Wed, Mar 14, 2018 at 5:32 AM, kant kodali <kanth...@gmail.com> wrote:
>>
>> Tried this
>>
>>
>>
>>  ./spark-shell --master yarn --deploy-mode client --executor-memory 4g
>>
>>
>>
>> Same issue. Keeps going forever..
>>
>>
>>
>> 18/03/14 09:31:25 INFO Client:
>>
>> client token: N/A
>>
>> diagnostics: N/A
>>
>> ApplicationMaster host: N/A
>>
>> ApplicationMaster RPC port: -1
>>
>> queue: default
>>
>> start time: 1521019884656
>>
>> final status: UNDEFINED
>>
>> tracking URL: http://ip-172-31-0-54:8088/proxy/application_1521014458020_
>> 0004/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__ip-2D172-2D31-2D0-2D54-3A8088_proxy_application-5F1521014458020-5F0004_=DwMFaQ=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0=Eq5RVldAnerufWd7pgeydUZWtdXr2XJoEncqgUV5McE=>
>>
>> user: centos
>>
>>
>>
>> 18/03/14 09:30:08 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:09 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:10 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:11 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:12 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:13 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:14 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:15 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>>
>>
>> On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony <femib...@gmail.com> wrote:
>>
>> Make sure you have enough memory allocated for Spark workers, try
>> specifying executor memory as follows:
>>
>> --executor-memory 
>>
>> to spark-submit.
>>
>>
>>
>> On Wed, Mar 14, 2018 at 3:25 AM, kant kodali <kanth...@gmail.com> wrote:
>>
>> I am using spark 2.3.0 and hadoop 2.7.3.
>>
>>
>>
>> Also I have done the following and restarted all. But I still
>> see ACCEPTED: waiting for AM container to be allocated, launched and
>> register with RM. And i am unable to spawn spark-shell.
>>
>>
>>
>> editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the
>> following property value from 0.1 to something higher. I changed to 0.5
>> (50%)
>>
>> 
>>
>> ya

Re: How to run spark shell using YARN

2018-03-14 Thread kant kodali 
Hi,

I am not using emr. And yes I restarted several times.

On Wed, Mar 14, 2018 at 6:35 AM, Anthony, Olufemi <
olufemi.anth...@capitalone.com> wrote:

> After you updated your yarn-site.xml  file, did you restart the YARN
> resource manager ?
>
>
>
> https://aws.amazon.com/premiumsupport/knowledge-
> center/restart-service-emr/
>
>
>
> Femi
>
>
>
> *From: *kant kodali <kanth...@gmail.com>
> *Date: *Wednesday, March 14, 2018 at 6:16 AM
> *To: *Femi Anthony <femib...@gmail.com>
> *Cc: *vermanurag <anurag.ve...@fnmathlogic.com>, "user @spark" <
> user@spark.apache.org>
> *Subject: *Re: How to run spark shell using YARN
>
>
>
> 16GB RAM.  AWS m4.xlarge. It's a three node cluster and I only have YARN
> and  HDFS running. Resources are barely used however I believe there is
> something in my config that is preventing YARN to see that I have good
> amount of resources I think (thats my guess I never worked with YARN
> before). My mapred-site.xml is empty. Do I even need this? if so, what
> should I set it to?
>
>
>
> On Wed, Mar 14, 2018 at 2:46 AM, Femi Anthony <femib...@gmail.com> wrote:
>
> What's the hardware configuration of the box you're running on i.e. how
> much memory does it have ?
>
>
>
> Femi
>
>
>
> On Wed, Mar 14, 2018 at 5:32 AM, kant kodali <kanth...@gmail.com> wrote:
>
> Tried this
>
>
>
>  ./spark-shell --master yarn --deploy-mode client --executor-memory 4g
>
>
>
> Same issue. Keeps going forever..
>
>
>
> 18/03/14 09:31:25 INFO Client:
>
> client token: N/A
>
> diagnostics: N/A
>
> ApplicationMaster host: N/A
>
> ApplicationMaster RPC port: -1
>
> queue: default
>
> start time: 1521019884656
>
> final status: UNDEFINED
>
> tracking URL: http://ip-172-31-0-54:8088/proxy/application_
> 1521014458020_0004/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__ip-2D172-2D31-2D0-2D54-3A8088_proxy_application-5F1521014458020-5F0004_=DwMFaQ=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0=Eq5RVldAnerufWd7pgeydUZWtdXr2XJoEncqgUV5McE=>
>
> user: centos
>
>
>
> 18/03/14 09:30:08 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:09 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:10 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:11 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:12 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:13 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:14 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:15 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
>
>
> On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony <femib...@gmail.com> wrote:
>
> Make sure you have enough memory allocated for Spark workers, try
> specifying executor memory as follows:
>
> --executor-memory 
>
> to spark-submit.
>
>
>
> On Wed, Mar 14, 2018 at 3:25 AM, kant kodali <kanth...@gmail.com> wrote:
>
> I am using spark 2.3.0 and hadoop 2.7.3.
>
>
>
> Also I have done the following and restarted all. But I still
> see ACCEPTED: waiting for AM container to be allocated, launched and
> register with RM. And i am unable to spawn spark-shell.
>
>
>
> editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the
> following property value from 0.1 to something higher. I changed to 0.5
> (50%)
>
> 
>
> yarn.scheduler.capacity.maximum-am-resource-percent
>
> 0.5
>
> 
>
> Maximum percent of resources in the cluster which can be used to run 
> application masters i.e. controls number of concurrent running applications.
>
> 
>
> 
>
> You may have to allocate more memory to YARN by editing yarn-site.xml by
> updating the following property:
>
> 
>
> yarn.nodemanager.resource.memory-mb
>
> 8192
>
> 
>
> https://stackoverflow.com/questions/45687607/waiting-
> for-am-container-to-be-allocated-launched-and-register-with-rm
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_45687607_waiting-2Dfor-2Dam-2Dcontainer-2Dto-2Dbe-2Dallocated-2Dlaunched-2Dand-2Dregist

Re: How to run spark shell using YARN

2018-03-14 Thread Anthony, Olufemi 
After you updated your yarn-site.xml  file, did you restart the YARN resource 
manager ?

https://aws.amazon.com/premiumsupport/knowledge-center/restart-service-emr/

Femi

From: kant kodali <kanth...@gmail.com>
Date: Wednesday, March 14, 2018 at 6:16 AM
To: Femi Anthony <femib...@gmail.com>
Cc: vermanurag <anurag.ve...@fnmathlogic.com>, "user @spark" 
<user@spark.apache.org>
Subject: Re: How to run spark shell using YARN

16GB RAM.  AWS m4.xlarge. It's a three node cluster and I only have YARN and  
HDFS running. Resources are barely used however I believe there is something in 
my config that is preventing YARN to see that I have good amount of resources I 
think (thats my guess I never worked with YARN before). My mapred-site.xml is 
empty. Do I even need this? if so, what should I set it to?

On Wed, Mar 14, 2018 at 2:46 AM, Femi Anthony 
<femib...@gmail.com<mailto:femib...@gmail.com>> wrote:
What's the hardware configuration of the box you're running on i.e. how much 
memory does it have ?

Femi

On Wed, Mar 14, 2018 at 5:32 AM, kant kodali 
<kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote:
Tried this


 ./spark-shell --master yarn --deploy-mode client --executor-memory 4g



Same issue. Keeps going forever..



18/03/14 09:31:25 INFO Client:

client token: N/A

diagnostics: N/A

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: default

start time: 1521019884656

final status: UNDEFINED

tracking URL: 
http://ip-172-31-0-54:8088/proxy/application_1521014458020_0004/<https://urldefense.proofpoint.com/v2/url?u=http-3A__ip-2D172-2D31-2D0-2D54-3A8088_proxy_application-5F1521014458020-5F0004_=DwMFaQ=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0=Eq5RVldAnerufWd7pgeydUZWtdXr2XJoEncqgUV5McE=>

user: centos



18/03/14 09:30:08 INFO Client: Application report for 
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:09 INFO Client: Application report for 
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:10 INFO Client: Application report for 
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:11 INFO Client: Application report for 
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:12 INFO Client: Application report for 
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:13 INFO Client: Application report for 
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:14 INFO Client: Application report for 
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:15 INFO Client: Application report for 
application_1521014458020_0003 (state: ACCEPTED)

On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony 
<femib...@gmail.com<mailto:femib...@gmail.com>> wrote:
Make sure you have enough memory allocated for Spark workers, try specifying 
executor memory as follows:

--executor-memory 

to spark-submit.

On Wed, Mar 14, 2018 at 3:25 AM, kant kodali 
<kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote:
I am using spark 2.3.0 and hadoop 2.7.3.

Also I have done the following and restarted all. But I still see ACCEPTED: 
waiting for AM container to be allocated, launched and register with RM. And i 
am unable to spawn spark-shell.


editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the following 
property value from 0.1 to something higher. I changed to 0.5 (50%)



yarn.scheduler.capacity.maximum-am-resource-percent

0.5



Maximum percent of resources in the cluster which can be used to run 
application masters i.e. controls number of concurrent running applications.





You may have to allocate more memory to YARN by editing yarn-site.xml by 
updating the following property:



yarn.nodemanager.resource.memory-mb

8192


https://stackoverflow.com/questions/45687607/waiting-for-am-container-to-be-allocated-launched-and-register-with-rm<https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_45687607_waiting-2Dfor-2Dam-2Dcontainer-2Dto-2Dbe-2Dallocated-2Dlaunched-2Dand-2Dregister-2Dwith-2Drm=DwMFaQ=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0=i8R5_ASmKyL_OccyAC0AtMDz7VWncp0UO27XuXBnfXs=>



On Wed, Mar 14, 2018 at 12:12 AM, kant kodali 
<kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote:
any idea?

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali 
<kanth...@gmail.com<mailto:kanth...@gmail.com>> wrote:
I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this 
website<https://urldefense.proofpoint.com/v2/url?u=https-3A__dwbi.org_etl_bigdata_183-2Dsetup-2Dhadoop-2Dcluster=DwMFaQ=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa=oOFBWIVhH_T4NwkrNL0

Re: How to run spark shell using YARN

2018-03-14 Thread kant kodali 
16GB RAM.  AWS m4.xlarge. It's a three node cluster and I only have YARN
and  HDFS running. Resources are barely used however I believe there is
something in my config that is preventing YARN to see that I have good
amount of resources I think (thats my guess I never worked with YARN
before). My mapred-site.xml is empty. Do I even need this? if so, what
should I set it to?

On Wed, Mar 14, 2018 at 2:46 AM, Femi Anthony  wrote:

> What's the hardware configuration of the box you're running on i.e. how
> much memory does it have ?
>
> Femi
>
> On Wed, Mar 14, 2018 at 5:32 AM, kant kodali  wrote:
>
>> Tried this
>>
>>  ./spark-shell --master yarn --deploy-mode client --executor-memory 4g
>>
>>
>> Same issue. Keeps going forever..
>>
>>
>> 18/03/14 09:31:25 INFO Client:
>>
>> client token: N/A
>>
>> diagnostics: N/A
>>
>> ApplicationMaster host: N/A
>>
>> ApplicationMaster RPC port: -1
>>
>> queue: default
>>
>> start time: 1521019884656
>>
>> final status: UNDEFINED
>>
>> tracking URL: http://ip-172-31-0-54:8088/proxy/application_1521014458020_
>> 0004/
>>
>> user: centos
>>
>>
>> 18/03/14 09:30:08 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:09 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:10 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:11 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:12 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:13 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:14 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:15 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony  wrote:
>>
>>> Make sure you have enough memory allocated for Spark workers, try
>>> specifying executor memory as follows:
>>>
>>> --executor-memory 
>>>
>>> to spark-submit.
>>>
>>> On Wed, Mar 14, 2018 at 3:25 AM, kant kodali  wrote:
>>>
 I am using spark 2.3.0 and hadoop 2.7.3.

 Also I have done the following and restarted all. But I still
 see ACCEPTED: waiting for AM container to be allocated, launched and
 register with RM. And i am unable to spawn spark-shell.

 editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the
 following property value from 0.1 to something higher. I changed to 0.5
 (50%)

 
 yarn.scheduler.capacity.maximum-am-resource-percent
 0.5
 
 Maximum percent of resources in the cluster which can be used to 
 run application masters i.e. controls number of concurrent running 
 applications.
 
 

 You may have to allocate more memory to YARN by editing yarn-site.xml
 by updating the following property:

 
 yarn.nodemanager.resource.memory-mb
 8192
 

 https://stackoverflow.com/questions/45687607/waiting-for-am-
 container-to-be-allocated-launched-and-register-with-rm



 On Wed, Mar 14, 2018 at 12:12 AM, kant kodali 
 wrote:

> any idea?
>
> On Wed, Mar 14, 2018 at 12:12 AM, kant kodali 
> wrote:
>
>> I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this
>> website  and
>> these are the only three files I changed Do I need to set or change
>> anything in mapred-site.xml (As of now I have not touched 
>> mapred-site.xml)?
>>
>> when I do yarn -node -list -all I can see both node manager and
>> resource managers are running fine.
>>
>> But when I run spark-shell --master yarn --deploy-mode client
>>
>>
>> it just keeps looping forever and never stops with the following
>> messages
>>
>> 18/03/14 07:07:47 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>> 18/03/14 07:07:48 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>> 18/03/14 07:07:49 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>> 18/03/14 07:07:50 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>> 18/03/14 07:07:51 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>> 18/03/14 07:07:52 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>>
>> when I go to RM UI I see this
>>
>> ACCEPTED:

Re: How to run spark shell using YARN

2018-03-14 Thread Femi Anthony 
What's the hardware configuration of the box you're running on i.e. how
much memory does it have ?

Femi

On Wed, Mar 14, 2018 at 5:32 AM, kant kodali  wrote:

> Tried this
>
>  ./spark-shell --master yarn --deploy-mode client --executor-memory 4g
>
>
> Same issue. Keeps going forever..
>
>
> 18/03/14 09:31:25 INFO Client:
>
> client token: N/A
>
> diagnostics: N/A
>
> ApplicationMaster host: N/A
>
> ApplicationMaster RPC port: -1
>
> queue: default
>
> start time: 1521019884656
>
> final status: UNDEFINED
>
> tracking URL: http://ip-172-31-0-54:8088/proxy/application_
> 1521014458020_0004/
>
> user: centos
>
>
> 18/03/14 09:30:08 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:09 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:10 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:11 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:12 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:13 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:14 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> 18/03/14 09:30:15 INFO Client: Application report for
> application_1521014458020_0003 (state: ACCEPTED)
>
> On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony  wrote:
>
>> Make sure you have enough memory allocated for Spark workers, try
>> specifying executor memory as follows:
>>
>> --executor-memory 
>>
>> to spark-submit.
>>
>> On Wed, Mar 14, 2018 at 3:25 AM, kant kodali  wrote:
>>
>>> I am using spark 2.3.0 and hadoop 2.7.3.
>>>
>>> Also I have done the following and restarted all. But I still
>>> see ACCEPTED: waiting for AM container to be allocated, launched and
>>> register with RM. And i am unable to spawn spark-shell.
>>>
>>> editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the
>>> following property value from 0.1 to something higher. I changed to 0.5
>>> (50%)
>>>
>>> 
>>> yarn.scheduler.capacity.maximum-am-resource-percent
>>> 0.5
>>> 
>>> Maximum percent of resources in the cluster which can be used to 
>>> run application masters i.e. controls number of concurrent running 
>>> applications.
>>> 
>>> 
>>>
>>> You may have to allocate more memory to YARN by editing yarn-site.xml by
>>> updating the following property:
>>>
>>> 
>>> yarn.nodemanager.resource.memory-mb
>>> 8192
>>> 
>>>
>>> https://stackoverflow.com/questions/45687607/waiting-for-am-
>>> container-to-be-allocated-launched-and-register-with-rm
>>>
>>>
>>>
>>> On Wed, Mar 14, 2018 at 12:12 AM, kant kodali 
>>> wrote:
>>>
 any idea?

 On Wed, Mar 14, 2018 at 12:12 AM, kant kodali 
 wrote:

> I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website
>  and these are
> the only three files I changed Do I need to set or change anything in
> mapred-site.xml (As of now I have not touched mapred-site.xml)?
>
> when I do yarn -node -list -all I can see both node manager and
> resource managers are running fine.
>
> But when I run spark-shell --master yarn --deploy-mode client
>
>
> it just keeps looping forever and never stops with the following
> messages
>
> 18/03/14 07:07:47 INFO Client: Application report for
> application_1521011212550_0001 (state: ACCEPTED)
> 18/03/14 07:07:48 INFO Client: Application report for
> application_1521011212550_0001 (state: ACCEPTED)
> 18/03/14 07:07:49 INFO Client: Application report for
> application_1521011212550_0001 (state: ACCEPTED)
> 18/03/14 07:07:50 INFO Client: Application report for
> application_1521011212550_0001 (state: ACCEPTED)
> 18/03/14 07:07:51 INFO Client: Application report for
> application_1521011212550_0001 (state: ACCEPTED)
> 18/03/14 07:07:52 INFO Client: Application report for
> application_1521011212550_0001 (state: ACCEPTED)
>
> when I go to RM UI I see this
>
> ACCEPTED: waiting for AM container to be allocated, launched and
> register with RM.
>
>
>
>
> On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <
> anurag.ve...@fnmathlogic.com> wrote:
>
>> This does not look like Spark error. Looks like yarn has not been
>> able to
>> allocate resources for spark driver. If you check resource manager UI
>> you
>> are likely to see this as spark application waiting for resources. Try
>> reducing the driver node memory and/ or other bottlenecks based on
>> what you
>> see in the resource manager UI.
>>
>>
>>

Re: How to run spark shell using YARN

2018-03-14 Thread kant kodali 
Tried this

 ./spark-shell --master yarn --deploy-mode client --executor-memory 4g


Same issue. Keeps going forever..


18/03/14 09:31:25 INFO Client:

client token: N/A

diagnostics: N/A

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: default

start time: 1521019884656

final status: UNDEFINED

tracking URL:
http://ip-172-31-0-54:8088/proxy/application_1521014458020_0004/

user: centos


18/03/14 09:30:08 INFO Client: Application report for
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:09 INFO Client: Application report for
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:10 INFO Client: Application report for
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:11 INFO Client: Application report for
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:12 INFO Client: Application report for
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:13 INFO Client: Application report for
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:14 INFO Client: Application report for
application_1521014458020_0003 (state: ACCEPTED)

18/03/14 09:30:15 INFO Client: Application report for
application_1521014458020_0003 (state: ACCEPTED)

On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony  wrote:

> Make sure you have enough memory allocated for Spark workers, try
> specifying executor memory as follows:
>
> --executor-memory 
>
> to spark-submit.
>
> On Wed, Mar 14, 2018 at 3:25 AM, kant kodali  wrote:
>
>> I am using spark 2.3.0 and hadoop 2.7.3.
>>
>> Also I have done the following and restarted all. But I still
>> see ACCEPTED: waiting for AM container to be allocated, launched and
>> register with RM. And i am unable to spawn spark-shell.
>>
>> editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the
>> following property value from 0.1 to something higher. I changed to 0.5
>> (50%)
>>
>> 
>> yarn.scheduler.capacity.maximum-am-resource-percent
>> 0.5
>> 
>> Maximum percent of resources in the cluster which can be used to run 
>> application masters i.e. controls number of concurrent running applications.
>> 
>> 
>>
>> You may have to allocate more memory to YARN by editing yarn-site.xml by
>> updating the following property:
>>
>> 
>> yarn.nodemanager.resource.memory-mb
>> 8192
>> 
>>
>> https://stackoverflow.com/questions/45687607/waiting-for-am-
>> container-to-be-allocated-launched-and-register-with-rm
>>
>>
>>
>> On Wed, Mar 14, 2018 at 12:12 AM, kant kodali  wrote:
>>
>>> any idea?
>>>
>>> On Wed, Mar 14, 2018 at 12:12 AM, kant kodali 
>>> wrote:
>>>
 I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website
  and these are
 the only three files I changed Do I need to set or change anything in
 mapred-site.xml (As of now I have not touched mapred-site.xml)?

 when I do yarn -node -list -all I can see both node manager and
 resource managers are running fine.

 But when I run spark-shell --master yarn --deploy-mode client


 it just keeps looping forever and never stops with the following
 messages

 18/03/14 07:07:47 INFO Client: Application report for
 application_1521011212550_0001 (state: ACCEPTED)
 18/03/14 07:07:48 INFO Client: Application report for
 application_1521011212550_0001 (state: ACCEPTED)
 18/03/14 07:07:49 INFO Client: Application report for
 application_1521011212550_0001 (state: ACCEPTED)
 18/03/14 07:07:50 INFO Client: Application report for
 application_1521011212550_0001 (state: ACCEPTED)
 18/03/14 07:07:51 INFO Client: Application report for
 application_1521011212550_0001 (state: ACCEPTED)
 18/03/14 07:07:52 INFO Client: Application report for
 application_1521011212550_0001 (state: ACCEPTED)

 when I go to RM UI I see this

 ACCEPTED: waiting for AM container to be allocated, launched and
 register with RM.




 On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <
 anurag.ve...@fnmathlogic.com> wrote:

> This does not look like Spark error. Looks like yarn has not been able
> to
> allocate resources for spark driver. If you check resource manager UI
> you
> are likely to see this as spark application waiting for resources. Try
> reducing the driver node memory and/ or other bottlenecks based on
> what you
> see in the resource manager UI.
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

>>>
>>
>
>
> --
> http://www.femibyte.com/twiki5/bin/view/Tech/
> http://www.nextmatrix.com
> "Great spirits have always encountered

Re: How to run spark shell using YARN

2018-03-14 Thread Femi Anthony 
Make sure you have enough memory allocated for Spark workers, try
specifying executor memory as follows:

--executor-memory 

to spark-submit.

On Wed, Mar 14, 2018 at 3:25 AM, kant kodali  wrote:

> I am using spark 2.3.0 and hadoop 2.7.3.
>
> Also I have done the following and restarted all. But I still
> see ACCEPTED: waiting for AM container to be allocated, launched and
> register with RM. And i am unable to spawn spark-shell.
>
> editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the
> following property value from 0.1 to something higher. I changed to 0.5
> (50%)
>
> 
> yarn.scheduler.capacity.maximum-am-resource-percent
> 0.5
> 
> Maximum percent of resources in the cluster which can be used to run 
> application masters i.e. controls number of concurrent running applications.
> 
> 
>
> You may have to allocate more memory to YARN by editing yarn-site.xml by
> updating the following property:
>
> 
> yarn.nodemanager.resource.memory-mb
> 8192
> 
>
> https://stackoverflow.com/questions/45687607/waiting-
> for-am-container-to-be-allocated-launched-and-register-with-rm
>
>
>
> On Wed, Mar 14, 2018 at 12:12 AM, kant kodali  wrote:
>
>> any idea?
>>
>> On Wed, Mar 14, 2018 at 12:12 AM, kant kodali  wrote:
>>
>>> I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website
>>>  and these are
>>> the only three files I changed Do I need to set or change anything in
>>> mapred-site.xml (As of now I have not touched mapred-site.xml)?
>>>
>>> when I do yarn -node -list -all I can see both node manager and resource
>>> managers are running fine.
>>>
>>> But when I run spark-shell --master yarn --deploy-mode client
>>>
>>>
>>> it just keeps looping forever and never stops with the following messages
>>>
>>> 18/03/14 07:07:47 INFO Client: Application report for
>>> application_1521011212550_0001 (state: ACCEPTED)
>>> 18/03/14 07:07:48 INFO Client: Application report for
>>> application_1521011212550_0001 (state: ACCEPTED)
>>> 18/03/14 07:07:49 INFO Client: Application report for
>>> application_1521011212550_0001 (state: ACCEPTED)
>>> 18/03/14 07:07:50 INFO Client: Application report for
>>> application_1521011212550_0001 (state: ACCEPTED)
>>> 18/03/14 07:07:51 INFO Client: Application report for
>>> application_1521011212550_0001 (state: ACCEPTED)
>>> 18/03/14 07:07:52 INFO Client: Application report for
>>> application_1521011212550_0001 (state: ACCEPTED)
>>>
>>> when I go to RM UI I see this
>>>
>>> ACCEPTED: waiting for AM container to be allocated, launched and
>>> register with RM.
>>>
>>>
>>>
>>>
>>> On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <
>>> anurag.ve...@fnmathlogic.com> wrote:
>>>
 This does not look like Spark error. Looks like yarn has not been able
 to
 allocate resources for spark driver. If you check resource manager UI
 you
 are likely to see this as spark application waiting for resources. Try
 reducing the driver node memory and/ or other bottlenecks based on what
 you
 see in the resource manager UI.



 --
 Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

 -
 To unsubscribe e-mail: user-unsubscr...@spark.apache.org


>>>
>>
>


-- 
http://www.femibyte.com/twiki5/bin/view/Tech/
http://www.nextmatrix.com
"Great spirits have always encountered violent opposition from mediocre
minds." - Albert Einstein.

Re: How to run spark shell using YARN

2018-03-14 Thread kant kodali 
I am using spark 2.3.0 and hadoop 2.7.3.

Also I have done the following and restarted all. But I still see ACCEPTED:
waiting for AM container to be allocated, launched and register with RM.
And i am unable to spawn spark-shell.

editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the
following property value from 0.1 to something higher. I changed to 0.5
(50%)


yarn.scheduler.capacity.maximum-am-resource-percent
0.5

Maximum percent of resources in the cluster which can be used
to run application masters i.e. controls number of concurrent running
applications.



You may have to allocate more memory to YARN by editing yarn-site.xml by
updating the following property:


yarn.nodemanager.resource.memory-mb
8192


https://stackoverflow.com/questions/45687607/waiting-for-am-container-to-be-allocated-launched-and-register-with-rm



On Wed, Mar 14, 2018 at 12:12 AM, kant kodali  wrote:

> any idea?
>
> On Wed, Mar 14, 2018 at 12:12 AM, kant kodali  wrote:
>
>> I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website
>>  and these are
>> the only three files I changed Do I need to set or change anything in
>> mapred-site.xml (As of now I have not touched mapred-site.xml)?
>>
>> when I do yarn -node -list -all I can see both node manager and resource
>> managers are running fine.
>>
>> But when I run spark-shell --master yarn --deploy-mode client
>>
>>
>> it just keeps looping forever and never stops with the following messages
>>
>> 18/03/14 07:07:47 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>> 18/03/14 07:07:48 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>> 18/03/14 07:07:49 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>> 18/03/14 07:07:50 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>> 18/03/14 07:07:51 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>> 18/03/14 07:07:52 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>>
>> when I go to RM UI I see this
>>
>> ACCEPTED: waiting for AM container to be allocated, launched and register
>> with RM.
>>
>>
>>
>>
>> On Mon, Mar 12, 2018 at 7:16 PM, vermanurag > > wrote:
>>
>>> This does not look like Spark error. Looks like yarn has not been able to
>>> allocate resources for spark driver. If you check resource manager UI you
>>> are likely to see this as spark application waiting for resources. Try
>>> reducing the driver node memory and/ or other bottlenecks based on what
>>> you
>>> see in the resource manager UI.
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>>>
>>> -
>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>>
>>>
>>
>

Re: How to run spark shell using YARN

2018-03-14 Thread kant kodali 
any idea?

On Wed, Mar 14, 2018 at 12:12 AM, kant kodali  wrote:

> I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website
>  and these are the
> only three files I changed Do I need to set or change anything in
> mapred-site.xml (As of now I have not touched mapred-site.xml)?
>
> when I do yarn -node -list -all I can see both node manager and resource
> managers are running fine.
>
> But when I run spark-shell --master yarn --deploy-mode client
>
>
> it just keeps looping forever and never stops with the following messages
>
> 18/03/14 07:07:47 INFO Client: Application report for
> application_1521011212550_0001 (state: ACCEPTED)
> 18/03/14 07:07:48 INFO Client: Application report for
> application_1521011212550_0001 (state: ACCEPTED)
> 18/03/14 07:07:49 INFO Client: Application report for
> application_1521011212550_0001 (state: ACCEPTED)
> 18/03/14 07:07:50 INFO Client: Application report for
> application_1521011212550_0001 (state: ACCEPTED)
> 18/03/14 07:07:51 INFO Client: Application report for
> application_1521011212550_0001 (state: ACCEPTED)
> 18/03/14 07:07:52 INFO Client: Application report for
> application_1521011212550_0001 (state: ACCEPTED)
>
> when I go to RM UI I see this
>
> ACCEPTED: waiting for AM container to be allocated, launched and register
> with RM.
>
>
>
>
> On Mon, Mar 12, 2018 at 7:16 PM, vermanurag 
> wrote:
>
>> This does not look like Spark error. Looks like yarn has not been able to
>> allocate resources for spark driver. If you check resource manager UI you
>> are likely to see this as spark application waiting for resources. Try
>> reducing the driver node memory and/ or other bottlenecks based on what
>> you
>> see in the resource manager UI.
>>
>>
>>
>> --
>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>>
>> -
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>
>>
>

Re: How to run spark shell using YARN

2018-03-14 Thread kant kodali 
I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website
 and these are the
only three files I changed Do I need to set or change anything in
mapred-site.xml (As of now I have not touched mapred-site.xml)?

when I do yarn -node -list -all I can see both node manager and resource
managers are running fine.

But when I run spark-shell --master yarn --deploy-mode client


it just keeps looping forever and never stops with the following messages

18/03/14 07:07:47 INFO Client: Application report for
application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:48 INFO Client: Application report for
application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:49 INFO Client: Application report for
application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:50 INFO Client: Application report for
application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:51 INFO Client: Application report for
application_1521011212550_0001 (state: ACCEPTED)
18/03/14 07:07:52 INFO Client: Application report for
application_1521011212550_0001 (state: ACCEPTED)

when I go to RM UI I see this

ACCEPTED: waiting for AM container to be allocated, launched and register
with RM.




On Mon, Mar 12, 2018 at 7:16 PM, vermanurag 
wrote:

> This does not look like Spark error. Looks like yarn has not been able to
> allocate resources for spark driver. If you check resource manager UI you
> are likely to see this as spark application waiting for resources. Try
> reducing the driver node memory and/ or other bottlenecks based on what you
> see in the resource manager UI.
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

Re: How to run spark shell using YARN

2018-03-12 Thread vermanurag 
This does not look like Spark error. Looks like yarn has not been able to
allocate resources for spark driver. If you check resource manager UI you
are likely to see this as spark application waiting for resources. Try
reducing the driver node memory and/ or other bottlenecks based on what you
see in the resource manager UI. 



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: How to run spark shell using YARN

2018-03-12 Thread Marcelo Vanzin 
Looks like you either have a misconfigured HDFS service, or you're
using the wrong configuration on the client.

BTW, as I said in the previous response, the message you saw initially
is *not* an error. If you're just trying things out, you don't need to
do anything and Spark should still work.

On Mon, Mar 12, 2018 at 6:13 PM, kant kodali  wrote:
> Hi,
>
> I read that doc several times now. I am stuck with the below error message
> when I run ./spark-shell --master yarn --deploy-mode client.
>
> I have my HADOOP_CONF_DIR set to /usr/local/hadoop-2.7.3/etc/hadoop and
> SPARK_HOME set to /usr/local/spark on all 3 machines (1 node for Resource
> Manager and NameNode, 2 Nodes for Node Manager and DataNodes).
>
> Any idea?
>
>
>
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /user/centos/.sparkStaging/application_1520898664848_0003/__spark_libs__2434167523839846774.zip
> could only be replicated to 0 nodes instead of minReplication (=1).  There
> are 2 datanode(s) running and no node(s) are excluded in this operation.
>
>
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> java.security.AccessController.doPrivileged(Native Method)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> javax.security.auth.Subject.doAs(Subject.java:422)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> 18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> 18/03/13
>
>
> Thanks!
>
>
> On Mon, Mar 12, 2018 at 4:46 PM, Marcelo Vanzin  wrote:
>>
>> That's not an error, just a warning. The docs [1] have more info about
>> the config options mentioned in that message.
>>
>> [1] http://spark.apache.org/docs/latest/running-on-yarn.html
>>
>> On Mon, Mar 12, 2018 at 4:42 PM, kant kodali  wrote:
>> > Hi All,
>> >
>> > I am trying to use YARN for the very first time. I believe I configured
>> > all
>> > the resource manager and name node fine. And then I run the below
>> > command
>> >
>> > ./spark-shell --master yarn --deploy-mode client
>> >
>> > I get the below output and it hangs there forever (I had been waiting
>> > over
>> > 10 minutes)
>> >
>> > 18/03/12 23:36:32 WARN Client: Neither spark.yarn.jars nor
>> > spark.yarn.archive is set, falling back to uploading libraries under
>> > SPARK_HOME.
>> >
>> > Any idea?
>> >
>> > Thanks!
>>
>>
>>
>> --
>> Marcelo
>
>



-- 
Marcelo

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: How to run spark shell using YARN

2018-03-12 Thread kant kodali 
Hi,

I read that doc several times now. I am stuck with the below error message
when I run ./spark-shell --master yarn --deploy-mode client.

I have my HADOOP_CONF_DIR set to /usr/local/hadoop-2.7.3/etc/hadoop and
SPARK_HOME set to /usr/local/spark on all 3 machines (1 node for Resource
Manager and NameNode, 2 Nodes for Node Manager and DataNodes).

Any idea?



*18/03/13 00:19:13 INFO LineBufferedStream: stdout:
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/user/centos/.sparkStaging/application_1520898664848_0003/__spark_libs__2434167523839846774.zip
could only be replicated to 0 nodes instead of minReplication (=1).  There
are 2 datanode(s) running and no node(s) are excluded in this operation.*


18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
java.security.AccessController.doPrivileged(Native Method)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
javax.security.auth.Subject.doAs(Subject.java:422)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
18/03/13 00:19:13 INFO LineBufferedStream: stdout:  at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
18/03/13


Thanks!


On Mon, Mar 12, 2018 at 4:46 PM, Marcelo Vanzin  wrote:

> That's not an error, just a warning. The docs [1] have more info about
> the config options mentioned in that message.
>
> [1] http://spark.apache.org/docs/latest/running-on-yarn.html
>
> On Mon, Mar 12, 2018 at 4:42 PM, kant kodali  wrote:
> > Hi All,
> >
> > I am trying to use YARN for the very first time. I believe I configured
> all
> > the resource manager and name node fine. And then I run the below command
> >
> > ./spark-shell --master yarn --deploy-mode client
> >
> > I get the below output and it hangs there forever (I had been waiting
> over
> > 10 minutes)
> >
> > 18/03/12 23:36:32 WARN Client: Neither spark.yarn.jars nor
> > spark.yarn.archive is set, falling back to uploading libraries under
> > SPARK_HOME.
> >
> > Any idea?
> >
> > Thanks!
>
>
>
> --
> Marcelo
>

Re: How to run spark shell using YARN

2018-03-12 Thread Marcelo Vanzin 
That's not an error, just a warning. The docs [1] have more info about
the config options mentioned in that message.

[1] http://spark.apache.org/docs/latest/running-on-yarn.html

On Mon, Mar 12, 2018 at 4:42 PM, kant kodali  wrote:
> Hi All,
>
> I am trying to use YARN for the very first time. I believe I configured all
> the resource manager and name node fine. And then I run the below command
>
> ./spark-shell --master yarn --deploy-mode client
>
> I get the below output and it hangs there forever (I had been waiting over
> 10 minutes)
>
> 18/03/12 23:36:32 WARN Client: Neither spark.yarn.jars nor
> spark.yarn.archive is set, falling back to uploading libraries under
> SPARK_HOME.
>
> Any idea?
>
> Thanks!



-- 
Marcelo

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

How to run spark shell using YARN

2018-03-12 Thread kant kodali 
Hi All,

I am trying to use YARN for the very first time. I believe I configured all
the resource manager and name node fine. And then I run the below command

./spark-shell --master yarn --deploy-mode client

*I get the below output and it hangs there forever *(I had been waiting
over 10 minutes)

18/03/12 23:36:32 WARN Client: Neither spark.yarn.jars nor
spark.yarn.archive is set, falling back to uploading libraries under
SPARK_HOME.

Any idea?

Thanks!