Re: sqoop import job not working when spark thrift server is running.

[akshay naidu]

Thanks Jörn,

Fairscheduler is already enabled in yarn-site.xml

yarn.resourcemanager.scheduler.class -
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

yarn.scheduler.fair.allow-undeclared-pools -
true

yarn.scheduler.fair.user-as-default-queue
true

yarn.scheduler.fair.preemption
true

yarn.scheduler.fair.preemption.cluster-utilization-threshold
0.8

On Sat, Feb 24, 2018 at 6:26 PM, Jörn Franke  wrote:

> Fairscheduler in yarn provides you the possibility to use more resources
> than configured if they are available
>
> On 24. Feb 2018, at 13:47, akshay naidu  wrote:
>
> it sure is not able to get sufficient resources from YARN to start the
>> containers.
>>
> that's right. I worked when I reduced executors from thrift but it also
> reduced thrift's performance.
>
> But it is not the solution i am looking forward to. my sqoop import job
> runs just once a day, and thrift apps will running for 24/7 for
> fetching-processing-displaying online reports on website. reducing
> executors and keeping some in spare is helping in running more jobs other
> than thrift parallely but it's wasting the core when other jobs are not
> working.
>
> is there something which can help in allocating resources dynamically?
> which will automatically allocate maximum resources to thrift when there
> are no other jobs running, and automatically share resources with jobs/apps
> other than thrift.?
>
> I've heard of property in yarn - dynamicAlloction , can this help?
>
>
> Thanks.
>
> On Sat, Feb 24, 2018 at 7:14 AM, vijay.bvp  wrote:
>
>> it sure is not able to get sufficient resources from YARN to start the
>> containers.
>> is it only with this import job or if you submit any other job its failing
>> to start.
>>
>> As a test just try to run another spark job or a mapredue job  and see if
>> the job can be started.
>>
>> Reduce the thrift server executors and see overall there is available
>> cluster capacity for new jobs.
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>>
>> -
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>
>>
>

Re: sqoop import job not working when spark thrift server is running.

[Jörn Franke]

Fairscheduler in yarn provides you the possibility to use more resources than 
configured if they are available 

On 24. Feb 2018, at 13:47, akshay naidu  wrote:

>> it sure is not able to get sufficient resources from YARN to start the
>> containers. 
> that's right. I worked when I reduced executors from thrift but it also 
> reduced thrift's performance. 
> 
> But it is not the solution i am looking forward to. my sqoop import job runs 
> just once a day, and thrift apps will running for 24/7 for 
> fetching-processing-displaying online reports on website. reducing executors 
> and keeping some in spare is helping in running more jobs other than thrift 
> parallely but it's wasting the core when other jobs are not working.
> 
> is there something which can help in allocating resources dynamically?
> which will automatically allocate maximum resources to thrift when there are 
> no other jobs running, and automatically share resources with jobs/apps other 
> than thrift.?
> 
> I've heard of property in yarn - dynamicAlloction , can this help?
> 
> 
> Thanks.
> 
>> On Sat, Feb 24, 2018 at 7:14 AM, vijay.bvp  wrote:
>> it sure is not able to get sufficient resources from YARN to start the
>> containers.
>> is it only with this import job or if you submit any other job its failing
>> to start.
>> 
>> As a test just try to run another spark job or a mapredue job  and see if
>> the job can be started.
>> 
>> Reduce the thrift server executors and see overall there is available
>> cluster capacity for new jobs.
>> 
>> 
>> 
>> 
>> 
>> --
>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>> 
>> -
>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>> 
> 

Re: sqoop import job not working when spark thrift server is running.

[akshay naidu]

>
> it sure is not able to get sufficient resources from YARN to start the
> containers.
>
that's right. I worked when I reduced executors from thrift but it also
reduced thrift's performance.

But it is not the solution i am looking forward to. my sqoop import job
runs just once a day, and thrift apps will running for 24/7 for
fetching-processing-displaying online reports on website. reducing
executors and keeping some in spare is helping in running more jobs other
than thrift parallely but it's wasting the core when other jobs are not
working.

is there something which can help in allocating resources dynamically?
which will automatically allocate maximum resources to thrift when there
are no other jobs running, and automatically share resources with jobs/apps
other than thrift.?

I've heard of property in yarn - dynamicAlloction , can this help?


Thanks.

On Sat, Feb 24, 2018 at 7:14 AM, vijay.bvp  wrote:

> it sure is not able to get sufficient resources from YARN to start the
> containers.
> is it only with this import job or if you submit any other job its failing
> to start.
>
> As a test just try to run another spark job or a mapredue job  and see if
> the job can be started.
>
> Reduce the thrift server executors and see overall there is available
> cluster capacity for new jobs.
>
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

Re: sqoop import job not working when spark thrift server is running.

[vijay.bvp]

it sure is not able to get sufficient resources from YARN to start the
containers.
is it only with this import job or if you submit any other job its failing
to start.

As a test just try to run another spark job or a mapredue job  and see if
the job can be started.

Reduce the thrift server executors and see overall there is available
cluster capacity for new jobs.





--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: sqoop import job not working when spark thrift server is running.

[akshay naidu]

hello vijay,
appreciate your reply.

  what was the error when you are trying to run mapreduce import job when
> the
> thrift server is running.


it didnt throw any error, it just gets stuck at
INFO mapreduce.Job: Running job: job_151911053

and resumes the moment i kill Thrift .

thanks

On Tue, Feb 20, 2018 at 1:48 PM, vijay.bvp  wrote:

> what was the error when you are trying to run mapreduce import job when the
> thrift server is running.
> this is only config changed? what was the config before...
> also share the spark thrift server job config such as no of executors,
> cores
> memory etc.
>
> My guess is your mapreduce job is unable to get sufficient resources,
> container couldn't be launched and so failing to start, this could either
> because of non availability sufficient cores or RAM
>
> 9 worker nodes 12GB RAM each with 6 cores (max allowed cores 4 per
> container)
> you have to keep some room for operation system and other daemons.
>
> if thrift server is setup to have 11 executors with 3 cores each = 33 cores
> for workers and 1 for driver so 34 cores required for spark job and rest
> for
> any other jobs.
>
> spark driver and worker memory is ~9GB
> with 9 12 GB RAM worker nodes not sure how much you can allocate.
>
> thanks
> Vijay
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>

Re: sqoop import job not working when spark thrift server is running.

[vijay.bvp]

what was the error when you are trying to run mapreduce import job when the
thrift server is running.
this is only config changed? what was the config before... 
also share the spark thrift server job config such as no of executors, cores
memory etc.

My guess is your mapreduce job is unable to get sufficient resources,
container couldn't be launched and so failing to start, this could either
because of non availability sufficient cores or RAM

9 worker nodes 12GB RAM each with 6 cores (max allowed cores 4 per
container)
you have to keep some room for operation system and other daemons. 

if thrift server is setup to have 11 executors with 3 cores each = 33 cores
for workers and 1 for driver so 34 cores required for spark job and rest for
any other jobs. 

spark driver and worker memory is ~9GB 
with 9 12 GB RAM worker nodes not sure how much you can allocate.

thanks
Vijay



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org