Rack locality

2015-04-13 Thread rcharaya
I want to use Rack locality feature of Apache Spark in my application.

Is YARN the only resource manager which supports it as of now?

After going through the source code, I found that default implementation of
getRackForHost() method returns NONE in TaskSchedulerImpl which (I suppose)
would be used by standalone mode.

On the other hand, it is overriden in YarnScheduler.scala to fetch the rack
information by invoking RackResolver api of hadoop which would be used when
its run on YARN.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Rack-locality-tp22483.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org



Re: Rack locality

2015-04-13 Thread Sandy Ryza
Hi Riya,

As far as I know, that is correct, unless Mesos fine-grained mode handles
this in some mysterious way.

-Sandy

On Mon, Apr 13, 2015 at 2:09 PM, rcharaya riya.char...@gmail.com wrote:

 I want to use Rack locality feature of Apache Spark in my application.

 Is YARN the only resource manager which supports it as of now?

 After going through the source code, I found that default implementation of
 getRackForHost() method returns NONE in TaskSchedulerImpl which (I suppose)
 would be used by standalone mode.

 On the other hand, it is overriden in YarnScheduler.scala to fetch the rack
 information by invoking RackResolver api of hadoop which would be used when
 its run on YARN.



 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/Rack-locality-tp22483.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org