[ https://issues.apache.org/jira/browse/SPARK-10149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yun Zhao updated SPARK-10149: ----------------------------- Description: Locality Level is ANY on "Details for Stage" WebUI page When a sc.textFile(XX) program is running, Locality Level is ANY which should be NODE_LOCAL on stage 0. org.apache.spark.scheduler.TaskSetManager {quote} // Check for node-local tasks if (TaskLocality.isAllowed(locality, TaskLocality.NODE_LOCAL)) { for (index <- speculatableTasks if canRunOnHost(index)) { val locations = tasks(index).preferredLocations.map(_.host) if (locations.contains(host)) { speculatableTasks -= index return Some((index, TaskLocality.NODE_LOCAL)) } } } {quote} The variable "locations" is hostname of HDFS split, which is from InetAddress.getHostName. The variable "host" is ip of Executor, which is from InetAddress.getLocalHost.getHostAddress. org.apache.spark.deploy.worker.WorkerArguments {quote} var host = Utils.localHostName() {quote} was: Locality Level is ANY on "Details for Stage" WebUI page When a sc.textFile(XX) program is running, Locality Level is ANY which should be NODE_LOCAL on stage 0. org.apache.spark.scheduler.TaskSetManager // Check for node-local tasks if (TaskLocality.isAllowed(locality, TaskLocality.NODE_LOCAL)) { for (index <- speculatableTasks if canRunOnHost(index)) { val locations = tasks(index).preferredLocations.map(_.host) if (locations.contains(host)) { speculatableTasks -= index return Some((index, TaskLocality.NODE_LOCAL)) } } } The variable "locations" is hostname of HDFS split, which is from InetAddress.getHostName. The variable "host" is ip of Executor, which is from InetAddress.getLocalHost.getHostAddress. org.apache.spark.deploy.worker.WorkerArguments var host = Utils.localHostName() > Locality Level is ANY on "Details for Stage" WebUI page > ------------------------------------------------------- > > Key: SPARK-10149 > URL: https://issues.apache.org/jira/browse/SPARK-10149 > Project: Spark > Issue Type: Bug > Components: Spark Core, Web UI > Affects Versions: 1.4.1 > Reporter: Yun Zhao > > Locality Level is ANY on "Details for Stage" WebUI page > When a sc.textFile(XX) program is running, Locality Level is ANY which should > be NODE_LOCAL on stage 0. > org.apache.spark.scheduler.TaskSetManager > {quote} > // Check for node-local tasks > if (TaskLocality.isAllowed(locality, TaskLocality.NODE_LOCAL)) { > for (index <- speculatableTasks if canRunOnHost(index)) { > val locations = tasks(index).preferredLocations.map(_.host) > if (locations.contains(host)) { > speculatableTasks -= index > return Some((index, TaskLocality.NODE_LOCAL)) > } > } > } > {quote} > The variable "locations" is hostname of HDFS split, which is from > InetAddress.getHostName. > The variable "host" is ip of Executor, which is from > InetAddress.getLocalHost.getHostAddress. > org.apache.spark.deploy.worker.WorkerArguments > {quote} > var host = Utils.localHostName() > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org