[
https://issues.apache.org/jira/browse/SPARK-10149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Zhao updated SPARK-10149:
-----------------------------
Description:
Locality Level is ANY on "Details for Stage" WebUI page
When a sc.textFile(XX) program is running, Locality Level is ANY which should
be NODE_LOCAL on stage 0.
org.apache.spark.scheduler.TaskSetManager
{quote}
// Check for node-local tasks
if (TaskLocality.isAllowed(locality, TaskLocality.NODE_LOCAL)) {
for (index <- speculatableTasks if canRunOnHost(index)) {
val locations = tasks(index).preferredLocations.map(_.host)
if (locations.contains(host)) {
speculatableTasks -= index
return Some((index, TaskLocality.NODE_LOCAL))
}
}
}
{quote}
The variable "locations" is hostname of HDFS split, which is from
InetAddress.getHostName.
The variable "host" is ip of Executor, which is from
InetAddress.getLocalHost.getHostAddress.
org.apache.spark.deploy.worker.WorkerArguments
{quote}
var host = Utils.localHostName()
{quote}
was:
Locality Level is ANY on "Details for Stage" WebUI page
When a sc.textFile(XX) program is running, Locality Level is ANY which should
be NODE_LOCAL on stage 0.
org.apache.spark.scheduler.TaskSetManager
// Check for node-local tasks
if (TaskLocality.isAllowed(locality, TaskLocality.NODE_LOCAL)) {
for (index <- speculatableTasks if canRunOnHost(index)) {
val locations = tasks(index).preferredLocations.map(_.host)
if (locations.contains(host)) {
speculatableTasks -= index
return Some((index, TaskLocality.NODE_LOCAL))
}
}
}
The variable "locations" is hostname of HDFS split, which is from
InetAddress.getHostName.
The variable "host" is ip of Executor, which is from
InetAddress.getLocalHost.getHostAddress.
org.apache.spark.deploy.worker.WorkerArguments
var host = Utils.localHostName()
> Locality Level is ANY on "Details for Stage" WebUI page
> -------------------------------------------------------
>
> Key: SPARK-10149
> URL: https://issues.apache.org/jira/browse/SPARK-10149
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Web UI
> Affects Versions: 1.4.1
> Reporter: Yun Zhao
>
> Locality Level is ANY on "Details for Stage" WebUI page
> When a sc.textFile(XX) program is running, Locality Level is ANY which should
> be NODE_LOCAL on stage 0.
> org.apache.spark.scheduler.TaskSetManager
> {quote}
> // Check for node-local tasks
> if (TaskLocality.isAllowed(locality, TaskLocality.NODE_LOCAL)) {
> for (index <- speculatableTasks if canRunOnHost(index)) {
> val locations = tasks(index).preferredLocations.map(_.host)
> if (locations.contains(host)) {
> speculatableTasks -= index
> return Some((index, TaskLocality.NODE_LOCAL))
> }
> }
> }
> {quote}
> The variable "locations" is hostname of HDFS split, which is from
> InetAddress.getHostName.
> The variable "host" is ip of Executor, which is from
> InetAddress.getLocalHost.getHostAddress.
> org.apache.spark.deploy.worker.WorkerArguments
> {quote}
> var host = Utils.localHostName()
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
