[
https://issues.apache.org/jira/browse/HDFS-13720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sekhon updated HDFS-13720:
-------------------------------
Summary: HDFS dataset Anti-Affinity Block Placement across all DataNodes
for data local task optimization (improve Spark executor utilization &
performance) (was: HDFS dataset Anti-Affinity Block Placement across DataNodes
for data local task optimization (improve Spark executor utilization &
performance))
> HDFS dataset Anti-Affinity Block Placement across all DataNodes for data
> local task optimization (improve Spark executor utilization & performance)
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-13720
> URL: https://issues.apache.org/jira/browse/HDFS-13720
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, block placement, performance
> Affects Versions: 2.7.3
> Environment: Hortonworks HDP 2.6
> Reporter: Hari Sekhon
> Priority: Major
>
> Improvement Request for Anti-Affinity Block Placement across datanodes such
> that for a given data set the blocks are distributed evenly across all
> available datanodes in order to improve task scheduling while maintaining
> data locality.
> Methods to be implemented:
> # balancer command switch combined with a target path to files or directories
> # client side write flag
> Both options should proactively (re)distribute the given data set as evenly
> as possible across all datanodes in the cluster.
> See this following Spark issue which causes massive under-utilisation across
> jobs. Only 30-50% of executor cores were being used for tasks due to data
> locality targeting. Many executors doing literally nothing, while holding
> significant cluster resources, because the data set, which in at least one
> job was large enough to have 30,000 tasks churning though slowly on only a
> subset of the available executors. The workaround in the end was to disable
> data local tasks in Spark, but if everyone did that the bottleneck would go
> back to being the network and it undermines Hadoop's first premise of don't
> move the data to compute. For performance critical jobs, returning containers
> to Yarn because they cannot find any data to execute on locally isn't a good
> idea either, they want the jobs to use all the resources available and
> allocated to the job, not just the resources on a subset of nodes that hold a
> given dataset or disabling data local task execution to pull half the blocks
> across the network to make use of the other half of the nodes.
> https://issues.apache.org/jira/browse/SPARK-24474
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
