[ https://issues.apache.org/jira/browse/HDFS-13720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Sekhon updated HDFS-13720: ------------------------------- Summary: HDFS dataset Anti-Affinity Block Placement across all DataNodes for data local task optimization (improve Spark executor utilization & performance) (was: HDFS dataset Anti-Affinity Block Placement across DataNodes for data local task optimization (improve Spark executor utilization & performance)) > HDFS dataset Anti-Affinity Block Placement across all DataNodes for data > local task optimization (improve Spark executor utilization & performance) > --------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: HDFS-13720 > URL: https://issues.apache.org/jira/browse/HDFS-13720 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover, block placement, performance > Affects Versions: 2.7.3 > Environment: Hortonworks HDP 2.6 > Reporter: Hari Sekhon > Priority: Major > > Improvement Request for Anti-Affinity Block Placement across datanodes such > that for a given data set the blocks are distributed evenly across all > available datanodes in order to improve task scheduling while maintaining > data locality. > Methods to be implemented: > # balancer command switch combined with a target path to files or directories > # client side write flag > Both options should proactively (re)distribute the given data set as evenly > as possible across all datanodes in the cluster. > See this following Spark issue which causes massive under-utilisation across > jobs. Only 30-50% of executor cores were being used for tasks due to data > locality targeting. Many executors doing literally nothing, while holding > significant cluster resources, because the data set, which in at least one > job was large enough to have 30,000 tasks churning though slowly on only a > subset of the available executors. The workaround in the end was to disable > data local tasks in Spark, but if everyone did that the bottleneck would go > back to being the network and it undermines Hadoop's first premise of don't > move the data to compute. For performance critical jobs, returning containers > to Yarn because they cannot find any data to execute on locally isn't a good > idea either, they want the jobs to use all the resources available and > allocated to the job, not just the resources on a subset of nodes that hold a > given dataset or disabling data local task execution to pull half the blocks > across the network to make use of the other half of the nodes. > https://issues.apache.org/jira/browse/SPARK-24474 -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org