Lifeng Wang created SPARK-18738:
-----------------------------------
Summary: Some Spark SQL queries has poor performance on HDFS
Erasure Coding feature when enabling dynamic allocation.
Key: SPARK-18738
URL: https://issues.apache.org/jira/browse/SPARK-18738
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.0.2
Reporter: Lifeng Wang
Fix For: 2.2.0
We run TPCx-BB with Spark SQL engine on local cluster using Spark 2.0.3 trunk
and Hadoop 3.0 alpha 2 trunk. We run Spark SQL queries with same data size on
both Erasure Coding and 3-replication. The test results show that some queries
has much worse performance on EC compared to 3-replication. After initial
investigations, we found spark starts one third executors to execute queries on
EC compared to 3-replication.
We use query 30 as example, our cluster can totally launch 108 executors. When
we run the query from 3-replication database, spark will start all 108
executors to execute the query. When we run the query from Erasure Coding
database, spark will launch 108 executors and kill 72 executors due to they’re
idle, at last there are only 36 executors to execute the query which leads to
poor performance.
This issues only happens when we enable dynamic allocations mechanism. When we
disable the dynamic allocations, Spark SQL query on EC has the similar
performance with on 3-replication.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
