[ 
https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350813#comment-16350813
 ] 

Thomas Graves commented on SPARK-23309:
---------------------------------------

I'm still seeing spark 2.3 slower by about 15% for the larger dataset.  I tried 

=> --conf spark.sql.orc.impl=hive --conf spark.sql.orc.filterPushdown=true 
--conf spark.sql.hive.convertMetastoreOrc=false

and then also tried setting the vectoried reader to false

=> --conf spark.sql.orc.impl=hive --conf spark.sql.orc.filterPushdown=true 
--conf spark.sql.hive.convertMetastoreOrc=false --conf 
spark.sql.inMemoryColumnarStorage.enableVectorizedReader=false

Note the # of partitions its processing is now the same since turning off the 
native orc impl.

 

> Spark 2.3 cached query performance 20-30% worse then spark 2.2
> --------------------------------------------------------------
>
>                 Key: SPARK-23309
>                 URL: https://issues.apache.org/jira/browse/SPARK-23309
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Thomas Graves
>            Priority: Blocker
>
> I was testing spark 2.3 rc2 and I am seeing a performance regression in sql 
> queries on cached data.
> The size of the data: 10.4GB input from hive orc files /18.8 GB cached/5592 
> partitions
> Here is the example query:
> val dailycached = spark.sql("select something from table where dt = 
> '20170301' AND something IS NOT NULL")
> dailycached.createOrReplaceTempView("dailycached") 
> spark.catalog.cacheTable("dailyCached")
> spark.sql("SELECT COUNT(DISTINCT(something)) from dailycached").show()
>  
> On spark 2.2 I see queries times average 13 seconds
> On the same nodes I see spark 2.3 queries times average 17 seconds
> Note these are times of queries after the initial caching.  so just running 
> the last line again: 
> spark.sql("SELECT COUNT(DISTINCT(something)) from dailycached").show() 
> multiple times.
>  
> I also ran a query over more data (335GB input/587.5 GB cached) and saw a 
> similar discrepancy in the performance of querying cached data between spark 
> 2.3 and spark 2.2, where 2.2 was better by like 20%.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to