[ https://issues.apache.org/jira/browse/SPARK-13664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Armbrust reopened SPARK-13664: -------------------------------------- > Simplify and Speedup HadoopFSRelation > ------------------------------------- > > Key: SPARK-13664 > URL: https://issues.apache.org/jira/browse/SPARK-13664 > Project: Spark > Issue Type: Improvement > Components: SQL > Reporter: Michael Armbrust > Assignee: Michael Armbrust > Priority: Blocker > Fix For: 2.0.0 > > > A majority of Spark SQL queries likely run though {{HadoopFSRelation}}, > however there are currently several complexity and performance problems with > this code path: > - The class mixes the concerns of file management, schema reconciliation, > scan building, bucketing, partitioning, and writing data. > - For very large tables, we are broadcasting the entire list of files to > every executor. [SPARK-11441] > - For partitioned tables, we always do an extra projection. This results > not only in a copy, but undoes much of the performance gains that we are > going to get from vectorized reads. > This is an umbrella ticket to track a set of improvements to this codepath. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org