[
https://issues.apache.org/jira/browse/HIVE-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934965#comment-15934965
]
Sahil Takiar commented on HIVE-14919:
-------------------------------------
[~kellyzly] my guess is that any Hive table should benefit from using the
DataFrames API. I'm not a Spark expert, but I believe RDDs are just a
distributed, collection of objects, but those objects don't have a defined
schema. A DataFrame is similar to a table in a database, it has a set of named
columns. So naturally I would think Hive fits more into the DataFrames model
since it works with tables that have a set of pre-defined columns.
According to some blog posts on the DataFrames API, it has a number of
performance optimizations built in due to the fact that column types are known.
These optimizations were not possible with RDDs because RDDs don't have a
schema.
>From a DataBricks blog post:
{quote}
It can also perform lower level optimizations such as eliminating expensive
object allocations and reducing virtual function calls. As a result, we expect
performance improvements for existing Spark programs when they migrate to
DataFrames.
{quote}
Sources:
https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html
https://databricks.com/blog/2015/02/17/introducing-dataframes-in-spark-for-large-scale-data-science.html
> Improve the performance of Hive on Spark 2.0.0
> ----------------------------------------------
>
> Key: HIVE-14919
> URL: https://issues.apache.org/jira/browse/HIVE-14919
> Project: Hive
> Issue Type: Improvement
> Reporter: Ferdinand Xu
> Assignee: Ferdinand Xu
>
> In HIVE-14029, we have updated Spark dependency to 2.0.0. We use Intel
> BigBench[1] to run benchmark with Spark 2.0 over 1 TB data set comparing with
> Spark 1.6. We can see performance improvments about 5.4% in general and 45%
> for the best case. However, some queries doesn't have significant performance
> improvements. This JIRA is the umbrella ticket addressing those performance
> issues.
> [1] https://github.com/intel-hadoop/Big-Data-Benchmark-for-Big-Bench
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
