[
https://issues.apache.org/jira/browse/HIVE-10550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541662#comment-14541662
]
Chengxiang Li commented on HIVE-10550:
--------------------------------------
New added configuration:
||name||default value||
|hive.spark.dynamic.rdd.caching|true|
|hive.spark.dynamic.rdd.caching.threshold|100 * 1024 * 1024L(100M)|
> Dynamic RDD caching optimization for HoS.[Spark Branch]
> -------------------------------------------------------
>
> Key: HIVE-10550
> URL: https://issues.apache.org/jira/browse/HIVE-10550
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Chengxiang Li
> Assignee: Chengxiang Li
> Attachments: HIVE-10550.1.patch
>
>
> A Hive query may try to scan the same table multi times, like self-join,
> self-union, or even share the same subquery, [TPC-DS
> Q39|https://github.com/hortonworks/hive-testbench/blob/hive14/sample-queries-tpcds/query39.sql]
> is an example. As you may know that, Spark support cache RDD data, which
> mean Spark would put the calculated RDD data in memory and get the data from
> memory directly for next time, this avoid the calculation cost of this
> RDD(and all the cost of its dependencies) at the cost of more memory usage.
> Through analyze the query context, we should be able to understand which part
> of query could be shared, so that we can reuse the cached RDD in the
> generated Spark job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
