[
https://issues.apache.org/jira/browse/HIVE-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199423#comment-14199423
]
Xuefu Zhang commented on HIVE-7768:
-----------------------------------
Given that SPARK-3174 is resolved, this research should continue. Eventually,
we need to figure out how to integrate with it.
[~venki387], would you have time on this? Thanks.
> Research growing/shrinking our Spark Application [Spark Branch]
> ---------------------------------------------------------------
>
> Key: HIVE-7768
> URL: https://issues.apache.org/jira/browse/HIVE-7768
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Brock Noland
> Assignee: Venki Korukanti
> Priority: Critical
>
> Scenario:
> A user connects to Hive and runs a query on a small time. Our SC is sized for
> that small table. They then run a query on a much larger table. We'll need to
> "re-size" the SC which I don't think Spark supports today, so we need to
> research what is available today in Spark and how Tez works.
> More details:
> Similar to Tez, it's likely our "SparkContext" is going to be long lived and
> process many queries. Some queries will be large and some small. Additionally
> the SC might be idle for long periods of time.
> In this JIRA we will research the following:
> * How Spark decides the number of slaves for a given RDD today
> * Given a SC when you create a new RDD based on a much larger input dataset,
> does the SC adjust?
> * How Tez increases/decreases the size of the running YARN application (set
> of slaves)
> * How Tez handles scenarios when it has a running set of slaves in YARN and
> requests more resources for a query and fails to get additional resources
> * How Tez decides to timeout idle slaves
> This will guide requirements we'll need from Spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
