[
https://issues.apache.org/jira/browse/SPARK-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14340950#comment-14340950
]
Xuefu Zhang commented on SPARK-3691:
------------------------------------
Hive is using spark.master=local-cluster for unit tests. It also built a mini
yarn cluster to test Hive on Spark. Thus, we don't need this any more.
> Provide a mini cluster for testing system built on Spark
> --------------------------------------------------------
>
> Key: SPARK-3691
> URL: https://issues.apache.org/jira/browse/SPARK-3691
> Project: Spark
> Issue Type: Test
> Components: Spark Core
> Affects Versions: 1.1.0
> Reporter: Xuefu Zhang
>
> Most Hadoop components such MR, DFS, Tez, and Yarn provide a mini cluster
> that can be used to test the external systems that rely on those frameworks,
> such as Pig and Hive. While Spark's local mode can be used to do such testing
> and is friendly for debugging, it's too far from a real Spark cluster and a
> lot of problems cannot be discovered. Thus, an equivalent of Hadoop MR mini
> cluster in Spark would be very helpful in testing system such as Hive/Pig on
> Spark.
> Spark's local-cluster is considered for this purpose but it doesn't fit well
> because it requires a Spark installation on the box where the tests run.
> Also, local-cluster isn't exposed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
