[
https://issues.apache.org/jira/browse/HIVE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16195041#comment-16195041
]
Aihua Xu commented on HIVE-17111:
---------------------------------
TestSparkCliDriver is testing on local cluster. TestMiniSparkOnYarnCliDriver is
testing against yarn. I see the value to add TestLocalSparkCliDriver to cover
LocalHiveSparkClient and also it will make debugging Spark easier. Of course
we just need one or two tests to cover it as the patch does.
+1 from me.
[~lirui], [~xuefuz] How do you think?
> Add TestLocalSparkCliDriver
> ---------------------------
>
> Key: HIVE-17111
> URL: https://issues.apache.org/jira/browse/HIVE-17111
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Attachments: HIVE-17111.1.patch
>
>
> The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but
> the HoS still uses decides to use the RemoteHiveSparkClient rather than the
> LocalHiveSparkClient.
> The issue is with the following check in HiveSparkClientFactory:
> {code}
> if (master.equals("local") || master.startsWith("local[")) {
> // With local spark context, all user sessions share the same spark
> context.
> return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
> } else {
> return new RemoteHiveSparkClient(hiveconf, sparkConf);
> }
> {code}
> When {{master.startsWith("local[")}} it checks the value of spark.master and
> sees that it doesn't start with {{local[}} and then decides to use the
> RemoteHiveSparkClient.
> We should fix this so that the LocalHiveSparkClient is used. It should speed
> up some of the tests, and also makes qtests easier to debug since everything
> will now be run in the same process.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
