[
https://issues.apache.org/jira/browse/HIVE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16195249#comment-16195249
]
Sahil Takiar commented on HIVE-17111:
-------------------------------------
[~xuefuz]
{quote} 1. Other test driver class is generated, so I'm wondering why we
actually have a nongenerated class for TestLocalSparkCliDriver. {quote} I think
that was the old way of making these classes, now you can just directly create
a new Java class to make a new CLI Driver. Unless I am misunderstanding the
question.
{quote} 2. With this change, are we able to run any .q test using this test
driver class? {quote}. Yes, the idea is that you can run any .q file with these
new CLI Driver, which should help developers debug any HoS issues.
> Add TestLocalSparkCliDriver
> ---------------------------
>
> Key: HIVE-17111
> URL: https://issues.apache.org/jira/browse/HIVE-17111
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Attachments: HIVE-17111.1.patch
>
>
> The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but
> the HoS still uses decides to use the RemoteHiveSparkClient rather than the
> LocalHiveSparkClient.
> The issue is with the following check in HiveSparkClientFactory:
> {code}
> if (master.equals("local") || master.startsWith("local[")) {
> // With local spark context, all user sessions share the same spark
> context.
> return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
> } else {
> return new RemoteHiveSparkClient(hiveconf, sparkConf);
> }
> {code}
> When {{master.startsWith("local[")}} it checks the value of spark.master and
> sees that it doesn't start with {{local[}} and then decides to use the
> RemoteHiveSparkClient.
> We should fix this so that the LocalHiveSparkClient is used. It should speed
> up some of the tests, and also makes qtests easier to debug since everything
> will now be run in the same process.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
