Sahil Takiar created HIVE-17111:
-----------------------------------
Summary: TestSparkCliDriver does not use LocalHiveSparkClient
Key: HIVE-17111
URL: https://issues.apache.org/jira/browse/HIVE-17111
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Sahil Takiar
Assignee: Sahil Takiar
The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but the
HoS still uses decides to use the RemoteHiveSparkClient rather than the
LocalHiveSparkClient.
The issue is with the following check in HiveSparkClientFactory:
{code}
if (master.equals("local") || master.startsWith("local[")) {
// With local spark context, all user sessions share the same spark
context.
return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
} else {
return new RemoteHiveSparkClient(hiveconf, sparkConf);
}
{code}
When {{master.startsWith("local[")}} it checks the value of spark.master and
sees that it doesn't start with {{local[}} and then decides to use the
RemoteHiveSparkClient.
We should fix this so that the LocalHiveSparkClient is used. It should speed up
some of the tests, and also makes qtests easier to debug since everything will
now be run in the same process.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
