[
https://issues.apache.org/jira/browse/HIVE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17111:
--------------------------------
Attachment: HIVE-17111.1.patch
Attaching a patch that creates a new CLI Driver called
{{TestLocalSparkCliDriver}} which sets {{spark.master=local[*]}}. Adding a very
simple q test, with a few basic queries. This provides some test coverage for
{{LocalHiveSparkClient}}.
The main advantage is that this new CLI Driver runs the entire HoS query inside
a single process. This makes debugging HoS much easier. Users can set
breakpoints in portions of the HoS code that are only invoked at runtime. While
this does provide some coverage for {{LocalHiveSparkClient}}, I think that main
advantage is that it makes debugging HoS easier for developers, especially new
developers who may not be as familiar with the HoS code and want to debug
things via an IDE like IntelliJ.
The patch doesn't modify anything related to the other Spark CLI Drivers.
> TestSparkCliDriver does not use LocalHiveSparkClient
> ----------------------------------------------------
>
> Key: HIVE-17111
> URL: https://issues.apache.org/jira/browse/HIVE-17111
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Attachments: HIVE-17111.1.patch
>
>
> The TestSparkCliDriver sets the spark.master to local-cluster[2,2,1024] but
> the HoS still uses decides to use the RemoteHiveSparkClient rather than the
> LocalHiveSparkClient.
> The issue is with the following check in HiveSparkClientFactory:
> {code}
> if (master.equals("local") || master.startsWith("local[")) {
> // With local spark context, all user sessions share the same spark
> context.
> return LocalHiveSparkClient.getInstance(generateSparkConf(sparkConf));
> } else {
> return new RemoteHiveSparkClient(hiveconf, sparkConf);
> }
> {code}
> When {{master.startsWith("local[")}} it checks the value of spark.master and
> sees that it doesn't start with {{local[}} and then decides to use the
> RemoteHiveSparkClient.
> We should fix this so that the LocalHiveSparkClient is used. It should speed
> up some of the tests, and also makes qtests easier to debug since everything
> will now be run in the same process.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
