[
https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16466816#comment-16466816
]
Rui Li commented on HIVE-18533:
-------------------------------
Hi [~stakiar], my concern for SparkLauncherFuture is mainly about the cancel
logic. SparkLauncherFuture::cancel calls SparkAppHandle::stop, which I think is
an async method. So it doesn't immediately unblock threads waiting on
SparkLauncherFuture::get. And subsequent calls to isCancelled and isDone may
not return true. Besides, JavaDoc mentions SparkAppHandle::stop is only best
effort to ask the app to stop, so it doesn't even guarantee a state change.
Another issue is SparkLauncherFuture::isCancelled considers all failed states
as cancelled. So it may return true even if cancel is not called.
I know this might not be an issue according to the way AbstractSparkClient
works at the moment. But if we want to make changes to AbstractSparkClient in
the future, it's better if the two sub-classes behave consistently and both
honor the Future contracts.
If we use a FutureTask, we can interrupt the thread when we cancel the Future.
The thread can handle the interrupt exception and call SparkAppHandle::stop
(probably need to cancel the RPC as well) -- similar to what we do in
SparkSubmitSparkClient.
> Add option to use InProcessLauncher to submit spark jobs
> --------------------------------------------------------
>
> Key: HIVE-18533
> URL: https://issues.apache.org/jira/browse/HIVE-18533
> Project: Hive
> Issue Type: Improvement
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
> Attachments: HIVE-18533.1.patch, HIVE-18533.2.patch,
> HIVE-18533.3.patch, HIVE-18533.4.patch, HIVE-18533.5.patch,
> HIVE-18533.6.patch, HIVE-18533.7.patch, HIVE-18533.8.patch,
> HIVE-18533.9.patch, HIVE-18533.91.patch, HIVE-18831.93.patch
>
>
> See discussion in HIVE-16484 for details.
> I think this will help with reducing the amount of time it takes to open a
> HoS session + debuggability (no need launch a separate process to run a Spark
> app).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
