[
https://issues.apache.org/jira/browse/HIVE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655458#comment-16655458
]
Sahil Takiar edited comment on HIVE-20737 at 10/18/18 3:49 PM:
---------------------------------------------------------------
{quote}Opening of a Spark session and Job submission should be done as an
atomic operation.
{quote}
Well we intentionally don't do that. HoS by design de-couples opening a Spark
session and submitting a job. There are a few reasons:
(1) In order to support static allocation in Spark, we have to open a session
before we even have a job to submit (e.g. see {{SetSparkReducerParallelism}})
(2) At some point I think we should implement HIVE-17927; the reason is that
opening a Spark session causes a Spark application to be created, which
requires resource negotiation with YARN and the spawning of the Spark driver,
which takes a non-trivial amount of time
{quote}Not to have case when we submit something having already closed session
{quote}
We could just re-open the session if we try to submit a job on a closed session.
was (Author: stakiar):
{quote} Opening of a Spark session and Job submission should be done as an
atomic operation. {quote}
Well we intentionally don't do that. HoS by design de-couples opening a Spark
session and submitting a job. There are a few reasons:
(1) In order to support static allocation in Spark, we have to open a session
before we even have a job to submit (e.g. see {{SetSparkReducerParallelism}})
(2) At some point I think we should implement HIVE-17927; the reason is that
opening a Spark session causes a Spark application to be created, which
requires resource negotiation with YARN and the spawning of the Spark driver,
which takes a non-trivial amount of time
{quote} Not to have case when we submit something having already closed session
{quote}
We could just re-open the session if we try to submit a job on a closed session.
> Local SparkContext is shared between user sessions and should be closed only
> when there is no active
> ----------------------------------------------------------------------------------------------------
>
> Key: HIVE-20737
> URL: https://issues.apache.org/jira/browse/HIVE-20737
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Reporter: Denys Kuzmenko
> Assignee: Denys Kuzmenko
> Priority: Major
> Attachments: HIVE-20737.1.patch, HIVE-20737.10.patch,
> HIVE-20737.11.patch, HIVE-20737.12.patch, HIVE-20737.2.patch,
> HIVE-20737.5.patch, HIVE-20737.6.patch, HIVE-20737.7.patch,
> HIVE-20737.8.patch, HIVE-20737.9.patch
>
>
> 1. Local SparkContext is shared between user sessions and should be closed
> only when there is no active.
> 2. Possible race condition in SparkSession.open() in case when user queries
> run in parallel within the same session.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
