[
https://issues.apache.org/jira/browse/HIVE-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16312200#comment-16312200
]
Sahil Takiar commented on HIVE-18214:
-------------------------------------
[~pvary] thanks for taking a look.
* In production, {{RemoteDriver}} is run in a dedicated container; however, we
have some unit tests which run it in the local process; so in production its
not really possible to hit this issue
* I'm not a fan of exposing these methods publicly either, I can add a
{{\@VisibleForTesting}} annotation; {{RemoteDriver}} is already marked as
{{\@Private}}
The only other way I can think of doing this is to change the
{{TestSparkClient}} so it runs the {{RemoteDriver}} in a dedicated process (so
similar to what we do in production). The test will take longer to run, but we
won't hit this issue.
> Flaky test: TestSparkClient
> ---------------------------
>
> Key: HIVE-18214
> URL: https://issues.apache.org/jira/browse/HIVE-18214
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Attachments: HIVE-18214.1.patch
>
>
> Looks like there is a race condition in {{TestSparkClient#runTest}}. The test
> creates a {{RemoteDriver}} in memory, which creates a {{JavaSparkContext}}. A
> new {{JavaSparkContext}} is created for each test that is run. There is a
> race condition where the {{RemoteDriver}} isn't given enough time to
> shutdown, so when the next test starts running it creates another
> {{JavaSparkContext}} which causes an exception like
> {{org.apache.spark.SparkException: Only one SparkContext may be running in
> this JVM (see SPARK-2243)}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
