Yep, you shouldn't enable *spark.driver.allowMultipleContexts* since it has
the potential to cause extremely difficult-to-debug task failures; it was
originally introduced as an escape-hatch to allow users whose workloads
happened to work "by accident" to continue using multiple active contexts,
but I would not write any new code which uses it.
On Tue, Dec 1, 2015 at 5:45 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> Looks like #2 is better choice.
>
> On Tue, Dec 1, 2015 at 4:51 PM, Anfernee Xu <anfernee...@gmail.com> wrote:
>
>> Thanks Ted, so 1) is off from the table, can I go with 2), yarn-cluster
>> mode? As the driver is running as a Yarn container, it's should be OK for
>> my usercase, isn't it?
>>
>> Anfernee
>>
>> On Tue, Dec 1, 2015 at 4:48 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>>
>>> For #1, looks like the config is used in test suites:
>>>
>>> .set("spark.driver.allowMultipleContexts", "true")
>>>
>>> ./sql/core/src/test/scala/org/apache/spark/sql/MultiSQLContextsSuite.scala
>>> .set("spark.driver.allowMultipleContexts", "true")
>>>
>>> ./sql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeCoordinatorSuite.scala
>>> .set("spark.driver.allowMultipleContexts", "false")
>>> val conf = new SparkConf().set("spark.driver.allowMultipleContexts",
>>> "false")
>>> .set("spark.driver.allowMultipleContexts", "true")
>>> .set("spark.driver.allowMultipleContexts", "true"))
>>> ./core/src/test/scala/org/apache/spark/SparkContextSuite.scala
>>>
>>> FYI
>>>
>>> On Tue, Dec 1, 2015 at 3:32 PM, Anfernee Xu <anfernee...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a doubt regarding yarn-cluster mode and spark.driver.
>>>> allowMultipleContexts for below usercases.
>>>>
>>>> I have a long running backend server where I will create a short-lived
>>>> Spark job in response to each user request, base on the fact that by
>>>> default multiple Spark Context cannot be created in the same JVM, looks
>>>> like I have 2 choices
>>>>
>>>> 1) enable spark.driver.allowMultipleContexts
>>>>
>>>> 2) run my jobs in yarn-cluster mode instead yarn-client
>>>>
>>>> For 1) I cannot find any official document, so looks like it's not
>>>> encouraged, isn't it?
>>>> For 2), I want to make sure yarn-cluster will NOT hit such
>>>> limitation(single SparkContext per VM), apparently I have to something in
>>>> driver side to push the result set back to my application.
>>>>
>>>> Thanks
>>>>
>>>> --
>>>> --Anfernee
>>>>
>>>
>>>
>>
>>
>> --
>> --Anfernee
>>
>
>
