Yeah, but I don't need to crash entire app, I want to fail several tasks or
executors and then wait for completion.
вс, 10 февр. 2019 г. в 21:49, Gabor Somogyi <gabor.g.somo...@gmail.com>:
> Another approach is adding artificial exception into the application's
> source code like this:
>
> val query = input.toDS.map(_ / 0).writeStream.format("console").start()
>
> G
>
>
> On Sun, Feb 10, 2019 at 9:36 PM Serega Sheypak <serega.shey...@gmail.com>
> wrote:
>
>> Hi BR,
>> thanks for your reply. I want to mimic the issue and kill tasks at a
>> certain stage. Killing executor is also an option for me.
>> I'm curious how do core spark contributors test spark fault tolerance?
>>
>>
>> вс, 10 февр. 2019 г. в 16:57, Gabor Somogyi <gabor.g.somo...@gmail.com>:
>>
>>> Hi Serega,
>>>
>>> If I understand your problem correctly you would like to kill one
>>> executor only and the rest of the app has to be untouched.
>>> If that's true yarn -kill is not what you want because it stops the
>>> whole application.
>>>
>>> I've done similar thing when tested/testing Spark's HA features.
>>> - jps -vlm | grep
>>> "org.apache.spark.executor.CoarseGrainedExecutorBackend.*applicationid"
>>> - kill -9 pidofoneexecutor
>>>
>>> Be aware if it's a multi-node cluster check whether at least one process
>>> runs on a specific node(it's not required).
>>> Happy killing...
>>>
>>> BR,
>>> G
>>>
>>>
>>> On Sun, Feb 10, 2019 at 4:19 PM Jörn Franke <jornfra...@gmail.com>
>>> wrote:
>>>
>>>> yarn application -kill applicationid ?
>>>>
>>>> > Am 10.02.2019 um 13:30 schrieb Serega Sheypak <
>>>> serega.shey...@gmail.com>:
>>>> >
>>>> > Hi there!
>>>> > I have weird issue that appears only when tasks fail at specific
>>>> stage. I would like to imitate failure on my own.
>>>> > The plan is to run problematic app and then kill entire executor or
>>>> some tasks when execution reaches certain stage.
>>>> >
>>>> > Is it do-able?
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>>>
>>>>
