What blocks you to put if conditions inside the mentioned map function?
On Mon, Feb 11, 2019 at 10:31 AM Serega Sheypak <serega.shey...@gmail.com>
wrote:
> Yeah, but I don't need to crash entire app, I want to fail several tasks
> or executors and then wait for completion.
>
> вс, 10 февр. 2019 г. в 21:49, Gabor Somogyi <gabor.g.somo...@gmail.com>:
>
>> Another approach is adding artificial exception into the application's
>> source code like this:
>>
>> val query = input.toDS.map(_ / 0).writeStream.format("console").start()
>>
>> G
>>
>>
>> On Sun, Feb 10, 2019 at 9:36 PM Serega Sheypak <serega.shey...@gmail.com>
>> wrote:
>>
>>> Hi BR,
>>> thanks for your reply. I want to mimic the issue and kill tasks at a
>>> certain stage. Killing executor is also an option for me.
>>> I'm curious how do core spark contributors test spark fault tolerance?
>>>
>>>
>>> вс, 10 февр. 2019 г. в 16:57, Gabor Somogyi <gabor.g.somo...@gmail.com>:
>>>
>>>> Hi Serega,
>>>>
>>>> If I understand your problem correctly you would like to kill one
>>>> executor only and the rest of the app has to be untouched.
>>>> If that's true yarn -kill is not what you want because it stops the
>>>> whole application.
>>>>
>>>> I've done similar thing when tested/testing Spark's HA features.
>>>> - jps -vlm | grep
>>>> "org.apache.spark.executor.CoarseGrainedExecutorBackend.*applicationid"
>>>> - kill -9 pidofoneexecutor
>>>>
>>>> Be aware if it's a multi-node cluster check whether at least one
>>>> process runs on a specific node(it's not required).
>>>> Happy killing...
>>>>
>>>> BR,
>>>> G
>>>>
>>>>
>>>> On Sun, Feb 10, 2019 at 4:19 PM Jörn Franke <jornfra...@gmail.com>
>>>> wrote:
>>>>
>>>>> yarn application -kill applicationid ?
>>>>>
>>>>> > Am 10.02.2019 um 13:30 schrieb Serega Sheypak <
>>>>> serega.shey...@gmail.com>:
>>>>> >
>>>>> > Hi there!
>>>>> > I have weird issue that appears only when tasks fail at specific
>>>>> stage. I would like to imitate failure on my own.
>>>>> > The plan is to run problematic app and then kill entire executor or
>>>>> some tasks when execution reaches certain stage.
>>>>> >
>>>>> > Is it do-able?
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>>>>
>>>>>
