Is there anything in the executor logs?
On Tue, Jul 18, 2017 at 2:39 PM, Thodoris Zois <z...@ics.forth.gr> wrote:
> Hello, i am trying to run an executor from jar file. But every time i
> launch a task it fails and scheduler terminates `Aborting because task 0 is
> in unexpected state TASK_FAILED with reason 'REASON_EXECUTOR_TERMINATED'
> from source 'SOURCE_SLAVE' with message 'Executor terminated'` could it be
> an env var?
>
> It's really weird.. Messages from agent and master are these below:
>
> Master:
> I0719 00:18:37.676826 133214208 master.cpp:6598] Executor 'default' of
> framework aa9bbc8c-6444-422c-95e3-deade1c82671-0001 on agent
> aa9bbc8c-6444-422c-95e3-deade1c82671-S0 at slave(1)@127.0.0.1:5051
> (localhost): exited with status 1
>
> I0719 00:18:37.676916 133214208 master.cpp:8691] Removing executor
> 'default' with resources {} of framework
> aa9bbc8c-6444-422c-95e3-deade1c82671-0001
> on agent aa9bbc8c-6444-422c-95e3-deade1c82671-S0 at slave(1)@
> 127.0.0.1:5051 (localhost)
>
>
>
> And from Agent:
>
> 0719 00:09:01.913131 255987712 paths.cpp:573] Trying to chown
> '/var/lib/mesos/slaves/aa9bbc8c-6444-422c-95e3-deade1c82671-S0/frameworks/
> aa9bbc8c-6444-422c-95e3-deade1c82671-0000/executors/
> default/runs/e67bb039-3c47-4cf8-a068-89216c29cfdf' to user 'zois'
>
> I0719 00:09:01.917273 255987712 slave.cpp:6933] Launching executor
> 'default' of framework aa9bbc8c-6444-422c-95e3-deade1c82671-0000 with
> resources {} in work directory '/var/lib/mesos/slaves/
> aa9bbc8c-6444-422c-95e3-deade1c82671-S0/frameworks/
> aa9bbc8c-6444-422c-95e3-deade1c82671-0000/executors/
> default/runs/e67bb039-3c47-4cf8-a068-89216c29cfdf'
>
> I0719 00:09:01.925673 255451136 containerizer.cpp:1001] Starting container
> e67bb039-3c47-4cf8-a068-89216c29cfdf for executor 'default' of framework
> aa9bbc8c-6444-422c-95e3-deade1c82671-0000
>
> I0719 00:09:01.926951 255987712 slave.cpp:2310] Queued task '0' for
> executor 'default' of framework aa9bbc8c-6444-422c-95e3-deade1c82671-0000
>
> I0719 00:09:01.927599 255987712 slave.cpp:2081] Launching task '1' for
> framework aa9bbc8c-6444-422c-95e3-deade1c82671-0000
>
> I0719 00:09:01.928652 255987712 slave.cpp:2310] Queued task '1' for
> executor 'default' of framework aa9bbc8c-6444-422c-95e3-deade1c82671-0000
>
> I0719 00:09:01.965394 255451136 launcher.cpp:140] Forked child with pid
> '2960' for container 'e67bb039-3c47-4cf8-a068-89216c29cfdf'
>
> I0719 00:09:01.966382 255451136 containerizer.cpp:1623] Checkpointing
> container's forked pid 2960 to '/var/lib/mesos/meta/slaves/
> aa9bbc8c-6444-422c-95e3-deade1c82671-S0/frameworks/
> aa9bbc8c-6444-422c-95e3-deade1c82671-0000/executors/
> default/runs/e67bb039-3c47-4cf8-a068-89216c29cfdf/pids/forked.pid'
>
> I0719 00:09:02.097687 258134016 containerizer.cpp:2508] Container
> e67bb039-3c47-4cf8-a068-89216c29cfdf has exited
>
> I0719 00:09:02.097818 258134016 containerizer.cpp:2102] Destroying
> container e67bb039-3c47-4cf8-a068-89216c29cfdf in RUNNING state
>
> I0719 00:09:02.099333 258134016 launcher.cpp:156] Asked to destroy
> container e67bb039-3c47-4cf8-a068-89216c29cfdf
>
> I0719 00:09:02.134932 258670592 slave.cpp:5168] Executor 'default' of
> framework aa9bbc8c-6444-422c-95e3-deade1c82671-0000 exited with status 1
>
> I0719 00:09:02.143437 258670592 slave.cpp:4215] Handling status update
> TASK_FAILED (UUID: 2cd78079-46d0-47b5-9a50-909d539fe094) for task 0 of
> framework aa9bbc8c-6444-422c-95e3-deade1c82671-0000 from @0.0.0.0:0
>
> I0719 00:09:02.147959 258670592 slave.cpp:4215] Handling status update
> TASK_FAILED (UUID: 2b844b59-4e70-46c5-9984-8992c9286804) for task 1 of
> framework aa9bbc8c-6444-422c-95e3-deade1c82671-0000 from @0.0.0.0:0
>
> W0719 00:09:02.149605 255987712 containerizer.cpp:1917] Ignoring update
> for unknown container e67bb039-3c47-4cf8-a068-89216c29cfdf
>
> W0719 00:09:02.150360 258134016 containerizer.cpp:1917] Ignoring update
> for unknown container e67bb039-3c47-4cf8-a068-89216c29cfdf
> I0719 00:09:02.151563 257597440 status_update_manager.cpp:323] Received
> status update TASK_FAILED (UUID: 2cd78079-46d0-47b5-9a50-909d539fe094)
> for task 0 of framework aa9bbc8c-6444-422c-95e3-deade1c82671-0000
>
> I0719 00:09:02.154439 257597440 status_update_manager.cpp:832]
> Checkpointing UPDATE for status update TASK_FAILED (UUID:
> 2cd78079-46d0-47b5-9a50-909d539fe094) for task 0 of framework
> aa9bbc8c-6444-422c-95e3-deade1c82671-0000
>
> Seems like it can’t get resources…
> I really have not idea what it might causing this… Could it be libmesos?
>
> Any help would be appreciated,
> Thank you very much
>
>
