I noticed other errors so I put all LIBPROCESS_SSL_ also in the
/etc/default/marathon/ and after putting back
executor_environment_variables, I still have this, task goes immediately
to delayed.
I0218 21:35:13.301903 5296 slave.cpp:2035] Got assigned task
'server.b13d17c6-33bc-11e9-b70f-0050563001a1' for framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0218 21:35:13.302404 5295 gc.cpp:129] Unscheduling
'/var/lib/mesos/slaves/53336fcb-7756-4673-b9c7-177e04f34c3b-S2/framework
s/d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000' from gc
I0218 21:35:13.302502 5295 gc.cpp:129] Unscheduling
'/var/lib/mesos/meta/slaves/53336fcb-7756-4673-b9c7-177e04f34c3b-S2/fram
eworks/d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000' from gc
I0218 21:35:13.303890 5296 slave.cpp:2409] Authorizing task
'server.b13d17c6-33bc-11e9-b70f-0050563001a1' for framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0218 21:35:13.305584 5296 slave.cpp:2852] Launching task
'server.b13d17c6-33bc-11e9-b70f-0050563001a1' for framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0218 21:35:13.306473 5296 paths.cpp:745] Creating sandbox
'/var/lib/mesos/slaves/53336fcb-7756-4673-b9c7-177e04f34c3b-S2/framework
s/d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000/executors/server.b13d17c6-33
bc-11e9-b70f-0050563001a1/runs/7d1322ba-6fb6-47ba-a4ed-fd09131902b4' for
user 'nobody'
I0218 21:35:13.307333 5296 paths.cpp:748] Creating sandbox
'/var/lib/mesos/meta/slaves/53336fcb-7756-4673-b9c7-177e04f34c3b-S2/fram
eworks/d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000/executors/server.b13d17
c6-33bc-11e9-b70f-0050563001a1/runs/7d1322ba-6fb6-47ba-a4ed-fd09131902b4
'
I0218 21:35:13.307591 5296 slave.cpp:9020] Launching executor
'server.b13d17c6-33bc-11e9-b70f-0050563001a1' of framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000 with resources
[{"allocation_info":{"role":"marathon"},"name":"cpus","scalar":{"value":
0.1},"type":"SCALAR"},{"allocation_info":{"role":"marathon"},"name":"mem
","scalar":{"value":32.0},"type":"SCALAR"}] in work directory
'/var/lib/mesos/slaves/53336fcb-7756-4673-b9c7-177e04f34c3b-S2/framework
s/d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000/executors/server.b13d17c6-33
bc-11e9-b70f-0050563001a1/runs/7d1322ba-6fb6-47ba-a4ed-fd09131902b4'
I0218 21:35:13.308032 5296 slave.cpp:3530] Launching container
7d1322ba-6fb6-47ba-a4ed-fd09131902b4 for executor
'server.b13d17c6-33bc-11e9-b70f-0050563001a1' of framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0218 21:35:13.308653 5296 slave.cpp:3049] Queued task
'server.b13d17c6-33bc-11e9-b70f-0050563001a1' for executor
'server.b13d17c6-33bc-11e9-b70f-0050563001a1' of framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0218 21:35:13.309053 5294 containerizer.cpp:1286] Starting container
7d1322ba-6fb6-47ba-a4ed-fd09131902b4
I0218 21:35:13.309458 5294 containerizer.cpp:3134] Transitioning the
state of container 7d1322ba-6fb6-47ba-a4ed-fd09131902b4 from
PROVISIONING to PREPARING
I0218 21:35:13.311450 5300 switchboard.cpp:316] Container logger module
finished preparing container 7d1322ba-6fb6-47ba-a4ed-fd09131902b4;
IOSwitchboard server is not required
I0218 21:35:13.312512 5295 linux_launcher.cpp:492] Launching container
7d1322ba-6fb6-47ba-a4ed-fd09131902b4 and cloning with namespaces
CLONE_NEWNS
I0218 21:35:13.321563 5299 containerizer.cpp:2056] Checkpointing
container's forked pid 5728 to
'/var/lib/mesos/meta/slaves/53336fcb-7756-4673-b9c7-177e04f34c3b-S2/fram
eworks/d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000/executors/server.b13d17
c6-33bc-11e9-b70f-0050563001a1/runs/7d1322ba-6fb6-47ba-a4ed-fd09131902b4
/pids/forked.pid'
I0218 21:35:13.322213 5299 containerizer.cpp:3134] Transitioning the
state of container 7d1322ba-6fb6-47ba-a4ed-fd09131902b4 from PREPARING
to ISOLATING
I0218 21:35:13.323691 5299 containerizer.cpp:3134] Transitioning the
state of container 7d1322ba-6fb6-47ba-a4ed-fd09131902b4 from ISOLATING
to FETCHING
I0218 21:35:13.324301 5299 containerizer.cpp:3134] Transitioning the
state of container 7d1322ba-6fb6-47ba-a4ed-fd09131902b4 from FETCHING to
RUNNING
I0218 21:35:13.526005 5296 containerizer.cpp:2973] Container
7d1322ba-6fb6-47ba-a4ed-fd09131902b4 has exited
I0218 21:35:13.526057 5296 containerizer.cpp:2467] Destroying container
7d1322ba-6fb6-47ba-a4ed-fd09131902b4 in RUNNING state
I0218 21:35:13.526080 5296 containerizer.cpp:3134] Transitioning the
state of container 7d1322ba-6fb6-47ba-a4ed-fd09131902b4 from RUNNING to
DESTROYING
I0218 21:35:13.526208 5296 linux_launcher.cpp:576] Asked to destroy
container 7d1322ba-6fb6-47ba-a4ed-fd09131902b4
I0218 21:35:13.526435 5296 linux_launcher.cpp:618] Destroying cgroup
'/sys/fs/cgroup/freezer/mesos/7d1322ba-6fb6-47ba-a4ed-fd09131902b4'
I0218 21:35:13.526872 5296 cgroups.cpp:2854] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/7d1322ba-6fb6-47ba-a4ed-fd09131902b4
I0218 21:35:13.527145 5296 cgroups.cpp:1242] Successfully froze cgroup
/sys/fs/cgroup/freezer/mesos/7d1322ba-6fb6-47ba-a4ed-fd09131902b4 after
235008ns
I0218 21:35:13.527376 5296 cgroups.cpp:2872] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/7d1322ba-6fb6-47ba-a4ed-fd09131902b4
I0218 21:35:13.527496 5296 cgroups.cpp:1271] Successfully thawed cgroup
/sys/fs/cgroup/freezer/mesos/7d1322ba-6fb6-47ba-a4ed-fd09131902b4 after
78080ns
I0218 21:35:13.531431 5297 linux_launcher.cpp:650] Destroying cgroup
'/sys/fs/cgroup/systemd/mesos/7d1322ba-6fb6-47ba-a4ed-fd09131902b4'
I0218 21:35:13.534024 5295 slave.cpp:6331] Executor
'server.b13d17c6-33bc-11e9-b70f-0050563001a1' of framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000 exited with status 1
I0218 21:35:13.535235 5295 slave.cpp:5311] Handling status update
TASK_FAILED (Status UUID: 837b9ecf-ff27-4367-8715-4d0981bdeabb) for task
server.b13d17c6-33bc-11e9-b70f-0050563001a1 of framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000 from @0.0.0.0:0
W0218 21:35:13.535614 5296 containerizer.cpp:2266] Ignoring update for
unknown container 7d1322ba-6fb6-47ba-a4ed-fd09131902b4
I0218 21:35:13.536057 5295 task_status_update_manager.cpp:328] Received
task status update TASK_FAILED (Status UUID:
837b9ecf-ff27-4367-8715-4d0981bdeabb) for task
server.b13d17c6-33bc-11e9-b70f-0050563001a1 of framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0218 21:35:13.536970 5295 task_status_update_manager.cpp:842]
Checkpointing UPDATE for task status update TASK_FAILED (Status UUID:
837b9ecf-ff27-4367-8715-4d0981bdeabb) for task
server.b13d17c6-33bc-11e9-b70f-0050563001a1 of framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0218 21:35:13.537187 5300 slave.cpp:5803] Forwarding the update
TASK_FAILED (Status UUID: 837b9ecf-ff27-4367-8715-4d0981bdeabb) for task
server.b13d17c6-33bc-11e9-b70f-0050563001a1 of framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000 to [email protected]:5050
I0218 21:35:13.559950 5294 task_status_update_manager.cpp:401] Received
task status update acknowledgement (UUID:
837b9ecf-ff27-4367-8715-4d0981bdeabb) for task
server.b13d17c6-33bc-11e9-b70f-0050563001a1 of framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0218 21:35:13.560015 5294 task_status_update_manager.cpp:842]
Checkpointing ACK for task status update TASK_FAILED (Status UUID:
837b9ecf-ff27-4367-8715-4d0981bdeabb) for task
server.b13d17c6-33bc-11e9-b70f-0050563001a1 of framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0218 21:35:13.560214 5294 slave.cpp:6429] Cleaning up executor
'server.b13d17c6-33bc-11e9-b70f-0050563001a1' of framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
I0218 21:35:13.560338 5299 gc.cpp:95] Scheduling
'/var/lib/mesos/meta/slaves/53336fcb-7756-4673-b9c7-177e04f34c3b-S2/fram
eworks/d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000/executors/server.b13d17
c6-33bc-11e9-b70f-0050563001a1/runs/7d1322ba-6fb6-47ba-a4ed-fd09131902b4
/tasks/server.b13d17c6-33bc-11e9-b70f-0050563001a1' for gc
6.99999351645926days in the future
I0218 21:35:13.560730 5294 slave.cpp:6558] Cleaning up framework
d5168fcd-51be-48c3-ba64-ade27ab23c4e-0000
==============
Feb 18 21:15:22 m01 zookeeper: 2019-02-18 21:15:22,127 [myid:] - INFO
[ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got
user-level KeeperException when processing sessionid:0x169021324e4002a
type:setData cxid:0xe4 zxid:0x1d20 txntype:-1 reqpath:n/a Error
Path:/marathon/state/apps/d/server/2019-02-18T20:15:22.112Z
Error:KeeperErrorCode = NoNode for
/marathon/state/apps/d/server/2019-02-18T20:15:22.112Z
Feb 18 21:15:22 m01 zookeeper: 2019-02-18 21:15:22,127 [myid:] - INFO
[ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got
user-level KeeperException when processing sessionid:0x169021324e4002a
type:setData cxid:0xe4 zxid:0x1d20 txntype:-1 reqpath:n/a Error
Path:/marathon/state/apps/d/server/2019-02-18T20:15:22.112Z
Error:KeeperErrorCode = NoNode for
/marathon/state/apps/d/server/2019-02-18T20:15:22.112Z
Feb 18 21:15:22 m01 zookeeper: 2019-02-18 21:15:22,181 [myid:] - INFO
[ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got
user-level KeeperException when processing sessionid:0x169021324e4002a
type:setData cxid:0xe6 zxid:0x1d22 txntype:-1 reqpath:n/a Error
Path:/marathon/state/group/2/root/2019-02-18T20:15:22.112Z
Error:KeeperErrorCode = NoNode for
/marathon/state/group/2/root/2019-02-18T20:15:22.112Z
Feb 18 21:15:22 m01 zookeeper: 2019-02-18 21:15:22,181 [myid:] - INFO
[ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got
user-level KeeperException when processing sessionid:0x169021324e4002a
type:setData cxid:0xe6 zxid:0x1d22 txntype:-1 reqpath:n/a Error
Path:/marathon/state/group/2/root/2019-02-18T20:15:22.112Z
Error:KeeperErrorCode = NoNode for
/marathon/state/group/2/root/2019-02-18T20:15:22.112Z
Feb 18 21:15:22 m01 zookeeper: 2019-02-18 21:15:22,412 [myid:] - INFO
[ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got
user-level KeeperException when processing sessionid:0x169021324e4002a
type:setData cxid:0xe8 zxid:0x1d24 txntype:-1 reqpath:n/a Error
Path:/marathon/state/deployment/a/aeeb05b3-f6d9-44e4-a012-4af35ebb5554
Error:KeeperErrorCode = NoNode for
/marathon/state/deployment/a/aeeb05b3-f6d9-44e4-a012-4af35ebb5554
Feb 18 21:15:22 m01 zookeeper: 2019-02-18 21:15:22,412 [myid:] - INFO
[ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got
user-level KeeperException when processing sessionid:0x169021324e4002a
type:setData cxid:0xe8 zxid:0x1d24 txntype:-1 reqpath:n/a Error
Path:/marathon/state/deployment/a/aeeb05b3-f6d9-44e4-a012-4af35ebb5554
Error:KeeperErrorCode = NoNode for
/marathon/state/deployment/a/aeeb05b3-f6d9-44e4-a012-4af35ebb5554
Feb 18 21:15:22 m01 zookeeper: 2019-02-18 21:15:22,501 [myid:] - INFO
[ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got
user-level KeeperException when processing sessionid:0x169021324e4002a
type:setData cxid:0xec zxid:0x1d28 txntype:-1 reqpath:n/a Error
Path:/marathon/state/instance/9/server.marathon-eb87000e-33b9-11e9-8454-
0050563001a1 Error:KeeperErrorCode = NoNode for
/marathon/state/instance/9/server.marathon-eb87000e-33b9-11e9-8454-00505
63001a1
Feb 18 21:15:22 m01 zookeeper: 2019-02-18 21:15:22,501 [myid:] - INFO
[ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got
user-level KeeperException when processing sessionid:0x169021324e4002a
type:setData cxid:0xec zxid:0x1d28 txntype:-1 reqpath:n/a Error
Path:/marathon/state/instance/9/server.marathon-eb87000e-33b9-11e9-8454-
0050563001a1 Error:KeeperErrorCode = NoNode for
/marathon/state/instance/9/server.marathon-eb87000e-33b9-11e9-8454-00505
63001a1
Feb 18 21:15:22 m01 mesos-slave[27381]: W0218 21:15:22.741439 27416
containerizer.cpp:2266] Ignoring update for unknown container
3b5f3dfd-b47c-4ec8-a425-25c3e653029c
Feb 18 21:15:22 m01 mesos-slave[27381]: W0218 21:15:22.741439 27416
containerizer.cpp:2266] Ignoring update for unknown container
3b5f3dfd-b47c-4ec8-a425-25c3e653029c
Feb 18 21:15:22 m01 zookeeper: 2019-02-18 21:15:22,761 [myid:] - INFO
[ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got
user-level KeeperException when processing sessionid:0x169021324e4002a
type:setData cxid:0xf2 zxid:0x1d2d txntype:-1 reqpath:n/a Error
Path:/marathon/state/taskFailures/d/server Error:KeeperErrorCode =
NoNode for /marathon/state/taskFailures/d/server
Feb 18 21:15:22 m01 zookeeper: 2019-02-18 21:15:22,761 [myid:] - INFO
[ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got
user-level KeeperException when processing sessionid:0x169021324e4002a
type:setData cxid:0xf3 zxid:0x1d2e txntype:-1 reqpath:n/a Error
Path:/marathon/state/taskFailures/d/server/2019-02-18T20:15:22.112Z
Error:KeeperErrorCode = NoNode for
/marathon/state/taskFailures/d/server/2019-02-18T20:15:22.112Z
Feb 18 21:15:22 m01 zookeeper: 2019-02-18 21:15:22,761 [myid:] - INFO
[ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got
user-level KeeperException when processing sessionid:0x169021324e4002a
type:setData cxid:0xf2 zxid:0x1d2d txntype:-1 reqpath:n/a Error
Path:/marathon/state/taskFailures/d/server Error:KeeperErrorCode =
NoNode for /marathon/state/taskFailures/d/server
Feb 18 21:15:22 m01 zookeeper: 2019-02-18 21:15:22,761 [myid:] - INFO
[ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got
user-level KeeperException when processing sessionid:0x169021324e4002a
type:setData cxid:0xf3 zxid:0x1d2e txntype:-1 reqpath:n/a Error
Path:/marathon/state/taskFailures/d/server/2019-02-18T20:15:22.112Z
Error:KeeperErrorCode = NoNode for
/marathon/state/taskFailures/d/server/2019-02-18T20:15:22.112Z
Feb 18 21:15:22 m01 marathon: [2019-02-18 21:15:22,760] WARN New
instance [instance
[server.marathon-eb87000e-33b9-11e9-8454-0050563001a1]] failed during
app /server scaling, queueing another instance
(mesosphere.marathon.core.deployment.impl.TaskStartActor:marathon-akka.a
ctor.default-dispatcher-12)
Feb 18 21:15:22 m01 marathon: [2019-02-18 21:15:22,760] WARN New
instance [instance
[server.marathon-eb87000e-33b9-11e9-8454-0050563001a1]] failed during
app /server scaling, queueing another instance
(mesosphere.marathon.core.deployment.impl.TaskStartActor:marathon-akka.a
ctor.default-dispatcher-12)
-----Original Message-----
Sent: 18 February 2019 18:49
To: user
Subject: Re: ssl mesos-executor not using /etc/default/mesos
> On Feb 16, 2019, at 9:46 AM, Marc Roos <[email protected]>
wrote:
>
>
>
> Looks like the mesos-executor is not using /etc/default/mesos
> environment variables
Depending on your configuration, the executor runs inside the container,
which means that /etc/default/mesos is probably not available.
>
> If I export the variables in /etc/default/mesos manually, I can run
> the task.
>
> mesos-execute --master=x.x.x.x:5050 --principal=xxx --secret=xxx
> --name=ls --command="ls -lRrt /*; sleep 60"
> --env=file:///test/env.json
>
> How should this be resolved? I tried setting
> --executor_environment_variables=/etc/mesos/executor-env.json
And what happened when you did this?
