Benno Evers created MESOS-9260:
----------------------------------
Summary:
StorageLocalResourceProviderTest.ReconcileUnacknowledgedTerminalOperation
Key: MESOS-9260
URL: https://issues.apache.org/jira/browse/MESOS-9260
Project: Mesos
Issue Type: Bug
Reporter: Benno Evers
Observed in an internal CI system:
{noformat}
../../src/tests/storage_local_resource_provider_tests.cpp:4286
Failed to wait 15secs for offers
{noformat}
Full log:
{noformat}
[ RUN ]
StorageLocalResourceProviderTest.ReconcileUnacknowledgedTerminalOperation
I0926 00:01:11.465590 11812 cluster.cpp:173] Creating default 'local' authorizer
I0926 00:01:11.466861 1833 master.cpp:413] Master
f0a77094-641d-45e5-a890-46f2c2d8202c (ip-172-16-10-227.ec2.internal) started on
172.16.10.227:43103
I0926 00:01:11.466879 1833 master.cpp:416] Flags at startup: --acls=""
--agent_ping_timeout="15secs" --agent_reregister_timeout="10mins"
--allocation_interval="50ms" --allocator="hierarchical"
--authenticate_agents="true" --authenticate_frameworks="true"
--authenticate_http_frameworks="true" --authenticate_http_readonly="true"
--authenticate_http_readwrite="true" --authentication_v0_timeout="15secs"
--authenticators="crammd5" --authorizers="local"
--credentials="/tmp/r7LBlp/credentials" --filter_gpu_resources="true"
--framework_sorter="drf" --help="false" --hostname_lookup="true"
--http_authenticators="basic" --http_framework_authenticators="basic"
--initialize_driver_logging="true" --log_auto_initialize="true"
--logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5"
--max_completed_frameworks="50" --max_completed_tasks_per_framework="1000"
--max_unreachable_tasks_per_framework="1000" --memory_profiling="false"
--min_allocatable_resources="cpus:0.01|mem:32" --port="5050" --quiet="false"
--recovery_agent_removal_limit="100%" --registry="in_memory"
--registry_fetch_timeout="1mins" --registry_gc_interval="15mins"
--registry_max_agent_age="2weeks" --registry_max_agent_count="102400"
--registry_store_timeout="100secs" --registry_strict="false"
--require_agent_domain="false" --role_sorter="drf" --root_submissions="true"
--version="false" --webui_dir="/usr/local/share/mesos/webui"
--work_dir="/tmp/r7LBlp/master" --zk_session_timeout="10secs"
I0926 00:01:11.467025 1833 master.cpp:465] Master only allowing authenticated
frameworks to register
I0926 00:01:11.467034 1833 master.cpp:471] Master only allowing authenticated
agents to register
I0926 00:01:11.467041 1833 master.cpp:477] Master only allowing authenticated
HTTP frameworks to register
I0926 00:01:11.467047 1833 credentials.hpp:37] Loading credentials for
authentication from '/tmp/r7LBlp/credentials'
I0926 00:01:11.467128 1833 master.cpp:521] Using default 'crammd5'
authenticator
I0926 00:01:11.467170 1833 http.cpp:1037] Creating default 'basic' HTTP
authenticator for realm 'mesos-master-readonly'
I0926 00:01:11.467207 1833 http.cpp:1037] Creating default 'basic' HTTP
authenticator for realm 'mesos-master-readwrite'
I0926 00:01:11.467229 1833 http.cpp:1037] Creating default 'basic' HTTP
authenticator for realm 'mesos-master-scheduler'
I0926 00:01:11.467248 1833 master.cpp:602] Authorization enabled
I0926 00:01:11.467972 1833 hierarchical.cpp:182] Initialized hierarchical
allocator process
I0926 00:01:11.468001 1833 whitelist_watcher.cpp:77] No whitelist given
I0926 00:01:11.468333 1833 master.cpp:2083] Elected as the leading master!
I0926 00:01:11.468343 1833 master.cpp:1638] Recovering from registrar
I0926 00:01:11.468374 1833 registrar.cpp:339] Recovering registrar
I0926 00:01:11.468508 1833 registrar.cpp:383] Successfully fetched the
registry (0B) in 0ns
I0926 00:01:11.468541 1833 registrar.cpp:487] Applied 1 operations in 8882ns;
attempting to update the registry
I0926 00:01:11.468659 1833 registrar.cpp:544] Successfully updated the
registry in 0ns
I0926 00:01:11.468688 1833 registrar.cpp:416] Successfully recovered registrar
I0926 00:01:11.468765 1833 master.cpp:1752] Recovered 0 agents from the
registry (176B); allowing 10mins for agents to reregister
I0926 00:01:11.468796 1833 hierarchical.cpp:220] Skipping recovery of
hierarchical allocator: nothing to recover
W0926 00:01:11.470585 11812 process.cpp:2810] Attempted to spawn already
running process files@172.16.10.227:43103
I0926 00:01:11.470960 11812 containerizer.cpp:305] Using isolation {
cgroups/cpu, filesystem/posix, environment_secret, cgroups/mem, network/cni }
I0926 00:01:11.472955 11812 linux_launcher.cpp:144] Using
/sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
I0926 00:01:11.473356 11812 provisioner.cpp:298] Using default backend 'overlay'
I0926 00:01:11.478726 11812 cluster.cpp:485] Creating default 'local' authorizer
I0926 00:01:11.479349 1840 slave.cpp:267] Mesos agent started on
(771)@172.16.10.227:43103
I0926 00:01:11.479365 1840 slave.cpp:268] Flags at startup: --acls=""
--appc_simple_discovery_uri_prefix="http://";
--appc_store_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/store/appc"
--authenticate_http_executors="true" --authenticate_http_readonly="true"
--authenticate_http_readwrite="true" --authenticatee="crammd5"
--authentication_backoff_factor="1secs" --authentication_timeout_max="1mins"
--authentication_timeout_min="5secs" --authorizer="local"
--cgroups_cpu_enable_pids_and_tids_count="false"
--cgroups_destroy_timeout="1mins" --cgroups_enable_cfs="false"
--cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false"
--cgroups_root="mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff"
--container_disk_watch_interval="15secs" --containerizers="mesos"
--credential="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/credential"
--default_role="*" --disallow_sharing_agent_pid_namespace="false"
--disk_profile_adaptor="org_apache_mesos_UriDiskProfileAdaptor"
--disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true"
--docker_registry="https://registry-1.docker.io"; --docker_remove_delay="6hrs"
--docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns"
--docker_store_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/store/docker"
--docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume"
--enforce_container_disk_quota="false" --executor_registration_timeout="1mins"
--executor_reregistration_timeout="2secs"
--executor_shutdown_grace_period="5secs"
--fetcher_cache_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/fetch"
--fetcher_cache_size="2GB" --fetcher_stall_timeout="1mins"
--frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1"
--gc_non_executor_container_sandboxes="false" --help="false"
--hostname_lookup="true" --http_command_executor="false"
--http_credentials="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/http_credentials"
--http_heartbeat_interval="30secs" --initialize_driver_logging="true"
--isolation="cgroups/cpu,cgroups/mem"
--jwt_secret_key="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/jwt_secret_key"
--launcher="linux"
--launcher_dir="/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src"
--logbufsecs="0" --logging_level="INFO"
--max_completed_executors_per_framework="150" --memory_profiling="false"
--network_cni_metrics="true" --oversubscribed_resources_interval="15secs"
--perf_duration="10secs" --perf_interval="1mins" --port="5051"
--qos_correction_interval_min="0ns" --quiet="false"
--reconfiguration_policy="equal" --recover="reconnect"
--recovery_timeout="15mins" --registration_backoff_factor="10ms"
--resource_provider_config_dir="/tmp/r7LBlp/resource_provider_configs"
--resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]"
--revocable_cpu_low_priority="true"
--runtime_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO"
--sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true"
--systemd_enable_support="true"
--systemd_runtime_directory="/run/systemd/system" --version="false"
--work_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86"
--zk_session_timeout="10secs"
I0926 00:01:11.479526 1840 credentials.hpp:86] Loading credential for
authentication from
'/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/credential'
I0926 00:01:11.479574 1840 slave.cpp:300] Agent using credential for:
test-principal
I0926 00:01:11.479584 1840 credentials.hpp:37] Loading credentials for
authentication from
'/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/http_credentials'
I0926 00:01:11.479640 1840 http.cpp:1037] Creating default 'basic' HTTP
authenticator for realm 'mesos-agent-executor'
I0926 00:01:11.479673 1840 http.cpp:1058] Creating default 'jwt' HTTP
authenticator for realm 'mesos-agent-executor'
I0926 00:01:11.479717 1840 http.cpp:1037] Creating default 'basic' HTTP
authenticator for realm 'mesos-agent-readonly'
I0926 00:01:11.479737 1840 http.cpp:1058] Creating default 'jwt' HTTP
authenticator for realm 'mesos-agent-readonly'
I0926 00:01:11.479763 1840 http.cpp:1037] Creating default 'basic' HTTP
authenticator for realm 'mesos-agent-readwrite'
I0926 00:01:11.479780 1840 http.cpp:1058] Creating default 'jwt' HTTP
authenticator for realm 'mesos-agent-readwrite'
I0926 00:01:11.479838 1840 disk_profile_adaptor.cpp:84] Creating disk profile
adaptor module 'org_apache_mesos_UriDiskProfileAdaptor'
I0926 00:01:11.480406 1840 slave.cpp:615] Agent resources:
[{"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}]
I0926 00:01:11.480468 1840 slave.cpp:623] Agent attributes: [ ]
I0926 00:01:11.480478 1840 slave.cpp:632] Agent hostname:
ip-172-16-10-227.ec2.internal
I0926 00:01:11.480512 1837 uri_disk_profile_adaptor.cpp:304] Updated disk
profile mapping to 1 active profiles
I0926 00:01:11.480752 1837 task_status_update_manager.cpp:181] Pausing sending
task status updates
I0926 00:01:11.480829 1837 state.cpp:66] Recovering state from
'/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/meta'
I0926 00:01:11.480952 1838 slave.cpp:6909] Finished recovering checkpointed
state from
'/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/meta',
beginning agent recovery
I0926 00:01:11.480989 1838 task_status_update_manager.cpp:207] Recovering task
status update manager
I0926 00:01:11.481066 1838 containerizer.cpp:727] Recovering Mesos containers
I0926 00:01:11.481117 1838 linux_launcher.cpp:286] Recovering Linux launcher
I0926 00:01:11.481254 1840 containerizer.cpp:1053] Recovering isolators
I0926 00:01:11.481770 1837 containerizer.cpp:1092] Recovering provisioner
I0926 00:01:11.481986 1835 provisioner.cpp:494] Provisioner recovery complete
I0926 00:01:11.482172 1833 composing.cpp:339] Finished recovering all
containerizers
I0926 00:01:11.482262 1834 slave.cpp:7138] Recovering executors
I0926 00:01:11.482285 1834 slave.cpp:7291] Finished recovery
I0926 00:01:11.482527 1836 task_status_update_manager.cpp:181] Pausing sending
task status updates
I0926 00:01:11.482537 1837 slave.cpp:1254] New master detected at
master@172.16.10.227:43103
I0926 00:01:11.482563 1837 slave.cpp:1319] Detecting new master
I0926 00:01:11.482641 1833 slave.cpp:1346] Authenticating with master
master@172.16.10.227:43103
I0926 00:01:11.482664 1833 slave.cpp:1355] Using default CRAM-MD5 authenticatee
I0926 00:01:11.482738 1836 authenticatee.cpp:121] Creating new client SASL
connection
I0926 00:01:11.483110 1840 master.cpp:9653] Authenticating
slave(771)@172.16.10.227:43103
I0926 00:01:11.483165 1835 authenticator.cpp:414] Starting authentication
session for crammd5-authenticatee(1491)@172.16.10.227:43103
I0926 00:01:11.483230 1835 authenticator.cpp:98] Creating new server SASL
connection
I0926 00:01:11.483600 1835 authenticatee.cpp:213] Received SASL authentication
mechanisms: CRAM-MD5
I0926 00:01:11.483618 1835 authenticatee.cpp:239] Attempting to authenticate
with mechanism 'CRAM-MD5'
I0926 00:01:11.483647 1835 authenticator.cpp:204] Received SASL authentication
start
I0926 00:01:11.483681 1835 authenticator.cpp:326] Authentication requires more
steps
I0926 00:01:11.483709 1835 authenticatee.cpp:259] Received SASL authentication
step
I0926 00:01:11.483744 1835 authenticator.cpp:232] Received SASL authentication
step
I0926 00:01:11.483757 1835 auxprop.cpp:109] Request to lookup properties for
user: 'test-principal' realm: 'ip-172-16-10-227.ec2.internal' server FQDN:
'ip-172-16-10-227.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false
SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I0926 00:01:11.483767 1835 auxprop.cpp:181] Looking up auxiliary property
'*userPassword'
I0926 00:01:11.483778 1835 auxprop.cpp:181] Looking up auxiliary property
'*cmusaslsecretCRAM-MD5'
I0926 00:01:11.483786 1835 auxprop.cpp:109] Request to lookup properties for
user: 'test-principal' realm: 'ip-172-16-10-227.ec2.internal' server FQDN:
'ip-172-16-10-227.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false
SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I0926 00:01:11.483793 1835 auxprop.cpp:131] Skipping auxiliary property
'*userPassword' since SASL_AUXPROP_AUTHZID == true
I0926 00:01:11.483798 1835 auxprop.cpp:131] Skipping auxiliary property
'*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I0926 00:01:11.483811 1835 authenticator.cpp:318] Authentication success
I0926 00:01:11.483865 1840 authenticatee.cpp:299] Authentication success
I0926 00:01:11.483928 1835 master.cpp:9685] Successfully authenticated
principal 'test-principal' at slave(771)@172.16.10.227:43103
I0926 00:01:11.483942 1839 authenticator.cpp:432] Authentication session
cleanup for crammd5-authenticatee(1491)@172.16.10.227:43103
I0926 00:01:11.484007 1833 slave.cpp:1446] Successfully authenticated with
master master@172.16.10.227:43103
I0926 00:01:11.484109 1833 slave.cpp:1877] Will retry registration in
14.703994ms if necessary
I0926 00:01:11.484150 1835 master.cpp:6605] Received register agent message
from slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal)
I0926 00:01:11.484203 1835 master.cpp:3964] Authorizing agent providing
resources 'cpus:2; mem:1024; disk:1024; ports:[31000-32000]' with principal
'test-principal'
I0926 00:01:11.484340 1840 master.cpp:6672] Authorized registration of agent
at slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal)
I0926 00:01:11.484375 1840 master.cpp:6787] Registering agent at
slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal) with id
f0a77094-641d-45e5-a890-46f2c2d8202c-S0
I0926 00:01:11.484493 1840 registrar.cpp:487] Applied 1 operations in 38466ns;
attempting to update the registry
I0926 00:01:11.484637 1835 registrar.cpp:544] Successfully updated the
registry in 0ns
I0926 00:01:11.484691 1834 master.cpp:6835] Admitted agent
f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103
(ip-172-16-10-227.ec2.internal)
I0926 00:01:11.484822 1834 master.cpp:6880] Registered agent
f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103
(ip-172-16-10-227.ec2.internal) with cpus:2; mem:1024; disk:1024;
ports:[31000-32000]
I0926 00:01:11.484916 1834 hierarchical.cpp:601] Added agent
f0a77094-641d-45e5-a890-46f2c2d8202c-S0 (ip-172-16-10-227.ec2.internal) with
cpus:2; mem:1024; disk:1024; ports:[31000-32000] (allocated: {})
I0926 00:01:11.484963 1834 hierarchical.cpp:1564] Performed allocation for 1
agents in 9501ns
I0926 00:01:11.484994 1834 slave.cpp:1479] Registered with master
master@172.16.10.227:43103; given agent ID
f0a77094-641d-45e5-a890-46f2c2d8202c-S0
I0926 00:01:11.485126 1834 slave.cpp:1499] Checkpointing SlaveInfo to
'/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/meta/slaves/f0a77094-641d-45e5-a890-46f2c2d8202c-S0/slave.info'
I0926 00:01:11.485360 1835 task_status_update_manager.cpp:188] Resuming
sending task status updates
I0926 00:01:11.485404 1834 slave.cpp:1548] Forwarding agent update
{"operations":{},"resource_version_uuid":{"value":"TUGtB/13S1q/oBij0hB5rA=="},"slave_id":{"value":"f0a77094-641d-45e5-a890-46f2c2d8202c-S0"},"update_oversubscribed_resources":false}
I0926 00:01:11.485775 1836 master.cpp:7939] Ignoring update on agent
f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103
(ip-172-16-10-227.ec2.internal) as it reports no changes
I0926 00:01:11.487349 1836 process.cpp:3569] Handling HTTP event for process
'slave(771)' with path: '/slave(771)/api/v1'
I0926 00:01:11.487823 1840 http.cpp:1177] HTTP POST for /slave(771)/api/v1
from 172.16.10.227:35910
I0926 00:01:11.487898 1840 http.cpp:2074] Processing GET_CONTAINERS call
I0926 00:01:11.490869 1836 container_daemon.cpp:121] Launching container
'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
I0926 00:01:11.491497 1833 process.cpp:3569] Handling HTTP event for process
'slave(771)' with path: '/slave(771)/api/v1'
I0926 00:01:11.491904 1833 http.cpp:1177] HTTP POST for /slave(771)/api/v1
from 172.16.10.227:35912
I0926 00:01:11.492103 1833 http.cpp:2440] Processing LAUNCH_CONTAINER call for
container
'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
I0926 00:01:11.492333 1833 http.cpp:2556] Creating sandbox
'/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
I0926 00:01:11.493259 1836 containerizer.cpp:1280] Starting container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:11.493693 1836 containerizer.cpp:1446] Checkpointed
ContainerConfig at
'/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE/config'
I0926 00:01:11.493710 1836 containerizer.cpp:3118] Transitioning the state of
container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
from PROVISIONING to PREPARING
I0926 00:01:11.493806 1834 cgroups.cpp:429] Creating cgroup at
'/sys/fs/cgroup/cpu,cpuacct/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
for container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:11.494220 1834 cgroups.cpp:429] Creating cgroup at
'/sys/fs/cgroup/memory/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
for container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:11.494746 1834 memory.cpp:478] Started listening for OOM events
for container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:11.494796 1834 memory.cpp:590] Started listening on 'low' memory
pressure events for container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:11.494828 1834 memory.cpp:590] Started listening on 'medium'
memory pressure events for container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:11.494858 1834 memory.cpp:590] Started listening on 'critical'
memory pressure events for container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:11.495095 1836 memory.cpp:198] Updated
'memory.soft_limit_in_bytes' to 1GB for container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:11.495132 1834 cpu.cpp:92] Updated 'cpu.shares' to 102 (cpus 0.1)
for container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:11.495165 1836 memory.cpp:227] Updated 'memory.limit_in_bytes' to
1GB for container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:11.495867 1834 containerizer.cpp:1939] Launching
'mesos-containerizer' with flags '--help="false"
--launch_info="{"command":{"arguments":["/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin","--available_capacity=4GB","--volumes=","--work_dir=/tmp/r7LBlp/test_csi_plugin"],"shell":false,"value":"/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin"},"environment":{"variables":[{"name":"MESOS_SANDBOX","type":"VALUE","value":"/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE"},{"name":"CSI_ENDPOINT","type":"VALUE","value":"unix:///tmp/mesos-csi-wGzvgE/endpoint.sock"}]},"task_environment":{},"working_directory":"/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE"}"
--pipe_read="75" --pipe_write="76"
--runtime_directory="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE"
--unshare_namespace_mnt="false"'
I0926 00:01:11.496028 1840 linux_launcher.cpp:492] Launching container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
and cloning with namespaces
I0926 00:01:11.521980 1834 containerizer.cpp:3118] Transitioning the state of
container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
from PREPARING to ISOLATING
I0926 00:01:11.523646 1839 containerizer.cpp:3118] Transitioning the state of
container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
from ISOLATING to FETCHING
I0926 00:01:11.523715 1839 fetcher.cpp:369] Starting to fetch URIs for
container:
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE,
directory:
/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:11.523888 1839 containerizer.cpp:3118] Transitioning the state of
container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
from FETCHING to RUNNING
I0926 00:01:11.525076 1835 container_daemon.cpp:140] Invoking post-start hook
for container
'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
I0926 00:01:11.525574 1835 hierarchical.cpp:1564] Performed allocation for 1
agents in 18948ns
I0926 00:01:11.576836 1836 hierarchical.cpp:1564] Performed allocation for 1
agents in 19631ns
I0926 00:01:11.627815 1835 hierarchical.cpp:1564] Performed allocation for 1
agents in 21050ns
I0926 00:01:11.639616 6260 test_csi_plugin.cpp:302] ProbeRequest '{}'
I0926 00:01:11.640031 1839 container_daemon.cpp:171] Waiting for container
'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
I0926 00:01:11.640367 6260 test_csi_plugin.cpp:274] GetPluginInfoRequest '{}'
I0926 00:01:11.640795 1836 process.cpp:3569] Handling HTTP event for process
'slave(771)' with path: '/slave(771)/api/v1'
I0926 00:01:11.640939 1836 provider.cpp:2178] Node plugin loaded:
{"name":"org.apache.mesos.csi.test","vendorVersion":"1.8.0"}
I0926 00:01:11.641376 1834 http.cpp:1177] HTTP POST for /slave(771)/api/v1
from 172.16.10.227:35914
I0926 00:01:11.641413 6261 test_csi_plugin.cpp:288]
GetPluginCapabilitiesRequest '{}'
I0926 00:01:11.641479 1834 http.cpp:2670] Processing WAIT_CONTAINER call for
container
'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
I0926 00:01:11.641993 6261 test_csi_plugin.cpp:274] GetPluginInfoRequest '{}'
I0926 00:01:11.642236 1839 provider.cpp:2219] Controller plugin loaded:
{"name":"org.apache.mesos.csi.test","vendorVersion":"1.8.0"}
I0926 00:01:11.642463 6261 test_csi_plugin.cpp:593]
ControllerGetCapabilitiesRequest '{}'
I0926 00:01:11.642946 6261 test_csi_plugin.cpp:870] NodeGetCapabilitiesRequest
'{}'
I0926 00:01:11.643378 6261 test_csi_plugin.cpp:857] NodeGetIdRequest '{}'
I0926 00:01:11.643676 1839 provider.cpp:661] Finished recovery for resource
provider with type 'org.apache.mesos.rp.local.storage' and name 'test
I0926 00:01:11.643731 1834 status_update_manager_process.hpp:379] Pausing
operation status update manager
I0926 00:01:11.643767 1839 http_connection.hpp:227] New endpoint detected at
http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
I0926 00:01:11.644228 1838 http_connection.hpp:283] Connected with the remote
endpoint at http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
I0926 00:01:11.644340 1839 provider.cpp:524] Connected to resource provider
manager
I0926 00:01:11.644498 1835 http_connection.hpp:131] Sending 1 call to
http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
I0926 00:01:11.644778 1837 process.cpp:3569] Handling HTTP event for process
'slave(771)' with path: '/slave(771)/api/v1/resource_provider'
I0926 00:01:11.645036 1840 http.cpp:1177] HTTP POST for
/slave(771)/api/v1/resource_provider from 172.16.10.227:35918
I0926 00:01:11.645167 1840 manager.cpp:736] Subscribing resource provider
{"default_reservations":[{"role":"storage","type":"DYNAMIC"}],"name":"test","storage":{"plugin":{"containers":[{"command":{"arguments":["/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin","--available_capacity=4GB","--volumes=","--work_dir=/tmp/r7LBlp/test_csi_plugin"],"shell":false,"value":"/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin"},"resources":[{"name":"cpus","scalar":{"value":0.1},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"}],"services":["CONTROLLER_SERVICE","NODE_SERVICE"]}],"name":"test_csi_plugin","type":"org.apache.mesos.csi.test"}},"type":"org.apache.mesos.rp.local.storage"}
I0926 00:01:11.647924 1840 slave.cpp:7643] Handling resource provider message
'SUBSCRIBE:
{"default_reservations":[{"role":"storage","type":"DYNAMIC"}],"id":{"value":"520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2"},"name":"test","storage":{"plugin":{"containers":[{"command":{"arguments":["/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin","--available_capacity=4GB","--volumes=","--work_dir=/tmp/r7LBlp/test_csi_plugin"],"shell":false,"value":"/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin"},"resources":[{"name":"cpus","scalar":{"value":0.1},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"}],"services":["CONTROLLER_SERVICE","NODE_SERVICE"]}],"name":"test_csi_plugin","type":"org.apache.mesos.csi.test"}},"type":"org.apache.mesos.rp.local.storage"}'
I0926 00:01:11.648219 1837 provider.cpp:546] Received SUBSCRIBED event
I0926 00:01:11.648238 1837 provider.cpp:1496] Subscribed with ID
520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
I0926 00:01:11.648468 1836 status_update_manager_process.hpp:314] Recovering
operation status update manager
I0926 00:01:11.648780 6261 test_csi_plugin.cpp:531] ListVolumesRequest '{}'
I0926 00:01:11.649070 1839 provider.cpp:3519] Sending UPDATE_STATE call with
resources '{}' and 0 operations to agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0
I0926 00:01:11.649124 1839 provider.cpp:1110] Resource provider
520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 is in READY state
I0926 00:01:11.649148 1834 http_connection.hpp:131] Sending 3 call to
http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
I0926 00:01:11.649207 1839 status_update_manager_process.hpp:385] Resuming
operation status update manager
I0926 00:01:11.649250 1835 provider.cpp:1422] Updating profiles { test } for
resource provider 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
I0926 00:01:11.649550 1837 process.cpp:3569] Handling HTTP event for process
'slave(771)' with path: '/slave(771)/api/v1/resource_provider'
I0926 00:01:11.649752 1837 http.cpp:1177] HTTP POST for
/slave(771)/api/v1/resource_provider from 172.16.10.227:35916
I0926 00:01:11.649860 1837 manager.cpp:917] Received UPDATE_STATE call with
resources '[]' and 0 operations from resource provider
520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
I0926 00:01:11.649793 6261 test_csi_plugin.cpp:559] GetCapacityRequest
'{"volumeCapabilities":[{"mount":{},"accessMode":{"mode":"SINGLE_NODE_WRITER"}}]}'
I0926 00:01:11.649943 1837 slave.cpp:7643] Handling resource provider message
'UPDATE_STATE: 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 {}'
I0926 00:01:11.649966 1837 slave.cpp:7763] Forwarding new total resources
cpus:2; mem:1024; disk:1024; ports:[31000-32000]
I0926 00:01:11.650372 1837 provider.cpp:1297] Removing '{}' and adding
'disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096' to the total
resources
I0926 00:01:11.650480 1833 hierarchical.cpp:740] Grew agent
f0a77094-641d-45e5-a890-46f2c2d8202c-S0 by {} (total), { } (used)
I0926 00:01:11.650538 1833 hierarchical.cpp:697] Agent
f0a77094-641d-45e5-a890-46f2c2d8202c-S0 (ip-172-16-10-227.ec2.internal) updated
with total resources cpus:2; mem:1024; disk:1024; ports:[31000-32000]
I0926 00:01:11.650691 1837 provider.cpp:3519] Sending UPDATE_STATE call with
resources 'disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096' and 0
operations to agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0
I0926 00:01:11.650811 1833 http_connection.hpp:131] Sending 3 call to
http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
I0926 00:01:11.651093 1840 process.cpp:3569] Handling HTTP event for process
'slave(771)' with path: '/slave(771)/api/v1/resource_provider'
I0926 00:01:11.651756 11812 scheduler.cpp:189] Version: 1.8.0
I0926 00:01:11.652252 1838 scheduler.cpp:355] Using default 'basic' HTTP
authenticatee
I0926 00:01:11.652364 1836 scheduler.cpp:538] New master detected at
master@172.16.10.227:43103
I0926 00:01:11.652377 1836 scheduler.cpp:547] Waiting for 0ns before
initiating a re-(connection) attempt with the master
I0926 00:01:11.652827 1835 scheduler.cpp:429] Connected with the master at
http://172.16.10.227:43103/master/api/v1/scheduler
I0926 00:01:11.653015 1838 scheduler.cpp:248] Sending SUBSCRIBE call to
http://172.16.10.227:43103/master/api/v1/scheduler
I0926 00:01:11.653323 1833 process.cpp:3569] Handling HTTP event for process
'master' with path: '/master/api/v1/scheduler'
I0926 00:01:11.653650 1840 http.cpp:1177] HTTP POST for
/master/api/v1/scheduler from 172.16.10.227:35922
I0926 00:01:11.653709 1840 master.cpp:2502] Received subscription request for
HTTP framework 'default'
I0926 00:01:11.653733 1840 master.cpp:2155] Authorizing framework principal
'test-principal' to receive offers for roles '{ storage }'
I0926 00:01:11.653841 1834 master.cpp:2637] Subscribing framework 'default'
with checkpointing disabled and capabilities [ MULTI_ROLE,
RESERVATION_REFINEMENT ]
I0926 00:01:11.654402 1834 master.cpp:9883] Adding framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) with roles { } suppressed
I0926 00:01:11.654527 1837 hierarchical.cpp:306] Added framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000
I0926 00:01:11.654933 1836 scheduler.cpp:845] Enqueuing event SUBSCRIBED
received from http://172.16.10.227:43103/master/api/v1/scheduler
I0926 00:01:11.654948 1837 hierarchical.cpp:1564] Performed allocation for 1
agents in 140494ns
I0926 00:01:11.655035 1835 master.cpp:9468] Sending offers [
f0a77094-641d-45e5-a890-46f2c2d8202c-O0 ] to framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
I0926 00:01:11.689805 1839 http.cpp:1177] HTTP POST for
/slave(771)/api/v1/resource_provider from 172.16.10.227:35916
I0926 00:01:13.959466 1834 manager.cpp:917] Received UPDATE_STATE call with
resources
'[{"disk":{"source":{"profile":"test","type":"RAW"}},"name":"disk","provider_id":{"value":"520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2"},"reservations":[{"role":"storage","type":"DYNAMIC"}],"scalar":{"value":4096.0},"type":"SCALAR"}]'
and 0 operations from resource provider 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
I0926 00:01:13.960250 1839 slave.cpp:7643] Handling resource provider message
'UPDATE_STATE: 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 disk(reservations:
[(DYNAMIC,storage)])[RAW(,test)]:4096'
I0926 00:01:13.960312 1839 slave.cpp:7763] Forwarding new total resources
cpus:2; mem:1024; disk:1024; ports:[31000-32000]; disk(reservations:
[(DYNAMIC,storage)])[RAW(,test)]:4096
I0926 00:01:13.960785 1838 hierarchical.cpp:697] Agent
f0a77094-641d-45e5-a890-46f2c2d8202c-S0 (ip-172-16-10-227.ec2.internal) updated
with total resources cpus:2; mem:1024; disk:1024; ports:[31000-32000];
disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096
I0926 00:01:13.960876 1838 hierarchical.cpp:1564] Performed allocation for 1
agents in 40437ns
I0926 00:01:13.965555 1836 scheduler.cpp:845] Enqueuing event HEARTBEAT
received from http://172.16.10.227:43103/master/api/v1/scheduler
I0926 00:01:13.966929 1836 scheduler.cpp:845] Enqueuing event OFFERS received
from http://172.16.10.227:43103/master/api/v1/scheduler
I0926 00:01:13.967149 1838 scheduler.cpp:248] Sending DECLINE call to
http://172.16.10.227:43103/master/api/v1/scheduler
I0926 00:01:13.967648 1835 process.cpp:3569] Handling HTTP event for process
'master' with path: '/master/api/v1/scheduler'
I0926 00:01:13.967850 1837 hierarchical.cpp:1564] Performed allocation for 1
agents in 33598ns
I0926 00:01:13.968111 1833 http.cpp:1177] HTTP POST for
/master/api/v1/scheduler from 172.16.10.227:35920
I0926 00:01:13.968158 1833 master.cpp:5850] Processing DECLINE call for
offers: [ f0a77094-641d-45e5-a890-46f2c2d8202c-O0 ] for framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
I0926 00:01:13.968245 1833 master.cpp:11462] Removing offer
f0a77094-641d-45e5-a890-46f2c2d8202c-O0
I0926 00:01:13.968375 1833 hierarchical.cpp:1236] Recovered cpus(allocated:
storage):2; mem(allocated: storage):1024; disk(allocated: storage):1024;
ports(allocated: storage):[31000-32000] (total: cpus:2; mem:1024; disk:1024;
ports:[31000-32000]; disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096,
allocated: {}) on agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0 from framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000
I0926 00:01:13.968408 1833 hierarchical.cpp:1282] Framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000 filtered agent
f0a77094-641d-45e5-a890-46f2c2d8202c-S0 for 5secs
../../src/tests/storage_local_resource_provider_tests.cpp:4286: Failure
Failed to wait 15secs for offers
I0926 00:01:28.977944 1833 master.cpp:1366] Framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) disconnected
I0926 00:01:28.977985 1833 master.cpp:3230] Deactivating framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
I0926 00:01:28.978009 1833 master.cpp:3207] Disconnecting framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
I0926 00:01:28.978024 1833 master.cpp:1381] Giving framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) 0ns to failover
I0926 00:01:28.978085 1833 hierarchical.cpp:420] Deactivated framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000
I0926 00:01:28.978204 1837 master.cpp:9261] Framework failover timeout,
removing framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
I0926 00:01:28.978226 1837 master.cpp:10197] Removing framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
I0926 00:01:28.978302 1840 slave.cpp:3896] Asked to shut down framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000 by master@172.16.10.227:43103
I0926 00:01:28.978327 1840 slave.cpp:3911] Cannot shut down unknown framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000
I0926 00:01:28.978341 1837 hierarchical.cpp:359] Removed framework
f0a77094-641d-45e5-a890-46f2c2d8202c-0000
../../src/tests/storage_local_resource_provider_tests.cpp:4266: Failure
Actual function call count doesn't match EXPECT_CALL(*scheduler, offers(_,
v1::scheduler::OffersHaveAnyResource(isRaw)))...
Expected: to be called once
Actual: never called - unsatisfied and active
I0926 00:01:28.978569 11812 slave.cpp:909] Agent terminating
I0926 00:01:28.978735 11812 manager.cpp:161] Terminating resource provider
520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
E0926 00:01:28.979059 1836 http_connection.hpp:452] End-Of-File received
I0926 00:01:28.979130 1836 http_connection.hpp:217] Re-detecting endpoint
I0926 00:01:28.979238 1836 http_connection.hpp:227] New endpoint detected at
http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
I0926 00:01:28.979274 1839 provider.cpp:536] Disconnected from resource
provider manager
I0926 00:01:28.979276 1836 http_connection.hpp:338] Ignoring disconnection
attempt from stale connection
I0926 00:01:28.979300 1836 http_connection.hpp:338] Ignoring disconnection
attempt from stale connection
I0926 00:01:28.979308 1839 status_update_manager_process.hpp:379] Pausing
operation status update manager
I0926 00:01:28.979518 1834 master.cpp:1251] Agent
f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103
(ip-172-16-10-227.ec2.internal) disconnected
I0926 00:01:28.979540 1834 master.cpp:3267] Disconnecting agent
f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103
(ip-172-16-10-227.ec2.internal)
I0926 00:01:28.979560 1834 master.cpp:3286] Deactivating agent
f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103
(ip-172-16-10-227.ec2.internal)
I0926 00:01:28.979599 1834 hierarchical.cpp:795] Agent
f0a77094-641d-45e5-a890-46f2c2d8202c-S0 deactivated
I0926 00:01:28.979914 1839 containerizer.cpp:2455] Destroying container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
in RUNNING state
I0926 00:01:28.979934 1839 containerizer.cpp:3118] Transitioning the state of
container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
from RUNNING to DESTROYING
I0926 00:01:28.980154 1839 linux_launcher.cpp:580] Asked to destroy container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:28.980207 1839 linux_launcher.cpp:622] Destroying cgroup
'/sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
I0926 00:01:28.980270 1840 http_connection.hpp:283] Connected with the remote
endpoint at http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
I0926 00:01:28.980406 1840 hierarchical.cpp:1564] Performed allocation for 1
agents in 26143ns
I0926 00:01:28.980520 1836 provider.cpp:524] Connected to resource provider
manager
I0926 00:01:28.980541 1838 cgroups.cpp:2838] Freezing cgroup
/sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:28.980742 1835 http_connection.hpp:131] Sending 1 call to
http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
I0926 00:01:28.980916 1838 cgroups.cpp:1229] Successfully froze cgroup
/sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
after 349184ns
I0926 00:01:28.981250 1837 cgroups.cpp:2856] Thawing cgroup
/sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:28.981411 1837 cgroups.cpp:1258] Successfully thawed cgroup
/sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
after 124928ns
I0926 00:01:28.981528 1841 process.cpp:2735] Returning '404 Not Found' for
'/slave(771)/api/v1/resource_provider'
E0926 00:01:28.981875 1840 provider.cpp:1067] Failed to subscribe resource
provider with type 'org.apache.mesos.rp.local.storage' and name 'test':
Received '404 Not Found' ()
I0926 00:01:29.031200 1835 hierarchical.cpp:1564] Performed allocation for 1
agents in 26888ns
I0926 00:01:29.081390 1836 hierarchical.cpp:1564] Performed allocation for 1
agents in 21841ns
I0926 00:01:29.081869 1836 containerizer.cpp:2957] Container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
has exited
I0926 00:01:29.083161 1839 provisioner.cpp:597] Ignoring destroy request for
unknown container
org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
I0926 00:01:29.083875 1834 container_daemon.cpp:189] Invoking post-stop hook
for container
'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
I0926 00:01:29.084064 1834 container_daemon.cpp:121] Launching container
'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
I0926 00:01:29.086195 1841 process.cpp:2735] Returning '404 Not Found' for
'/slave(771)/api/v1'
E0926 00:01:29.086671 1833 container_daemon.cpp:150] Failed to launch
container
'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE':
Failed to launch container
'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE':
Unexpected response '404 Not Found' ()
I0926 00:01:29.099272 11812 master.cpp:1093] Master terminating
I0926 00:01:29.099445 1836 hierarchical.cpp:637] Removed agent
f0a77094-641d-45e5-a890-46f2c2d8202c-S0
I0926 00:01:29.113350 1837 cgroups.cpp:2838] Freezing cgroup
/sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff
I0926 00:01:29.113492 1837 cgroups.cpp:1229] Successfully froze cgroup
/sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff after
99072ns
I0926 00:01:29.113646 1837 cgroups.cpp:2856] Thawing cgroup
/sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff
I0926 00:01:29.113750 1837 cgroups.cpp:1258] Successfully thawed cgroup
/sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff after
72960ns
[ FAILED ]
StorageLocalResourceProviderTest.ReconcileUnacknowledgedTerminalOperation
(17703 ms)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
