[
https://issues.apache.org/jira/browse/MESOS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chun-Hung Hsiao reassigned MESOS-9260:
--------------------------------------
Assignee: Chun-Hung Hsiao
> StorageLocalResourceProviderTest.ReconcileUnacknowledgedTerminalOperation is
> flaky
> ----------------------------------------------------------------------------------
>
> Key: MESOS-9260
> URL: https://issues.apache.org/jira/browse/MESOS-9260
> Project: Mesos
> Issue Type: Bug
> Reporter: Benno Evers
> Assignee: Chun-Hung Hsiao
> Priority: Major
> Labels: flaky-test
>
> Observed in an internal CI system:
> {noformat}
> ../../src/tests/storage_local_resource_provider_tests.cpp:4286
> Failed to wait 15secs for offers
> {noformat}
> Full log:
> {noformat}
> [ RUN ]
> StorageLocalResourceProviderTest.ReconcileUnacknowledgedTerminalOperation
> I0926 00:01:11.465590 11812 cluster.cpp:173] Creating default 'local'
> authorizer
> I0926 00:01:11.466861 1833 master.cpp:413] Master
> f0a77094-641d-45e5-a890-46f2c2d8202c (ip-172-16-10-227.ec2.internal) started
> on 172.16.10.227:43103
> I0926 00:01:11.466879 1833 master.cpp:416] Flags at startup: --acls=""
> --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins"
> --allocation_interval="50ms" --allocator="hierarchical"
> --authenticate_agents="true" --authenticate_frameworks="true"
> --authenticate_http_frameworks="true" --authenticate_http_readonly="true"
> --authenticate_http_readwrite="true" --authentication_v0_timeout="15secs"
> --authenticators="crammd5" --authorizers="local"
> --credentials="/tmp/r7LBlp/credentials" --filter_gpu_resources="true"
> --framework_sorter="drf" --help="false" --hostname_lookup="true"
> --http_authenticators="basic" --http_framework_authenticators="basic"
> --initialize_driver_logging="true" --log_auto_initialize="true"
> --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5"
> --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000"
> --max_unreachable_tasks_per_framework="1000" --memory_profiling="false"
> --min_allocatable_resources="cpus:0.01|mem:32" --port="5050" --quiet="false"
> --recovery_agent_removal_limit="100%" --registry="in_memory"
> --registry_fetch_timeout="1mins" --registry_gc_interval="15mins"
> --registry_max_agent_age="2weeks" --registry_max_agent_count="102400"
> --registry_store_timeout="100secs" --registry_strict="false"
> --require_agent_domain="false" --role_sorter="drf" --root_submissions="true"
> --version="false" --webui_dir="/usr/local/share/mesos/webui"
> --work_dir="/tmp/r7LBlp/master" --zk_session_timeout="10secs"
> I0926 00:01:11.467025 1833 master.cpp:465] Master only allowing
> authenticated frameworks to register
> I0926 00:01:11.467034 1833 master.cpp:471] Master only allowing
> authenticated agents to register
> I0926 00:01:11.467041 1833 master.cpp:477] Master only allowing
> authenticated HTTP frameworks to register
> I0926 00:01:11.467047 1833 credentials.hpp:37] Loading credentials for
> authentication from '/tmp/r7LBlp/credentials'
> I0926 00:01:11.467128 1833 master.cpp:521] Using default 'crammd5'
> authenticator
> I0926 00:01:11.467170 1833 http.cpp:1037] Creating default 'basic' HTTP
> authenticator for realm 'mesos-master-readonly'
> I0926 00:01:11.467207 1833 http.cpp:1037] Creating default 'basic' HTTP
> authenticator for realm 'mesos-master-readwrite'
> I0926 00:01:11.467229 1833 http.cpp:1037] Creating default 'basic' HTTP
> authenticator for realm 'mesos-master-scheduler'
> I0926 00:01:11.467248 1833 master.cpp:602] Authorization enabled
> I0926 00:01:11.467972 1833 hierarchical.cpp:182] Initialized hierarchical
> allocator process
> I0926 00:01:11.468001 1833 whitelist_watcher.cpp:77] No whitelist given
> I0926 00:01:11.468333 1833 master.cpp:2083] Elected as the leading master!
> I0926 00:01:11.468343 1833 master.cpp:1638] Recovering from registrar
> I0926 00:01:11.468374 1833 registrar.cpp:339] Recovering registrar
> I0926 00:01:11.468508 1833 registrar.cpp:383] Successfully fetched the
> registry (0B) in 0ns
> I0926 00:01:11.468541 1833 registrar.cpp:487] Applied 1 operations in
> 8882ns; attempting to update the registry
> I0926 00:01:11.468659 1833 registrar.cpp:544] Successfully updated the
> registry in 0ns
> I0926 00:01:11.468688 1833 registrar.cpp:416] Successfully recovered
> registrar
> I0926 00:01:11.468765 1833 master.cpp:1752] Recovered 0 agents from the
> registry (176B); allowing 10mins for agents to reregister
> I0926 00:01:11.468796 1833 hierarchical.cpp:220] Skipping recovery of
> hierarchical allocator: nothing to recover
> W0926 00:01:11.470585 11812 process.cpp:2810] Attempted to spawn already
> running process [email protected]:43103
> I0926 00:01:11.470960 11812 containerizer.cpp:305] Using isolation {
> cgroups/cpu, filesystem/posix, environment_secret, cgroups/mem, network/cni }
> I0926 00:01:11.472955 11812 linux_launcher.cpp:144] Using
> /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
> I0926 00:01:11.473356 11812 provisioner.cpp:298] Using default backend
> 'overlay'
> I0926 00:01:11.478726 11812 cluster.cpp:485] Creating default 'local'
> authorizer
> I0926 00:01:11.479349 1840 slave.cpp:267] Mesos agent started on
> (771)@172.16.10.227:43103
> I0926 00:01:11.479365 1840 slave.cpp:268] Flags at startup: --acls=""
> --appc_simple_discovery_uri_prefix="http://";
> --appc_store_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/store/appc"
> --authenticate_http_executors="true" --authenticate_http_readonly="true"
> --authenticate_http_readwrite="true" --authenticatee="crammd5"
> --authentication_backoff_factor="1secs" --authentication_timeout_max="1mins"
> --authentication_timeout_min="5secs" --authorizer="local"
> --cgroups_cpu_enable_pids_and_tids_count="false"
> --cgroups_destroy_timeout="1mins" --cgroups_enable_cfs="false"
> --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false"
> --cgroups_root="mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff"
> --container_disk_watch_interval="15secs" --containerizers="mesos"
> --credential="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/credential"
> --default_role="*" --disallow_sharing_agent_pid_namespace="false"
> --disk_profile_adaptor="org_apache_mesos_UriDiskProfileAdaptor"
> --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true"
> --docker_registry="https://registry-1.docker.io"; --docker_remove_delay="6hrs"
> --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns"
> --docker_store_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/store/docker"
> --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume"
> --enforce_container_disk_quota="false"
> --executor_registration_timeout="1mins"
> --executor_reregistration_timeout="2secs"
> --executor_shutdown_grace_period="5secs"
> --fetcher_cache_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/fetch"
> --fetcher_cache_size="2GB" --fetcher_stall_timeout="1mins"
> --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1"
> --gc_non_executor_container_sandboxes="false" --help="false"
> --hostname_lookup="true" --http_command_executor="false"
> --http_credentials="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/http_credentials"
> --http_heartbeat_interval="30secs" --initialize_driver_logging="true"
> --isolation="cgroups/cpu,cgroups/mem"
> --jwt_secret_key="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/jwt_secret_key"
> --launcher="linux"
> --launcher_dir="/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src"
> --logbufsecs="0" --logging_level="INFO"
> --max_completed_executors_per_framework="150" --memory_profiling="false"
> --network_cni_metrics="true" --oversubscribed_resources_interval="15secs"
> --perf_duration="10secs" --perf_interval="1mins" --port="5051"
> --qos_correction_interval_min="0ns" --quiet="false"
> --reconfiguration_policy="equal" --recover="reconnect"
> --recovery_timeout="15mins" --registration_backoff_factor="10ms"
> --resource_provider_config_dir="/tmp/r7LBlp/resource_provider_configs"
> --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]"
> --revocable_cpu_low_priority="true"
> --runtime_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO"
> --sandbox_directory="/mnt/mesos/sandbox" --strict="true"
> --switch_user="true" --systemd_enable_support="true"
> --systemd_runtime_directory="/run/systemd/system" --version="false"
> --work_dir="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86"
> --zk_session_timeout="10secs"
> I0926 00:01:11.479526 1840 credentials.hpp:86] Loading credential for
> authentication from
> '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/credential'
> I0926 00:01:11.479574 1840 slave.cpp:300] Agent using credential for:
> test-principal
> I0926 00:01:11.479584 1840 credentials.hpp:37] Loading credentials for
> authentication from
> '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/http_credentials'
> I0926 00:01:11.479640 1840 http.cpp:1037] Creating default 'basic' HTTP
> authenticator for realm 'mesos-agent-executor'
> I0926 00:01:11.479673 1840 http.cpp:1058] Creating default 'jwt' HTTP
> authenticator for realm 'mesos-agent-executor'
> I0926 00:01:11.479717 1840 http.cpp:1037] Creating default 'basic' HTTP
> authenticator for realm 'mesos-agent-readonly'
> I0926 00:01:11.479737 1840 http.cpp:1058] Creating default 'jwt' HTTP
> authenticator for realm 'mesos-agent-readonly'
> I0926 00:01:11.479763 1840 http.cpp:1037] Creating default 'basic' HTTP
> authenticator for realm 'mesos-agent-readwrite'
> I0926 00:01:11.479780 1840 http.cpp:1058] Creating default 'jwt' HTTP
> authenticator for realm 'mesos-agent-readwrite'
> I0926 00:01:11.479838 1840 disk_profile_adaptor.cpp:84] Creating disk
> profile adaptor module 'org_apache_mesos_UriDiskProfileAdaptor'
> I0926 00:01:11.480406 1840 slave.cpp:615] Agent resources:
> [{"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}]
> I0926 00:01:11.480468 1840 slave.cpp:623] Agent attributes: [ ]
> I0926 00:01:11.480478 1840 slave.cpp:632] Agent hostname:
> ip-172-16-10-227.ec2.internal
> I0926 00:01:11.480512 1837 uri_disk_profile_adaptor.cpp:304] Updated disk
> profile mapping to 1 active profiles
> I0926 00:01:11.480752 1837 task_status_update_manager.cpp:181] Pausing
> sending task status updates
> I0926 00:01:11.480829 1837 state.cpp:66] Recovering state from
> '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/meta'
> I0926 00:01:11.480952 1838 slave.cpp:6909] Finished recovering checkpointed
> state from
> '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/meta',
> beginning agent recovery
> I0926 00:01:11.480989 1838 task_status_update_manager.cpp:207] Recovering
> task status update manager
> I0926 00:01:11.481066 1838 containerizer.cpp:727] Recovering Mesos containers
> I0926 00:01:11.481117 1838 linux_launcher.cpp:286] Recovering Linux launcher
> I0926 00:01:11.481254 1840 containerizer.cpp:1053] Recovering isolators
> I0926 00:01:11.481770 1837 containerizer.cpp:1092] Recovering provisioner
> I0926 00:01:11.481986 1835 provisioner.cpp:494] Provisioner recovery complete
> I0926 00:01:11.482172 1833 composing.cpp:339] Finished recovering all
> containerizers
> I0926 00:01:11.482262 1834 slave.cpp:7138] Recovering executors
> I0926 00:01:11.482285 1834 slave.cpp:7291] Finished recovery
> I0926 00:01:11.482527 1836 task_status_update_manager.cpp:181] Pausing
> sending task status updates
> I0926 00:01:11.482537 1837 slave.cpp:1254] New master detected at
> [email protected]:43103
> I0926 00:01:11.482563 1837 slave.cpp:1319] Detecting new master
> I0926 00:01:11.482641 1833 slave.cpp:1346] Authenticating with master
> [email protected]:43103
> I0926 00:01:11.482664 1833 slave.cpp:1355] Using default CRAM-MD5
> authenticatee
> I0926 00:01:11.482738 1836 authenticatee.cpp:121] Creating new client SASL
> connection
> I0926 00:01:11.483110 1840 master.cpp:9653] Authenticating
> slave(771)@172.16.10.227:43103
> I0926 00:01:11.483165 1835 authenticator.cpp:414] Starting authentication
> session for crammd5-authenticatee(1491)@172.16.10.227:43103
> I0926 00:01:11.483230 1835 authenticator.cpp:98] Creating new server SASL
> connection
> I0926 00:01:11.483600 1835 authenticatee.cpp:213] Received SASL
> authentication mechanisms: CRAM-MD5
> I0926 00:01:11.483618 1835 authenticatee.cpp:239] Attempting to authenticate
> with mechanism 'CRAM-MD5'
> I0926 00:01:11.483647 1835 authenticator.cpp:204] Received SASL
> authentication start
> I0926 00:01:11.483681 1835 authenticator.cpp:326] Authentication requires
> more steps
> I0926 00:01:11.483709 1835 authenticatee.cpp:259] Received SASL
> authentication step
> I0926 00:01:11.483744 1835 authenticator.cpp:232] Received SASL
> authentication step
> I0926 00:01:11.483757 1835 auxprop.cpp:109] Request to lookup properties for
> user: 'test-principal' realm: 'ip-172-16-10-227.ec2.internal' server FQDN:
> 'ip-172-16-10-227.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false
> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
> I0926 00:01:11.483767 1835 auxprop.cpp:181] Looking up auxiliary property
> '*userPassword'
> I0926 00:01:11.483778 1835 auxprop.cpp:181] Looking up auxiliary property
> '*cmusaslsecretCRAM-MD5'
> I0926 00:01:11.483786 1835 auxprop.cpp:109] Request to lookup properties for
> user: 'test-principal' realm: 'ip-172-16-10-227.ec2.internal' server FQDN:
> 'ip-172-16-10-227.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false
> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
> I0926 00:01:11.483793 1835 auxprop.cpp:131] Skipping auxiliary property
> '*userPassword' since SASL_AUXPROP_AUTHZID == true
> I0926 00:01:11.483798 1835 auxprop.cpp:131] Skipping auxiliary property
> '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
> I0926 00:01:11.483811 1835 authenticator.cpp:318] Authentication success
> I0926 00:01:11.483865 1840 authenticatee.cpp:299] Authentication success
> I0926 00:01:11.483928 1835 master.cpp:9685] Successfully authenticated
> principal 'test-principal' at slave(771)@172.16.10.227:43103
> I0926 00:01:11.483942 1839 authenticator.cpp:432] Authentication session
> cleanup for crammd5-authenticatee(1491)@172.16.10.227:43103
> I0926 00:01:11.484007 1833 slave.cpp:1446] Successfully authenticated with
> master [email protected]:43103
> I0926 00:01:11.484109 1833 slave.cpp:1877] Will retry registration in
> 14.703994ms if necessary
> I0926 00:01:11.484150 1835 master.cpp:6605] Received register agent message
> from slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal)
> I0926 00:01:11.484203 1835 master.cpp:3964] Authorizing agent providing
> resources 'cpus:2; mem:1024; disk:1024; ports:[31000-32000]' with principal
> 'test-principal'
> I0926 00:01:11.484340 1840 master.cpp:6672] Authorized registration of agent
> at slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal)
> I0926 00:01:11.484375 1840 master.cpp:6787] Registering agent at
> slave(771)@172.16.10.227:43103 (ip-172-16-10-227.ec2.internal) with id
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0
> I0926 00:01:11.484493 1840 registrar.cpp:487] Applied 1 operations in
> 38466ns; attempting to update the registry
> I0926 00:01:11.484637 1835 registrar.cpp:544] Successfully updated the
> registry in 0ns
> I0926 00:01:11.484691 1834 master.cpp:6835] Admitted agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103
> (ip-172-16-10-227.ec2.internal)
> I0926 00:01:11.484822 1834 master.cpp:6880] Registered agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103
> (ip-172-16-10-227.ec2.internal) with cpus:2; mem:1024; disk:1024;
> ports:[31000-32000]
> I0926 00:01:11.484916 1834 hierarchical.cpp:601] Added agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0 (ip-172-16-10-227.ec2.internal) with
> cpus:2; mem:1024; disk:1024; ports:[31000-32000] (allocated: {})
> I0926 00:01:11.484963 1834 hierarchical.cpp:1564] Performed allocation for 1
> agents in 9501ns
> I0926 00:01:11.484994 1834 slave.cpp:1479] Registered with master
> [email protected]:43103; given agent ID
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0
> I0926 00:01:11.485126 1834 slave.cpp:1499] Checkpointing SlaveInfo to
> '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/meta/slaves/f0a77094-641d-45e5-a890-46f2c2d8202c-S0/slave.info'
> I0926 00:01:11.485360 1835 task_status_update_manager.cpp:188] Resuming
> sending task status updates
> I0926 00:01:11.485404 1834 slave.cpp:1548] Forwarding agent update
> {"operations":{},"resource_version_uuid":{"value":"TUGtB/13S1q/oBij0hB5rA=="},"slave_id":{"value":"f0a77094-641d-45e5-a890-46f2c2d8202c-S0"},"update_oversubscribed_resources":false}
> I0926 00:01:11.485775 1836 master.cpp:7939] Ignoring update on agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103
> (ip-172-16-10-227.ec2.internal) as it reports no changes
> I0926 00:01:11.487349 1836 process.cpp:3569] Handling HTTP event for process
> 'slave(771)' with path: '/slave(771)/api/v1'
> I0926 00:01:11.487823 1840 http.cpp:1177] HTTP POST for /slave(771)/api/v1
> from 172.16.10.227:35910
> I0926 00:01:11.487898 1840 http.cpp:2074] Processing GET_CONTAINERS call
> I0926 00:01:11.490869 1836 container_daemon.cpp:121] Launching container
> 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:11.491497 1833 process.cpp:3569] Handling HTTP event for process
> 'slave(771)' with path: '/slave(771)/api/v1'
> I0926 00:01:11.491904 1833 http.cpp:1177] HTTP POST for /slave(771)/api/v1
> from 172.16.10.227:35912
> I0926 00:01:11.492103 1833 http.cpp:2440] Processing LAUNCH_CONTAINER call
> for container
> 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:11.492333 1833 http.cpp:2556] Creating sandbox
> '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:11.493259 1836 containerizer.cpp:1280] Starting container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.493693 1836 containerizer.cpp:1446] Checkpointed
> ContainerConfig at
> '/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE/config'
> I0926 00:01:11.493710 1836 containerizer.cpp:3118] Transitioning the state
> of container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> from PROVISIONING to PREPARING
> I0926 00:01:11.493806 1834 cgroups.cpp:429] Creating cgroup at
> '/sys/fs/cgroup/cpu,cpuacct/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> for container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.494220 1834 cgroups.cpp:429] Creating cgroup at
> '/sys/fs/cgroup/memory/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> for container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.494746 1834 memory.cpp:478] Started listening for OOM events
> for container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.494796 1834 memory.cpp:590] Started listening on 'low' memory
> pressure events for container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.494828 1834 memory.cpp:590] Started listening on 'medium'
> memory pressure events for container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.494858 1834 memory.cpp:590] Started listening on 'critical'
> memory pressure events for container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.495095 1836 memory.cpp:198] Updated
> 'memory.soft_limit_in_bytes' to 1GB for container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.495132 1834 cpu.cpp:92] Updated 'cpu.shares' to 102 (cpus
> 0.1) for container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.495165 1836 memory.cpp:227] Updated 'memory.limit_in_bytes'
> to 1GB for container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.495867 1834 containerizer.cpp:1939] Launching
> 'mesos-containerizer' with flags '--help="false"
> --launch_info="{"command":{"arguments":["/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin","--available_capacity=4GB","--volumes=","--work_dir=/tmp/r7LBlp/test_csi_plugin"],"shell":false,"value":"/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin"},"environment":{"variables":[{"name":"MESOS_SANDBOX","type":"VALUE","value":"/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE"},{"name":"CSI_ENDPOINT","type":"VALUE","value":"unix:///tmp/mesos-csi-wGzvgE/endpoint.sock"}]},"task_environment":{},"working_directory":"/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE"}"
> --pipe_read="75" --pipe_write="76"
> --runtime_directory="/tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_06lIVO/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE"
> --unshare_namespace_mnt="false"'
> I0926 00:01:11.496028 1840 linux_launcher.cpp:492] Launching container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> and cloning with namespaces
> I0926 00:01:11.521980 1834 containerizer.cpp:3118] Transitioning the state
> of container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> from PREPARING to ISOLATING
> I0926 00:01:11.523646 1839 containerizer.cpp:3118] Transitioning the state
> of container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> from ISOLATING to FETCHING
> I0926 00:01:11.523715 1839 fetcher.cpp:369] Starting to fetch URIs for
> container:
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE,
> directory:
> /tmp/StorageLocalResourceProviderTest_ReconcileUnacknowledgedTerminalOperation_YdUE86/containers/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:11.523888 1839 containerizer.cpp:3118] Transitioning the state
> of container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> from FETCHING to RUNNING
> I0926 00:01:11.525076 1835 container_daemon.cpp:140] Invoking post-start
> hook for container
> 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:11.525574 1835 hierarchical.cpp:1564] Performed allocation for 1
> agents in 18948ns
> I0926 00:01:11.576836 1836 hierarchical.cpp:1564] Performed allocation for 1
> agents in 19631ns
> I0926 00:01:11.627815 1835 hierarchical.cpp:1564] Performed allocation for 1
> agents in 21050ns
> I0926 00:01:11.639616 6260 test_csi_plugin.cpp:302] ProbeRequest '{}'
> I0926 00:01:11.640031 1839 container_daemon.cpp:171] Waiting for container
> 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:11.640367 6260 test_csi_plugin.cpp:274] GetPluginInfoRequest '{}'
> I0926 00:01:11.640795 1836 process.cpp:3569] Handling HTTP event for process
> 'slave(771)' with path: '/slave(771)/api/v1'
> I0926 00:01:11.640939 1836 provider.cpp:2178] Node plugin loaded:
> {"name":"org.apache.mesos.csi.test","vendorVersion":"1.8.0"}
> I0926 00:01:11.641376 1834 http.cpp:1177] HTTP POST for /slave(771)/api/v1
> from 172.16.10.227:35914
> I0926 00:01:11.641413 6261 test_csi_plugin.cpp:288]
> GetPluginCapabilitiesRequest '{}'
> I0926 00:01:11.641479 1834 http.cpp:2670] Processing WAIT_CONTAINER call for
> container
> 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:11.641993 6261 test_csi_plugin.cpp:274] GetPluginInfoRequest '{}'
> I0926 00:01:11.642236 1839 provider.cpp:2219] Controller plugin loaded:
> {"name":"org.apache.mesos.csi.test","vendorVersion":"1.8.0"}
> I0926 00:01:11.642463 6261 test_csi_plugin.cpp:593]
> ControllerGetCapabilitiesRequest '{}'
> I0926 00:01:11.642946 6261 test_csi_plugin.cpp:870]
> NodeGetCapabilitiesRequest '{}'
> I0926 00:01:11.643378 6261 test_csi_plugin.cpp:857] NodeGetIdRequest '{}'
> I0926 00:01:11.643676 1839 provider.cpp:661] Finished recovery for resource
> provider with type 'org.apache.mesos.rp.local.storage' and name 'test
> I0926 00:01:11.643731 1834 status_update_manager_process.hpp:379] Pausing
> operation status update manager
> I0926 00:01:11.643767 1839 http_connection.hpp:227] New endpoint detected at
> http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:11.644228 1838 http_connection.hpp:283] Connected with the
> remote endpoint at
> http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:11.644340 1839 provider.cpp:524] Connected to resource provider
> manager
> I0926 00:01:11.644498 1835 http_connection.hpp:131] Sending 1 call to
> http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:11.644778 1837 process.cpp:3569] Handling HTTP event for process
> 'slave(771)' with path: '/slave(771)/api/v1/resource_provider'
> I0926 00:01:11.645036 1840 http.cpp:1177] HTTP POST for
> /slave(771)/api/v1/resource_provider from 172.16.10.227:35918
> I0926 00:01:11.645167 1840 manager.cpp:736] Subscribing resource provider
> {"default_reservations":[{"role":"storage","type":"DYNAMIC"}],"name":"test","storage":{"plugin":{"containers":[{"command":{"arguments":["/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin","--available_capacity=4GB","--volumes=","--work_dir=/tmp/r7LBlp/test_csi_plugin"],"shell":false,"value":"/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin"},"resources":[{"name":"cpus","scalar":{"value":0.1},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"}],"services":["CONTROLLER_SERVICE","NODE_SERVICE"]}],"name":"test_csi_plugin","type":"org.apache.mesos.csi.test"}},"type":"org.apache.mesos.rp.local.storage"}
> I0926 00:01:11.647924 1840 slave.cpp:7643] Handling resource provider
> message 'SUBSCRIBE:
> {"default_reservations":[{"role":"storage","type":"DYNAMIC"}],"id":{"value":"520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2"},"name":"test","storage":{"plugin":{"containers":[{"command":{"arguments":["/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin","--available_capacity=4GB","--volumes=","--work_dir=/tmp/r7LBlp/test_csi_plugin"],"shell":false,"value":"/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-16.04/mesos/build/src/test-csi-plugin"},"resources":[{"name":"cpus","scalar":{"value":0.1},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"}],"services":["CONTROLLER_SERVICE","NODE_SERVICE"]}],"name":"test_csi_plugin","type":"org.apache.mesos.csi.test"}},"type":"org.apache.mesos.rp.local.storage"}'
> I0926 00:01:11.648219 1837 provider.cpp:546] Received SUBSCRIBED event
> I0926 00:01:11.648238 1837 provider.cpp:1496] Subscribed with ID
> 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
> I0926 00:01:11.648468 1836 status_update_manager_process.hpp:314] Recovering
> operation status update manager
> I0926 00:01:11.648780 6261 test_csi_plugin.cpp:531] ListVolumesRequest '{}'
> I0926 00:01:11.649070 1839 provider.cpp:3519] Sending UPDATE_STATE call with
> resources '{}' and 0 operations to agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0
> I0926 00:01:11.649124 1839 provider.cpp:1110] Resource provider
> 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 is in READY state
> I0926 00:01:11.649148 1834 http_connection.hpp:131] Sending 3 call to
> http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:11.649207 1839 status_update_manager_process.hpp:385] Resuming
> operation status update manager
> I0926 00:01:11.649250 1835 provider.cpp:1422] Updating profiles { test } for
> resource provider 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
> I0926 00:01:11.649550 1837 process.cpp:3569] Handling HTTP event for process
> 'slave(771)' with path: '/slave(771)/api/v1/resource_provider'
> I0926 00:01:11.649752 1837 http.cpp:1177] HTTP POST for
> /slave(771)/api/v1/resource_provider from 172.16.10.227:35916
> I0926 00:01:11.649860 1837 manager.cpp:917] Received UPDATE_STATE call with
> resources '[]' and 0 operations from resource provider
> 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
> I0926 00:01:11.649793 6261 test_csi_plugin.cpp:559] GetCapacityRequest
> '{"volumeCapabilities":[{"mount":{},"accessMode":{"mode":"SINGLE_NODE_WRITER"}}]}'
> I0926 00:01:11.649943 1837 slave.cpp:7643] Handling resource provider
> message 'UPDATE_STATE: 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2 {}'
> I0926 00:01:11.649966 1837 slave.cpp:7763] Forwarding new total resources
> cpus:2; mem:1024; disk:1024; ports:[31000-32000]
> I0926 00:01:11.650372 1837 provider.cpp:1297] Removing '{}' and adding
> 'disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096' to the total
> resources
> I0926 00:01:11.650480 1833 hierarchical.cpp:740] Grew agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0 by {} (total), { } (used)
> I0926 00:01:11.650538 1833 hierarchical.cpp:697] Agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0 (ip-172-16-10-227.ec2.internal)
> updated with total resources cpus:2; mem:1024; disk:1024; ports:[31000-32000]
> I0926 00:01:11.650691 1837 provider.cpp:3519] Sending UPDATE_STATE call with
> resources 'disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096' and 0
> operations to agent f0a77094-641d-45e5-a890-46f2c2d8202c-S0
> I0926 00:01:11.650811 1833 http_connection.hpp:131] Sending 3 call to
> http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:11.651093 1840 process.cpp:3569] Handling HTTP event for process
> 'slave(771)' with path: '/slave(771)/api/v1/resource_provider'
> I0926 00:01:11.651756 11812 scheduler.cpp:189] Version: 1.8.0
> I0926 00:01:11.652252 1838 scheduler.cpp:355] Using default 'basic' HTTP
> authenticatee
> I0926 00:01:11.652364 1836 scheduler.cpp:538] New master detected at
> [email protected]:43103
> I0926 00:01:11.652377 1836 scheduler.cpp:547] Waiting for 0ns before
> initiating a re-(connection) attempt with the master
> I0926 00:01:11.652827 1835 scheduler.cpp:429] Connected with the master at
> http://172.16.10.227:43103/master/api/v1/scheduler
> I0926 00:01:11.653015 1838 scheduler.cpp:248] Sending SUBSCRIBE call to
> http://172.16.10.227:43103/master/api/v1/scheduler
> I0926 00:01:11.653323 1833 process.cpp:3569] Handling HTTP event for process
> 'master' with path: '/master/api/v1/scheduler'
> I0926 00:01:11.653650 1840 http.cpp:1177] HTTP POST for
> /master/api/v1/scheduler from 172.16.10.227:35922
> I0926 00:01:11.653709 1840 master.cpp:2502] Received subscription request
> for HTTP framework 'default'
> I0926 00:01:11.653733 1840 master.cpp:2155] Authorizing framework principal
> 'test-principal' to receive offers for roles '{ storage }'
> I0926 00:01:11.653841 1834 master.cpp:2637] Subscribing framework 'default'
> with checkpointing disabled and capabilities [ MULTI_ROLE,
> RESERVATION_REFINEMENT ]
> I0926 00:01:11.654402 1834 master.cpp:9883] Adding framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) with roles { } suppressed
> I0926 00:01:11.654527 1837 hierarchical.cpp:306] Added framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000
> I0926 00:01:11.654933 1836 scheduler.cpp:845] Enqueuing event SUBSCRIBED
> received from http://172.16.10.227:43103/master/api/v1/scheduler
> I0926 00:01:11.654948 1837 hierarchical.cpp:1564] Performed allocation for 1
> agents in 140494ns
> I0926 00:01:11.655035 1835 master.cpp:9468] Sending offers [
> f0a77094-641d-45e5-a890-46f2c2d8202c-O0 ] to framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
> I0926 00:01:11.689805 1839 http.cpp:1177] HTTP POST for
> /slave(771)/api/v1/resource_provider from 172.16.10.227:35916
> I0926 00:01:13.959466 1834 manager.cpp:917] Received UPDATE_STATE call with
> resources
> '[{"disk":{"source":{"profile":"test","type":"RAW"}},"name":"disk","provider_id":{"value":"520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2"},"reservations":[{"role":"storage","type":"DYNAMIC"}],"scalar":{"value":4096.0},"type":"SCALAR"}]'
> and 0 operations from resource provider 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
> I0926 00:01:13.960250 1839 slave.cpp:7643] Handling resource provider
> message 'UPDATE_STATE: 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
> disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096'
> I0926 00:01:13.960312 1839 slave.cpp:7763] Forwarding new total resources
> cpus:2; mem:1024; disk:1024; ports:[31000-32000]; disk(reservations:
> [(DYNAMIC,storage)])[RAW(,test)]:4096
> I0926 00:01:13.960785 1838 hierarchical.cpp:697] Agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0 (ip-172-16-10-227.ec2.internal)
> updated with total resources cpus:2; mem:1024; disk:1024;
> ports:[31000-32000]; disk(reservations: [(DYNAMIC,storage)])[RAW(,test)]:4096
> I0926 00:01:13.960876 1838 hierarchical.cpp:1564] Performed allocation for 1
> agents in 40437ns
> I0926 00:01:13.965555 1836 scheduler.cpp:845] Enqueuing event HEARTBEAT
> received from http://172.16.10.227:43103/master/api/v1/scheduler
> I0926 00:01:13.966929 1836 scheduler.cpp:845] Enqueuing event OFFERS
> received from http://172.16.10.227:43103/master/api/v1/scheduler
> I0926 00:01:13.967149 1838 scheduler.cpp:248] Sending DECLINE call to
> http://172.16.10.227:43103/master/api/v1/scheduler
> I0926 00:01:13.967648 1835 process.cpp:3569] Handling HTTP event for process
> 'master' with path: '/master/api/v1/scheduler'
> I0926 00:01:13.967850 1837 hierarchical.cpp:1564] Performed allocation for 1
> agents in 33598ns
> I0926 00:01:13.968111 1833 http.cpp:1177] HTTP POST for
> /master/api/v1/scheduler from 172.16.10.227:35920
> I0926 00:01:13.968158 1833 master.cpp:5850] Processing DECLINE call for
> offers: [ f0a77094-641d-45e5-a890-46f2c2d8202c-O0 ] for framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
> I0926 00:01:13.968245 1833 master.cpp:11462] Removing offer
> f0a77094-641d-45e5-a890-46f2c2d8202c-O0
> I0926 00:01:13.968375 1833 hierarchical.cpp:1236] Recovered cpus(allocated:
> storage):2; mem(allocated: storage):1024; disk(allocated: storage):1024;
> ports(allocated: storage):[31000-32000] (total: cpus:2; mem:1024; disk:1024;
> ports:[31000-32000]; disk(reservations:
> [(DYNAMIC,storage)])[RAW(,test)]:4096, allocated: {}) on agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0 from framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000
> I0926 00:01:13.968408 1833 hierarchical.cpp:1282] Framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000 filtered agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0 for 5secs
> ../../src/tests/storage_local_resource_provider_tests.cpp:4286: Failure
> Failed to wait 15secs for offers
> I0926 00:01:28.977944 1833 master.cpp:1366] Framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) disconnected
> I0926 00:01:28.977985 1833 master.cpp:3230] Deactivating framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
> I0926 00:01:28.978009 1833 master.cpp:3207] Disconnecting framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
> I0926 00:01:28.978024 1833 master.cpp:1381] Giving framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default) 0ns to failover
> I0926 00:01:28.978085 1833 hierarchical.cpp:420] Deactivated framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000
> I0926 00:01:28.978204 1837 master.cpp:9261] Framework failover timeout,
> removing framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
> I0926 00:01:28.978226 1837 master.cpp:10197] Removing framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000 (default)
> I0926 00:01:28.978302 1840 slave.cpp:3896] Asked to shut down framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000 by [email protected]:43103
> I0926 00:01:28.978327 1840 slave.cpp:3911] Cannot shut down unknown
> framework f0a77094-641d-45e5-a890-46f2c2d8202c-0000
> I0926 00:01:28.978341 1837 hierarchical.cpp:359] Removed framework
> f0a77094-641d-45e5-a890-46f2c2d8202c-0000
> ../../src/tests/storage_local_resource_provider_tests.cpp:4266: Failure
> Actual function call count doesn't match EXPECT_CALL(*scheduler, offers(_,
> v1::scheduler::OffersHaveAnyResource(isRaw)))...
> Expected: to be called once
> Actual: never called - unsatisfied and active
> I0926 00:01:28.978569 11812 slave.cpp:909] Agent terminating
> I0926 00:01:28.978735 11812 manager.cpp:161] Terminating resource provider
> 520ef77a-5d14-4c9f-8077-0a6c4ae0b0b2
> E0926 00:01:28.979059 1836 http_connection.hpp:452] End-Of-File received
> I0926 00:01:28.979130 1836 http_connection.hpp:217] Re-detecting endpoint
> I0926 00:01:28.979238 1836 http_connection.hpp:227] New endpoint detected at
> http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:28.979274 1839 provider.cpp:536] Disconnected from resource
> provider manager
> I0926 00:01:28.979276 1836 http_connection.hpp:338] Ignoring disconnection
> attempt from stale connection
> I0926 00:01:28.979300 1836 http_connection.hpp:338] Ignoring disconnection
> attempt from stale connection
> I0926 00:01:28.979308 1839 status_update_manager_process.hpp:379] Pausing
> operation status update manager
> I0926 00:01:28.979518 1834 master.cpp:1251] Agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103
> (ip-172-16-10-227.ec2.internal) disconnected
> I0926 00:01:28.979540 1834 master.cpp:3267] Disconnecting agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103
> (ip-172-16-10-227.ec2.internal)
> I0926 00:01:28.979560 1834 master.cpp:3286] Deactivating agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0 at slave(771)@172.16.10.227:43103
> (ip-172-16-10-227.ec2.internal)
> I0926 00:01:28.979599 1834 hierarchical.cpp:795] Agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0 deactivated
> I0926 00:01:28.979914 1839 containerizer.cpp:2455] Destroying container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> in RUNNING state
> I0926 00:01:28.979934 1839 containerizer.cpp:3118] Transitioning the state
> of container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> from RUNNING to DESTROYING
> I0926 00:01:28.980154 1839 linux_launcher.cpp:580] Asked to destroy
> container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:28.980207 1839 linux_launcher.cpp:622] Destroying cgroup
> '/sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:28.980270 1840 http_connection.hpp:283] Connected with the
> remote endpoint at
> http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:28.980406 1840 hierarchical.cpp:1564] Performed allocation for 1
> agents in 26143ns
> I0926 00:01:28.980520 1836 provider.cpp:524] Connected to resource provider
> manager
> I0926 00:01:28.980541 1838 cgroups.cpp:2838] Freezing cgroup
> /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:28.980742 1835 http_connection.hpp:131] Sending 1 call to
> http://172.16.10.227:43103/slave(771)/api/v1/resource_provider
> I0926 00:01:28.980916 1838 cgroups.cpp:1229] Successfully froze cgroup
> /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> after 349184ns
> I0926 00:01:28.981250 1837 cgroups.cpp:2856] Thawing cgroup
> /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:28.981411 1837 cgroups.cpp:1258] Successfully thawed cgroup
> /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff/org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> after 124928ns
> I0926 00:01:28.981528 1841 process.cpp:2735] Returning '404 Not Found' for
> '/slave(771)/api/v1/resource_provider'
> E0926 00:01:28.981875 1840 provider.cpp:1067] Failed to subscribe resource
> provider with type 'org.apache.mesos.rp.local.storage' and name 'test':
> Received '404 Not Found' ()
> I0926 00:01:29.031200 1835 hierarchical.cpp:1564] Performed allocation for 1
> agents in 26888ns
> I0926 00:01:29.081390 1836 hierarchical.cpp:1564] Performed allocation for 1
> agents in 21841ns
> I0926 00:01:29.081869 1836 containerizer.cpp:2957] Container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> has exited
> I0926 00:01:29.083161 1839 provisioner.cpp:597] Ignoring destroy request for
> unknown container
> org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE
> I0926 00:01:29.083875 1834 container_daemon.cpp:189] Invoking post-stop hook
> for container
> 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:29.084064 1834 container_daemon.cpp:121] Launching container
> 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE'
> I0926 00:01:29.086195 1841 process.cpp:2735] Returning '404 Not Found' for
> '/slave(771)/api/v1'
> E0926 00:01:29.086671 1833 container_daemon.cpp:150] Failed to launch
> container
> 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE':
> Failed to launch container
> 'org-apache-mesos-rp-local-storage-test--org-apache-mesos-csi-test-test_csi_plugin--CONTROLLER_SERVICE-NODE_SERVICE':
> Unexpected response '404 Not Found' ()
> I0926 00:01:29.099272 11812 master.cpp:1093] Master terminating
> I0926 00:01:29.099445 1836 hierarchical.cpp:637] Removed agent
> f0a77094-641d-45e5-a890-46f2c2d8202c-S0
> I0926 00:01:29.113350 1837 cgroups.cpp:2838] Freezing cgroup
> /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff
> I0926 00:01:29.113492 1837 cgroups.cpp:1229] Successfully froze cgroup
> /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff after
> 99072ns
> I0926 00:01:29.113646 1837 cgroups.cpp:2856] Thawing cgroup
> /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff
> I0926 00:01:29.113750 1837 cgroups.cpp:1258] Successfully thawed cgroup
> /sys/fs/cgroup/freezer/mesos_test_a71fca40-7489-4600-b9b0-b37c10625cff after
> 72960ns
> [ FAILED ]
> StorageLocalResourceProviderTest.ReconcileUnacknowledgedTerminalOperation
> (17703 ms)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
