[
https://issues.apache.org/jira/browse/MESOS-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156668#comment-16156668
]
Benjamin Bannier commented on MESOS-7111:
-----------------------------------------
I see a similar failure on fedora25 with clang-trunk from
{{HttpFaultToleranceTest.SchedulerExit}}.
> HttpFaultToleranceTest.SchedulerFailoverFrameworkToExecutorMessage segfaults
> ----------------------------------------------------------------------------
>
> Key: MESOS-7111
> URL: https://issues.apache.org/jira/browse/MESOS-7111
> Project: Mesos
> Issue Type: Bug
> Components: test
> Affects Versions: 1.2.0
> Environment: ubuntu-16
> Reporter: Benjamin Bannier
> Labels: mesosphere
>
> We observed a segfault in
> {{HttpFaultToleranceTest.SchedulerFailoverFrameworkToExecutorMessage}} in
> internal CI on an ubuntu16 machine. Note that ubuntu16 uses gcc-6.
> {code}
> [ RUN ]
> HttpFaultToleranceTest.SchedulerFailoverFrameworkToExecutorMessage
> I0210 02:47:31.260174 19578 cluster.cpp:160] Creating default 'local'
> authorizer
> I0210 02:47:31.261225 19597 master.cpp:383] Master
> d8129420-2a04-48e7-9b28-6b0a0af73168 (ip-10-150-111-24.ec2.internal) started
> on 10.150.111.24:33608
> I0210 02:47:31.261281 19597 master.cpp:385] Flags at startup: --acls=""
> --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins"
> --allocation_interval="1secs" --allocator="HierarchicalDRF"
> --authenticate_agents="true" --authenticate_frameworks="false"
> --authenticate_http_frameworks="true" --authenticate_http_readonly="true"
> --authenticate_http_readwrite="true" --authenticators="crammd5"
> --authorizers="local" --credentials="/tmp/fBrqHi/credentials"
> --framework_sorter="drf" --help="false" --hostname_lookup="true"
> --http_authenticators="basic" --http_framework_authenticators="basic"
> --initialize_driver_logging="true" --log_auto_initialize="true"
> --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5"
> --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000"
> --max_unreachable_tasks_per_framework="1000" --quiet="false"
> --recovery_agent_removal_limit="100%" --registry="in_memory"
> --registry_fetch_timeout="1mins" --registry_gc_interval="15mins"
> --registry_max_agent_age="2weeks" --registry_max_agent_count="102400"
> --registry_store_timeout="100secs" --registry_strict="false"
> --root_submissions="true" --user_sorter="drf" --version="false"
> --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/fBrqHi/master"
> --zk_session_timeout="10secs"
> I0210 02:47:31.261404 19597 master.cpp:437] Master allowing unauthenticated
> frameworks to register
> I0210 02:47:31.261411 19597 master.cpp:449] Master only allowing
> authenticated agents to register
> I0210 02:47:31.261415 19597 master.cpp:462] Master only allowing
> authenticated HTTP frameworks to register
> I0210 02:47:31.261420 19597 credentials.hpp:37] Loading credentials for
> authentication from '/tmp/fBrqHi/credentials'
> I0210 02:47:31.261488 19597 master.cpp:507] Using default 'crammd5'
> authenticator
> I0210 02:47:31.261530 19597 http.cpp:919] Using default 'basic' HTTP
> authenticator for realm 'mesos-master-readonly'
> I0210 02:47:31.261591 19597 http.cpp:919] Using default 'basic' HTTP
> authenticator for realm 'mesos-master-readwrite'
> I0210 02:47:31.261631 19597 http.cpp:919] Using default 'basic' HTTP
> authenticator for realm 'mesos-master-scheduler'
> I0210 02:47:31.261698 19597 master.cpp:587] Authorization enabled
> I0210 02:47:31.261754 19601 whitelist_watcher.cpp:77] No whitelist given
> I0210 02:47:31.261754 19602 hierarchical.cpp:161] Initialized hierarchical
> allocator process
> I0210 02:47:31.262462 19597 master.cpp:2124] Elected as the leading master!
> I0210 02:47:31.262482 19597 master.cpp:1646] Recovering from registrar
> I0210 02:47:31.262545 19603 registrar.cpp:329] Recovering registrar
> I0210 02:47:31.262774 19602 registrar.cpp:362] Successfully fetched the
> registry (0B) in 201984ns
> I0210 02:47:31.262809 19602 registrar.cpp:461] Applied 1 operations in
> 2963ns; attempting to update the registry
> I0210 02:47:31.263062 19599 registrar.cpp:506] Successfully updated the
> registry in 214016ns
> I0210 02:47:31.263119 19599 registrar.cpp:392] Successfully recovered
> registrar
> I0210 02:47:31.263267 19597 master.cpp:1762] Recovered 0 agents from the
> registry (172B); allowing 10mins for agents to re-register
> I0210 02:47:31.263295 19598 hierarchical.cpp:188] Skipping recovery of
> hierarchical allocator: nothing to recover
> I0210 02:47:31.264645 19578 cluster.cpp:446] Creating default 'local'
> authorizer
> I0210 02:47:31.265029 19598 slave.cpp:211] Mesos agent started on
> (105)@10.150.111.24:33608
> I0210 02:47:31.265187 19578 scheduler.cpp:184] Version: 1.3.0
> I0210 02:47:31.265043 19598 slave.cpp:212] Flags at startup: --acls=""
> --appc_simple_discovery_uri_prefix="http://";
> --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="true"
> --authenticate_http_readwrite="true" --authenticatee="crammd5"
> --authentication_backoff_factor="1secs" --authorizer="local"
> --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false"
> --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false"
> --cgroups_root="mesos" --container_disk_watch_interval="15secs"
> --containerizers="mesos"
> --credential="/tmp/HttpFaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_I1eg7W/credential"
> --default_role="*" --disk_watch_interval="1mins" --docker="docker"
> --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io";
> --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock"
> --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker"
> --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume"
> --enforce_container_disk_quota="false"
> --executor_registration_timeout="1mins"
> --executor_shutdown_grace_period="5secs"
> --fetcher_cache_dir="/tmp/HttpFaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_I1eg7W/fetch"
> --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks"
> --gc_disk_headroom="0.1" --hadoop_home="" --help="false"
> --hostname_lookup="true" --http_authenticators="basic"
> --http_command_executor="false"
> --http_credentials="/tmp/HttpFaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_I1eg7W/http_credentials"
> --http_heartbeat_interval="30secs" --initialize_driver_logging="true"
> --isolation="posix/cpu,posix/mem" --launcher="linux"
> --launcher_dir="/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/Plain/label/mesos-ec2-ubuntu-16.04/mesos/build/src"
> --logbufsecs="0" --logging_level="INFO"
> --max_completed_executors_per_framework="150"
> --oversubscribed_resources_interval="15secs" --perf_duration="10secs"
> --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false"
> --recover="reconnect" --recovery_timeout="15mins"
> --registration_backoff_factor="10ms"
> --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]"
> --revocable_cpu_low_priority="true"
> --runtime_dir="/tmp/HttpFaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_I1eg7W"
> --sandbox_directory="/mnt/mesos/sandbox" --strict="true"
> --switch_user="true" --systemd_enable_support="true"
> --systemd_runtime_directory="/run/systemd/system" --version="false"
> --work_dir="/tmp/HttpFaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_g77kUC"
> I0210 02:47:31.265250 19598 credentials.hpp:86] Loading credential for
> authentication from
> '/tmp/HttpFaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_I1eg7W/credential'
> I0210 02:47:31.265290 19598 slave.cpp:354] Agent using credential for:
> test-principal
> I0210 02:47:31.265303 19598 credentials.hpp:37] Loading credentials for
> authentication from
> '/tmp/HttpFaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_I1eg7W/http_credentials'
> I0210 02:47:31.265349 19598 http.cpp:919] Using default 'basic' HTTP
> authenticator for realm 'mesos-agent-readonly'
> I0210 02:47:31.265388 19598 http.cpp:919] Using default 'basic' HTTP
> authenticator for realm 'mesos-agent-readwrite'
> I0210 02:47:31.265653 19599 scheduler.cpp:470] New master detected at
> [email protected]:33608
> I0210 02:47:31.265671 19599 scheduler.cpp:479] Waiting for 0ns before
> initiating a re-(connection) attempt with the master
> I0210 02:47:31.266012 19598 slave.cpp:541] Agent resources: cpus(*):2;
> mem(*):1024; disk(*):1024; ports(*):[31000-32000]
> I0210 02:47:31.266049 19598 slave.cpp:549] Agent attributes: [ ]
> I0210 02:47:31.266055 19598 slave.cpp:554] Agent hostname:
> ip-10-150-111-24.ec2.internal
> I0210 02:47:31.266111 19602 status_update_manager.cpp:177] Pausing sending
> status updates
> I0210 02:47:31.266312 19604 state.cpp:62] Recovering state from
> '/tmp/HttpFaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_g77kUC/meta'
> I0210 02:47:31.266428 19597 status_update_manager.cpp:203] Recovering status
> update manager
> I0210 02:47:31.266559 19600 slave.cpp:5555] Finished recovery
> I0210 02:47:31.266736 19600 slave.cpp:5729] Querying resource estimator for
> oversubscribable resources
> I0210 02:47:31.266881 19600 status_update_manager.cpp:177] Pausing sending
> status updates
> I0210 02:47:31.266877 19598 slave.cpp:931] New master detected at
> [email protected]:33608
> I0210 02:47:31.266935 19598 slave.cpp:966] Detecting new master
> I0210 02:47:31.266990 19598 slave.cpp:5743] Received oversubscribable
> resources {} from the resource estimator
> I0210 02:47:31.267329 19603 scheduler.cpp:361] Connected with the master at
> http://10.150.111.24:33608/master/api/v1/scheduler
> I0210 02:47:31.267763 19601 scheduler.cpp:243] Sending SUBSCRIBE call to
> http://10.150.111.24:33608/master/api/v1/scheduler
> I0210 02:47:31.268118 19597 process.cpp:3704] Handling HTTP event for process
> 'master' with path: '/master/api/v1/scheduler'
> I0210 02:47:31.268470 19600 http.cpp:420] HTTP POST for
> /master/api/v1/scheduler from 10.150.111.24:58632
> I0210 02:47:31.268532 19600 master.cpp:2423] Received subscription request
> for HTTP framework 'default'
> I0210 02:47:31.268556 19600 master.cpp:2160] Authorizing framework principal
> 'test-principal' to receive offers for role '*'
> I0210 02:47:31.268748 19599 master.cpp:2537] Subscribing framework 'default'
> with checkpointing disabled and capabilities [ ]
> I0210 02:47:31.268887 19597 master.hpp:2119] Sending heartbeat to
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000
> I0210 02:47:31.268909 19604 hierarchical.cpp:286] Added framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000
> I0210 02:47:31.268954 19604 hierarchical.cpp:1810] No allocations performed
> I0210 02:47:31.268977 19604 hierarchical.cpp:1900] No inverse offers to send
> out!
> I0210 02:47:31.268992 19604 hierarchical.cpp:1394] Performed allocation for 0
> agents in 45649ns
> I0210 02:47:31.269172 19598 scheduler.cpp:676] Enqueuing event SUBSCRIBED
> received from http://10.150.111.24:33608/master/api/v1/scheduler
> I0210 02:47:31.269356 19598 scheduler.cpp:676] Enqueuing event HEARTBEAT
> received from http://10.150.111.24:33608/master/api/v1/scheduler
> I0210 02:47:31.273422 19602 slave.cpp:993] Authenticating with master
> [email protected]:33608
> I0210 02:47:31.273448 19602 slave.cpp:1004] Using default CRAM-MD5
> authenticatee
> I0210 02:47:31.273509 19599 authenticatee.cpp:121] Creating new client SASL
> connection
> I0210 02:47:31.273859 19599 master.cpp:7173] Authenticating
> slave(105)@10.150.111.24:33608
> I0210 02:47:31.273927 19598 authenticator.cpp:414] Starting authentication
> session for crammd5-authenticatee(241)@10.150.111.24:33608
> I0210 02:47:31.274020 19601 authenticator.cpp:98] Creating new server SASL
> connection
> I0210 02:47:31.274341 19601 authenticatee.cpp:213] Received SASL
> authentication mechanisms: CRAM-MD5
> I0210 02:47:31.274369 19601 authenticatee.cpp:239] Attempting to authenticate
> with mechanism 'CRAM-MD5'
> I0210 02:47:31.274416 19601 authenticator.cpp:204] Received SASL
> authentication start
> I0210 02:47:31.274457 19601 authenticator.cpp:326] Authentication requires
> more steps
> I0210 02:47:31.274494 19601 authenticatee.cpp:259] Received SASL
> authentication step
> I0210 02:47:31.274559 19598 authenticator.cpp:232] Received SASL
> authentication step
> I0210 02:47:31.274586 19598 auxprop.cpp:109] Request to lookup properties for
> user: 'test-principal' realm: 'ip-10-150-111-24.ec2.internal' server FQDN:
> 'ip-10-150-111-24.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false
> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
> I0210 02:47:31.274600 19598 auxprop.cpp:181] Looking up auxiliary property
> '*userPassword'
> I0210 02:47:31.274615 19598 auxprop.cpp:181] Looking up auxiliary property
> '*cmusaslsecretCRAM-MD5'
> I0210 02:47:31.274629 19598 auxprop.cpp:109] Request to lookup properties for
> user: 'test-principal' realm: 'ip-10-150-111-24.ec2.internal' server FQDN:
> 'ip-10-150-111-24.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false
> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
> I0210 02:47:31.274641 19598 auxprop.cpp:131] Skipping auxiliary property
> '*userPassword' since SASL_AUXPROP_AUTHZID == true
> I0210 02:47:31.274682 19598 auxprop.cpp:131] Skipping auxiliary property
> '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
> I0210 02:47:31.274701 19598 authenticator.cpp:318] Authentication success
> I0210 02:47:31.274760 19603 authenticatee.cpp:299] Authentication success
> I0210 02:47:31.274780 19601 authenticator.cpp:432] Authentication session
> cleanup for crammd5-authenticatee(241)@10.150.111.24:33608
> I0210 02:47:31.274793 19598 master.cpp:7203] Successfully authenticated
> principal 'test-principal' at slave(105)@10.150.111.24:33608
> I0210 02:47:31.274907 19603 slave.cpp:1088] Successfully authenticated with
> master [email protected]:33608
> I0210 02:47:31.274969 19603 slave.cpp:1515] Will retry registration in
> 1.825425ms if necessary
> I0210 02:47:31.275038 19602 master.cpp:5388] Registering agent at
> slave(105)@10.150.111.24:33608 (ip-10-150-111-24.ec2.internal) with id
> d8129420-2a04-48e7-9b28-6b0a0af73168-S0
> I0210 02:47:31.275151 19602 registrar.cpp:461] Applied 1 operations in
> 12479ns; attempting to update the registry
> I0210 02:47:31.275413 19600 registrar.cpp:506] Successfully updated the
> registry in 237056ns
> I0210 02:47:31.275595 19601 slave.cpp:4343] Received ping from
> slave-observer(98)@10.150.111.24:33608
> I0210 02:47:31.275621 19600 master.cpp:5462] Registered agent
> d8129420-2a04-48e7-9b28-6b0a0af73168-S0 at slave(105)@10.150.111.24:33608
> (ip-10-150-111-24.ec2.internal) with cpus(*):2; mem(*):1024; disk(*):1024;
> ports(*):[31000-32000]
> I0210 02:47:31.275703 19597 hierarchical.cpp:516] Added agent
> d8129420-2a04-48e7-9b28-6b0a0af73168-S0 (ip-10-150-111-24.ec2.internal) with
> cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: {})
> I0210 02:47:31.275722 19601 slave.cpp:1134] Registered with master
> [email protected]:33608; given agent ID
> d8129420-2a04-48e7-9b28-6b0a0af73168-S0
> I0210 02:47:31.275737 19601 fetcher.cpp:90] Clearing fetcher cache
> I0210 02:47:31.275805 19599 status_update_manager.cpp:184] Resuming sending
> status updates
> I0210 02:47:31.275915 19601 slave.cpp:1162] Checkpointing SlaveInfo to
> '/tmp/HttpFaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_g77kUC/meta/slaves/d8129420-2a04-48e7-9b28-6b0a0af73168-S0/slave.info'
> I0210 02:47:31.275993 19597 hierarchical.cpp:1900] No inverse offers to send
> out!
> I0210 02:47:31.276023 19597 hierarchical.cpp:1394] Performed allocation for 1
> agents in 263025ns
> I0210 02:47:31.276077 19601 slave.cpp:1200] Forwarding total oversubscribed
> resources {}
> I0210 02:47:31.276083 19602 master.cpp:7002] Sending 1 offers to framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (default)
> I0210 02:47:31.276250 19602 master.cpp:6019] Received update of agent
> d8129420-2a04-48e7-9b28-6b0a0af73168-S0 at slave(105)@10.150.111.24:33608
> (ip-10-150-111-24.ec2.internal) with total oversubscribed resources {}
> I0210 02:47:31.276347 19599 hierarchical.cpp:589] Agent
> d8129420-2a04-48e7-9b28-6b0a0af73168-S0 (ip-10-150-111-24.ec2.internal)
> updated with oversubscribed resources {} (total: cpus(*):2; mem(*):1024;
> disk(*):1024; ports(*):[31000-32000], allocated: cpus(*)(allocated: *):2;
> mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated:
> *):[31000-32000])
> I0210 02:47:31.276420 19599 hierarchical.cpp:1810] No allocations performed
> I0210 02:47:31.276445 19599 hierarchical.cpp:1900] No inverse offers to send
> out!
> I0210 02:47:31.276461 19599 hierarchical.cpp:1394] Performed allocation for 1
> agents in 56740ns
> I0210 02:47:31.276593 19598 scheduler.cpp:676] Enqueuing event OFFERS
> received from http://10.150.111.24:33608/master/api/v1/scheduler
> I0210 02:47:31.277225 19601 scheduler.cpp:243] Sending ACCEPT call to
> http://10.150.111.24:33608/master/api/v1/scheduler
> I0210 02:47:31.277544 19597 process.cpp:3704] Handling HTTP event for process
> 'master' with path: '/master/api/v1/scheduler'
> I0210 02:47:31.277899 19598 http.cpp:420] HTTP POST for
> /master/api/v1/scheduler from 10.150.111.24:58630
> I0210 02:47:31.278055 19598 master.cpp:3776] Processing ACCEPT call for
> offers: [ d8129420-2a04-48e7-9b28-6b0a0af73168-O0 ] on agent
> d8129420-2a04-48e7-9b28-6b0a0af73168-S0 at slave(105)@10.150.111.24:33608
> (ip-10-150-111-24.ec2.internal) for framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (default)
> I0210 02:47:31.278100 19598 master.cpp:3352] Authorizing framework principal
> 'test-principal' to launch task 512412f2-4982-435f-a963-8dc60a3fb03e
> W0210 02:47:31.278590 19598 validation.cpp:1064] Executor 'default' for task
> '512412f2-4982-435f-a963-8dc60a3fb03e' uses less CPUs (None) than the minimum
> required (0.01). Please update your executor, as this will be mandatory in
> future releases.
> W0210 02:47:31.278615 19598 validation.cpp:1076] Executor 'default' for task
> '512412f2-4982-435f-a963-8dc60a3fb03e' uses less memory (None) than the
> minimum required (32MB). Please update your executor, as this will be
> mandatory in future releases.
> I0210 02:47:31.278728 19598 master.cpp:9063] Adding task
> 512412f2-4982-435f-a963-8dc60a3fb03e with resources cpus(*)(allocated: *):2;
> mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated:
> *):[31000-32000] on agent d8129420-2a04-48e7-9b28-6b0a0af73168-S0 at
> slave(105)@10.150.111.24:33608 (ip-10-150-111-24.ec2.internal)
> I0210 02:47:31.278810 19598 master.cpp:4426] Launching task
> 512412f2-4982-435f-a963-8dc60a3fb03e of framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (default) with resources
> cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated:
> *):1024; ports(*)(allocated: *):[31000-32000] on agent
> d8129420-2a04-48e7-9b28-6b0a0af73168-S0 at slave(105)@10.150.111.24:33608
> (ip-10-150-111-24.ec2.internal)
> I0210 02:47:31.278962 19599 slave.cpp:1625] Got assigned task
> '512412f2-4982-435f-a963-8dc60a3fb03e' for framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000
> I0210 02:47:31.279145 19599 slave.cpp:1785] Launching task
> '512412f2-4982-435f-a963-8dc60a3fb03e' for framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000
> I0210 02:47:31.279255 19603 hierarchical.cpp:807] Updated allocation of
> framework d8129420-2a04-48e7-9b28-6b0a0af73168-0000 on agent
> d8129420-2a04-48e7-9b28-6b0a0af73168-S0 from cpus(*)(allocated: *):2;
> mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated:
> *):[31000-32000] to cpus(*)(allocated: *):2; mem(*)(allocated: *):1024;
> disk(*)(allocated: *):1024; ports(*)(allocated: *):[31000-32000]
> I0210 02:47:31.279412 19599 paths.cpp:547] Trying to chown
> '/tmp/HttpFaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_g77kUC/slaves/d8129420-2a04-48e7-9b28-6b0a0af73168-S0/frameworks/d8129420-2a04-48e7-9b28-6b0a0af73168-0000/executors/default/runs/f287deb2-0a21-4be0-ad54-613d7a35087d'
> to user 'root'
> I0210 02:47:31.279538 19599 slave.cpp:6477] Launching executor 'default' of
> framework d8129420-2a04-48e7-9b28-6b0a0af73168-0000 with resources {} in work
> directory
> '/tmp/HttpFaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_g77kUC/slaves/d8129420-2a04-48e7-9b28-6b0a0af73168-S0/frameworks/d8129420-2a04-48e7-9b28-6b0a0af73168-0000/executors/default/runs/f287deb2-0a21-4be0-ad54-613d7a35087d'
> I0210 02:47:31.279744 19599 slave.cpp:2115] Queued task
> '512412f2-4982-435f-a963-8dc60a3fb03e' for executor 'default' of framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000
> I0210 02:47:31.279788 19599 slave.cpp:884] Successfully attached file
> '/tmp/HttpFaultToleranceTest_SchedulerFailoverFrameworkToExecutorMessage_g77kUC/slaves/d8129420-2a04-48e7-9b28-6b0a0af73168-S0/frameworks/d8129420-2a04-48e7-9b28-6b0a0af73168-0000/executors/default/runs/f287deb2-0a21-4be0-ad54-613d7a35087d'
> I0210 02:47:31.280185 19604 executor.cpp:192] Version: 1.3.0
> I0210 02:47:31.280642 19597 executor.cpp:401] Connected with the agent
> I0210 02:47:31.280867 19603 executor.cpp:302] Sending SUBSCRIBE call to
> http://10.150.111.24:33608/slave(105)/api/v1/executor
> I0210 02:47:31.281189 19597 process.cpp:3704] Handling HTTP event for process
> 'slave(105)' with path: '/slave(105)/api/v1/executor'
> I0210 02:47:31.281433 19597 http.cpp:307] HTTP POST for
> /slave(105)/api/v1/executor from 10.150.111.24:58634
> I0210 02:47:31.281486 19597 slave.cpp:3166] Received Subscribe request for
> HTTP executor 'default' of framework d8129420-2a04-48e7-9b28-6b0a0af73168-0000
> I0210 02:47:31.281761 19597 slave.cpp:2328] Sending queued task
> '512412f2-4982-435f-a963-8dc60a3fb03e' to executor 'default' of framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (via HTTP)
> I0210 02:47:31.282199 19602 executor.cpp:716] Enqueuing event SUBSCRIBED
> received from http://10.150.111.24:33608/slave(105)/api/v1/executor
> I0210 02:47:31.282369 19602 executor.cpp:716] Enqueuing event LAUNCH received
> from http://10.150.111.24:33608/slave(105)/api/v1/executor
> I0210 02:47:31.282842 19578 scheduler.cpp:184] Version: 1.3.0
> I0210 02:47:31.283253 19604 scheduler.cpp:470] New master detected at
> [email protected]:33608
> I0210 02:47:31.283275 19604 scheduler.cpp:479] Waiting for 0ns before
> initiating a re-(connection) attempt with the master
> I0210 02:47:31.284811 19602 scheduler.cpp:361] Connected with the master at
> http://10.150.111.24:33608/master/api/v1/scheduler
> I0210 02:47:31.285277 19599 scheduler.cpp:243] Sending SUBSCRIBE call to
> http://10.150.111.24:33608/master/api/v1/scheduler
> I0210 02:47:31.285604 19604 process.cpp:3704] Handling HTTP event for process
> 'master' with path: '/master/api/v1/scheduler'
> I0210 02:47:31.285926 19602 http.cpp:420] HTTP POST for
> /master/api/v1/scheduler from 10.150.111.24:58640
> I0210 02:47:31.285991 19602 master.cpp:2423] Received subscription request
> for HTTP framework 'default'
> I0210 02:47:31.286016 19602 master.cpp:2160] Authorizing framework principal
> 'test-principal' to receive offers for role '*'
> I0210 02:47:31.286171 19598 master.cpp:2537] Subscribing framework 'default'
> with checkpointing disabled and capabilities [ ]
> I0210 02:47:31.286193 19598 master.cpp:2583] Updating info for framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000
> I0210 02:47:31.286348 19603 master.hpp:2119] Sending heartbeat to
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000
> I0210 02:47:31.286393 19601 slave.cpp:2824] Updating framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000 pid to @0.0.0.0:0
> I0210 02:47:31.286487 19603 status_update_manager.cpp:184] Resuming sending
> status updates
> I0210 02:47:31.286502 19600 scheduler.cpp:676] Enqueuing event ERROR received
> from http://10.150.111.24:33608/master/api/v1/scheduler
> I0210 02:47:31.286542 19603 master.cpp:1237] Ignoring disconnection for
> framework d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (default) as it has
> already reconnected
> I0210 02:47:31.286800 19603 scheduler.cpp:676] Enqueuing event SUBSCRIBED
> received from http://10.150.111.24:33608/master/api/v1/scheduler
> E0210 02:47:31.286847 19602 scheduler.cpp:649] End-Of-File received from
> master. The master closed the event stream
> I0210 02:47:31.287020 19601 scheduler.cpp:444] Re-detecting master
> I0210 02:47:31.287117 19603 scheduler.cpp:676] Enqueuing event HEARTBEAT
> received from http://10.150.111.24:33608/master/api/v1/scheduler
> I0210 02:47:31.287179 19601 scheduler.cpp:395] Ignoring disconnection attempt
> from stale connection
> I0210 02:47:31.287207 19601 scheduler.cpp:395] Ignoring disconnection attempt
> from stale connection
> I0210 02:47:31.287338 19598 scheduler.cpp:470] New master detected at
> [email protected]:33608
> I0210 02:47:31.287356 19598 scheduler.cpp:479] Waiting for 0ns before
> initiating a re-(connection) attempt with the master
> I0210 02:47:31.287582 19598 scheduler.cpp:243] Sending MESSAGE call to
> http://10.150.111.24:33608/master/api/v1/scheduler
> I0210 02:47:31.288050 19600 process.cpp:3704] Handling HTTP event for process
> 'master' with path: '/master/api/v1/scheduler'
> I0210 02:47:31.288568 19604 http.cpp:420] HTTP POST for
> /master/api/v1/scheduler from 10.150.111.24:58638
> I0210 02:47:31.288591 19601 scheduler.cpp:361] Connected with the master at
> http://10.150.111.24:33608/master/api/v1/scheduler
> I0210 02:47:31.288615 19604 master.cpp:5271] Processing MESSAGE call from
> framework d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (default) to agent
> d8129420-2a04-48e7-9b28-6b0a0af73168-S0 at slave(105)@10.150.111.24:33608
> (ip-10-150-111-24.ec2.internal)
> I0210 02:47:31.289027 19603 executor.cpp:716] Enqueuing event MESSAGE
> received from http://10.150.111.24:33608/slave(105)/api/v1/executor
> I0210 02:47:31.289861 19597 master.cpp:1381] Framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (default) disconnected
> I0210 02:47:31.289906 19597 master.cpp:3081] Deactivating framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (default)
> I0210 02:47:31.289947 19597 master.cpp:3058] Disconnecting framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (default)
> I0210 02:47:31.289978 19601 hierarchical.cpp:415] Deactivated framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000
> I0210 02:47:31.290050 19597 master.cpp:1396] Giving framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (default) 0ns to failover
> I0210 02:47:31.291277 19598 master.cpp:6842] Framework failover timeout,
> removing framework d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (default)
> I0210 02:47:31.291296 19598 master.cpp:7727] Removing framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (default)
> I0210 02:47:31.291358 19598 master.cpp:8312] Updating the state of task
> 512412f2-4982-435f-a963-8dc60a3fb03e of framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (latest state: TASK_KILLED, status
> update state: TASK_KILLED)
> I0210 02:47:31.291368 19599 slave.cpp:2655] Asked to shut down framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000 by [email protected]:33608
> I0210 02:47:31.291404 19599 slave.cpp:2680] Shutting down framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000
> I0210 02:47:31.291416 19599 slave.cpp:5080] Shutting down executor 'default'
> of framework d8129420-2a04-48e7-9b28-6b0a0af73168-0000 (via HTTP)
> I0210 02:47:31.291527 19598 master.cpp:8406] Removing task
> 512412f2-4982-435f-a963-8dc60a3fb03e with resources cpus(*)(allocated: *):2;
> mem(*)(allocated: *):1024; disk(*)(allocated: *):1024; ports(*)(allocated:
> *):[31000-32000] of framework d8129420-2a04-48e7-9b28-6b0a0af73168-0000 on
> agent d8129420-2a04-48e7-9b28-6b0a0af73168-S0 at
> slave(105)@10.150.111.24:33608 (ip-10-150-111-24.ec2.internal)
> I0210 02:47:31.291599 19598 master.cpp:8435] Removing executor 'default' with
> resources {} of framework d8129420-2a04-48e7-9b28-6b0a0af73168-0000 on agent
> d8129420-2a04-48e7-9b28-6b0a0af73168-S0 at slave(105)@10.150.111.24:33608
> (ip-10-150-111-24.ec2.internal)
> I0210 02:47:31.291600 19603 hierarchical.cpp:1064] Recovered
> cpus(*)(allocated: *):2; mem(*)(allocated: *):1024; disk(*)(allocated:
> *):1024; ports(*)(allocated: *):[31000-32000] (total: cpus(*):2; mem(*):1024;
> disk(*):1024; ports(*):[31000-32000], allocated: {}) on agent
> d8129420-2a04-48e7-9b28-6b0a0af73168-S0 from framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000
> I0210 02:47:31.291684 19603 hierarchical.cpp:362] Removed framework
> d8129420-2a04-48e7-9b28-6b0a0af73168-0000
> I0210 02:47:31.292042 19603 executor.cpp:716] Enqueuing event SHUTDOWN
> received from http://10.150.111.24:33608/slave(105)/api/v1/executor
> *** Aborted at 1486694851 (unix time) try "date -d @1486694851" if you are
> using GNU date ***
> PC: @ 0x55e75dde42b2
> testing::internal::UntypedFunctionMockerBase::SetOwnerAndName()
> *** SIGBUS (@0x0) received by PID 19578 (TID 0x7fd07ac1f700) from PID 0;
> stack trace: ***
> @ 0x7fd082e30390 (unknown)
> @ 0x55e75dde42b2
> testing::internal::UntypedFunctionMockerBase::SetOwnerAndName()
> @ 0x55e75d11ebe6
> mesos::internal::tests::executor::TestMesos<>::events()
> @ 0x55e75d10f34f std::_Function_handler<>::_M_invoke()
> @ 0x7fd084a840b2 process::AsyncExecutorProcess::execute<>()
> @ 0x7fd084a9595a
> _ZNSt17_Function_handlerIFvPN7process11ProcessBaseEEZNS0_8dispatchI7NothingNS0_20AsyncExecutorProcessERKSt8functionIFvRKSt5queueIN5mesos2v18executor5EventESt5dequeISC_SaISC_EEEEESG_PvSK_SG_SN_EENS0_6FutureIT_EERKNS0_3PIDIT0_EEMSS_FSP_T1_T2_T3_ET4_T5_T6_EUlS2_E_E9_M_invokeERKSt9_Any_dataOS2_
> @ 0x7fd0857b340c process::ProcessBase::visit()
> @ 0x7fd0857c5c33 process::ProcessManager::resume()
> @ 0x7fd0857d06e7
> _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEE6_M_runEv
> @ 0x7fd083613c80 (unknown)
> @ 0x7fd082e266ba start_thread
> @ 0x7fd082b5c82d (unknown)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
