[ https://issues.apache.org/jira/browse/MESOS-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157704#comment-15157704 ]
Alexander Rojas commented on MESOS-4047: ---------------------------------------- Reproduced again with following message (CentOS 6.7): {noformat} [==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from MemoryPressureMesosTest 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.000394345 s, 2.7 GB/s [ RUN ] MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery I0222 09:32:20.622694 20868 leveldb.cpp:174] Opened db in 5.153509ms I0222 09:32:20.624688 20868 leveldb.cpp:181] Compacted db in 1.914323ms I0222 09:32:20.624778 20868 leveldb.cpp:196] Created db iterator in 24549ns I0222 09:32:20.624795 20868 leveldb.cpp:202] Seeked to beginning of db in 2610ns I0222 09:32:20.624804 20868 leveldb.cpp:271] Iterated through 0 keys in the db in 323ns I0222 09:32:20.624874 20868 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0222 09:32:20.625977 20888 recover.cpp:447] Starting replica recovery I0222 09:32:20.626901 20888 recover.cpp:473] Replica is in EMPTY status I0222 09:32:20.634701 20889 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (11193)@127.0.0.1:54769 I0222 09:32:20.634953 20888 master.cpp:376] Master 17b7da64-0c4d-4e46-ae1f-2b356dc5f266 (localhost) started on 127.0.0.1:54769 I0222 09:32:20.634986 20888 master.cpp:378] Flags at startup: --acls="" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_http="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/0rXncF/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" --registry_strict="true" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/0rXncF/master" --zk_session_timeout="10secs" W0222 09:32:20.635417 20888 master.cpp:381] ************************************************** Master bound to loopback interface! Cannot communicate with remote schedulers or slaves. You might want to set '--ip' flag to a routable IP address. ************************************************** I0222 09:32:20.635587 20888 master.cpp:423] Master only allowing authenticated frameworks to register I0222 09:32:20.635601 20888 master.cpp:428] Master only allowing authenticated slaves to register I0222 09:32:20.635622 20888 credentials.hpp:35] Loading credentials for authentication from '/tmp/0rXncF/credentials' I0222 09:32:20.636018 20888 master.cpp:468] Using default 'crammd5' authenticator I0222 09:32:20.636190 20888 master.cpp:537] Using default 'basic' HTTP authenticator I0222 09:32:20.636174 20887 recover.cpp:193] Received a recover response from a replica in EMPTY status I0222 09:32:20.636425 20888 master.cpp:571] Authorization enabled I0222 09:32:20.637810 20885 recover.cpp:564] Updating replica status to STARTING I0222 09:32:20.640805 20887 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 2.741248ms I0222 09:32:20.640964 20887 replica.cpp:320] Persisted replica status to STARTING I0222 09:32:20.641525 20885 recover.cpp:473] Replica is in STARTING status I0222 09:32:20.642133 20888 master.cpp:1712] The newly elected leader is master@127.0.0.1:54769 with id 17b7da64-0c4d-4e46-ae1f-2b356dc5f266 I0222 09:32:20.642236 20888 master.cpp:1725] Elected as the leading master! I0222 09:32:20.642253 20888 master.cpp:1470] Recovering from registrar I0222 09:32:20.642496 20885 registrar.cpp:307] Recovering registrar I0222 09:32:20.643162 20889 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from (11195)@127.0.0.1:54769 I0222 09:32:20.643590 20885 recover.cpp:193] Received a recover response from a replica in STARTING status I0222 09:32:20.644120 20887 recover.cpp:564] Updating replica status to VOTING I0222 09:32:20.646817 20889 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 1.190281ms I0222 09:32:20.646870 20889 replica.cpp:320] Persisted replica status to VOTING I0222 09:32:20.647094 20885 recover.cpp:578] Successfully joined the Paxos group I0222 09:32:20.647337 20885 recover.cpp:462] Recover process terminated I0222 09:32:20.647781 20887 log.cpp:659] Attempting to start the writer I0222 09:32:20.648854 20890 replica.cpp:493] Replica received implicit promise request from (11196)@127.0.0.1:54769 with proposal 1 I0222 09:32:20.650074 20890 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 1.17538ms I0222 09:32:20.650133 20890 replica.cpp:342] Persisted promised to 1 I0222 09:32:20.650879 20883 coordinator.cpp:238] Coordinator attempting to fill missing positions I0222 09:32:20.652029 20889 replica.cpp:388] Replica received explicit promise request from (11197)@127.0.0.1:54769 for position 0 with proposal 2 I0222 09:32:20.653084 20889 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 1.006874ms I0222 09:32:20.653152 20889 replica.cpp:712] Persisted action at 0 I0222 09:32:20.654109 20888 replica.cpp:537] Replica received write request for position 0 from (11198)@127.0.0.1:54769 I0222 09:32:20.654157 20888 leveldb.cpp:436] Reading position from leveldb took 19941ns I0222 09:32:20.655339 20888 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 914288ns I0222 09:32:20.655375 20888 replica.cpp:712] Persisted action at 0 I0222 09:32:20.655944 20890 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 I0222 09:32:20.657099 20890 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 1.124148ms I0222 09:32:20.657166 20890 replica.cpp:712] Persisted action at 0 I0222 09:32:20.657232 20890 replica.cpp:697] Replica learned NOP action at position 0 I0222 09:32:20.657861 20890 log.cpp:675] Writer started with ending position 0 I0222 09:32:20.658861 20886 leveldb.cpp:436] Reading position from leveldb took 30090ns I0222 09:32:20.659822 20888 registrar.cpp:340] Successfully fetched the registry (0B) in 17.27488ms I0222 09:32:20.660028 20888 registrar.cpp:439] Applied 1 operations in 123880ns; attempting to update the 'registry' I0222 09:32:20.660781 20886 log.cpp:683] Attempting to append 160 bytes to the log I0222 09:32:20.661144 20885 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 I0222 09:32:20.663028 20890 replica.cpp:537] Replica received write request for position 1 from (11199)@127.0.0.1:54769 I0222 09:32:20.666872 20890 leveldb.cpp:341] Persisting action (179 bytes) to leveldb took 3.746467ms I0222 09:32:20.666934 20890 replica.cpp:712] Persisted action at 1 I0222 09:32:20.667937 20890 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 I0222 09:32:20.670030 20890 leveldb.cpp:341] Persisting action (181 bytes) to leveldb took 2.029339ms I0222 09:32:20.670084 20890 replica.cpp:712] Persisted action at 1 I0222 09:32:20.670105 20890 replica.cpp:697] Replica learned APPEND action at position 1 I0222 09:32:20.671183 20889 registrar.cpp:484] Successfully updated the 'registry' in 11.046144ms I0222 09:32:20.671344 20889 registrar.cpp:370] Successfully recovered registrar I0222 09:32:20.671454 20890 log.cpp:702] Attempting to truncate the log to 1 I0222 09:32:20.672252 20889 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 I0222 09:32:20.672366 20885 master.cpp:1522] Recovered 0 slaves from the Registry (122B) ; allowing 10mins for slaves to re-register I0222 09:32:20.673305 20884 replica.cpp:537] Replica received write request for position 2 from (11200)@127.0.0.1:54769 I0222 09:32:20.675009 20884 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 1.632734ms I0222 09:32:20.675067 20884 replica.cpp:712] Persisted action at 2 I0222 09:32:20.675851 20883 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 I0222 09:32:20.677466 20883 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 1.548407ms I0222 09:32:20.677618 20883 leveldb.cpp:399] Deleting ~1 keys from leveldb took 69924ns I0222 09:32:20.677649 20883 replica.cpp:712] Persisted action at 2 I0222 09:32:20.677686 20883 replica.cpp:697] Replica learned TRUNCATE action at position 2 I0222 09:32:20.685191 20868 containerizer.cpp:149] Using isolation: cgroups/mem,filesystem/posix I0222 09:32:20.692539 20868 linux_launcher.cpp:101] Using /cgroup/freezer as the freezer hierarchy for the Linux launcher I0222 09:32:20.717495 20889 slave.cpp:193] Slave started on 247)@127.0.0.1:54769 I0222 09:32:20.717723 20889 slave.cpp:194] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos_test_7adedc17-08f0-4c8b-9974-66c009b1da18" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="https://auth.docker.io" --docker_kill_orphans="true" --docker_puller_timeout="60" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="cgroups/mem" --launcher_dir="/home/alexander/workspace/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI" W0222 09:32:20.718128 20889 slave.cpp:197] ************************************************** Slave bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address. ************************************************** I0222 09:32:20.718827 20889 credentials.hpp:83] Loading credential for authentication from '/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/credential' I0222 09:32:20.719223 20889 slave.cpp:324] Slave using credential for: test-principal W0222 09:32:20.719915 20868 sched.cpp:1642] ************************************************** Scheduler driver bound to loopback interface! Cannot communicate with remote master(s). You might want to set 'LIBPROCESS_IP' environment variable to use a routable IP address. ************************************************** I0222 09:32:20.720688 20889 slave.cpp:464] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0222 09:32:20.720878 20889 slave.cpp:472] Slave attributes: [ ] I0222 09:32:20.720979 20889 slave.cpp:477] Slave hostname: localhost I0222 09:32:20.721812 20868 sched.cpp:222] Version: 0.28.0 I0222 09:32:20.722302 20889 state.cpp:58] Recovering state from '/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/meta' I0222 09:32:20.722332 20886 sched.cpp:326] New master detected at master@127.0.0.1:54769 I0222 09:32:20.725953 20885 status_update_manager.cpp:200] Recovering status update manager I0222 09:32:20.726177 20886 sched.cpp:382] Authenticating with master master@127.0.0.1:54769 I0222 09:32:20.726552 20886 sched.cpp:389] Using default CRAM-MD5 authenticatee I0222 09:32:20.726557 20889 containerizer.cpp:407] Recovering containerizer I0222 09:32:20.727334 20884 authenticatee.cpp:121] Creating new client SASL connection I0222 09:32:20.727854 20886 master.cpp:5526] Authenticating scheduler-9fd9adc7-27ff-4054-a6ed-d613cf04ec62@127.0.0.1:54769 I0222 09:32:20.728514 20889 authenticator.cpp:98] Creating new server SASL connection I0222 09:32:20.728842 20889 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0222 09:32:20.728987 20889 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0222 09:32:20.729140 20889 authenticator.cpp:203] Received SASL authentication start I0222 09:32:20.729305 20889 authenticator.cpp:325] Authentication requires more steps I0222 09:32:20.729557 20889 authenticatee.cpp:258] Received SASL authentication step I0222 09:32:20.729763 20889 authenticator.cpp:231] Received SASL authentication step I0222 09:32:20.730895 20889 authenticator.cpp:317] Authentication success I0222 09:32:20.731767 20889 master.cpp:5556] Successfully authenticated principal 'test-principal' at scheduler-9fd9adc7-27ff-4054-a6ed-d613cf04ec62@127.0.0.1:54769 I0222 09:32:20.731767 20885 authenticatee.cpp:298] Authentication success I0222 09:32:20.735591 20885 sched.cpp:471] Successfully authenticated with master master@127.0.0.1:54769 I0222 09:32:20.736048 20890 provisioner.cpp:245] Provisioner recovery complete I0222 09:32:20.739279 20889 slave.cpp:4565] Finished recovery I0222 09:32:20.739347 20887 master.cpp:2280] Received SUBSCRIBE call for framework 'default' at scheduler-9fd9adc7-27ff-4054-a6ed-d613cf04ec62@127.0.0.1:54769 I0222 09:32:20.739572 20887 master.cpp:1751] Authorizing framework principal 'test-principal' to receive offers for role '*' I0222 09:32:20.740470 20889 master.cpp:2351] Subscribing framework default with checkpointing enabled and capabilities [ ] I0222 09:32:20.740790 20887 slave.cpp:796] New master detected at master@127.0.0.1:54769 I0222 09:32:20.741118 20885 hierarchical.cpp:265] Added framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:20.741659 20883 sched.cpp:703] Framework registered with 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:20.740885 20886 status_update_manager.cpp:174] Pausing sending status updates I0222 09:32:20.741971 20887 slave.cpp:859] Authenticating with master master@127.0.0.1:54769 I0222 09:32:20.742007 20887 slave.cpp:864] Using default CRAM-MD5 authenticatee I0222 09:32:20.742254 20887 slave.cpp:832] Detecting new master I0222 09:32:20.742301 20889 authenticatee.cpp:121] Creating new client SASL connection I0222 09:32:20.742700 20885 master.cpp:5526] Authenticating slave(247)@127.0.0.1:54769 I0222 09:32:20.743103 20884 authenticator.cpp:98] Creating new server SASL connection I0222 09:32:20.743357 20884 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 I0222 09:32:20.743404 20884 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' I0222 09:32:20.743563 20884 authenticator.cpp:203] Received SASL authentication start I0222 09:32:20.743628 20884 authenticator.cpp:325] Authentication requires more steps I0222 09:32:20.743881 20886 authenticatee.cpp:258] Received SASL authentication step I0222 09:32:20.743976 20886 authenticator.cpp:231] Received SASL authentication step I0222 09:32:20.744055 20886 authenticator.cpp:317] Authentication success I0222 09:32:20.744320 20886 authenticatee.cpp:298] Authentication success I0222 09:32:20.744364 20883 master.cpp:5556] Successfully authenticated principal 'test-principal' at slave(247)@127.0.0.1:54769 I0222 09:32:20.744813 20883 slave.cpp:927] Successfully authenticated with master master@127.0.0.1:54769 I0222 09:32:20.745275 20889 master.cpp:4240] Registering slave at slave(247)@127.0.0.1:54769 (localhost) with id 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 I0222 09:32:20.745687 20883 registrar.cpp:439] Applied 1 operations in 46410ns; attempting to update the 'registry' I0222 09:32:20.746387 20883 log.cpp:683] Attempting to append 327 bytes to the log I0222 09:32:20.746670 20889 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 I0222 09:32:20.747566 20886 replica.cpp:537] Replica received write request for position 3 from (11220)@127.0.0.1:54769 I0222 09:32:20.749735 20886 leveldb.cpp:341] Persisting action (346 bytes) to leveldb took 2.033429ms I0222 09:32:20.749877 20886 replica.cpp:712] Persisted action at 3 I0222 09:32:20.751174 20890 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 I0222 09:32:20.752976 20883 master.cpp:4228] Ignoring register slave message from slave(247)@127.0.0.1:54769 (localhost) as admission is already in progress I0222 09:32:20.753378 20890 leveldb.cpp:341] Persisting action (348 bytes) to leveldb took 2.085441ms I0222 09:32:20.753435 20890 replica.cpp:712] Persisted action at 3 I0222 09:32:20.753473 20890 replica.cpp:697] Replica learned APPEND action at position 3 I0222 09:32:20.755097 20888 registrar.cpp:484] Successfully updated the 'registry' in 9.328128ms I0222 09:32:20.755506 20884 log.cpp:702] Attempting to truncate the log to 3 I0222 09:32:20.755728 20888 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 I0222 09:32:20.756162 20886 master.cpp:4308] Registered slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769 (localhost) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0222 09:32:20.756397 20883 hierarchical.cpp:473] Added slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 (localhost) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: ) I0222 09:32:20.756494 20889 slave.cpp:971] Registered with master master@127.0.0.1:54769; given slave ID 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 I0222 09:32:20.756803 20888 status_update_manager.cpp:181] Resuming sending status updates I0222 09:32:20.757164 20889 slave.cpp:1030] Forwarding total oversubscribed resources I0222 09:32:20.757809 20883 replica.cpp:537] Replica received write request for position 4 from (11221)@127.0.0.1:54769 I0222 09:32:20.757522 20888 master.cpp:5355] Sending 1 offers to framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 (default) at scheduler-9fd9adc7-27ff-4054-a6ed-d613cf04ec62@127.0.0.1:54769 I0222 09:32:20.758471 20888 master.cpp:4649] Received update of slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769 (localhost) with total oversubscribed resources I0222 09:32:20.759210 20883 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 1.362326ms I0222 09:32:20.759312 20883 replica.cpp:712] Persisted action at 4 I0222 09:32:20.760362 20888 hierarchical.cpp:531] Slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 (localhost) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]) I0222 09:32:20.760388 20889 replica.cpp:691] Replica received learned notice for position 4 from @0.0.0.0:0 I0222 09:32:20.762329 20888 master.cpp:3138] Processing ACCEPT call for offers: [ 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-O0 ] on slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769 (localhost) for framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 (default) at scheduler-9fd9adc7-27ff-4054-a6ed-d613cf04ec62@127.0.0.1:54769 I0222 09:32:20.762415 20888 master.cpp:2825] Authorizing framework principal 'test-principal' to launch task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d as user 'root' I0222 09:32:20.762817 20889 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 2.201003ms I0222 09:32:20.764372 20889 leveldb.cpp:399] Deleting ~2 keys from leveldb took 1.136088ms I0222 09:32:20.764535 20889 replica.cpp:712] Persisted action at 4 I0222 09:32:20.764650 20889 replica.cpp:697] Replica learned TRUNCATE action at position 4 I0222 09:32:20.765740 20884 master.hpp:176] Adding task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d with resources cpus(*):1; mem(*):256; disk(*):1024 on slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 (localhost) I0222 09:32:20.766749 20884 master.cpp:3623] Launching task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 (default) at scheduler-9fd9adc7-27ff-4054-a6ed-d613cf04ec62@127.0.0.1:54769 with resources cpus(*):1; mem(*):256; disk(*):1024 on slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769 (localhost) I0222 09:32:20.768129 20888 slave.cpp:1361] Got assigned task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d for framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:20.769053 20888 slave.cpp:1480] Launching task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d for framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:20.769868 20888 paths.cpp:474] Trying to chown '/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/slaves/17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0/frameworks/17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000/executors/24458ae6-dde3-40e0-a4f7-84a901bd9f7d/runs/0e8d2278-bb2b-4480-bfb1-39b47abcb707' to user 'root' I0222 09:32:20.774318 20888 slave.cpp:5367] Launching executor 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/slaves/17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0/frameworks/17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000/executors/24458ae6-dde3-40e0-a4f7-84a901bd9f7d/runs/0e8d2278-bb2b-4480-bfb1-39b47abcb707' I0222 09:32:20.775251 20890 containerizer.cpp:666] Starting container '0e8d2278-bb2b-4480-bfb1-39b47abcb707' for executor '24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework '17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000' I0222 09:32:20.775647 20888 slave.cpp:1698] Queuing task '24458ae6-dde3-40e0-a4f7-84a901bd9f7d' for executor '24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:20.779909 20883 mem.cpp:602] Started listening for OOM events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707 I0222 09:32:20.780675 20883 mem.cpp:722] Started listening on low memory pressure events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707 I0222 09:32:20.781409 20883 mem.cpp:722] Started listening on medium memory pressure events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707 I0222 09:32:20.781992 20883 mem.cpp:722] Started listening on critical memory pressure events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707 I0222 09:32:20.782661 20883 mem.cpp:353] Updated 'memory.soft_limit_in_bytes' to 288MB for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707 I0222 09:32:20.783485 20883 mem.cpp:388] Updated 'memory.limit_in_bytes' to 288MB for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707 I0222 09:32:20.785579 20890 linux_launcher.cpp:304] Cloning child process with flags = I0222 09:32:20.788130 20890 containerizer.cpp:1104] Checkpointing executor's forked pid 26589 to '/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/meta/slaves/17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0/frameworks/17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000/executors/24458ae6-dde3-40e0-a4f7-84a901bd9f7d/runs/0e8d2278-bb2b-4480-bfb1-39b47abcb707/pids/forked.pid' I0222 09:32:20.911543 26589 exec.cpp:143] Version: 0.28.0 I0222 09:32:20.920542 20884 slave.cpp:2643] Got registration for executor '24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 from executor(1)@127.0.0.1:49224 I0222 09:32:20.923596 20884 mem.cpp:353] Updated 'memory.soft_limit_in_bytes' to 288MB for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707 I0222 09:32:20.923749 26619 exec.cpp:217] Executor registered on slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 I0222 09:32:20.924268 20886 slave.cpp:1863] Sending queued task '24458ae6-dde3-40e0-a4f7-84a901bd9f7d' to executor '24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 at executor(1)@127.0.0.1:49224 Registered executor on localhost Starting task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d sh -c 'while true; do dd count=512 bs=1M if=/dev/zero of=./temp; done' Forked command at 26627 I0222 09:32:20.929365 20884 slave.cpp:3002] Handling status update TASK_RUNNING (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 from executor(1)@127.0.0.1:49224 I0222 09:32:20.930857 20886 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:20.931435 20886 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_RUNNING (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:20.935792 20887 slave.cpp:3400] Forwarding the update TASK_RUNNING (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 to master@127.0.0.1:54769 I0222 09:32:20.936246 20887 slave.cpp:3310] Sending acknowledgement for status update TASK_RUNNING (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 to executor(1)@127.0.0.1:49224 I0222 09:32:20.936532 20886 master.cpp:4794] Status update TASK_RUNNING (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 from slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769 (localhost) I0222 09:32:20.936666 20886 master.cpp:4842] Forwarding status update TASK_RUNNING (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:20.936980 20886 master.cpp:6450] Updating the state of task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING) I0222 09:32:20.938391 20885 master.cpp:3952] Processing ACKNOWLEDGE call e57e1d63-60c7-4132-812e-4d980eeb3039 for task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 (default) at scheduler-9fd9adc7-27ff-4054-a6ed-d613cf04ec62@127.0.0.1:54769 on slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 I0222 09:32:20.939096 20883 status_update_manager.cpp:392] Received status update acknowledgement (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:20.939386 20868 slave.cpp:668] Slave terminating I0222 09:32:20.939414 20883 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_RUNNING (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:20.939702 20889 master.cpp:1174] Slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769 (localhost) disconnected I0222 09:32:20.939720 20889 master.cpp:2635] Disconnecting slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769 (localhost) I0222 09:32:20.939901 20889 master.cpp:2654] Deactivating slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769 (localhost) I0222 09:32:20.940335 20887 hierarchical.cpp:560] Slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 deactivated I0222 09:32:20.944319 20868 containerizer.cpp:149] Using isolation: cgroups/mem,filesystem/posix I0222 09:32:20.949115 20868 linux_launcher.cpp:101] Using /cgroup/freezer as the freezer hierarchy for the Linux launcher I0222 09:32:21.021484 20890 slave.cpp:193] Slave started on 248)@127.0.0.1:54769 I0222 09:32:21.021664 20890 slave.cpp:194] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos_test_7adedc17-08f0-4c8b-9974-66c009b1da18" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_auth_server="https://auth.docker.io" --docker_kill_orphans="true" --docker_puller_timeout="60" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="cgroups/mem" --launcher_dir="/home/alexander/workspace/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI" W0222 09:32:21.021950 20890 slave.cpp:197] ************************************************** Slave bound to loopback interface! Cannot communicate with remote master(s). You might want to set '--ip' flag to a routable IP address. ************************************************** I0222 09:32:21.021999 20890 credentials.hpp:83] Loading credential for authentication from '/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/credential' I0222 09:32:21.022282 20890 slave.cpp:324] Slave using credential for: test-principal I0222 09:32:21.022814 20890 slave.cpp:464] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0222 09:32:21.022861 20890 slave.cpp:472] Slave attributes: [ ] I0222 09:32:21.022871 20890 slave.cpp:477] Slave hostname: localhost I0222 09:32:21.024055 20890 state.cpp:58] Recovering state from '/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/meta' I0222 09:32:21.024163 20890 state.cpp:698] No checkpointed resources found at '/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/meta/resources/resources.info' I0222 09:32:21.026868 20889 slave.cpp:4653] Recovering framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:21.026999 20889 slave.cpp:5476] Recovering executor '24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:21.028094 20884 status_update_manager.cpp:200] Recovering status update manager I0222 09:32:21.028137 20884 status_update_manager.cpp:208] Recovering executor '24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:21.031335 20887 containerizer.cpp:407] Recovering containerizer I0222 09:32:21.031548 20887 containerizer.cpp:462] Recovering container '0e8d2278-bb2b-4480-bfb1-39b47abcb707' for executor '24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:21.035538 20885 mem.cpp:602] Started listening for OOM events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707 I0222 09:32:21.036461 20885 mem.cpp:722] Started listening on low memory pressure events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707 I0222 09:32:21.037284 20885 mem.cpp:722] Started listening on medium memory pressure events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707 I0222 09:32:21.038064 20885 mem.cpp:722] Started listening on critical memory pressure events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707 I0222 09:32:21.040393 20886 provisioner.cpp:245] Provisioner recovery complete I0222 09:32:21.041451 20883 slave.cpp:4505] Sending reconnect request to executor '24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 at executor(1)@127.0.0.1:49224 I0222 09:32:21.044780 26624 exec.cpp:263] Received reconnect request from slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 I0222 09:32:21.048050 20886 slave.cpp:2792] Re-registering executor '24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 I0222 09:32:21.051213 26620 exec.cpp:240] Executor re-registered on slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 I0222 09:32:21.052510 20888 mem.cpp:353] Updated 'memory.soft_limit_in_bytes' to 288MB for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707 Re-registered executor on localhost I0222 09:32:21.053912 20888 mem.cpp:388] Updated 'memory.limit_in_bytes' to 288MB for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707 W0222 09:32:21.210090 20886 master.cpp:3869] Cannot kill task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 (default) at scheduler-9fd9adc7-27ff-4054-a6ed-d613cf04ec62@127.0.0.1:54769 because the slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769 (localhost) is disconnected. Kill will be retried if the slave re-registers ../../src/tests/containerizer/memory_pressure_tests.cpp:300: Failure Expected: (usage.get().mem_low_pressure_counter()) >= (usage.get().mem_medium_pressure_counter()), actual: 6 vs 8 *** Aborted at 1456162341 (unix time) try "date -d @1456162341" if you are using GNU date *** PC: @ 0x16b24b0 testing::UnitTest::AddTestPartResult() *** SIGSEGV (@0x0) received by PID 20868 (TID 0x7f0ee95ae840) from PID 0; stack trace: *** @ 0x7f0ee14ee790 (unknown) @ 0x16b24b0 testing::UnitTest::AddTestPartResult() @ 0x16a6ee9 testing::internal::AssertHelper::operator=() @ 0x1688fc0 mesos::internal::tests::MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_Test::TestBody() @ 0x16cff10 testing::internal::HandleSehExceptionsInMethodIfSupported<>() @ 0x16caf3e testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x16ac289 testing::Test::Run() @ 0x16aca17 testing::TestInfo::Run() @ 0x16ad052 testing::TestCase::Run() @ 0x16b39a1 testing::internal::UnitTestImpl::RunAllTests() @ 0x16d0b9f testing::internal::HandleSehExceptionsInMethodIfSupported<>() @ 0x16cbaca testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x16b26d1 testing::UnitTest::Run() @ 0xe3bcd6 RUN_ALL_TESTS() @ 0xe3b8ec main @ 0x7f0ee0356d5d __libc_start_main @ 0x9c4d89 (unknown) I0222 09:32:21.558915 26624 exec.cpp:463] Slave exited, but framework has checkpointing enabled. Waiting 15mins to reconnect with slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 I0222 09:32:21.559326 26620 exec.cpp:472] Slave exited ... shutting down Shutting down Sending SIGTERM to process tree at pid 26627 /var/tmp/sclcApPT3: line 8: 20868 Segmentation fault './bin/mesos-tests.sh' '--gtest_filter=MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery' '--gtest_repeat=1000' '--gtest_break_on_failure' Sent SIGTERM to the following process trees: [ -+- 26627 sh -c while true; do dd count=512 bs=1M if=/dev/zero of=./temp; done \--- 26628 dd count=512 bs=1M if=/dev/zero of=./temp ] Command terminated with signal Terminated (pid: 26627) E0222 09:32:21.704880 26626 process.cpp:1963] Failed to shutdown socket with fd 9: Transport endpoint is not connected {noformat} after running: {code} MESOS_VERBOSE=1 sudo .libs/mesos-tests --gtest_filter="MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery" --gtest_repeat=1000 --gtest_break_on_failure {code} > MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery is flaky > ----------------------------------------------------------- > > Key: MESOS-4047 > URL: https://issues.apache.org/jira/browse/MESOS-4047 > Project: Mesos > Issue Type: Bug > Components: test > Affects Versions: 0.26.0 > Environment: Ubuntu 14, gcc 4.8.4 > Reporter: Joseph Wu > Assignee: Joseph Wu > Labels: flaky, flaky-test > Fix For: 0.27.0 > > > {code:title=Output from passed test} > [----------] 1 test from MemoryPressureMesosTest > 1+0 records in > 1+0 records out > 1048576 bytes (1.0 MB) copied, 0.000430889 s, 2.4 GB/s > [ RUN ] MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery > I1202 11:09:14.319327 5062 exec.cpp:134] Version: 0.27.0 > I1202 11:09:14.333317 5079 exec.cpp:208] Executor registered on slave > bea15b35-9aa1-4b57-96fb-29b5f70638ac-S0 > Registered executor on ubuntu > Starting task 4e62294c-cfcf-4a13-b699-c6a4b7ac5162 > sh -c 'while true; do dd count=512 bs=1M if=/dev/zero of=./temp; done' > Forked command at 5085 > I1202 11:09:14.391739 5077 exec.cpp:254] Received reconnect request from > slave bea15b35-9aa1-4b57-96fb-29b5f70638ac-S0 > I1202 11:09:14.398598 5082 exec.cpp:231] Executor re-registered on slave > bea15b35-9aa1-4b57-96fb-29b5f70638ac-S0 > Re-registered executor on ubuntu > Shutting down > Sending SIGTERM to process tree at pid 5085 > Killing the following process trees: > [ > -+- 5085 sh -c while true; do dd count=512 bs=1M if=/dev/zero of=./temp; done > \--- 5086 dd count=512 bs=1M if=/dev/zero of=./temp > ] > [ OK ] MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery (1096 ms) > {code} > {code:title=Output from failed test} > [----------] 1 test from MemoryPressureMesosTest > 1+0 records in > 1+0 records out > 1048576 bytes (1.0 MB) copied, 0.000404489 s, 2.6 GB/s > [ RUN ] MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery > I1202 11:09:15.509950 5109 exec.cpp:134] Version: 0.27.0 > I1202 11:09:15.568183 5123 exec.cpp:208] Executor registered on slave > 88734acc-718e-45b0-95b9-d8f07cea8a9e-S0 > Registered executor on ubuntu > Starting task 14b6bab9-9f60-4130-bdc4-44efba262bc6 > Forked command at 5132 > sh -c 'while true; do dd count=512 bs=1M if=/dev/zero of=./temp; done' > I1202 11:09:15.665498 5129 exec.cpp:254] Received reconnect request from > slave 88734acc-718e-45b0-95b9-d8f07cea8a9e-S0 > I1202 11:09:15.670995 5123 exec.cpp:381] Executor asked to shutdown > Shutting down > Sending SIGTERM to process tree at pid 5132 > ../../src/tests/containerizer/memory_pressure_tests.cpp:283: Failure > (usage).failure(): Unknown container: ebe90e15-72fa-4519-837b-62f43052c913 > *** Aborted at 1449083355 (unix time) try "date -d @1449083355" if you are > using GNU date *** > {code} > Notice that in the failed test, the executor is asked to shutdown when it > tries to reconnect to the agent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)