[
https://issues.apache.org/jira/browse/MESOS-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400420#comment-16400420
]
Alexander Rojas commented on MESOS-8677:
----------------------------------------
The issue seem to be using
[{{hashset}}|https://github.com/apache/mesos/blob/master/src/common/http.cpp#L871]
to ensure only one entry per action while constructing the
{{ObjectApprovers}}. We were relying that different iterations would show the
{{hashset}} contents in the same order, but this was not true in macOS (nor can
we ever rely on that), particularly after the {{hashset}} was copied.
Replacing the {{hashset}} for a {{set}} which does have a strong order
guarantee, solves the issue.
> FaulToleranceTest.ReregisterCompletedFrameworks crashes on macOS
> ----------------------------------------------------------------
>
> Key: MESOS-8677
> URL: https://issues.apache.org/jira/browse/MESOS-8677
> Project: Mesos
> Issue Type: Bug
> Components: test
> Environment: macOS 10.13.3 with LLVM 6.0.0 as well as with Apple LLVM
> version 9.0.0 (clang-900.0.39.2)
> Reporter: Jan Schlicht
> Priority: Major
>
> Here's a {{GLOG_v=1}} run of the test:
> {noformat}
> [ RUN ] FaultToleranceTest.ReregisterCompletedFrameworks
> I0314 14:30:11.240077 2290090816 cluster.cpp:172] Creating default 'local'
> authorizer
> I0314 14:30:11.241261 55140352 master.cpp:463] Master
> 025f775d-9c75-43f6-9ee6-079a605fbf01 (Jenkinss-Mac-mini.local) started on
> 10.0.49.4:54648
> I0314 14:30:11.241287 55140352 master.cpp:465] Flags at startup: --acls=""
> --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins"
> --allocation_interval="1secs" --allocator="HierarchicalDRF"
> --authenticate_agents="true" --authenticate_frameworks="true"
> --authenticate_http_frameworks="true" --authenticate_http_readonly="true"
> --authenticate_http_readwrite="true" --authenticators="crammd5"
> --authorizers="local"
> --credentials="/private/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/ZyMWb1/credentials"
> --filter_gpu_resources="true" --framework_sorter="drf" --help="false"
> --hostname_lookup="true" --http_authenticators="basic"
> --http_framework_authenticators="basic" --initialize_driver_logging="true"
> --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO"
> --max_agent_ping_timeouts="5" --max_completed_frameworks="50"
> --max_completed_tasks_per_framework="1000"
> --max_unreachable_tasks_per_framework="1000" --port="5050" --quiet="false"
> --recovery_agent_removal_limit="100%" --registry="in_memory"
> --registry_fetch_timeout="1mins" --registry_gc_interval="15mins"
> --registry_max_agent_age="2weeks" --registry_max_agent_count="102400"
> --registry_store_timeout="100secs" --registry_strict="false"
> --require_agent_domain="false" --root_submissions="true" --user_sorter="drf"
> --version="false" --webui_dir="/usr/local/share/mesos/webui"
> --work_dir="/private/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/ZyMWb1/master"
> --zk_session_timeout="10secs"
> I0314 14:30:11.241439 55140352 master.cpp:514] Master only allowing
> authenticated frameworks to register
> I0314 14:30:11.241447 55140352 master.cpp:520] Master only allowing
> authenticated agents to register
> I0314 14:30:11.241452 55140352 master.cpp:526] Master only allowing
> authenticated HTTP frameworks to register
> I0314 14:30:11.241461 55140352 credentials.hpp:37] Loading credentials for
> authentication from
> '/private/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/ZyMWb1/credentials'
> I0314 14:30:11.241678 55140352 master.cpp:570] Using default 'crammd5'
> authenticator
> I0314 14:30:11.241739 55140352 http.cpp:957] Creating default 'basic' HTTP
> authenticator for realm 'mesos-master-readonly'
> I0314 14:30:11.241824 55140352 http.cpp:957] Creating default 'basic' HTTP
> authenticator for realm 'mesos-master-readwrite'
> I0314 14:30:11.241873 55140352 http.cpp:957] Creating default 'basic' HTTP
> authenticator for realm 'mesos-master-scheduler'
> I0314 14:30:11.241919 55140352 master.cpp:649] Authorization enabled
> I0314 14:30:11.242066 52457472 whitelist_watcher.cpp:77] No whitelist given
> I0314 14:30:11.242079 51920896 hierarchical.cpp:175] Initialized hierarchical
> allocator process
> I0314 14:30:11.243557 52994048 master.cpp:2119] Elected as the leading master!
> I0314 14:30:11.243574 52994048 master.cpp:1678] Recovering from registrar
> I0314 14:30:11.243640 51920896 registrar.cpp:347] Recovering registrar
> I0314 14:30:11.243852 52457472 registrar.cpp:391] Successfully fetched the
> registry (0B) in 190976ns
> I0314 14:30:11.243928 52457472 registrar.cpp:495] Applied 1 operations in
> 28606ns; attempting to update the registry
> I0314 14:30:11.244163 52457472 registrar.cpp:552] Successfully updated the
> registry in 194816ns
> I0314 14:30:11.244222 52457472 registrar.cpp:424] Successfully recovered
> registrar
> I0314 14:30:11.244408 54067200 master.cpp:1792] Recovered 0 agents from the
> registry (155B); allowing 10mins for agents to reregister
> I0314 14:30:11.244443 52994048 hierarchical.cpp:213] Skipping recovery of
> hierarchical allocator: nothing to recover
> W0314 14:30:11.247259 2290090816 process.cpp:2805] Attempted to spawn already
> running process [email protected]:54648
> I0314 14:30:11.247681 2290090816 cluster.cpp:460] Creating default 'local'
> authorizer
> I0314 14:30:11.248837 55676928 slave.cpp:265] Mesos agent started on
> (50)@10.0.49.4:54648
> I0314 14:30:11.248865 55676928 slave.cpp:266] Flags at startup: --acls=""
> --appc_simple_discovery_uri_prefix="http://";
> --appc_store_dir="/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/FaultToleranceTest_ReregisterCompletedFrameworks_UqvwBG/store/appc"
> --authenticate_http_executors="true" --authenticate_http_readonly="true"
> --authenticate_http_readwrite="true" --authenticatee="crammd5"
> --authentication_backoff_factor="1secs" --authorizer="local"
> --container_disk_watch_interval="15secs" --containerizers="mesos"
> --credential="/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/FaultToleranceTest_ReregisterCompletedFrameworks_UqvwBG/credential"
> --default_role="*" --disk_watch_interval="1mins" --docker="docker"
> --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io";
> --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock"
> --docker_stop_timeout="0ns"
> --docker_store_dir="/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/FaultToleranceTest_ReregisterCompletedFrameworks_UqvwBG/store/docker"
> --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume"
> --enforce_container_disk_quota="false"
> --executor_registration_timeout="1mins"
> --executor_reregistration_timeout="2secs"
> --executor_shutdown_grace_period="5secs"
> --fetcher_cache_dir="/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/FaultToleranceTest_ReregisterCompletedFrameworks_UqvwBG/fetch"
> --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks"
> --gc_disk_headroom="0.1" --hadoop_home="" --help="false"
> --hostname_lookup="true" --http_command_executor="false"
> --http_credentials="/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/FaultToleranceTest_ReregisterCompletedFrameworks_UqvwBG/http_credentials"
> --http_heartbeat_interval="30secs" --initialize_driver_logging="true"
> --isolation="posix/cpu,posix/mem"
> --jwt_secret_key="/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/FaultToleranceTest_ReregisterCompletedFrameworks_UqvwBG/jwt_secret_key"
> --launcher="posix"
> --launcher_dir="/Users/jenkins/workspace/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mac/mesos/build/src"
> --logbufsecs="0" --logging_level="INFO"
> --max_completed_executors_per_framework="150"
> --oversubscribed_resources_interval="15secs" --port="5051"
> --qos_correction_interval_min="0ns" --quiet="false"
> --reconfiguration_policy="equal" --recover="reconnect"
> --recovery_timeout="15mins" --registration_backoff_factor="10ms"
> --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]"
> --runtime_dir="/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/FaultToleranceTest_ReregisterCompletedFrameworks_UqvwBG"
> --sandbox_directory="/mnt/mesos/sandbox" --strict="true"
> --switch_user="true" --version="false"
> --work_dir="/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/FaultToleranceTest_ReregisterCompletedFrameworks_1NHm2G"
> --zk_session_timeout="10secs"
> I0314 14:30:11.249145 55676928 credentials.hpp:86] Loading credential for
> authentication from
> '/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/FaultToleranceTest_ReregisterCompletedFrameworks_UqvwBG/credential'
> I0314 14:30:11.249279 55676928 slave.cpp:298] Agent using credential for:
> test-principal
> I0314 14:30:11.249297 55676928 credentials.hpp:37] Loading credentials for
> authentication from
> '/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/FaultToleranceTest_ReregisterCompletedFrameworks_UqvwBG/http_credentials'
> I0314 14:30:11.249531 55676928 http.cpp:957] Creating default 'basic' HTTP
> authenticator for realm 'mesos-agent-executor'
> I0314 14:30:11.249590 55676928 http.cpp:978] Creating default 'jwt' HTTP
> authenticator for realm 'mesos-agent-executor'
> I0314 14:30:11.249687 55676928 http.cpp:957] Creating default 'basic' HTTP
> authenticator for realm 'mesos-agent-readonly'
> I0314 14:30:11.249727 55676928 http.cpp:978] Creating default 'jwt' HTTP
> authenticator for realm 'mesos-agent-readonly'
> I0314 14:30:11.249786 55676928 http.cpp:957] Creating default 'basic' HTTP
> authenticator for realm 'mesos-agent-readwrite'
> I0314 14:30:11.249827 55676928 http.cpp:978] Creating default 'jwt' HTTP
> authenticator for realm 'mesos-agent-readwrite'
> I0314 14:30:11.249979 52457472 process.cpp:3564] Handling HTTP event for
> process 'master' with path: '/master/state'
> I0314 14:30:11.250329 55676928 slave.cpp:615] Agent resources:
> [{"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}]
> I0314 14:30:11.250526 54603776 http.cpp:1097] HTTP GET for /master/state from
> 10.0.49.4:55065
> I0314 14:30:11.250535 55676928 slave.cpp:623] Agent attributes: [ ]
> I0314 14:30:11.250561 55676928 slave.cpp:632] Agent hostname:
> Jenkinss-Mac-mini.local
> I0314 14:30:11.250732 51920896 task_status_update_manager.cpp:181] Pausing
> sending task status updates
> F0314 14:30:11.251163 53530624 authorizer.cpp:321] Check failed: object->task
> != nullptr || object->task_info != nullptr
> *** Check failure stack trace: ***
> I0314 14:30:11.251318 52457472 state.cpp:66] Recovering state from
> '/var/folders/6w/rw03zh013y38ys6cyn8qppf80000gn/T/FaultToleranceTest_ReregisterCompletedFrameworks_1NHm2G/meta'
> I0314 14:30:11.251448 54603776 task_status_update_manager.cpp:207] Recovering
> task status update manager
> I0314 14:30:11.251709 54067200 slave.cpp:7218] Finished recovery
> I0314 14:30:11.252424 52994048 task_status_update_manager.cpp:181] Pausing
> sending task status updates
> I0314 14:30:11.252446 54603776 slave.cpp:1266] New master detected at
> [email protected]:54648
> I0314 14:30:11.252493 54603776 slave.cpp:1321] Detecting new master
> I0314 14:30:11.262979 54067200 slave.cpp:1348] Authenticating with master
> [email protected]:54648
> I0314 14:30:11.263046 54067200 slave.cpp:1357] Using default CRAM-MD5
> authenticatee
> I0314 14:30:11.263149 55140352 authenticatee.cpp:121] Creating new client
> SASL connection
> @ 0x11af0ff0a google::LogMessage::Fail()
> @ 0x11af0dbbc google::LogMessage::SendToLog()
> @ 0x11af0e949 google::LogMessage::Flush()
> @ 0x11af17bc9 google::LogMessageFatal::~LogMessageFatal()
> @ 0x11af103f5 google::LogMessageFatal::~LogMessageFatal()
> @ 0x1173931a2
> mesos::internal::LocalAuthorizerObjectApprover::approved()
> @ 0x117ccf66b
> _ZN5mesos15ObjectApprovers8approvedILNS_13authorization6ActionE18EJEEEbDpRKT0_
> @ 0x117ccddfe
> _ZZZNK5mesos8internal6master6Master4Http5stateERKN7process4http7RequestERK6OptionINS5_14authentication9PrincipalEEENK4$_15clERKNS4_5OwnedINS_15ObjectApproversEEEENKUlPN4JSON12ObjectWriterEE_clESN_
> @ 0x117ccc3dc
> _ZZN4JSON8internal7jsonifyIZZNK5mesos8internal6master6Master4Http5stateERKN7process4http7RequestERK6OptionINS8_14authentication9PrincipalEEENK4$_15clERKNS7_5OwnedINS2_15ObjectApproversEEEEUlPNS_12ObjectWriterEE_vEENSt3__18functionIFvPNSR_13basic_ostreamIcNSR_11char_traitsIcEEEEEEERKT_NS0_6PreferEENKUlSX_E_clESX_
> @ 0x117ccc370
> _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4JSON8internal7jsonifyIZZNK5mesos8internal6master6Master4Http5stateERKN7process4http7RequestERK6OptionINSC_14authentication9PrincipalEEENK4$_15clERKNSB_5OwnedINS6_15ObjectApproversEEEEUlPNS3_12ObjectWriterEE_vEENS_8functionIFvPNS_13basic_ostreamIcNS_11char_traitsIcEEEEEEERKT_NS4_6PreferEEUlS10_E_S10_EEEvDpOT_
> @ 0x117ccc269
> _ZNSt3__110__function6__funcIZN4JSON8internal7jsonifyIZZNK5mesos8internal6master6Master4Http5stateERKN7process4http7RequestERK6OptionINSB_14authentication9PrincipalEEENK4$_15clERKNSA_5OwnedINS5_15ObjectApproversEEEEUlPNS2_12ObjectWriterEE_vEENS_8functionIFvPNS_13basic_ostreamIcNS_11char_traitsIcEEEEEEERKT_NS3_6PreferEEUlSZ_E_NS_9allocatorIS16_EES10_EclEOSZ_
> @ 0x108544677 std::__1::function<>::operator()()
> @ 0x1173b8f34 JSON::operator<<()
> @ 0x11a937c97 process::http::OK::OK()
> @ 0x11a9389c5 process::http::OK::OK()
> @ 0x117ccae1f
> mesos::internal::master::Master::Http::state()::$_15::operator()()
> @ 0x117ce51a4
> _ZN5cpp176invokeIZNK5mesos8internal6master6Master4Http5stateERKN7process4http7RequestERK6OptionINS7_14authentication9PrincipalEEE4$_15JNS6_5OwnedINS1_15ObjectApproversEEEEEEDTclclsr3stdE7forwardIT_Efp_Espclsr3stdE7forwardIT0_Efp0_EEEOSL_DpOSM_
> @ 0x117ce5162
> _ZN6lambda8internal7PartialIZNK5mesos8internal6master6Master4Http5stateERKN7process4http7RequestERK6OptionINS8_14authentication9PrincipalEEE4$_15JNS7_5OwnedINS2_15ObjectApproversEEEEE13invoke_expandISI_NSt3__15tupleIJSL_EEENSP_IJEEEJLm0EEEEDTclsr5cpp17E6invokeclsr3stdE7forwardIT_Efp_Espcl6expandclsr3stdE3getIXT2_EEclsr3stdE7forwardIT0_Efp0_EEclsr3stdE7forwardIT1_Efp2_EEEEOSS_OST_N5cpp1416integer_sequenceImJXspT2_EEEEOSU_
> @ 0x117ce50c0
> _ZNO6lambda8internal7PartialIZNK5mesos8internal6master6Master4Http5stateERKN7process4http7RequestERK6OptionINS8_14authentication9PrincipalEEE4$_15JNS7_5OwnedINS2_15ObjectApproversEEEEEclIJEEEDTcl13invoke_expandclL_ZNSt3__14moveIRSI_EEONSO_16remove_referenceIT_E4typeEOSS_EdtdefpT1fEclL_ZNSP_IRNSO_5tupleIJSL_EEEEESV_SW_EdtdefpT10bound_argsEcvN5cpp1416integer_sequenceImJLm0EEEE_Eclsr3stdE16forward_as_tuplespclsr3stdE7forwardIT_Efp_EEEEDpOS13_
> @ 0x117ce5054
> _ZN5cpp176invokeIN6lambda8internal7PartialIZNK5mesos8internal6master6Master4Http5stateERKN7process4http7RequestERK6OptionINSA_14authentication9PrincipalEEE4$_15JNS9_5OwnedINS4_15ObjectApproversEEEEEEJEEEDTclclsr3stdE7forwardIT_Efp_Espclsr3stdE7forwardIT0_Efp0_EEEOSP_DpOSQ_
> @ 0x117ce4fb3
> _ZN6lambda8internal6InvokeIN7process6FutureINS2_4http8ResponseEEEEclINS0_7PartialIZNK5mesos8internal6master6Master4Http5stateERKNS4_7RequestERK6OptionINS4_14authentication9PrincipalEEE4$_15JNS2_5OwnedINSA_15ObjectApproversEEEEEEJEEES6_OT_DpOT0_
> @ 0x117ce4e6d
> _ZNO6lambda12CallableOnceIFN7process6FutureINS1_4http8ResponseEEEvEE10CallableFnINS_8internal7PartialIZNK5mesos8internal6master6Master4Http5stateERKNS3_7RequestERK6OptionINS3_14authentication9PrincipalEEE4$_15JNS1_5OwnedINSB_15ObjectApproversEEEEEEEclEv
> @ 0x1174a87e4
> _ZNO6lambda12CallableOnceIFN7process6FutureINS1_4http8ResponseEEEvEEclEv
> @ 0x1174a8620
> _ZZN7process8internal8DispatchINS_6FutureINS_4http8ResponseEEEEclIN6lambda12CallableOnceIFS5_vEEEEES5_RKNS_4UPIDEOT_ENKUlNSt3__110unique_ptrINS_7PromiseIS4_EENSH_14default_deleteISK_EEEEOSB_PNS_11ProcessBaseEE_clESN_SO_SQ_
> @ 0x1174a821f
> _ZN5cpp176invokeIZN7process8internal8DispatchINS1_6FutureINS1_4http8ResponseEEEEclIN6lambda12CallableOnceIFS7_vEEEEES7_RKNS1_4UPIDEOT_EUlNSt3__110unique_ptrINS1_7PromiseIS6_EENSJ_14default_deleteISM_EEEEOSD_PNS1_11ProcessBaseEE_JSP_SD_SS_EEEDTclclsr3stdE7forwardISH_Efp_Espclsr3stdE7forwardIT0_Efp0_EEESI_DpOSU_
> @ 0x1174a8052
> _ZN6lambda8internal7PartialIZN7process8internal8DispatchINS2_6FutureINS2_4http8ResponseEEEEclINS_12CallableOnceIFS8_vEEEEES8_RKNS2_4UPIDEOT_EUlNSt3__110unique_ptrINS2_7PromiseIS7_EENSJ_14default_deleteISM_EEEEOSD_PNS2_11ProcessBaseEE_JSP_SD_NSJ_12placeholders4__phILi1EEEEE13invoke_expandIST_NSJ_5tupleIJSP_SD_SW_EEENSZ_IJOSS_EEEJLm0ELm1ELm2EEEEDTclsr5cpp17E6invokeclsr3stdE7forwardISH_Efp_Espcl6expandclsr3stdE3getIXT2_EEclsr3stdE7forwardIT0_Efp0_EEclsr3stdE7forwardIT1_Efp2_EEEESI_OS13_N5cpp1416integer_sequenceImJXspT2_EEEEOS14_
> @ 0x1174a7f33
> _ZNO6lambda8internal7PartialIZN7process8internal8DispatchINS2_6FutureINS2_4http8ResponseEEEEclINS_12CallableOnceIFS8_vEEEEES8_RKNS2_4UPIDEOT_EUlNSt3__110unique_ptrINS2_7PromiseIS7_EENSJ_14default_deleteISM_EEEEOSD_PNS2_11ProcessBaseEE_JSP_SD_NSJ_12placeholders4__phILi1EEEEEclIJSS_EEEDTcl13invoke_expandclL_ZNSJ_4moveIRST_EEONSJ_16remove_referenceISH_E4typeESI_EdtdefpT1fEclL_ZNSZ_IRNSJ_5tupleIJSP_SD_SW_EEEEES14_SI_EdtdefpT10bound_argsEcvN5cpp1416integer_sequenceImJLm0ELm1ELm2EEEE_Eclsr3stdE16forward_as_tuplespclsr3stdE7forwardIT_Efp_EEEEDpOS1B_
> @ 0x1174a7dfd
> _ZN5cpp176invokeIN6lambda8internal7PartialIZN7process8internal8DispatchINS4_6FutureINS4_4http8ResponseEEEEclINS1_12CallableOnceIFSA_vEEEEESA_RKNS4_4UPIDEOT_EUlNSt3__110unique_ptrINS4_7PromiseIS9_EENSL_14default_deleteISO_EEEEOSF_PNS4_11ProcessBaseEE_JSR_SF_NSL_12placeholders4__phILi1EEEEEEJSU_EEEDTclclsr3stdE7forwardISJ_Efp_Espclsr3stdE7forwardIT0_Efp0_EEESK_DpOS10_
> @ 0x1174a7dc1
> _ZN6lambda8internal6InvokeIvEclINS0_7PartialIZN7process8internal8DispatchINS5_6FutureINS5_4http8ResponseEEEEclINS_12CallableOnceIFSB_vEEEEESB_RKNS5_4UPIDEOT_EUlNSt3__110unique_ptrINS5_7PromiseISA_EENSM_14default_deleteISP_EEEEOSG_PNS5_11ProcessBaseEE_JSS_SG_NSM_12placeholders4__phILi1EEEEEEJSV_EEEvSL_DpOT0_
> @ 0x1174a7a46
> _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEE10CallableFnINS_8internal7PartialIZNS1_8internal8DispatchINS1_6FutureINS1_4http8ResponseEEEEclINS0_IFSE_vEEEEESE_RKNS1_4UPIDEOT_EUlNSt3__110unique_ptrINS1_7PromiseISD_EENSO_14default_deleteISR_EEEEOSI_S3_E_JSU_SI_NSO_12placeholders4__phILi1EEEEEEEclEOS3_
> @ 0x11ac9a28f
> _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEEclES3_
> @ 0x11ac9a104 process::ProcessBase::consume()
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
