The branch, master has been updated via 3ee348a9663 ctdb-scripts: Convert 40.vsftpd to use threshold-based fail counting via 8303c3a534f ctdb-scripts: Implement failcount handling with thresholds via 4981984dd47 ctdb-scripts: Avoid errors for uninitialised counters via 7c468d9d284 ctdb-doc: Add some subsection names in description via 749bc568764 ctdb-doc: Update CTDB manual pages to UTF-8 from 7b6c17359ba tests/krb5: Test that the correct Asserted Identity SID is added when inner FX‐FAST padata is used
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 3ee348a9663c69e20fe985d6a1fad6909579fca1 Author: Martin Schwenke <mschwe...@ddn.com> Date: Tue Jul 25 09:52:56 2023 +1000 ctdb-scripts: Convert 40.vsftpd to use threshold-based fail counting This effectively provides simple testing for the threshold-based approach. Add new script option CTDB_VSFTPD_MONITOR_THRESHOLDS. Signed-off-by: Martin Schwenke <mschwe...@ddn.com> Reviewed-by: Amitay Isaacs <ami...@gmail.com> Autobuild-User(master): Amitay Isaacs <ami...@samba.org> Autobuild-Date(master): Tue Oct 3 04:53:38 UTC 2023 on atb-devel-224 commit 8303c3a534fe2d31687d9d6386ba9c8a341c7a06 Author: Martin Schwenke <mschwe...@ddn.com> Date: Fri Mar 3 17:49:05 2023 +1100 ctdb-scripts: Implement failcount handling with thresholds This can be used for simple failure counting, without restarts, as used in the 40.vsftpd event script. That case will subsequently be converted and this functionality can also be used elsewhere. Add documentation to ctdb-script.options(5) to allow parameters that use this to be more easily described. Signed-off-by: Martin Schwenke <mschwe...@ddn.com> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 4981984dd4794d0b8cb5b3cf434fc1d1ae183bfd Author: Martin Schwenke <mschwe...@ddn.com> Date: Wed Mar 15 17:56:40 2023 +1100 ctdb-scripts: Avoid errors for uninitialised counters Uninitialised counters are treated as 0, but still produce an error. The redirect to stderr needs to come before the redirect for a missing counter file. The seemingly saner alternative of moving it outside the subshell works when dash is /bin/sh (e.g. on Debian) but does not work when bash is /bin/sh (e.g. on Fedora). Signed-off-by: Martin Schwenke <mschwe...@ddn.com> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 7c468d9d2846f6aab3f4963085413478e6aba895 Author: Martin Schwenke <mschwe...@ddn.com> Date: Tue Jul 25 12:19:07 2023 +1000 ctdb-doc: Add some subsection names in description A subsequent commit will add a new section, which looks out of place without these new sections. Best reviewed with "git show -w". Signed-off-by: Martin Schwenke <mschwe...@ddn.com> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 749bc568764bb20d2be3450a030cf68bc223a4b7 Author: Martin Schwenke <mschwe...@ddn.com> Date: Tue Jul 25 11:42:32 2023 +1000 ctdb-doc: Update CTDB manual pages to UTF-8 This will allow Unicode characters to be used, resulting in more readable source files. Signed-off-by: Martin Schwenke <mschwe...@ddn.com> Reviewed-by: Amitay Isaacs <ami...@gmail.com> ----------------------------------------------------------------------- Summary of changes: ctdb/config/events/legacy/40.vsftpd.script | 17 ++- ctdb/config/functions | 110 +++++++++++++++- ctdb/doc/ctdb-etcd.7.xml | 2 +- ctdb/doc/ctdb-script.options.5.xml | 141 +++++++++++++++------ ctdb/doc/ctdb-statistics.7.xml | 2 +- ctdb/doc/ctdb-tunables.7.xml | 2 +- ctdb/doc/ctdb.1.xml | 2 +- ctdb/doc/ctdb.7.xml | 2 +- ctdb/doc/ctdb.conf.5.xml | 2 +- ctdb/doc/ctdb.sysconfig.5.xml | 2 +- ctdb/doc/ctdb_diagnostics.1.xml | 2 +- ctdb/doc/ctdb_mutex_ceph_rados_helper.7.xml | 2 +- ctdb/doc/ctdbd.1.xml | 2 +- ctdb/doc/ltdbtool.1.xml | 2 +- ctdb/doc/onnode.1.xml | 2 +- ctdb/doc/ping_pong.1.xml | 2 +- .../UNIT/eventscripts/40.vsftpd.monitor.002.sh | 39 +++++- ctdb/tests/UNIT/eventscripts/scripts/40.vsftpd.sh | 2 + 18 files changed, 269 insertions(+), 66 deletions(-) Changeset truncated at 500 lines: diff --git a/ctdb/config/events/legacy/40.vsftpd.script b/ctdb/config/events/legacy/40.vsftpd.script index 1202812c3cd..2d2aac47033 100755 --- a/ctdb/config/events/legacy/40.vsftpd.script +++ b/ctdb/config/events/legacy/40.vsftpd.script @@ -19,11 +19,13 @@ load_script_options ctdb_setup_state_dir "service" "$service_name" +port_21="vsftpd listening on TCP port 21" + case "$1" in startup) service "$service_name" stop > /dev/null 2>&1 service "$service_name" start - ctdb_counter_init + failcount_init "$port_21" ;; shutdown) @@ -42,15 +44,12 @@ ipreallocated) monitor) if ctdb_check_tcp_ports 21 ; then - ctdb_counter_init + failcount_reset "$port_21" else - ctdb_counter_incr - num_fails=$(ctdb_counter_get) - if [ "$num_fails" -ge 2 ] ; then - die "ERROR: ${num_fails} consecutive failures for vsftpd, marking node unhealthy" - elif [ "$num_fails" -eq 1 ] ; then - echo "WARNING: vsftpd not listening but less than 2 consecutive failures, not unhealthy yet" - fi + # Set defaults, if unset + : "${CTDB_VSFTPD_MONITOR_THRESHOLDS:=1:2}" + + failcount_incr "$port_21" "$CTDB_VSFTPD_MONITOR_THRESHOLDS" fi ;; esac diff --git a/ctdb/config/functions b/ctdb/config/functions index 56105aab165..d8f7f57b84c 100755 --- a/ctdb/config/functions +++ b/ctdb/config/functions @@ -754,12 +754,120 @@ ctdb_counter_get() { _ctdb_counter_common "$1" # unary counting! - _val=$(wc -c <"$_counter_file" 2>/dev/null || echo 0) + _val=$(wc -c 2>/dev/null <"$_counter_file" || echo 0) # Strip leading spaces from output of wc (on freebsd) # shellcheck disable=SC2086 echo $_val } +# +# Fail counter/threshold combination to control warnings and node unhealthy +# + +_failcount_validate_threshold() +{ + case "$1" in + "") return 1 ;; # A failure that doesn't need a warning + *) + if echo "$1" | grep -qx '[0-9]*'; then + return 0 + fi + + echo "WARNING: ${1} is an invalid threshold in \"${2}\" check" + return 1 + ;; + esac +} + +_failcount_common() +{ + _thing="$1" + + _counter=$(echo "$_thing" | sed -e 's@/@_SLASH_@g' -e 's@ @_@g') +} + +failcount_init() +{ + _thing="$1" + + _failcount_common "$_thing" + + ctdb_counter_init "$_counter" +} + +failcount_reset() +{ + _thing="$1" + + _failcount_common "$_thing" + + _failcount=$(ctdb_counter_get "$_counter") + if [ "$_failcount" -eq 0 ]; then + return + fi + + printf 'NOTICE: %s: no longer failing\n' "$_thing" + ctdb_counter_init "$_counter" +} + +failcount_incr() +{ + _thing="$1" + _thresholds="$2" + _output="$3" + + _failcount_common "$_thing" + + ctdb_counter_incr "$_counter" + _failcount=$(ctdb_counter_get "$_counter") + + case "$_thresholds" in + *:*) + _warn_threshold="${_thresholds%:*}" + _unhealthy_threshold="${_thresholds#*:}" + ;; + "") + _warn_threshold=1 + _unhealthy_threshold="" + ;; + *) + _warn_threshold="$_thresholds" + _unhealthy_threshold="" + ;; + esac + + if _failcount_validate_threshold "$_unhealthy_threshold" "$_thing"; then + if [ "$_failcount" -ge "$_unhealthy_threshold" ]; then + printf 'ERROR: %s: fail count %d >= threshold %d\n' \ + "$_thing" \ + "$_failcount" \ + "$_unhealthy_threshold" + # Only print output when exceeding the + # unhealthy threshold + if [ "$_failcount" -eq "$_unhealthy_threshold" ] && \ + [ -n "$_output" ]; then + echo "$_output" + fi + exit 1 + fi + fi + + if _failcount_validate_threshold "$_warn_threshold" "$_thing"; then + if [ "$_failcount" -lt "$_warn_threshold" ]; then + return 0 + fi + fi + + printf 'WARNING: %s: fail count %d >= threshold %d\n' \ + "$_thing" \ + "$_failcount" \ + "$_warn_threshold" + if [ "$_failcount" -eq "$_warn_threshold" ] && [ -n "$_output" ]; then + # Only print output when exceeding the warning threshold + echo "$_output" + fi +} + ######################################################## # ctdb_setup_state_dir <type> <name> diff --git a/ctdb/doc/ctdb-etcd.7.xml b/ctdb/doc/ctdb-etcd.7.xml index f84989f854f..af343dbaa57 100644 --- a/ctdb/doc/ctdb-etcd.7.xml +++ b/ctdb/doc/ctdb-etcd.7.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> diff --git a/ctdb/doc/ctdb-script.options.5.xml b/ctdb/doc/ctdb-script.options.5.xml index 700d1f7bd35..a01b10a949c 100644 --- a/ctdb/doc/ctdb-script.options.5.xml +++ b/ctdb/doc/ctdb-script.options.5.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> @@ -20,51 +20,81 @@ <refsect1> <title>DESCRIPTION</title> - <para> - Each CTDB script has 2 possible locations for its configuration options: - </para> + <refsect2> + <title>Location</title> + <para> + Each CTDB script has 2 possible locations for its configuration options: + </para> - <variablelist> + <variablelist> - <varlistentry> - <term> - <filename>/usr/local/etc/ctdb/script.options</filename> - </term> - <listitem> - <para> - This is a catch-all global file for general purpose - scripts and for options that are used in multiple event - scripts. - </para> - </listitem> - </varlistentry> + <varlistentry> + <term> + <filename>/usr/local/etc/ctdb/script.options</filename> + </term> + <listitem> + <para> + This is a catch-all global file for general purpose + scripts and for options that are used in multiple event + scripts. + </para> + </listitem> + </varlistentry> - <varlistentry> - <term> - <parameter>SCRIPT</parameter>.options - </term> - <listitem> - <para> - That is, options for - <filename><parameter>SCRIPT</parameter></filename> are - placed in a file alongside the script, with a ".script" - suffix added. This style is usually recommended for event - scripts. - </para> + <varlistentry> + <term> + <parameter>SCRIPT</parameter>.options + </term> + <listitem> + <para> + That is, options for + <filename><parameter>SCRIPT</parameter></filename> are + placed in a file alongside the script, with a ".script" + suffix added. This style is usually recommended for event + scripts. + </para> - <para> - Options in this script-specific file override those in - the global file. - </para> - </listitem> - </varlistentry> + <para> + Options in this script-specific file override those in + the global file. + </para> + </listitem> + </varlistentry> - </variablelist> + </variablelist> + </refsect2> - <para> - These files should include simple shell-style variable - assignments and shell-style comments. - </para> + <refsect2> + <title>Contents</title> + + <para> + These files should include simple shell-style variable + assignments and shell-style comments. + </para> + </refsect2> + + <refsect2> + <title>Monitoring Thresholds</title> + + <para> + Event scripts can monitor resources or services. When a + problem is detected, it may be better to warn about a problem + rather than to immediately fail monitoring and mark a node as + unhealthy. CTDB provides support for event scripts to do + threshold-based monitoring. + </para> + + <para> + A threshold setting looks like + <parameter>WARNING_THRESHOLD<optional>:ERROR_THRESHOLD</optional></parameter>. + If the number of problems is ≥ WARNING_THRESHOLD then the + script will log a warning and continue. If the number + problems is ≥ ERROR_THRESHOLD then the script will log an + error and exit with failure, causing monitoring to fail. Note + that ERROR_THRESHOLD is optional, and follows the optional + colon (:) separator. + </para> + </refsect2> </refsect1> @@ -581,6 +611,37 @@ CTDB_PER_IP_ROUTING_TABLE_ID_HIGH=9000 </refsect2> + <refsect2> + <title>40.vsftpd</title> + + <para> + Provides CTDB's vsftpd service management. + </para> + + <variablelist> + <varlistentry> + <term> + CTDB_VSFTPD_MONITOR_THRESHOLDS=<parameter>THRESHOLDS</parameter> + </term> + <listitem> + <para> + THRESHOLDS indicates how many consecutive monitoring + attempts need to report that vsftpd is not listening on + TCP port 21 before a warning is logged and before + monitoring fails. See the <citetitle>Monitoring + Thresholds</citetitle> for a description of how + monitoring thresholds work. + </para> + <para> + Default is 1:2. + </para> + </listitem> + </varlistentry> + + </variablelist> + + </refsect2> + <refsect2> <title>48.netbios</title> diff --git a/ctdb/doc/ctdb-statistics.7.xml b/ctdb/doc/ctdb-statistics.7.xml index 387852d5498..0d10484f866 100644 --- a/ctdb/doc/ctdb-statistics.7.xml +++ b/ctdb/doc/ctdb-statistics.7.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> diff --git a/ctdb/doc/ctdb-tunables.7.xml b/ctdb/doc/ctdb-tunables.7.xml index df7b0ea9dc9..725c781e47e 100644 --- a/ctdb/doc/ctdb-tunables.7.xml +++ b/ctdb/doc/ctdb-tunables.7.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> diff --git a/ctdb/doc/ctdb.1.xml b/ctdb/doc/ctdb.1.xml index 2973fbf2832..75934ef50e3 100644 --- a/ctdb/doc/ctdb.1.xml +++ b/ctdb/doc/ctdb.1.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> diff --git a/ctdb/doc/ctdb.7.xml b/ctdb/doc/ctdb.7.xml index 351f3e8b6f7..0f3fbc6bf6a 100644 --- a/ctdb/doc/ctdb.7.xml +++ b/ctdb/doc/ctdb.7.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> diff --git a/ctdb/doc/ctdb.conf.5.xml b/ctdb/doc/ctdb.conf.5.xml index 87a7ea594ea..b9bf3a6d08b 100644 --- a/ctdb/doc/ctdb.conf.5.xml +++ b/ctdb/doc/ctdb.conf.5.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> diff --git a/ctdb/doc/ctdb.sysconfig.5.xml b/ctdb/doc/ctdb.sysconfig.5.xml index 26189348360..b4cdaaf633e 100644 --- a/ctdb/doc/ctdb.sysconfig.5.xml +++ b/ctdb/doc/ctdb.sysconfig.5.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> diff --git a/ctdb/doc/ctdb_diagnostics.1.xml b/ctdb/doc/ctdb_diagnostics.1.xml index 274c70d10e4..2f28131c86c 100644 --- a/ctdb/doc/ctdb_diagnostics.1.xml +++ b/ctdb/doc/ctdb_diagnostics.1.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> diff --git a/ctdb/doc/ctdb_mutex_ceph_rados_helper.7.xml b/ctdb/doc/ctdb_mutex_ceph_rados_helper.7.xml index dd3dbabdd50..f558f873d9a 100644 --- a/ctdb/doc/ctdb_mutex_ceph_rados_helper.7.xml +++ b/ctdb/doc/ctdb_mutex_ceph_rados_helper.7.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> diff --git a/ctdb/doc/ctdbd.1.xml b/ctdb/doc/ctdbd.1.xml index 314964e8ca6..c0462945857 100644 --- a/ctdb/doc/ctdbd.1.xml +++ b/ctdb/doc/ctdbd.1.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> diff --git a/ctdb/doc/ltdbtool.1.xml b/ctdb/doc/ltdbtool.1.xml index 790db0ce2fe..875131a0ddb 100644 --- a/ctdb/doc/ltdbtool.1.xml +++ b/ctdb/doc/ltdbtool.1.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> diff --git a/ctdb/doc/onnode.1.xml b/ctdb/doc/onnode.1.xml index f7da04aedbe..8e58ee78cf5 100644 --- a/ctdb/doc/onnode.1.xml +++ b/ctdb/doc/onnode.1.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> diff --git a/ctdb/doc/ping_pong.1.xml b/ctdb/doc/ping_pong.1.xml index 47e90e896a7..cbb30b5a7d3 100644 --- a/ctdb/doc/ping_pong.1.xml +++ b/ctdb/doc/ping_pong.1.xml @@ -1,4 +1,4 @@ -<?xml version="1.0" encoding="iso-8859-1"?> +<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> diff --git a/ctdb/tests/UNIT/eventscripts/40.vsftpd.monitor.002.sh b/ctdb/tests/UNIT/eventscripts/40.vsftpd.monitor.002.sh index 4426e1b103c..f825be4d02a 100755 --- a/ctdb/tests/UNIT/eventscripts/40.vsftpd.monitor.002.sh +++ b/ctdb/tests/UNIT/eventscripts/40.vsftpd.monitor.002.sh @@ -2,18 +2,51 @@ . "${TEST_SCRIPTS_DIR}/unit.sh" -define_test "managed, down - once, twice" +define_test "up once, down with recovery" + +setup "up" + +ok_null +simple_test setup "down" ok <<EOF vsftpd not listening on TCP port 21 -WARNING: vsftpd not listening but less than 2 consecutive failures, not unhealthy yet +WARNING: vsftpd listening on TCP port 21: fail count 1 >= threshold 1 +EOF +simple_test + +setup "up" + +ok <<EOF +NOTICE: vsftpd listening on TCP port 21: no longer failing +EOF +simple_test + +setup "down" + +ok <<EOF -- Samba Shared Repository