The branch, master has been updated via ffd64de ctdb-daemon: Drop --lvs option and support for CTDB_CAP_LVS via 951e818 ctdb-daemon: Drop --single-public-ip option and related code via dfe8de9 ctdb-scripts: Simplify "ctdb lvs ..." output via e3a7178 ctdb-tools: Change ctdb CLI to have a single "lvs" command via 9543e35 ctdb-tool: Change ctdb lvs/lvsmaster CLI commands to use ctdb_lvs helper via a92394e ctdb-scripts: Add monitoring of CTDB_LVS_PUBLIC_IFACE via 798b131 ctdb-doc: Add sample LVS configuration via ce33a7c ctdb-scripts: Call out to ctdb_lvs helper from 91.lvs via b737686 ctdb-scripts: Move ctdb_get_ip_address() to functions file via 0352224 ctdb-tools: Add new ctdb_lvs helper via 7d52279 ctdb-scripts: LVS eventscript cleanups via 5fc9ca91 ctdb-tests: Add unit tests for LVS eventscript via 89dc86e ctdb-tests: LVS support for ctdb tool stub via e2ecdd2 ctdb-tests: Add ipvsadm test stub via 47a68ff ctdb-tests: Add 32-bit netmask support to "ip addr show" stub via 8305c8d ctdb-tests: Add loopback support for "ip link show" stub via 6237b08 ctdb-tests: Allow scope to be specified in "ip addr add" stub via 39cbe07 ctdb-scripts: Drop "recovered" event from 91.lvs via a0ef57f ctdb-scripts: LVS eventscript error redirection improvements via 600211d ctdb-scripts: Drop hardcoded /sbin and /proc paths in LVS eventscript via 6ff9060 ctdb-scripts: die() should output to stderr from 43ea097 s3:libsmb: Fix illegal memory access after memory has been deleted.
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit ffd64de7729cbe078d8dbc2546fe8ccb472b4bb7 Author: Martin Schwenke <mar...@meltin.net> Date: Mon Apr 11 08:10:23 2016 +1000 ctdb-daemon: Drop --lvs option and support for CTDB_CAP_LVS Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> Autobuild-User(master): Amitay Isaacs <ami...@samba.org> Autobuild-Date(master): Fri Apr 15 09:35:51 CEST 2016 on sn-devel-144 commit 951e8180a9857484cd30ac03b4b2d9761c91ec99 Author: Martin Schwenke <mar...@meltin.net> Date: Mon Apr 11 08:02:36 2016 +1000 ctdb-daemon: Drop --single-public-ip option and related code This has been replaced by scripts. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit dfe8de9e4a6a22fd0dc8e0ce7f74df9d0cba9491 Author: Martin Schwenke <mar...@meltin.net> Date: Wed Apr 13 17:47:45 2016 +1000 ctdb-scripts: Simplify "ctdb lvs ..." output For "master", if there is a master then print the PNN, otherwise print nothing. For "list", print the PNN and IP addresses without a colon in between. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit e3a71785110316eac0dd55fbd5f2e76be31049b4 Author: Martin Schwenke <mar...@meltin.net> Date: Tue Apr 12 11:31:41 2016 +1000 ctdb-tools: Change ctdb CLI to have a single "lvs" command This simply calls out to the wrapper, so that commands are changed as follows: ctdb lvsmaster -> ctdb lvs master ctdb lvs -> ctdb lvs list This provides a simple, extensible interface and means that "ctdb lvs status" is also available. Unit tests are streamlined so that there is a single test for each CTDB state. Each test does "master", "list" and "status" sub-tests. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 9543e35557840e997793fd951c24aa8dad4e890b Author: Martin Schwenke <mar...@meltin.net> Date: Fri Apr 8 17:14:44 2016 +1000 ctdb-tool: Change ctdb lvs/lvsmaster CLI commands to use ctdb_lvs helper Update tests accordingly. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit a92394e69ce08f2b49f09ef0be1f56a4a6004ed1 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Apr 14 13:35:07 2016 +1000 ctdb-scripts: Add monitoring of CTDB_LVS_PUBLIC_IFACE ... and unit tests. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 798b131400e9d8867741b43576a17def0c53dfce Author: Amitay Isaacs <ami...@gmail.com> Date: Thu Apr 14 16:30:38 2016 +1000 ctdb-doc: Add sample LVS configuration Signed-off-by: Amitay Isaacs <ami...@gmail.com> Reviewed-by: Martin Schwenke <mar...@meltin.net> commit ce33a7cb1e6d91b56bc9261efb4bf6d53371f77e Author: Martin Schwenke <mar...@meltin.net> Date: Fri Apr 8 16:21:08 2016 +1000 ctdb-scripts: Call out to ctdb_lvs helper from 91.lvs To keep this commit comprehensible, 91.lvs and the CTDB CLI tool are temporarily inconsistent. The tool will be made consistent in a subsequent commit. LVS now uses a configuration file specified by CTDB_LVS_NODES and supports the same slave-only syntax as CTDB_NATGW_NODES. LVS also uses new variable CTDB_LVS_PUBLIC_IFACE instead of CTDB_PUBLIC_INTERFACE. Update unit tests and documentation. Note that the --lvs and --single-public-ip daemon options are no longer used. These will be removed and relevant documentation updated in a subsequent commit. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit b7376861cbbcdddff9764f622f8c0bc9526b6690 Author: Martin Schwenke <mar...@meltin.net> Date: Fri Apr 8 15:53:47 2016 +1000 ctdb-scripts: Move ctdb_get_ip_address() to functions file Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 035222426264d9f12b099612d7ce8c20d1d607e1 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Apr 7 17:30:28 2016 +1000 ctdb-tools: Add new ctdb_lvs helper This will replace the ctdb CLI tool "lvs" and "lvsmaster" options. It also makes LVS daemon support unnecessary. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 7d522794ef9a70e43d6ac85decb619c90e115144 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Apr 7 06:11:49 2016 +1000 ctdb-scripts: LVS eventscript cleanups Stylistic changes and cleaner code. No functional changes. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 5fc9ca91d7cdc3a641065f2606188a25485e4ccb Author: Martin Schwenke <mar...@meltin.net> Date: Thu Apr 7 06:18:21 2016 +1000 ctdb-tests: Add unit tests for LVS eventscript Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 89dc86e0fd77857926d78bb6b0bd4f4c72a718f4 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Apr 7 13:54:37 2016 +1000 ctdb-tests: LVS support for ctdb tool stub Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit e2ecdd2401585cdb7251791004c107ab5fa74bda Author: Martin Schwenke <mar...@meltin.net> Date: Wed Apr 6 21:19:37 2016 +1000 ctdb-tests: Add ipvsadm test stub Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 47a68ff440ceb3affb2dcca5bea79938a154f5be Author: Martin Schwenke <mar...@meltin.net> Date: Thu Apr 7 12:06:52 2016 +1000 ctdb-tests: Add 32-bit netmask support to "ip addr show" stub In this case do not show broadcast address. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 8305c8d36346764f2a5a165bfe9d8de275117a95 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Apr 7 12:05:33 2016 +1000 ctdb-tests: Add loopback support for "ip link show" stub Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 6237b08f5335ccec1c3385f2732d16f13c307506 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Apr 7 10:02:09 2016 +1000 ctdb-tests: Allow scope to be specified in "ip addr add" stub Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 39cbe07ef08279d5a9f37d7ca9617e06dfd9b290 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Apr 14 16:26:39 2016 +1000 ctdb-scripts: Drop "recovered" event from 91.lvs Having both "recovered" and "ipreallocated" means that everything happens twice when there is a recovery. No need for that. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit a0ef57fe840583383205eb561c45baa0ad870246 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Apr 7 13:55:08 2016 +1000 ctdb-scripts: LVS eventscript error redirection improvements Basic error redirection improvements before writing tests. Deleting the service during "startup" will usually fail because the service has never been setup, so redirect output to avoid logging an error. Similarly, deleting the service in "ipreallocated" will always fail the first time, which would cause an error to be logged. Given the simplicity of the script, there's no sane way to avoid the error sometimes and log it if it actually matters. This could potentially be tidied up in the future by making 91.lvs stateful, in a similar way to 11.natgw. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 600211d11b8c6dff34cc1e0d6f332fdf85ca9a34 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Apr 7 05:50:27 2016 +1000 ctdb-scripts: Drop hardcoded /sbin and /proc paths in LVS eventscript It can now be unit tested. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 6ff90602850b42b6d8c505c969816430a0232757 Author: Martin Schwenke <mar...@meltin.net> Date: Fri Apr 8 17:14:23 2016 +1000 ctdb-scripts: die() should output to stderr Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> ----------------------------------------------------------------------- Summary of changes: ctdb/config/ctdbd_wrapper | 1 - ctdb/config/events.d/11.natgw | 13 -- ctdb/config/events.d/91.lvs | 139 +++++++++----- ctdb/config/functions | 15 +- ctdb/doc/ctdb.1.xml | 73 ++++--- ctdb/doc/ctdb.7.xml | 101 ++++++---- ctdb/doc/ctdbd.1.xml | 32 +--- ctdb/doc/ctdbd.conf.5.xml | 85 +++++++- ctdb/doc/examples/lvs.conf | 90 +++++++++ ctdb/include/ctdb_private.h | 3 - ctdb/packaging/RPM/ctdb.spec.in | 1 + ctdb/protocol/protocol.h | 3 +- ctdb/server/ctdb_takeover.c | 74 +------ ctdb/server/ctdbd.c | 21 -- ctdb/tests/eventscripts/91.lvs.001.sh | 55 ++++++ .../tests/eventscripts/91.lvs.ipreallocated.011.sh | 15 ++ .../tests/eventscripts/91.lvs.ipreallocated.012.sh | 18 ++ .../tests/eventscripts/91.lvs.ipreallocated.013.sh | 18 ++ .../tests/eventscripts/91.lvs.ipreallocated.014.sh | 28 +++ ctdb/tests/eventscripts/91.lvs.monitor.001.sh | 12 ++ ctdb/tests/eventscripts/91.lvs.monitor.002.sh | 15 ++ ctdb/tests/eventscripts/91.lvs.monitor.003.sh | 20 ++ ctdb/tests/eventscripts/91.lvs.shutdown.001.sh | 12 ++ ctdb/tests/eventscripts/91.lvs.shutdown.002.sh | 19 ++ ctdb/tests/eventscripts/91.lvs.startup.001.sh | 12 ++ ctdb/tests/eventscripts/91.lvs.startup.002.sh | 15 ++ ctdb/tests/eventscripts/scripts/local.sh | 77 ++++++++ ctdb/tests/eventscripts/stubs/ctdb | 12 +- ctdb/tests/eventscripts/stubs/ctdb_lvs | 52 +++++ ctdb/tests/eventscripts/stubs/ip | 44 +++-- ctdb/tests/eventscripts/stubs/ipvsadm | 126 ++++++++++++ ctdb/tests/src/ctdb_test_stubs.c | 5 +- ctdb/tests/tool/scripts/local.sh | 24 +++ ctdb/tests/tool/stubby.getcapabilities.001.sh | 1 - ctdb/tests/tool/stubby.getcapabilities.002.sh | 1 - ctdb/tests/tool/stubby.getcapabilities.004.sh | 5 +- ctdb/tests/tool/stubby.lvs.001.sh | 31 ++- ctdb/tests/tool/stubby.lvs.002.sh | 50 ++++- ctdb/tests/tool/stubby.lvs.003.sh | 43 ++++- ctdb/tests/tool/stubby.lvs.004.sh | 62 ++++++ ctdb/tests/tool/stubby.lvs.005.sh | 63 ++++++ ctdb/tests/tool/stubby.lvs.006.sh | 61 ++++++ ctdb/tests/tool/stubby.lvs.007.sh | 59 ++++++ ctdb/tests/tool/stubby.lvs.008.sh | 66 +++++++ ctdb/tests/tool/stubby.lvsmaster.001.sh | 27 --- ctdb/tests/tool/stubby.lvsmaster.002.sh | 27 --- ctdb/tests/tool/stubby.lvsmaster.003.sh | 27 --- ctdb/tests/tool/stubby.lvsmaster.004.sh | 27 --- ctdb/tests/tool/stubby.lvsmaster.005.sh | 27 --- ctdb/tests/tool/stubby.lvsmaster.006.sh | 27 --- ctdb/tests/tool/stubby.lvsmaster.007.sh | 27 --- ctdb/tools/ctdb.c | 213 ++------------------- ctdb/tools/ctdb_lvs | 201 +++++++++++++++++++ ctdb/wscript | 7 + 54 files changed, 1581 insertions(+), 701 deletions(-) create mode 100644 ctdb/doc/examples/lvs.conf create mode 100755 ctdb/tests/eventscripts/91.lvs.001.sh create mode 100755 ctdb/tests/eventscripts/91.lvs.ipreallocated.011.sh create mode 100755 ctdb/tests/eventscripts/91.lvs.ipreallocated.012.sh create mode 100755 ctdb/tests/eventscripts/91.lvs.ipreallocated.013.sh create mode 100755 ctdb/tests/eventscripts/91.lvs.ipreallocated.014.sh create mode 100755 ctdb/tests/eventscripts/91.lvs.monitor.001.sh create mode 100755 ctdb/tests/eventscripts/91.lvs.monitor.002.sh create mode 100755 ctdb/tests/eventscripts/91.lvs.monitor.003.sh create mode 100755 ctdb/tests/eventscripts/91.lvs.shutdown.001.sh create mode 100755 ctdb/tests/eventscripts/91.lvs.shutdown.002.sh create mode 100755 ctdb/tests/eventscripts/91.lvs.startup.001.sh create mode 100755 ctdb/tests/eventscripts/91.lvs.startup.002.sh create mode 100755 ctdb/tests/eventscripts/stubs/ctdb_lvs create mode 100755 ctdb/tests/eventscripts/stubs/ipvsadm create mode 100755 ctdb/tests/tool/stubby.lvs.004.sh create mode 100755 ctdb/tests/tool/stubby.lvs.005.sh create mode 100755 ctdb/tests/tool/stubby.lvs.006.sh create mode 100755 ctdb/tests/tool/stubby.lvs.007.sh create mode 100755 ctdb/tests/tool/stubby.lvs.008.sh delete mode 100755 ctdb/tests/tool/stubby.lvsmaster.001.sh delete mode 100755 ctdb/tests/tool/stubby.lvsmaster.002.sh delete mode 100755 ctdb/tests/tool/stubby.lvsmaster.003.sh delete mode 100755 ctdb/tests/tool/stubby.lvsmaster.004.sh delete mode 100755 ctdb/tests/tool/stubby.lvsmaster.005.sh delete mode 100755 ctdb/tests/tool/stubby.lvsmaster.006.sh delete mode 100755 ctdb/tests/tool/stubby.lvsmaster.007.sh create mode 100755 ctdb/tools/ctdb_lvs Changeset truncated at 500 lines: diff --git a/ctdb/config/ctdbd_wrapper b/ctdb/config/ctdbd_wrapper index be251e6..b1f71c3 100755 --- a/ctdb/config/ctdbd_wrapper +++ b/ctdb/config/ctdbd_wrapper @@ -163,7 +163,6 @@ build_ctdb_options () maybe_set "--start-as-stopped " "$CTDB_START_AS_STOPPED" "yes" maybe_set "--no-recmaster" "$CTDB_CAPABILITY_RECMASTER" "no" maybe_set "--no-lmaster" "$CTDB_CAPABILITY_LMASTER" "no" - maybe_set "--lvs --single-public-ip" "$CTDB_LVS_PUBLIC_IP" maybe_set "--script-log-level" "$CTDB_SCRIPT_LOG_LEVEL" maybe_set "--max-persistent-check-errors" "$CTDB_MAX_PERSISTENT_CHECK_ERRORS" } diff --git a/ctdb/config/events.d/11.natgw b/ctdb/config/events.d/11.natgw index 25cf27f..bc00788 100755 --- a/ctdb/config/events.d/11.natgw +++ b/ctdb/config/events.d/11.natgw @@ -23,19 +23,6 @@ natgw_cfg_new="${service_state_dir}/cfg_new" natgw_cfg_old="${service_state_dir}/cfg_old" natgw_master_old="${service_state_dir}/master_old" -# Cached retrieval of private IP address from local node. This never -# changes. Sets $ip_address to avoid an unnecessary subprocess. -ctdb_get_ip_address () -{ - _ip_addr_file="${service_state_dir}/my-ip-address" - if [ ! -f "$_ip_addr_file" ] ; then - ctdb -X nodestatus | - awk -F '|' 'NR == 2 { print $3 }' >"$_ip_addr_file" - fi - - read ip_address <"$_ip_addr_file" -} - ctdb_natgw_slave_only () { ctdb_get_ip_address diff --git a/ctdb/config/events.d/91.lvs b/ctdb/config/events.d/91.lvs index 57ee2c1..141f964 100755 --- a/ctdb/config/events.d/91.lvs +++ b/ctdb/config/events.d/91.lvs @@ -8,82 +8,119 @@ loadconfig ctdb -[ -z "$CTDB_LVS_PUBLIC_IP" ] && exit 0 -[ -z "$CTDB_PUBLIC_INTERFACE" ] && exit 0 +[ -n "$CTDB_LVS_NODES" ] || exit 0 +export CTDB_LVS_NODES -[ -x /sbin/ipvsadm ] || { - echo "LVS configured but /sbin/ipvsadm is not installed." - exit 0 +if ! type ipvsadm >/dev/null 2>&1 ; then + echo "LVS configured but ipvsadm not found" + exit 0 +fi + + +lvs_slave_only () +{ + ctdb_get_ip_address + + awk -v my_ip="$ip_address" \ + '$1 == my_ip { if ($2 ~ "slave-only") { exit 0 } else { exit 1 } }' \ + "$CTDB_LVS_NODES" } -case "$1" in - startup) - ipvsadm -D -t $CTDB_LVS_PUBLIC_IP:0 - ipvsadm -D -u $CTDB_LVS_PUBLIC_IP:0 +lvs_check_config () +{ + [ -r "$CTDB_LVS_NODES" ] || \ + die "error: CTDB_LVS_NODES=${CTDB_LVS_NODES} unreadable" + [ -n "$CTDB_LVS_PUBLIC_IP" ] || \ + die "Invalid configuration: CTDB_LVS_PUBLIC_IP not set" + if ! lvs_slave_only ; then + [ -n "$CTDB_LVS_PUBLIC_IFACE" ] || \ + die "Invalid configuration: CTDB_LVS_PUBLIC_IFACE not set" + fi + + if [ "$CTDB_PARTIALLY_ONLINE_INTERFACES" = "yes" ] ; then + die "Invalid configuration: CTDB_PARTIALLY_ONLINE_INTERFACES=yes incompatible with LVS" + fi +} + +case "$1" in +setup) + lvs_check_config + ;; +startup) + lvs_check_config - ip addr add $CTDB_LVS_PUBLIC_IP/32 dev lo scope host >/dev/null 2>/dev/null + ipvsadm -D -t "$CTDB_LVS_PUBLIC_IP" >/dev/null 2>&1 + ipvsadm -D -u "$CTDB_LVS_PUBLIC_IP" >/dev/null 2>&1 + + ip addr add "${CTDB_LVS_PUBLIC_IP}/32" dev lo scope host # do not respond to ARPs that are for ip addresses with scope 'host' - echo 3 > /proc/sys/net/ipv4/conf/all/arp_ignore + set_proc_maybe sys/net/ipv4/conf/all/arp_ignore 3 # do not send out arp requests from loopback addresses - echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce + set_proc_maybe sys/net/ipv4/conf/all/arp_announce 2 ;; - shutdown) - ipvsadm -D -t $CTDB_LVS_PUBLIC_IP:0 - ipvsadm -D -u $CTDB_LVS_PUBLIC_IP:0 +shutdown) + lvs_check_config + + ipvsadm -D -t "$CTDB_LVS_PUBLIC_IP" + ipvsadm -D -u "$CTDB_LVS_PUBLIC_IP" - # remove the ip - ip addr del $CTDB_LVS_PUBLIC_IP/32 dev lo >/dev/null 2>/dev/null + ip addr del "${CTDB_LVS_PUBLIC_IP}/32" dev lo >/dev/null 2>&1 - # flush our route cache - echo 1 > /proc/sys/net/ipv4/route/flush + flush_route_cache ;; - recovered|ipreallocated) - # kill off any tcp connections - ipvsadm -D -t $CTDB_LVS_PUBLIC_IP:0 - ipvsadm -D -u $CTDB_LVS_PUBLIC_IP:0 +ipreallocated) + lvs_check_config + + # Kill connections + ipvsadm -D -t "$CTDB_LVS_PUBLIC_IP" >/dev/null 2>&1 + ipvsadm -D -u "$CTDB_LVS_PUBLIC_IP" >/dev/null 2>&1 kill_tcp_connections_local_only \ - "$CTDB_PUBLIC_INTERFACE" "$CTDB_LVS_PUBLIC_IP" + "$CTDB_LVS_PUBLIC_IFACE" "$CTDB_LVS_PUBLIC_IP" ctdb_get_pnn - LVSMASTER=`ctdb lvsmaster | sed -e "s/.*Node //" -e "s/ .*//"` - - [ "$pnn" != "$LVSMASTER" ] && { - # we are not the lvs master so we have to - # change the ip address to have scope host so we wont respond - # to arps - ip addr del $CTDB_LVS_PUBLIC_IP/32 dev lo >/dev/null 2>/dev/null - ip addr add $CTDB_LVS_PUBLIC_IP/32 dev lo scope host >/dev/null 2>/dev/null + lvsmaster=$("${CTDB_HELPER_BINDIR}/ctdb_lvs" master) + if [ "$pnn" != "$lvsmaster" ] ; then + # This node is not the LVS master so change the IP address + # to have scope "host" so this node won't respond to ARPs + ip addr del "${CTDB_LVS_PUBLIC_IP}/32" dev lo >/dev/null 2>&1 + ip addr add "${CTDB_LVS_PUBLIC_IP}/32" dev lo scope host exit 0 - } + fi - # change the scope so we start responding to arps - ip addr del $CTDB_LVS_PUBLIC_IP/32 dev lo >/dev/null 2>/dev/null - ip addr add $CTDB_LVS_PUBLIC_IP/32 dev lo >/dev/null 2>/dev/null + # Change the scope so this node starts responding to ARPs + ip addr del "${CTDB_LVS_PUBLIC_IP}/32" dev lo >/dev/null 2>&1 + ip addr add "${CTDB_LVS_PUBLIC_IP}/32" dev lo >/dev/null 2>&1 - ipvsadm -A -t $CTDB_LVS_PUBLIC_IP:0 -p 1999999 -s lc - ipvsadm -A -u $CTDB_LVS_PUBLIC_IP:0 -p 1999999 -s lc + ipvsadm -A -t "$CTDB_LVS_PUBLIC_IP" -p 1999999 -s lc + ipvsadm -A -u "$CTDB_LVS_PUBLIC_IP" -p 1999999 -s lc - # add all nodes (except ourselves) to the lvs config - ctdb lvs | egrep -v "^${pnn}:" | sed -e "s/.*://" | while read IP; do - ipvsadm -a -t $CTDB_LVS_PUBLIC_IP:0 -r $IP -g - ipvsadm -a -u $CTDB_LVS_PUBLIC_IP:0 -r $IP -g + # Add all nodes (except this node) as LVS servers + "${CTDB_HELPER_BINDIR}/ctdb_lvs" list | + awk -v pnn="$pnn" '$1 != pnn { print $2 }' | + while read ip ; do + ipvsadm -a -t "$CTDB_LVS_PUBLIC_IP" -r $ip -g + ipvsadm -a -u "$CTDB_LVS_PUBLIC_IP" -r $ip -g done - # and add the localhost too - ipvsadm -a -t $CTDB_LVS_PUBLIC_IP:0 -r 127.0.0.1 - ipvsadm -a -u $CTDB_LVS_PUBLIC_IP:0 -r 127.0.0.1 - # send out a gratious arp so our peers will update their arp tables - ctdb gratiousarp $CTDB_LVS_PUBLIC_IP $CTDB_PUBLIC_INTERFACE >/dev/null 2>/dev/null + # Add localhost too... + ipvsadm -a -t "$CTDB_LVS_PUBLIC_IP" -r 127.0.0.1 + ipvsadm -a -u "$CTDB_LVS_PUBLIC_IP" -r 127.0.0.1 - # flush our route cache - echo 1 > /proc/sys/net/ipv4/route/flush + ctdb gratiousarp \ + "$CTDB_LVS_PUBLIC_IP" "$CTDB_LVS_PUBLIC_IFACE" >/dev/null 2>&1 + + flush_route_cache ;; - *) - ctdb_standard_event_handler "$@" +monitor) + lvs_check_config + + if [ -n "$CTDB_LVS_PUBLIC_IFACE" ] ; then + interface_monitor "$CTDB_LVS_PUBLIC_IFACE" || exit 1 + fi ;; esac diff --git a/ctdb/config/functions b/ctdb/config/functions index 7c28e4c..782978d 100755 --- a/ctdb/config/functions +++ b/ctdb/config/functions @@ -115,7 +115,7 @@ die () _msg="$1" _rc="${2:-1}" - echo "$_msg" + echo "$_msg" >&2 exit $_rc } @@ -247,6 +247,19 @@ ctdb_get_pnn () read pnn <"$_pnn_file" } +# Cached retrieval of private IP address from local node. This never +# changes. Sets $ip_address to avoid an unnecessary subprocess. +ctdb_get_ip_address () +{ + _ip_addr_file="${CTDB_SCRIPT_VARDIR}/my-ip-address" + if [ ! -f "$_ip_addr_file" ] ; then + ctdb -X nodestatus | + awk -F '|' 'NR == 2 { print $3 }' >"$_ip_addr_file" + fi + + read ip_address <"$_ip_addr_file" +} + ###################################################### # wrapper around /proc/ settings to allow them to be hooked # for testing diff --git a/ctdb/doc/ctdb.1.xml b/ctdb/doc/ctdb.1.xml index 7c5822c..7cab96f 100644 --- a/ctdb/doc/ctdb.1.xml +++ b/ctdb/doc/ctdb.1.xml @@ -723,45 +723,63 @@ MonitorInterval = 15 </refsect2> <refsect2> - <title>lvsmaster</title> + <title>lvs {master|list|status}</title> <para> - This command shows which node is currently the LVSMASTER. The - LVSMASTER is the node in the cluster which drives the LVS system and - which receives all incoming traffic from clients. - </para> - <para> - LVS is the mode where the entire CTDB/Samba cluster uses a single - ip address for the entire cluster. In this mode all clients connect to - one specific node which will then multiplex/loadbalance the clients - evenly onto the other nodes in the cluster. This is an alternative to using - public ip addresses. See the manpage for ctdbd for more information - about LVS. + This command shows different aspects of LVS status. For an + overview of CTDB's LVS functionality please see the + <citetitle>LVS</citetitle> section in + <citerefentry><refentrytitle>ctdb</refentrytitle> + <manvolnum>7</manvolnum></citerefentry>. </para> - </refsect2> - <refsect2> - <title>lvs</title> + <variablelist> + <varlistentry> + <term>master</term> + <listitem> + <para> + Shows the PNN of the current LVS master node. + </para> <para> - This command shows which nodes in the cluster are currently active in the - LVS configuration. I.e. which nodes we are currently loadbalancing - the single ip address across. + Example output: </para> - + <screen> +2 + </screen> + </listitem> + </varlistentry> + <varlistentry> + <term>list</term> + <listitem> + <para> + Lists the currently usable LVS nodes. + </para> <para> - LVS will by default only loadbalance across those nodes that are both - LVS capable and also HEALTHY. Except if all nodes are UNHEALTHY in which - case LVS will loadbalance across all UNHEALTHY nodes as well. - LVS will never use nodes that are DISCONNECTED, STOPPED, BANNED or - DISABLED. + Example output: </para> - + <screen> +2 10.0.0.13 +3 10.0.0.14 + </screen> + </listitem> + </varlistentry> + <varlistentry> + <term>status</term> + <listitem> + <para> + List the nodes in the current LVS group and their status. + </para> <para> Example output: </para> <screen> -2:10.0.0.13 -3:10.0.0.14 +pnn:0 10.0.0.11 UNHEALTHY (THIS NODE) +pnn:1 10.0.0.12 UNHEALTHY +pnn:2 10.0.0.13 OK +pnn:3 10.0.0.14 OK </screen> + </listitem> + </varlistentry> + </variablelist> </refsect2> @@ -782,7 +800,6 @@ MonitorInterval = 15 <screen> RECMASTER: YES LMASTER: YES -LVS: NO </screen> </refsect2> diff --git a/ctdb/doc/ctdb.7.xml b/ctdb/doc/ctdb.7.xml index 40d3387..51222ad 100644 --- a/ctdb/doc/ctdb.7.xml +++ b/ctdb/doc/ctdb.7.xml @@ -436,21 +436,6 @@ Node 3:/usr/local/etc/ctdb/public_addresses </listitem> </varlistentry> - <varlistentry> - <term>LVS</term> - <listitem> - <para> - Indicates that a node is configued in Linux Virtual Server - (LVS) mode. In this mode the entire CTDB cluster uses one - single public address for the entire cluster instead of - using multiple public addresses in failover mode. This is - an alternative to using a load-balancing layer-4 switch. - See the <citetitle>LVS</citetitle> section for more - details. - </para> - </listitem> - </varlistentry> - </variablelist> <para> @@ -477,25 +462,35 @@ Node 3:/usr/local/etc/ctdb/public_addresses </para> <para> - In this mode the cluster selects a set of nodes in the cluster - and loadbalance all client access to the LVS address across this - set of nodes. This set of nodes are all LVS capable nodes that - are HEALTHY, or if no HEALTHY nodes exists all LVS capable nodes - regardless of health status. LVS will however never loadbalance - traffic to nodes that are BANNED, STOPPED, DISABLED or - DISCONNECTED. The <command>ctdb lvs</command> command is used to - show which nodes are currently load-balanced across. + One extra LVS public address is assigned on the public network + to each LVS group. Each LVS group is a set of nodes in the + cluster that presents the same LVS address public address to the + outside world. Normally there would only be one LVS group + spanning an entire cluster, but in situations where one CTDB + cluster spans multiple physical sites it might be useful to have + one LVS group for each site. There can be multiple LVS groups + in a cluster but each node can only be member of one LVS group. + </para> + + <para> + Client access to the cluster is load-balanced across the HEALTHY + nodes in an LVS group. If no HEALTHY nodes exists then all + nodes in the group are used, regardless of health status. CTDB + will, however never load-balance LVS traffic to nodes that are + BANNED, STOPPED, DISABLED or DISCONNECTED. The <command>ctdb + lvs</command> command is used to show which nodes are currently + load-balanced across. </para> <para> - One of the these nodes are elected as the LVSMASTER. This node - receives all traffic from clients coming in to the LVS address - and multiplexes it across the internal network to one of the - nodes that LVS is using. When responding to the client, that - node will send the data back directly to the client, bypassing - the LVSMASTER node. The command <command>ctdb - lvsmaster</command> will show which node is the current - LVSMASTER. + In each LVS group, one of the nodes is selected by CTDB to be + the LVS master. This node receives all traffic from clients + coming in to the LVS public address and multiplexes it across + the internal network to one of the nodes that LVS is using. + When responding to the client, that node will send the data back + directly to the client, bypassing the LVS master node. The + command <command>ctdb lvsmaster</command> will show which node + is the current LVS master. </para> <para> @@ -525,7 +520,7 @@ Node 3:/usr/local/etc/ctdb/public_addresses </orderedlist> </para> - <para> + <para> This means that all incoming traffic to the cluster will pass through one physical node, which limits scalability. You can send more data to the LVS address that one physical node can @@ -565,20 +560,50 @@ Node 3:/usr/local/etc/ctdb/public_addresses <para> To activate LVS on a CTDB node you must specify the - <varname>CTDB_PUBLIC_INTERFACE</varname> and - <varname>CTDB_LVS_PUBLIC_IP</varname> configuration variables. - Setting the latter variable also enables the LVS capability on - the node at startup. + <varname>CTDB_LVS_PUBLIC_IFACE</varname>, + <varname>CTDB_LVS_PUBLIC_IP</varname> and + <varname>CTDB_LVS_NODES</varname> configuration variables. + <varname>CTDB_LVS_NODES</varname> specifies a file containing + the private address of all nodes in the current node's LVS + group. </para> - + <para> Example: <screen format="linespecific"> -CTDB_PUBLIC_INTERFACE=eth1 +CTDB_LVS_PUBLIC_IFACE=eth1 CTDB_LVS_PUBLIC_IP=10.1.1.237 +CTDB_LVS_NODES=/usr/local/etc/ctdb/lvs_nodes </screen> </para> + <para> + Example <filename>/usr/local/etc/ctdb/lvs_nodes</filename>: + </para> + <screen format="linespecific"> +192.168.1.2 +192.168.1.3 +192.168.1.4 + </screen> + + <para> + Normally any node in an LVS group can act as the LVS master. + Nodes that are highly loaded due to other demands maybe + flagged with the "slave-only" option in the + <varname>CTDB_LVS_NODES</varname> file to limit the LVS + functionality of those nodes. + </para> + + <para> + LVS nodes file that excludes 192.168.1.4 from being + the LVS master node: + </para> + <screen format="linespecific"> +192.168.1.2 +192.168.1.3 +192.168.1.4 slave-only + </screen> + </refsect2> </refsect1> diff --git a/ctdb/doc/ctdbd.1.xml b/ctdb/doc/ctdbd.1.xml index ae61792..0f75f77 100644 --- a/ctdb/doc/ctdbd.1.xml +++ b/ctdb/doc/ctdbd.1.xml @@ -229,19 +229,6 @@ </varlistentry> <varlistentry> - <term>--lvs</term> -- Samba Shared Repository