The branch, master has been updated
via 1e55591bc54 ctdb-tests: Add a new fetch ring test that also checks
hot keys
via fb38252677d ctdb-tests: Update fetch_ring to take database and key
on command line
via 53b73b9b0f9 ctdb-daemon: Fix sorting of hot keys
via 5c8dfbbf9be ctdb-daemon: Add extra logging of hot keys
via baf058dcf72 ctdb-daemon: Update hot key logging
via 1ab39b3270e ctdb-daemon: Fix bug in slot 0 comparison optimisation
via f9f60c2a604 ctdb-daemon: Switch some variables to unsigned
via 21b9844bcbb ctdb-daemon: Add separate hot keys array for database
statistics
via c28914bfa70 ctdb-build: Fix a typo
from c68133d41c3 smbd: prepare DH disconnect for AT-names
https://git.samba.org/?p=samba.git;a=shortlog;h=master
- Log -----------------------------------------------------------------
commit 1e55591bc54949848eeb01a8be4378a18bc9e6c3
Author: Martin Schwenke <[email protected]>
Date: Mon May 4 17:15:19 2020 +1000
ctdb-tests: Add a new fetch ring test that also checks hot keys
Signed-off-by: Martin Schwenke <[email protected]>
Reviewed-by: Amitay Isaacs <[email protected]>
Autobuild-User(master): Amitay Isaacs <[email protected]>
Autobuild-Date(master): Fri May 22 08:05:54 UTC 2020 on sn-devel-184
commit fb38252677d0dab299f3fab051eb031167b87242
Author: Martin Schwenke <[email protected]>
Date: Mon May 4 16:58:38 2020 +1000
ctdb-tests: Update fetch_ring to take database and key on command line
Signed-off-by: Martin Schwenke <[email protected]>
Reviewed-by: Amitay Isaacs <[email protected]>
commit 53b73b9b0f99c9683fca043f4fee74be6a488c66
Author: Martin Schwenke <[email protected]>
Date: Thu Apr 23 18:59:47 2020 +1000
ctdb-daemon: Fix sorting of hot keys
The current code only ever swaps with slot 0. This will only ever
happen with slots 0 and 1, so probably never sorts.
Replace with qsort().
Signed-off-by: Martin Schwenke <[email protected]>
Reviewed-by: Amitay Isaacs <[email protected]>
commit 5c8dfbbf9bea05c35fe1ce454ee47dbee5172722
Author: Martin Schwenke <[email protected]>
Date: Fri May 1 16:44:22 2020 +1000
ctdb-daemon: Add extra logging of hot keys
ctdbd currently only logs when a new hot key is added. If a key gets
hotter then nothing new is logged.
Log hot key updates when the number of migrations has doubled since
the last time that key was logged.
Signed-off-by: Martin Schwenke <[email protected]>
Reviewed-by: Amitay Isaacs <[email protected]>
commit baf058dcf72e4072ecc2f4bf775f04c271a6e10b
Author: Martin Schwenke <[email protected]>
Date: Fri May 1 16:24:27 2020 +1000
ctdb-daemon: Update hot key logging
This message indicates that a hot key was added, so say that. After
all the hot key slots have been filled the id will always be 0, so
don't bother logging it.
Signed-off-by: Martin Schwenke <[email protected]>
Reviewed-by: Amitay Isaacs <[email protected]>
commit 1ab39b3270ea34a1953e79d63f3f6d9e04983898
Author: Martin Schwenke <[email protected]>
Date: Thu May 14 20:25:22 2020 +1000
ctdb-daemon: Fix bug in slot 0 comparison optimisation
This is only valid if all slots are in use.
Signed-off-by: Martin Schwenke <[email protected]>
Reviewed-by: Amitay Isaacs <[email protected]>
commit f9f60c2a60464e8773663848572ec9ec08be0ee5
Author: Martin Schwenke <[email protected]>
Date: Thu Apr 23 18:59:24 2020 +1000
ctdb-daemon: Switch some variables to unsigned
These should be unsigned but luck is currently on our side.
Signed-off-by: Martin Schwenke <[email protected]>
Reviewed-by: Amitay Isaacs <[email protected]>
commit 21b9844bcbb4d8c18d82df063d6c143b435cdfb7
Author: Martin Schwenke <[email protected]>
Date: Thu Apr 23 18:51:40 2020 +1000
ctdb-daemon: Add separate hot keys array for database statistics
There are 2 reasons for this. Sorting of hot keys is broken and will
be changed to an implementation that needs a named (i.e. not
anonymous) structure. Also, at least one non-protocol field will be
added to facilitate more useful logging.
Signed-off-by: Martin Schwenke <[email protected]>
Reviewed-by: Amitay Isaacs <[email protected]>
commit c28914bfa7020227066387b637e0e90b8590510e
Author: Martin Schwenke <[email protected]>
Date: Thu May 21 20:19:17 2020 +1000
ctdb-build: Fix a typo
Signed-off-by: Martin Schwenke <[email protected]>
Reviewed-by: Amitay Isaacs <[email protected]>
-----------------------------------------------------------------------
Summary of changes:
ctdb/include/ctdb_private.h | 7 +
ctdb/server/ctdb_call.c | 83 +++++++----
ctdb/server/ctdb_ltdb_server.c | 22 ++-
ctdb/tests/INTEGRATION/database/fetch.001.ring.sh | 3 +-
.../INTEGRATION/database/fetch.002.ring-hotkeys.sh | 161 +++++++++++++++++++++
ctdb/tests/src/fetch_ring.c | 30 ++--
ctdb/wscript | 2 +-
7 files changed, 259 insertions(+), 49 deletions(-)
create mode 100755 ctdb/tests/INTEGRATION/database/fetch.002.ring-hotkeys.sh
Changeset truncated at 500 lines:
diff --git a/ctdb/include/ctdb_private.h b/ctdb/include/ctdb_private.h
index e532323bb34..9ca87332d61 100644
--- a/ctdb/include/ctdb_private.h
+++ b/ctdb/include/ctdb_private.h
@@ -342,6 +342,12 @@ struct ctdb_context {
struct lock_context *lock_pending;
};
+struct ctdb_db_hot_key {
+ uint32_t count;
+ TDB_DATA key;
+ uint32_t last_logged_count;
+};
+
struct ctdb_db_context {
struct ctdb_db_context *next, *prev;
struct ctdb_context *ctdb;
@@ -375,6 +381,7 @@ struct ctdb_db_context {
struct trbt_tree *defer_dmaster;
struct ctdb_db_statistics_old statistics;
+ struct ctdb_db_hot_key hot_keys[MAX_HOT_KEYS];
struct lock_context *lock_current;
struct lock_context *lock_pending;
diff --git a/ctdb/server/ctdb_call.c b/ctdb/server/ctdb_call.c
index f8bf2d8d41e..14baa797bd6 100644
--- a/ctdb/server/ctdb_call.c
+++ b/ctdb/server/ctdb_call.c
@@ -820,31 +820,62 @@ ctdb_defer_pinned_down_request(struct ctdb_context *ctdb,
struct ctdb_db_context
return 0;
}
+static int hot_key_cmp(const void *a, const void *b)
+{
+ const struct ctdb_db_hot_key *ka = (const struct ctdb_db_hot_key *)a;
+ const struct ctdb_db_hot_key *kb = (const struct ctdb_db_hot_key *)b;
+
+ if (ka->count < kb->count) {
+ return -1;
+ }
+ if (ka->count > kb->count) {
+ return 1;
+ }
+
+ return 0;
+}
+
static void
ctdb_update_db_stat_hot_keys(struct ctdb_db_context *ctdb_db, TDB_DATA key,
unsigned int count)
{
- int i, id;
+ unsigned int i, id;
char *keystr;
- /* smallest value is always at index 0 */
- if (count <= ctdb_db->statistics.hot_keys[0].count) {
+ /*
+ * If all slots are being used then only need to compare
+ * against the count in the 0th slot, since it contains the
+ * smallest count.
+ */
+ if (ctdb_db->statistics.num_hot_keys == MAX_HOT_KEYS &&
+ count <= ctdb_db->hot_keys[0].count) {
return;
}
/* see if we already know this key */
for (i = 0; i < MAX_HOT_KEYS; i++) {
- if (key.dsize != ctdb_db->statistics.hot_keys[i].key.dsize) {
+ if (key.dsize != ctdb_db->hot_keys[i].key.dsize) {
continue;
}
- if (memcmp(key.dptr, ctdb_db->statistics.hot_keys[i].key.dptr,
key.dsize)) {
+ if (memcmp(key.dptr, ctdb_db->hot_keys[i].key.dptr, key.dsize))
{
continue;
}
/* found an entry for this key */
- if (count <= ctdb_db->statistics.hot_keys[i].count) {
+ if (count <= ctdb_db->hot_keys[i].count) {
return;
}
- ctdb_db->statistics.hot_keys[i].count = count;
+ if (count >= (2 * ctdb_db->hot_keys[i].last_logged_count)) {
+ keystr = hex_encode_talloc(ctdb_db,
+ (unsigned char *)key.dptr,
+ key.dsize);
+ D_NOTICE("Updated hot key database=%s key=%s
count=%d\n",
+ ctdb_db->db_name,
+ keystr ? keystr : "" ,
+ count);
+ TALLOC_FREE(keystr);
+ ctdb_db->hot_keys[i].last_logged_count = count;
+ }
+ ctdb_db->hot_keys[i].count = count;
goto sort_keys;
}
@@ -855,35 +886,29 @@ ctdb_update_db_stat_hot_keys(struct ctdb_db_context
*ctdb_db, TDB_DATA key,
id = 0;
}
- if (ctdb_db->statistics.hot_keys[id].key.dptr != NULL) {
- talloc_free(ctdb_db->statistics.hot_keys[id].key.dptr);
+ if (ctdb_db->hot_keys[id].key.dptr != NULL) {
+ talloc_free(ctdb_db->hot_keys[id].key.dptr);
}
- ctdb_db->statistics.hot_keys[id].key.dsize = key.dsize;
- ctdb_db->statistics.hot_keys[id].key.dptr = talloc_memdup(ctdb_db,
key.dptr, key.dsize);
- ctdb_db->statistics.hot_keys[id].count = count;
+ ctdb_db->hot_keys[id].key.dsize = key.dsize;
+ ctdb_db->hot_keys[id].key.dptr = talloc_memdup(ctdb_db,
+ key.dptr,
+ key.dsize);
+ ctdb_db->hot_keys[id].count = count;
keystr = hex_encode_talloc(ctdb_db,
(unsigned char *)key.dptr, key.dsize);
- DEBUG(DEBUG_NOTICE,("Updated hot key database=%s key=%s id=%d "
- "count=%d\n", ctdb_db->db_name,
- keystr ? keystr : "" , id, count));
+ D_NOTICE("Added hot key database=%s key=%s count=%d\n",
+ ctdb_db->db_name,
+ keystr ? keystr : "" ,
+ count);
talloc_free(keystr);
+ ctdb_db->hot_keys[id].last_logged_count = count;
sort_keys:
- for (i = 1; i < MAX_HOT_KEYS; i++) {
- if (ctdb_db->statistics.hot_keys[i].count == 0) {
- continue;
- }
- if (ctdb_db->statistics.hot_keys[i].count <
ctdb_db->statistics.hot_keys[0].count) {
- count = ctdb_db->statistics.hot_keys[i].count;
- ctdb_db->statistics.hot_keys[i].count =
ctdb_db->statistics.hot_keys[0].count;
- ctdb_db->statistics.hot_keys[0].count = count;
-
- key = ctdb_db->statistics.hot_keys[i].key;
- ctdb_db->statistics.hot_keys[i].key =
ctdb_db->statistics.hot_keys[0].key;
- ctdb_db->statistics.hot_keys[0].key = key;
- }
- }
+ qsort(&ctdb_db->hot_keys[0],
+ ctdb_db->statistics.num_hot_keys,
+ sizeof(struct ctdb_db_hot_key),
+ hot_key_cmp);
}
/*
diff --git a/ctdb/server/ctdb_ltdb_server.c b/ctdb/server/ctdb_ltdb_server.c
index ce3569fe7b1..b6c7630bea3 100644
--- a/ctdb/server/ctdb_ltdb_server.c
+++ b/ctdb/server/ctdb_ltdb_server.c
@@ -1597,13 +1597,15 @@ int ctdb_set_db_sticky(struct ctdb_context *ctdb,
struct ctdb_db_context *ctdb_d
void ctdb_db_statistics_reset(struct ctdb_db_context *ctdb_db)
{
- struct ctdb_db_statistics_old *s = &ctdb_db->statistics;
- int i;
+ unsigned int i;
for (i=0; i<MAX_HOT_KEYS; i++) {
- if (s->hot_keys[i].key.dsize > 0) {
- talloc_free(s->hot_keys[i].key.dptr);
+ if (ctdb_db->hot_keys[i].key.dsize > 0) {
+ TALLOC_FREE(ctdb_db->hot_keys[i].key.dptr);
+ ctdb_db->hot_keys[i].key.dsize = 0;
}
+ ctdb_db->hot_keys[i].count = 0;
+ ctdb_db->hot_keys[i].last_logged_count = 0;
}
ZERO_STRUCT(ctdb_db->statistics);
@@ -1615,8 +1617,8 @@ int32_t ctdb_control_get_db_statistics(struct
ctdb_context *ctdb,
{
struct ctdb_db_context *ctdb_db;
struct ctdb_db_statistics_old *stats;
- int i;
- int len;
+ unsigned int i;
+ size_t len;
char *ptr;
ctdb_db = find_ctdb_db(ctdb, db_id);
@@ -1627,7 +1629,13 @@ int32_t ctdb_control_get_db_statistics(struct
ctdb_context *ctdb,
len = offsetof(struct ctdb_db_statistics_old, hot_keys_wire);
for (i = 0; i < MAX_HOT_KEYS; i++) {
- len += ctdb_db->statistics.hot_keys[i].key.dsize;
+ struct ctdb_db_statistics_old *s = &ctdb_db->statistics;
+
+ s->hot_keys[i].key.dsize = ctdb_db->hot_keys[i].key.dsize;
+ s->hot_keys[i].key.dptr = ctdb_db->hot_keys[i].key.dptr;
+ s->hot_keys[i].count = ctdb_db->hot_keys[i].count;
+
+ len += s->hot_keys[i].key.dsize;
}
stats = talloc_size(outdata, len);
diff --git a/ctdb/tests/INTEGRATION/database/fetch.001.ring.sh
b/ctdb/tests/INTEGRATION/database/fetch.001.ring.sh
index 05632024cb5..33df07b7d88 100755
--- a/ctdb/tests/INTEGRATION/database/fetch.001.ring.sh
+++ b/ctdb/tests/INTEGRATION/database/fetch.001.ring.sh
@@ -23,7 +23,8 @@ try_command_on_node 0 "$CTDB listnodes | wc -l"
num_nodes="$out"
echo "Running fetch_ring on all $num_nodes nodes."
-try_command_on_node -v -p all $CTDB_TEST_WRAPPER $VALGRIND fetch_ring -n
$num_nodes
+testprog_onnode -v -p all \
+ fetch_ring -n "$num_nodes" -D "fetch_ring.tdb" -k "testkey"
pat='^(Waiting for cluster|Fetch\[[[:digit:]]+\]:
[[:digit:]]+(\.[[:digit:]]+)? msgs/sec)$'
sanity_check_output 1 "$pat"
diff --git a/ctdb/tests/INTEGRATION/database/fetch.002.ring-hotkeys.sh
b/ctdb/tests/INTEGRATION/database/fetch.002.ring-hotkeys.sh
new file mode 100755
index 00000000000..c4e0023e11c
--- /dev/null
+++ b/ctdb/tests/INTEGRATION/database/fetch.002.ring-hotkeys.sh
@@ -0,0 +1,161 @@
+#!/bin/bash
+
+# Run the fetch_ring test, sanity check the output and check hot keys
+# statistics
+
+. "${TEST_SCRIPTS_DIR}/integration.bash"
+
+set -e
+
+ctdb_test_init
+
+testdb="fetch_ring.tdb"
+
+ctdb_get_all_pnns
+# $all_pnns is set above
+# shellcheck disable=SC2154
+num_nodes=$(echo "$all_pnns" | wc -w | tr -d '[:space:]')
+first=$(echo "$all_pnns" | sed -n -e '1p')
+
+get_key ()
+{
+ _n="$1"
+
+ echo "testkey${_n}"
+}
+
+run_fetch_ring ()
+{
+ _timelimit="$1"
+ _key_num="$2"
+
+ _key=$(get_key "$_key_num")
+ _base_cmd="fetch_ring -n ${num_nodes} -D ${testdb}"
+ _cmd="${_base_cmd} -t ${_timelimit} -k ${_key}"
+ echo "Running \"${_cmd}\" on all $num_nodes nodes."
+ testprog_onnode -v -p all "$_cmd"
+
+ _pat='^(Waiting for cluster|Fetch\[[[:digit:]]+\]:
[[:digit:]]+(\.[[:digit:]]+)? msgs/sec)$'
+ sanity_check_output 1 "$_pat"
+
+ # Get the last line of output.
+ # $outfile is set above by testprog_onnode()
+ # shellcheck disable=SC2154
+ _last=$(tail -n 1 "$outfile")
+
+ # $last should look like this:
+ # Fetch[1]: 10670.93 msgs/sec
+ _stuff="${_last##*Fetch\[*\]: }"
+ _mps="${_stuff% msgs/sec*}"
+
+ if [ "${_mps%.*}" -ge 10 ] ; then
+ echo "OK: ${_mps} msgs/sec >= 10 msgs/sec"
+ else
+ ctdb_test_fail "BAD: ${_mps} msgs/sec < 10 msgs/sec"
+ fi
+}
+
+check_hot_keys ()
+{
+ _pnn="$1"
+ _first_key="$2"
+ _num_keys="$3"
+
+ echo
+ echo "Checking hot keys on node ${_pnn}"
+
+ ctdb_onnode "$_pnn" dbstatistics "$testdb"
+
+ # Get hot keys with a non-empty key
+ _hotkeys=$(grep -Ex '[[:space:]]+Count:[[:digit:]]+ Key:[[:xdigit:]]+' \
+ "$outfile") || true
+
+ # Check that there are the right number of non-empty slots
+ if [ -z "$_hotkeys" ] ; then
+ _num=0
+ else
+ _num=$(echo "$_hotkeys" | wc -l | tr -d '[:space:]')
+ fi
+ _msg="hot key slots in use = ${_num}"
+ if [ "$_num_keys" -ne "$_num" ] ; then
+ echo
+ cat "$outfile"
+ ctdb_test_fail "BAD: ${_msg} (expected ${_num_keys})"
+ fi
+ echo "GOOD: ${_msg}"
+
+ # No hot keys? Done...
+ if [ "$_num" = 0 ] ; then
+ return
+ fi
+
+ # Check that hot key counts are correctly sorted
+ #
+ # Try to be as POSIX as possible
+ # shellcheck disable=SC2001
+ _counts=$(echo "$_hotkeys" | \
+ sed -e 's|.*Count:\([[:digit:]][[:digit:]]*\).*|\1|')
+ _counts_sorted=$(echo "$_counts" | sort -n)
+ if [ "$_counts" != "$_counts_sorted" ] ; then
+ echo
+ cat "$outfile"
+ ctdb_test_fail "BAD: hot keys not sorted"
+ fi
+ echo "GOOD: hot key counts are correctly sorted"
+
+ # Check that all keys are considered hot
+ for _j in $(seq "$_first_key" $((_first_key + _num_keys - 1))) ; do
+ _key=$(get_key "$_j")
+ _key_hex=$(printf '%s' "$_key" | \
+ od -A n -t x1 | \
+ tr -d '[:space:]')
+ if ! echo "$_hotkeys" | grep -q "Key:${_key_hex}\$" ; then
+ echo
+ cat "$outfile"
+ ctdb_test_fail "BAD: key \"${_key}\" is not a hot key"
+ fi
+ done
+ echo "GOOD: all keys are listed as hot keys"
+}
+
+# Run fetch_ring for each of 10 keys. After each run confirm that all
+# keys used so far are considered hot keys (and do other hot key
+# sanity checks) on all nodes.
+for i in $(seq 1 10) ; do
+ run_fetch_ring 5 "$i"
+
+ for pnn in $all_pnns ; do
+ check_hot_keys "$pnn" 1 "$i"
+ done
+
+ echo
+done
+
+echo
+echo "Resetting statistics on node ${first}"
+ctdb_onnode "$first" statisticsreset
+
+# Ensure that only node $first has had statistics reset
+for pnn in $all_pnns ; do
+ if [ "$pnn" = "$first" ] ; then
+ check_hot_keys "$pnn" 1 0
+ else
+ check_hot_keys "$pnn" 1 10
+ fi
+done
+
+echo
+
+# Run fetch_ring for each of 3 new keys. After each run confirm that
+# the new keys used so far are considered hot keys (and do other hot
+# key sanity checks) on node $first.
+#
+# Note that nothing can be said about hot keys on other nodes, since
+# they may be an arbitrary blend of old and new keys.
+for i in $(seq 1 3) ; do
+ run_fetch_ring 5 $((100 + i))
+
+ check_hot_keys 0 101 "$i"
+
+ echo
+done
diff --git a/ctdb/tests/src/fetch_ring.c b/ctdb/tests/src/fetch_ring.c
index a8d103ed3f2..f1786ef370b 100644
--- a/ctdb/tests/src/fetch_ring.c
+++ b/ctdb/tests/src/fetch_ring.c
@@ -28,9 +28,6 @@
#include "tests/src/test_options.h"
#include "tests/src/cluster_wait.h"
-#define TESTDB "fetch_ring.tdb"
-#define TESTKEY "testkey"
-
#define MSG_ID_FETCH 0
static uint32_t next_node(struct ctdb_client_context *client, uint32_t
num_nodes)
@@ -63,7 +60,9 @@ static struct tevent_req *fetch_ring_send(TALLOC_CTX *mem_ctx,
struct tevent_context *ev,
struct ctdb_client_context *client,
struct ctdb_db_context *ctdb_db,
- uint32_t num_nodes, int timelimit,
+ const char *keystr,
+ uint32_t num_nodes,
+ int timelimit,
int interactive)
{
struct tevent_req *req, *subreq;
@@ -80,8 +79,8 @@ static struct tevent_req *fetch_ring_send(TALLOC_CTX *mem_ctx,
state->num_nodes = num_nodes;
state->timelimit = timelimit;
state->interactive = interactive;
- state->key.dptr = discard_const(TESTKEY);
- state->key.dsize = strlen(TESTKEY);
+ state->key.dptr = discard_const(keystr);
+ state->key.dsize = strlen(keystr);
subreq = ctdb_client_set_message_handler_send(
state, ev, client, MSG_ID_FETCH,
@@ -334,7 +333,7 @@ int main(int argc, const char *argv[])
setup_logging("fetch_ring", DEBUG_STDERR);
- status = process_options_basic(argc, argv, &opts);
+ status = process_options_database(argc, argv, &opts);
if (! status) {
exit(1);
}
@@ -362,15 +361,24 @@ int main(int argc, const char *argv[])
exit(1);
}
- ret = ctdb_attach(ev, client, tevent_timeval_zero(), TESTDB, 0,
+ ret = ctdb_attach(ev,
+ client,
+ tevent_timeval_zero(),
+ opts->dbname,
+ 0,
&ctdb_db);
if (ret != 0) {
- fprintf(stderr, "Failed to attach to DB %s\n", TESTDB);
+ fprintf(stderr, "Failed to attach to DB %s\n", opts->dbname);
exit(1);
}
- req = fetch_ring_send(mem_ctx, ev, client, ctdb_db,
- opts->num_nodes, opts->timelimit,
+ req = fetch_ring_send(mem_ctx,
+ ev,
+ client,
+ ctdb_db,
+ opts->keystr,
+ opts->num_nodes,
+ opts->timelimit,
opts->interactive);
if (req == NULL) {
fprintf(stderr, "Memory allocation error\n");
diff --git a/ctdb/wscript b/ctdb/wscript
index 2273853c74f..5f93f3a0668 100644
--- a/ctdb/wscript
+++ b/ctdb/wscript
@@ -33,7 +33,7 @@ samba_dist.DIST_DIRS('''ctdb:. lib/replace:lib/replace
lib/talloc:lib/talloc
lib/util:lib/util lib/tdb_wrap:lib/tdb_wrap
lib/ccan:lib/ccan libcli/util:libcli/util
lib/async_req:lib/async_req
- lib/pthreadpool:lib/pthreadpoool
+ lib/pthreadpool:lib/pthreadpool
lib/messaging:lib/messaging
buildtools:buildtools
third_party/waf:third_party/waf''')
--
Samba Shared Repository
