The branch, master has been updated via 57a32cebdd8 ctdb-recoverd: Pass SIGHUP to running helper via 8e949a60828 ctdb-recoverd: Record helper PID in recovery daemon context via 97a45f6f25f ctdb-recoverd: Add log reopening on SIGHUP to helpers via 51f0380e83c ctdb-daemon: Enable log reopening for event daemon via 4f14d7c0b9b ctdb-event: Reopen logs on SIGHUP via c554a325fe8 ctdb-daemon: Enable log reopening for recovery daemon via 4acfefed61f ctdb-recoverd: Add basic log reopening via 4ed37de82b1 ctdb-daemon: Add basic top-level log reopening via 72773853901 ctdb-common: Add support for reopening logs via d0a19778cdb ctdb-common: Separate sock_daemon's SIGHUP and SIGUSR1 handling via 10d15c9e5df ctdb-common: Use Samba's DEBUG_FILE logging via 666a0487074 ctdb-common: Switch initial debug type to DEBUG_DEFAULT_STDERR from 5f9dbf3decd s3: smbd: Add missing pop_sec_ctx() in error code path of close_directory()
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit 57a32cebdd834e24e69f524f8ffaa980472cde33 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Sep 30 21:16:44 2021 +1000 ctdb-recoverd: Pass SIGHUP to running helper The recovery and takeover helpers can run for a while and generate non-trivial logs, so have them reopen their logs to support log rotation. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> Autobuild-User(master): Amitay Isaacs <ami...@samba.org> Autobuild-Date(master): Mon Jan 17 04:36:30 UTC 2022 on sn-devel-184 commit 8e949a60828bae47a3636f051dcc86387b5fce23 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Sep 30 21:15:56 2021 +1000 ctdb-recoverd: Record helper PID in recovery daemon context Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 97a45f6f25f34fff97a9be7ba2b346a4e8b93218 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Sep 30 21:14:14 2021 +1000 ctdb-recoverd: Add log reopening on SIGHUP to helpers Recovery and takeover helpers can run for a while and generate non-trivial logs. They should support log reopening. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 51f0380e83c93702a08e29c230bb8d87d472b616 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Sep 30 21:10:33 2021 +1000 ctdb-daemon: Enable log reopening for event daemon Add and call hook to pass on SIGHUP to eventd. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 4f14d7c0b9bf1f122a1e9f92c1b8bdc57c4c9e68 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Sep 30 21:11:44 2021 +1000 ctdb-event: Reopen logs on SIGHUP Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit c554a325fe81aab90f4816600849f6c4f901b2f9 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Sep 30 21:08:25 2021 +1000 ctdb-daemon: Enable log reopening for recovery daemon Pass on a SIGHUP to the recovery daemon, which will then reopen its logs. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 4acfefed61f2c9e069963ac76b2001469a694461 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Sep 30 21:03:15 2021 +1000 ctdb-recoverd: Add basic log reopening Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 4ed37de82b1be4732f6e5936e149aae718855513 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Sep 30 21:06:16 2021 +1000 ctdb-daemon: Add basic top-level log reopening Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 72773853901832e4728e42588570d69c93976ce1 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Sep 30 20:55:27 2021 +1000 ctdb-common: Add support for reopening logs Now that CTDB uses Samba's file logging it is possible to reopen the logs, so that log rotation can be supported. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit d0a19778cdbd04f8b6fca43199d7b12e1d4933b7 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Nov 18 21:17:39 2021 +1100 ctdb-common: Separate sock_daemon's SIGHUP and SIGUSR1 handling SIGHUP is for reopening logs, SIGUSR1 is for reconfigure. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 10d15c9e5dfe4e8595d0b322c96f474fc7078f46 Author: Martin Schwenke <mar...@meltin.net> Date: Thu Sep 23 18:37:57 2021 +1000 ctdb-common: Use Samba's DEBUG_FILE logging This has support for log rotation (or re-opening). The log format is updated to use an RFC5424 timestamp and to include a hostname. The addition of the hostname allows trivial merging of log files from multiple cluster nodes. The hostname is faked from the CTDB_BASE environment variable during testing, as per the comment in the code. It is currently faked in a similar manner in local_daemons.sh when printing logs, so drop this. Unit tests need updating because stderr logging no longer produces a "PROGNAME[PID]: " header. Signed-off-by: Martin Schwenke <mar...@meltin.net> Reviewed-by: Amitay Isaacs <ami...@gmail.com> commit 666a048707436a8ba3b9a9cb297a4fd504fa1e54 Author: Martin Schwenke <mar...@meltin.net> Date: Fri Sep 24 22:17:53 2021 +1000 ctdb-common: Switch initial debug type to DEBUG_DEFAULT_STDERR This can be overridden by DEBUG_FILE, whereas DEBUG_STDERR can not. Signed-off-by: Martin Schwenke <mar...@meltin.net> Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Amitay Isaacs <ami...@gmail.com> ----------------------------------------------------------------------- Summary of changes: ctdb/common/logging.c | 190 +++++++++++++++----------- ctdb/common/logging.h | 7 + ctdb/common/sock_daemon.c | 61 ++++++++- ctdb/common/sock_daemon.h | 15 +- ctdb/event/event_daemon.c | 10 ++ ctdb/include/ctdb_private.h | 2 + ctdb/server/ctdb_daemon.c | 21 +++ ctdb/server/ctdb_recoverd.c | 25 ++++ ctdb/server/ctdb_recovery_helper.c | 7 + ctdb/server/ctdb_takeover_helper.c | 8 ++ ctdb/server/eventscript.c | 7 + ctdb/tests/UNIT/cunit/sock_daemon_test_001.sh | 141 ++++++++++--------- ctdb/tests/UNIT/eventd/eventd_051.sh | 2 +- ctdb/tests/UNIT/eventd/scripts/local.sh | 3 +- ctdb/tests/local_daemons.sh | 9 +- ctdb/tests/src/sock_daemon_test.c | 100 +++++++++++++- 16 files changed, 448 insertions(+), 160 deletions(-) Changeset truncated at 500 lines: diff --git a/ctdb/common/logging.c b/ctdb/common/logging.c index a4321b68ba9..1b91cdcc92b 100644 --- a/ctdb/common/logging.c +++ b/ctdb/common/logging.c @@ -111,50 +111,6 @@ int debug_level_from_string(const char *log_string) * file logging backend */ -struct file_log_state { - const char *app_name; - int fd; - char buffer[1024]; -}; - -static void file_log(void *private_data, int level, const char *msg) -{ - struct file_log_state *state = talloc_get_type_abort( - private_data, struct file_log_state); - struct timeval tv; - struct timeval_buf tvbuf; - int ret; - - if (state->fd == STDERR_FILENO) { - ret = snprintf(state->buffer, sizeof(state->buffer), - "%s[%u]: %s\n", - state->app_name, (unsigned)getpid(), msg); - } else { - GetTimeOfDay(&tv); - timeval_str_buf(&tv, false, true, &tvbuf); - - ret = snprintf(state->buffer, sizeof(state->buffer), - "%s %s[%u]: %s\n", tvbuf.buf, - state->app_name, (unsigned)getpid(), msg); - } - if (ret < 0) { - return; - } - - state->buffer[sizeof(state->buffer)-1] = '\0'; - - sys_write_v(state->fd, state->buffer, strlen(state->buffer)); -} - -static int file_log_state_destructor(struct file_log_state *state) -{ - if (state->fd != -1 && state->fd != STDERR_FILENO) { - close(state->fd); - state->fd = -1; - } - return 0; -} - static bool file_log_validate(const char *option) { char *t, *dir; @@ -185,46 +141,49 @@ static bool file_log_validate(const char *option) return true; } -static int file_log_setup(TALLOC_CTX *mem_ctx, const char *option, +static int file_log_setup(TALLOC_CTX *mem_ctx, + const char *option, const char *app_name) { - struct file_log_state *state; - - state = talloc_zero(mem_ctx, struct file_log_state); - if (state == NULL) { - return ENOMEM; - } - - state->app_name = app_name; + struct debug_settings settings = { + .debug_syslog_format = true, + .debug_hires_timestamp = true, + }; + const char *t = NULL; if (option == NULL || strcmp(option, "-") == 0) { - int ret; - - state->fd = STDERR_FILENO; - ret = dup2(STDERR_FILENO, STDOUT_FILENO); - if (ret == -1) { - int save_errno = errno; - talloc_free(state); - return save_errno; - } - - } else { - state->fd = open(option, O_WRONLY|O_APPEND|O_CREAT, 0644); - if (state->fd == -1) { - int save_errno = errno; - talloc_free(state); - return save_errno; - } - - if (! set_close_on_exec(state->fd)) { - int save_errno = errno; - talloc_free(state); - return save_errno; + /* + * Logging to stderr is the default and has already + * been done in logging init + */ + return 0; + } + + /* + * Support logging of fake hostname in local daemons. This + * hostname is basename(getenv(CTDB_BASE)). + */ + t = getenv("CTDB_TEST_MODE"); + if (t != NULL) { + t = getenv("CTDB_BASE"); + if (t != NULL) { + const char *p = strrchr(t, '/'); + if (p != NULL) { + p++; + if (p[0] == '\0') { + p = "unknown"; + } + } else { + p = t; + } + + debug_set_hostname(p); } } - talloc_set_destructor(state, file_log_state_destructor); - debug_set_callback(state, file_log); + debug_set_settings(&settings, "file", 0, false); + debug_set_logfile(option); + setup_logging(app_name, DEBUG_FILE); return 0; } @@ -677,7 +636,7 @@ int logging_init(TALLOC_CTX *mem_ctx, const char *logging, int level; int ret; - setup_logging(app_name, DEBUG_STDERR); + setup_logging(app_name, DEBUG_DEFAULT_STDERR); if (debug_level == NULL) { debug_level = getenv("CTDB_DEBUGLEVEL"); @@ -708,3 +667,78 @@ int logging_init(TALLOC_CTX *mem_ctx, const char *logging, talloc_free(option); return ret; } + +bool logging_reopen_logs(void) +{ + bool status; + + status = reopen_logs_internal(); + + return status; +} + +struct logging_reopen_logs_data { + void (*hook)(void *private_data); + void *private_data; +}; + +static void logging_sig_hup_handler(struct tevent_context *ev, + struct tevent_signal *se, + int signum, + int count, + void *dont_care, + void *private_data) +{ + bool status; + + if (private_data != NULL) { + struct logging_reopen_logs_data *data = talloc_get_type_abort( + private_data, struct logging_reopen_logs_data); + + if (data->hook != NULL) { + data->hook(data->private_data); + } + } + + status = logging_reopen_logs(); + if (!status) { + D_WARNING("Failed to reopen logs\n"); + return; + } + + D_NOTICE("Reopened logs\n"); + +} + +bool logging_setup_sighup_handler(struct tevent_context *ev, + TALLOC_CTX *talloc_ctx, + void (*hook)(void *private_data), + void *private_data) +{ + struct logging_reopen_logs_data *data = NULL; + struct tevent_signal *se; + + if (hook != NULL) { + data = talloc(talloc_ctx, struct logging_reopen_logs_data); + if (data == NULL) { + return false; + } + + data->hook = hook; + data->private_data = private_data; + } + + + se = tevent_add_signal(ev, + talloc_ctx, + SIGHUP, + 0, + logging_sig_hup_handler, + data); + if (se == NULL) { + talloc_free(data); + return false; + } + + return true; +} diff --git a/ctdb/common/logging.h b/ctdb/common/logging.h index 368171fe406..542b4a3723d 100644 --- a/ctdb/common/logging.h +++ b/ctdb/common/logging.h @@ -21,6 +21,7 @@ #define __CTDB_LOGGING_H__ #include <talloc.h> +#include <tevent.h> #include "lib/util/debug.h" #define DEBUG_ERR DBGLVL_ERR @@ -41,4 +42,10 @@ bool logging_validate(const char *logging); int logging_init(TALLOC_CTX *mem_ctx, const char *logging, const char *debuglevel, const char *app_name); +bool logging_reopen_logs(void); +bool logging_setup_sighup_handler(struct tevent_context *ev, + TALLOC_CTX *talloc_ctx, + void (*hook)(void *private_data), + void *private_data); + #endif /* __CTDB_LOGGING_H__ */ diff --git a/ctdb/common/sock_daemon.c b/ctdb/common/sock_daemon.c index 6298653f4ec..e31a36445b5 100644 --- a/ctdb/common/sock_daemon.c +++ b/ctdb/common/sock_daemon.c @@ -555,6 +555,8 @@ static void sock_daemon_run_signal_handler(struct tevent_context *ev, void *private_data); static void sock_daemon_run_reconfigure(struct tevent_req *req); static void sock_daemon_run_reconfigure_done(struct tevent_req *subreq); +static void sock_daemon_run_reopen_logs(struct tevent_req *req); +static void sock_daemon_run_reopen_logs_done(struct tevent_req *subreq); static void sock_daemon_run_shutdown(struct tevent_req *req); static void sock_daemon_run_shutdown_done(struct tevent_req *subreq); static void sock_daemon_run_exit(struct tevent_req *req); @@ -735,11 +737,16 @@ static void sock_daemon_run_signal_handler(struct tevent_context *ev, D_NOTICE("Received signal %d\n", signum); - if (signum == SIGHUP || signum == SIGUSR1) { + if (signum == SIGUSR1) { sock_daemon_run_reconfigure(req); return; } + if (signum == SIGHUP) { + sock_daemon_run_reopen_logs(req); + return; + } + if (signum == SIGINT || signum == SIGTERM) { state->exit_code = EINTR; sock_daemon_run_shutdown(req); @@ -798,6 +805,58 @@ static void sock_daemon_run_reconfigure_done(struct tevent_req *subreq) D_NOTICE("reconfigure completed successfully\n"); } +static void sock_daemon_run_reopen_logs(struct tevent_req *req) +{ + struct tevent_req *subreq; + struct sock_daemon_run_state *state = tevent_req_data( + req, struct sock_daemon_run_state); + struct sock_daemon_context *sockd = state->sockd; + + if (sockd->funcs != NULL && sockd->funcs->reopen_logs_send != NULL && + sockd->funcs->reopen_logs_recv != NULL) { + subreq = sockd->funcs->reopen_logs_send(state, state->ev, + sockd->private_data); + if (tevent_req_nomem(subreq, req)) { + return; + } + tevent_req_set_callback(subreq, + sock_daemon_run_reopen_logs_done, req); + return; + } + + if (sockd->funcs != NULL && sockd->funcs->reopen_logs != NULL) { + int ret; + + ret = sockd->funcs->reopen_logs(sockd->private_data); + if (ret != 0) { + D_ERR("reopen logs, ret=%d\n", ret); + return; + } + + D_NOTICE("reopen logs completed successfully\n"); + } +} + +static void sock_daemon_run_reopen_logs_done(struct tevent_req *subreq) +{ + struct tevent_req *req = tevent_req_callback_data( + subreq, struct tevent_req); + struct sock_daemon_run_state *state = tevent_req_data( + req, struct sock_daemon_run_state); + struct sock_daemon_context *sockd = state->sockd; + int ret; + bool status; + + status = sockd->funcs->reopen_logs_recv(subreq, &ret); + TALLOC_FREE(subreq); + if (! status) { + D_ERR("reopen logs failed, ret=%d\n", ret); + return; + } + + D_NOTICE("reopen logs completed successfully\n"); +} + static void sock_daemon_run_shutdown(struct tevent_req *req) { struct tevent_req *subreq; diff --git a/ctdb/common/sock_daemon.h b/ctdb/common/sock_daemon.h index 7aa9d5dfd34..7653d2c1023 100644 --- a/ctdb/common/sock_daemon.h +++ b/ctdb/common/sock_daemon.h @@ -54,12 +54,18 @@ struct sock_client_context; * * startup_send()/startup_recv() is the async version of startup() * - * reconfigure() is called when the daemon receives SIGUSR1 or SIGHUP + * reconfigure() is called when the daemon receives SIGUSR1 * reconfigure() should return 0 for success, non-zero value on failure * On failure, sock_daemon_run() will continue to run. * * reconfigure_send()/reconfigure_recv() is the async version of reconfigure() * + * reopen_logs() is called when the daemon receives SIGHUP + * reopen_logs() should return 0 for success, non-zero value on failure + * On failure, sock_daemon_run() will continue to run. + * + * reopen_logs_send()/reopen_logs_recv() is the async version of reopen_logs() + * * shutdown() is called when process receives SIGINT or SIGTERM or * when wait computation has finished * @@ -90,6 +96,13 @@ struct sock_daemon_funcs { void *private_data); bool (*reconfigure_recv)(struct tevent_req *req, int *perr); + int (*reopen_logs)(void *private_data); + + struct tevent_req * (*reopen_logs_send)(TALLOC_CTX *mem_ctx, + struct tevent_context *ev, + void *private_data); + bool (*reopen_logs_recv)(struct tevent_req *req, int *perr); + void (*shutdown)(void *private_data); struct tevent_req * (*shutdown_send)(TALLOC_CTX *mem_ctx, diff --git a/ctdb/event/event_daemon.c b/ctdb/event/event_daemon.c index b6c01761e91..d96ff6f172f 100644 --- a/ctdb/event/event_daemon.c +++ b/ctdb/event/event_daemon.c @@ -74,6 +74,15 @@ static int event_daemon_reconfigure(void *private_data) return 0; } +static int event_daemon_reopen_logs(void *private_data) +{ + bool status; + + status = logging_reopen_logs(); + + return status ? 0 : 1; +} + static void event_daemon_shutdown(void *private_data) { struct event_daemon_state *e_state = talloc_get_type_abort( @@ -307,6 +316,7 @@ int main(int argc, const char **argv) daemon_funcs = (struct sock_daemon_funcs) { .startup = event_daemon_startup, .reconfigure = event_daemon_reconfigure, + .reopen_logs = event_daemon_reopen_logs, .shutdown = event_daemon_shutdown, }; diff --git a/ctdb/include/ctdb_private.h b/ctdb/include/ctdb_private.h index f5e647f08a5..b0469f7f14a 100644 --- a/ctdb/include/ctdb_private.h +++ b/ctdb/include/ctdb_private.h @@ -1035,4 +1035,6 @@ int ctdb_event_script_args(struct ctdb_context *ctdb, int ctdb_event_script(struct ctdb_context *ctdb, enum ctdb_event call); +void ctdb_event_reopen_logs(struct ctdb_context *ctdb); + #endif diff --git a/ctdb/server/ctdb_daemon.c b/ctdb/server/ctdb_daemon.c index 0896ba08f90..be5a6922c79 100644 --- a/ctdb/server/ctdb_daemon.c +++ b/ctdb/server/ctdb_daemon.c @@ -1488,6 +1488,17 @@ static void fork_only(void) } } +static void sighup_hook(void *private_data) +{ + struct ctdb_context *ctdb = talloc_get_type_abort(private_data, + struct ctdb_context); + + if (ctdb->recoverd_pid > 0) { + kill(ctdb->recoverd_pid, SIGHUP); + } + ctdb_event_reopen_logs(ctdb); +} + /* start the protocol going as a daemon */ @@ -1495,6 +1506,7 @@ int ctdb_start_daemon(struct ctdb_context *ctdb, bool interactive, bool test_mode_enabled) { + bool status; int ret; struct tevent_fd *fde; @@ -1548,6 +1560,15 @@ int ctdb_start_daemon(struct ctdb_context *ctdb, ctdb_tevent_trace_init(); tevent_set_trace_callback(ctdb->ev, ctdb_tevent_trace, ctdb); + status = logging_setup_sighup_handler(ctdb->ev, + ctdb, + sighup_hook, + ctdb); + if (!status) { + D_ERR("Failed to set up signal handler for SIGHUP\n"); + exit(1); + } + /* set up a handler to pick up sigchld */ if (ctdb_init_sigchld(ctdb) == NULL) { DEBUG(DEBUG_CRIT,("Failed to set up signal handler for SIGCHLD\n")); diff --git a/ctdb/server/ctdb_recoverd.c b/ctdb/server/ctdb_recoverd.c index dfa6d0d089b..761f00177ca 100644 --- a/ctdb/server/ctdb_recoverd.c +++ b/ctdb/server/ctdb_recoverd.c @@ -263,6 +263,7 @@ struct ctdb_recoverd { struct ctdb_node_capabilities *caps; bool frozen_on_inactive; struct ctdb_recovery_lock_handle *recovery_lock_handle; + pid_t helper_pid; }; #define CONTROL_TIMEOUT() timeval_current_ofs(ctdb->tunable.recover_timeout, 0) @@ -899,6 +900,7 @@ static int helper_run(struct ctdb_recoverd *rec, TALLOC_CTX *mem_ctx, close(state->fd[1]); state->fd[1] = -1; + rec->helper_pid = state->pid; state->done = false; fde = tevent_add_fd(rec->ctdb->ev, state, state->fd[0], @@ -927,6 +929,7 @@ static int helper_run(struct ctdb_recoverd *rec, TALLOC_CTX *mem_ctx, goto fail; } + rec->helper_pid = -1; ctdb_kill(rec->ctdb, state->pid, SIGKILL); talloc_free(state); return 0; @@ -938,6 +941,7 @@ fail: if (state->fd[1] != -1) { close(state->fd[1]); } + rec->helper_pid = -1; if (state->pid != -1) { ctdb_kill(rec->ctdb, state->pid, SIGKILL); } @@ -2909,6 +2913,16 @@ done: } -- Samba Shared Repository