The branch, v4-7-test has been updated via 2d1d00b s3: libsmb: Fix reversing of oldname/newname paths when creating a reparse point symlink on Windows from smbclient. via 3f1f2a8 s3: client: Rename <oldname> to <link_target> in cmd_symlink() and cli_posix_symlink(). via 61140f4 pthreadpool: Fix deadlock via 086b453 pthreadpool: Add test for pthread_create failure via a677717 wscript: Add check for --wrap linker flag via fcc8616 pthreadpool: Undo put_job when returning error via c43c888 pthreadpool: Move creating of thread to new function via 97a9e81 ctdb-daemon: Send STARTUP control after startup event via 6f7215f ctdb-takeover: Send tcp tickles immediately on STARTUP control via 0fdc82e ctdb-takeover: Refactor code to send tickle lists for all public IPs from 5bb2b9c vfs_zfsacl: fix compilation error
https://git.samba.org/?p=samba.git;a=shortlog;h=v4-7-test - Log ----------------------------------------------------------------- commit 2d1d00bdd0f1acf66c6700488f4a3e3970b722a1 Author: Jeremy Allison <j...@samba.org> Date: Wed Nov 29 13:16:43 2017 -0800 s3: libsmb: Fix reversing of oldname/newname paths when creating a reparse point symlink on Windows from smbclient. This happened as smbd doesn't support reparse points so we couldn't test. This was the reverse of the (tested) symlink parameters in the unix extensions symlink command. Rename parameters to link_target instead of oldname so this is clearer. BUG: https://bugzilla.samba.org/show_bug.cgi?id=13172 Signed-off-by: Jeremy Allison <j...@samba.org> Reviewed-by: Volker Lendecke <v...@samba.org> (cherry picked from commit abbc9b9ab793d22bca6a37828f4375ef38c56dd3) Autobuild-User(v4-7-test): Karolin Seeger <ksee...@samba.org> Autobuild-Date(v4-7-test): Wed Dec 13 14:19:59 CET 2017 on sn-devel-144 commit 3f1f2a82896fb7f506b4671a95613365e9250f46 Author: Jeremy Allison <j...@samba.org> Date: Wed Nov 29 13:10:25 2017 -0800 s3: client: Rename <oldname> to <link_target> in cmd_symlink() and cli_posix_symlink(). Stops us from mixing up the old and new names. Only behavior change is correcting the names printed in the error messages. BUG: https://bugzilla.samba.org/show_bug.cgi?id=13172 Signed-off-by: Jeremy Allison <j...@samba.org> Reviewed-by: Volker Lendecke <v...@samba.org> (cherry picked from commit 8448dcaa8da78bcb84fca6a000c75e256bce1e77) commit 61140f4b47488018ec3d505a390f55640dd724d8 Author: Volker Lendecke <v...@samba.org> Date: Tue Dec 12 23:07:39 2017 +0100 pthreadpool: Fix deadlock Christof's idea from https://lists.samba.org/archive/samba-technical/2017-December/124384.html was that the thread already exited. It could also be that the thread is not yet idle when the new pthreadpool_add_jobs comes around the corner. Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Christof Schmitt <c...@samba.org> Autobuild-User(master): Christof Schmitt <c...@samba.org> Autobuild-Date(master): Wed Dec 13 04:46:12 CET 2017 on sn-devel-144 (cherry picked from commit dfc4670640341761b346065922a62a3e755e9e58) BUG: https://bugzilla.samba.org/show_bug.cgi?id=13170 commit 086b45389fe825f461f04de00b90c98ff5f335b9 Author: Christof Schmitt <c...@samba.org> Date: Wed Dec 6 15:10:23 2017 -0700 pthreadpool: Add test for pthread_create failure This is implemented using cmocka and the __wrap override for pthread_create. BUG: https://bugzilla.samba.org/show_bug.cgi?id=13170 Signed-off-by: Christof Schmitt <c...@samba.org Reviewed-by: Andreas Schneider <a...@samba.org> Autobuild-User(master): Christof Schmitt <c...@samba.org> Autobuild-Date(master): Fri Dec 8 13:54:20 CET 2017 on sn-devel-144 (cherry picked from commit 8cdb3995caf7a21d0c27a56a0bf0c1efd5b491e4) commit a6777171c0a4e0a8bdda9bf160193773c6b93838 Author: Christof Schmitt <c...@samba.org> Date: Thu Dec 7 10:42:30 2017 -0700 wscript: Add check for --wrap linker flag BUG: https://bugzilla.samba.org/show_bug.cgi?id=13170 Signed-off-by: Christof Schmitt <c...@samba.org> Reviewed-by: Andreas Schneider <a...@samba.org> (cherry picked from commit 8e17be1c3df09c238560c8a7e62c17e9f9ff9bc7) commit fcc8616450f16850ba1d35fb3d7529c784b615f2 Author: Christof Schmitt <c...@samba.org> Date: Tue Nov 28 10:59:06 2017 -0700 pthreadpool: Undo put_job when returning error When an error is returned to the caller of pthreadpool_add_job, the job should not be kept in the internal job array. Otherwise the caller might free the data structure and a later worker thread would still reference it. When it is not possible to create a single worker thread, the system might be out of resources or hitting a configured limit. In this case fall back to calling the job function synchronously instead of raising the error to the caller and possibly back to the SMB client. BUG: https://bugzilla.samba.org/show_bug.cgi?id=13170 Signed-off-by: Christof Schmitt <c...@samba.org> Reviewed-by: Volker Lendecke <v...@samba.org> (cherry picked from commit 065fb5d94d25d19fc85832bb85aa9e379e8551cc) commit c43c8888ce263fd88ac91b042b3785d3a9864360 Author: Christof Schmitt <c...@samba.org> Date: Tue Nov 28 10:49:36 2017 -0700 pthreadpool: Move creating of thread to new function No functional change, but this simplifies error handling. BUG: https://bugzilla.samba.org/show_bug.cgi?id=13170 Signed-off-by: Christof Schmitt <c...@samba.org> Reviewed-by: Volker Lendecke <v...@samba.org> (cherry picked from commit 949ccc3ea9073a3d38bff28345f644d39177256f) commit 97a9e81db09a1e884c0f1ad22171851a9b437772 Author: Amitay Isaacs <ami...@gmail.com> Date: Mon Nov 20 15:27:52 2017 +1100 ctdb-daemon: Send STARTUP control after startup event BUG: https://bugzilla.samba.org/show_bug.cgi?id=13154 STARTUP control is primarily used to synchronise tcp tickles from running nodes to a node which has just started up. Earlier STARTUP control was sent (using BROADCAST_ALL) after setup event. Once the other nodes in the cluster connected to this node, the queued up messages would be sent and the tcp tickles would get synchronised. Recent fix to drop messages to disconnected or not-yet-connected nodes, the STARTUP control was never sent to the remote nodes and the tcp tickles did not get synchronised. To fix this problem send the STARTUP control (using BROADCAST_CONNECTED) after startup event. By this time all the running nodes in the cluster are connected. Signed-off-by: Amitay Isaacs <ami...@gmail.com> Reviewed-by: Martin Schwenke <mar...@meltin.net> Autobuild-User(master): Martin Schwenke <mart...@samba.org> Autobuild-Date(master): Thu Nov 30 15:29:48 CET 2017 on sn-devel-144 (cherry picked from commit d7a5cd589b7b16d625dbc64dac21a1384519e32b) commit 6f7215fdee31de1cdc253db3586ff234e27fecda Author: Amitay Isaacs <ami...@gmail.com> Date: Mon Nov 20 15:37:39 2017 +1100 ctdb-takeover: Send tcp tickles immediately on STARTUP control BUG: https://bugzilla.samba.org/show_bug.cgi?id=13154 Signed-off-by: Amitay Isaacs <ami...@gmail.com> Reviewed-by: Martin Schwenke <mar...@meltin.net> (cherry picked from commit 73e261b48c4abc91e00775ac7437752c9640e5bd) commit 0fdc82e1167c9a178131c4241d29d59f3c79b9bd Author: Amitay Isaacs <ami...@gmail.com> Date: Mon Nov 20 15:17:15 2017 +1100 ctdb-takeover: Refactor code to send tickle lists for all public IPs BUG: https://bugzilla.samba.org/show_bug.cgi?id=13154 Signed-off-by: Amitay Isaacs <ami...@gmail.com> Reviewed-by: Martin Schwenke <mar...@meltin.net> (cherry picked from commit 2b253f6b1bc4e765f3fcb614a3b67b14084a625d) ----------------------------------------------------------------------- Summary of changes: ctdb/server/ctdb_daemon.c | 6 -- ctdb/server/ctdb_monitor.c | 6 ++ ctdb/server/ctdb_takeover.c | 59 ++++++++------ lib/pthreadpool/pthreadpool.c | 103 ++++++++++++++++-------- lib/pthreadpool/tests_cmocka.c | 179 +++++++++++++++++++++++++++++++++++++++++ lib/pthreadpool/wscript_build | 7 ++ source3/client/client.c | 15 ++-- source3/libsmb/clifile.c | 15 ++-- source3/libsmb/clisymlink.c | 14 ++-- source3/selftest/tests.py | 7 ++ wscript | 4 + 11 files changed, 329 insertions(+), 86 deletions(-) create mode 100644 lib/pthreadpool/tests_cmocka.c Changeset truncated at 500 lines: diff --git a/ctdb/server/ctdb_daemon.c b/ctdb/server/ctdb_daemon.c index de38542..3abbee4 100644 --- a/ctdb/server/ctdb_daemon.c +++ b/ctdb/server/ctdb_daemon.c @@ -1081,12 +1081,6 @@ static void ctdb_setup_event_callback(struct ctdb_context *ctdb, int status, } ctdb_run_notification_script(ctdb, "setup"); - /* tell all other nodes we've just started up */ - ctdb_daemon_send_control(ctdb, CTDB_BROADCAST_ALL, - 0, CTDB_CONTROL_STARTUP, 0, - CTDB_CTRL_FLAG_NOREPLY, - tdb_null, NULL, NULL); - /* Start the recovery daemon */ if (ctdb_start_recoverd(ctdb) != 0) { DEBUG(DEBUG_ALERT,("Failed to start recovery daemon\n")); diff --git a/ctdb/server/ctdb_monitor.c b/ctdb/server/ctdb_monitor.c index 738acb1..1864887 100644 --- a/ctdb/server/ctdb_monitor.c +++ b/ctdb/server/ctdb_monitor.c @@ -243,6 +243,12 @@ static void ctdb_startup_callback(struct ctdb_context *ctdb, int status, void *p ctdb->monitor->monitoring_mode = CTDB_MONITORING_ENABLED; + /* tell all other nodes we've just started up */ + ctdb_daemon_send_control(ctdb, CTDB_BROADCAST_CONNECTED, + 0, CTDB_CONTROL_STARTUP, 0, + CTDB_CTRL_FLAG_NOREPLY, + tdb_null, NULL, NULL); + tevent_add_timer(ctdb->ev, ctdb->monitor->monitor_context, timeval_current_ofs(ctdb->monitor->next_interval, 0), ctdb_check_health, ctdb); diff --git a/ctdb/server/ctdb_takeover.c b/ctdb/server/ctdb_takeover.c index c359d66..a7aa8db 100644 --- a/ctdb/server/ctdb_takeover.c +++ b/ctdb/server/ctdb_takeover.c @@ -1495,24 +1495,23 @@ int32_t ctdb_control_tcp_remove(struct ctdb_context *ctdb, TDB_DATA indata) } +static void ctdb_send_set_tcp_tickles_for_all(struct ctdb_context *ctdb, + bool force); + /* Called when another daemon starts - causes all tickles for all public addresses we are serving to be sent to the new node on the - next check. This actually causes the next scheduled call to - tdb_update_tcp_tickles() to update all nodes. This is simple and + next check. This actually causes the tickles to be sent to the + other node immediately. In case there is an error, the periodic + timer will send the updates on timer event. This is simple and doesn't require careful error handling. */ int32_t ctdb_control_startup(struct ctdb_context *ctdb, uint32_t pnn) { - struct ctdb_vnn *vnn; - DEBUG(DEBUG_INFO, ("Received startup control from node %lu\n", (unsigned long) pnn)); - for (vnn = ctdb->vnn; vnn != NULL; vnn = vnn->next) { - vnn->tcp_update_needed = true; - } - + ctdb_send_set_tcp_tickles_for_all(ctdb, true); return 0; } @@ -1995,43 +1994,53 @@ static int ctdb_send_set_tcp_tickles_for_ip(struct ctdb_context *ctdb, return ret; } - -/* - perform tickle updates if required - */ -static void ctdb_update_tcp_tickles(struct tevent_context *ev, - struct tevent_timer *te, - struct timeval t, void *private_data) +static void ctdb_send_set_tcp_tickles_for_all(struct ctdb_context *ctdb, + bool force) { - struct ctdb_context *ctdb = talloc_get_type(private_data, struct ctdb_context); - int ret; struct ctdb_vnn *vnn; + int ret; - for (vnn=ctdb->vnn;vnn;vnn=vnn->next) { - /* we only send out updates for public addresses that + for (vnn = ctdb->vnn; vnn != NULL; vnn = vnn->next) { + /* we only send out updates for public addresses that we have taken over */ if (ctdb->pnn != vnn->pnn) { continue; } + /* We only send out the updates if we need to */ - if (!vnn->tcp_update_needed) { + if (!force && !vnn->tcp_update_needed) { continue; } + ret = ctdb_send_set_tcp_tickles_for_ip(ctdb, &vnn->public_address, vnn->tcp_array); if (ret != 0) { - DEBUG(DEBUG_ERR,("Failed to send the tickle update for public address %s\n", - ctdb_addr_to_str(&vnn->public_address))); + D_ERR("Failed to send the tickle update for ip %s\n", + ctdb_addr_to_str(&vnn->public_address)); + vnn->tcp_update_needed = true; } else { - DEBUG(DEBUG_INFO, - ("Sent tickle update for public address %s\n", - ctdb_addr_to_str(&vnn->public_address))); + D_INFO("Sent tickle update for ip %s\n", + ctdb_addr_to_str(&vnn->public_address)); vnn->tcp_update_needed = false; } } +} + +/* + perform tickle updates if required + */ +static void ctdb_update_tcp_tickles(struct tevent_context *ev, + struct tevent_timer *te, + struct timeval t, void *private_data) +{ + struct ctdb_context *ctdb = talloc_get_type( + private_data, struct ctdb_context); + + ctdb_send_set_tcp_tickles_for_all(ctdb, false); + tevent_add_timer(ctdb->ev, ctdb->tickle_update_context, timeval_current_ofs(ctdb->tunable.tickle_update_interval, 0), ctdb_update_tcp_tickles, ctdb); diff --git a/lib/pthreadpool/pthreadpool.c b/lib/pthreadpool/pthreadpool.c index 23885aa..309aba9 100644 --- a/lib/pthreadpool/pthreadpool.c +++ b/lib/pthreadpool/pthreadpool.c @@ -429,6 +429,11 @@ static bool pthreadpool_put_job(struct pthreadpool *p, return true; } +static void pthreadpool_undo_put_job(struct pthreadpool *p) +{ + p->num_jobs -= 1; +} + static void *pthreadpool_server(void *arg) { struct pthreadpool *pool = (struct pthreadpool *)arg; @@ -521,14 +526,56 @@ static void *pthreadpool_server(void *arg) } } -int pthreadpool_add_job(struct pthreadpool *pool, int job_id, - void (*fn)(void *private_data), void *private_data) +static int pthreadpool_create_thread(struct pthreadpool *pool) { pthread_attr_t thread_attr; pthread_t thread_id; int res; sigset_t mask, omask; + /* + * Create a new worker thread. It should not receive any signals. + */ + + sigfillset(&mask); + + res = pthread_attr_init(&thread_attr); + if (res != 0) { + return res; + } + + res = pthread_attr_setdetachstate( + &thread_attr, PTHREAD_CREATE_DETACHED); + if (res != 0) { + pthread_attr_destroy(&thread_attr); + return res; + } + + res = pthread_sigmask(SIG_BLOCK, &mask, &omask); + if (res != 0) { + pthread_attr_destroy(&thread_attr); + return res; + } + + res = pthread_create(&thread_id, &thread_attr, pthreadpool_server, + (void *)pool); + + assert(pthread_sigmask(SIG_SETMASK, &omask, NULL) == 0); + + pthread_attr_destroy(&thread_attr); + + if (res == 0) { + pool->num_threads += 1; + } + + return res; +} + +int pthreadpool_add_job(struct pthreadpool *pool, int job_id, + void (*fn)(void *private_data), void *private_data) +{ + int res; + res = pthread_mutex_lock(&pool->mutex); if (res != 0) { return res; @@ -557,6 +604,9 @@ int pthreadpool_add_job(struct pthreadpool *pool, int job_id, * We have idle threads, wake one. */ res = pthread_cond_signal(&pool->condvar); + if (res != 0) { + pthreadpool_undo_put_job(pool); + } pthread_mutex_unlock(&pool->mutex); return res; } @@ -570,43 +620,28 @@ int pthreadpool_add_job(struct pthreadpool *pool, int job_id, return 0; } - /* - * Create a new worker thread. It should not receive any signals. - */ - - sigfillset(&mask); - - res = pthread_attr_init(&thread_attr); - if (res != 0) { - pthread_mutex_unlock(&pool->mutex); - return res; - } - - res = pthread_attr_setdetachstate( - &thread_attr, PTHREAD_CREATE_DETACHED); + res = pthreadpool_create_thread(pool); if (res != 0) { - pthread_attr_destroy(&thread_attr); - pthread_mutex_unlock(&pool->mutex); - return res; - } + if (pool->num_threads == 0) { + /* + * No thread could be created to run job, + * fallback to sync call. + */ + pthreadpool_undo_put_job(pool); + pthread_mutex_unlock(&pool->mutex); - res = pthread_sigmask(SIG_BLOCK, &mask, &omask); - if (res != 0) { - pthread_attr_destroy(&thread_attr); - pthread_mutex_unlock(&pool->mutex); - return res; - } + fn(private_data); + return pool->signal_fn(job_id, fn, private_data, + pool->signal_fn_private_data); + } - res = pthread_create(&thread_id, &thread_attr, pthreadpool_server, - (void *)pool); - if (res == 0) { - pool->num_threads += 1; + /* + * At least one thread is still available, let + * that one run the queued job. + */ + res = 0; } - assert(pthread_sigmask(SIG_SETMASK, &omask, NULL) == 0); - - pthread_attr_destroy(&thread_attr); - pthread_mutex_unlock(&pool->mutex); return res; } diff --git a/lib/pthreadpool/tests_cmocka.c b/lib/pthreadpool/tests_cmocka.c new file mode 100644 index 0000000..9753d21 --- /dev/null +++ b/lib/pthreadpool/tests_cmocka.c @@ -0,0 +1,179 @@ +/* + * Unix SMB/CIFS implementation. + * cmocka tests for thread pool implementation + * Copyright (C) Christof Schmitt 2017 + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 3 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see <http://www.gnu.org/licenses/>. + */ + +#include <errno.h> +#include <pthread.h> +#include <setjmp.h> +#include <stdlib.h> +#include <string.h> + +#include <talloc.h> +#include <tevent.h> +#include <pthreadpool_tevent.h> + +#include <cmocka.h> +#include <poll.h> + +struct pthreadpool_tevent_test { + struct tevent_context *ev; + struct pthreadpool_tevent *pool; +}; + +static int setup_pthreadpool_tevent(void **state) +{ + struct pthreadpool_tevent_test *t; + int ret; + + t = talloc(NULL, struct pthreadpool_tevent_test); + assert_non_null(t); + + t->ev = tevent_context_init(t); + assert_non_null(t->ev); + + ret = pthreadpool_tevent_init(t->ev, 0, &t->pool); + assert_return_code(ret, 0); + + *state = t; + + return 0; +} + +static int teardown_pthreadpool_tevent(void **state) +{ + struct pthreadpool_tevent_test *t = *state; + + TALLOC_FREE(t); + + return 0; +} + +int __wrap_pthread_create(pthread_t *thread, const pthread_attr_t *attr, + void *(*start_routine) (void *), void *arg); +int __real_pthread_create(pthread_t *thread, const pthread_attr_t *attr, + void *(*start_routine) (void *), void *arg); + +int __wrap_pthread_create(pthread_t *thread, const pthread_attr_t *attr, + void *(*start_routine) (void *), void *arg) +{ + int error; + + error = mock_type(int); + if (error != 0) { + return error; + } + + return __real_pthread_create(thread, attr, start_routine, arg); +} + +static void test_job_threadid(void *ptr) +{ + pthread_t *threadid = ptr; + + *threadid = pthread_self(); +} + +static int test_create_do(struct tevent_context *ev, + struct pthreadpool_tevent *pool, + bool *in_main_thread) +{ + struct tevent_req *req; + pthread_t main_thread, worker_thread; + bool ok; + int ret; + + main_thread = pthread_self(); + + req = pthreadpool_tevent_job_send( + ev, ev, pool, test_job_threadid, &worker_thread); + if (req == NULL) { + fprintf(stderr, "pthreadpool_tevent_job_send failed\n"); + TALLOC_FREE(ev); + return ENOMEM; + } + + ok = tevent_req_poll(req, ev); + if (!ok) { + ret = errno; + fprintf(stderr, "tevent_req_poll failed: %s\n", + strerror(ret)); + TALLOC_FREE(ev); + return ret; + } + + + ret = pthreadpool_tevent_job_recv(req); + TALLOC_FREE(req); + if (ret != 0) { + fprintf(stderr, "tevent_req_recv failed: %s\n", + strerror(ret)); + TALLOC_FREE(ev); + return ret; + } + *in_main_thread = pthread_equal(worker_thread, main_thread); + + return 0; +} + +static void test_create(void **state) +{ + struct pthreadpool_tevent_test *t = *state; + bool in_main_thread; + int ret; + + /* + * When pthreadpool cannot create the first worker thread, + * this job will run in the sync fallback in the main thread. + */ + will_return(__wrap_pthread_create, EAGAIN); + ret = test_create_do(t->ev, t->pool, &in_main_thread); + assert_return_code(ret, 0); + assert_true(in_main_thread); + + /* + * When a thread can be created, the job will run in the worker thread. + */ + will_return(__wrap_pthread_create, 0); + ret = test_create_do(t->ev, t->pool, &in_main_thread); + assert_return_code(ret, 0); + assert_false(in_main_thread); + + poll(NULL, 0, 10); + + /* + * Workerthread will still be active for a second; immediately + * running another job will also use the worker thread, even + * if a new thread cannot be created. + */ + ret = test_create_do(t->ev, t->pool, &in_main_thread); + assert_return_code(ret, 0); + assert_false(in_main_thread); +} + +int main(int argc, char **argv) +{ + const struct CMUnitTest tests[] = { + cmocka_unit_test_setup_teardown(test_create, + setup_pthreadpool_tevent, + teardown_pthreadpool_tevent), + }; + + cmocka_set_message_output(CM_OUTPUT_SUBUNIT); + + return cmocka_run_group_tests(tests, NULL, NULL); +} diff --git a/lib/pthreadpool/wscript_build b/lib/pthreadpool/wscript_build index d530463..57df255 100644 --- a/lib/pthreadpool/wscript_build +++ b/lib/pthreadpool/wscript_build @@ -21,3 +21,10 @@ bld.SAMBA_BINARY('pthreadpooltest', deps='PTHREADPOOL', enabled=bld.env.WITH_PTHREADPOOL, install=False) + +bld.SAMBA_BINARY('pthreadpooltest_cmocka', + source='tests_cmocka.c', + deps='PTHREADPOOL cmocka', + ldflags='-Wl,--wrap=pthread_create', + enabled=bld.env.WITH_PTHREADPOOL and bld.env['HAVE_LDWRAP'], + install=False) diff --git a/source3/client/client.c b/source3/client/client.c index 9c57375..0ee084a 100644 --- a/source3/client/client.c +++ b/source3/client/client.c @@ -3523,7 +3523,7 @@ static int cmd_readlink(void) static int cmd_symlink(void) { TALLOC_CTX *ctx = talloc_tos(); - char *oldname = NULL; + char *link_target = NULL; char *newname = NULL; char *buf = NULL; char *buf2 = NULL; -- Samba Shared Repository