The branch, master has been updated via debf8ba vfs_fileid: add fileid:algorithm = fsname_norootdir via 1468dd2 vfs_fileid: add fileid:nolockinode parameter via b599cb2 vfs_fileid: add fileid:algorithm = fsname_nodirs via 495c646 vfs_fileid: add fileid:algorithm = hostname via 6a8764e vfs_fileid: convert dev argument of the device_mapping_fn to SMB_STRUCT_STAT via 9962495 vfs_fileid: add "fstype/mntdir deny/allow list" option via 326df16 vfs_fileid: preserve errno in an error code path via 5cce620 vfs_fileid: add a DEBUG message to log dev and inode via d0def3b tests: The pthreadpooltests do not need a full environment via 74dbeba dnscli: Make a few functions static via 361ea74 samba: Only use async signal-safe functions in signal handler via e1fb902 s4/torture: test vfs_fruit "fruit:time machine max size" option via 74eebac vfs_fruit: add "time machine max size" option from 3cbeaf4 docs-xml: add basic Makefile dependencies for targets that use xsltproc
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit debf8ba799ea7a4535e29c20a5f9377932c81938 Author: Ralph Boehme <s...@samba.org> Date: Thu Jan 4 17:22:16 2018 +0100 vfs_fileid: add fileid:algorithm = fsname_norootdir Based-on-a-patch-by: Ralph Wuerthner <ralph.wuerth...@de.ibm.com> Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> Autobuild-User(master): Jeremy Allison <j...@samba.org> Autobuild-Date(master): Sat Jan 6 04:41:24 CET 2018 on sn-devel-144 commit 1468dd21c37445cff044583229b107254d53b2b9 Author: Ralph Boehme <s...@samba.org> Date: Thu Jan 4 17:09:21 2018 +0100 vfs_fileid: add fileid:nolockinode parameter Based-on-a-patch-by: Ralph Wuerthner <ralph.wuerth...@de.ibm.com> Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit b599cb216815415a63504ec69be3f70f08ea58d5 Author: Ralph Boehme <s...@samba.org> Date: Thu Jan 4 17:02:53 2018 +0100 vfs_fileid: add fileid:algorithm = fsname_nodirs Enabling fileid:algorithm = fsname_nodirs uses the hostname algorithm for directories and thus breaks cluster lock coherence for directories. Based-on-a-patch-by: Christian Ambach <a...@samba.org> Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 495c646ec5cbd61539f6c547cd6048d7ff167d30 Author: Ralph Boehme <s...@samba.org> Date: Thu Jan 4 16:59:54 2018 +0100 vfs_fileid: add fileid:algorithm = hostname Using fileid:algorithm = hostname makes fileid generate fileids based on the hostname. This breaks cluster lock coherence. Based-on-a-patch-by: Christian Ambach <a...@samba.org> Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 6a8764ebcc0de57e7dd0dc22eaf4a9d201c0dca9 Author: Ralph Boehme <s...@samba.org> Date: Thu Jan 4 16:35:38 2018 +0100 vfs_fileid: convert dev argument of the device_mapping_fn to SMB_STRUCT_STAT This is in preperation of adding an additional mapping function that acts differently depending of the file type. No change in behaviour. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 996249571d0e8f9285d1b714b1c36d66e7649271 Author: Ralph Wuerthner <ralph.wuerth...@de.ibm.com> Date: Tue Jan 12 16:00:24 2016 +0100 vfs_fileid: add "fstype/mntdir deny/allow list" option When using the fsname or fsid algorithm a stat() and statfs() call is required for all mounted file systems to generate the file_id. If e.g. an NFS file system is unresponsive such a call might block and the smbd process will become unresponsive. Add "fileid:fstype deny", "fileid:fstype allow", "fileid:mntdir deny", and "fileid:mntdir allow" options to ignore potentially unresponsive file systems. See also https://lists.samba.org/archive/samba-technical/2016-January/111553.html for a discussion about why this is useful. Signed-off-by: Ralph Wuerthner <ralph.wuerth...@de.ibm.com> Reviewed-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 326df161736abc16fb3bd35a18a3e55a44fb3c5d Author: Ralph Boehme <s...@samba.org> Date: Fri Jan 5 10:23:30 2018 +0100 vfs_fileid: preserve errno in an error code path Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 5cce620bc1eac9a3f0bbf58084b7df2acde6bb15 Author: Ralph Boehme <s...@samba.org> Date: Thu Jan 4 17:25:07 2018 +0100 vfs_fileid: add a DEBUG message to log dev and inode Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit d0def3b2179cfbfca6f35237f2bb23d226a72814 Author: Volker Lendecke <v...@samba.org> Date: Fri Jan 5 10:45:41 2018 +0100 tests: The pthreadpooltests do not need a full environment Makes "make test TESTS=pthreadpool" faster Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Andreas Schneider <a...@samba.org> commit 74dbeba723c642bd207dad9d6881e8aa803b7509 Author: Volker Lendecke <v...@samba.org> Date: Thu Jan 4 21:26:58 2018 +0100 dnscli: Make a few functions static We might want to use the tcp flavor in the future in the forwarder for a single, persistent TCP connection. Then we can easily re-publish it. Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Andreas Schneider <a...@samba.org> commit 361ea743576cf125d7957a97ed78a0446dab1a19 Author: Volker Lendecke <v...@samba.org> Date: Thu Jan 4 21:06:02 2018 +0100 samba: Only use async signal-safe functions in signal handler Otherwise shutdown can hang Signed-off-by: Volker Lendecke <v...@samba.org> Reviewed-by: Andreas Schneider <a...@samba.org> commit e1fb902ca408aeecf26ecfd1926ca7824b25a3e7 Author: Ralph Boehme <s...@samba.org> Date: Tue Jan 2 19:09:04 2018 +0100 s4/torture: test vfs_fruit "fruit:time machine max size" option Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> commit 74eebac975ee913e49c95b29c13a329edb0744e5 Author: Ralph Boehme <s...@samba.org> Date: Fri Nov 3 10:56:29 2017 +0100 vfs_fruit: add "time machine max size" option This can be used to configure a per client filesystem size limit on TimeMachine shares. It's a nasty hack but it was reportedly working well in Netatalk where it's taken from. Signed-off-by: Ralph Boehme <s...@samba.org> Reviewed-by: Jeremy Allison <j...@samba.org> ----------------------------------------------------------------------- Summary of changes: docs-xml/manpages/vfs_fileid.8.xml | 77 ++++++- docs-xml/manpages/vfs_fruit.8.xml | 18 ++ libcli/dns/dns.c | 36 ++-- libcli/dns/libdns.h | 38 ---- selftest/target/Samba3.pm | 8 + source3/modules/vfs_fileid.c | 235 +++++++++++++++++++- source3/modules/vfs_fruit.c | 428 +++++++++++++++++++++++++++++++++++++ source3/selftest/tests.py | 8 +- source4/smbd/server.c | 4 +- source4/torture/vfs/fruit.c | 102 +++++++++ source4/torture/vfs/vfs.c | 1 + 11 files changed, 881 insertions(+), 74 deletions(-) Changeset truncated at 500 lines: diff --git a/docs-xml/manpages/vfs_fileid.8.xml b/docs-xml/manpages/vfs_fileid.8.xml index 5a3a70e..edfdef2 100644 --- a/docs-xml/manpages/vfs_fileid.8.xml +++ b/docs-xml/manpages/vfs_fileid.8.xml @@ -40,6 +40,15 @@ generates the device number based on the configured algorithm (see the "fileid:algorithm" option). </para> + + <para>When using the fsname or fsid algorithm a + <command>stat()</command> and <command>statfs()</command> call is + required for all mounted file systems to generate the file_id. If e.g. + an NFS file system is unresponsive such a call might block and the smbd + process will become unresponsive. Use the "fileid:fstype deny", + "fileid:fstype allow", "fileid:mntdir deny", or "fileid:mntdir allow" + options to ignore potentially unresponsive file systems. + </para> </refsect1> @@ -51,17 +60,33 @@ <varlistentry> <term>fileid:algorithm = ALGORITHM</term> <listitem> - <para>Available algorithms are <command>fsname</command> - and <command>fsid</command>. The default value is + <para>Available algorithms are <command>fsname</command>, + <command>fsname_nodirs</command>, <command>fsid</command> and + <command>hostname</command>. The default value is <command>fsname</command>. </para> <para>The <command>fsname</command> algorithm generates device id by hashing the kernel device name. </para> + <para>The <command>fsname_nodirs</command> algorithm generates + device id by hashing the kernel device name for files and by hashing + the hostname for directories. This can be used to deliberately + break lock coherency for directories in a cluster. + </para> <para>The <command>fsid</command> algorithm generates the device id from the <command>f_fsid</command> returned from the <command>statfs()</command> syscall. </para> + <para>The <command>hostname</command> algorithm generates device + id by hashing the hostname. This can be used to deliberately + break lock coherency in a cluster. + </para> + <para>The <command>fsname_norootdir</command> algorithm + generates device ids by hashing the kernel device name, except + for the root directory of shares where it will use the hostname + algorithm. This can be used to deliberately break lock coherency + in a cluster for the root directory of a share. + </para> </listitem> </varlistentry> @@ -75,6 +100,54 @@ </listitem> </varlistentry> + <varlistentry> + <term>fileid:fstype deny = LIST</term> + <listitem> + <para>List of file system types to be ignored for file_id + generation. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>fileid:fstype allow = LIST</term> + <listitem> + <para>List of file system types to be allowed for file_id + generation. If this option is set, file system types not listed + here are ignored. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>fileid:mntdir deny = LIST</term> + <listitem> + <para>List of file system mount points to be ignored for + file_id generation. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>fileid:mntdir allow = LIST</term> + <listitem> + <para>List of file system mount points to be allowed for file_id + generation. If this option is set, file system mount points + not listed here are ignored. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>fileid:nolockinode</term> + <listitem> + <para>This option triggers use of the fileid hostname algorithm + for the configured inode which can be used to deliberately break + lock coherency for the corresponding file or directory in a + cluster. + </para> + </listitem> + </varlistentry> </variablelist> </refsect1> diff --git a/docs-xml/manpages/vfs_fruit.8.xml b/docs-xml/manpages/vfs_fruit.8.xml index fcaf173..7f6a0e7 100644 --- a/docs-xml/manpages/vfs_fruit.8.xml +++ b/docs-xml/manpages/vfs_fruit.8.xml @@ -242,6 +242,24 @@ </varlistentry> <varlistentry> + <term>fruit:time machine max size = SIZE [K|M|G|T|P]</term> + <listitem> + <para>Useful for Time Machine: limits the reported disksize, thus + preventing Time Machine from using the whole real disk space for + backup. The option takes a number plus an optional unit.</para> + <para><emphasis>IMPORTANT</emphasis>: This is an approximated + calculation that only takes into account the contents of Time + Machine sparsebundle images. Therefor you <emphasis>MUST + NOT</emphasis> use this volume to store other content when using + this option, because it would NOT be accounted.</para> + <para>The calculation works by reading the band size from the + Info.plist XML file of the sparsebundle, reading the bands/ + directory counting the number of band files, and then multiplying + one with the other.</para> + </listitem> + </varlistentry> + + <varlistentry> <term>fruit:metadata = [ stream | netatalk ]</term> <listitem> <para>Controls where the OS X metadata stream is stored:</para> diff --git a/libcli/dns/dns.c b/libcli/dns/dns.c index 6404cb8..c30de2d 100644 --- a/libcli/dns/dns.c +++ b/libcli/dns/dns.c @@ -45,11 +45,11 @@ struct dns_udp_request_state { static void dns_udp_request_get_reply(struct tevent_req *subreq); static void dns_udp_request_done(struct tevent_req *subreq); -struct tevent_req *dns_udp_request_send(TALLOC_CTX *mem_ctx, - struct tevent_context *ev, - const char *server_addr_string, - const uint8_t *query, - size_t query_len) +static struct tevent_req *dns_udp_request_send(TALLOC_CTX *mem_ctx, + struct tevent_context *ev, + const char *server_addr_string, + const uint8_t *query, + size_t query_len) { struct tevent_req *req, *subreq; struct dns_udp_request_state *state; @@ -158,10 +158,10 @@ static void dns_udp_request_done(struct tevent_req *subreq) tevent_req_done(req); } -int dns_udp_request_recv(struct tevent_req *req, - TALLOC_CTX *mem_ctx, - uint8_t **reply, - size_t *reply_len) +static int dns_udp_request_recv(struct tevent_req *req, + TALLOC_CTX *mem_ctx, + uint8_t **reply, + size_t *reply_len) { struct dns_udp_request_state *state = tevent_req_data(req, struct dns_udp_request_state); @@ -201,11 +201,11 @@ static int dns_tcp_request_next_vector(struct tstream_context *stream, size_t *_count); static void dns_tcp_request_received(struct tevent_req *subreq); -struct tevent_req *dns_tcp_request_send(TALLOC_CTX *mem_ctx, - struct tevent_context *ev, - const char *server_addr_string, - const uint8_t *query, - size_t query_len) +static struct tevent_req *dns_tcp_request_send(TALLOC_CTX *mem_ctx, + struct tevent_context *ev, + const char *server_addr_string, + const uint8_t *query, + size_t query_len) { struct tevent_req *req, *subreq; struct dns_tcp_request_state *state; @@ -377,10 +377,10 @@ static void dns_tcp_request_received(struct tevent_req *subreq) tevent_req_done(req); } -int dns_tcp_request_recv(struct tevent_req *req, - TALLOC_CTX *mem_ctx, - uint8_t **reply, - size_t *reply_len) +static int dns_tcp_request_recv(struct tevent_req *req, + TALLOC_CTX *mem_ctx, + uint8_t **reply, + size_t *reply_len) { struct dns_tcp_request_state *state = tevent_req_data( req, struct dns_tcp_request_state); diff --git a/libcli/dns/libdns.h b/libcli/dns/libdns.h index 1b7c404..15ca257 100644 --- a/libcli/dns/libdns.h +++ b/libcli/dns/libdns.h @@ -26,44 +26,6 @@ #include "lib/util/time.h" #include "librpc/gen_ndr/dns.h" -/** Send an dns request to a dns server using UDP - * - *@param mem_ctx talloc memory context to use - *@param ev tevent context to use - *@param server_address address of the server as a string - *@param query dns query to send - *@param query_len length of the query - *@return tevent_req with the active request or NULL on out-of-memory - */ -struct tevent_req *dns_udp_request_send(TALLOC_CTX *mem_ctx, - struct tevent_context *ev, - const char *server_address, - const uint8_t *query, - size_t query_len); - -/** Get the dns response from a dns server via UDP - * - *@param req tevent_req struct returned from dns_udp_request_send - *@param mem_ctx talloc memory context to use for the reply string - *@param reply buffer that will be allocated and filled with the dns reply - *@param reply_len length of the reply buffer - *@return 0/errno - */ -int dns_udp_request_recv(struct tevent_req *req, - TALLOC_CTX *mem_ctx, - uint8_t **reply, - size_t *reply_len); - -struct tevent_req *dns_tcp_request_send(TALLOC_CTX *mem_ctx, - struct tevent_context *ev, - const char *server_addr_string, - const uint8_t *query, - size_t query_len); -int dns_tcp_request_recv(struct tevent_req *req, - TALLOC_CTX *mem_ctx, - uint8_t **reply, - size_t *reply_len); - /* * DNS request with fallback to TCP on truncation */ diff --git a/selftest/target/Samba3.pm b/selftest/target/Samba3.pm index c9888c2..1e652d8e 100755 --- a/selftest/target/Samba3.pm +++ b/selftest/target/Samba3.pm @@ -1957,6 +1957,14 @@ sub provision($$$$$$$$$) path = $shrdir vfs objects = streams_depot acl_xattr +[vfs_fruit_timemachine] + path = $shrdir + vfs objects = fruit streams_xattr acl_xattr + fruit:resource = file + fruit:metadata = stream + fruit:time machine = yes + fruit:time machine max size = 32K + [badname-tmp] path = $badnames_shrdir guest ok = yes diff --git a/source3/modules/vfs_fileid.c b/source3/modules/vfs_fileid.c index a7c4a49..98cc32d 100644 --- a/source3/modules/vfs_fileid.c +++ b/source3/modules/vfs_fileid.c @@ -37,11 +37,64 @@ struct fileid_mount_entry { struct fileid_handle_data { uint64_t (*device_mapping_fn)(struct fileid_handle_data *data, - SMB_DEV_T dev); + const SMB_STRUCT_STAT *sbuf); + char **fstype_deny_list; + char **fstype_allow_list; + char **mntdir_deny_list; + char **mntdir_allow_list; unsigned num_mount_entries; struct fileid_mount_entry *mount_entries; + ino_t nolockinode; }; +/* check if a mount entry is allowed based on fstype and mount directory */ +static bool fileid_mount_entry_allowed(struct fileid_handle_data *data, + struct mntent *m) +{ + int i; + char **fstype_deny = data->fstype_deny_list; + char **fstype_allow = data->fstype_allow_list; + char **mntdir_deny = data->mntdir_deny_list; + char **mntdir_allow = data->mntdir_allow_list; + + if (fstype_deny != NULL) { + for (i = 0; fstype_deny[i] != NULL; i++) { + if (strcmp(m->mnt_type, fstype_deny[i]) == 0) { + return false; + } + } + } + if (fstype_allow != NULL) { + for (i = 0; fstype_allow[i] != NULL; i++) { + if (strcmp(m->mnt_type, fstype_allow[i]) == 0) { + break; + } + } + if (fstype_allow[i] == NULL) { + return false; + } + } + if (mntdir_deny != NULL) { + for (i=0; mntdir_deny[i] != NULL; i++) { + if (strcmp(m->mnt_dir, mntdir_deny[i]) == 0) { + return false; + } + } + } + if (mntdir_allow != NULL) { + for (i=0; mntdir_allow[i] != NULL; i++) { + if (strcmp(m->mnt_dir, mntdir_allow[i]) == 0) { + break; + } + } + if (mntdir_allow[i] == NULL) { + return false; + } + } + return true; +} + + /* load all the mount entries from the mtab */ static void fileid_load_mount_entries(struct fileid_handle_data *data) { @@ -58,7 +111,13 @@ static void fileid_load_mount_entries(struct fileid_handle_data *data) struct stat st; struct statfs sfs; struct fileid_mount_entry *cur; + bool allowed; + allowed = fileid_mount_entry_allowed(data, m); + if (!allowed) { + DBG_DEBUG("skipping mount entry %s\n", m->mnt_dir); + continue; + } if (stat(m->mnt_dir, &st) != 0) continue; if (statfs(m->mnt_dir, &sfs) != 0) continue; @@ -136,12 +195,12 @@ static uint64_t fileid_uint64_hash(const uint8_t *s, size_t len) /* a device mapping using a fsname */ static uint64_t fileid_device_mapping_fsname(struct fileid_handle_data *data, - SMB_DEV_T dev) + const SMB_STRUCT_STAT *sbuf) { struct fileid_mount_entry *m; - m = fileid_find_mount_entry(data, dev); - if (!m) return dev; + m = fileid_find_mount_entry(data, sbuf->st_ex_dev); + if (!m) return sbuf->st_ex_dev; if (m->devid == (uint64_t)-1) { m->devid = fileid_uint64_hash((const uint8_t *)m->mnt_fsname, @@ -151,14 +210,55 @@ static uint64_t fileid_device_mapping_fsname(struct fileid_handle_data *data, return m->devid; } +/* a device mapping using a hostname */ +static uint64_t fileid_device_mapping_hostname(struct fileid_handle_data *data, + const SMB_STRUCT_STAT *sbuf) +{ + char hostname[HOST_NAME_MAX+1]; + char *devname = NULL; + uint64_t id; + size_t devname_len; + int rc; + + rc = gethostname(hostname, HOST_NAME_MAX+1); + if (rc != 0) { + DBG_ERR("gethostname failed\n"); + return UINT64_MAX; + } + + devname = talloc_asprintf(talloc_tos(), "%s%lu", + hostname, sbuf->st_ex_dev); + if (devname == NULL) { + DBG_ERR("talloc_asprintf failed\n"); + return UINT64_MAX; + } + devname_len = talloc_array_length(devname) - 1; + TALLOC_FREE(devname); + + id = fileid_uint64_hash((uint8_t *)devname, devname_len); + return id; +} + +/* a device mapping using a fsname for files and hostname for dirs */ +static uint64_t fileid_device_mapping_fsname_nodirs( + struct fileid_handle_data *data, + const SMB_STRUCT_STAT *sbuf) +{ + if (S_ISDIR(sbuf->st_ex_mode)) { + return fileid_device_mapping_hostname(data, sbuf); + } + + return fileid_device_mapping_fsname(data, sbuf); +} + /* device mapping functions using a fsid */ static uint64_t fileid_device_mapping_fsid(struct fileid_handle_data *data, - SMB_DEV_T dev) + const SMB_STRUCT_STAT *sbuf) { struct fileid_mount_entry *m; - m = fileid_find_mount_entry(data, dev); - if (!m) return dev; + m = fileid_find_mount_entry(data, sbuf->st_ex_dev); + if (!m) return sbuf->st_ex_dev; if (m->devid == (uint64_t)-1) { if (sizeof(fsid_t) > sizeof(uint64_t)) { @@ -178,11 +278,43 @@ static uint64_t fileid_device_mapping_fsid(struct fileid_handle_data *data, return m->devid; } +static int get_connectpath_ino(struct vfs_handle_struct *handle, + ino_t *ino) +{ + struct smb_filename *fname = NULL; + int ret; + + fname = synthetic_smb_fname(talloc_tos(), + handle->conn->connectpath, + NULL, + NULL, + 0); + if (fname == NULL) { + DBG_ERR("synthetic_smb_fname failed\n"); + return -1; + } + + ret = SMB_VFS_NEXT_STAT(handle, fname); + TALLOC_FREE(fname); + if (ret != 0) { + DBG_ERR("stat failed for %s with %s\n", + handle->conn->connectpath, strerror(errno)); + return -1; + } + + return 0; +} + static int fileid_connect(struct vfs_handle_struct *handle, const char *service, const char *user) { struct fileid_handle_data *data; const char *algorithm; + const char **fstype_deny_list = NULL; + const char **fstype_allow_list = NULL; + const char **mntdir_deny_list = NULL; + const char **mntdir_allow_list = NULL; + int saved_errno; int ret = SMB_VFS_NEXT_CONNECT(handle, service, user); if (ret < 0) { @@ -191,11 +323,15 @@ static int fileid_connect(struct vfs_handle_struct *handle, data = talloc_zero(handle->conn, struct fileid_handle_data); if (!data) { + saved_errno = errno; SMB_VFS_NEXT_DISCONNECT(handle); DEBUG(0, ("talloc_zero() failed\n")); + errno = saved_errno; return -1; } + data->nolockinode = 0; + /* * "fileid:mapping" is only here as fallback for old setups * "fileid:algorithm" is the option new setups should use @@ -208,20 +344,89 @@ static int fileid_connect(struct vfs_handle_struct *handle, algorithm); if (strcmp("fsname", algorithm) == 0) { data->device_mapping_fn = fileid_device_mapping_fsname; + } else if (strcmp("fsname_nodirs", algorithm) == 0) { -- Samba Shared Repository