The branch, v3-6-test has been updated
via 16f900c Allow vfs_aio_pthread to build as a static module.
via c738f0e Update man page to fix typo vfs_aio_fork ->
vfs_aio_pthread, add aio read size, aio write size examples. (cherry picked
from commit 12b614a9298974ba5daee7aa8d1aa47006de01e2)
via 0ba64e6 Add vfs_aio_pthread code.
via caa7cca Ensure we always free aio_ex on all error paths by moving
the TALLOC_FREE call out of smbd_aio_complete_aio_ex() and into the caller.
via 00d59a0 Add man page for vfs_aio_pthread module. (cherry picked
from commit d8c699190d2cc0ce64395c7b2b10bb25c98a2943)
via b306267 Change the signature of pthreadpool_finished_job() to
return 0 on success, errno on fail and return the jobid in a separate variable.
from 5bfe963 s3:smb2_server: fix a logic error, we should sign non guest
sessions
http://gitweb.samba.org/?p=samba.git;a=shortlog;h=v3-6-test
- Log -----------------------------------------------------------------
commit 16f900cb94a69a47d627666751d374f097f092f4
Author: Christian Ambach <[email protected]>
Date: Fri Jan 27 10:25:13 2012 -0800
Allow vfs_aio_pthread to build as a static module.
The last 6 patches address bug #8723 (Add pthread-based aio module to
3.6.3.).
commit c738f0ea9e1c2356eab1dac778ceb94f22036f0a
Author: Jeremy Allison <[email protected]>
Date: Wed Jan 25 17:17:48 2012 -0800
Update man page to fix typo vfs_aio_fork -> vfs_aio_pthread, add aio read
size, aio write size examples. (cherry picked from commit
12b614a9298974ba5daee7aa8d1aa47006de01e2)
commit 0ba64e6bc78404b2f75af638c22b52007159d96b
Author: Jeremy Allison <[email protected]>
Date: Wed Jan 25 16:54:39 2012 -0800
Add vfs_aio_pthread code.
commit caa7ccae10f9be77cf28890aadff735ca83de93e
Author: Jeremy Allison <[email protected]>
Date: Wed Jan 25 16:27:54 2012 -0800
Ensure we always free aio_ex on all error paths by moving the TALLOC_FREE
call out of smbd_aio_complete_aio_ex() and into the caller.
commit 00d59a043dc4008f25cdf44dc233d181114dfa2d
Author: Jeremy Allison <[email protected]>
Date: Wed Jan 25 14:11:12 2012 -0800
Add man page for vfs_aio_pthread module. (cherry picked from commit
d8c699190d2cc0ce64395c7b2b10bb25c98a2943)
commit b30626720405c435ad48abf8e1445ee8f4b859a3
Author: Jeremy Allison <[email protected]>
Date: Wed Dec 21 20:38:32 2011 -0800
Change the signature of pthreadpool_finished_job() to return 0 on success,
errno on fail and return the jobid in a separate variable.
I need this fix for my vfs_aio_pthread.c module.
Autobuild-User: Jeremy Allison <[email protected]>
Autobuild-Date: Thu Dec 22 12:12:33 CET 2011 on sn-devel-104
(cherry picked from commit 711c18c2301d1bea35cac1144080a94e6b89be27)
-----------------------------------------------------------------------
Summary of changes:
docs-xml/manpages-3/vfs_aio_pthread.8.xml | 120 ++++++
source3/Makefile.in | 5 +
source3/configure.in | 4 +
source3/lib/fncall.c | 3 +-
source3/lib/pthreadpool/pthreadpool.c | 9 +-
source3/lib/pthreadpool/pthreadpool.h | 5 +-
source3/lib/pthreadpool/tests.c | 18 +-
source3/modules/vfs_aio_fork.c | 1 +
source3/modules/vfs_aio_pthread.c | 625 +++++++++++++++++++++++++++++
source3/smbd/aio.c | 3 +-
10 files changed, 775 insertions(+), 18 deletions(-)
create mode 100644 docs-xml/manpages-3/vfs_aio_pthread.8.xml
create mode 100644 source3/modules/vfs_aio_pthread.c
Changeset truncated at 500 lines:
diff --git a/docs-xml/manpages-3/vfs_aio_pthread.8.xml
b/docs-xml/manpages-3/vfs_aio_pthread.8.xml
new file mode 100644
index 0000000..3e41ee9
--- /dev/null
+++ b/docs-xml/manpages-3/vfs_aio_pthread.8.xml
@@ -0,0 +1,120 @@
+<?xml version="1.0" encoding="iso-8859-1"?>
+<!DOCTYPE refentry PUBLIC "-//Samba-Team//DTD DocBook V4.2-Based Variant
V1.0//EN" "http://www.samba.org/samba/DTD/samba-doc";>
+<refentry id="vfs_aio_pthread.8">
+
+<refmeta>
+ <refentrytitle>vfs_aio_pthread</refentrytitle>
+ <manvolnum>8</manvolnum>
+ <refmiscinfo class="source">Samba</refmiscinfo>
+ <refmiscinfo class="manual">System Administration tools</refmiscinfo>
+ <refmiscinfo class="version">3.6</refmiscinfo>
+</refmeta>
+
+
+<refnamediv>
+ <refname>vfs_aio_pthread</refname>
+ <refpurpose>implement async I/O in Samba vfs using a pthread
pool</refpurpose>
+</refnamediv>
+
+<refsynopsisdiv>
+ <cmdsynopsis>
+ <command>vfs objects = aio_pthread</command>
+ </cmdsynopsis>
+</refsynopsisdiv>
+
+<refsect1>
+ <title>DESCRIPTION</title>
+
+ <para>This VFS module is part of the
+ <citerefentry><refentrytitle>samba</refentrytitle>
+ <manvolnum>7</manvolnum></citerefentry> suite.</para>
+
+ <para>The <command>aio_pthread</command> VFS module enables asynchronous
+ I/O for Samba on platforms which have the pthreads API available,
+ without using the Posix AIO interface. Posix AIO can suffer from severe
+ limitations. For example, on some Linux versions the
+ real-time signals that it uses are broken under heavy load.
+ Other systems only allow AIO when special kernel modules are
+ loaded or only allow a certain system-wide amount of async
+ requests being scheduled. Systems based on glibc (most Linux
+ systems) only allow a single outstanding request per file
+ descriptor which essentially makes Posix AIO useless on systems
+ using the glibc implementation.</para>
+
+ <para>To work around all these limitations, the aio_pthread module
+ was written. It uses a pthread pool instead of the
+ internal Posix AIO interface to allow read and write calls
+ to be process asynchronously. A pthread pool is created
+ which expands dynamically by creating new threads as work is
+ given to it to a maximum of 100 threads per smbd process.
+ To change this limit see the "aio num threads" parameter
+ below. New threads are not created if idle threads are
+ available when a new read or write request is received,
+ the new work is given to an existing idle thread. Threads
+ terminate themselves if idle for one second.
+ </para>
+
+ <para>
+ Note that the smb.conf parameters <command>aio read size</command>
+ and <command>aio write size</command> must also be set appropriately
+ for this module to be active.
+ </para>
+
+ <para>This module MUST be listed last in any module stack as
+ the Samba VFS pread/pwrite interface is not thread-safe. This
+ module makes direct pread and pwrite system calls and does
+ NOT call the Samba VFS pread and pwrite interfaces.</para>
+
+</refsect1>
+
+
+<refsect1>
+ <title>EXAMPLES</title>
+
+ <para>Straight forward use:</para>
+
+<programlisting>
+ <smbconfsection name="[cooldata]"/>
+ <smbconfoption name="path">/data/ice</smbconfoption>
+ <smbconfoption name="aio read size">1024</smbconfoption>
+ <smbconfoption name="aio write size">1024</smbconfoption>
+ <smbconfoption name="vfs objects">aio_pthread</smbconfoption>
+</programlisting>
+
+</refsect1>
+
+<refsect1>
+ <title>OPTIONS</title>
+
+ <variablelist>
+
+ <varlistentry>
+ <term>aio_pthread:aio num threads = INTEGER</term>
+ <listitem>
+ <para>Limit the maximum number of threads per smbd that
+ will be created in the thread pool to service IO requests.
+ </para>
+ <para>By default this is set to 100.</para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+</refsect1>
+<refsect1>
+ <title>VERSION</title>
+
+ <para>This man page is correct for version 3.6.3 of the Samba suite.
+ </para>
+</refsect1>
+
+<refsect1>
+ <title>AUTHOR</title>
+
+ <para>The original Samba software and related utilities
+ were created by Andrew Tridgell. Samba is now developed
+ by the Samba Team as an Open Source project similar
+ to the way the Linux kernel is developed.</para>
+
+</refsect1>
+
+</refentry>
diff --git a/source3/Makefile.in b/source3/Makefile.in
index 7e98db7..007c82d 100644
--- a/source3/Makefile.in
+++ b/source3/Makefile.in
@@ -829,6 +829,7 @@ VFS_READAHEAD_OBJ = modules/vfs_readahead.o
VFS_TSMSM_OBJ = modules/vfs_tsmsm.o
VFS_FILEID_OBJ = modules/vfs_fileid.o
VFS_AIO_FORK_OBJ = modules/vfs_aio_fork.o
+VFS_AIO_PTHREAD_OBJ = modules/vfs_aio_pthread.o
VFS_PREOPEN_OBJ = modules/vfs_preopen.o
VFS_SYNCOPS_OBJ = modules/vfs_syncops.o
VFS_ACL_XATTR_OBJ = modules/vfs_acl_xattr.o
@@ -3029,6 +3030,10 @@ bin/aio_fork.@SHLIBEXT@: $(BINARY_PREREQS)
$(VFS_AIO_FORK_OBJ)
@echo "Building plugin $@"
@$(SHLD_MODULE) $(VFS_AIO_FORK_OBJ)
+bin/aio_pthread.@SHLIBEXT@: $(BINARY_PREREQS) $(VFS_AIO_PTHREAD_OBJ)
lib/pthreadpool/pthreadpool.o
+ @echo "Building plugin $@"
+ @$(SHLD_MODULE) $(VFS_AIO_PTHREAD_OBJ) lib/pthreadpool/pthreadpool.o
-lpthread
+
bin/preopen.@SHLIBEXT@: $(BINARY_PREREQS) $(VFS_PREOPEN_OBJ)
@echo "Building plugin $@"
@$(SHLD_MODULE) $(VFS_PREOPEN_OBJ)
diff --git a/source3/configure.in b/source3/configure.in
index 398a4f8..d8d3a1f 100644
--- a/source3/configure.in
+++ b/source3/configure.in
@@ -6674,6 +6674,9 @@ if test x"$enable_pthreadpool" = x"yes" -a
x"$samba_cv_HAVE_PTHREAD" = x"yes"; t
AC_SUBST(PTHREADPOOL_OBJ, "lib/pthreadpool/pthreadpool.o")
PTHREADPOOLTEST="bin/pthreadpooltest\$(EXEEXT)"
AC_SUBST(PTHREADPOOLTEST)
+ if test x"$samba_cv_HAVE_AIO" = x"yes"; then
+ default_shared_modules="$default_shared_modules vfs_aio_pthread"
+ fi
fi
#################################################
@@ -6901,6 +6904,7 @@ SMB_MODULE(vfs_readahead, \$(VFS_READAHEAD_OBJ),
"bin/readahead.$SHLIBEXT", VFS)
SMB_MODULE(vfs_tsmsm, \$(VFS_TSMSM_OBJ), "bin/tsmsm.$SHLIBEXT", VFS)
SMB_MODULE(vfs_fileid, \$(VFS_FILEID_OBJ), "bin/fileid.$SHLIBEXT", VFS)
SMB_MODULE(vfs_aio_fork, \$(VFS_AIO_FORK_OBJ), "bin/aio_fork.$SHLIBEXT", VFS)
+SMB_MODULE(vfs_aio_pthread, \$(VFS_AIO_PTHREAD_OBJ),
"bin/aio_pthread.$SHLIBEXT", VFS)
SMB_MODULE(vfs_preopen, \$(VFS_PREOPEN_OBJ), "bin/preopen.$SHLIBEXT", VFS)
SMB_MODULE(vfs_syncops, \$(VFS_SYNCOPS_OBJ), "bin/syncops.$SHLIBEXT", VFS)
SMB_MODULE(vfs_zfsacl, \$(VFS_ZFSACL_OBJ), "bin/zfsacl.$SHLIBEXT", VFS)
diff --git a/source3/lib/fncall.c b/source3/lib/fncall.c
index 4a013e9..13bf093 100644
--- a/source3/lib/fncall.c
+++ b/source3/lib/fncall.c
@@ -280,8 +280,7 @@ static void fncall_handler(struct tevent_context *ev,
struct tevent_fd *fde,
int i, num_pending;
int job_id;
- job_id = pthreadpool_finished_job(ctx->pool);
- if (job_id <= 0) {
+ if (pthreadpool_finished_job(ctx->pool, &job_id) != 0) {
return;
}
diff --git a/source3/lib/pthreadpool/pthreadpool.c
b/source3/lib/pthreadpool/pthreadpool.c
index 7538fb7..c2dd92a 100644
--- a/source3/lib/pthreadpool/pthreadpool.c
+++ b/source3/lib/pthreadpool/pthreadpool.c
@@ -284,16 +284,16 @@ static void pthreadpool_join_children(struct pthreadpool
*pool)
* Fetch a finished job number from the signal pipe
*/
-int pthreadpool_finished_job(struct pthreadpool *pool)
+int pthreadpool_finished_job(struct pthreadpool *pool, int *jobid)
{
- int result;
+ int ret_jobid;
ssize_t nread;
nread = -1;
errno = EINTR;
while ((nread == -1) && (errno == EINTR)) {
- nread = read(pool->sig_pipe[0], &result, sizeof(int));
+ nread = read(pool->sig_pipe[0], &ret_jobid, sizeof(int));
}
if (nread == -1) {
return errno;
@@ -301,7 +301,8 @@ int pthreadpool_finished_job(struct pthreadpool *pool)
if (nread != sizeof(int)) {
return EINVAL;
}
- return result;
+ *jobid = ret_jobid;
+ return 0;
}
/*
diff --git a/source3/lib/pthreadpool/pthreadpool.h
b/source3/lib/pthreadpool/pthreadpool.h
index 79704ea..0fde3c8 100644
--- a/source3/lib/pthreadpool/pthreadpool.h
+++ b/source3/lib/pthreadpool/pthreadpool.h
@@ -90,8 +90,9 @@ int pthreadpool_signal_fd(struct pthreadpool *pool);
* pthreadpool_signal_fd() is readable.
*
* @param[in] pool The pool to query for finished jobs
- * @return The job_id of the finished job
+ * @param[out] pjobid The job_id of the finished job
+ * @return success: 0, failure: errno
*/
-int pthreadpool_finished_job(struct pthreadpool *pool);
+int pthreadpool_finished_job(struct pthreadpool *pool, int *jobid);
#endif
diff --git a/source3/lib/pthreadpool/tests.c b/source3/lib/pthreadpool/tests.c
index 667ee01..95d37b6 100644
--- a/source3/lib/pthreadpool/tests.c
+++ b/source3/lib/pthreadpool/tests.c
@@ -68,12 +68,13 @@ static int test_jobs(int num_threads, int num_jobs)
}
for (i=0; i<num_jobs; i++) {
- ret = pthreadpool_finished_job(p);
- if ((ret < 0) || (ret >= num_jobs)) {
- fprintf(stderr, "invalid job number %d\n", ret);
+ int jobid = -1;
+ ret = pthreadpool_finished_job(p, &jobid);
+ if ((ret != 0) || (jobid >= num_jobs)) {
+ fprintf(stderr, "invalid job number %d\n", jobid);
return -1;
}
- finished[ret] += 1;
+ finished[jobid] += 1;
}
for (i=0; i<num_jobs; i++) {
@@ -275,18 +276,19 @@ static int test_threaded_addjob(int num_pools, int
num_threads, int poolsize,
}
for (j=0; j<num_pools; j++) {
+ int jobid = -1;
if ((pfds[j].revents & (POLLIN|POLLHUP)) == 0) {
continue;
}
- ret = pthreadpool_finished_job(pools[j]);
- if ((ret < 0) || (ret >= num_jobs * num_threads)) {
+ ret = pthreadpool_finished_job(pools[j], &jobid);
+ if ((ret != 0) || (jobid >= num_jobs * num_threads)) {
fprintf(stderr, "invalid job number %d\n",
- ret);
+ jobid);
return -1;
}
- finished[ret] += 1;
+ finished[jobid] += 1;
received += 1;
}
}
diff --git a/source3/modules/vfs_aio_fork.c b/source3/modules/vfs_aio_fork.c
index 41b5a89..7f6a021 100644
--- a/source3/modules/vfs_aio_fork.c
+++ b/source3/modules/vfs_aio_fork.c
@@ -434,6 +434,7 @@ static void handle_aio_completion(struct event_context
*event_ctx,
aio_ex = (struct aio_extra
*)child->aiocb->aio_sigevent.sigev_value.sival_ptr;
smbd_aio_complete_aio_ex(aio_ex);
+ TALLOC_FREE(aio_ex);
}
static int aio_child_destructor(struct aio_child *child)
diff --git a/source3/modules/vfs_aio_pthread.c
b/source3/modules/vfs_aio_pthread.c
new file mode 100644
index 0000000..ceef822
--- /dev/null
+++ b/source3/modules/vfs_aio_pthread.c
@@ -0,0 +1,625 @@
+/*
+ * Simulate Posix AIO using pthreads.
+ *
+ * Based on the aio_fork work from Volker and Volker's pthreadpool library.
+ *
+ * Copyright (C) Volker Lendecke 2008
+ * Copyright (C) Jeremy Allison 2012
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ */
+
+#include "includes.h"
+#include "system/filesys.h"
+#include "system/shmem.h"
+#include "smbd/smbd.h"
+#include "lib/pthreadpool/pthreadpool.h"
+
+struct aio_extra;
+static struct pthreadpool *pool;
+static int aio_pthread_jobid;
+
+struct aio_private_data {
+ struct aio_private_data *prev, *next;
+ int jobid;
+ SMB_STRUCT_AIOCB *aiocb;
+ ssize_t ret_size;
+ int ret_errno;
+ bool cancelled;
+ bool write_command;
+};
+
+/* List of outstanding requests we have. */
+static struct aio_private_data *pd_list;
+
+static void aio_pthread_handle_completion(struct event_context *event_ctx,
+ struct fd_event *event,
+ uint16 flags,
+ void *p);
+
+/************************************************************************
+ How many threads to initialize ?
+ 100 per process seems insane as a default until you realize that
+ (a) Threads terminate after 1 second when idle.
+ (b) Throttling is done in SMB2 via the crediting algorithm.
+ (c) SMB1 clients are limited to max_mux (50) outstanding requests and
+ Windows clients don't use this anyway.
+ Essentially we want this to be unlimited unless smb.conf says different.
+***********************************************************************/
+
+static int aio_get_num_threads(struct vfs_handle_struct *handle)
+{
+ return lp_parm_int(SNUM(handle->conn),
+ "aio_pthread", "aio num threads", 100);
+}
+
+/************************************************************************
+ Ensure thread pool is initialized.
+***********************************************************************/
+
+static bool init_aio_threadpool(struct vfs_handle_struct *handle)
+{
+ struct fd_event *sock_event = NULL;
+ int ret = 0;
+ int num_threads;
+
+ if (pool) {
+ return true;
+ }
+
+ num_threads = aio_get_num_threads(handle);
+ ret = pthreadpool_init(num_threads, &pool);
+ if (ret) {
+ errno = ret;
+ return false;
+ }
+ sock_event = tevent_add_fd(server_event_context(),
+ NULL,
+ pthreadpool_signal_fd(pool),
+ TEVENT_FD_READ,
+ aio_pthread_handle_completion,
+ NULL);
+ if (sock_event == NULL) {
+ pthreadpool_destroy(pool);
+ pool = NULL;
+ return false;
+ }
+
+ DEBUG(10,("init_aio_threadpool: initialized with up to %d threads\n",
+ num_threads));
+
+ return true;
+}
+
+
+/************************************************************************
+ Worker function - core of the pthread aio engine.
+ This is the function that actually does the IO.
+***********************************************************************/
+
+static void aio_worker(void *private_data)
+{
+ struct aio_private_data *pd =
+ (struct aio_private_data *)private_data;
+
+ if (pd->write_command) {
+ pd->ret_size = sys_pwrite(pd->aiocb->aio_fildes,
+ (const void *)pd->aiocb->aio_buf,
+ pd->aiocb->aio_nbytes,
+ pd->aiocb->aio_offset);
+ if (pd->ret_size == -1 && errno == ESPIPE) {
+ /* Maintain the fiction that pipes can
+ be seeked (sought?) on. */
+ pd->ret_size = sys_write(pd->aiocb->aio_fildes,
+ (const void *)pd->aiocb->aio_buf,
+ pd->aiocb->aio_nbytes);
+ }
+ } else {
+ pd->ret_size = sys_pread(pd->aiocb->aio_fildes,
+ (void *)pd->aiocb->aio_buf,
+ pd->aiocb->aio_nbytes,
+ pd->aiocb->aio_offset);
+ if (pd->ret_size == -1 && errno == ESPIPE) {
+ /* Maintain the fiction that pipes can
+ be seeked (sought?) on. */
+ pd->ret_size = sys_read(pd->aiocb->aio_fildes,
+ (void *)pd->aiocb->aio_buf,
+ pd->aiocb->aio_nbytes);
+ }
+ }
+ if (pd->ret_size == -1) {
+ pd->ret_errno = errno;
+ } else {
+ pd->ret_errno = 0;
+ }
+}
+
+/************************************************************************
+ Private data destructor.
+***********************************************************************/
+
+static int pd_destructor(struct aio_private_data *pd)
+{
+ DLIST_REMOVE(pd_list, pd);
+ return 0;
+}
+
+/************************************************************************
+ Create and initialize a private data struct.
+***********************************************************************/
+
+static struct aio_private_data *create_private_data(TALLOC_CTX *ctx,
+ SMB_STRUCT_AIOCB *aiocb)
+{
+ struct aio_private_data *pd = talloc_zero(ctx, struct aio_private_data);
+ if (!pd) {
+ return NULL;
+ }
+ pd->jobid = aio_pthread_jobid++;
+ pd->aiocb = aiocb;
+ pd->ret_size = -1;
+ pd->ret_errno = EINPROGRESS;
+ talloc_set_destructor(pd, pd_destructor);
+ DLIST_ADD_END(pd_list, pd, struct aio_private_data *);
+ return pd;
+}
+
+/************************************************************************
+ Spin off a threadpool (if needed) and initiate a pread call.
+***********************************************************************/
+
+static int aio_pthread_read(struct vfs_handle_struct *handle,
+ struct files_struct *fsp,
+ SMB_STRUCT_AIOCB *aiocb)
+{
+ struct aio_extra *aio_ex = (struct aio_extra
*)aiocb->aio_sigevent.sigev_value.sival_ptr;
+ struct aio_private_data *pd = NULL;
+ int ret;
+
+ if (!init_aio_threadpool(handle)) {
+ return -1;
+ }
+
+ pd = create_private_data(aio_ex, aiocb);
+ if (pd == NULL) {
+ DEBUG(10, ("aio_pthread_read: Could not create private
data.\n"));
+ return -1;
+ }
+
+ ret = pthreadpool_add_job(pool, pd->jobid, aio_worker, (void *)pd);
--
Samba Shared Repository
