And another. I have now implemented the subdirectories-in-hugetlbfs
model for security. There are basically two options for setting up
sharing:
- Set up your own directory, with whatever permission model
you want, and explicitly point libhugetlbfs there by setting
HUGETLB_SHARE_PATH. If this environment variable exists, it checks
that the given path exists and is on a hugetlbfs, but otherwise
assumes you know what you're doing.
- Use the default approach: libhugetlbfs will use a
subdirectory of mode 0700 called 'elflink-uid-XXX' in the hugetlbfs
mount, where XXX is the process's uid. If the directory doesn't
exist, it creates it. If it does exist, it checks that its owner is
the same as the process's uid and that its mode does not permit write
access by other uids.
Index: libhugetlbfs/HOWTO
===================================================================
--- libhugetlbfs.orig/HOWTO 2006-10-27 17:14:39.000000000 +1000
+++ libhugetlbfs/HOWTO 2006-10-27 17:14:51.000000000 +1000
@@ -326,42 +326,26 @@ libhugetlbfs:
Sharing remapped segments:
--------------------------
-Sharing of program segments among processes is a two-fold process.
-
-First, a system-wide daemon must be started, hugetlbd. The executable
-(which will be installed into $PREFIX/sbin by make install) should be
-run as root and will automatically spawn a child process to wait for
-segment sharing into the background. The process will log any output
-requested to /tmp/hugetlbd.log. The complete usage for hugetlbd is:
-
-hugetlbd [-v] [-V verbosity] [-p timeout1] [-s timeout2]
- -v: display this help and exit
- -V: specify log level (1-3) [default=1]
- 1: only log errors
- 2: log errors and warnings
- 3: log all output
- -p: specify how long to wait before unlinking files in the hugetlbfs
- mountpoint (seconds) [default=600]
- -s: specify when a file last needs to have been shared for it not to be
- unlinked (seconds) [default=600]
-
-Second, in the applications that wish to share segments, the
-HUGETLB_SHARE environment variable must be set appropriately. The
-variable takes on three values: 0, the default, indicates no segments
-should be shared; 1 indicates that only read-only segments (text) should
-be shared; and 2 indicates that all segments should be shared. Note, as
-this is a per-process variable, it is important to keep its usage
-consistent with HUGETLB_MINIMAL_COPY. If MINIMAL_COPY is used by the
-first process to share a particular executable's segments, it should be
-used by all subsequent ones, to guarantee no failures while copying
-segments. Similarly, if MINIMAL_COPY is not used by the first process
-sharing, it should not be used by subsequent ones.
-
-Having taken these two steps, the same application run twice with the
-same environment variable settings will use the same file both times
-(reducing the overall hugetlbfs file usage). If, for any reason, segment
-sharing is requested but unavailable, the library will fallback to
-unlinked files.
+Sharing of program segments among processes is a simple process. For
+the applications that wish to share segments, the HUGETLB_SHARE
+environment variable must be set appropriately. The variable takes on
+three values: 0, the default, indicates no segments should be shared;
+1 indicates that only read-only segments (text) should be shared; and
+2 indicates that all segments should be shared. Note, as this is a
+per-process variable, it is important to keep its usage consistent
+with HUGETLB_MINIMAL_COPY. If MINIMAL_COPY is used by the first
+process to share a particular executable's segments, it should be used
+by all subsequent ones, to guarantee no failures while copying
+segments. Similarly, if MINIMAL_COPY is not used by the first process
+sharing, it should not be used by subsequent ones. Additionally, as a
+security precaution, sharing is only possible between users in the
+same group.
+
+Now, the same application run twice with the same environment variable
+settings will use the same file both times (reducing the overall
+hugetlbfs file usage). If, for any reason, segment sharing is
+requested but unavailable, the library will fallback to unlinked
+files.
Examples
========
Index: libhugetlbfs/Makefile
===================================================================
--- libhugetlbfs.orig/Makefile 2006-10-27 17:14:39.000000000 +1000
+++ libhugetlbfs/Makefile 2006-10-27 17:14:51.000000000 +1000
@@ -2,7 +2,6 @@ PREFIX = /usr/local
BASEOBJS = hugeutils.o version.o
LIBOBJS = $(BASEOBJS) elflink.o morecore.o debug.o
-SBINOBJS = hugetlbd
INSTALL_OBJ_LIBS = libhugetlbfs.so libhugetlbfs.a
LDSCRIPT_TYPES = B BDT
LDSCRIPT_DIST_ELF = elf32ppclinux elf64ppc elf_i386 elf_x86_64
@@ -84,14 +83,13 @@ endif
DEPFILES = $(LIBOBJS:%.o=%.d)
-all: libs sbin tests
+all: libs tests
.PHONY: tests libs
libs: $(foreach file,$(INSTALL_OBJ_LIBS),$(OBJDIRS:%=%/$(file)))
-sbin: $(foreach file,$(SBINOBJS),$(OBJDIRS:%=%/$(file)))
-tests: libs sbin # Force make to build the library first
+tests: libs # Force make to build the library first
tests: tests/all
tests/%:
@@ -168,14 +166,6 @@ obj64/%.s: %.c
@$(VECHO) CC64 -S $@
$(CC64) $(CPPFLAGS) $(CFLAGS) -o $@ -S $<
-obj32/hugetlbd: hugetlbd.c $(BASEOBJS:%=obj32/%)
- @$(VECHO) CC32 $@
- $(CC32) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS32) -o $@ $^
-
-obj64/hugetlbd: hugetlbd.c $(BASEOBJS:%=obj64/%)
- @$(VECHO) CC64 $@
- $(CC64) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS64) -o $@ $^
-
clean:
@$(VECHO) CLEAN
rm -f *~ *.o *.so *.a *.d *.i core a.out $(VERSION)
Index: libhugetlbfs/elflink.c
===================================================================
--- libhugetlbfs.orig/elflink.c 2006-10-27 17:14:39.000000000 +1000
+++ libhugetlbfs/elflink.c 2006-10-27 17:14:51.000000000 +1000
@@ -1,6 +1,6 @@
/*
* libhugetlbfs - Easy use of Linux hugepages
- * Copyright (C) 2005-2006 David Gibson & Adam Litke, IBM Corporation.
+ * Copyright (C) 2005-2006 David Gibson & dam Litke, IBM Corporation.
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public License
@@ -24,17 +24,20 @@
#include <stdlib.h>
#include <malloc.h>
#include <string.h>
+#include <unistd.h>
+#include <fcntl.h>
#include <signal.h>
#include <sys/syscall.h>
+#include <sys/file.h>
#include <linux/unistd.h>
#include <sys/mman.h>
#include <errno.h>
+#include <limits.h>
#include <elf.h>
#include <dlfcn.h>
#include "hugetlbfs.h"
#include "libhugetlbfs_internal.h"
-
#ifdef __LP64__
#define Elf_Ehdr Elf64_Ehdr
#define Elf_Phdr Elf64_Phdr
@@ -177,19 +180,97 @@ static void unmapped_abort(const char *f
}
#endif /* __syscall_return */
+static char share_path[PATH_MAX+1];
+
#define MAX_HTLB_SEGS 2
+struct seg_info {
+ void *vaddr;
+ unsigned long filesz, memsz;
+ int prot;
+ int fd;
+ int phdr;
+};
+
static struct seg_info htlb_seg_table[MAX_HTLB_SEGS];
static int htlb_num_segs;
static int minimal_copy = 1;
+static int sharing; /* =0 */
int __debug = 0;
static Elf_Ehdr *ehdr;
+/**
+ * assemble_path - handy wrapper around snprintf() for building paths
+ * @dst: buffer of size PATH_MAX+1 to assemble string into
+ * @...: strings to concatenate into the path, terminated with a NULL
+ */
+static void assemble_path(char *dst, const char *fmt, ...)
+{
+ va_list ap;
+ int len;
+
+ va_start(ap, fmt);
+ len = vsnprintf(dst, PATH_MAX+1, fmt, ap);
+ va_end(ap);
+
+ if (len > PATH_MAX) {
+ ERROR("Overflow assembling path");
+ abort();
+ }
+}
+
+static int check_env(void)
+{
+ char *env;
+
+ env = getenv("HUGETLB_ELFMAP");
+ if (env && (strcasecmp(env, "no") == 0)) {
+ DEBUG("HUGETLB_ELFMAP=%s, not attempting to remap program "
+ "segments\n", env);
+ return -1;
+ }
+
+ env = getenv("LD_PRELOAD");
+ if (env && strstr(env, "libhugetlbfs")) {
+ ERROR("LD_PRELOAD is incompatible with segment remapping\n");
+ ERROR("Segment remapping has been DISABLED\n");
+ return -1;
+ }
+
+ env = getenv("HUGETLB_MINIMAL_COPY");
+ if (env && (strcasecmp(env, "no") == 0)) {
+ DEBUG("HUGETLB_MINIMAL_COPY=%s, disabling filesz copy "
+ "optimization\n", env);
+ minimal_copy = 0;
+ }
+
+ env = getenv("HUGETLB_SHARE");
+ if (env)
+ sharing = atoi(env);
+ DEBUG("HUGETLB_SHARE=%d, sharing ", sharing);
+ if (sharing == 2) {
+ DEBUG_CONT("enabled for all segments\n");
+ } else {
+ if (sharing == 1) {
+ DEBUG_CONT("enabled for only read-only segments\n");
+ } else {
+ DEBUG_CONT("disabled\n");
+ }
+ }
+
+ env = getenv("HUGETLB_DEBUG");
+ if (env) {
+ DEBUG("HUGETLB_DEBUG=%s, enabling extra checking\n", env);
+ __debug = 1;
+ }
+
+ return 0;
+}
+
/*
* Parse an ELF header and record segment information for any segments
* which contain hugetlb information.
*/
-
static void parse_phdrs(Elf_Ehdr *ehdr)
{
Elf_Phdr *phdr = (Elf_Phdr *)((char *)ehdr + ehdr->e_phoff);
@@ -231,14 +312,67 @@ static void parse_phdrs(Elf_Ehdr *ehdr)
htlb_seg_table[htlb_num_segs].filesz = filesz;
htlb_seg_table[htlb_num_segs].memsz = memsz;
htlb_seg_table[htlb_num_segs].prot = prot;
+ htlb_seg_table[htlb_num_segs].phdr = i;
htlb_num_segs++;
}
}
+static int find_or_create_share_path(void)
+{
+ char *env;
+ struct stat sb;
+ int ret;
+
+ env = getenv("HUGETLB_SHARE_PATH");
+ if (env) {
+ /* Given an explicit path */
+ if (hugetlbfs_test_path(env) != 0) {
+ ERROR("HUGETLB_SHARE_PATH %s is not on a hugetlbfs\n",
+ share_path);
+ return -1;
+ }
+ assemble_path(share_path, "%s", env);
+ return 0;
+ }
+
+ assemble_path(share_path, "%s/elflink-uid-%d",
+ hugetlbfs_find_path(), getuid());
+
+ ret = mkdir(share_path, 0700);
+ if ((ret != 0) && (errno != EEXIST)) {
+ ERROR("Error creating share directory %s", share_path);
+ return -1;
+ }
+
+ /* Check the share directory is sane */
+ ret = lstat(share_path, &sb);
+ if (ret != 0) {
+ ERROR("Couldn't stat() %s: %s\n", share_path, strerror(errno));
+ return -1;
+ }
+
+ if (! S_ISDIR(sb.st_mode)) {
+ ERROR("%s is not a directory", share_path);
+ return -1;
+ }
+
+ if (sb.st_uid != getuid()) {
+ ERROR("%s has wrong owner (uid=%d instead of %d)\n",
+ share_path, sb.st_uid, getuid());
+ return -1;
+ }
+ if (sb.st_mode & (S_IWGRP | S_IWOTH)) {
+ ERROR("%s has bad permissions 0%03o\n",
+ share_path, sb.st_mode);
+ return -1;
+ }
+
+ return 0;
+}
+
/*
* Look for non-zero BSS data inside a range and print out any matches
*/
-
static void check_bss(unsigned long *start, unsigned long *end)
{
unsigned long *addr;
@@ -249,13 +383,52 @@ static void check_bss(unsigned long *sta
}
}
+/**
+ * get_shared_file_name - create a shared file name from program name, gid and
phdr
+ * @htlb_seg_info: pointer to program's segment data
+ * @file_path: pointer to a PATH_MAX array to store filename in
+ *
+ * The file name created is *not* intended to be unique, except when the name,
+ * gid or phdr number differ. The goal here is to have a standar means of
+ * accessing particular segments of particular executables.
+ *
+ * returns:
+ * -1, on failure
+ * 0, on success
+ */
+static int get_shared_file_name(struct seg_info *htlb_seg_info, char
*file_path)
+{
+ int ret;
+ char binary[PATH_MAX+1];
+ char *binary2;
+
+ memset(binary, 0, sizeof(binary));
+ ret = readlink("/proc/self/exe", binary, PATH_MAX);
+ if (ret < 0) {
+ ERROR("shared_file: readlink() on /proc/self/exe "
+ "failed: %s\n", strerror(errno));
+ return -1;
+ }
+
+ binary2 = basename(binary);
+ if (!binary2) {
+ ERROR("shared_file: basename() on %s failed: %s\n",
+ binary, strerror(errno));
+ return -1;
+ }
+
+ assemble_path(file_path, "%s/%s_%zd_%d", share_path, binary2,
+ sizeof(unsigned long) * 8, htlb_seg_info->phdr);
+
+ return 0;
+}
+
/*
* Subtle: Since libhugetlbfs depends on glibc, we allow it
* it to be loaded before us. As part of its init functions, it
* initializes stdin, stdout, and stderr in the bss. We need to
* include these initialized variables in our copy.
*/
-
static void get_extracopy(struct seg_info *seg, void **extra_start,
void **extra_end)
{
@@ -387,7 +560,6 @@ bail3:
* smallest amount of data possible, unless the user disables this
* optimization via the HUGETLB_ELFMAP environment variable.
*/
-
static int prepare_segment(struct seg_info *seg)
{
int hpage_size = gethugepagesize();
@@ -437,6 +609,138 @@ static int prepare_segment(struct seg_in
return 0;
}
+/**
+ * find_or_prepare_shared_file - get one shareable file
+ * @htlb_seg_info: pointer to program's segment data
+ *
+ * returns:
+ * -1, on failure
+ * 0, on success
+ */
+static int find_or_prepare_shared_file(struct seg_info *htlb_seg_info)
+{
+ int fdx, fds;
+ int errnox, errnos;
+ int ret;
+ char final_path[PATH_MAX+1];
+ char tmp_path[PATH_MAX+1];
+
+ ret = get_shared_file_name(htlb_seg_info, final_path);
+ if (ret < 0)
+ return -1;
+ assemble_path(tmp_path, "%s.tmp", final_path);
+
+ do {
+ /* NB: mode is modified by umask */
+ fdx = open(tmp_path, O_CREAT | O_EXCL | O_RDWR, 0666);
+ errnox = errno;
+ fds = open(final_path, O_RDONLY);
+ errnos = errno;
+
+ if (fds > 0) {
+ /* Got an already-prepared file -> use it */
+ if (fdx > 0) {
+ /* Also got an exclusive file -> clean up */
+ ret = unlink(tmp_path);
+ if (ret != 0)
+ ERROR("shared_file: unable to clean up"
+ " unneeded file %s: %s\n",
+ tmp_path, strerror(errno));
+ close(fdx);
+ } else if (errnox != EEXIST) {
+ WARNING("shared_file: Unexpected failure on
exclusive"
+ " open of %s: %s\n", tmp_path,
+ strerror(errnox));
+ }
+ htlb_seg_info->fd = fds;
+ return 0;
+ }
+
+ if (fdx > 0) {
+ /* It's our job to prepare */
+ if (errnos != ENOENT)
+ WARNING("shared_file: Unexpected failure on"
+ " shared open of %s: %s\n", final_path,
+ strerror(errnos));
+
+ htlb_seg_info->fd = fdx;
+
+ DEBUG("Got unpopulated shared fd -- Preparing\n");
+ ret = prepare_segment(htlb_seg_info);
+ if (ret < 0)
+ goto fail;
+
+ DEBUG("Prepare succeeded\n");
+ /* move to permanent location */
+ ret = rename(tmp_path, final_path);
+ if (ret != 0) {
+ ERROR("shared_file: unable to rename %s"
+ " to %s: %s\n", tmp_path, final_path,
+ strerror(errno));
+ goto fail;
+ }
+
+ return 0;
+ }
+
+ /* Both opens failed, somebody else is still preparing */
+ /* Wait and try again */
+ sleep(1);
+ /* FIXME: should have a timeout */
+ } while (1);
+
+ fail:
+ if (fdx > 0) {
+ ret = unlink(tmp_path);
+ if (ret != 0)
+ ERROR("shared_file: Unable to clean up temp file %s on"
+ " failure: %s\n", tmp_path, strerror(errno));
+ close(fdx);
+ }
+ if (fds > 0)
+ close(fds);
+
+ return -1;
+}
+
+/**
+ * obtain_prepared_file - multiplex callers depending on if
+ * sharing or not
+ * @htlb_seg_info: pointer to program's segment data
+ *
+ * returns:
+ * -1, on error
+ * 0, on success
+ */
+static int obtain_prepared_file(struct seg_info *htlb_seg_info)
+{
+ int fd = -1;
+ int ret;
+
+ /* Either share all segments or share only read-only segments */
+ if ((sharing == 2) || ((sharing == 1) &&
+ !(htlb_seg_info->prot & PROT_WRITE))) {
+ /* first, try to share */
+ ret = find_or_prepare_shared_file(htlb_seg_info);
+ if (ret == 0)
+ return 0;
+ /* but, fall through to unlinked files, if sharing fails */
+ DEBUG("Falling back to unlinked files\n");
+ }
+ fd = hugetlbfs_unlinked_fd();
+ if (fd < 0)
+ return -1;
+ htlb_seg_info->fd = fd;
+
+ ret = prepare_segment(htlb_seg_info);
+ if (ret < 0) {
+ DEBUG("Failed to prepare segment\n");
+ return -1;
+ }
+ DEBUG("Prepare succeeded\n");
+ return 0;
+}
+
static void remap_segments(struct seg_info *seg, int num)
{
int hpage_size = gethugepagesize();
@@ -482,83 +786,6 @@ static void remap_segments(struct seg_in
*/
}
-static int maybe_prepare(int fd_state, struct seg_info *seg)
-{
- int ret, reply = 0;
-
- switch (fd_state) {
- case 0:
- DEBUG("Got populated shared fd -- Not preparing\n");
- return 0;
-
- case 1:
- DEBUG("Got unpopulated shared fd -- Preparing\n");
- reply = 1;
- break;
- case 2:
- DEBUG("Got unshared fd, wanted shared -- Preparing\n");
- break;
- case 3:
- DEBUG("Got unshared fd as expected -- Preparing\n");
- break;
-
- default:
- ERROR("Unexpected fd state: %d in maybe_prepare\n",
- fd_state);
- return -1;
- }
-
- ret = prepare_segment(seg);
- if (ret < 0) {
- /* notify daemon of failed prepare */
- DEBUG("Failed to prepare segment\n");
- finished_prepare(seg, -1);
- return -1;
- }
- DEBUG("Prepare succeeded\n");
- if (reply) {
- ret = finished_prepare(seg, 0);
- if (ret < 0)
- DEBUG("Failed to communicate successful "
- "prepare to hugetlbd\n");
- }
- return ret;
-}
-
-static int check_env(void)
-{
- char *env;
-
- env = getenv("HUGETLB_ELFMAP");
- if (env && (strcasecmp(env, "no") == 0)) {
- DEBUG("HUGETLB_ELFMAP=%s, not attempting to remap program "
- "segments\n", env);
- return -1;
- }
-
- env = getenv("LD_PRELOAD");
- if (env && strstr(env, "libhugetlbfs")) {
- ERROR("LD_PRELOAD is incompatible with segment remapping\n");
- ERROR("Segment remapping has been DISABLED\n");
- return -1;
- }
-
- env = getenv("HUGETLB_MINIMAL_COPY");
- if (env && (strcasecmp(env, "no") == 0)) {
- DEBUG("HUGETLB_MINIMAL_COPY=%s, disabling filesz copy "
- "optimization\n", env);
- minimal_copy = 0;
- }
-
- env = getenv("HUGETLB_DEBUG");
- if (env) {
- DEBUG("HUGETLB_DEBUG=%s, enabling extra checking\n", env);
- __debug = 1;
- }
-
- return 0;
-}
-
static void __attribute__ ((constructor)) setup_elflink(void)
{
extern Elf_Ehdr __executable_start __attribute__((weak));
@@ -581,20 +808,18 @@ static void __attribute__ ((constructor)
return;
}
- /* Step 1. Get access to the files we're going to mmap for the
- * segments */
- for (i = 0; i < htlb_num_segs; i++) {
- ret = hugetlbfs_set_fd(&htlb_seg_table[i]);
- if (ret < 0) {
- DEBUG("Failed to setup hugetlbfs file\n");
+ /* Do we need to find a share directory */
+ if (sharing) {
+ ret = find_or_create_share_path();
+ if (ret != 0)
return;
- }
+ }
- /* Step 2. Map the hugetlbfs files anywhere to copy data, if we
- * need to prepare at all */
- ret = maybe_prepare(ret, &htlb_seg_table[i]);
+ /* Step 1. Obtain hugepage files with our program data */
+ for (i = 0; i < htlb_num_segs; i++) {
+ ret = obtain_prepared_file(&htlb_seg_table[i]);
if (ret < 0) {
- DEBUG("Failed to prepare hugetlbfs file\n");
+ DEBUG("Failed to setup hugetlbfs file\n");
return;
}
}
Index: libhugetlbfs/hugetlbd.c
===================================================================
--- libhugetlbfs.orig/hugetlbd.c 2006-10-27 17:14:39.000000000 +1000
+++ /dev/null 1970-01-01 00:00:00.000000000 +0000
@@ -1,872 +0,0 @@
-/*
- * libhugetlbfs - Easy use of Linux hugepages
- * Copyright (C) 2006 Nishanth Aravamudan, IBM Corporation
- *
- * This library is free software; you can redistribute it and/or
- * modify it under the terms of the GNU Lesser General Public License
- * as published by the Free Software Foundation; either version 2.1 of
- * the License, or (at your option) any later version.
- *
- * This library is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * Lesser General Public License for more details.
- *
- * You should have received a copy of the GNU Lesser General Public
- * License along with this library; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
- */
-
-#define _GNU_SOURCE
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdint.h>
-#include <errno.h>
-#include <getopt.h>
-#include <signal.h>
-#include <time.h>
-#include <unistd.h>
-#include <fcntl.h>
-
-#include <sys/mman.h>
-#include <sys/poll.h>
-#include <sys/socket.h>
-#include <sys/stat.h>
-#include <sys/types.h>
-#include <sys/un.h>
-
-#include "libhugetlbfs_internal.h"
-#include "hugetlbfs.h"
-#include "hugetlbd.h"
-
-#define QUEUE_LENGTH SOMAXCONN
-#define HUGETLBD_PORT 60046 /* some random number */
-/* how often to reap due to idleness */
-#define DEFAULT_POLL_TIMEOUT 600000 /* milliseconds */
-/* how long ago does a share need to have occurred to reap? */
-#define DEFAULT_SHARE_TIMEOUT 600 /* seconds */
-
-struct shared_mapping {
- /* unique identifier */
- exe_ident_t id;
- /* filename */
- char path_to_file[PATH_MAX+1];
- /* virtual address of segments */
- vaddr_t vaddr;
- /* timestamp for file cleanup */
- time_t timestamp;
- /* linked-list pointer */
- struct shared_mapping *next;
-};
-
-static struct shared_mapping *head;
-
-/* visible for multiple manipulation points */
-static int sock;
-/* visible for SIGALRM handling */
-static int timeout;
-/* visible for error handling */
-static int shared_fd;
-static FILE *logfd;
-static unsigned int share_timeout;
-static unsigned int poll_timeout;
-
-int __hugetlbfs_verbose = 0;
-
-static void usage()
-{
- printf("Usage: hugetlbd [-v] [-V verbosity] [-p timeout1] [-s
timeout2]\n"
- "Start libhugetlbfs daemon for sharing program segments\n"
- " -v: display this help and exit\n"
- " -V: specify log level (1-3) [default=1]\n"
- " 1: only log errors\n"
- " 2: log errors and warnings\n"
- " 3: log all output\n"
- " -p: specify how long to wait before unlinking files in the
hugetlbfs\n"
- " mountpoint (seconds) [default=600]\n"
- " -s: specify when a file last needs to have been shared for
it not to\n"
- " be unlinked (seconds) [default=600]\n"
- "Report bugs to [EMAIL PROTECTED]");
-}
-
-/**
- * reap_files - unlink all shared files
- * @quitting: 1 if exiting daemon
- *
- * no return value
- */
-static void reap_files(int quitting)
-{
- struct shared_mapping *smptr, *nextsmptr;
- time_t now = time(NULL);
-
- DEBUG("Reaping files\n");
- smptr = head;
- /* list is sorted by timestamp, youngest is at head */
- while (smptr != NULL) {
- /*
- * Iterate through list until we get to an old enough
- * sharing
- */
- if (!quitting && (now - smptr->timestamp < share_timeout)) {
- smptr = smptr->next;
- continue;
- }
- /* keep pointer to the next one */
- nextsmptr = smptr->next;
- unlink(smptr->path_to_file);
- /* if we're deleting head, head has to move */
- if (smptr == head) {
- head = nextsmptr;
- }
- free(smptr);
- smptr = nextsmptr;
- }
-}
-
-/**
- * kill_daemon - exit the daemon process
- * @ret: if 0, expected quit
- * 1, unexpected quit
- * also functions as return value to exit with
- *
- * no return value
- */
-static void kill_daemon(int ret)
-{
- DEBUG("Killing daemon\n");
- reap_files(1);
- /*
- * Close the socket here, because we might dying due to an
- * asynchronous signal.
- */
- close(sock);
- exit(ret);
-}
-
-/**
- * signal_handler - respond to catchable signals
- * @signal: the signal to handle
- *
- * no return value
- */
-static void signal_handler(int signal)
-{
- switch (signal) {
- case SIGHUP:
- /* reap files */
- DEBUG("Caught SIGHUP\n");
- reap_files(1);
- break;
- case SIGINT:
- /* expectedly quit */
- kill_daemon(0);
- break;
- case SIGALRM:
- timeout = 1;
- break;
- }
-}
-
-/**
- * idcpy - copy id from src to dst
- * @src: source id
- * @dst: destination id
- *
- * Would need to be changed if exe_ident_t is changed to not be a basic
- * type.
- *
- * no return value
- */
-static void idcpy(exe_ident_t *dst, exe_ident_t *src)
-{
- *dst = *src;
-}
-
-/**
- * idcmp - compare id1 to id2
- * @id1: first id
- * @dst: second id
- *
- * Would need to be changed if exe_ident_t is changed to not be a basic
- * type.
- *
- * returns: 0, if ids match
- * 1, if ids don't match
- */
-static int idcmp(exe_ident_t *i1, exe_ident_t *i2)
-{
- if (*i1 == *i2)
- return 0;
- else
- return 1;
-}
-
-/**
- * set_path_to_file - create new unique file in hugetlbfs
- * @tgt: target character array to store filename in
- * @size: size of @tgt
- *
- * Creates a new uniquely named file in the hugetlbfs mount point. File
- * is opened in mkstemp, and the file descriptor is stored in a globally
- * visible variable, because the caller's return value is already
- * overloaded.
- *
- * Effectively identical to get_unlinked_fd().
- *
- * no return value
- */
-static void set_path_to_file(char *tgt, int size)
-{
- const char *path;
- shared_fd = -1;
-
- tgt[size - 1] = '\0';
-
- path = hugetlbfs_find_path();
- if (!path) {
- shared_fd = -1;
- return;
- }
-
- strcpy(tgt, path);
- strncat(tgt, "/libhugetlbfs.tmp.XXXXXX", size - 1);
-
- shared_fd = mkstemp(tgt);
- if (shared_fd > 0) {
- /* prevent access to shared file */
- int ret = chmod(tgt, 0);
- if (ret < 0) {
- DEBUG("chmod failed\n");
- shared_fd = -1;
- }
- }
-}
-
-/**
- * do_shared_file - find shareable mapping, or create new one
- * @req: contains executable id and virtual addresses of segments
- * @resp: contains filename and whether prepare() needs to be run
- *
- * The core routine for sharing mappings. The request and response
- * structures allow for accepting many parameters and returning many
- * response values independent of the actual return value, which acts
- * kind of like metadata about @resp.
- *
- * returns: -1, if failed to create mapping
- * NULL, if sharing mapping already at head of list
- * pointer to previous entry in linked list, else
- */
-static struct shared_mapping *do_shared_file(struct client_request *creq,
- struct daemon_response *dresp)
-{
- struct shared_mapping *smptr, *prevsmptr;
-
- if (head == NULL) {
- /* allocating first member of linked list */
- goto create_mapping;
- }
-
- prevsmptr = NULL;
- smptr = head;
-
- while (smptr != NULL) {
- if (!idcmp(&(smptr->id), &(creq->id))
- && (smptr->vaddr == creq->vaddr))
- /* found match */
- goto share_mapping;
-
- /*
- * Keep track of the previous entry in the list, because
- * we're going to need to move the one we're currently
- * at. This prevents an extra loop later.
- */
- prevsmptr = smptr;
- smptr = smptr->next;
- }
-
-create_mapping:
- /* first attempt to map this segment, create a new mapping */
- smptr = (struct shared_mapping *)malloc(sizeof(struct shared_mapping));
- if (smptr == NULL) {
- ERROR("Failed to allocate new shared mapping: %s\n",
- strerror(errno));
- return (struct shared_mapping *)-1;
- }
-
- set_path_to_file(smptr->path_to_file, sizeof(smptr->path_to_file));
- if (shared_fd < 0) {
- ERROR("Failed to create new shared file\n");
- free(smptr);
- return (struct shared_mapping *)-1;
- }
-
- DEBUG("created shared mapping to %s\n", smptr->path_to_file);
-
- /* grab data out of request */
- idcpy(&(smptr->id), &(creq->id));
- smptr->vaddr = creq->vaddr;
- /* timestamp will be set upon insert */
-
- /* client will need to prepare file */
- dresp->need_to_prepare = 1;
- strcpy(dresp->path_to_file, smptr->path_to_file);
-
- /*
- * Cannot be NULL. Caller is responsible for freeing if an error
- * occurs
- */
- return smptr;
-
-share_mapping:
- /* store data in response */
- dresp->need_to_prepare = 0;
- strcpy(dresp->path_to_file, smptr->path_to_file);
-
- DEBUG("sharing mapping to %s\n", smptr->path_to_file);
-
- /*
- * Can be NULL, if sharing at the head of the list. Caller does
- * not need to deal with freeing, but must deal with both NULL
- * and non-NULL cases.
- * We are moving smptr to the front of the list if the sharing
- * succeeds, but send prevsmptr back so that we don't need to
- * iterate to do the move.
- */
- return prevsmptr;
-}
-
-static int parse_args(int argc, char *argv[]) {
- int opt;
- int ret = 0;
- while (1) {
- opt = getopt(argc, argv, ":p:s:dV:v");
- if (opt == -1)
- break;
-
- switch (opt) {
- case 'p':
- /* needs to be in milliseconds */
- poll_timeout = 1000*atoi(optarg);
- break;
- case 's':
- share_timeout = atoi(optarg);
- break;
- case 'd':
- /* do not daemonize */
- ret = 1;
- break;
- case 'V':
- __hugetlbfs_verbose = atoi(optarg);
- break;
- case 'v':
- usage();
- return -1;
- case ':':
- printf("getopt: missing parameter for option
%c\n", optopt);
- usage();
- return -1;
- case '?':
- printf("getopt: unknown option: %c\n", optopt);
- usage();
- return -1;
- default:
- printf("getopt: returned character code
0%o\n", opt);
- usage();
- return -1;
- }
- }
-
- if (poll_timeout == 0) {
- poll_timeout = DEFAULT_POLL_TIMEOUT;
- }
- if (share_timeout == 0) {
- share_timeout = DEFAULT_SHARE_TIMEOUT;
- }
-
- return ret;
-}
-
-static void daemonize()
-{
- pid_t pid, sid;
-
- pid = fork();
- if (pid < 0) {
- ERROR("daemonize: fork() failed\n");
- exit(1);
- }
- if (pid > 0)
- exit(0);
-
- umask(0);
-
- sid = setsid();
- if (sid < 0) {
- ERROR("daemonize: Failed to set session ID\n");
- exit(1);
- }
-
- if ((chdir("/")) < 0) {
- ERROR("daemonize: Failed to change directory\n");
- exit(1);
- }
-
- logfd = fopen(LOGFILE, "a+");
- if (!logfd) {
- ERROR("daemonize: Failed to open log file %s\n", LOGFILE);
- exit(1);
- }
-
- close(STDIN_FILENO);
- close(STDOUT_FILENO);
- close(STDERR_FILENO);
- stderr = logfd;
-}
-
-/**
- * read_client_request - receive request from client
- * @local_sock: socket to read from
- * @cresp: pointer to store request in
- *
- * Wait for client's request for sharing
- *
- * returns 0, on success
- * negative values, on failure
- */
-static int read_client_request(int local_sock, struct client_request
- *creq)
-{
- int ret;
- alarm(60);
- ret = read(local_sock, creq, sizeof(*creq));
- /* cancel any alarms */
- alarm(0);
- if (ret < 0) {
- /* timeout == 1 means that the SIGALRM handler ran */
- if (timeout) {
- DEBUG("read timed out\n");
- return -1;
- }
- ERROR("read failed: %s\n", strerror(errno));
- return -2;
- }
- if (ret != sizeof(*creq)) {
- /*
- * This should never happen, but is useful for
- * debugging size issues with mixed environments
- */
- DEBUG("short first read (got %d, should have been "
- "%zu\n", ret, sizeof(*creq));
- return -1;
- }
- return 0;
-}
-
-/**
- * write_daemon_response - respond to client
- * @local_sock: socket to write to
- * @cresp: pointer to response
- *
- * Acknowledge a client's request with a filename and whether the caller
- * needs to prepare
- *
- * returns 0, on success
- * negative values, on failure
- */
-static int write_daemon_response(int local_sock, struct daemon_response
- *dresp)
-{
- int ret;
- alarm(60);
- ret = write(local_sock, dresp, sizeof(*dresp));
- alarm(0);
- /*
- * If dresp.need_to_prepare is set, then we know we
- * dynamically allocated an element (which is pointed to
- * by smptr). We need to backout any changes it made to
- * the system state (open file, memory consumption) on
- * errors.
- */
- if (ret < 0) {
- if (timeout) {
- DEBUG("write timed out\n");
- if (dresp->need_to_prepare)
- return -1;
- return -2;
- }
- ERROR("write failed: %s\n", strerror(errno));
- if (dresp->need_to_prepare)
- return -3;
- return -4;
- }
- if (ret != sizeof(*dresp)) {
- DEBUG("short write (got %d, should have been %zu\n",
- ret, sizeof(*dresp));
- if (dresp->need_to_prepare)
- return -1;
- return -2;
- } /* else successfully wrote */
- return 0;
-}
-
-/**
- * read_client_response - wait for a client's ACK
- * @local_sock: socket to read from
- * @dresp: pointer to daemon's previous reponse
- * @cresp: pointer to store response in
- *
- * Wait for notification that file is prepared and segments have been
- * remapped. We need @dresp so that we can backout on errors
- * appropriately.
- *
- * returns 0, on success
- * negative values, on failure
- */
-static int read_client_response(int local_sock, struct daemon_response
- *dresp, struct client_response *cresp)
-{
- int ret;
- DEBUG("waiting for client's response\n");
- /* this may be unnecessarily long */
- alarm(300);
- /*
- */
- ret = read(local_sock, cresp, sizeof(*cresp));
- alarm(0);
- if (ret < 0) {
- if (timeout) {
- DEBUG("read timed out\n");
- if (dresp->need_to_prepare)
- return -1;
- return -2;
- }
- ERROR("read failed: %s\n", strerror(errno));
- if (dresp->need_to_prepare)
- return -3;
- return -4;
- }
- if (ret != sizeof(*cresp)) {
- DEBUG("short second read (got %d, should have been "
- "%zu\n", ret, sizeof(*cresp));
- if (dresp->need_to_prepare)
- return -1;
- return -2;
- }
- return 0;
-}
-
-/**
- * sharing_control_loop - main event loop for sharing
- * @sock: socket to wait on for connections
- *
- * This is expected to be an infinite loop. If we we ever return, it is
- * a fail case.
- *
- * returns 1
- */
-static int sharing_control_loop(int sock)
-{
- int ret;
- struct sockaddr_un from;
- socklen_t len = (socklen_t)sizeof(from);
- /*
- * Needed for socket messages -- could switch to dynamic
- * allocation
- */
- struct client_request creq;
- struct daemon_response dresp;
- struct client_response cresp;
- int local_sock;
- /* handle on some state that may need to be freed */
- struct shared_mapping *smptr;
- mode_t mode;
- /* needed for poll */
- struct pollfd ufds;
- /* used to allow for handling SIGALRM and SIGHUP */
- struct sigaction sigact = {
- .sa_handler = signal_handler,
- .sa_flags = 0,
- };
- struct sigaction prevact;
-
- sigemptyset(&(sigact.sa_mask));
-
- /* We will poll on sock for any incoming connections */
- ufds.fd = sock;
- ufds.events = POLLIN;
-
- /* install our custom signal-handler */
- sigaction(SIGINT, &sigact, NULL);
-
- for (;;) {
- /* gets set by SIGALRM handler */
- timeout = 0;
- /* not sure if this is necessary, but ok */
- ufds.revents = 0;
- /* helps with error handling */
- dresp.need_to_prepare = 0;
- /* reset mode of shared file */
- mode = 0;
-
- sigaction(SIGHUP, &sigact, &prevact);
-
-do_poll:
- ret = poll(&ufds, 1, poll_timeout);
- if (ret < 0) {
- /*
- * catch SIGHUP as non-fail case
- * SIGINT will have made us exit already
- */
- if (errno == EINTR)
- goto do_poll;
- ERROR("poll() failed: %s\n", strerror(errno));
- goto die;
- } else if (ret == 0) {
- /* poll timeout, reap any old files */
- DEBUG("poll() timed out\n");
- reap_files(0);
- continue;
- }
-
- switch (ufds.revents) {
- case POLLERR:
- ERROR("poll returned a POLLERR event\n");
- goto die;
- break;
- case POLLNVAL:
- ERROR("poll returned a POLLNVAL event\n");
- goto die;
- break;
- case POLLHUP:
- ERROR("poll returned a POLLHUP event\n");
- goto die;
- break;
- case POLLIN:
- break;
- default:
- ERROR("poll() returned an event we did not
expect: %d\n", ufds.revents);
- goto die;
- }
-
- local_sock = accept(sock, (struct sockaddr *)(&from), &len);
- if (local_sock < 0) {
- if (errno == ECONNABORTED) {
- /*
- * I think this is ok as a non-failure
- * case, but not positive, since the
- * client is what failed, not us.
- */
- DEBUG("accept() returned connection aborted\n");
- continue;
- }
- /*
- * Don't fail on a SIGHUP
- */
- if (errno == EINTR)
- goto do_poll;
- ERROR("accept() failed: %s\n", strerror(errno));
- goto die;
- }
-
- /*
- * Disable the races due to removing files that may be
- * sent to clients for sharing by just not handling
- * SIGHUP for a bit
- */
- sigaction(SIGHUP, &prevact, NULL);
- sigaction(SIGALRM, &sigact, NULL);
-
- ret = read_client_request(local_sock, &creq);
- switch (ret) {
- case 0:
- break;
- case -1:
- goto next;
- case -2:
- goto close_and_die;
- }
-
- /* Use the request to figure out the mapping ... */
- smptr = do_shared_file(&creq, &dresp);
- if (smptr == (struct shared_mapping *)-1) {
- ERROR("do_shared_file failed\n");
- /*
- * Just because we didn't get a hugetlbfs file,
- * doesn't mean we should fail outright. For
- * instance, if mkstemp() failed for some reason
- * we should still continue sharing any
- * previously shared segments.
- */
- goto next;
- }
-
- mode = S_IRUSR | S_IRGRP | S_IROTH;
- /* only allow preparer to write to file */
- if (dresp.need_to_prepare)
- mode |= S_IWUSR | S_IWGRP | S_IWOTH;
- ret = chmod(dresp.path_to_file, mode);
- if (ret < 0) {
- DEBUG("chmod(%s) to read (maybe writable) failed:
%s\n", dresp.path_to_file, strerror(errno));
- if (dresp.need_to_prepare)
- goto failed_next;
- goto next;
- }
-
- ret = write_daemon_response(local_sock, &dresp);
- switch (ret) {
- case 0:
- break;
- case -1:
- goto failed_next;
- case -2:
- goto next;
- case -3:
- goto failed_close_and_die;
- case -4:
- goto failed_next;
- }
-
- ret = read_client_response(local_sock, &dresp, &cresp);
- switch (ret) {
- case 0:
- break;
- case -1:
- goto failed_next;
- case -2:
- goto next;
- case -3:
- goto failed_close_and_die;
- case -4:
- goto failed_next;
- }
-
- /*
- * Determine if the segments were prepared/remapped as
- * expected.
- */
- if (cresp.succeeded != 0) {
- DEBUG("file not prepared\n");
- if (dresp.need_to_prepare)
- goto failed_next;
- goto next;
- }
-
- DEBUG("File prepared / opened successfully\n");
- if (dresp.need_to_prepare) {
- mode = S_IRUSR | S_IRGRP | S_IROTH;
- ret = chmod(dresp.path_to_file, mode);
- if (ret < 0) {
- DEBUG("chmod(%s) to read-only failed: %s\n",
dresp.path_to_file, strerror(errno));
- goto failed_next;
- }
- /*
- * Insert new element at head of list
- */
- if (head != NULL)
- smptr->next = head;
- else
- smptr->next = NULL;
- head = smptr;
-
- close(shared_fd);
- } else {
- /*
- * Move element to head of list. We have pointer
- * to the element just before the element to
- * move. If it is NULL, then we are sharing at
- * the head of the list, and no moving is
- * necessary. Else, shift the pointers around.
- * We obviously also know that head != NULL.
- */
- if (smptr != NULL) {
- struct shared_mapping *temp;
- temp = smptr->next;
- smptr->next = temp->next;
- temp->next = head;
- head = temp;
- }
- }
- /*
- * Shared mapping is at the head of the list now, update
- * timestamp.
- */
- head->timestamp = time(NULL);
-
- goto next;
-
-failed_next:
- close(shared_fd);
- unlink(dresp.path_to_file);
- free(smptr);
-next:
- close(local_sock);
- }
-
-failed_close_and_die:
- close(shared_fd);
- unlink(dresp.path_to_file);
- free(smptr);
-close_and_die:
- close(local_sock);
-die:
- kill_daemon(1);
-
- /* indicate failure if we ever get this far */
- return 1;
-}
-
-/**
- * main - set up socket and enter central control loop
- * @argc: number of command line arguments
- * @argv: command line arguments
- *
- * returns: 0, on success
- * -1, on failure
- */
-int main(int argc, char *argv[])
-{
- int ret;
- /* local socket structures */
- struct sockaddr_un sun;
-
- ret = parse_args(argc, argv);
- if (ret < 0)
- exit(1);
- if (ret == 0)
- /* Perform all the steps to become a real daemon */
- daemonize();
-
- sock = socket(AF_UNIX, SOCK_STREAM, 0);
- if (sock < 0) {
- ERROR("socket() failed: %s\n", strerror(errno));
- return -1;
- }
-
- sun.sun_family = AF_UNIX;
- /* clear out any previous socket */
- unlink("/tmp/libhugetlbfs-sock");
- strcpy(sun.sun_path, "/tmp/libhugetlbfs-sock");
- ret = bind(sock, (struct sockaddr *)(&sun), sizeof(sun));
- if (ret < 0) {
- ERROR("bind() failed: %s\n", strerror(errno));
- goto die;
- }
-
- chmod("/tmp/libhugetlbfs-sock", 0666);
-
- ret = listen(sock, QUEUE_LENGTH);
- if (ret < 0) {
- ERROR("listen() failed: %s\n", strerror(errno));
- goto die;
- }
-
- return sharing_control_loop(sock);
-
-die:
- kill_daemon(1);
-
- /* indicate failure if we ever get this far */
- return 1;
-}
Index: libhugetlbfs/hugetlbd.h
===================================================================
--- libhugetlbfs.orig/hugetlbd.h 2006-10-27 17:14:39.000000000 +1000
+++ /dev/null 1970-01-01 00:00:00.000000000 +0000
@@ -1,60 +0,0 @@
-/*
- * libhugetlbfs - Easy use of Linux hugepages
- * Copyright (C) 2006 Nishanth Aravamudan, IBM Corporation.
- *
- * This library is free software; you can redistribute it and/or
- * modify it under the terms of the GNU Lesser General Public License
- * as published by the Free Software Foundation; either version 2.1 of
- * the License, or (at your option) any later version.
- *
- * This library is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * Lesser General Public License for more details.
- *
- * You should have received a copy of the GNU Lesser General Public
- * License along with this library; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
- */
-
-#ifndef _HUGETLBD_H
-#define _HUGETLBD_H
-
-#ifndef __LIBHUGETLBFS__
-#error This header should not be included by library users.
-#endif /* __LIBHUGETLBFS__ */
-
-#define ID_KEY 0x56
-#define LOGFILE "/tmp/hugetlbd.log"
-
-/*
- * Ideally, would like to deal with this better, so that a 32-bit daemon
- * only deals in 32-bit addresses. But, for now, this makes 32-bit and
- * 64-bit apps deal ok with the socket messages.
- */
-typedef uint64_t vaddr_t;
-typedef uint64_t exe_ident_t;
-
-struct client_request {
- /*
- * unique identifer
- * if id == 0, then the request is actually a response that the
- * file is prepared
- */
- exe_ident_t id;
- /*
- * to identify the segment
- */
- vaddr_t vaddr;
-};
-
-struct client_response {
- int succeeded;
-};
-
-struct daemon_response {
- int need_to_prepare;
- char path_to_file[PATH_MAX+1];
-};
-
-#endif /* _HUGETLBD_H */
Index: libhugetlbfs/hugeutils.c
===================================================================
--- libhugetlbfs.orig/hugeutils.c 2006-10-27 17:14:50.000000000 +1000
+++ libhugetlbfs/hugeutils.c 2006-10-27 17:14:51.000000000 +1000
@@ -36,18 +36,14 @@
#include <sys/vfs.h>
#include <sys/statfs.h>
#include <sys/types.h>
-#include <sys/ipc.h>
-#include <sys/socket.h>
-#include <sys/un.h>
#include <sys/mman.h>
+#include <sys/file.h>
#include "libhugetlbfs_internal.h"
#include "hugetlbfs.h"
-#include "hugetlbd.h"
static int hpage_size; /* = 0 */
static char htlb_mount[PATH_MAX+1]; /* = 0 */
-int sharing; /* =0 */
/********************************************************************/
/* Internal functions */
@@ -247,297 +243,6 @@ int hugetlbfs_unlinked_fd(void)
return fd;
}
-/**
- * create_id - create unique id for current executable
- *
- * Uses IPC ftok() to generate a roughly unique identifier for our own
- * executable. This is used to match shareable mappings.
- *
- * returns: -1, on error
- * a unique identifier, otherwise
- */
-static exe_ident_t create_id(void)
-{
- exe_ident_t id;
- int ret;
- char my_exe[PATH_MAX];
-
- ret = readlink("/proc/self/exe", my_exe, PATH_MAX);
- if (ret < 0) {
- ERROR("readlink failed: %s\n", strerror(errno));
- return -1;
- }
-
- id = (exe_ident_t)ftok(my_exe, ID_KEY);
- if (id < 0) {
- ERROR("ftok failed: %s\n", strerror(errno));
- return -1;
- }
-
- return id;
-}
-
-static int sock;
-static int timeout;
-/* global to allow multiple callsites to set members */
-
-static void signal_handler(int signal)
-{
- switch (signal) {
- case SIGALRM:
- timeout = 1;
- }
-}
-
-static int client_complete(struct seg_info *htlb_seg_info, int success);
-
-/**
- * hugetlbfs_shared_file - communicate with daemon to get one shareable file
- * @htlb_seg_info: pointer to program's segment data
- *
- * returns:
- * -1, on failure
- * 0, on success, and caller does not need to prepare the segments
- * 1, on success, and caller does need to prepare the segments
- */
-static int hugetlbfs_shared_file(struct seg_info *htlb_seg_info)
-{
- int ret, fd;
- struct client_request creq;
- struct daemon_response dresp;
- struct sockaddr_un sun;
- struct sigaction sigact = {
- .sa_handler = signal_handler,
- .sa_flags = 0,
- };
- struct sigaction prev;
- int open_flags;
-
- sock = socket(AF_UNIX, SOCK_STREAM, 0);
- if (sock < 0) {
- ERROR("socket() failed: %s\n", strerror(errno));
- return -1;
- }
-
- sun.sun_family = AF_UNIX;
- strcpy(sun.sun_path, "/tmp/libhugetlbfs-sock");
- ret = connect(sock, &sun, sizeof(sun));
- if (ret < 0) {
- ERROR("connect() failed: %s\n", strerror(errno));
- return -1;
- }
-
- sigemptyset(&(sigact.sa_mask));
-
- sigaction(SIGALRM, &sigact, &prev);
-
- /* add data to request */
- creq.id = create_id();
- /*
- * This results in (non-harmful) compilation warning for
- * 32-bit applications
- */
- creq.vaddr = (vaddr_t)htlb_seg_info->vaddr;
-
- alarm(60);
- ret = write(sock, &creq, sizeof(creq));
- /* cancel any alarms */
- alarm(0);
- if (ret < 0) {
- if (timeout) {
- DEBUG("write timed out\n");
- goto fail;
- } else {
- ERROR("client request write failed: %s\n",
- strerror(errno));
- goto fail;
- }
- } else if (ret != sizeof(creq)) {
- DEBUG("short write (got %d, should have been %zu)\n",
- ret, sizeof(creq));
- goto fail;
- } /* else successfully wrote */
-
- /* wait slightly longer for the response */
- alarm(120);
- ret = read(sock, &dresp, sizeof(dresp));
- alarm(0);
- if (ret < 0) {
- if (timeout) {
- DEBUG("read timed out\n");
- goto fail;
- } else {
- ERROR("read failed: %s\n", strerror(errno));
- goto fail;
- }
- } else if (ret != sizeof(dresp)) {
- DEBUG("short read (got %d, should have been %zu)\n",
- ret, sizeof(dresp));
- goto fail;
- } /* else successfully read */
-
- sigaction(SIGALRM, &prev, NULL);
-
- /* Convert daemon response to open mode bits */
- if (dresp.need_to_prepare) {
- open_flags = O_RDWR;
- } else {
- open_flags = O_RDONLY;
- }
-
- /* get local handle on file */
- fd = open(dresp.path_to_file, open_flags);
- if (fd < 0) {
- ERROR("open(%s) failed: %s\n", dresp.path_to_file,
- strerror(errno));
- goto fail;
- }
-
- if (dresp.need_to_prepare) {
- DEBUG("daemon said to prepare\n");
- } else {
- /* ACK right away (closes socket) */
- DEBUG("daemon said not to prepare\n");
- ret = client_complete(htlb_seg_info, 0);
- if (ret < 0)
- goto fail;
- }
-
- htlb_seg_info->fd = fd;
-
- /* leave socket open until finished_prepare is called */
- return dresp.need_to_prepare;
-
-fail:
- close(sock);
- return -1;
-}
-
-/**
- * client_complete - send ACK to daemon that we no longer need file
- * @htlb_seg_info: pointer to program's segment data
- * @success: if 0, successful prepare, else failed
- *
- * returns: 0, if successful transmission (none occurs if
- * not sharing)
- * -1, if failed transmission
- */
-static int client_complete(struct seg_info *htlb_seg_info, int success)
-{
- struct sigaction sigact = {
- .sa_handler = signal_handler,
- .sa_flags = 0,
- };
- struct sigaction prev;
- struct client_response cresp;
- int ret;
-
- sigemptyset(&(sigact.sa_mask));
-
- sigaction(SIGALRM, &sigact, &prev);
-
- cresp.succeeded = success;
-
- alarm(60);
- ret = write(sock, &cresp, sizeof(cresp));
- alarm(0);
- if (ret < 0) {
- if (timeout) {
- DEBUG("write timed out\n");
- goto fail;
- } else {
- ERROR("client response write failed: %s\n",
- strerror(errno));
- goto fail;
- }
- } else if (ret != sizeof(cresp)) {
- DEBUG("short write (got %d, should have been %zu)\n",
- ret, sizeof(cresp));
- goto fail;
- } /* else successfully wrote */
-
- sigaction(SIGALRM, &prev, NULL);
-
- if (success < 0)
- /* back out open'd fd since we failed */
- close(htlb_seg_info->fd);
-
- close(sock);
- return 0;
-
-fail:
- close(htlb_seg_info->fd);
- close(sock);
- return -1;
-}
-
-/**
- * finished_prepare - callback for ACK'ing prepare is done
- * @htlb_seg_info: pointer to program's segment data
- * @success: if 0, successful prepare, else failed
- *
- * returns: 0, if successful transmission (none occurs if
- * not sharing)
- * -1, if failed transmission
- */
-int finished_prepare(struct seg_info *htlb_seg_info, int success)
-{
- if (sharing)
- return client_complete(htlb_seg_info, success);
- return 0;
-}
-
-/**
- * hugetlbfs_set_fds - multiplex callers depending on if sharing or not
- * @htlb_seg_info: pointer to program's segment data
- *
- * returns:
- * -1, on error
- * 0, on success, caller expects and gets shared, does not need to prepare
- * 1, on success, caller expects and gets shared, does need to prepare
- * 2, on success, caller expects shared and gets unlinked, does need to
prepare
- * 3, on success, caller expects and gets unlinked, does need to prepare
- */
-int hugetlbfs_set_fd(struct seg_info *htlb_seg_info)
-{
- char *env;
- int fd;
- /* assume unlinked, all unlinked files need to be prepared */
- int ret = 3;
-
- env = getenv("HUGETLB_SHARE");
- if (env)
- sharing = atoi(env);
- DEBUG("HUGETLB_SHARE=%d, sharing ", sharing);
- if (sharing == 2) {
- DEBUG_CONT("enabled for all segments\n");
- } else {
- if (sharing == 1) {
- DEBUG_CONT("enabled for only read-only segments\n");
- } else {
- DEBUG_CONT("disabled\n");
- }
- }
-
- /* Either share all segments or share only read-only segments */
- if ((sharing == 2) || ((sharing == 1) &&
- !(htlb_seg_info->prot & PROT_WRITE))) {
- /* first, try to share */
- ret = hugetlbfs_shared_file(htlb_seg_info);
- if (ret >= 0)
- return ret;
- /* but, fall through to unlinked files, if sharing fails */
- ret = 2;
- DEBUG("Falling back to unlinked files\n");
- }
- fd = hugetlbfs_unlinked_fd();
- if (fd < 0)
- return -1;
- htlb_seg_info->fd = fd;
- return ret;
-}
-
-
/********************************************************************/
/* Library user visible DIAGNOSES/DEBUGGING ONLY functions */
/********************************************************************/
Index: libhugetlbfs/libhugetlbfs_internal.h
===================================================================
--- libhugetlbfs.orig/libhugetlbfs_internal.h 2006-10-27 17:14:39.000000000
+1000
+++ libhugetlbfs/libhugetlbfs_internal.h 2006-10-27 17:14:51.000000000
+1000
@@ -30,8 +30,6 @@
extern int __hugetlbfs_verbose;
-extern int sharing;
-
#define ERROR(...) \
if (__hugetlbfs_verbose >= 1) \
fprintf(stderr, "libhugetlbfs: ERROR: " __VA_ARGS__)
@@ -48,16 +46,6 @@ extern int sharing;
if (__hugetlbfs_verbose >= 3) \
fprintf(stderr, __VA_ARGS__)
-struct seg_info {
- void *vaddr;
- unsigned long filesz, memsz;
- int prot;
- int fd;
-};
-
-int finished_prepare(struct seg_info *htlb_seg_info, int success);
-int hugetlbfs_set_fd(struct seg_info *htlb_seg_info);
-
#if defined(__powerpc64__) && !defined(__LP64__)
/* Older binutils fail to provide this symbol */
#define __LP64__
Index: libhugetlbfs/tests/run_tests.sh
===================================================================
--- libhugetlbfs.orig/tests/run_tests.sh 2006-10-27 17:14:50.000000000
+1000
+++ libhugetlbfs/tests/run_tests.sh 2006-10-27 17:14:51.000000000 +1000
@@ -13,6 +13,14 @@ function free_hpages() {
echo "$H"
}
+function hugetlbfs_path() {
+ if [ -n "$HUGETLB_PATH" ]; then
+ echo "$HUGETLB_PATH"
+ else
+ grep hugetlbfs /proc/mounts | cut -f2 -d' '
+ fi
+}
+
TOTAL_HPAGES=$(grep 'HugePages_Total:' /proc/meminfo | cut -f2 -d:)
[ -z "$TOTAL_HPAGES" ] && TOTAL_HPAGES=0
HPAGE_SIZE=$(grep 'Hugepagesize:' /proc/meminfo | awk '{print $2}')
@@ -64,14 +72,12 @@ elfshare_test () {
baseprog="${args[$N]}"
unset args[$N]
set -- "[EMAIL PROTECTED]"
+ # Run each elfshare test invocation independently - clean up the
+ # sharefiles before and after:
NUM_THREADS=2
- killall -HUP hugetlbd
run_test HUGETLB_SHARE=2 "$@" "xB.$baseprog" $NUM_THREADS
- killall -HUP hugetlbd
run_test HUGETLB_SHARE=1 "$@" "xB.$baseprog" $NUM_THREADS
- killall -HUP hugetlbd
run_test HUGETLB_SHARE=2 "$@" "xBDT.$baseprog" $NUM_THREADS
- killall -HUP hugetlbd
run_test HUGETLB_SHARE=1 "$@" "xBDT.$baseprog" $NUM_THREADS
}
@@ -131,21 +137,11 @@ functional_tests () {
elflink_test linkhuge
# Sharing tests
- # stop all running instances for clean testing
- killall -INT hugetlbd
- # start the daemon in the bg
- PATH=../obj32:../obj64:$PATH hugetlbd
- # XXX: Wait for daemon to start
- sleep 5
elfshare_test linkshare
- # stop our instance of the daemon
- killall -INT hugetlbd
# Accounting bug tests
# reset free hpages because sharing will have held some
# alternatively, use
-# killall -HUP hugetlbd
-# to make the sharing daemon give up the files
run_test chunk-overcommit `free_hpages`
run_test alloc-instantiate-race `free_hpages`
run_test truncate_reserve_wraparound
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Libhugetlbfs-devel mailing list
Libhugetlbfs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libhugetlbfs-devel
