On 13.10.2006 [12:47:25 -0700], Nishanth Aravamudan wrote:
> On 13.10.2006 [12:02:44 +1000], David Gibson wrote:
> > Nish, or someone else who understands the sharing implementation:
> >
> > Can you explain to me the need for the hugetlbd? i.e. why is it
> > necessary to have a daemon to manage the shared mappings, rather than
> > just using a naming convention within the hugetlbfs to allow the
> > libhugetlbfs instances to directly find the right shared file?
>
> This is a very good question, and one I don't have the best answer to
> right now. As I thought of various reasons that using a naming
> convention *wouldn't* work, I immediately came up with ways to work
> around issues.
>
> Let me work up a patch that might get rid of *a lot* of code and see
> if I can't test it a bit. Give me a few days, though.
Hrm, some Fridays are better than others, apparently.
Description: Remove all of the daemonized sharing code and replace it
with a filename-based approach. The general flow of the new sharing code
is now:
1) First sharer (aka preparer) will open() the hugetlbfs file with
O_EXCL|O_CREAT. All other sharers will fail this open and open the file
O_RDONLY only.
2) The preparer will then LOCK_EX flock() the file, to cause the other
sharers to block on their flock() calls, as they try to use LOCK_SH.
3) The preparer then prepares the file, that is it copies the data as
before. If that fails, the preparer unlinks the file.
4) Regardless of the success of the prepare, the preparer will LOCK_SH
and LOCK_UN the file, to trigger all the other sharers to continue. If
the prepare failed, all the sharers will try to access() the file and
fail, and fall back to unlinked files. If the file does exist, though,
it is assumed to be safe to use as a copy of the segment.
This seems noticeably faster than the hugetlbd method. The files in the
hugetlbfs mount point are uniquely named as:
$(executable name)_$(gid of user)_$(word size)_$(program header)
Thus, sharing is only possible between members of the same group
(security concern), and 32-bit executables won't share the same file as
64-bit executables. The program header is what differentiates between
multiple segments of the same executable.
Compile and run-tested via make func on ppc64 and x86_64.
This is still an RFC, though, as I'm not sure of a good way to remove
the files from the hugetlbfs mount point, without having effectively a
daemon very similar to hugetlbd, but whose only job is to track atimes
of files in the hugetlbfs mount point and unlink "old" ones (as
specified at daemon start time). If that seems like a reasonable
approach, I will modify hugetlbd accordingly.
Many thanks to David Gibson for the original idea and Adam Litke for
discussing the implementation.
Signed-off-by: Nishanth Aravamudan <[EMAIL PROTECTED]>
---
HOWTO | 56 +--
Makefile | 14
elflink.c | 10
hugetlbd.c | 872 ------------------------------------------------
hugetlbd.h | 60 ---
hugeutils.c | 319 ++++++-----------
libhugetlbfs_internal.h | 3
tests/run_tests.sh | 14
8 files changed, 155 insertions(+), 1193 deletions(-)
diff --git a/HOWTO b/HOWTO
index 2c0c47a..7e8188d 100644
--- a/HOWTO
+++ b/HOWTO
@@ -323,42 +323,26 @@ libhugetlbfs:
Sharing remapped segments:
--------------------------
-Sharing of program segments among processes is a two-fold process.
-
-First, a system-wide daemon must be started, hugetlbd. The executable
-(which will be installed into $PREFIX/sbin by make install) should be
-run as root and will automatically spawn a child process to wait for
-segment sharing into the background. The process will log any output
-requested to /tmp/hugetlbd.log. The complete usage for hugetlbd is:
-
-hugetlbd [-v] [-V verbosity] [-p timeout1] [-s timeout2]
- -v: display this help and exit
- -V: specify log level (1-3) [default=1]
- 1: only log errors
- 2: log errors and warnings
- 3: log all output
- -p: specify how long to wait before unlinking files in the hugetlbfs
- mountpoint (seconds) [default=600]
- -s: specify when a file last needs to have been shared for it not to be
- unlinked (seconds) [default=600]
-
-Second, in the applications that wish to share segments, the
-HUGETLB_SHARE environment variable must be set appropriately. The
-variable takes on three values: 0, the default, indicates no segments
-should be shared; 1 indicates that only read-only segments (text) should
-be shared; and 2 indicates that all segments should be shared. Note, as
-this is a per-process variable, it is important to keep its usage
-consistent with HUGETLB_MINIMAL_COPY. If MINIMAL_COPY is used by the
-first process to share a particular executable's segments, it should be
-used by all subsequent ones, to guarantee no failures while copying
-segments. Similarly, if MINIMAL_COPY is not used by the first process
-sharing, it should not be used by subsequent ones.
-
-Having taken these two steps, the same application run twice with the
-same environment variable settings will use the same file both times
-(reducing the overall hugetlbfs file usage). If, for any reason, segment
-sharing is requested but unavailable, the library will fallback to
-unlinked files.
+Sharing of program segments among processes is a simple process. For
+the applications that wish to share segments, the HUGETLB_SHARE
+environment variable must be set appropriately. The variable takes on
+three values: 0, the default, indicates no segments should be shared;
+1 indicates that only read-only segments (text) should be shared; and
+2 indicates that all segments should be shared. Note, as this is a
+per-process variable, it is important to keep its usage consistent
+with HUGETLB_MINIMAL_COPY. If MINIMAL_COPY is used by the first
+process to share a particular executable's segments, it should be used
+by all subsequent ones, to guarantee no failures while copying
+segments. Similarly, if MINIMAL_COPY is not used by the first process
+sharing, it should not be used by subsequent ones. Additionally, as a
+security precaution, sharing is only possible between users in the
+same group.
+
+Now, the same application run twice with the same environment variable
+settings will use the same file both times (reducing the overall
+hugetlbfs file usage). If, for any reason, segment sharing is
+requested but unavailable, the library will fallback to unlinked
+files.
Examples
========
diff --git a/Makefile b/Makefile
index 3868dea..567aba7 100644
--- a/Makefile
+++ b/Makefile
@@ -2,7 +2,6 @@ PREFIX = /usr/local
BASEOBJS = hugeutils.o version.o
LIBOBJS = $(BASEOBJS) elflink.o morecore.o debug.o
-SBINOBJS = hugetlbd
INSTALL_OBJ_LIBS = libhugetlbfs.so libhugetlbfs.a
LDSCRIPT_TYPES = B BDT
INSTALL_OBJSCRIPT = ld.hugetlbfs
@@ -89,14 +88,13 @@ endif
DEPFILES = $(LIBOBJS:%.o=%.d)
-all: libs sbin tests
+all: libs tests
.PHONY: tests libs
libs: $(foreach file,$(INSTALL_OBJ_LIBS),$(OBJDIRS:%=%/$(file)))
-sbin: $(foreach file,$(SBINOBJS),$(OBJDIRS:%=%/$(file)))
-tests: libs sbin # Force make to build the library first
+tests: libs # Force make to build the library first
tests: tests/all
tests/%:
@@ -173,14 +171,6 @@ obj64/%.s: %.c
@$(VECHO) CC64 -S $@
$(CC64) $(CPPFLAGS) $(CFLAGS) -o $@ -S $<
-obj32/hugetlbd: hugetlbd.c $(BASEOBJS:%=obj32/%)
- @$(VECHO) CC32 $@
- $(CC32) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS32) -o $@ $^
-
-obj64/hugetlbd: hugetlbd.c $(BASEOBJS:%=obj64/%)
- @$(VECHO) CC64 $@
- $(CC64) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS64) -o $@ $^
-
clean:
@$(VECHO) CLEAN
rm -f *~ *.o *.so *.a *.d *.i core a.out $(VERSION)
diff --git a/elflink.c b/elflink.c
index 41a1fe9..6196ba3 100644
--- a/elflink.c
+++ b/elflink.c
@@ -231,6 +231,7 @@ static void parse_phdrs(Elf_Ehdr *ehdr)
htlb_seg_table[htlb_num_segs].filesz = filesz;
htlb_seg_table[htlb_num_segs].memsz = memsz;
htlb_seg_table[htlb_num_segs].prot = prot;
+ htlb_seg_table[htlb_num_segs].phdr = i;
htlb_num_segs++;
}
}
@@ -503,17 +504,16 @@ static int maybe_prepare(int fd_state, s
ret = prepare_segment(seg);
if (ret < 0) {
- /* notify daemon of failed prepare */
DEBUG("Failed to prepare segment\n");
- finished_prepare(seg, -1);
+ finished_prepare(seg);
return -1;
}
DEBUG("Prepare succeeded\n");
if (reply) {
- ret = finished_prepare(seg, 0);
+ ret = finished_prepare(seg);
if (ret < 0)
- DEBUG("Failed to communicate successful "
- "prepare to hugetlbd\n");
+ DEBUG("Failed to successfully unlock the "
+ "hugetlbfs file\n");
}
return ret;
}
diff --git a/hugetlbd.c b/hugetlbd.c
deleted file mode 100644
index 72bf251..0000000
--- a/hugetlbd.c
+++ /dev/null
@@ -1,872 +0,0 @@
-/*
- * libhugetlbfs - Easy use of Linux hugepages
- * Copyright (C) 2006 Nishanth Aravamudan, IBM Corporation
- *
- * This library is free software; you can redistribute it and/or
- * modify it under the terms of the GNU Lesser General Public License
- * as published by the Free Software Foundation; either version 2.1 of
- * the License, or (at your option) any later version.
- *
- * This library is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * Lesser General Public License for more details.
- *
- * You should have received a copy of the GNU Lesser General Public
- * License along with this library; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
- */
-
-#define _GNU_SOURCE
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdint.h>
-#include <errno.h>
-#include <getopt.h>
-#include <signal.h>
-#include <time.h>
-#include <unistd.h>
-#include <fcntl.h>
-
-#include <sys/mman.h>
-#include <sys/poll.h>
-#include <sys/socket.h>
-#include <sys/stat.h>
-#include <sys/types.h>
-#include <sys/un.h>
-
-#include "libhugetlbfs_internal.h"
-#include "hugetlbfs.h"
-#include "hugetlbd.h"
-
-#define QUEUE_LENGTH SOMAXCONN
-#define HUGETLBD_PORT 60046 /* some random number */
-/* how often to reap due to idleness */
-#define DEFAULT_POLL_TIMEOUT 600000 /* milliseconds */
-/* how long ago does a share need to have occurred to reap? */
-#define DEFAULT_SHARE_TIMEOUT 600 /* seconds */
-
-struct shared_mapping {
- /* unique identifier */
- exe_ident_t id;
- /* filename */
- char path_to_file[PATH_MAX+1];
- /* virtual address of segments */
- vaddr_t vaddr;
- /* timestamp for file cleanup */
- time_t timestamp;
- /* linked-list pointer */
- struct shared_mapping *next;
-};
-
-static struct shared_mapping *head;
-
-/* visible for multiple manipulation points */
-static int sock;
-/* visible for SIGALRM handling */
-static int timeout;
-/* visible for error handling */
-static int shared_fd;
-static FILE *logfd;
-static unsigned int share_timeout;
-static unsigned int poll_timeout;
-
-int __hugetlbfs_verbose = 0;
-
-static void usage()
-{
- printf("Usage: hugetlbd [-v] [-V verbosity] [-p timeout1] [-s
timeout2]\n"
- "Start libhugetlbfs daemon for sharing program segments\n"
- " -v: display this help and exit\n"
- " -V: specify log level (1-3) [default=1]\n"
- " 1: only log errors\n"
- " 2: log errors and warnings\n"
- " 3: log all output\n"
- " -p: specify how long to wait before unlinking files in the
hugetlbfs\n"
- " mountpoint (seconds) [default=600]\n"
- " -s: specify when a file last needs to have been shared for
it not to\n"
- " be unlinked (seconds) [default=600]\n"
- "Report bugs to [EMAIL PROTECTED]");
-}
-
-/**
- * reap_files - unlink all shared files
- * @quitting: 1 if exiting daemon
- *
- * no return value
- */
-static void reap_files(int quitting)
-{
- struct shared_mapping *smptr, *nextsmptr;
- time_t now = time(NULL);
-
- DEBUG("Reaping files\n");
- smptr = head;
- /* list is sorted by timestamp, youngest is at head */
- while (smptr != NULL) {
- /*
- * Iterate through list until we get to an old enough
- * sharing
- */
- if (!quitting && (now - smptr->timestamp < share_timeout)) {
- smptr = smptr->next;
- continue;
- }
- /* keep pointer to the next one */
- nextsmptr = smptr->next;
- unlink(smptr->path_to_file);
- /* if we're deleting head, head has to move */
- if (smptr == head) {
- head = nextsmptr;
- }
- free(smptr);
- smptr = nextsmptr;
- }
-}
-
-/**
- * kill_daemon - exit the daemon process
- * @ret: if 0, expected quit
- * 1, unexpected quit
- * also functions as return value to exit with
- *
- * no return value
- */
-static void kill_daemon(int ret)
-{
- DEBUG("Killing daemon\n");
- reap_files(1);
- /*
- * Close the socket here, because we might dying due to an
- * asynchronous signal.
- */
- close(sock);
- exit(ret);
-}
-
-/**
- * signal_handler - respond to catchable signals
- * @signal: the signal to handle
- *
- * no return value
- */
-static void signal_handler(int signal)
-{
- switch (signal) {
- case SIGHUP:
- /* reap files */
- DEBUG("Caught SIGHUP\n");
- reap_files(1);
- break;
- case SIGINT:
- /* expectedly quit */
- kill_daemon(0);
- break;
- case SIGALRM:
- timeout = 1;
- break;
- }
-}
-
-/**
- * idcpy - copy id from src to dst
- * @src: source id
- * @dst: destination id
- *
- * Would need to be changed if exe_ident_t is changed to not be a basic
- * type.
- *
- * no return value
- */
-static void idcpy(exe_ident_t *dst, exe_ident_t *src)
-{
- *dst = *src;
-}
-
-/**
- * idcmp - compare id1 to id2
- * @id1: first id
- * @dst: second id
- *
- * Would need to be changed if exe_ident_t is changed to not be a basic
- * type.
- *
- * returns: 0, if ids match
- * 1, if ids don't match
- */
-static int idcmp(exe_ident_t *i1, exe_ident_t *i2)
-{
- if (*i1 == *i2)
- return 0;
- else
- return 1;
-}
-
-/**
- * set_path_to_file - create new unique file in hugetlbfs
- * @tgt: target character array to store filename in
- * @size: size of @tgt
- *
- * Creates a new uniquely named file in the hugetlbfs mount point. File
- * is opened in mkstemp, and the file descriptor is stored in a globally
- * visible variable, because the caller's return value is already
- * overloaded.
- *
- * Effectively identical to get_unlinked_fd().
- *
- * no return value
- */
-static void set_path_to_file(char *tgt, int size)
-{
- const char *path;
- shared_fd = -1;
-
- tgt[size - 1] = '\0';
-
- path = hugetlbfs_find_path();
- if (!path) {
- shared_fd = -1;
- return;
- }
-
- strcpy(tgt, path);
- strncat(tgt, "/libhugetlbfs.tmp.XXXXXX", size - 1);
-
- shared_fd = mkstemp(tgt);
- if (shared_fd > 0) {
- /* prevent access to shared file */
- int ret = chmod(tgt, 0);
- if (ret < 0) {
- DEBUG("chmod failed\n");
- shared_fd = -1;
- }
- }
-}
-
-/**
- * do_shared_file - find shareable mapping, or create new one
- * @req: contains executable id and virtual addresses of segments
- * @resp: contains filename and whether prepare() needs to be run
- *
- * The core routine for sharing mappings. The request and response
- * structures allow for accepting many parameters and returning many
- * response values independent of the actual return value, which acts
- * kind of like metadata about @resp.
- *
- * returns: -1, if failed to create mapping
- * NULL, if sharing mapping already at head of list
- * pointer to previous entry in linked list, else
- */
-static struct shared_mapping *do_shared_file(struct client_request *creq,
- struct daemon_response *dresp)
-{
- struct shared_mapping *smptr, *prevsmptr;
-
- if (head == NULL) {
- /* allocating first member of linked list */
- goto create_mapping;
- }
-
- prevsmptr = NULL;
- smptr = head;
-
- while (smptr != NULL) {
- if (!idcmp(&(smptr->id), &(creq->id))
- && (smptr->vaddr == creq->vaddr))
- /* found match */
- goto share_mapping;
-
- /*
- * Keep track of the previous entry in the list, because
- * we're going to need to move the one we're currently
- * at. This prevents an extra loop later.
- */
- prevsmptr = smptr;
- smptr = smptr->next;
- }
-
-create_mapping:
- /* first attempt to map this segment, create a new mapping */
- smptr = (struct shared_mapping *)malloc(sizeof(struct shared_mapping));
- if (smptr == NULL) {
- ERROR("Failed to allocate new shared mapping: %s\n",
- strerror(errno));
- return (struct shared_mapping *)-1;
- }
-
- set_path_to_file(smptr->path_to_file, sizeof(smptr->path_to_file));
- if (shared_fd < 0) {
- ERROR("Failed to create new shared file\n");
- free(smptr);
- return (struct shared_mapping *)-1;
- }
-
- DEBUG("created shared mapping to %s\n", smptr->path_to_file);
-
- /* grab data out of request */
- idcpy(&(smptr->id), &(creq->id));
- smptr->vaddr = creq->vaddr;
- /* timestamp will be set upon insert */
-
- /* client will need to prepare file */
- dresp->need_to_prepare = 1;
- strcpy(dresp->path_to_file, smptr->path_to_file);
-
- /*
- * Cannot be NULL. Caller is responsible for freeing if an error
- * occurs
- */
- return smptr;
-
-share_mapping:
- /* store data in response */
- dresp->need_to_prepare = 0;
- strcpy(dresp->path_to_file, smptr->path_to_file);
-
- DEBUG("sharing mapping to %s\n", smptr->path_to_file);
-
- /*
- * Can be NULL, if sharing at the head of the list. Caller does
- * not need to deal with freeing, but must deal with both NULL
- * and non-NULL cases.
- * We are moving smptr to the front of the list if the sharing
- * succeeds, but send prevsmptr back so that we don't need to
- * iterate to do the move.
- */
- return prevsmptr;
-}
-
-static int parse_args(int argc, char *argv[]) {
- int opt;
- int ret = 0;
- while (1) {
- opt = getopt(argc, argv, ":p:s:dV:v");
- if (opt == -1)
- break;
-
- switch (opt) {
- case 'p':
- /* needs to be in milliseconds */
- poll_timeout = 1000*atoi(optarg);
- break;
- case 's':
- share_timeout = atoi(optarg);
- break;
- case 'd':
- /* do not daemonize */
- ret = 1;
- break;
- case 'V':
- __hugetlbfs_verbose = atoi(optarg);
- break;
- case 'v':
- usage();
- return -1;
- case ':':
- printf("getopt: missing parameter for option
%c\n", optopt);
- usage();
- return -1;
- case '?':
- printf("getopt: unknown option: %c\n", optopt);
- usage();
- return -1;
- default:
- printf("getopt: returned character code
0%o\n", opt);
- usage();
- return -1;
- }
- }
-
- if (poll_timeout == 0) {
- poll_timeout = DEFAULT_POLL_TIMEOUT;
- }
- if (share_timeout == 0) {
- share_timeout = DEFAULT_SHARE_TIMEOUT;
- }
-
- return ret;
-}
-
-static void daemonize()
-{
- pid_t pid, sid;
-
- pid = fork();
- if (pid < 0) {
- ERROR("daemonize: fork() failed\n");
- exit(1);
- }
- if (pid > 0)
- exit(0);
-
- umask(0);
-
- sid = setsid();
- if (sid < 0) {
- ERROR("daemonize: Failed to set session ID\n");
- exit(1);
- }
-
- if ((chdir("/")) < 0) {
- ERROR("daemonize: Failed to change directory\n");
- exit(1);
- }
-
- logfd = fopen(LOGFILE, "a+");
- if (!logfd) {
- ERROR("daemonize: Failed to open log file %s\n", LOGFILE);
- exit(1);
- }
-
- close(STDIN_FILENO);
- close(STDOUT_FILENO);
- close(STDERR_FILENO);
- stderr = logfd;
-}
-
-/**
- * read_client_request - receive request from client
- * @local_sock: socket to read from
- * @cresp: pointer to store request in
- *
- * Wait for client's request for sharing
- *
- * returns 0, on success
- * negative values, on failure
- */
-static int read_client_request(int local_sock, struct client_request
- *creq)
-{
- int ret;
- alarm(60);
- ret = read(local_sock, creq, sizeof(*creq));
- /* cancel any alarms */
- alarm(0);
- if (ret < 0) {
- /* timeout == 1 means that the SIGALRM handler ran */
- if (timeout) {
- DEBUG("read timed out\n");
- return -1;
- }
- ERROR("read failed: %s\n", strerror(errno));
- return -2;
- }
- if (ret != sizeof(*creq)) {
- /*
- * This should never happen, but is useful for
- * debugging size issues with mixed environments
- */
- DEBUG("short first read (got %d, should have been "
- "%zu\n", ret, sizeof(*creq));
- return -1;
- }
- return 0;
-}
-
-/**
- * write_daemon_response - respond to client
- * @local_sock: socket to write to
- * @cresp: pointer to response
- *
- * Acknowledge a client's request with a filename and whether the caller
- * needs to prepare
- *
- * returns 0, on success
- * negative values, on failure
- */
-static int write_daemon_response(int local_sock, struct daemon_response
- *dresp)
-{
- int ret;
- alarm(60);
- ret = write(local_sock, dresp, sizeof(*dresp));
- alarm(0);
- /*
- * If dresp.need_to_prepare is set, then we know we
- * dynamically allocated an element (which is pointed to
- * by smptr). We need to backout any changes it made to
- * the system state (open file, memory consumption) on
- * errors.
- */
- if (ret < 0) {
- if (timeout) {
- DEBUG("write timed out\n");
- if (dresp->need_to_prepare)
- return -1;
- return -2;
- }
- ERROR("write failed: %s\n", strerror(errno));
- if (dresp->need_to_prepare)
- return -3;
- return -4;
- }
- if (ret != sizeof(*dresp)) {
- DEBUG("short write (got %d, should have been %zu\n",
- ret, sizeof(*dresp));
- if (dresp->need_to_prepare)
- return -1;
- return -2;
- } /* else successfully wrote */
- return 0;
-}
-
-/**
- * read_client_response - wait for a client's ACK
- * @local_sock: socket to read from
- * @dresp: pointer to daemon's previous reponse
- * @cresp: pointer to store response in
- *
- * Wait for notification that file is prepared and segments have been
- * remapped. We need @dresp so that we can backout on errors
- * appropriately.
- *
- * returns 0, on success
- * negative values, on failure
- */
-static int read_client_response(int local_sock, struct daemon_response
- *dresp, struct client_response *cresp)
-{
- int ret;
- DEBUG("waiting for client's response\n");
- /* this may be unnecessarily long */
- alarm(300);
- /*
- */
- ret = read(local_sock, cresp, sizeof(*cresp));
- alarm(0);
- if (ret < 0) {
- if (timeout) {
- DEBUG("read timed out\n");
- if (dresp->need_to_prepare)
- return -1;
- return -2;
- }
- ERROR("read failed: %s\n", strerror(errno));
- if (dresp->need_to_prepare)
- return -3;
- return -4;
- }
- if (ret != sizeof(*cresp)) {
- DEBUG("short second read (got %d, should have been "
- "%zu\n", ret, sizeof(*cresp));
- if (dresp->need_to_prepare)
- return -1;
- return -2;
- }
- return 0;
-}
-
-/**
- * sharing_control_loop - main event loop for sharing
- * @sock: socket to wait on for connections
- *
- * This is expected to be an infinite loop. If we we ever return, it is
- * a fail case.
- *
- * returns 1
- */
-static int sharing_control_loop(int sock)
-{
- int ret;
- struct sockaddr_un from;
- socklen_t len = (socklen_t)sizeof(from);
- /*
- * Needed for socket messages -- could switch to dynamic
- * allocation
- */
- struct client_request creq;
- struct daemon_response dresp;
- struct client_response cresp;
- int local_sock;
- /* handle on some state that may need to be freed */
- struct shared_mapping *smptr;
- mode_t mode;
- /* needed for poll */
- struct pollfd ufds;
- /* used to allow for handling SIGALRM and SIGHUP */
- struct sigaction sigact = {
- .sa_handler = signal_handler,
- .sa_flags = 0,
- };
- struct sigaction prevact;
-
- sigemptyset(&(sigact.sa_mask));
-
- /* We will poll on sock for any incoming connections */
- ufds.fd = sock;
- ufds.events = POLLIN;
-
- /* install our custom signal-handler */
- sigaction(SIGINT, &sigact, NULL);
-
- for (;;) {
- /* gets set by SIGALRM handler */
- timeout = 0;
- /* not sure if this is necessary, but ok */
- ufds.revents = 0;
- /* helps with error handling */
- dresp.need_to_prepare = 0;
- /* reset mode of shared file */
- mode = 0;
-
- sigaction(SIGHUP, &sigact, &prevact);
-
-do_poll:
- ret = poll(&ufds, 1, poll_timeout);
- if (ret < 0) {
- /*
- * catch SIGHUP as non-fail case
- * SIGINT will have made us exit already
- */
- if (errno == EINTR)
- goto do_poll;
- ERROR("poll() failed: %s\n", strerror(errno));
- goto die;
- } else if (ret == 0) {
- /* poll timeout, reap any old files */
- DEBUG("poll() timed out\n");
- reap_files(0);
- continue;
- }
-
- switch (ufds.revents) {
- case POLLERR:
- ERROR("poll returned a POLLERR event\n");
- goto die;
- break;
- case POLLNVAL:
- ERROR("poll returned a POLLNVAL event\n");
- goto die;
- break;
- case POLLHUP:
- ERROR("poll returned a POLLHUP event\n");
- goto die;
- break;
- case POLLIN:
- break;
- default:
- ERROR("poll() returned an event we did not
expect: %d\n", ufds.revents);
- goto die;
- }
-
- local_sock = accept(sock, (struct sockaddr *)(&from), &len);
- if (local_sock < 0) {
- if (errno == ECONNABORTED) {
- /*
- * I think this is ok as a non-failure
- * case, but not positive, since the
- * client is what failed, not us.
- */
- DEBUG("accept() returned connection aborted\n");
- continue;
- }
- /*
- * Don't fail on a SIGHUP
- */
- if (errno == EINTR)
- goto do_poll;
- ERROR("accept() failed: %s\n", strerror(errno));
- goto die;
- }
-
- /*
- * Disable the races due to removing files that may be
- * sent to clients for sharing by just not handling
- * SIGHUP for a bit
- */
- sigaction(SIGHUP, &prevact, NULL);
- sigaction(SIGALRM, &sigact, NULL);
-
- ret = read_client_request(local_sock, &creq);
- switch (ret) {
- case 0:
- break;
- case -1:
- goto next;
- case -2:
- goto close_and_die;
- }
-
- /* Use the request to figure out the mapping ... */
- smptr = do_shared_file(&creq, &dresp);
- if (smptr == (struct shared_mapping *)-1) {
- ERROR("do_shared_file failed\n");
- /*
- * Just because we didn't get a hugetlbfs file,
- * doesn't mean we should fail outright. For
- * instance, if mkstemp() failed for some reason
- * we should still continue sharing any
- * previously shared segments.
- */
- goto next;
- }
-
- mode = S_IRUSR | S_IRGRP | S_IROTH;
- /* only allow preparer to write to file */
- if (dresp.need_to_prepare)
- mode |= S_IWUSR | S_IWGRP | S_IWOTH;
- ret = chmod(dresp.path_to_file, mode);
- if (ret < 0) {
- DEBUG("chmod(%s) to read (maybe writable) failed:
%s\n", dresp.path_to_file, strerror(errno));
- if (dresp.need_to_prepare)
- goto failed_next;
- goto next;
- }
-
- ret = write_daemon_response(local_sock, &dresp);
- switch (ret) {
- case 0:
- break;
- case -1:
- goto failed_next;
- case -2:
- goto next;
- case -3:
- goto failed_close_and_die;
- case -4:
- goto failed_next;
- }
-
- ret = read_client_response(local_sock, &dresp, &cresp);
- switch (ret) {
- case 0:
- break;
- case -1:
- goto failed_next;
- case -2:
- goto next;
- case -3:
- goto failed_close_and_die;
- case -4:
- goto failed_next;
- }
-
- /*
- * Determine if the segments were prepared/remapped as
- * expected.
- */
- if (cresp.succeeded != 0) {
- DEBUG("file not prepared\n");
- if (dresp.need_to_prepare)
- goto failed_next;
- goto next;
- }
-
- DEBUG("File prepared / opened successfully\n");
- if (dresp.need_to_prepare) {
- mode = S_IRUSR | S_IRGRP | S_IROTH;
- ret = chmod(dresp.path_to_file, mode);
- if (ret < 0) {
- DEBUG("chmod(%s) to read-only failed: %s\n",
dresp.path_to_file, strerror(errno));
- goto failed_next;
- }
- /*
- * Insert new element at head of list
- */
- if (head != NULL)
- smptr->next = head;
- else
- smptr->next = NULL;
- head = smptr;
-
- close(shared_fd);
- } else {
- /*
- * Move element to head of list. We have pointer
- * to the element just before the element to
- * move. If it is NULL, then we are sharing at
- * the head of the list, and no moving is
- * necessary. Else, shift the pointers around.
- * We obviously also know that head != NULL.
- */
- if (smptr != NULL) {
- struct shared_mapping *temp;
- temp = smptr->next;
- smptr->next = temp->next;
- temp->next = head;
- head = temp;
- }
- }
- /*
- * Shared mapping is at the head of the list now, update
- * timestamp.
- */
- head->timestamp = time(NULL);
-
- goto next;
-
-failed_next:
- close(shared_fd);
- unlink(dresp.path_to_file);
- free(smptr);
-next:
- close(local_sock);
- }
-
-failed_close_and_die:
- close(shared_fd);
- unlink(dresp.path_to_file);
- free(smptr);
-close_and_die:
- close(local_sock);
-die:
- kill_daemon(1);
-
- /* indicate failure if we ever get this far */
- return 1;
-}
-
-/**
- * main - set up socket and enter central control loop
- * @argc: number of command line arguments
- * @argv: command line arguments
- *
- * returns: 0, on success
- * -1, on failure
- */
-int main(int argc, char *argv[])
-{
- int ret;
- /* local socket structures */
- struct sockaddr_un sun;
-
- ret = parse_args(argc, argv);
- if (ret < 0)
- exit(1);
- if (ret == 0)
- /* Perform all the steps to become a real daemon */
- daemonize();
-
- sock = socket(AF_UNIX, SOCK_STREAM, 0);
- if (sock < 0) {
- ERROR("socket() failed: %s\n", strerror(errno));
- return -1;
- }
-
- sun.sun_family = AF_UNIX;
- /* clear out any previous socket */
- unlink("/tmp/libhugetlbfs-sock");
- strcpy(sun.sun_path, "/tmp/libhugetlbfs-sock");
- ret = bind(sock, (struct sockaddr *)(&sun), sizeof(sun));
- if (ret < 0) {
- ERROR("bind() failed: %s\n", strerror(errno));
- goto die;
- }
-
- chmod("/tmp/libhugetlbfs-sock", 0666);
-
- ret = listen(sock, QUEUE_LENGTH);
- if (ret < 0) {
- ERROR("listen() failed: %s\n", strerror(errno));
- goto die;
- }
-
- return sharing_control_loop(sock);
-
-die:
- kill_daemon(1);
-
- /* indicate failure if we ever get this far */
- return 1;
-}
diff --git a/hugetlbd.h b/hugetlbd.h
deleted file mode 100644
index 04423cd..0000000
--- a/hugetlbd.h
+++ /dev/null
@@ -1,60 +0,0 @@
-/*
- * libhugetlbfs - Easy use of Linux hugepages
- * Copyright (C) 2006 Nishanth Aravamudan, IBM Corporation.
- *
- * This library is free software; you can redistribute it and/or
- * modify it under the terms of the GNU Lesser General Public License
- * as published by the Free Software Foundation; either version 2.1 of
- * the License, or (at your option) any later version.
- *
- * This library is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * Lesser General Public License for more details.
- *
- * You should have received a copy of the GNU Lesser General Public
- * License along with this library; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
- */
-
-#ifndef _HUGETLBD_H
-#define _HUGETLBD_H
-
-#ifndef __LIBHUGETLBFS__
-#error This header should not be included by library users.
-#endif /* __LIBHUGETLBFS__ */
-
-#define ID_KEY 0x56
-#define LOGFILE "/tmp/hugetlbd.log"
-
-/*
- * Ideally, would like to deal with this better, so that a 32-bit daemon
- * only deals in 32-bit addresses. But, for now, this makes 32-bit and
- * 64-bit apps deal ok with the socket messages.
- */
-typedef uint64_t vaddr_t;
-typedef uint64_t exe_ident_t;
-
-struct client_request {
- /*
- * unique identifer
- * if id == 0, then the request is actually a response that the
- * file is prepared
- */
- exe_ident_t id;
- /*
- * to identify the segment
- */
- vaddr_t vaddr;
-};
-
-struct client_response {
- int succeeded;
-};
-
-struct daemon_response {
- int need_to_prepare;
- char path_to_file[PATH_MAX+1];
-};
-
-#endif /* _HUGETLBD_H */
diff --git a/hugeutils.c b/hugeutils.c
index e1872fa..553d94f 100644
--- a/hugeutils.c
+++ b/hugeutils.c
@@ -36,14 +36,11 @@
#include <sys/vfs.h>
#include <sys/statfs.h>
#include <sys/types.h>
-#include <sys/ipc.h>
-#include <sys/socket.h>
-#include <sys/un.h>
#include <sys/mman.h>
+#include <sys/file.h>
#include "libhugetlbfs_internal.h"
#include "hugetlbfs.h"
-#include "hugetlbd.h"
static int hpage_size; /* = 0 */
static char htlb_mount[PATH_MAX+1]; /* = 0 */
@@ -248,48 +245,55 @@ int hugetlbfs_unlinked_fd(void)
}
/**
- * create_id - create unique id for current executable
+ * get_shared_file_name - create a shared file name from program name, gid and
phdr
+ * @htlb_seg_info: pointer to program's segment data
+ * @file_path: pointer to a PATH_MAX array to store filename in
*
- * Uses IPC ftok() to generate a roughly unique identifier for our own
- * executable. This is used to match shareable mappings.
+ * The file name created is *not* intended to be unique, except when the name,
+ * gid or phdr number differ. The goal here is to have a standar means of
+ * accessing particular segments of particular executables.
*
- * returns: -1, on error
- * a unique identifier, otherwise
+ * returns:
+ * -1, on failure
+ * 0, on success
*/
-static exe_ident_t create_id(void)
+static int get_shared_file_name(struct seg_info *htlb_seg_info, char
*file_path)
{
- exe_ident_t id;
int ret;
- char my_exe[PATH_MAX];
+ char binary[PATH_MAX];
+ char *binary2;
+ const char *hugetlbfs_path;
- ret = readlink("/proc/self/exe", my_exe, PATH_MAX);
+ ret = readlink("/proc/self/exe", binary, PATH_MAX);
if (ret < 0) {
- ERROR("readlink failed: %s\n", strerror(errno));
+ ERROR("shared_file: readlink() on /proc/self/exe "
+ "failed: %s\n", strerror(errno));
return -1;
}
- id = (exe_ident_t)ftok(my_exe, ID_KEY);
- if (id < 0) {
- ERROR("ftok failed: %s\n", strerror(errno));
+ binary2 = basename(binary);
+ if (!binary2) {
+ ERROR("shared_file: basename() on %s failed: %s\n",
+ binary, strerror(errno));
return -1;
}
- return id;
-}
-
-static int sock;
-static int timeout;
-/* global to allow multiple callsites to set members */
+ hugetlbfs_path = hugetlbfs_find_path();
+ if (!hugetlbfs_path) {
+ ERROR("shared_file: hugetlbfs_find_path() failed\n");
+ return -1;
+ }
-static void signal_handler(int signal)
-{
- switch (signal) {
- case SIGALRM:
- timeout = 1;
+ ret = snprintf(file_path, PATH_MAX, "%s/%s_%d_%zd_%d",
+ hugetlbfs_path, binary2, getgid(), sizeof(unsigned long) * 8,
+ htlb_seg_info->phdr);
+ if (ret < 0) {
+ ERROR("shared_file: snprintf() failed\n");
+ return -1;
}
-}
-static int client_complete(struct seg_info *htlb_seg_info, int success);
+ return 0;
+}
/**
* hugetlbfs_shared_file - communicate with daemon to get one shareable file
@@ -302,196 +306,125 @@ static int client_complete(struct seg_in
*/
static int hugetlbfs_shared_file(struct seg_info *htlb_seg_info)
{
- int ret, fd;
- struct client_request creq;
- struct daemon_response dresp;
- struct sockaddr_un sun;
- struct sigaction sigact = {
- .sa_handler = signal_handler,
- .sa_flags = 0,
- };
- struct sigaction prev;
- int open_flags;
-
- sock = socket(AF_UNIX, SOCK_STREAM, 0);
- if (sock < 0) {
- ERROR("socket() failed: %s\n", strerror(errno));
- return -1;
- }
+ int fd, ret;
+ char file_path[PATH_MAX];
- sun.sun_family = AF_UNIX;
- strcpy(sun.sun_path, "/tmp/libhugetlbfs-sock");
- ret = connect(sock, &sun, sizeof(sun));
- if (ret < 0) {
- ERROR("connect() failed: %s\n", strerror(errno));
+ ret = get_shared_file_name(htlb_seg_info, file_path);
+ if (ret < 0)
return -1;
- }
-
- sigemptyset(&(sigact.sa_mask));
-
- sigaction(SIGALRM, &sigact, &prev);
-
- /* add data to request */
- creq.id = create_id();
- /*
- * This results in (non-harmful) compilation warning for
- * 32-bit applications
- */
- creq.vaddr = (vaddr_t)htlb_seg_info->vaddr;
-
- alarm(60);
- ret = write(sock, &creq, sizeof(creq));
- /* cancel any alarms */
- alarm(0);
- if (ret < 0) {
- if (timeout) {
- DEBUG("write timed out\n");
- goto fail;
- } else {
- ERROR("client request write failed: %s\n",
- strerror(errno));
- goto fail;
- }
- } else if (ret != sizeof(creq)) {
- DEBUG("short write (got %d, should have been %zu)\n",
- ret, sizeof(creq));
- goto fail;
- } /* else successfully wrote */
-
- /* wait slightly longer for the response */
- alarm(120);
- ret = read(sock, &dresp, sizeof(dresp));
- alarm(0);
- if (ret < 0) {
- if (timeout) {
- DEBUG("read timed out\n");
- goto fail;
- } else {
- ERROR("read failed: %s\n", strerror(errno));
- goto fail;
- }
- } else if (ret != sizeof(dresp)) {
- DEBUG("short read (got %d, should have been %zu)\n",
- ret, sizeof(dresp));
- goto fail;
- } /* else successfully read */
-
- sigaction(SIGALRM, &prev, NULL);
-
- /* Convert daemon response to open mode bits */
- if (dresp.need_to_prepare) {
- open_flags = O_RDWR;
- } else {
- open_flags = O_RDONLY;
- }
- /* get local handle on file */
- fd = open(dresp.path_to_file, open_flags);
+ fd = open(file_path, O_CREAT | O_EXCL | O_RDWR, 0640);
if (fd < 0) {
- ERROR("open(%s) failed: %s\n", dresp.path_to_file,
- strerror(errno));
- goto fail;
- }
-
- if (dresp.need_to_prepare) {
- DEBUG("daemon said to prepare\n");
+ if (errno == EEXIST) {
+ fd = open(file_path, O_RDONLY);
+ if (fd < 0) {
+ ERROR("shared_file: nonexclusive open "
+ "unexpectedly failed: %s\n",
+ strerror(errno));
+ return -1;
+ }
+ ret = flock(fd, LOCK_SH);
+ if (ret < 0) {
+ ERROR("shared_file: sharer lock failed: %s\n",
+ strerror(errno));
+ close(fd);
+ return -1;
+ }
+ if (access(file_path, F_OK)) {
+ /* check to make sure prepare succeeded */
+ ERROR("shared_file: sharer can't access "
+ "file: %s\n",
+ strerror(errno));
+ close(fd);
+ return -1;
+ }
+ htlb_seg_info->fd = fd;
+ return 0;
+ }
+ ERROR("shared_file: exclusive open unexpectedly failed: %s\n",
+ strerror(errno));
+ return -1;
} else {
- /* ACK right away (closes socket) */
- DEBUG("daemon said not to prepare\n");
- ret = client_complete(htlb_seg_info, 0);
- if (ret < 0)
- goto fail;
+ ret = flock(fd, LOCK_EX);
+ if (ret < 0) {
+ ERROR("shared_file: preparer exclusive lock "
+ "failed: %s\n", strerror(errno));
+ close(fd);
+ return -1;
+ }
+ /* we are the first to get the lock */
+ htlb_seg_info->fd = fd;
+ return 1;
}
-
- htlb_seg_info->fd = fd;
-
- /* leave socket open until finished_prepare is called */
- return dresp.need_to_prepare;
-
-fail:
- close(sock);
- return -1;
}
/**
- * client_complete - send ACK to daemon that we no longer need file
+ * finished_prepare - callback for ACK'ing prepare is done
* @htlb_seg_info: pointer to program's segment data
- * @success: if 0, successful prepare, else failed
*
- * returns: 0, if successful transmission (none occurs if
- * not sharing)
- * -1, if failed transmission
+ * returns: 0, if successful
+ * -1, if failed
*/
-static int client_complete(struct seg_info *htlb_seg_info, int success)
+int finished_prepare(struct seg_info *htlb_seg_info)
{
- struct sigaction sigact = {
- .sa_handler = signal_handler,
- .sa_flags = 0,
- };
- struct sigaction prev;
- struct client_response cresp;
int ret;
+ char file_path[PATH_MAX];
- sigemptyset(&(sigact.sa_mask));
-
- sigaction(SIGALRM, &sigact, &prev);
+ if (sharing) {
- cresp.succeeded = success;
-
- alarm(60);
- ret = write(sock, &cresp, sizeof(cresp));
- alarm(0);
- if (ret < 0) {
- if (timeout) {
- DEBUG("write timed out\n");
- goto fail;
- } else {
- ERROR("client response write failed: %s\n",
- strerror(errno));
- goto fail;
+ ret = get_shared_file_name(htlb_seg_info, file_path);
+ if (ret < 0) {
+ ERROR("finished_prepare: get_shared_file() "
+ "failed\n");
+ (void)flock(htlb_seg_info->fd, LOCK_SH);
+ (void)flock(htlb_seg_info->fd, LOCK_UN);
+ goto fail1;
}
- } else if (ret != sizeof(cresp)) {
- DEBUG("short write (got %d, should have been %zu)\n",
- ret, sizeof(cresp));
- goto fail;
- } /* else successfully wrote */
-
- sigaction(SIGALRM, &prev, NULL);
-
- if (success < 0)
- /* back out open'd fd since we failed */
- close(htlb_seg_info->fd);
- close(sock);
+ /*
+ * reduce permissions on file
+ * should be a no-op in the failure case
+ */
+
+ ret = fchmod(htlb_seg_info->fd, 0440);
+ if (ret < 0) {
+ ERROR("finished_prepare: fchmod failed: "
+ "%s\n", strerror(errno));
+ /*
+ * we're going to bomb out anyways,
+ * so ignore return values
+ */
+ (void)flock(htlb_seg_info->fd, LOCK_SH);
+ (void)flock(htlb_seg_info->fd, LOCK_UN);
+ goto fail2;
+ }
+ ret = flock(htlb_seg_info->fd, LOCK_SH);
+ if (ret < 0) {
+ ERROR("finished_prepare: preparer share lock "
+ "failed: %s\n", strerror(errno));
+ goto fail2;
+ }
+ ret = flock(htlb_seg_info->fd, LOCK_UN);
+ if (ret < 0) {
+ ERROR("finished_prepare: preparer unlock "
+ "failed: %s\n", strerror(errno));
+ goto fail2;
+ }
+ }
return 0;
-fail:
- close(htlb_seg_info->fd);
- close(sock);
+fail2:
+ (void)unlink(file_path);
+fail1:
+ (void)close(htlb_seg_info->fd);
return -1;
}
/**
- * finished_prepare - callback for ACK'ing prepare is done
- * @htlb_seg_info: pointer to program's segment data
- * @success: if 0, successful prepare, else failed
- *
- * returns: 0, if successful transmission (none occurs if
- * not sharing)
- * -1, if failed transmission
- */
-int finished_prepare(struct seg_info *htlb_seg_info, int success)
-{
- if (sharing)
- return client_complete(htlb_seg_info, success);
- return 0;
-}
-
-/**
* hugetlbfs_set_fds - multiplex callers depending on if sharing or not
* @htlb_seg_info: pointer to program's segment data
*
- * returns:
+ * returns:
* -1, on error
* 0, on success, caller expects and gets shared, does not need to prepare
* 1, on success, caller expects and gets shared, does need to prepare
diff --git a/libhugetlbfs_internal.h b/libhugetlbfs_internal.h
index 5de2e1d..9f95ae3 100644
--- a/libhugetlbfs_internal.h
+++ b/libhugetlbfs_internal.h
@@ -53,9 +53,10 @@ struct seg_info {
unsigned long filesz, memsz;
int prot;
int fd;
+ int phdr;
};
-int finished_prepare(struct seg_info *htlb_seg_info, int success);
+int finished_prepare(struct seg_info *htlb_seg_info);
int hugetlbfs_set_fd(struct seg_info *htlb_seg_info);
#if defined(__powerpc64__) && !defined(__LP64__)
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
index 459c7d9..5f7516e 100755
--- a/tests/run_tests.sh
+++ b/tests/run_tests.sh
@@ -65,13 +65,9 @@ elfshare_test () {
unset args[$N]
set -- "[EMAIL PROTECTED]"
NUM_THREADS=$((`free_hpages` / 15 - 1))
- killall -HUP hugetlbd
run_test HUGETLB_SHARE=2 "$@" "xB.$baseprog" $NUM_THREADS
- killall -HUP hugetlbd
run_test HUGETLB_SHARE=1 "$@" "xB.$baseprog" $NUM_THREADS
- killall -HUP hugetlbd
run_test HUGETLB_SHARE=2 "$@" "xBDT.$baseprog" $NUM_THREADS
- killall -HUP hugetlbd
run_test HUGETLB_SHARE=1 "$@" "xBDT.$baseprog" $NUM_THREADS
}
@@ -129,21 +125,11 @@ functional_tests () {
elflink_test linkhuge
# Sharing tests
- # stop all running instances for clean testing
- killall -INT hugetlbd
- # start the daemon in the bg
- PATH=../obj32:../obj64:$PATH hugetlbd
- # XXX: Wait for daemon to start
- sleep 5
elfshare_test linkshare
- # stop our instance of the daemon
- killall -INT hugetlbd
# Accounting bug tests
# reset free hpages because sharing will have held some
# alternatively, use
-# killall -HUP hugetlbd
-# to make the sharing daemon give up the files
run_test chunk-overcommit `free_hpages`
run_test alloc-instantiate-race `free_hpages`
}
--
Nishanth Aravamudan <[EMAIL PROTECTED]>
IBM Linux Technology Center
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Libhugetlbfs-devel mailing list
Libhugetlbfs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libhugetlbfs-devel
