Re: [PATCH v1] CLD replication (WIP)

2009-07-31 Thread Jeff Garzik
Jeff Garzik wrote: Sage Weil wrote: Hi Jeff, Do you still plan to replace bdb (and it's replication) with a something based on paxos? I'm considering replacing the Ceph monitors (which currently implement paxos, but in a very ceph-specific way) with cld if it can meet the basic

CLD future plans (was Re: [PATCH v1] CLD replication (WIP))

2009-07-31 Thread Jeff Garzik
Sage Weil wrote: On Fri, 31 Jul 2009, Jeff Garzik wrote: Sage Weil wrote: Hi Jeff, Do you still plan to replace bdb (and it's replication) with a something based on paxos? I'm considering replacing the Ceph monitors (which currently implement paxos, but in a very ceph-specific way) with cld

chunkd authentication checks added

2009-07-31 Thread Jeff Garzik
Just added per-operation authentication checks to chunkd. If the authentication user does not own an object, they cannot GET or DELETE it. LIST only shows objects owned by the authenticated user (making the Owner tag a bit redundant). The newly added 'auth' test verified that this works,

cld/chunkd/tabled release URLs moved a bit

2009-08-01 Thread Jeff Garzik
The URL for the release tarballs of Project Hail, http://www.kernel.org/pub/software/network/distsrv/ was getting a little crowded, with all the tarballs in that one directory. The tarballs have been moved into per-project sub-directories. cld tarballs will live in

Re: [Patch 1/3] cldcli: Add default CLD host

2009-08-03 Thread Jeff Garzik
Pete Zaitcev wrote: It's convenient to run cldcli without arguments if your DNS is set up. Signed-off-by: Pete Zaitcev zait...@redhat.com applied -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info

Re: [Patch 3/3] cldcli: drop useless comment

2009-08-03 Thread Jeff Garzik
Pete Zaitcev wrote: This seems copy-pasted from a daemon like Chunk that has a comment to the tune of now that we have arguments parsed we can switch to syslog. But in cldcli it's meaningless. Signed-off-by: Pete Zaitcev zait...@redhat.com applied -- To unsubscribe from this list: send the

Re: [Patch] chunkd: use port xxx82 to build

2009-08-04 Thread Jeff Garzik
Pete Zaitcev wrote: Let's change build ports to xxx82 to deconflict with tabled. Signed-Off-By: Pete Zaitcev zait...@redhat.com diff --git a/test/server-test.cfg b/test/server-test.cfg index 38fcc43..02c39d4 100644 --- a/test/server-test.cfg +++ b/test/server-test.cfg @@ -5,11 +5,11 @@ /SSL

Re: [Patch] chunkd: use port xxx82 to build

2009-08-04 Thread Jeff Garzik
Pete Zaitcev wrote: On Tue, 04 Aug 2009 21:00:36 -0400, Jeff Garzik j...@garzik.org wrote: Speaking of chunkd... I don't think I will have time this week to fix chunkd's failure on big endian machines. If you (or anyone else listening...) have an opportunity to look into that, that would

Re: [Patch] chunkd: use port xxx82 to build

2009-08-04 Thread Jeff Garzik
Pete Zaitcev wrote: On Tue, 04 Aug 2009 21:40:22 -0400, Jeff Garzik j...@garzik.org wrote: Another avenue is to sign up for a Fedora Account (http://fedoraproject.org/wiki/Join) and get access to scratch builds via koji (https://fedoraproject.org/wiki/Using_the_Koji_build_system). Chunk

[PATCH v1] CLD big message support (WIP)

2009-08-06 Thread Jeff Garzik
Below is my current work-in-progress, finishing the packet / message logical separation. This will permit messages larger than a single packet, which, in turn, permits GET and PUT messages to be more easily handled as a single, contiguous message. DATA messages are eliminated. This is a major

[PATCH v3] CLD replication (WIP)

2009-08-10 Thread Jeff Garzik
Attached is the current CLD replication patch, which takes CLD from being a single-node service to a fully replicated, highly available service. The server implementation should be complete. The current merge blocker is needed code in libcldc, which does not yet properly hunt for a master,

Re: [Patch] libcldc: transition to applog, phase 1

2009-08-11 Thread Jeff Garzik
Pete Zaitcev wrote: This patch transitions from printf-like to syslog-like API for libcldc diagnostics. In the future, this allows to drop app_log() from chunkd (tabled didn't even grow it). There's a small complication: one stdarg function cannot call another, so we cannot just insert some

Re: [Patch 3/4] chunkd: write our contact information into CLD

2009-08-11 Thread Jeff Garzik
Pete Zaitcev wrote: Add our listening information into the parameter file in CLD. For now we only do that, and do not try to reorganize the CLD namespace that Chunk uses, use full path as a parameter, and other planned things. We do switch to asprintf though, it makes the code more robust (no

Re: [Patch] libcldc: correct length checks in open_end_cb

2009-08-12 Thread Jeff Garzik
Pete Zaitcev wrote: Without this patch, open errors cannot be delivered to application and opens get stuck forever. Also, who added -EINVAL again?! Stop doing that, the user wants to know what failed: syscall or CLD protocol. Signed-Off-By: Pete Zaitcev zait...@redhat.com applied -- To

Re: BabuDB

2009-08-16 Thread Jeff Garzik
Fabian Deutsch wrote: Hey. Reading about the release of XtremFS 1.0 [1] i stumbled accross babudb [2]. babudb provides some log based format, requested in the hail wiki [3] for CLD. So I just wanted to let you know :) - fabian [1] http://www.xtreemfs.org/ Both are interesting.. I note

Re: [Patch] tabled: parse Chunk parameters in CLD v2

2009-08-16 Thread Jeff Garzik
On 08/16/2009 09:10 AM, Pete Zaitcev wrote: This patch probably has all that's necessary to flip the configuration to use CLD to configure available Chunk nodes and dispose with StorageNode forever. However, it did not complete all tests that's necessary to declare victory. So for now we

Re: Interested in contributing

2009-08-26 Thread Jeff Garzik
On 08/25/2009 09:13 PM, John Quigley wrote: Hey Folks: I've just recently discovered the existence of this project, and am excited by its prospects. I hopefully have relevant experience to help with the effort, as I've spent the last four years working on a network distributed storage

Re: [Patch] libcldc,cldcli: Use humanized error messages

2009-08-26 Thread Jeff Garzik
[slight change, error messages kept in array -jg] Signed-off-by: Jeff Garzik jgar...@redhat.com diff --git a/include/cld_msg.h b/include/cld_msg.h index 01bda16..e4c8f28 100644 --- a/include/cld_msg.h +++ b/include/cld_msg.h @@ -257,5 +257,6 @@ struct cld_msg_event { extern unsigned long long

Re: [Patch 1/3] tabled: Employ sesible start-up retries

2009-08-29 Thread Jeff Garzik
On 08/28/2009 10:41 PM, Pete Zaitcev wrote: With this patch, daemons that tabled needs may come up at any time relatively to each other and tabled should be able to initialize itself. To this end, we remove the extra sleep command, now unnecessary. Unfortunately, one sleep is still needed for

Re: [Patch 2/2] tabled: tweak chunk-test.conf

2009-10-02 Thread Jeff Garzik
On 09/30/2009 09:42 PM, Pete Zaitcev wrote: - Add the fresh option ForceHost, for the sake of Fedora builds. - Change the NID just to make it more interesting (this helps to make sure we're not using 1 by accident). Signed-off-by: Pete Zaitcevzait...@redhat.com applied 1-2 -- To

Re: Hail usable as a limited local AWS S3 and doc contribution?

2009-10-14 Thread Jeff Garzik
On 10/13/2009 01:27 PM, Zack Perry wrote: Hi, While trying to find a less expensive way to validate my application targeting AWS S3, I stumbled across Project Hail. After reading the wiki, I downloaded cld, chunkd, and tabled tar balls, built them on my Ubuntu 9.04 based VPS successfully.

[PATCH] chunkd: update backend to support binary keys

2009-11-03 Thread Jeff Garzik
(yeah, the summary line needs s/cld:/chunkd:/) Just committed the following change to chunkd's storage backend API, preparing chunkd for variable-length opaque binary keys. commit 0a8c3225355de86484eba71279c646f7790d5bb1 Author: Jeff Garzik j...@garzik.org Date: Wed Nov 4 00:44:55 2009

Re: [PATCH] chunkd: update backend to support binary keys

2009-11-03 Thread Jeff Garzik
On 11/04/2009 12:51 AM, Jeff Garzik wrote: (yeah, the summary line needs s/cld:/chunkd:/) Just committed the following change to chunkd's storage backend API, preparing chunkd for variable-length opaque binary keys. commit 0a8c3225355de86484eba71279c646f7790d5bb1 Author: Jeff Garzikj

command line interface to chunkd, tabled

2009-11-08 Thread Jeff Garzik
A command line client for chunkd, chcli, was just committed. It offers the ability to GET (dumps to stdout) and PUT (input from cmd line arg). Also, some quick searching revealed a command line tool for S3 (and thus tabled), s3cmd: http://s3tools.org/s3cmd Jeff -- To

chunkd design genesis, storage tech, and support for multiple key/value tables

2009-11-10 Thread Jeff Garzik
You wrote this insightful and pointed comment on IRC... Comparing with every k/v service out there assumes that you're growing a generic key/value service out of Chunk. You're essentially admitting it openly. This is an excellent point to raise. So let the begin at the beginning, cover the

Re: [Patch 1/7] tabled: Fix error path in bucket_del

2009-11-14 Thread Jeff Garzik
On 11/14/2009 01:30 AM, Pete Zaitcev wrote: Signed-off-by: Pete Zaitcevzait...@redhat.com applied 1-2 -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at

Re: [Patch 3/7] tabled: Reduce verbosity in CLD client

2009-11-14 Thread Jeff Garzik
On 11/14/2009 01:32 AM, Pete Zaitcev wrote: I got into habit of watching logs and there was too much useless messaging in them. Some messages were stubs and reminders, so I left them in the code, but commented out. Signed-off-by: Pete Zaitcevzait...@redhat.com I applied this, just to keep

Re: [Patch 4/7] tabled: Fix buffer pointer confusion in object.c

2009-11-14 Thread Jeff Garzik
On 11/14/2009 01:33 AM, Pete Zaitcev wrote: This bug leads to object's data corrupt whenever the write is partial. Signed-off-by: Pete Zaitcevzait...@redhat.com applied -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to

Re: [Patch 5/7] tabled: Add replication daemon

2009-11-14 Thread Jeff Garzik
On 11/14/2009 01:36 AM, Pete Zaitcev wrote: This patch adds what amounts to a background process that maintains redundancy for object data. It is far from the complete solution. For one thing, it does not verify checksums. But it's a start. There's no way to turn this off, by intention. The

btest2.c

2009-11-14 Thread Jeff Garzik
This is a simple test to determine memory usage for storing object locations in RAM, indexed by node id, and optimized for - rapid object id insertion, deletion - easy method to determine all objects in a single node Compile with gcc -O -Wall `pkg-config glib-2.0

[PATCH] CLD replication (WIP)

2009-11-18 Thread Jeff Garzik
|8 +- server/server.c | 287 ++--- test/pid-exists | 24 +++-- test/prep-db | 19 ++-- test/start-daemon | 24 - test/stop-daemon | 18 ++-- 9 files changed, 419 insertions(+), 59 deletions(-) Jeff Garzik (1): cld

Re: [PATCH] CLD replication (WIP)

2009-11-18 Thread Jeff Garzik
On 11/18/2009 12:00 PM, Pete Zaitcev wrote: On Wed, 18 Nov 2009 03:45:40 -0500, Jeff Garzikj...@garzik.org wrote: I just updated the current CLD replication code to the latest upstream cld.git code. It is now living on the replica branch of git://git.kernel.org/pub/scm/daemon/cld/cld.git

Re: [Patch 1/1] tabled: Add replication daemon

2009-11-27 Thread Jeff Garzik
On 11/26/2009 09:39 PM, Pete Zaitcev wrote: This patch adds what amounts to a background process that maintains redundancy for object data. It is far from the complete solution. For one thing, it does not verify checksums. But it's a start. There's no way to turn this off, by intention. The

Re: [Patch 1/2] tabled: drop dependency on libevent from libcldc

2009-11-28 Thread Jeff Garzik
On 11/28/2009 02:07 AM, Pete Zaitcev wrote: On Sat, 28 Nov 2009 00:09:09 -0500 Jeff Garzikj...@garzik.org wrote: On 11/27/2009 10:52 PM, Pete Zaitcev wrote: The patched tabled can build and work with the old libcldc too. Part of the reason for that is the continued presence of

Re: [Patch 1/2] tabled: drop dependency on libevent from libcldc

2009-11-28 Thread Jeff Garzik
On 11/27/2009 10:52 PM, Pete Zaitcev wrote: This patch makes it so that tabled stops relying on libcldc using libevent. Thus, libcldc would be free to drop libevent dependency. The patched tabled can build and work with the old libcldc too. Signed-Off-By: Pete Zaitcevzait...@redhat.com ---

Re: [PATCH] Some minor CLD test program fixes

2009-11-28 Thread Jeff Garzik
On 11/27/2009 06:20 PM, cmcc...@alumni.cmu.edu wrote: --- a/lib/common.c +++ b/lib/common.c @@ -56,6 +56,37 @@ const char *cld_errstr(enum cle_err_codes ecode) return cld_errlist[ecode]; } +/** Read from a file descriptor, resuming after interruptions. + * + * @param fd The

Re: [PATCH v2 2/2] cld: read the cld.port file using g_file_get_contents

2009-11-28 Thread Jeff Garzik
On 11/27/2009 09:23 PM, Colin McCabe wrote: Signed-off-by: Colin McCabecmcc...@alumni.cmu.edu --- lib/common.c | 29 +++-- 1 files changed, 15 insertions(+), 14 deletions(-) diff --git a/lib/common.c b/lib/common.c index 68f60f8..db20e2a 100644 --- a/lib/common.c

Re: [PATCH v2 2/2] cld: read the cld.port file using g_file_get_contents

2009-11-28 Thread Jeff Garzik
On 11/28/2009 05:37 AM, Jeff Garzik wrote: 1) 'buf' is no longer nul-terminated, which means strtol() has become a buffer overrun. Whoops, it seems I am incorrect on this, according to http://library.gnome.org/devel/glib/stable/glib-File-Utilities.html#g-file-get-contents Ignore that point

Re: [PATCH v3 2/2] cld: read the cld.port file using g_file_get_contents

2009-11-28 Thread Jeff Garzik
On 11/28/2009 07:34 AM, Colin McCabe wrote: Signed-off-by: Colin McCabecmcc...@alumni.cmu.edu --- lib/common.c | 27 +-- 1 files changed, 13 insertions(+), 14 deletions(-) applied, thanks for the revisions -- To unsubscribe from this list: send the line

Re: [Patch 1/2] CLD: factor timers out into a library

2009-11-29 Thread Jeff Garzik
On 11/29/2009 01:35 AM, Pete Zaitcev wrote: From: Jeff Garzikjgar...@redhat.com Move timer_init and friends so that they can be used by test/*. Signed-Off-By: Pete Zaitcevzait...@redhat.com applied 1-2 -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a

Re: [Patch 1/2] CLD: factor timers out into a library

2009-11-29 Thread Jeff Garzik
On 11/29/2009 01:35 AM, Pete Zaitcev wrote: From: Jeff Garzikjgar...@redhat.com Move timer_init and friends so that they can be used by test/*. Signed-Off-By: Pete Zaitcevzait...@redhat.com hmmm... cld now segfaults reliably in koji:

Re: [Patch 1/2] CLD: factor timers out into a library

2009-11-29 Thread Jeff Garzik
On 11/29/2009 02:34 PM, Pete Zaitcev wrote: On Sun, 29 Nov 2009 06:11:44 -0500 Jeff Garzikj...@garzik.org wrote: hmmm... cld now segfaults reliably in koji: http://koji.fedoraproject.org/koji/taskinfo?taskID=1836079 Curious. It works fine here, of course (make distcheck). Did you try to

Re: [Patch 1/1] CLD: fix crash in __mutex_get_max (libdb-4.7.so) on F13

2009-11-29 Thread Jeff Garzik
On 11/29/2009 08:17 PM, Pete Zaitcev wrote: Fedora 13 comes with db4.8 and apparently the compat-db4.7 is bust. Let us link with 4.8 as a workaround. Signed-Off-By: Pete Zaitcevzait...@redhat.com --- configure.ac |5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) Not sure how

Re: [Patch 1/1] CLD: fix crash in __mutex_get_max (libdb-4.7.so) on F13

2009-12-02 Thread Jeff Garzik
On 12/01/2009 08:18 PM, Pete Zaitcev wrote: On Sun, 29 Nov 2009 20:38:45 -0500 Jeff Garzikj...@garzik.org wrote: Interesting... I recall the root cause clearly, now: /usr/include/db.h always refers to the latest installed db4, even if compat-db{,45,46} is installed. Our configure recipe

Re: XDR print?

2009-12-06 Thread Jeff Garzik
On 12/06/2009 07:39 PM, Colin McCabe wrote: I'm wondering if there's a function that will take a buffer full of XDR data and an xdrproc_t, and print out a human readable version of the data. It seems like an obvious debugging tool, but I can't find a reference to anything like this in the XDR

Re: [PATCH 0/6 v2] logging refactoring

2009-12-14 Thread Jeff Garzik
On 12/14/2009 02:23 AM, Colin McCabe wrote: Hi all, I think I'm going to respin this patch to be a lot smaller. Just refactor one or two things, but leave the existing verbose and applog system in place. Let me know what you think-- do you think I should push ahead with HAIL_DEBUG etc.,

Re: [PATCH 1/8 v3] cld: Declare common.c functions in common.h

2009-12-15 Thread Jeff Garzik
On 12/14/2009 07:32 PM, Colin McCabe wrote: Signed-off-by: Colin McCabecmcc...@alumni.cmu.edu applied, after fixing two problems: * did not survive 'make distcheck'. common.h was needed in include/Makefile.am. * introduced build warnings in the server. added #include in server/cld.h.

Re: [PATCH 2/8 v3] cld: create logging macros

2009-12-15 Thread Jeff Garzik
On 12/14/2009 07:32 PM, Colin McCabe wrote: Signed-off-by: Colin McCabecmcc...@alumni.cmu.edu applied, after fixing two minor problems: * added to include/Makefile.am * #include'd stdbool.h -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to

Re: [PATCH 3/8 v3] cld: modify client code to use logging macros

2009-12-15 Thread Jeff Garzik
On 12/14/2009 07:32 PM, Colin McCabe wrote: Signed-off-by: Colin McCabecmcc...@alumni.cmu.edu --- include/cldc.h |5 +-- lib/cldc.c | 98 +++- tools/cldcli.c |2 +- 3 files changed, 43 insertions(+), 62 deletions(-) applied

Re: [PATCH 4/8 v3] cld: modify server code to use logging macros

2009-12-15 Thread Jeff Garzik
On 12/14/2009 07:32 PM, Colin McCabe wrote: Signed-off-by: Colin McCabecmcc...@alumni.cmu.edu applied, with minor changes: * include hail_log.h from server/cld.h, the header file common to all server code. remove individual #include's in each .c file. * fixed up #include of

Re: [PATCH 5/8 v3] cld: modify cld-dns to use logging macros

2009-12-15 Thread Jeff Garzik
On 12/14/2009 07:32 PM, Colin McCabe wrote: Signed-off-by: Colin McCabecmcc...@alumni.cmu.edu --- include/cldc.h |7 ++--- lib/cldc-dns.c | 80 +--- tools/cldcli.c | 12 +--- 3 files changed, 41 insertions(+), 58 deletions(-)

Re: [PATCH 9/8 v3] cld: Export hail_log.h to the include directory

2009-12-15 Thread Jeff Garzik
On 12/14/2009 07:55 PM, Colin McCabe wrote: I almost forgot-- Makefile.am needs to change. Signed-off-by: Colin McCabecmcc...@alumni.cmu.edu --- include/Makefile.am |4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/Makefile.am b/include/Makefile.am index

Re: [PATCH 8/8 v3] tabled: Use hail_log API to call libcldc

2009-12-15 Thread Jeff Garzik
On 12/14/2009 07:32 PM, Colin McCabe wrote: Signed-off-by: Colin McCabecmcc...@alumni.cmu.edu --- server/cldu.c | 14 -- server/tabled.h |1 + 2 files changed, 13 insertions(+), 2 deletions(-) ditto last message, for tabled -- To unsubscribe from this list: send the

Re: [Patch 3/3] tabled: drop addr_af

2009-12-15 Thread Jeff Garzik
On 12/16/2009 12:10 AM, Pete Zaitcev wrote: This flag was is a useless duplication and a fiction prompted by getaddrinfo(). In reality it's not useful even if we decide to run CLD over DCCP. Signed-off-by: Pete Zaitcevzait...@redhat.com --- server/storage.c |8 server/tabled.h

Fedora SRPMs for Hail

2009-12-15 Thread Jeff Garzik
zaitcev and I typically maintain recent versions of cld/chunkd/tabled in Fedora rawhide. This can be a useful source of SRPMs for Fedora 10/11/12, if people find those useful. Usually it takes a few days for the built [S]RPMs to trickle out to the Fedora mirrors. Since we just committed

a boto-works test for tabled?

2009-12-18 Thread Jeff Garzik
Looking at the 'python-boto' package in Fedora, I see it ships with boto's basic test suite. Notably, it ships with a test module that exercises boto's S3 library routines. Is there anyone that would be interested in copying (or directly use)

Re: a boto-works test for tabled?

2009-12-18 Thread Jeff Garzik
On 12/18/2009 12:09 PM, Pete Zaitcev wrote: Since I have no clue about Python, I had trouble making their standard testing harness to work, so I just copied the test into a local file, and appended an invocation like this: from boto.s3.connection import S3Connection from boto.s3.connection

Re: a boto-works test for tabled?

2009-12-18 Thread Jeff Garzik
On 12/18/2009 12:16 PM, Jeff Darcy wrote: On 12/18/2009 12:09 PM, Pete Zaitcev wrote: 1. I fixed the calling format issue long time ago. Mind that you have to add a wildcard for it to work: ; Default port is 8081, we override it with -p 4499 in /etc/sysconfig/cld _cld._udp IN SRV

Re: [Patch 1/1] Chunk: fix stored checksums

2009-12-20 Thread Jeff Garzik
On 12/20/2009 09:45 PM, Pete Zaitcev wrote: Existing code writes checksums of something other than the object data. Fix by summing the object data. Signed-off-by: Pete Zaitcevzait...@redhat.com --- server/object.c |2 +- 1 file changed, 1 insertion(+), 1 deletion(-) This seems too

Re: Design challenges in chunkd self-checking

2009-12-22 Thread Jeff Garzik
On 12/22/2009 04:41 PM, Pete Zaitcev wrote: I'm looking into adding self-checking to chunkd. This involves basically a process that re-reads everything stored in the chunkserver and verifies that it's still ok. Nothing can be simpler, right? So, current problems for which I'd like input are:

Re: Design challenges in chunkd self-checking

2009-12-22 Thread Jeff Garzik
On 12/22/2009 08:40 PM, Pete Zaitcev wrote: On Tue, 22 Dec 2009 17:43:58 -0500 Jeff Garzikj...@garzik.org wrote: It is normal and reasonable to maintain global information about all in-progress operations. Caching systems do that, for example, to ensure multiple cache requests for object A

getsrvinfo(3)

2009-12-23 Thread Jeff Garzik
Working on libcldc's functionality requires moving all cldc_host searching and manipulation out from chunkd/tabled and into libcldc proper. Part of this involves changing the integration points related to SRV record lookups, so as to accomodate input from a file during testing (thus avoiding the

Re: [Patch 3/4] chunkd: make error paths more regular

2009-12-25 Thread Jeff Garzik
On 12/26/2009 01:34 AM, Pete Zaitcev wrote: diff --git a/server/be-fs.c b/server/be-fs.c index 89635f8..6551c83 100644 --- a/server/be-fs.c +++ b/server/be-fs.c @@ -75,7 +75,7 @@ int fs_open(void) } if (!tchdbsetmutex(hdb)) - goto out_hdb; + goto

Re: [Patch 4/4] chunkd: Drop unused forward declaration

2009-12-25 Thread Jeff Garzik
On 12/26/2009 01:34 AM, Pete Zaitcev wrote: Signed-off-by: Pete Zaitcevzait...@redhat.com --- server/chunkd.h |1 - 1 file changed, 1 deletion(-) commit 6fe147ad2f8833e162d92277ec6827520c5497ba Author: Masterzait...@lembas.zaitcev.lan Date: Fri Dec 25 23:18:50 2009 -0700 Unused

Re: [Patch 1/1] chunkd: split up fs_list_objs

2009-12-27 Thread Jeff Garzik
On 12/27/2009 06:58 PM, Pete Zaitcev wrote: This way we create a set of methods that can be used by self-check to list existing objects. Signed-off-by: Pete Zaitcevzait...@redhat.com applied 1-2 quickly, since those were easy to review and apply. I need more time to think about and review

Re: [PATCH] cld: check parameters for HAIL_LOG

2010-01-04 Thread Jeff Garzik
On 01/03/2010 07:45 PM, Colin McCabe wrote: Another attempt at escaping the overly-aggressive spam filters... GCC has a nifty ability to typecheck varargs functions that follow the printf conventions. This change uses that in HAIL_LOG. If you build with -Wall, gcc can now give warnings about

Re: [Patch 2/2] tabled: add a test for larger objects

2010-01-05 Thread Jeff Garzik
On 01/05/2010 02:27 AM, Pete Zaitcev wrote: Existing tests only excercised operations with relatively small objects. It did not test pipelining of object data in sufficient degree. So, let's have a better test case for this (large-object.c). We also change the existing basic-object.c to match.

Re: [Patch 1/2] tabled: fix bugs in streaming of data

2010-01-07 Thread Jeff Garzik
On 01/05/2010 11:02 PM, Pete Zaitcev wrote: On Tue, 05 Jan 2010 04:13:36 -0500 Jeff Garzikj...@garzik.org wrote: applied... chunkd needs these fixes also, yes? No, it appears correct as-is. Although, the large-object test in the suite does not check the data it transmits, so if chunkd were

Re: [PATCH] tabled: use new cld api

2010-01-11 Thread Jeff Garzik
On 01/10/2010 10:22 AM, Colin McCabe wrote: Signed-off-by: Colin McCabecmcc...@alumni.cmu.edu A very similar patch is needed for chunkd's server/cldu.c... did my spam filter eat it? Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a

Re: [PATCH] cld: use XDR for all messages

2010-01-11 Thread Jeff Garzik
On 01/10/2010 10:00 AM, Colin McCabe wrote: This patch moves CLD from using manual data serialization to using XDR (via rpcgen). Both the packet header and the message body are now serialized and deserialized using XDR. This makes it easy to have a variable-length packet header, as well as a

Re: [PATCH 2/6] chunkd: change the prefix length of object pathname from 4 to 3

2010-01-11 Thread Jeff Garzik
On 01/11/2010 07:25 AM, Akinobu Mita wrote: The volume directory may have 2^16 (=65536) directories because prefix length of object pathname is 4 bytes. But ext3 filesystem can only have EXT_LINK_MAX (32000) directories. This changes the prefix length from 4 bytes to 3 bytes and breaks the

Re: [Patch 4/4] chunkd: add self-checking

2010-01-12 Thread Jeff Garzik
On 12/27/2009 06:59 PM, Pete Zaitcev wrote: This patch adds a background process that periodically verifies the integrity of all known objects. Objects that are found found faulty are made invisible to applications. This way, only known good objects remain visible and applications may implement

Re: [PATCH 6/6] chunkd: fix PUT/GET for more than 2GB values

2010-01-12 Thread Jeff Garzik
On 01/11/2010 07:25 AM, Akinobu Mita wrote: - Define _FILE_OFFSET_BITS - Use uint64_t to hold object length and buffer length - Don't pass too much buffer to writev. writev returns -1 if the sum of the iov_len values overflows an ssize_t value Signed-off-by: Akinobu

Re: [PATCH] tabled: use LL to indicate 64-bit constant

2010-01-12 Thread Jeff Garzik
On 01/12/2010 01:25 PM, Colin McCabe wrote: util.c: In function ‘objid_init’: util.c:331: warning: integer constant is too large for ‘long’ type with [cmcc...@stargazer tabled]$ gcc --version gcc (GCC) 4.4.1 20090725 (Red Hat 4.4.1-2) Now that I wrote a little test program, I can see that gcc

Re: [PATCH 2/6] chunkd: change the prefix length of object pathname from 4 to 3

2010-01-13 Thread Jeff Garzik
On 01/12/2010 10:50 PM, Akinobu Mita wrote: This patch makes sense, but it also raises the question of whether or not we should move to a two-level directory scheme, eg. 123/456/7890ABCDEF rather than 123/4567890ABCDEF to limit the size of the top-level directories. It

Re: [PATCH] cld: use XDR for all messages

2010-01-13 Thread Jeff Garzik
On 01/10/2010 10:00 AM, Colin McCabe wrote: This patch moves CLD from using manual data serialization to using XDR (via rpcgen). Both the packet header and the message body are now serialized and deserialized using XDR. This makes it easy to have a variable-length packet header, as well as a

Re: the evils of the CLD api

2010-01-18 Thread Jeff Garzik
My general comment on the libcldc API is that it is a low-level API, and definitely not very user friendly at all. ncld or other APIs would improve life quite a bit. Making life easier for programmers, and cutting out tons of code, is a plus :) Jeff, just returned from camping

Re: [Patch 1/7] tabled: cleanup tabled.conf

2010-01-20 Thread Jeff Garzik
On 01/14/2010 11:03 PM, Pete Zaitcev wrote: The lib/ is a vestige of days when we stored files in a local filesystem, not Chunk. Just kill it. And as for CLD, it's a last resort configuration, not something we should encourage. So comment it out. Signed-Off-By: Pete Zaitcevzait...@redhat.com

Re: [Patch 2/7] tabled: add Cell element

2010-01-20 Thread Jeff Garzik
On 01/14/2010 11:11 PM, Pete Zaitcev wrote: Make the cell configurable. Among other things we need this for is that you cannot migrate a tabled over when DB format changes, short of running 2 CLDs. We also add a few thoughts to the documentation, prompted by re-reading of the document while

Re: [Patch 3/7] tabled: add the status display

2010-01-20 Thread Jeff Garzik
On 01/14/2010 11:12 PM, Pete Zaitcev wrote: I find it too tiresome to monitor the status with kill -USR1. This is more fun. The patch seems not too ugly regarding the factoring of the cli_xxx methods. Signed-Off-By: Pete Zaitcevzait...@redhat.com --- doc/etc.tabled.conf |3

Re: [Patch 4/7] tabled: retry conflicting locks

2010-01-20 Thread Jeff Garzik
On 01/14/2010 11:13 PM, Pete Zaitcev wrote: This problem was with us for a while, and even with this fix our start-up is not reliable. But at least we will not be 100% guaranteed to hang as before when restarting too quickly. So although the whole area needs some serious reworking, this specific

Re: [Patch 5/7] tabled: suppress repeating messages

2010-01-20 Thread Jeff Garzik
On 01/14/2010 11:13 PM, Pete Zaitcev wrote: Every time CLD crashes, I find my logs filled up with error messages. But running without them is no good either, so I don't want to remove this message entirely. Let us try some filtering. Actually, I am still considering if we should not reset the

Re: [PATCHv2 1/2] cld: fix CLD_INODE_NAME_MAX woes

2010-02-03 Thread Jeff Garzik
On 02/03/2010 08:45 AM, Colin McCabe wrote: When we create a static buffer for an inode name, and treat it like a null-terminated string, it needs to be of length CLD_INODE_NAME_MAX + 1 so that it can hold the NULL-terminator. In cldc_del and cldc_open, we should check that the user-submitted

Re: [Patch 4/7] tabled: retry conflicting locks

2010-02-03 Thread Jeff Garzik
On 01/20/2010 05:56 PM, Pete Zaitcev wrote: Is there a way to cancel an outstanding lock request? How? You seem to think that there's no problem. Actually I think an cmo_close on a handle that has outstanding requests of any kind should drop them, so I was incorrect about killing the session

Re: [PATCH] Re: [PATCHv2] cld: use XDR for all messages

2010-02-05 Thread Jeff Garzik
On 02/03/2010 04:45 PM, Colin McCabe wrote: On Tue, Feb 2, 2010 at 10:35 PM, Jeff Garzikj...@garzik.org wrote: I will continue whittling down the patch until it just contains the XDR changes themselves. In tools/Makefile.am, I don't think you need $(top_srcdir)/lib any more, since

Re: Post-XDR CLD cannot keep session up

2010-02-07 Thread Jeff Garzik
On 02/07/2010 02:00 AM, Pete Zaitcev wrote: Hi, Jeff Colin: It looks like you broke something in CLD, not sure if server or client. There are two possibly related bugs. But first, here's the messages (The chunkd is run with -D). Note that I have 2 servers listed in DNS (both on port 4499), but

Re: [Patch 1/1] CLD: Introduce the New CLD API

2010-02-08 Thread Jeff Garzik
On 02/07/2010 02:15 PM, Pete Zaitcev wrote: The traditional CLD API is too difficult to program. In particular, switching from a rigid Group policy to arbitrary paths in existing clients in Chunk and tabled turned out to be next to impossible. The issue is due to the fundamental nature of the

Re: [Patch 1/1] CLD: Introduce the New CLD API

2010-02-08 Thread Jeff Garzik
On 02/08/2010 08:46 PM, Pete Zaitcev wrote: On Mon, 08 Feb 2010 07:32:56 -0500 Jeff Garzikj...@garzik.org wrote: - log param in cldc-dns should not be made conditional as an undocumented side effect of ncld I pulled, and it appears that you merged a code that crashes at the first error

Re: Does this work on BSD too?

2010-02-08 Thread Jeff Garzik
On 02/08/2010 08:49 PM, Pete Zaitcev wrote: When I run autogen.sh, this message is printed: lib/Makefile.am:13: `%'-style pattern rules are a GNU make extension It's because of this: %_rpc.h: %_rpc.x rpcgen -h $$@ Is the above a concern? It is likely a concern, yes. For

Re: [Patch 1/1] CLD: Introduce the New CLD API

2010-02-08 Thread Jeff Garzik
On 02/08/2010 09:17 PM, Pete Zaitcev wrote: On Mon, 08 Feb 2010 21:06:17 -0500 Jeff Garzikj...@garzik.org wrote: - if (cldc_getaddr(host_list, hostb, NULL)) + if (cldc_getaddr(host_list, hostb,ncld_log)) return 1001; Logging pointer should be supplied by the

Re: [Patch 1/1] tabled: switch to ncld

2010-02-08 Thread Jeff Garzik
On 02/08/2010 11:42 PM, Pete Zaitcev wrote: No new function just yet, only a switch-over. Signed-Off-By: Pete Zaitcevzait...@redhat.com --- server/cldu.c | 789 +--- 1 file changed, 215 insertions(+), 574 deletions(-) Looks good, but needs a

Re: Post-XDR CLD cannot keep session up

2010-02-09 Thread Jeff Garzik
On 02/09/2010 05:34 AM, Jeff Garzik wrote: On 02/07/2010 02:00 AM, Pete Zaitcev wrote: Hi, Jeff Colin: It looks like you broke something in CLD, not sure if server or client. There are two possibly related bugs. But first, here's the messages (The chunkd is run with -D). Note that I have 2

Re: [Patch 1/1] CLD: Introduce the New CLD API

2010-02-17 Thread Jeff Garzik
; exit(0); commit 1c7d8de3e44e641651f766d6863780343ecaa5a8 Author: Jeff Garzik j...@garzik.org Date: Wed Feb 17 20:14:04 2010 -0500 Cosmetic renaming related to ncld API. s/fhp/fh/ s/ses/sess/ s/nsp/nsess/ And use a more standard way of calling callbacks

[PATCH] tabled: fix key corruption

2010-03-06 Thread Jeff Garzik
commit a58d541cf4357ab0aa7ae5e21765de35f65c6177 Author: Jeff Garzik j...@garzik.org Date: Sat Mar 6 17:44:51 2010 -0500 tabled: fix key corruption due to string overrun When obtaining the path (and thus, the object key) from the URI, it went unnoticed that the URI-based

[PATCH] tabled: disable debug output in tests

2010-03-07 Thread Jeff Garzik
commit cb0ed2014b5aa68e6d0a2ffdad031b2d9ee31159 Author: Jeff Garzik j...@garzik.org Date: Sun Mar 7 07:56:55 2010 -0500 [test] start-daemon: disable tabled debug output by default Signed-off-by: Jeff Garzik jgar...@redhat.com diff --git a/test/start-daemon b/test/start-daemon

Re: [PATCH] chunkd: fix debug output when the request key is not null-terminated

2010-03-15 Thread Jeff Garzik
On 03/15/2010 10:02 PM, Akinobu Mita wrote: If the request key is not null-terminated and the debug output is enabled, it prints invalid and unsafe value. Add a precision to fix it. Signed-off-by: Akinobu Mitaakinobu.m...@gmail.com --- server/server.c |3 ++- 1 files changed, 2

Re: [PATCH] chunkd: fix duplicate stc_object allocation in stc_parse_key()

2010-03-16 Thread Jeff Garzik
On 03/16/2010 05:59 AM, Akinobu Mita wrote: At the beginning of stc_parse_key(), st_object is allocated twice for the same variable. Signed-off-by: Akinobu Mitaakinobu.m...@gmail.com --- lib/chunkdc.c |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) good catch, applied -- To

Re: [Patch 1/1] tabled: fix a crash when looking up non-existing NID

2010-03-29 Thread Jeff Garzik
On 03/28/2010 09:57 PM, Pete Zaitcev wrote: Signed-off-by: Pete Zaitcevzait...@redhat.com --- server/storage.c |3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) applied -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to

Re: CLD doesn't build on db-4.3

2010-04-01 Thread Jeff Garzik
On 04/01/2010 07:01 AM, Samba - BoYang wrote: hi, * CLD doesn't build on db-4.3 on suse 11, since db-4.3 uses deprecated structure members DBC-c_xxx(c_close(), etc) instead of DBC-xxx. :-) It won't build on db-4.4, either. probably won't build on db-4.5, as db-5.0 says DBC-xxx was

Re: [Patch 1/3] CLD: End-to-end verbosity

2010-04-06 Thread Jeff Garzik
On 03/31/2010 08:43 PM, Pete Zaitcev wrote: diff --git a/server/server.c b/server/server.c index 3208e0f..2d68ee6 100644 --- a/server/server.c +++ b/server/server.c @@ -55,7 +55,7 @@ static struct argp_option options[] = { Store database environment in DIRECTORY. Default:

  1   2   >