erasing
truth of that great pr -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hail license changed
Hail license change was just pushed to the github hail repository. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Hail status and update (was Re: Question about hail)
Hi, Thank you for your mail, Pete and Jeff :) And also late to reply since tons of task and mail makes me busy (and a bit sick). On Tue, 27 Nov 2012 14:47:38 -0500 Jeff Garzik jgar...@pobox.com wrote: Q: Is this project is still alive? if so, where is the current main site. Could you tell me the status, please? The main site for source code is currently github. https://github.com/jgarzik/hail https://github.com/jgarzik/tabled https://github.com/jgarzik/itd https://github.com/jgarzik/nfs4d okay. Hail was collateral damage in a kernel.org hack. No data was lost or compromised, but it took kernel.org months to recover even basic account services and git access. wikis took months longer after that. I'm still waiting to see if anybody has an archive of old tarballs, because k.org was my canonical upstream storage location, with zero local ones. Hmm, I've expected it due to kernel.org compromise issue, terrible. Are kernel.org administrators still working? I know that restoring services and accounts are not easy task, and they also need to harden their system, but took _months_ is not usual... maybe ping to them is good. I've asked ftpadmin@ about hail archive before ask to you since it's described in top page, but got no reply :-( The tarballs can conceivably be recovered by checking out a git tag, re-running autogen.sh, and then make dist... but with autoconf/automake/libtool upgrades over the years, tarball checksums might change using that method. okay. History: hail was GPL-2 only, following the lead of the kernel. But it sounds like this is impractical given the few existing users, so I am in favor of relicensing to GPL-2 or later. :) I disagree with the interpretation vis a vis openssl and several others do too. However, if this is an impediment to use, I would be happy to accept a pull request adding the openssl exemption language. Probably ftpmasters in Debian archive would deny without it. Okay I'll do it later, thanks! -- Regards, Hideki Yamane henrich @ debian.or.jp/org http://wiki.debian.org/HidekiYamane -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hail status and update (was Re: Question about hail)
(CC'd hail-devel list) On 11/26/2012 02:28 AM, Hideki Yamane wrote: Hello hail upstream authors, I'm interested in porting hail (and Aeolus) to Debian, but have some questions about it. Cool! Q: Is this project is still alive? if so, where is the current main site. Could you tell me the status, please? The main site for source code is currently github. https://github.com/jgarzik/hail https://github.com/jgarzik/tabled https://github.com/jgarzik/itd https://github.com/jgarzik/nfs4d The Hail home page is https://hail.wiki.kernel.org/ but that clearly has some stale links on it (see below). The status is definitely long pause at the moment, as it got de-prioritized by my employer, but I still have a goal of completing the Paxos implementation in CLD, making it truly distributed. Patches are still accepted, and I want it to keep working on current platforms. - its upstream site hail.wiki.kernel.org points out some materials but those are empty at all. slide audio: http://www.kernel.org/pub/media/talks/hail/ git repo: http://git.kernel.org/?p=daemon/distsrv/itd.git source: http://www.kernel.org/pub/software/network/distsrv/ Hail was collateral damage in a kernel.org hack. No data was lost or compromised, but it took kernel.org months to recover even basic account services and git access. wikis took months longer after that. I'm still waiting to see if anybody has an archive of old tarballs, because k.org was my canonical upstream storage location, with zero local ones. Local git cryptographic integrity -- the canonical root of all the source code -- never disappeared, and was always verified and not-hacked. It briefly moved to github, but is now back at kernel.org. The tarballs can conceivably be recovered by checking out a git tag, re-running autogen.sh, and then make dist... but with autoconf/automake/libtool upgrades over the years, tarball checksums might change using that method. and there's no mail in http://news.gmane.org/gmane.comp.distributed.hail.devel except spam. Pete complained about this too. Not sure what to do about it -- the mail server is vger.kernel.org, same as LKML, with a postmaster who maintains the spam filter to similar setup and standards. I was surprised at Pete's spam report at first, too. I never see any of the spam, only the rare hail-devel message, because of good spam filtering. Of course, the M-L archiving bots seem to keep it all. Q: Can you add special exemptions for OpenSSL to its license, and add or later, too? - Its license is GPL-2, right? and it links to OpenSSL. However, openssl license is NOT compatible with GPL without special exemptions see http://lintian.debian.org/tags/possible-gpl-code-linked-with-openssl.html and http://www.openssl.org/support/faq.html#LEGAL2 So, we cannot distribute hail binary without its special exemptions. - GPL-2 and GPL-3 is NOT compatible. However, Image Warehouse (part of Aeolus) is licensed under GPL-3, and it seems to link to hail library (hstor.h). We should change a) hail to GPL-2 or later or GPL-3 (or later) b) iwhd to GPL-2 to avoid license incompatibility. History: hail was GPL-2 only, following the lead of the kernel. But it sounds like this is impractical given the few existing users, so I am in favor of relicensing to GPL-2 or later. I disagree with the interpretation vis a vis openssl and several others do too. However, if this is an impediment to use, I would be happy to accept a pull request adding the openssl exemption language. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Saia do SERASA/SPC!!!
Você possuí nome sujo no SPC/SERASA e deseja limpar o mesmo? Limpe agora mesmo! Agora você pode limpar seu nome sem pagar a divida toda. Trabalhamos com Seriedade e cumprimos o prometido. Envie ja um email soilicitando informações para:limpan...@r7.com -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Project Hail wikis alive again!
kernel.org fixed their wiki system, which means that all the k.org wikis are once again read-write! This includes Project Hail's home page, https://hail.wiki.kernel.org/ I hope to have the git repos moved back from https://github.com/jgarzik/ to kernel.org soon also. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Server Rental Service in HK
Dear All, We have our own datacenter in Hong Kong provide email/application/web rental service to clients.We are APNIC member provide clean IP to clients. Dell? PowerEdge? EnterpriseRack Mount Server -Intel(R) Xeon(R) E3-1240 Processor (3.3GHz, 8M Cache, Turbo, 4C/8T, 80W) -8GB RAM, 2x4GB, 1333MHz, DDR-3, Dual Ranked UDIMMs -500GB, 3.5, 6Gbps SAS x 2 -Raid 1 Mirroring Protection -Remote KVM (iDRAC6 Enterprise) Dell(TM) PowerEdge(TM) R410 Rack Mount Server -Intel(R) Quad Core E5606 Xeon(R) CPU, 2.13GHz, 4M Cache, 4.86 GT/s QPI -4GB Memory (2x2GB), 1333MHz Dual Ranked RDIMMs Fully-Buffered -500GB 7.2K RPM SATAII 3.5 Hard Drive x 2 -iDRAC6 Enterprise or Express (Remote KVM Management) Every Dedicated Server Hosting Solution Also Includes: Software Specification - CentOS / Fedora / Debian / FreeBSD / Ubuntu / Redhat Linux - Full root-level access - Data Center Facilities - Shared Local International Bandwidth - 2 IP Addresses Allocation - Un-interruptible Power Supply (UPS) backed up by private diesel generator - FM200¡§based fire suppression system - 24x7 CRAC Air Conditioning and Humidity Control - 24x7 Security Control - 24x7 Remote Hand Service Pls send us email for further information.Thanks, Boris bo...@cloudluca.com If you do not wish to further receive this event message, email borislam...@gmail.com to unsubscribe this message or remove your email from the list. -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Unable to log into Hail Wiki
On 01/25/2012 08:40 PM, Pete Zaitcev wrote: Jeff, looks like the wiki rots. The login points to this URL https://hail.wiki.kernel.org/articles/u/s/e/Special%7EUserLogin_94cd.html It returns 404. HALP? Yes -- all kernel.org wikis are _still_ read-only, even this many months after the kernel.org breakin. ata.wiki.kernel.org has similar behavior. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Unable to log into Hail Wiki
Jeff, looks like the wiki rots. The login points to this URL https://hail.wiki.kernel.org/articles/u/s/e/Special%7EUserLogin_94cd.html It returns 404. HALP? -- Pete -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch hail 1/1] Plug leak in hstor_parse_key
On 10/14/2011 01:34 PM, Pete Zaitcev wrote:
Signed-off-by: Pete Zaitcevzait...@kotori.zaitcev.us
---
lib/hstor.c |2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/hstor.c b/lib/hstor.c
index cb9c4da..5ce9b76 100644
--- a/lib/hstor.c
+++ b/lib/hstor.c
@@ -761,7 +761,7 @@ void hstor_free_keylist(struct hstor_keylist *keylist)
static void hstor_parse_key(xmlDocPtr doc, xmlNode *node,
struct hstor_keylist *keylist)
{
- struct hstor_object *obj = calloc(1, sizeof(*obj));
+ struct hstor_object *obj;
xmlChar *xs;
good catch... applied
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[patch hail 1/1] Plug leak in hstor_parse_key
Signed-off-by: Pete Zaitcev zait...@kotori.zaitcev.us
---
lib/hstor.c |2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/hstor.c b/lib/hstor.c
index cb9c4da..5ce9b76 100644
--- a/lib/hstor.c
+++ b/lib/hstor.c
@@ -761,7 +761,7 @@ void hstor_free_keylist(struct hstor_keylist *keylist)
static void hstor_parse_key(xmlDocPtr doc, xmlNode *node,
struct hstor_keylist *keylist)
{
- struct hstor_object *obj = calloc(1, sizeof(*obj));
+ struct hstor_object *obj;
xmlChar *xs;
obj = calloc(1, sizeof(*obj));
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hail git tree location
Are we going to have a git tree somewhere? It looks like our old one was purged from git.kernel.org. -- P -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Hail git tree location
On 10/04/2011 12:12 PM, Pete Zaitcev wrote:
Are we going to have a git tree somewhere? It looks like our old one
was purged from git.kernel.org.
Sorry, I should have posted. It was migrated along with the kernel.org
trees to
https://github.com/jgarzik/{hail,tabled,itd,nfs4d}
though kernel.org ones should be coming back.
Jeff
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
FYI, rpc/ is gone from Fedora 15
FYI, /usr/include/rpc/ no longer exists, as of F15's glibc-headers-2.13.90-10, so hail's lib/cld_msg_rpc.h will have to do something about this #include: $ grep rpc.h lib/cld_msg_rpc.h #include rpc/rpc.h -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: FYI, rpc/ is gone from Fedora 15
On 05/05/2011 10:14 AM, Jim Meyering wrote: FYI, /usr/include/rpc/ no longer exists, as of F15's glibc-headers-2.13.90-10, so hail's lib/cld_msg_rpc.h will have to do something about this #include: $ grep rpc.h lib/cld_msg_rpc.h #includerpc/rpc.h hm. Surely they did not delete sunrpc from glibc? That would be disappointing. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: FYI, rpc/ is gone from Fedora 15
On Thu, 05 May 2011 11:34:42 -0400 Jeff Garzik j...@garzik.org wrote: On 05/05/2011 10:14 AM, Jim Meyering wrote: FYI, /usr/include/rpc/ no longer exists, as of F15's glibc-headers-2.13.90-10, so hail's lib/cld_msg_rpc.h will have to do something about this #include: $ grep rpc.h lib/cld_msg_rpc.h #includerpc/rpc.h hm. Surely they did not delete sunrpc from glibc? Surely they did (and by they I mean Uli, obviously). The suggested fix is to use TI-RPC, whatever that is. -- Pete -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[hail] CLD conversion to TCP lands in git
I just pushed the CLD protocol change (UDP - TCP) to hail.git[1]. See the original post[2] for more details. It seems pretty solid from my beating on it, but it's still raw code. The focus will be on hammering out the kinks in this switch over the next 7-10 days, so expect some breakage and churn during that time. In particular, while tabled /should/ work, as the API has only seen minor changes, it will need a good stress test before I regain confidence in it. There was also an unrelated-to-CLD API change in libhail that requires a small tabled tweak[3], that will be attended-to this week. Jeff [1] http://git.kernel.org/?p=daemon/distsrv/hail.git;a=commit;h=3bdeaab68e1c2776a3488ac03f49f7b4bc2659c8 [2] http://marc.info/?l=hail-develm=129379489716486w=2 [3] http://git.kernel.org/?p=daemon/distsrv/hail.git;a=commit;h=59becbb9e329cdc20e4894f331fcb8dfc104c35a -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/3] CLD: switch network proto from UDP to TCP
On 01/02/2011 06:32 PM, Pete Zaitcev wrote: On Fri, 31 Dec 2010 05:57:28 -0500 Jeff Garzikj...@garzik.org wrote: + struct cldc_tcp *tcp = private; + ssize_t rc; + struct ubbp_header ubbp; + + memcpy(ubbp.magic, CLD1, 4); + ubbp.op_size = (buflen 8) | 1; +#ifdef WORDS_BIGENDIAN + swab32(ubbp.op_size); +#endif + + rc = write(tcp-fd,ubbp, sizeof(ubbp)); Why not this: unsigned int n; n = (buflen 8) | 1; ubbp.op_size = GUINT32_TO_LE(n); Yep. I used the #ifdef on the read(2) side, where I did not want to create an additional var... then I copied that onto the write(2) side, where it is less efficient as you point out. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Crash with db5
Looks like Rawhide throws this if libdb-devel is in use: make check-TESTS make[3]: Entering directory `/q/zaitcev/hail/hail-tip/test/cld' PASS: prep-db DB_ENV-lsn_reset: method not permitted before handle's open method DB_ENV-dbremove: method not permitted before handle's open method cld[11548]: SIGSEGV PASS: start-daemon port file not found. FAIL: pid-exists libdb-5.1.19-2.fc15.x86_64 -- Pete -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Crash with db5
On 01/02/2011 08:20 PM, Pete Zaitcev wrote: Looks like Rawhide throws this if libdb-devel is in use: make check-TESTS make[3]: Entering directory `/q/zaitcev/hail/hail-tip/test/cld' PASS: prep-db DB_ENV-lsn_reset: method not permitted before handle's open method DB_ENV-dbremove: method not permitted before handle's open method cld[11548]: SIGSEGV PASS: start-daemon port file not found. FAIL: pid-exists libdb-5.1.19-2.fc15.x86_64 Are you compiling with db4 headers, but linking with db5? Or vice versa? This is a problem I ran into, with F14. hail's configure script searches for the first libdb, which will always be libdb5 on = F14, because libdb5 is always installed due to dependencies. But... you can either have db4-devel or libdb-devel installed for the devel pkg. If you have db4-devel + libdb5... boom. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Crash with db5
On Sun, 02 Jan 2011 20:44:02 -0500 Jeff Garzik j...@garzik.org wrote: DB_ENV-dbremove: method not permitted before handle's open method cld[11548]: SIGSEGV Are you compiling with db4 headers, but linking with db5? Or vice versa? It turned out that Rawhide revved up the db5, so this band-aids it: commit 42d083416656e195baa79a924df0f11a3f0681c2 Author: Pete Zaitcev zait...@yahoo.com Date: Sun Jan 2 20:28:39 2011 -0700 Accept db-5.1. diff --git a/configure.ac b/configure.ac index 9cfad23..abc9330 100644 --- a/configure.ac +++ b/configure.ac @@ -75,7 +75,8 @@ dnl AC_TYPE_PID_T dnl - dnl Checks for required libraries dnl - -AC_CHECK_LIB(db-5.0, db_create, DB4_LIBS=-ldb-5.0, +AC_CHECK_LIB(db-5.1, db_create, DB4_LIBS=-ldb-5.1, + AC_CHECK_LIB(db-5.0, db_create, DB4_LIBS=-ldb-5.0, AC_CHECK_LIB(db-4.9, db_create, DB4_LIBS=-ldb-4.9, AC_CHECK_LIB(db-4.8, db_create, DB4_LIBS=-ldb-4.8, AC_CHECK_LIB(db-4.7, db_create, DB4_LIBS=-ldb-4.7, @@ -83,7 +84,7 @@ AC_CHECK_LIB(db-5.0, db_create, DB4_LIBS=-ldb-5.0, AC_CHECK_LIB(db-4.5, db_create, DB4_LIBS=-ldb-4.5, AC_CHECK_LIB(db-4.4, db_create, DB4_LIBS=-ldb-4.4, AC_CHECK_LIB(db-4.3, db_create, DB4_LIBS=-ldb-4.3, - [AC_MSG_ERROR([Missing required libdb 4.x])] + [AC_MSG_ERROR([Missing required libdb 4.x])]) AC_CHECK_LIB(event, event_base_new, EVENT_LIBS=-levent, [AC_MSG_ERROR([Missing required libevent])]) AC_CHECK_LIB(crypto, MD5_Init, CRYPTO_LIBS=-lcrypto) This needs to stop eventually, just don't know how. -- Pete -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/3] CLD: convert back to libevent
Switch CLD from hand-rolled server poll code, to libevent. Follows
similar techniques and rationale as chunkd commit
c1aed7464f237e5a6309351bf003162c77d69e27. This reverts ancient commit
90b3b5edcf5aa00577f4395fdbb490ed7e9be824.
Signed-off-by: Jeff Garzik jgar...@redhat.com
---
cld/Makefile.am |3 -
cld/cld.h | 22 +++
cld/server.c| 161
cld/session.c | 69
4 files changed, 118 insertions(+), 137 deletions(-)
diff --git a/cld/Makefile.am b/cld/Makefile.am
index 9a13ce0..30eea0b 100644
--- a/cld/Makefile.am
+++ b/cld/Makefile.am
@@ -12,7 +12,8 @@ cld_SOURCES = cldb.h cld.h \
cldb.c msg.c server.c session.c util.c
cld_LDADD = \
../lib/libhail.la @GLIB_LIBS@ @CRYPTO_LIBS@ \
- @SSL_LIBS@ @DB4_LIBS@ @XML_LIBS@ @LIBCURL@
+ @SSL_LIBS@ @DB4_LIBS@ @XML_LIBS@ @LIBCURL@ \
+ @EVENT_LIBS@
cldbadm_SOURCES= cldb.h cldbadm.c
cldbadm_LDADD = @CRYPTO_LIBS@ @GLIB_LIBS@ @DB4_LIBS@
diff --git a/cld/cld.h b/cld/cld.h
index 4c0099f..17f14b8 100644
--- a/cld/cld.h
+++ b/cld/cld.h
@@ -22,8 +22,9 @@
#include netinet/in.h
#include sys/time.h
-#include poll.h
+#include event.h
#include glib.h
+#include elist.h
#include cldb.h
#include cld_msg_rpc.h
#include cld_common.h
@@ -59,13 +60,13 @@ struct session {
uint64_tlast_contact;
uint64_tnext_fh;
- struct cld_timertimer;
+ struct eventtimer;
uint64_tnext_seqid_in;
uint64_tnext_seqid_out;
GList *out_q; /* outgoing pkts (to client) */
- struct cld_timerretry_timer;
+ struct eventretry_timer;
charuser[CLD_MAX_USERNAME];
@@ -85,10 +86,10 @@ struct server_stats {
unsigned long garbage;/* num. garbage pkts dropped */
};
-struct server_poll {
+struct server_socket {
int fd;
- bool(*cb)(int fd, short events, void *userdata);
- void*userdata;
+ struct eventev;
+ struct list_headsockets_node;
};
struct server {
@@ -103,14 +104,13 @@ struct server {
struct cldb cldb; /* database info */
- GArray *polls;
- GArray *poll_data;
+ struct event_base *evbase_main;
- GHashTable *sessions;
+ struct list_headsockets;
- struct cld_timer_list timers;
+ GHashTable *sessions;
- struct cld_timerchkpt_timer;/* db4 checkpoint timer */
+ struct eventchkpt_timer;/* db4 checkpoint timer */
struct server_stats stats; /* global statistics */
};
diff --git a/cld/server.c b/cld/server.c
index 7a57785..aed501b 100644
--- a/cld/server.c
+++ b/cld/server.c
@@ -559,7 +559,7 @@ static void simple_sendresp(int sock_fd, const struct
client *cli,
info-op);
}
-static bool udp_srv_event(int fd, short events, void *userdata)
+static void udp_srv_event(int fd, short events, void *userdata)
{
struct client cli;
char host[64];
@@ -586,7 +586,7 @@ static bool udp_srv_event(int fd, short events, void
*userdata)
rrc = recvmsg(fd, hdr, 0);
if (rrc 0) {
syslogerr(UDP recvmsg);
- return true; /* continue main loop; do NOT terminate server */
+ return;
}
cli.addr_len = hdr.msg_namelen;
@@ -601,59 +601,60 @@ static bool udp_srv_event(int fd, short events, void
*userdata)
if (!parse_pkt_header(raw_pkt, rrc, pkt, hdr_len)) {
cld_srv.stats.garbage++;
- return true;
+ return;
}
if (!get_pkt_info(pkt, raw_pkt, rrc, hdr_len, info)) {
xdr_free((xdrproc_t)xdr_cld_pkt_hdr, (char *)pkt);
cld_srv.stats.garbage++;
- return true;
+ return;
}
if (packet_is_dupe(info)) {
/* silently drop dupes */
xdr_free((xdrproc_t)xdr_cld_pkt_hdr, (char *)pkt);
- return true;
+ return;
}
err = validate_pkt_session(info, cli);
if (err) {
simple_sendresp(fd, cli, info, err);
xdr_free((xdrproc_t)xdr_cld_pkt_hdr, (char *)pkt);
- return true;
+ return;
}
err = pkt_chk_sig(raw_pkt, rrc, pkt);
if (err) {
simple_sendresp(fd, cli, info, err);
xdr_free((xdrproc_t)xdr_cld_pkt_hdr, (char *)pkt);
- return true;
+ return;
}
if (!(cld_srv.cldb.is_master
[PATCH 2/3] CLD: switch network proto from UDP to TCP
Convert CLD network protocol from UDP to TCP. Server, client lib,
and chunkd's cldu module are all updated. tabled's cldu module must
be updated also.
The original rationale for UDP use was following Google's lead, based
on the advice in the original Chubby paper, describing TCP's back-off
policies and other behavior during times of high network congestion.
This seems a bit dubious without further third party evidence, and TCP
vastly simplifies our lives. While the code remains open and modular
enough to support other protocols (hopefully RDMA or SCTP one day),
this upgrade from UDP to TCP promises to make the current codebase
much easier to use, while avoiding the reinvent TCP, by using UDP
problem, which was a rabbit hole threatening CLD.
Signed-off-by: Jeff Garzik jgar...@redhat.com
---
chunkd/cldu.c|6
cld/cld.h| 43 ++
cld/msg.c|4
cld/server.c | 356 ---
cld/session.c|4
configure.ac |1
include/Makefile.am |2
include/cld_common.h |4
include/cldc.h | 24 ++-
include/ncld.h |4
include/ubbp.h | 52 +++
lib/Makefile.am |2
lib/cldc-dns.c |2
lib/cldc-tcp.c | 185 ++
lib/cldc-udp.c | 141
lib/cldc.c | 54 +++
16 files changed, 595 insertions(+), 289 deletions(-)
diff --git a/chunkd/cldu.c b/chunkd/cldu.c
index 026c523..41f94b5 100644
--- a/chunkd/cldu.c
+++ b/chunkd/cldu.c
@@ -165,7 +165,7 @@ static void cldu_sess_event(void *priv, uint32_t what)
*/
if (cs-nsess) {
applog(LOG_ERR, Session failed, sid SIDFMT,
- SIDARG(cs-nsess-udp-sess-sid));
+ SIDARG(cs-nsess-tcp-sess-sid));
} else {
applog(LOG_ERR, Session open failed);
}
@@ -177,7 +177,7 @@ static void cldu_sess_event(void *priv, uint32_t what)
} else {
if (cs)
applog(LOG_INFO, cldc event 0x%x sid SIDFMT,
- what, SIDARG(cs-nsess-udp-sess-sid));
+ what, SIDARG(cs-nsess-tcp-sess-sid));
else
applog(LOG_INFO, cldc event 0x%x no sid, what);
}
@@ -372,7 +372,7 @@ static int cldu_set_cldc(struct cld_session *cs, int
newactive)
}
applog(LOG_INFO, New CLD session created, sid SIDFMT,
- SIDARG(cs-nsess-udp-sess-sid));
+ SIDARG(cs-nsess-tcp-sess-sid));
/*
* First, make sure the base directory exists.
diff --git a/cld/cld.h b/cld/cld.h
index 17f14b8..b1f9bbf 100644
--- a/cld/cld.h
+++ b/cld/cld.h
@@ -30,6 +30,7 @@
#include cld_common.h
#include hail_log.h
#include hail_private.h
+#include ubbp.h
struct client;
struct session_outpkt;
@@ -43,10 +44,39 @@ enum {
SFL_FOREGROUND = (1 0), /* run in foreground */
};
+struct atcp_read {
+ void*buf;
+ unsigned intbuf_size;
+ unsigned intbytes_wanted;
+ unsigned intbytes_read;
+
+ void(*cb)(void *, bool);
+ void*cb_data;
+
+ struct list_headnode;
+};
+
+struct atcp_read_state {
+ struct list_headq;
+};
+
struct client {
+ int fd;
+
+ struct eventev;
+ short ev_mask;/* EV_READ and/or EV_WRITE */
+
struct sockaddr_in6 addr; /* inet address */
socklen_t addr_len; /* inet address len */
charaddr_host[64]; /* ASCII version of inet addr */
+ charaddr_port[16]; /* ASCII version of inet addr */
+
+ struct atcp_read_state rst;
+
+ struct ubbp_header ubbp;
+
+ charraw_pkt[CLD_RAW_MSG_SZ];
+ unsigned intraw_size;
};
struct session {
@@ -124,6 +154,17 @@ struct pkt_info {
size_t hdr_len;
};
+#define ___constant_swab32(x) ((uint32_t)( \
+(((uint32_t)(x) (uint32_t)0x00ffUL) 24) |\
+(((uint32_t)(x) (uint32_t)0xff00UL) 8) |\
+(((uint32_t)(x) (uint32_t)0x00ffUL) 8) |\
+(((uint32_t)(x) (uint32_t)0xff00UL) 24)))
+
+static inline uint32_t swab32(uint32_t v)
+{
+ return ___constant_swab32(v);
+}
+
/* msg.c */
extern int inode_lock_rescan(DB_TXN *txn, cldino_t inum);
extern void msg_get(struct session *sess, const void *v);
@@ -178,7 +219,7 @@ extern int sess_load(GHashTable *ss);
extern struct server cld_srv;
extern struct hail_log srv_log;
extern struct timeval current_time;
-extern int udp_tx(int
Re: [patch tabled 6/8] Add filesystem back-end
On 11/28/2010 08:41 PM, Pete Zaitcev wrote:
This patch adds the first new back-end and makes some changes to the way
nodes are added, to make the invariants of storage_node more sensible.
The filesystem back-end itself is not intended for production use,
so it makes no attempt to run any asynchronous transfers.
We also add a test. Note that this differs from the preliminary versions
of this patch. We used to add both chunk and fs back-ends, so that tabled
replicates to both. This makes sense as a test of store path, but on
retrieval tabled selects any one of available storage nodes with the
object, randomly. It creates gaps in test coverage in any given run.
Therefore, we test two back-end types sequentially now.
Signed-off-by: Pete Zaitcevzait...@redhat.com
---
server/Makefile.am |2
server/stor_chunk.c | 21 -
server/stor_fs.c | 498 +
server/storage.c | 157 ++--
server/storparse.c | 97 +++
server/tabled.h | 31 ++
test/Makefile.am |3
test/be_fs-test.conf |5
test/combo-redux | 74 ++
test/prep-db |4
test/start-daemon|1
test/stop-daemon |9
12 files changed, 835 insertions(+), 67 deletions(-)
commit bccedeedabbe713e4053afa185314b3f57f3d204
Author: Pete Zaitcevzait...@yahoo.com
Date: Sun Nov 28 17:58:05 2010 -0700
Add fs back-end, with a test.
diff --git a/server/Makefile.am b/server/Makefile.am
index 52beec4..71bcb35 100644
--- a/server/Makefile.am
+++ b/server/Makefile.am
@@ -6,7 +6,7 @@ sbin_PROGRAMS = tabled tdbadm
tabled_SOURCES= tabled.h \
bucket.c cldu.c config.c metarep.c object.c replica.c \
server.c status.c storage.c storparse.c \
- stor_chunk.c util.c
+ stor_chunk.c stor_fs.c util.c
tabled_LDADD = ../lib/libtdb.a \
@HAIL_LIBS@ @PCRE_LIBS@ @GLIB_LIBS@ \
@CRYPTO_LIBS@ @DB4_LIBS@ @EVENT_LIBS@ @SSL_LIBS@
diff --git a/server/stor_chunk.c b/server/stor_chunk.c
index 815adcf..7462a9c 100644
--- a/server/stor_chunk.c
+++ b/server/stor_chunk.c
@@ -31,8 +31,7 @@
#includenetdb.h
#include tabled.h
-static const char stor_key_fmt[] = %016llx;
-#define STOR_KEY_SLEN 16
+static const char stor_key_fmt[] = STOR_KEY_FMT;
static int stor_new_stc(struct storage_node *stn, struct st_client **stcp)
{
@@ -66,24 +65,6 @@ static int stor_new_stc(struct storage_node *stn, struct
st_client **stcp)
return 0;
}
-static void stor_read_event(int fd, short events, void *userdata)
-{
- struct open_chunk *cep = userdata;
-
- cep-r_armed = false;/* no EV_PERSIST */
- if (cep-ocb)
- (*cep-ocb)(cep);
-}
-
-static void stor_write_event(int fd, short events, void *userdata)
-{
- struct open_chunk *cep = userdata;
-
- cep-w_armed = false;/* no EV_PERSIST */
- if (cep-ocb)
- (*cep-ocb)(cep);
-}
-
/*
* Open *cep using stn, set up chunk session if needed.
*/
diff --git a/server/stor_fs.c b/server/stor_fs.c
new file mode 100644
index 000..b433a67
--- /dev/null
+++ b/server/stor_fs.c
@@ -0,0 +1,498 @@
+
+/*
+ * Copyright 2010 Red Hat, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; see the file COPYING. If not, write to
+ * the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ */
+
+#define _GNU_SOURCE
+#include tabled-config.h
+
+#includesys/types.h
+#includesys/stat.h
+#includeerrno.h
+#includefcntl.h
+#includesyslog.h
+#includestring.h
+#includeglib.h
+#includeevent.h
+#include tabled.h
+
+static const char stor_key_fmt[] = STOR_KEY_FMT;
+
+static char *fs_obj_pathname(const char *base, uint64_t key)
+{
+ enum { PREFIX_LEN = 3 };
+ char prefix[PREFIX_LEN + 1];
+ char stckey[STOR_KEY_SLEN+1];
+ char *s;
+ int rc;
+
+ /* we know that stckey is going to be longer than PREFIX_LEN */
+ sprintf(stckey, stor_key_fmt, (unsigned long long) key);
+ memcpy(prefix, stckey, PREFIX_LEN);
+ prefix[PREFIX_LEN] = 0;
+
+ rc = asprintf(s, %s/%s/%s, base, prefix, stckey + PREFIX_LEN);
+ if (rc 0)
+ goto err_out;
+
+ return s;
+
+err_out:
+ return NULL;
+}
+
+static char *fs_ctl_pathname(const char *base, const char *file)
+{
+ char *s;
+ int rc;
+
+ rc = asprintf(s, %s/%s, base, file);
+ if
Re: [patch tabled 8/8] Add Swift back-end
On 11/28/2010 08:41 PM, Pete Zaitcev wrote: This patch allows to use tabled with OpenStack Swift object store as if it were our chunkserver, with some extra tricks. The configuration has to be entred manually into CLD, just like in case of filesystem back-end. The code is fairly experimental, so it retains extra messages. Also, since Swift authorizes by plaintext passwords, support for SSL is essential, but is currently missing. There is no build-time test for this, because it would require us to depend on OpenStack, which is untenable. Signed-off-by: Pete Zaitcevzait...@redhat.com applied patches 6-8. well done, this is a milestone for tabled! -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch tabled 6/8] Add filesystem back-end
On Mon, 13 Dec 2010 16:30:59 -0500 Jeff Garzik j...@garzik.org wrote: Current chunkd intentionally prevents two random users from put'ing the same object. The FS backend should do the same. Ouch, I completely forgot about it. Back then I hoped the DB4 transaction would do the job, but perhaps not. -- Pete -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch hail 1/2] Add subdomain calling format
On 12/05/2010 10:53 PM, Pete Zaitcev wrote: Amazon appears to give up on forcing users to migrate and bucket-in-path format is going to stay. However, they still refuse to list buckets from other regions on the default endpoint, which leads to annoying indirection (need to know the region somehow before listing). Easier just use the subdomain format in one invocation. Signed-off-by: Pete Zaitcevzait...@redhat.com --- include/hstor.h |6 + lib/hstor.c | 178 +- 2 files changed, 106 insertions(+), 78 deletions(-) applied 1-2 -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch tabled 1/8] Shuffle fields of storage nodes
On 11/28/2010 08:39 PM, Pete Zaitcev wrote: This helps copy-paste safer later, mostly. Signed-off-by: Pete Zaitcevzait...@redhat.com --- server/object.c |2 - server/storage.c | 79 ++--- server/tabled.h | 12 +++--- 3 files changed, 53 insertions(+), 40 deletions(-) applied 1-5 Gonna give the file backend a cursory test, and swift backend a slightly-more-than-cursory test, then merge those. Well done! Pluggable storage backends make tabled more interesting. -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [patch hail 1/2] Add subdomain calling format
On Tue, 07 Dec 2010 05:05:38 -0500 Jeff Garzik j...@garzik.org wrote: On 12/05/2010 10:53 PM, Pete Zaitcev wrote: Amazon appears to give up on forcing users to migrate and bucket-in-path format is going to stay. However, they still refuse to list buckets from other regions on the default endpoint, which leads to annoying indirection (need to know the region somehow before listing). Easier just use the subdomain format in one invocation. applied 1-2 Thanks. I eventually got around this by calling buckets foo-west, foo-east, but might as well just have this. -- Pete -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: AC_CONFIG_MACRO_DIR([m4])
Pete Zaitcev wrote: Hi, Jim: Autoconf printed a warning when reconfiguting Hail, so I gave up and added this: diff --git a/configure.ac b/configure.ac index 9cfad23..d378854 100644 --- a/configure.ac +++ b/configure.ac @@ -62,6 +62,8 @@ AC_PROG_GCC_TRADITIONAL AM_PROG_CC_C_O AM_PROG_LIBTOOL +AC_CONFIG_MACRO_DIR([m4]) + dnl Checks for header files. AC_HEADER_STDC dnl AC_CHECK_HEADERS(sys/ioctl.h unistd.h) Now I have a directory m4/ with symlinks... This does not seem to be helping any portability, unless I miss where the promised macro are being saved locally. What was this about, do you happen to know? Hi Pete, The symlinks are ok, since make dist dereferences them when creating a tarball. However, if for some reason you find a problem due to the use of symlinks (in that case, please let us know -- who knows, might have to change the default) you can add --copy (-c) to the libtoolize invocation in autogen.sh. You'll also want the following patch, so that aclocal knows where to find the .m4 files: diff --git a/Makefile.am b/Makefile.am index 38a1d92..e5bf438 100644 --- a/Makefile.am +++ b/Makefile.am @@ -2,7 +2,8 @@ ## Toplevel Makefile.am ## +ACLOCAL_AMFLAGS = -I m4 + SUBDIRS= doc lib include cld chunkd tools test EXTRA_DIST = autogen.sh Doxyfile COPYING LICENSE - -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: AC_CONFIG_MACRO_DIR([m4])
On 12/05/2010 04:56 PM, Pete Zaitcev wrote: Autoconf printed a warning when reconfiguting Hail, so I gave up and added this: [...] Now I have a directory m4/ with symlinks... This does not seem to be helping any portability, unless I miss where the promised macro are being saved locally. What was this about, do you happen to know? I presume you refer to this: [jgar...@bd hail]$ ./autogen.sh CFLAGS=-O2 -Wall -Wshadow -g -march=native ./configure --disable-shared libtoolize: putting auxiliary files in `.'. libtoolize: linking file `./ltmain.sh' libtoolize: You should add the contents of the following files to `aclocal.m4': libtoolize: `/usr/share/aclocal/libtool.m4' libtoolize: `/usr/share/aclocal/ltoptions.m4' libtoolize: `/usr/share/aclocal/ltversion.m4' libtoolize: `/usr/share/aclocal/lt~obsolete.m4' libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to configure.ac and libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree. libtoolize: Consider adding `-I m4' to ACLOCAL_AMFLAGS in Makefile.am. libtoolize: putting auxiliary files in `.'. libtoolize: linking file `./ltmain.sh' libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to configure.ac and libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree. libtoolize: Consider adding `-I m4' to ACLOCAL_AMFLAGS in Makefile.am. Think about what this implies: Keeping the correct libtool macros in-tree implies adding a pointless maintenance burden. The distro always gives us correct, up-to-date files. Why would hail want to potentially lag upstream's version of these macros, forcing us to manually track macros that are currently updated automatically for each ./autogen.sh invocation? It is this same silly logic that leads programmers to ship in-tree copies of (for example) zlib. Therefore, the requirement to rebuild hail's configure script is to have a recent distro. Users of tarballs never see this, so this is only an issue for those on oddball or ancient OS's, who are building release tarballs, or working directly out the git repo. And if someone is doing that, they have a lot more headaches than just outdated libtool to contend with. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: AC_CONFIG_MACRO_DIR([m4])
On 12/06/2010 12:44 PM, Pete Zaitcev wrote: On Mon, 06 Dec 2010 12:32:22 -0500 Jeff Garzikj...@garzik.org wrote: Keeping the correct libtool macros in-tree implies adding a pointless maintenance burden. The distro always gives us correct, up-to-date files. Why would hail want to potentially lag upstream's version of these macros, forcing us to manually track macros that are currently updated automatically for each ./autogen.sh invocation? I presumed that the important part is a compatibility between the syntax used in various .am files and the libtool scriptography that underpins them. Lagging upstream has no downside in this case (unlike zlib, where security fixes may exist). It does not seem optimal to run a current libtool with outdated macro files. In all cases except current one, you're checking in third party, maintained, versioned files to hail.git where they will be less-well maintained, and generally out-of-date vis a vis current [upstream | Fedora]. Where is the value in performing this additional work, besides silencing a warning seen only by git repo users? Users of tarballs never see this, so this is only an issue for those on oddball or ancient OS's, who are building release tarballs, or working directly out the git repo. Well, if you say so... Do you have knowledge to the contrary? Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: AC_CONFIG_MACRO_DIR([m4])
Jeff Garzik wrote: On 12/06/2010 12:44 PM, Pete Zaitcev wrote: On Mon, 06 Dec 2010 12:32:22 -0500 Jeff Garzikj...@garzik.org wrote: Keeping the correct libtool macros in-tree implies adding a pointless maintenance burden. The distro always gives us correct, up-to-date files. Why would hail want to potentially lag upstream's version of these macros, forcing us to manually track macros that are currently updated automatically for each ./autogen.sh invocation? I presumed that the important part is a compatibility between the syntax used in various .am files and the libtool scriptography that underpins them. Lagging upstream has no downside in this case (unlike zlib, where security fixes may exist). It does not seem optimal to run a current libtool with outdated macro files. In all cases except current one, you're checking in third party, maintained, versioned files to hail.git where they will be less-well maintained, and generally out-of-date vis a vis current [upstream | Fedora]. Hi Jeff, The patch that adds those two lines does not (and IMHO should not) imply that a project would version-control those files. Typically, those symlinks exist only in your working directory, and not in any project's VC repository. If you have no other files in m4/, you can simply .gitignore the entire m4/ directory. Where is the value in performing this additional work, besides silencing a warning seen only by git repo users? Yeah, either way it's not a big deal. -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
AC_CONFIG_MACRO_DIR([m4])
Hi, Jim: Autoconf printed a warning when reconfiguting Hail, so I gave up and added this: diff --git a/configure.ac b/configure.ac index 9cfad23..d378854 100644 --- a/configure.ac +++ b/configure.ac @@ -62,6 +62,8 @@ AC_PROG_GCC_TRADITIONAL AM_PROG_CC_C_O AM_PROG_LIBTOOL +AC_CONFIG_MACRO_DIR([m4]) + dnl Checks for header files. AC_HEADER_STDC dnl AC_CHECK_HEADERS(sys/ioctl.h unistd.h) Now I have a directory m4/ with symlinks... This does not seem to be helping any portability, unless I miss where the promised macro are being saved locally. What was this about, do you happen to know? -- Pete -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[patch hail 1/2] Add subdomain calling format
Amazon appears to give up on forcing users to migrate and bucket-in-path
format is going to stay. However, they still refuse to list buckets from
other regions on the default endpoint, which leads to annoying indirection
(need to know the region somehow before listing). Easier just use the
subdomain format in one invocation.
Signed-off-by: Pete Zaitcev zait...@redhat.com
---
include/hstor.h |6 +
lib/hstor.c | 178 +-
2 files changed, 106 insertions(+), 78 deletions(-)
diff --git a/include/hstor.h b/include/hstor.h
index b47387b..ca63f4a 100644
--- a/include/hstor.h
+++ b/include/hstor.h
@@ -25,6 +25,8 @@
#include curl/curl.h
#include glib.h
+enum hstor_calling_format { HFMT_ORDINARY, HFMT_SUBDOMAIN };
+
struct hstor_client {
CURL*curl;
char*acc;
@@ -32,6 +34,7 @@ struct hstor_client {
char*user;
char*key;
boolverbose;
+ boolsubdomain;
};
struct hstor_bucket {
@@ -165,6 +168,9 @@ extern void hstor_free_keylist(struct hstor_keylist
*keylist);
extern struct hstor_client *hstor_new(const char *service_acc,
const char *service_host, const char *user, const char *secret_key);
+extern bool hstor_set_format(struct hstor_client *hstor,
+enum hstor_calling_format f);
+
extern bool hstor_add_bucket(struct hstor_client *hstor, const char *name);
extern bool hstor_del_bucket(struct hstor_client *hstor, const char *name);
diff --git a/lib/hstor.c b/lib/hstor.c
index 7f638ec..2892a5e 100644
--- a/lib/hstor.c
+++ b/lib/hstor.c
@@ -86,6 +86,21 @@ err_out:
return NULL;
}
+bool hstor_set_format(struct hstor_client *hstor, enum hstor_calling_format f)
+{
+ switch (f) {
+ case HFMT_ORDINARY:
+ hstor-subdomain = false;
+ break;
+ case HFMT_SUBDOMAIN:
+ hstor-subdomain = true;
+ break;
+ default:
+ return false;
+ }
+ return true;
+}
+
static size_t all_data_cb(const void *ptr, size_t size, size_t nmemb,
void *user_data)
{
@@ -177,6 +192,51 @@ next:
}
}
+static bool hstor_resplit(const struct hstor_client *hstor,
+ const char *bucket, const char *key,
+ char **url, char **hosthdr, char **path)
+{
+ char *unesc_path;
+ int rc;
+
+ if (hstor-subdomain)
+ rc = asprintf(unesc_path, /%s, key);
+ else
+ rc = asprintf(unesc_path, /%s/%s, bucket, key);
+ if (rc 0)
+ goto err_spath;
+ *path = huri_field_escape(unesc_path, PATH_ESCAPE_MASK);
+ if (!*path)
+ goto err_epath;
+
+ if (hstor-subdomain)
+ rc = asprintf(hosthdr, Host: %s.%s, bucket, hstor-host);
+ else
+ rc = asprintf(hosthdr, Host: %s, hstor-host);
+ if (rc 0)
+ goto err_host;
+
+ if (hstor-subdomain)
+ rc = asprintf(url, http://%s.%s%s;, bucket, hstor-acc, *path);
+ else
+ rc = asprintf(url, http://%s%s;, hstor-acc, *path);
+ if (rc 0)
+ goto err_url;
+
+ free(unesc_path);
+ return true;
+
+ /* free(*url); */
+ err_url:
+ free(*hosthdr);
+ err_host:
+ free(*path);
+ err_epath:
+ free(unesc_path);
+ err_spath:
+ return false;
+}
+
struct hstor_blist *hstor_list_buckets(struct hstor_client *hstor)
{
struct http_req req;
@@ -318,8 +378,8 @@ static bool __hstor_ad_bucket(struct hstor_client *hstor,
const char *name,
struct curl_slist *headers = NULL;
int rc;
- if (asprintf(orig_path, /%s/, name) 0)
- goto err_spath;
+ if (!hstor_resplit(hstor, name, , url, host, orig_path))
+ goto err_split;
memset(req, 0, sizeof(req));
req.method = delete ? DELETE : PUT;
@@ -330,13 +390,9 @@ static bool __hstor_ad_bucket(struct hstor_client *hstor,
const char *name,
hreq_hdr_push(req, Date, timestr);
- hreq_sign(req, NULL, hstor-key, hmac);
+ hreq_sign(req, hstor-subdomain? name: NULL, hstor-key, hmac);
sprintf(auth, Authorization: AWS %s:%s, hstor-user, hmac);
- if (asprintf(host, Host: %s, hstor-host) 0)
- goto err_host;
- if (asprintf(url, http://%s/%s/;, hstor-acc, name) 0)
- goto err_url;
headers = curl_slist_append(headers, host);
headers = curl_slist_append(headers, datestr);
@@ -360,11 +416,7 @@ static bool __hstor_ad_bucket(struct hstor_client *hstor,
const char *name,
free(orig_path);
return (rc == 0);
-err_url:
- free(host);
-err_host:
- free(orig_path);
-err_spath:
+err_split:
return false;
}
@@ -384,16 +436,12 @@ bool hstor_get(struct hstor_client *hstor, const char
*bucket,
[patch tabled 1/8] Shuffle fields of storage nodes
This helps copy-paste safer later, mostly.
Signed-off-by: Pete Zaitcev zait...@redhat.com
---
server/object.c |2 -
server/storage.c | 79 ++---
server/tabled.h | 12 +++---
3 files changed, 53 insertions(+), 40 deletions(-)
commit 2d0645834468ea056223bee52d679c983945ce65
Author: Pete Zaitcev zait...@yahoo.com
Date: Thu Nov 11 11:24:00 2010 -0700
Field rearrangements. Layering violation improvement.
diff --git a/server/object.c b/server/object.c
index 71d5377..2920811 100644
--- a/server/object.c
+++ b/server/object.c
@@ -1008,7 +1008,7 @@ static bool object_get_poke(struct client *cli)
ssize_t bytes;
/* The checks for in_len in caller should protect us, but let's see. */
- if (!cli-in_ce.stc) {
+ if (!cli-in_ce.key) {
applog(LOG_ERR, read on closed chunk, in_len %ld,
(long) cli-in_len);
return false;
diff --git a/server/storage.c b/server/storage.c
index fc232ba..c472961 100644
--- a/server/storage.c
+++ b/server/storage.c
@@ -91,8 +91,8 @@ static void stor_read_event(int fd, short events, void
*userdata)
struct open_chunk *cep = userdata;
cep-r_armed = false; /* no EV_PERSIST */
- if (cep-rcb)
- (*cep-rcb)(cep);
+ if (cep-ocb)
+ (*cep-ocb)(cep);
}
static void stor_write_event(int fd, short events, void *userdata)
@@ -100,8 +100,8 @@ static void stor_write_event(int fd, short events, void
*userdata)
struct open_chunk *cep = userdata;
cep-w_armed = false; /* no EV_PERSIST */
- if (cep-wcb)
- (*cep-wcb)(cep);
+ if (cep-ocb)
+ (*cep-ocb)(cep);
}
/*
@@ -131,6 +131,8 @@ int stor_put_start(struct open_chunk *cep, void
(*cb)(struct open_chunk *),
{
char stckey[STOR_KEY_SLEN+1];
+ if (cep-key)
+ return -EBUSY;
if (!cep-stc)
return -EINVAL;
@@ -145,9 +147,10 @@ int stor_put_start(struct open_chunk *cep, void
(*cb)(struct open_chunk *),
cep-node-id, stckey, (long long) size);
return -EIO;
}
- cep-wtogo = size;
- cep-wkey = key;
- cep-wcb = cb;
+ cep-size = size;
+ cep-done = 0;
+ cep-key = key;
+ cep-ocb = cb;
event_set(cep-wevt, cep-wfd, EV_WRITE, stor_write_event, cep);
event_base_set(cep-evbase, cep-wevt);
@@ -167,13 +170,15 @@ int stor_open_read(struct open_chunk *cep, void
(*cb)(struct open_chunk *),
char stckey[STOR_KEY_SLEN+1];
uint64_t size;
+ if (cep-key)
+ return -EBUSY;
if (!cep-stc)
return -EINVAL;
- if (cep-rsize cep-roff != cep-rsize) {
+ if (cep-size cep-done != cep-size) {
applog(LOG_ERR, Unfinished Get (%ld,%ld),
- (long)cep-roff, (long)cep-rsize);
- cep-rsize = 0;
+ (long)cep-done, (long)cep-size);
+ cep-size = 0;
}
sprintf(stckey, stor_key_fmt, (unsigned long long) key);
@@ -184,9 +189,10 @@ int stor_open_read(struct open_chunk *cep, void
(*cb)(struct open_chunk *),
return -EIO;
}
*psize = size;
- cep-rsize = size;
- cep-roff = 0;
- cep-rcb = cb;
+ cep-size = size;
+ cep-done = 0;
+ cep-key = key;
+ cep-ocb = cb;
event_set(cep-revt, cep-rfd, EV_READ, stor_read_event, cep);
event_base_set(cep-evbase, cep-revt);
@@ -213,12 +219,14 @@ void stor_close(struct open_chunk *cep)
event_del(cep-revt);
cep-r_armed = false;
}
- cep-rsize = 0;
+ cep-size = 0;
if (cep-w_armed) {
event_del(cep-wevt);
cep-w_armed = false;
}
+
+ cep-key = 0;
}
/*
@@ -251,40 +259,44 @@ void stor_abort(struct open_chunk *cep)
if (debugging)
applog(LOG_INFO, Failed to reopen Chunk nid %u (%d),
cep-node-id, rc);
+
+ cep-size = 0;
+ cep-done = 0;
+ cep-key = 0;
return;
}
- if (cep-wtogo) {
- sprintf(stckey, stor_key_fmt, (unsigned long long) cep-wkey);
+ if (cep-done != cep-size) {
+ sprintf(stckey, stor_key_fmt, (unsigned long long) cep-key);
stc_delz(cep-stc, stckey);
- cep-wtogo = 0;
}
if (cep-r_armed) {
event_del(cep-revt);
cep-r_armed = false;
}
- cep-rsize = 0;
if (cep-w_armed) {
event_del(cep-wevt);
cep-w_armed = false;
}
+
+ cep-size = 0;
+ cep-done = 0;
+
+ cep-key = 0;
}
ssize_t stor_put_buf(struct open_chunk *cep, void *data, size_t len)
{
[patch tabled 2/8] Comment storage.c
I have a vague memory that It should be ok to return meant something
related to the way our event dispatch worked, but I cannot recall any
details. Ergo, useless comment.
Signed-off-by: Pete Zaitcev zait...@redhat.com
---
server/storage.c |6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
commit 8c4bed7bd56a5bb2c3a654a494a6bb545f0d7db2
Author: Pete Zaitcev zait...@yahoo.com
Date: Thu Nov 11 15:46:59 2010 -0700
Update comments.
diff --git a/server/storage.c b/server/storage.c
index c472961..5c6e3a7 100644
--- a/server/storage.c
+++ b/server/storage.c
@@ -161,9 +161,6 @@ int stor_put_start(struct open_chunk *cep, void
(*cb)(struct open_chunk *),
return 0;
}
-/*
- * It should be ok to return
- */
int stor_open_read(struct open_chunk *cep, void (*cb)(struct open_chunk *),
uint64_t key, uint64_t *psize)
{
@@ -379,6 +376,9 @@ int stor_obj_del(struct storage_node *stn, uint64_t key)
return rc;
}
+/*
+ * XXX WTF?! This accidentially tests a node instead of object! FIXME
+ */
bool stor_obj_test(struct open_chunk *cep, uint64_t key)
{
struct st_keylist *klist;
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[patch tabled 5/8] Rename in_storage to in_socket
The clause is called Socket, so the old name was confusing.
Signed-off-by: Pete Zaitcev zait...@redhat.com
---
server/storparse.c | 16
1 file changed, 8 insertions(+), 8 deletions(-)
commit 6536a193f1befcf3eed8b18535990e8566518479
Author: Pete Zaitcev zait...@yahoo.com
Date: Sun Nov 28 16:29:54 2010 -0700
Rename in_storage to in_socket, because Socket.
diff --git a/server/storparse.c b/server/storparse.c
index 5ed15ea..cbffe0f 100644
--- a/server/storparse.c
+++ b/server/storparse.c
@@ -36,7 +36,7 @@ struct config_context {
boolin_chunk;
boolin_chunk_reported;
- boolin_storage;
+ boolin_socket;
boolstor_encrypt;
char*stor_port;
char*stor_host;
@@ -82,8 +82,8 @@ static void cfg_elm_start (GMarkupParseContext *context,
}
if (!strcmp(element_name, Socket)) {
- if (!cc-in_storage)
- cc-in_storage = true;
+ if (!cc-in_socket)
+ cc-in_socket = true;
else
applog(LOG_ERR, %s: Nested Socket, cc-fname);
}
@@ -95,7 +95,7 @@ static void cfg_elm_start (GMarkupParseContext *context,
}
}
-static void cfg_elm_end_storage(struct config_context *cc)
+static void cfg_elm_end_socket(struct config_context *cc)
{
if (cc-text) {
applog(LOG_WARNING, %s: Extra text in Socket element: \%s\,
@@ -206,8 +206,8 @@ static void cfg_elm_end (GMarkupParseContext *context,
}
else if (!strcmp(element_name, Socket)) {
- cfg_elm_end_storage(cc);
- cc-in_storage = false;
+ cfg_elm_end_socket(cc);
+ cc-in_socket = false;
}
else if (!strcmp(element_name, Geo)) {
@@ -272,7 +272,7 @@ static void cfg_elm_end (GMarkupParseContext *context,
return;
}
- if (cc-in_storage) {
+ if (cc-in_socket) {
n = strtol(cc-text, NULL, 10);
if (n 0 n 65536) {
free(cc-stor_port);
@@ -296,7 +296,7 @@ static void cfg_elm_end (GMarkupParseContext *context,
return;
}
- if (cc-in_storage) {
+ if (cc-in_socket) {
free(cc-stor_host);
cc-stor_host = cc-text;
cc-text = NULL;
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[patch tabled 4/8] Split out chunk back-end
This patch is careful not to change anything but the strictly necessary
parts in the actual code.
Signed-off-by: Pete Zaitcev zait...@redhat.com
---
server/Makefile.am |3
server/stor_chunk.c | 409 ++
server/storage.c| 377 --
server/tabled.h |3
4 files changed, 414 insertions(+), 378 deletions(-)
commit 1eaf18ff243a2d4e291d0af2f13ec425ab0757c0
Author: Pete Zaitcev zait...@yahoo.com
Date: Thu Nov 11 16:30:52 2010 -0700
Split out the chunk-specific part, byte for byte, no changes.
diff --git a/server/Makefile.am b/server/Makefile.am
index 5b53a0a..52beec4 100644
--- a/server/Makefile.am
+++ b/server/Makefile.am
@@ -5,7 +5,8 @@ sbin_PROGRAMS = tabled tdbadm
tabled_SOURCES = tabled.h \
bucket.c cldu.c config.c metarep.c object.c replica.c \
- server.c status.c storage.c storparse.c util.c
+ server.c status.c storage.c storparse.c \
+ stor_chunk.c util.c
tabled_LDADD = ../lib/libtdb.a \
@HAIL_LIBS@ @PCRE_LIBS@ @GLIB_LIBS@ \
@CRYPTO_LIBS@ @DB4_LIBS@ @EVENT_LIBS@ @SSL_LIBS@
diff --git a/server/stor_chunk.c b/server/stor_chunk.c
new file mode 100644
index 000..815adcf
--- /dev/null
+++ b/server/stor_chunk.c
@@ -0,0 +1,409 @@
+
+/*
+ * Copyright 2010 Red Hat, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; see the file COPYING. If not, write to
+ * the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ */
+
+#define _GNU_SOURCE
+#include tabled-config.h
+
+#include sys/types.h
+// #include sys/socket.h
+#include errno.h
+#include syslog.h
+#include string.h
+#include glib.h
+#include event.h
+#include chunkc.h
+#include netdb.h
+#include tabled.h
+
+static const char stor_key_fmt[] = %016llx;
+#define STOR_KEY_SLEN 16
+
+static int stor_new_stc(struct storage_node *stn, struct st_client **stcp)
+{
+ struct st_client *stc;
+ struct sockaddr_in *a4;
+ struct sockaddr_in6 *a6;
+ unsigned short port;
+
+ if (stn-addr.sin6_family == AF_INET) {
+ a4 = (struct sockaddr_in *) stn-addr;
+ port = ntohs(a4-sin_port);
+ } else if (stn-addr.sin6_family == AF_INET6) {
+ a6 = stn-addr;
+ port = ntohs(a6-sin6_port);
+ } else {
+ return -EINVAL;
+ }
+
+ stc = stc_new(stn-hostname, port,
+ tabled_srv.chunk_user, tabled_srv.chunk_key,
+ false);
+ if (!stc)
+ return -EDOM;
+
+ if (!stc_table_openz(stc, tabled, CHF_TBL_CREAT)) {
+ stc_free(stc);
+ return -EDOM;
+ }
+
+ *stcp = stc;
+ return 0;
+}
+
+static void stor_read_event(int fd, short events, void *userdata)
+{
+ struct open_chunk *cep = userdata;
+
+ cep-r_armed = false; /* no EV_PERSIST */
+ if (cep-ocb)
+ (*cep-ocb)(cep);
+}
+
+static void stor_write_event(int fd, short events, void *userdata)
+{
+ struct open_chunk *cep = userdata;
+
+ cep-w_armed = false; /* no EV_PERSIST */
+ if (cep-ocb)
+ (*cep-ocb)(cep);
+}
+
+/*
+ * Open *cep using stn, set up chunk session if needed.
+ */
+static int chunk_open(struct open_chunk *cep, struct storage_node *stn,
+ struct event_base *ev_base)
+{
+ int rc;
+
+ if (cep-stc)
+ return 0;
+
+ if ((rc = stor_new_stc(stn, cep-stc)) 0)
+ return rc;
+
+ cep-evbase = ev_base;
+ cep-node = stor_node_get(stn);
+
+ /* cep-stc-verbose = 1; */
+
+ return 0;
+}
+
+static int chunk_put_start(struct open_chunk *cep,
+ void (*cb)(struct open_chunk *),
+ uint64_t key, uint64_t size)
+{
+ char stckey[STOR_KEY_SLEN+1];
+
+ if (cep-key)
+ return -EBUSY;
+ if (!cep-stc)
+ return -EINVAL;
+
+ /*
+* Set up the putting.
+*/
+ sprintf(stckey, stor_key_fmt, (unsigned long long) key);
+ if (!stc_put_startz(cep-stc, stckey, size, cep-wfd, 0)) {
+ if (debugging)
+ applog(LOG_INFO,
+ stor nid %u put %s new for %lld error,
+ cep-node-id, stckey, (long long) size);
+ return
[patch tabled 6/8] Add filesystem back-end
This patch adds the first new back-end and makes some changes to the way
nodes are added, to make the invariants of storage_node more sensible.
The filesystem back-end itself is not intended for production use,
so it makes no attempt to run any asynchronous transfers.
We also add a test. Note that this differs from the preliminary versions
of this patch. We used to add both chunk and fs back-ends, so that tabled
replicates to both. This makes sense as a test of store path, but on
retrieval tabled selects any one of available storage nodes with the
object, randomly. It creates gaps in test coverage in any given run.
Therefore, we test two back-end types sequentially now.
Signed-off-by: Pete Zaitcev zait...@redhat.com
---
server/Makefile.am |2
server/stor_chunk.c | 21 -
server/stor_fs.c | 498 +
server/storage.c | 157 ++--
server/storparse.c | 97 +++
server/tabled.h | 31 ++
test/Makefile.am |3
test/be_fs-test.conf |5
test/combo-redux | 74 ++
test/prep-db |4
test/start-daemon|1
test/stop-daemon |9
12 files changed, 835 insertions(+), 67 deletions(-)
commit bccedeedabbe713e4053afa185314b3f57f3d204
Author: Pete Zaitcev zait...@yahoo.com
Date: Sun Nov 28 17:58:05 2010 -0700
Add fs back-end, with a test.
diff --git a/server/Makefile.am b/server/Makefile.am
index 52beec4..71bcb35 100644
--- a/server/Makefile.am
+++ b/server/Makefile.am
@@ -6,7 +6,7 @@ sbin_PROGRAMS = tabled tdbadm
tabled_SOURCES = tabled.h \
bucket.c cldu.c config.c metarep.c object.c replica.c \
server.c status.c storage.c storparse.c \
- stor_chunk.c util.c
+ stor_chunk.c stor_fs.c util.c
tabled_LDADD = ../lib/libtdb.a \
@HAIL_LIBS@ @PCRE_LIBS@ @GLIB_LIBS@ \
@CRYPTO_LIBS@ @DB4_LIBS@ @EVENT_LIBS@ @SSL_LIBS@
diff --git a/server/stor_chunk.c b/server/stor_chunk.c
index 815adcf..7462a9c 100644
--- a/server/stor_chunk.c
+++ b/server/stor_chunk.c
@@ -31,8 +31,7 @@
#include netdb.h
#include tabled.h
-static const char stor_key_fmt[] = %016llx;
-#define STOR_KEY_SLEN 16
+static const char stor_key_fmt[] = STOR_KEY_FMT;
static int stor_new_stc(struct storage_node *stn, struct st_client **stcp)
{
@@ -66,24 +65,6 @@ static int stor_new_stc(struct storage_node *stn, struct
st_client **stcp)
return 0;
}
-static void stor_read_event(int fd, short events, void *userdata)
-{
- struct open_chunk *cep = userdata;
-
- cep-r_armed = false; /* no EV_PERSIST */
- if (cep-ocb)
- (*cep-ocb)(cep);
-}
-
-static void stor_write_event(int fd, short events, void *userdata)
-{
- struct open_chunk *cep = userdata;
-
- cep-w_armed = false; /* no EV_PERSIST */
- if (cep-ocb)
- (*cep-ocb)(cep);
-}
-
/*
* Open *cep using stn, set up chunk session if needed.
*/
diff --git a/server/stor_fs.c b/server/stor_fs.c
new file mode 100644
index 000..b433a67
--- /dev/null
+++ b/server/stor_fs.c
@@ -0,0 +1,498 @@
+
+/*
+ * Copyright 2010 Red Hat, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; see the file COPYING. If not, write to
+ * the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ */
+
+#define _GNU_SOURCE
+#include tabled-config.h
+
+#include sys/types.h
+#include sys/stat.h
+#include errno.h
+#include fcntl.h
+#include syslog.h
+#include string.h
+#include glib.h
+#include event.h
+#include tabled.h
+
+static const char stor_key_fmt[] = STOR_KEY_FMT;
+
+static char *fs_obj_pathname(const char *base, uint64_t key)
+{
+ enum { PREFIX_LEN = 3 };
+ char prefix[PREFIX_LEN + 1];
+ char stckey[STOR_KEY_SLEN+1];
+ char *s;
+ int rc;
+
+ /* we know that stckey is going to be longer than PREFIX_LEN */
+ sprintf(stckey, stor_key_fmt, (unsigned long long) key);
+ memcpy(prefix, stckey, PREFIX_LEN);
+ prefix[PREFIX_LEN] = 0;
+
+ rc = asprintf(s, %s/%s/%s, base, prefix, stckey + PREFIX_LEN);
+ if (rc 0)
+ goto err_out;
+
+ return s;
+
+err_out:
+ return NULL;
+}
+
+static char *fs_ctl_pathname(const char *base, const char *file)
+{
+ char *s;
+ int rc;
+
+ rc = asprintf(s, %s/%s, base, file);
+ if (rc 0)
+ return NULL;
+ return s;
+}
+
[patch tabled 8/8] Add Swift back-end
This patch allows to use tabled with OpenStack Swift object store as if it
were our chunkserver, with some extra tricks. The configuration has to be
entred manually into CLD, just like in case of filesystem back-end.
The code is fairly experimental, so it retains extra messages.
Also, since Swift authorizes by plaintext passwords, support for SSL is
essential, but is currently missing.
There is no build-time test for this, because it would require us to
depend on OpenStack, which is untenable.
Signed-off-by: Pete Zaitcev zait...@redhat.com
---
doc/etc.tabled.conf |5
server/Makefile.am |7
server/object.c |2
server/replica.c |4
server/server.c |8
server/stor_swift.c | 1420 +
server/storage.c |3
server/storparse.c |7
server/tabled.h | 30
test/swift-test.conf |8
10 files changed, 1487 insertions(+), 7 deletions(-)
commit f24e2bd00d10f074ca9c6f62794b0838bc2de9b5
Author: Pete Zaitcev zait...@yahoo.com
Date: Sun Nov 28 18:07:44 2010 -0700
Add Swift back-end. Sadly, no test.
diff --git a/doc/etc.tabled.conf b/doc/etc.tabled.conf
index 5112e8a..6e82fe4 100644
--- a/doc/etc.tabled.conf
+++ b/doc/etc.tabled.conf
@@ -43,8 +43,9 @@
/CLD
--
-ChunkUsertest/ChunkUser
-ChunkKeytest/ChunkKey
+!-- swift-auth-add-user -K devauth -a tabled tester1 tester1 --
+ChunkUsertester1/ChunkUser
+ChunkKeytester1/ChunkKey
!-- Monitoring interface. Best is not to expose this to Internet. --
StatusPort8084/StatusPort
diff --git a/server/Makefile.am b/server/Makefile.am
index 71bcb35..5651ec2 100644
--- a/server/Makefile.am
+++ b/server/Makefile.am
@@ -1,15 +1,16 @@
-INCLUDES = -I$(top_srcdir)/include @GLIB_CFLAGS@ @HAIL_CFLAGS@
+INCLUDES = -I$(top_srcdir)/include \
+ @LIBCURL_CPPFLAGS@ @GLIB_CFLAGS@ @HAIL_CFLAGS@
sbin_PROGRAMS = tabled tdbadm
tabled_SOURCES = tabled.h \
bucket.c cldu.c config.c metarep.c object.c replica.c \
server.c status.c storage.c storparse.c \
- stor_chunk.c stor_fs.c util.c
+ stor_chunk.c stor_fs.c stor_swift.c util.c
tabled_LDADD = ../lib/libtdb.a \
@HAIL_LIBS@ @PCRE_LIBS@ @GLIB_LIBS@ \
- @CRYPTO_LIBS@ @DB4_LIBS@ @EVENT_LIBS@ @SSL_LIBS@
+ @CRYPTO_LIBS@ @DB4_LIBS@ @EVENT_LIBS@ @LIBCURL@ @SSL_LIBS@
tdbadm_SOURCES = tdbadm.c
tdbadm_LDADD = ../lib/libtdb.a @GLIB_LIBS@ @DB4_LIBS@
diff --git a/server/object.c b/server/object.c
index 2920811..44f8d5c 100644
--- a/server/object.c
+++ b/server/object.c
@@ -700,6 +700,8 @@ static struct open_chunk *open_chunk1(struct storage_node
*stnode,
applog(LOG_ERR, OOM);
goto err_alloc;
}
+ INIT_LIST_HEAD(ochunk-evt_list);
+ INIT_LIST_HEAD(ochunk-buf_list);
rc = stor_open(ochunk, stnode, tabled_srv.evbase_main);
if (rc != 0) {
diff --git a/server/replica.c b/server/replica.c
index 7c31112..0d3f7a2 100644
--- a/server/replica.c
+++ b/server/replica.c
@@ -90,6 +90,10 @@ static struct rep_job *job_alloc(size_t klen, struct
db_obj_key *key)
job = malloc(len);
if (job) {
memset(job, 0, sizeof(struct rep_job));
+ INIT_LIST_HEAD(job-in_ce.evt_list);
+ INIT_LIST_HEAD(job-in_ce.buf_list);
+ INIT_LIST_HEAD(job-out_ce.evt_list);
+ INIT_LIST_HEAD(job-out_ce.buf_list);
memcpy(job+1, key, klen);
job-klen = klen;
job-key = (struct db_obj_key *)(job+1);
diff --git a/server/server.c b/server/server.c
index ba3e00a..de6975f 100644
--- a/server/server.c
+++ b/server/server.c
@@ -46,6 +46,7 @@
#include openssl/md5.h
#include openssl/hmac.h
#include openssl/ssl.h
+#include curl/curl.h
#include elist.h
#include chunkc.h
#include cldc.h
@@ -1245,6 +1246,8 @@ static struct client *cli_alloc(bool is_status)
applog(LOG_ERR, out of memory);
return NULL;
}
+ INIT_LIST_HEAD(cli-in_ce.evt_list);
+ INIT_LIST_HEAD(cli-in_ce.buf_list);
atcp_wr_init(cli-wst, libevent_wr_ops, cli-write_ev, cli);
@@ -2079,6 +2082,11 @@ int main (int argc, char *argv[])
SSL_library_init();
SSL_load_error_strings();
+ if (curl_global_init(CURL_GLOBAL_ALL)) {
+ fprintf(stderr, curl_global_init failed\n);
+ return 1;
+ }
+
stc_init();
cld_init();
diff --git a/server/stor_swift.c b/server/stor_swift.c
new file mode 100644
index 000..253c92c
--- /dev/null
+++ b/server/stor_swift.c
@@ -0,0 +1,1420 @@
+
+/*
+ * Copyright 2010 Red Hat, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed
[patch tabled 7/8] Clenaup stor_chunk.c
Signed-off-by: Pete Zaitcev zait...@redhat.com
---
server/stor_chunk.c |6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
commit ca0920cfe5978839ee1a35d6096754a87db6f9ac
Author: Pete Zaitcev zait...@yahoo.com
Date: Sun Nov 28 18:01:24 2010 -0700
Cleanup stor_chunk.
diff --git a/server/stor_chunk.c b/server/stor_chunk.c
index 7462a9c..9111750 100644
--- a/server/stor_chunk.c
+++ b/server/stor_chunk.c
@@ -21,14 +21,12 @@
#include tabled-config.h
#include sys/types.h
-// #include sys/socket.h
#include errno.h
#include syslog.h
#include string.h
#include glib.h
#include event.h
#include chunkc.h
-#include netdb.h
#include tabled.h
static const char stor_key_fmt[] = STOR_KEY_FMT;
@@ -117,7 +115,7 @@ static int chunk_put_start(struct open_chunk *cep,
event_base_set(cep-evbase, cep-wevt);
if (debugging)
- applog(LOG_INFO, stor nid %u put %s new for %lld,
+ applog(LOG_INFO, stor nid %u put %s size %lld,
cep-node-id, stckey, (long long) size);
return 0;
@@ -249,7 +247,7 @@ static ssize_t chunk_put_buf(struct open_chunk *cep, void
*data, size_t len)
int rc;
if (cep-done + len cep-size) {
- applog(LOG_ERR, Put size %ld remaining %ld,
+ /* P3 */ applog(LOG_ERR, Put length %ld remaining %ld,
(long) len, (long) (cep-size - cep-done));
if (cep-done == cep-size)
return -EIO;/* will spin otherwise, better error */
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[patch hail 1/2] use hail_log.h
I think forward declarations are not needed if we have a nice leaf header.
Signed-off-by: Pete Zaitcev zait...@redhat.com
---
include/cld_common.h |3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
commit 12f0fd5e0c6925299617082d834220396fae252e
Author: Pete Zaitcev zait...@yahoo.com
Date: Wed Nov 10 12:38:14 2010 -0700
Cleanup for hail_log.
diff --git a/include/cld_common.h b/include/cld_common.h
index 3ac18c6..84b1ec6 100644
--- a/include/cld_common.h
+++ b/include/cld_common.h
@@ -27,11 +27,10 @@
#include glib.h
#include openssl/sha.h
#include cld_msg_rpc.h
+#include hail_log.h
#define CLD_ALIGN8(n) ((8 - ((n) 7)) 7)
-struct hail_log;
-
struct cld_timer {
boolfired;
boolon_list;
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[patch hail 2/2] add -i to cldcli
This is purely cosmetic so that tabled build logs do not show useless
prompts. The way -i works copies common shells: one cannot force
non-interactive session.
Signed-off-by: Pete Zaitcev zait...@redhat.com
---
tools/cldcli.c | 14 +-
1 file changed, 13 insertions(+), 1 deletion(-)
commit ee2364add99af233a8fa5e9773a31087eb9caf00
Author: Pete Zaitcev zait...@yahoo.com
Date: Wed Nov 10 13:08:33 2010 -0700
Suppress prompts in cldcli when piped.
diff --git a/tools/cldcli.c b/tools/cldcli.c
index ff668ab..d347bf3 100644
--- a/tools/cldcli.c
+++ b/tools/cldcli.c
@@ -42,6 +42,8 @@ static struct argp_option options[] = {
Set debug output to LEVEL (0 = off, 2 = max verbose) },
{ host, 'h', HOST:PORT, 0,
Connect to remote CLD at specified HOST:PORT },
+ { interactive, 'i', NULL, 0,
+ Force an interactive session },
{ user, 'u', USER, 0,
Set username to USER },
{ verbose, 'v', NULL, 0,
@@ -75,6 +77,7 @@ struct cldcli_lock_info {
static GList *host_list;
static char clicwd[CLD_PATH_MAX + 1] = /;
static char our_user[CLD_MAX_USERNAME + 1] = cli_user;
+static int interactive = -1;
/* globals only for use in thread */
static struct ncld_sess *nsess;
@@ -646,6 +649,9 @@ static error_t parse_opt (int key, char *arg, struct
argp_state *state)
if (!push_host(arg))
argp_usage(state);
break;
+ case 'i':
+ interactive = 1;
+ break;
case 'u':
if (strlen(arg) = CLD_MAX_USERNAME) {
fprintf(stderr, TAG : invalid user: '%s'\n, arg);
@@ -667,6 +673,8 @@ static error_t parse_opt (int key, char *arg, struct
argp_state *state)
static void prompt(void)
{
+ if (!interactive)
+ return;
printf([%s %s]$ , our_user, clicwd);
fflush(stdout);
}
@@ -691,6 +699,9 @@ int main (int argc, char *argv[])
return 1;
}
+ if (interactive 0)
+ interactive = isatty(0);
+
if (!host_list) {
enum { hostsz = 64 };
char hostb[hostsz];
@@ -724,7 +735,8 @@ int main (int argc, char *argv[])
return 1;
}
- printf(Type 'help' at the prompt to list commands.\n);
+ if (interactive)
+ printf(Type 'help' at the prompt to list commands.\n);
prompt();
while (fgets(linebuf, sizeof(linebuf), stdin) != NULL) {
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[patch hail] remove duplicated stc_readport
Now that we have a common library for Hail, an opportunity opens to trim
some duplication, such as stc_readport. It even had a comment about it.
Note that we leave cld_readport in the API for a few weeks, while I get
my tabled trees and RPMs in order. Unfortunately we routinely neglect
to set specific version in RPM headers (e.g. no Requires: cld = 0.8.2).
Also, get rid of g_file_get_contents. Talk about pointless: it requires
caller to free memory, and it's not like code is any more compact or
easier to understand.
Signed-off-by: Pete Zaitcev zait...@redhat.com
---
chunkd/config.c |4 +--
include/chunkc.h |2 -
include/cld_common.h |3 +-
lib/chunkdc.c| 29 --
lib/common.c | 36 +++--
test/chunkd/auth.c |3 +-
test/chunkd/basic-object.c |3 +-
test/chunkd/cp.c |3 +-
test/chunkd/get-part.c |3 +-
test/chunkd/it-works.c |3 +-
test/chunkd/large-object.c |3 +-
test/chunkd/lotsa-objects.c |3 +-
test/chunkd/nop.c|3 +-
test/chunkd/selfcheck-unit.c |3 +-
test/cld/basic-io.c |2 -
test/cld/basic-session.c |2 -
test/cld/lock-file.c |2 -
17 files changed, 46 insertions(+), 61 deletions(-)
commit 56536e377f30a130f621beaa22dd90cb10e29d45
Author: Pete Zaitcev zait...@yahoo.com
Date: Tue Oct 26 13:39:50 2010 -0600
hail_readport.
diff --git a/chunkd/config.c b/chunkd/config.c
index 4c58768..c74e641 100644
--- a/chunkd/config.c
+++ b/chunkd/config.c
@@ -239,11 +239,11 @@ static void cfg_elm_end_cld(struct config_context *cc)
*/
if (cc-cld_port_file) {
int port;
- if ((port = cld_readport(cc-cld_port_file)) = 0) {
+ if ((port = hail_readport(cc-cld_port_file)) = 0) {
applog(LOG_INFO, Waiting for CLD PortFile %s,
cc-cld_port_file);
sleep(2);
- while ((port = cld_readport(cc-cld_port_file)) = 0)
+ while ((port = hail_readport(cc-cld_port_file)) = 0)
sleep(3);
applog(LOG_INFO, Using CLD port %u, port);
}
diff --git a/include/chunkc.h b/include/chunkc.h
index e3c2bb7..e9b6003 100644
--- a/include/chunkc.h
+++ b/include/chunkc.h
@@ -112,8 +112,6 @@ extern bool stc_check_status(struct st_client *stc,
extern struct st_keylist *stc_keys(struct st_client *stc);
-extern int stc_readport(const char *fname);
-
static inline void *stc_get_inlinez(struct st_client *stc,
const char *key,
size_t *len)
diff --git a/include/cld_common.h b/include/cld_common.h
index 8c11601..3ac18c6 100644
--- a/include/cld_common.h
+++ b/include/cld_common.h
@@ -64,7 +64,8 @@ static inline void cld_timer_init(struct cld_timer *timer,
const char *name,
extern unsigned long long cld_sid2llu(const uint8_t *sid);
extern void cld_rand64(void *p);
extern const char *cld_errstr(enum cle_err_codes ecode);
-extern int cld_readport(const char *fname);
+extern int cld_readport(const char *fname);/* deprecated */
+extern int hail_readport(const char *fname);
/*** Validate the HMAC signature of a byte buffer.
*
diff --git a/lib/chunkdc.c b/lib/chunkdc.c
index bebf87b..02cf850 100644
--- a/lib/chunkdc.c
+++ b/lib/chunkdc.c
@@ -1303,32 +1303,3 @@ void stc_init(void)
srand(time(NULL) ^ getpid()); // for cld_rand64 et.al.
}
-/*
- * Read a port number from a port file, return the value or negative error.
- * A 100% copy of cld_readport for now, but permits us not to link libcldc.
- */
-int stc_readport(const char *fname)
-{
- enum { LEN = 11 };
- char buf[LEN+1];
- long port;
- int fd;
- int rc;
-
- if ((fd = open(fname, O_RDONLY)) == -1)
- return -errno;
- rc = read(fd, buf, LEN);
- close(fd);
- if (rc 0)
- return -errno;
- if (rc == 0)
- return -EPIPE;
- buf[rc] = 0;
-
- port = strtol(buf, NULL, 10);
- if (port = 0 || port = 65636)
- return -EDOM;
-
- return (int)port;
-}
-
diff --git a/lib/common.c b/lib/common.c
index 30bc6dd..60b120d 100644
--- a/lib/common.c
+++ b/lib/common.c
@@ -81,27 +81,33 @@ const char *cld_errstr(enum cle_err_codes ecode)
/*
* Read a port number from a port file, return the value or negative error.
*/
-int cld_readport(const char *fname)
+int hail_readport(const char *fname)
{
+ enum { LEN = 11 };
+ char buf[LEN+1];
long port;
- gchar *buf;
- GError *err = NULL;
- gsize len;
-
- if (!g_file_get_contents(fname, buf, len, err)) {
- int ret = -1000 - err-code;
- g_error_free(err);
- return ret;
-
Re: [patch hail] remove duplicated stc_readport
On 10/26/2010 03:47 PM, Pete Zaitcev wrote: Now that we have a common library for Hail, an opportunity opens to trim some duplication, such as stc_readport. It even had a comment about it. Note that we leave cld_readport in the API for a few weeks, while I get my tabled trees and RPMs in order. Unfortunately we routinely neglect to set specific version in RPM headers (e.g. no Requires: cld= 0.8.2). Also, get rid of g_file_get_contents. Talk about pointless: it requires caller to free memory, and it's not like code is any more compact or easier to understand. Signed-off-by: Pete Zaitcevzait...@redhat.com applied it would be nice if a follow-up patch moved the hail_readport() definition into a more generic, not-CLD-specific header such as include/hail.h[1] Jeff [1] which doesn't exist yet. maybe we could rename hail_log.h to hail.h, and make hail.h a dumping ground for hail-generic items. -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] const-correctness tweaks
Jeff Garzik wrote: ... But even if curl were requiring some suboptimal signature, it would be nice not to impose that on all projects that use hail. Are there older curl headers that do require the const-free signature? If there are and you want to support them, too, let me know -- maybe I can cook up an autoconf test to make things work there, with minimal impact. Nah, I wouldn't worry about the const signature, it's probably just out of date documentation. If users appear running old OS's or OS versions, we can tackle autoconf'ing on a piecemeal basis as needs arise. Committed these patches of yours to hail.git and tabled.git. Thanks! -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
tabled + atcp
Just committed this: commit 57c4be44cdfa6c0cda6cf26d19e8048a945c5a78 Author: Jeff Garzik j...@garzik.org Date: Sat Oct 23 14:01:20 2010 -0400 Use libhail's atcp rather than our own async TCP write code. Should be functionally equivalent, as atcp originated from tabled code. Please test, and highlight any behavior differences with vanilla tabled v0.5.2. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] const-correctness tweaks
On 10/20/2010 04:53 AM, Jim Meyering wrote: Jeff Garzik wrote: ... Hi Jeff. Sorry I didn't notice that the first time. I built with ./autogen.sh ./configure make. It looks like you recommend -Wall -Wshadow. The two warnings above are the only ones I see with the patch, and they're easy to fix. When storing const pointer params into a struct like that, I've found that it's best to cast away the const, which really does reflect the semantics: by using const on the parameter, I view it as promising not to deref through the pointer *in that function*. Since it's usually not reasonable to make the struct member const (as you saw, it propagates too far and often ends up being contradictory), the lesser evil of the cast is preferable here. If you're still game, the following incremental patch seems to be enough for me: Let me know and I'll resubmit the full one. Well, my primary concern now originates from curl_easy_setopt(3) documentation: CURLOPT_WRITEFUNCTION Function pointer that should match the following prototype: size_t function( void *ptr, size_t size, size_t nmemb, void *stream); hstor's callback is passed directly to libcurl, so we seem to be bound by outside constraints, no? I compiled hail (with that patch) on F13 with -Wall -Wshadow with no warnings. That curl_easy_setopt documentation seems to be overly strict, or perhaps out of date?. When I compare with the code (curl/typecheck-gcc.h), I see all of the necessary const attributes: /* evaluates to true if expr is of type curl_write_callback or similar */ #define _curl_is_write_cb(expr) \ (_curl_is_read_cb(expr) ||\ __builtin_types_compatible_p(__typeof__(expr), __typeof__(fwrite)) || \ __builtin_types_compatible_p(__typeof__(expr), curl_write_callback) || \ _curl_callback_compatible((expr), _curl_write_callback1) ||\ _curl_callback_compatible((expr), _curl_write_callback2) ||\ _curl_callback_compatible((expr), _curl_write_callback3) ||\ _curl_callback_compatible((expr), _curl_write_callback4) ||\ _curl_callback_compatible((expr), _curl_write_callback5) ||\ _curl_callback_compatible((expr), _curl_write_callback6)) typedef size_t (_curl_write_callback1)(const char *, size_t, size_t, void*); typedef size_t (_curl_write_callback2)(const char *, size_t, size_t, const void*); typedef size_t (_curl_write_callback3)(const char *, size_t, size_t, FILE*); typedef size_t (_curl_write_callback4)(const void *, size_t, size_t, void*); typedef size_t (_curl_write_callback5)(const void *, size_t, size_t, const void*); typedef size_t (_curl_write_callback6)(const void *, size_t, size_t, FILE*); But even if curl were requiring some suboptimal signature, it would be nice not to impose that on all projects that use hail. Are there older curl headers that do require the const-free signature? If there are and you want to support them, too, let me know -- maybe I can cook up an autoconf test to make things work there, with minimal impact. Nah, I wouldn't worry about the const signature, it's probably just out of date documentation. If users appear running old OS's or OS versions, we can tackle autoconf'ing on a piecemeal basis as needs arise. Committed these patches of yours to hail.git and tabled.git. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
hail version 0.7.2 released
Home: https://hail.wiki.kernel.org/ Git: git://git.kernel.org/pub/scm/daemon/distsrv/hail.git Download: http://www.kernel.org/pub/software/network/distsrv/hail/ Version 0.7.2 release notes (NEWS): - cld: read overrun bug fix - chunkd: add checksum table to disk format, one checksum per 64k of obj data - chunkd, libhail: add new GET_PART operation for partial object retrieval - chunkd: bug fixes - chunkd: use libevent (again) for main loop polling - libhail: add async TCP network writing API, atcp_wr* - libhail: bug fixes This release includes incompatible API and on-disk format changes. Git shortlog attached. Jeff Garzik (12): chunkd: Add checksum table to on-disk format, one sum per 64k of data chunkd: checksum data prior to returning via GET chunk: Add Get-Partial-Object (GET_PART) operation lib/chunksrv.c: add FIXME chunkd: internal 32/64-bit type fixes test/chunkd/get-part: read and test segment of randomized memory libhail: add async TCP network writing API, atcp_wr* Use libevent in chunkd, rather than hand-rolled server-poll functionality. atcp: extract pre- and post-writev code into separate functions Merge branch 'chunkd-libevent' chunkd: properly checksum a multi-block range Release version 0.7.2. Jim Meyering (4): cld: don't expect inode name to be NUL-terminated (avoid read overrun) lib/hstor.c: avoid an unconditional leak in append_qparam chunkd: don't leak an FS object iterator libhail/hstor: const-correctness tweaks Pete Zaitcev (3): libhail: Fix calling convention of huri_field_escape Change cfgfile.txt into a real config file pkg: add doc/setup.txt to install
tabled version 0.5.2 released
Home: https://hail.wiki.kernel.org/ Git: git://git.kernel.org/pub/scm/daemon/distsrv/tabled.git Download: http://www.kernel.org/pub/software/network/distsrv/tabled/ Version 0.5.2 release notes (NEWS): - Permit randomly allocated TCP port, for db4 replication master - Install etc.tabled.conf as a useful example configuration - minor testsuite additions - many minor bug fixes Git shortlog attached.
Re: tabled version 0.5.2 released
On 10/22/2010 11:39 PM, Jeff Garzik wrote: Home: https://hail.wiki.kernel.org/ Git: git://git.kernel.org/pub/scm/daemon/distsrv/tabled.git Download: http://www.kernel.org/pub/software/network/distsrv/tabled/ Version 0.5.2 release notes (NEWS): - Permit randomly allocated TCP port, for db4 replication master - Install etc.tabled.conf as a useful example configuration - minor testsuite additions - many minor bug fixes Git shortlog attached. er, now it's attached. Jeff Garzik (2): test/.gitignore: ignore list-keys test Release version 0.5.2. Jim Meyering (6): server/server.c (net_write_port): Don't ignore write error. server/server.c: use sizeof(s) rather than equivalent 64 don't dereference NULL on OOM server/status.c: don't deref NULL on failed strdup server/bucket.c: don't deref NULL upon failed malloc adapt to changed signature of hstor_get's callback function Pete Zaitcev (8): Fix crash when stopping slave Clean name vs host cleanup a call to closelog() Support auto replicaton port test/start-daemon: Factor 3 pid-checking if blocks into a loop. test/start-daemon: Ignore stale .pid files. Add a test for hstor_keys Install etc.tabled.conf
Re: hail version 0.7.2 released
On 10/22/2010 11:22 PM, Jeff Garzik wrote: Home: https://hail.wiki.kernel.org/ Git: git://git.kernel.org/pub/scm/daemon/distsrv/hail.git Download: http://www.kernel.org/pub/software/network/distsrv/hail/ It seems that kernel.org mirroring is broken or extremely slow at the moment. The releases should appear as soon as kernel.org mirrors pick back up. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] const-correctness tweaks
Jeff Garzik wrote: On 10/06/2010 08:07 AM, Jim Meyering wrote: Make write_cb callback's buffer parameter const, like all write-like functions. Give a few char * parameters the const attribute. Signed-off-by: Jim Meyeringmeyer...@redhat.com --- It looks like most of hail's interfaces are const-correct, but one stood out because it provokes a warning when I tried to pass a const-correct write_cb function to hstor_get from iwhd: proxy.c:382: warning: passing argument 4 of 'hstor_get' from \ incompatible pointer type /usr/include/hstor.h:173: note: expected \ 'size_t (*)(void *, size_t, size_t, void *)' but argument is of type \ 'size_t (*)(const void *, size_t, size_t, void *)' In case you feel comfortable fixing this, here's a patch: Sorry for not getting back to this; I had hoped to solve some additional problems that cropped up, but didn't have time. So, to forestall further delay, libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/libxml2 -O2 -Wall -Wshadow -g -MT hutil.lo -MD -MP -MF .deps/hutil.Tpo -c hutil.c -o hutil.o hutil.c: In function ‘hreq_hdr_push’: hutil.c:145: warning: assignment discards qualifiers from pointer target type hutil.c:146: warning: assignment discards qualifiers from pointer target type warnings appear after this patch. When solving these warnings with const' markers, it quickly becomes a bit of a rat's nest. At a minimum, the write_cb callback signature must match libcurl's, which does not use 'const'. I can see this makes sense from libcurl implementation's perspective, even if it does not really match the constness one expects from a foo-get function. Hi Jeff. Sorry I didn't notice that the first time. I built with ./autogen.sh ./configure make. It looks like you recommend -Wall -Wshadow. The two warnings above are the only ones I see with the patch, and they're easy to fix. When storing const pointer params into a struct like that, I've found that it's best to cast away the const, which really does reflect the semantics: by using const on the parameter, I view it as promising not to deref through the pointer *in that function*. Since it's usually not reasonable to make the struct member const (as you saw, it propagates too far and often ends up being contradictory), the lesser evil of the cast is preferable here. If you're still game, the following incremental patch seems to be enough for me: Let me know and I'll resubmit the full one. diff --git a/lib/hutil.c b/lib/hutil.c index 13a8d5e..b74460b 100644 --- a/lib/hutil.c +++ b/lib/hutil.c @@ -142,8 +142,8 @@ int hreq_hdr_push(struct http_req *req, const char *key, const char *val) val++; hdr = req-hdr[req-n_hdr++]; - hdr-key = key; - hdr-val = val; + hdr-key = (char *) key; + hdr-val = (char *) val; return 0; } -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] const-correctness tweaks
On 10/20/2010 04:00 AM, Jim Meyering wrote: Jeff Garzik wrote: On 10/06/2010 08:07 AM, Jim Meyering wrote: Make write_cb callback's buffer parameter const, like all write-like functions. Give a few char * parameters the const attribute. Signed-off-by: Jim Meyeringmeyer...@redhat.com --- It looks like most of hail's interfaces are const-correct, but one stood out because it provokes a warning when I tried to pass a const-correct write_cb function to hstor_get from iwhd: proxy.c:382: warning: passing argument 4 of 'hstor_get' from \ incompatible pointer type /usr/include/hstor.h:173: note: expected \ 'size_t (*)(void *, size_t, size_t, void *)' but argument is of type \ 'size_t (*)(const void *, size_t, size_t, void *)' In case you feel comfortable fixing this, here's a patch: Sorry for not getting back to this; I had hoped to solve some additional problems that cropped up, but didn't have time. So, to forestall further delay, libtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/libxml2 -O2 -Wall -Wshadow -g -MT hutil.lo -MD -MP -MF .deps/hutil.Tpo -c hutil.c -o hutil.o hutil.c: In function ‘hreq_hdr_push’: hutil.c:145: warning: assignment discards qualifiers from pointer target type hutil.c:146: warning: assignment discards qualifiers from pointer target type warnings appear after this patch. When solving these warnings with const' markers, it quickly becomes a bit of a rat's nest. At a minimum, the write_cb callback signature must match libcurl's, which does not use 'const'. I can see this makes sense from libcurl implementation's perspective, even if it does not really match the constness one expects from a foo-get function. Hi Jeff. Sorry I didn't notice that the first time. I built with ./autogen.sh ./configure make. It looks like you recommend -Wall -Wshadow. The two warnings above are the only ones I see with the patch, and they're easy to fix. When storing const pointer params into a struct like that, I've found that it's best to cast away the const, which really does reflect the semantics: by using const on the parameter, I view it as promising not to deref through the pointer *in that function*. Since it's usually not reasonable to make the struct member const (as you saw, it propagates too far and often ends up being contradictory), the lesser evil of the cast is preferable here. If you're still game, the following incremental patch seems to be enough for me: Let me know and I'll resubmit the full one. Well, my primary concern now originates from curl_easy_setopt(3) documentation: CURLOPT_WRITEFUNCTION Function pointer that should match the following prototype: size_t function( void *ptr, size_t size, size_t nmemb, void *stream); hstor's callback is passed directly to libcurl, so we seem to be bound by outside constraints, no? Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] const-correctness tweaks
Jeff Garzik wrote: ... Hi Jeff. Sorry I didn't notice that the first time. I built with ./autogen.sh ./configure make. It looks like you recommend -Wall -Wshadow. The two warnings above are the only ones I see with the patch, and they're easy to fix. When storing const pointer params into a struct like that, I've found that it's best to cast away the const, which really does reflect the semantics: by using const on the parameter, I view it as promising not to deref through the pointer *in that function*. Since it's usually not reasonable to make the struct member const (as you saw, it propagates too far and often ends up being contradictory), the lesser evil of the cast is preferable here. If you're still game, the following incremental patch seems to be enough for me: Let me know and I'll resubmit the full one. Well, my primary concern now originates from curl_easy_setopt(3) documentation: CURLOPT_WRITEFUNCTION Function pointer that should match the following prototype: size_t function( void *ptr, size_t size, size_t nmemb, void *stream); hstor's callback is passed directly to libcurl, so we seem to be bound by outside constraints, no? I compiled hail (with that patch) on F13 with -Wall -Wshadow with no warnings. That curl_easy_setopt documentation seems to be overly strict, or perhaps out of date?. When I compare with the code (curl/typecheck-gcc.h), I see all of the necessary const attributes: /* evaluates to true if expr is of type curl_write_callback or similar */ #define _curl_is_write_cb(expr) \ (_curl_is_read_cb(expr) ||\ __builtin_types_compatible_p(__typeof__(expr), __typeof__(fwrite)) || \ __builtin_types_compatible_p(__typeof__(expr), curl_write_callback) || \ _curl_callback_compatible((expr), _curl_write_callback1) ||\ _curl_callback_compatible((expr), _curl_write_callback2) ||\ _curl_callback_compatible((expr), _curl_write_callback3) ||\ _curl_callback_compatible((expr), _curl_write_callback4) ||\ _curl_callback_compatible((expr), _curl_write_callback5) ||\ _curl_callback_compatible((expr), _curl_write_callback6)) typedef size_t (_curl_write_callback1)(const char *, size_t, size_t, void*); typedef size_t (_curl_write_callback2)(const char *, size_t, size_t, const void*); typedef size_t (_curl_write_callback3)(const char *, size_t, size_t, FILE*); typedef size_t (_curl_write_callback4)(const void *, size_t, size_t, void*); typedef size_t (_curl_write_callback5)(const void *, size_t, size_t, const void*); typedef size_t (_curl_write_callback6)(const void *, size_t, size_t, FILE*); But even if curl were requiring some suboptimal signature, it would be nice not to impose that on all projects that use hail. Are there older curl headers that do require the const-free signature? If there are and you want to support them, too, let me know -- maybe I can cook up an autoconf test to make things work there, with minimal impact. -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[patch tabled] distribute the sample tabled.conf
The doc/etc.tabled.conf is a vital piece of documentation but we forgot
to include it into the actual distribution. It was only available for
those pulling the git tree, not tarballs.
Also update the sample spec.
Signed-off-by: Pete Zaitcev zait...@redhat.com
---
doc/Makefile.am |1 +
pkg/tabled.spec |2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
commit 049a3242bd8a20837fbc7d8f091e19b447c364f1
Author: Pete Zaitcev zait...@yahoo.com
Date: Fri Oct 15 16:45:34 2010 -0600
Add the sample conf for distribution.
diff --git a/doc/Makefile.am b/doc/Makefile.am
index e344e04..d79a860 100644
--- a/doc/Makefile.am
+++ b/doc/Makefile.am
@@ -1,6 +1,7 @@
EXTRA_DIST = \
contributions.txt \
+ etc.tabled.conf \
setup.txt \
spec-todo.txt \
states.txt
diff --git a/pkg/tabled.spec b/pkg/tabled.spec
index ac93fb9..98004e6 100644
--- a/pkg/tabled.spec
+++ b/pkg/tabled.spec
@@ -83,7 +83,7 @@ fi
%files
%defattr(-,root,root,-)
-%doc AUTHORS COPYING LICENSE README NEWS doc/*.txt
+%doc AUTHORS COPYING LICENSE README NEWS doc/*.txt doc/etc.tabled.conf
%{_sbindir}/tabled
%{_sbindir}/tdbadm
%attr(0755,root,root) %{_initddir}/tabled
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] const-correctness tweaks
On 10/06/2010 08:07 AM, Jim Meyering wrote: Make write_cb callback's buffer parameter const, like all write-like functions. Give a few char * parameters the const attribute. Signed-off-by: Jim Meyeringmeyer...@redhat.com --- It looks like most of hail's interfaces are const-correct, but one stood out because it provokes a warning when I tried to pass a const-correct write_cb function to hstor_get from iwhd: proxy.c:382: warning: passing argument 4 of 'hstor_get' from \ incompatible pointer type /usr/include/hstor.h:173: note: expected \ 'size_t (*)(void *, size_t, size_t, void *)' but argument is of type \ 'size_t (*)(const void *, size_t, size_t, void *)' In case you feel comfortable fixing this, here's a patch: include/hstor.h |4 ++-- lib/hstor.c |5 +++-- lib/hutil.c |2 +- 3 files changed, 6 insertions(+), 5 deletions(-) This requires updating test/large-object.c in tabled, too. Would you mind sending along that companion patch? -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] const-correctness tweaks
Jeff Garzik wrote: On 10/06/2010 08:07 AM, Jim Meyering wrote: ... It looks like most of hail's interfaces are const-correct, but one stood out because it provokes a warning when I tried to pass a const-correct write_cb function to hstor_get from iwhd: ... include/hstor.h |4 ++-- lib/hstor.c |5 +++-- lib/hutil.c |2 +- 3 files changed, 6 insertions(+), 5 deletions(-) This requires updating test/large-object.c in tabled, too. Would you mind sending along that companion patch? Sure. Posting separately. With only one use, the tendrils were relatively short. -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH tabled] adapt to changed signature of hstor_get's callback function
* test/large-object.c: Hail has changed hstor_get's callback function
so that it now declares its buffer to be const, as all write-like
functions do. Adjust this file's hstor_get callback parameter and
propagate that, as required, to the local functions it uses to operate
on that now-read-only buffer.
Signed-off-by: Jim Meyering meyer...@redhat.com
---
test/large-object.c |9 +
1 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/test/large-object.c b/test/large-object.c
index dbe2027..fc7d03c 100644
--- a/test/large-object.c
+++ b/test/large-object.c
@@ -1,6 +1,6 @@
/*
- * Copyright 2008-2009 Red Hat, Inc.
+ * Copyright 2008-2010 Red Hat, Inc.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -60,7 +60,7 @@ static char key[] = Key of Large Object;
#define CSUM_INIT 0x
-static void incrsum(unsigned int *psum, unsigned char *data, size_t len)
+static void incrsum(unsigned int *psum, const unsigned char *data, size_t len)
{
unsigned int sum;
@@ -108,7 +108,7 @@ static size_t put_cb(void *ptr, size_t membsize, size_t
nmemb, void *user_data)
return rem;
}
-static size_t get_one(struct get_ctx *ctx, unsigned char *data, size_t len)
+static size_t get_one(struct get_ctx *ctx, const unsigned char *data, size_t
len)
{
unsigned num;
size_t rem;
@@ -143,7 +143,8 @@ static size_t get_one(struct get_ctx *ctx, unsigned char
*data, size_t len)
return rem;
}
-static size_t get_cb(void *ptr, size_t membsize, size_t nmemb, void *user_data)
+static size_t get_cb(const void *ptr, size_t membsize, size_t nmemb,
+void *user_data)
{
struct get_ctx *ctx = user_data;
size_t togo, len;
--
1.7.3.1.50.g1e633
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH hail] const-correctness tweaks
Make write_cb callback's buffer parameter const, like all write-like
functions.
Give a few char * parameters the const attribute.
Signed-off-by: Jim Meyering meyer...@redhat.com
---
It looks like most of hail's interfaces are const-correct,
but one stood out because it provokes a warning when I tried to
pass a const-correct write_cb function to hstor_get from iwhd:
proxy.c:382: warning: passing argument 4 of 'hstor_get' from \
incompatible pointer type
/usr/include/hstor.h:173: note: expected \
'size_t (*)(void *, size_t, size_t, void *)' but argument is of type \
'size_t (*)(const void *, size_t, size_t, void *)'
In case you feel comfortable fixing this, here's a patch:
include/hstor.h |4 ++--
lib/hstor.c |5 +++--
lib/hutil.c |2 +-
3 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/include/hstor.h b/include/hstor.h
index 8620d3b..b47387b 100644
--- a/include/hstor.h
+++ b/include/hstor.h
@@ -132,7 +132,7 @@ enum ReqACLC {
/* hutil.c */
extern char *hutil_time2str(char *buf, int len, time_t time);
extern time_t hutil_str2time(const char *timestr);
-extern int hreq_hdr_push(struct http_req *req, char *key, char *val);
+extern int hreq_hdr_push(struct http_req *req, const char *key, const char
*val);
extern char *hreq_hdr(struct http_req *req, const char *key);
extern void hreq_sign(struct http_req *req, const char *bucket, const char
*key,
char *b64hmac_out);
@@ -171,7 +171,7 @@ extern bool hstor_del_bucket(struct hstor_client *hstor,
const char *name);
extern struct hstor_blist *hstor_list_buckets(struct hstor_client *hstor);
extern bool hstor_get(struct hstor_client *hstor, const char *bucket, const
char *key,
-size_t (*write_cb)(void *, size_t, size_t, void *),
+size_t (*write_cb)(const void *, size_t, size_t, void *),
void *user_data, bool want_headers);
extern void *hstor_get_inline(struct hstor_client *hstor, const char *bucket,
const char *key, bool want_headers, size_t *len);
diff --git a/lib/hstor.c b/lib/hstor.c
index d0d87c7..7f638ec 100644
--- a/lib/hstor.c
+++ b/lib/hstor.c
@@ -86,7 +86,8 @@ err_out:
return NULL;
}
-static size_t all_data_cb(void *ptr, size_t size, size_t nmemb, void
*user_data)
+static size_t all_data_cb(const void *ptr, size_t size, size_t nmemb,
+ void *user_data)
{
GByteArray *all_data = user_data;
int len = size * nmemb;
@@ -378,7 +379,7 @@ bool hstor_del_bucket(struct hstor_client *hstor, const
char *name)
}
bool hstor_get(struct hstor_client *hstor, const char *bucket, const char *key,
-size_t (*write_cb)(void *, size_t, size_t, void *),
+size_t (*write_cb)(const void *, size_t, size_t, void *),
void *user_data, bool want_headers)
{
struct http_req req;
diff --git a/lib/hutil.c b/lib/hutil.c
index 7439d52..13a8d5e 100644
--- a/lib/hutil.c
+++ b/lib/hutil.c
@@ -131,7 +131,7 @@ static void cust_fin(struct custom_hdr_vec *cv)
/*
*/
-int hreq_hdr_push(struct http_req *req, char *key, char *val)
+int hreq_hdr_push(struct http_req *req, const char *key, const char *val)
{
struct http_hdr *hdr;
--
1.7.3.1.50.g1e633
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[hail patch 1/2] Change cfgfile.txt into a real config file
Not many people read Recipies page at the wiki, so let's change the in-tree cfgfile.txt so that it can be copied into /etc. Signed-off-by: Pete Zaitcev zait...@redhat.com --- doc/cfgfile.txt | 57 +++--- 1 file changed, 19 insertions(+), 38 deletions(-) diff --git a/doc/cfgfile.txt b/doc/cfgfile.txt index 6c55c9b..deb1507 100644 --- a/doc/cfgfile.txt +++ b/doc/cfgfile.txt @@ -1,8 +1,10 @@ - +!-- chunkd's configuration file is XML-like, but not strictly XML. +In particular, clause tags are case-sensitive. +Anyhow, just copy this file into /etc/chunkd.conf and supply it to -C option. +-- -listen addresses - +!-- You may specify multiple port, and/or multiple (address,port) combinations using Listen, Node, Port, and Encrypt: @@ -28,47 +30,26 @@ using Listen, Node, Port, and Encrypt: Port18080/Port Encrypttrue/Encrypt /Listen +-- +Listen Port8082/Port /Listen +!-- The default is usually acceptable -- +!-- PID/var/run/chunkd.pid/PID -- -Listen Record delimiter -Node Node address to which the TCP server will bind - (exclude Node for wildcard address) -Port Numeric port or service, to which TCP server will bind -Encrypt TLS is enabled on this port - - -volume served -= -You specify the volume served by this daemon using Path. - -Your basic setup, an empty directory: - Path/disk/massive/data/Path +Path/q/chunk-vega/Path !-- any /home directory will do -- +!-- Anything unique works: digits of IP address, time_t of creation. -- +NID13/NID -Path Path to volume +!-- + If you run tabled, vega is a Group tag where looks for chunks. + Otherwise, any path in CLD namespace works. + -- +InfoPath/chunk-vega/13/InfoPath - -SSL settings - -You specify server SSL attributes using SSL, PrivateKey, and Cert. - -Your basic setup: +!-- SSL works, although very few people/programs use it. Tabled doesn't. SSL PrivateKey/etc/pki/chunkd.pem/PrivateKey Cert/etc/pkt/cert.pem/Cert /SSL - - -SSL Record delimiter -PrivateKey Pathname of SSL private key PEM file -Cert Pathname of Certificate public key PEM file - - -miscellaneous settings -== - -PID file contained the process id of the daemon after forking, normally -written somewhere in /var/run: - - PID/var/run/chunkd.pid/PID - +-- -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
CLD multi-node status
Hi, is it correct that CLD is basically single-master right now? I can't find any trace of the mentioned Paxos implementation in the source. Regards, Geert -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: CLD multi-node status
On 09/30/2010 04:55 AM, Geert Jansen wrote: is it correct that CLD is basically single-master right now? I can't find any trace of the mentioned Paxos implementation in the source. The current main branch is single-master, correct. The 'replica' branch of hail.git contains the multi-node server -- where paxos implementation is imported from db4 replication engine. No multi-node client lib update exists, however. Look for this to change in the next 2 weeks, though! Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] chunkd: don't leak an FS object iterator
On 09/29/2010 11:20 AM, Jim Meyering wrote: chk_list_objs called fs_list_objs_open without also calling fs_list_objs_close. 32,808 bytes in 1 blocks are definitely lost in loss record 413 of 419 at 0x4A0515D: malloc (vg_replace_malloc.c:195) by 0x31BA8A26D0: __alloc_dir (opendir.c:184) by 0x405619: fs_list_objs_open (be-fs.c:974) by 0x40B202: chk_list_objs (selfcheck.c:41) by 0x40B575: chk_dbscan (selfcheck.c:131) by 0x40B628: chk_thread_scan (selfcheck.c:147) by 0x40B757: chk_thread_command (selfcheck.c:179) by 0x40B890: chk_thread_func (selfcheck.c:219) by 0x31BC464E83: g_thread_create_proxy (gthread.c:1893) by 0x31BB407760: start_thread (pthread_create.c:301) by 0x31BA8E151C: clone (clone.S:115) After seeing a few valgrind references from you, I'm curious... do you by chance happen to have a valgrind suppression file for openssl on Fedora? I've been wanting to run valgrind on chunkd, but each time I attempt it, I -- and valgrind -- have been overwhelmed by openssl false positives. openssl, deep in its RAND_xxx functions, intentionally does crazy stuff like using random, uninitialized stack contents as RNG entropy. Cute, but valgrind quite rightly complains loudly about it. It's a topic I've been meaning to research, because I currently lack the valgrind-fu necessary to have an effective valgrind+chunkd session. Thanks, Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] chunkd: don't leak an FS object iterator
On 09/29/2010 11:20 AM, Jim Meyering wrote: chk_list_objs called fs_list_objs_open without also calling fs_list_objs_close. 32,808 bytes in 1 blocks are definitely lost in loss record 413 of 419 at 0x4A0515D: malloc (vg_replace_malloc.c:195) by 0x31BA8A26D0: __alloc_dir (opendir.c:184) by 0x405619: fs_list_objs_open (be-fs.c:974) by 0x40B202: chk_list_objs (selfcheck.c:41) by 0x40B575: chk_dbscan (selfcheck.c:131) by 0x40B628: chk_thread_scan (selfcheck.c:147) by 0x40B757: chk_thread_command (selfcheck.c:179) by 0x40B890: chk_thread_func (selfcheck.c:219) by 0x31BC464E83: g_thread_create_proxy (gthread.c:1893) by 0x31BB407760: start_thread (pthread_create.c:301) by 0x31BA8E151C: clone (clone.S:115) Signed-off-by: Jim Meyeringmeyer...@redhat.com applied -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[chunkd patch] convert to libevent
For a nice code savings...
chunkd/Makefile.am |1
chunkd/chunkd.h| 28 +
chunkd/cldu.c | 64 +--
chunkd/server.c| 289 +
chunkd/util.c | 23
configure.ac |3
6 files changed, 116 insertions(+), 292 deletions(-)
diff --git a/chunkd/Makefile.am b/chunkd/Makefile.am
index 78bba72..a45a89b 100644
--- a/chunkd/Makefile.am
+++ b/chunkd/Makefile.am
@@ -10,4 +10,5 @@ chunkd_SOURCES= chunkd.h \
objcache.c
chunkd_LDADD = \
../lib/libhail.la @GLIB_LIBS@ @CRYPTO_LIBS@ \
+ @EVENT_LIBS@ \
@SSL_LIBS@ @TOKYOCABINET_LIBS@ @XML_LIBS@ @LIBCURL@
diff --git a/chunkd/chunkd.h b/chunkd/chunkd.h
index 937573c..5be155a 100644
--- a/chunkd/chunkd.h
+++ b/chunkd/chunkd.h
@@ -28,7 +28,7 @@
#include chunk_msg.h
#include hail_log.h
#include tchdb.h
-#include cldc.h /* for cld_timer */
+#include event.h
#include objcache.h
#ifndef ARRAY_SIZE
@@ -77,6 +77,8 @@ struct client {
charaddr_host[64]; /* ASCII version of inet addr */
charaddr_port[16]; /* ASCII version of port */
int fd; /* socket */
+ struct eventev;
+ short ev_mask;/* EV_READ and/or EV_WRITE */
charuser[CHD_USER_SZ + 1];
@@ -172,18 +174,10 @@ struct server_stats {
unsigned long opt_write; /* optimistic writes */
};
-struct server_poll {
- short events; /* POLL* from poll.h */
- boolbusy; /* if true, do not poll us */
-
- /* callback function, data */
- bool(*cb)(int fd, short events, void *userdata);
- void*userdata;
-};
-
struct server_socket {
int fd;
const struct listen_cfg *cfg;
+ struct eventev;
struct list_headsockets_node;
};
@@ -207,14 +201,15 @@ struct server {
char*pid_file; /* PID file */
int pid_fd;
+ struct event_base *evbase_main;
+
struct list_headlisteners;
struct list_headsockets;/* points into listeners */
- GHashTable *fd_info;
-
GThreadPool *workers; /* global thread worker pool */
int max_workers;
int worker_pipe[2];
+ struct eventworker_ev;
struct list_headwr_trash;
unsigned inttrash_sz;
@@ -311,11 +306,6 @@ extern void syslogerr(const char *prefix);
extern void strup(char *s);
extern int write_pid_file(const char *pid_fn);
extern int fsetflags(const char *prefix, int fd, int or_flags);
-extern void timer_init(struct cld_timer *timer, const char *name,
- void (*cb)(struct cld_timer *), void *userdata);
-extern void timer_add(struct cld_timer *timer, time_t expires);
-extern void timer_del(struct cld_timer *timer);
-extern time_t timers_run(void);
extern char *time2str(char *strbuf, time_t time);
extern void hexstr(const unsigned char *buf, size_t buf_len, char *outstr);
@@ -328,7 +318,7 @@ extern bool cli_err(struct client *cli, enum chunk_errcode
code, bool recycle_ok
extern int cli_writeq(struct client *cli, const void *buf, unsigned int buflen,
cli_write_func cb, void *cb_data);
extern bool cli_wr_sendfile(struct client *, cli_write_func);
-extern bool cli_rd_set_poll(struct client *cli, bool readable);
+extern void cli_rd_set_poll(struct client *cli, bool readable);
extern void cli_wr_set_poll(struct client *cli, bool writable);
extern bool cli_cb_free(struct client *cli, struct client_write *wr,
bool done);
@@ -336,7 +326,7 @@ extern bool cli_write_start(struct client *cli);
extern int cli_req_avail(struct client *cli);
extern int cli_poll_mod(struct client *cli);
extern bool worker_pipe_signal(struct worker_info *wi);
-extern bool tcp_cli_event(int fd, short events, void *userdata);
+extern void tcp_cli_event(int fd, short events, void *userdata);
extern void resp_init_req(struct chunksrv_resp *resp,
const struct chunksrv_req *req);
diff --git a/chunkd/cldu.c b/chunkd/cldu.c
index dd8b67c..026c523 100644
--- a/chunkd/cldu.c
+++ b/chunkd/cldu.c
@@ -21,6 +21,7 @@
#include hail-config.h
#include sys/types.h
+#include sys/time.h
#include sys/socket.h
#include glib.h
#include syslog.h
@@ -39,21 +40,23 @@ struct cld_host {
};
struct cld_session {
- bool forced_hosts; /* Administrator overrode default CLD */
- bool is_dead;
- struct ncld_sess *nsess;/* library state */
+ bool
Re: Autostart
On Wed, Sep 29, 2010 at 7:09 PM, Pete Zaitcev zait...@redhat.com wrote: An interesting question is what to do when iwhd exits. I decided not to kill what was started. So, we have a little self-contained cell of tabled, chunkd, S3, based off a certain local directory or other namespace anchor. Therefore, when iwhd restarts, it tests if the cell is still there, and uses that. It also tests if the service started successfuly, using the same method. As we see, for each service iwhd starts, it needs to verify that it's available (either before spawning it, or after). This would be done best by talking to the service. But iwhd only has S3 client, and no CLD client, so it cannot talk to cld (or chunkd). I had an idea: add an autostart feature to tabled. Tabled knows how to talk to both chunkd and cld, so it can verify that they are running. It would not be that much code. The downside is that it's clearly a special case, encoding of a policy. So I am asking how objectionable it would be (including do we want tabled -a for tests... they kinda run ok as they are). It seems like quite a special case feature. tabled is designed to use multiple chunkd nodes (and hopefully soon, multiple cld nodes). So having tabled start chunkd/cld seems misaligned with the existing design. That said, if it was possible to write a script or program that performed autostart without modifying tabled, it would be a nice addition to the git repository. tabled.autostart could be a simple program that pinged cld/chunkd, started if necessary, then exec'd the real tabled. Something modular and separate like that would be great. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH hail] chunkd: don't leak an FS object iterator
chk_list_objs called fs_list_objs_open without also calling fs_list_objs_close. 32,808 bytes in 1 blocks are definitely lost in loss record 413 of 419 at 0x4A0515D: malloc (vg_replace_malloc.c:195) by 0x31BA8A26D0: __alloc_dir (opendir.c:184) by 0x405619: fs_list_objs_open (be-fs.c:974) by 0x40B202: chk_list_objs (selfcheck.c:41) by 0x40B575: chk_dbscan (selfcheck.c:131) by 0x40B628: chk_thread_scan (selfcheck.c:147) by 0x40B757: chk_thread_command (selfcheck.c:179) by 0x40B890: chk_thread_func (selfcheck.c:219) by 0x31BC464E83: g_thread_create_proxy (gthread.c:1893) by 0x31BB407760: start_thread (pthread_create.c:301) by 0x31BA8E151C: clone (clone.S:115) Signed-off-by: Jim Meyering meyer...@redhat.com --- Thanks to Pete for catching my error. To make up for that, here's a real leak fix: chunkd/selfcheck.c |1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/chunkd/selfcheck.c b/chunkd/selfcheck.c index f3713da..86d3eb2 100644 --- a/chunkd/selfcheck.c +++ b/chunkd/selfcheck.c @@ -100,6 +100,7 @@ static void chk_list_objs(struct chk_tls *tls, uint32_t table_id) free(fn); } + fs_list_objs_close(lister); } static void chk_dbscan(struct chk_tls *tls) -- 1.7.3.293.gca9a76 -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Autostart
Jeff et.al.: I am working on a feature called autostart in Image Warehouse. It amounts to iwhd starting with a certain default configuration, but is a bit more: it also launches services necessary to support said default, e.g. tabled. Naturally, tabled needs cld and chunkd. An interesting question is what to do when iwhd exits. I decided not to kill what was started. So, we have a little self-contained cell of tabled, chunkd, S3, based off a certain local directory or other namespace anchor. Therefore, when iwhd restarts, it tests if the cell is still there, and uses that. It also tests if the service started successfuly, using the same method. As we see, for each service iwhd starts, it needs to verify that it's available (either before spawning it, or after). This would be done best by talking to the service. But iwhd only has S3 client, and no CLD client, so it cannot talk to cld (or chunkd). I had an idea: add an autostart feature to tabled. Tabled knows how to talk to both chunkd and cld, so it can verify that they are running. It would not be that much code. The downside is that it's clearly a special case, encoding of a policy. So I am asking how objectionable it would be (including do we want tabled -a for tests... they kinda run ok as they are). Cheers, -- Pete -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH hail] lib/hstor.c: avoid an unconditional leak in append_qparam
Signed-off-by: Jim Meyering meyer...@redhat.com
---
I would have preferred to insert a single line right before the
huri_field_escape call:
char *v = strdup(val);
[would result in a more compact, single-hunk patch]
but it looks like hail uses the anachronistic (pre-C99)
declare all vars at outer scope style, so I conformed.
lib/hstor.c |5 -
1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/lib/hstor.c b/lib/hstor.c
index 6c67bfa..79e0420 100644
--- a/lib/hstor.c
+++ b/lib/hstor.c
@@ -676,6 +676,7 @@ static GString *append_qparam(GString *str, const char
*key, const char *val,
char *arg_char)
{
char *stmp;
+ char *v;
str = g_string_append(str, arg_char);
arg_char[0] = '';
@@ -683,9 +684,11 @@ static GString *append_qparam(GString *str, const char
*key, const char *val,
str = g_string_append(str, key);
str = g_string_append(str, =);
- stmp = huri_field_escape(strdup(val), QUERY_ESCAPE_MASK);
+ v = strdup(val);
+ stmp = huri_field_escape(v, QUERY_ESCAPE_MASK);
str = g_string_append(str, stmp);
free(stmp);
+ free(v);
return str;
}
--
1.7.3.234.g7bba3
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] lib/hstor.c: avoid an unconditional leak in append_qparam
On Mon, 27 Sep 2010 10:53:06 +0200 Jim Meyering j...@meyering.net wrote: - stmp = huri_field_escape(strdup(val), QUERY_ESCAPE_MASK); + v = strdup(val); + stmp = huri_field_escape(v, QUERY_ESCAPE_MASK); str = g_string_append(str, stmp); free(stmp); + free(v); I think you may be fooled by the ridiculous calling convention of huri_field_escape(). It takes a pointer to heap, then either returns its argument, or reallocates it, frees the argument, and returns the reallocated area. It frees with g_free, so it assumes its equivalence with free(), haha. The end result, it either returns what strdup returned of frees it. Therefore if you free what strudup returned, you double-free it. I honestly think this madness must stop and huri_field_escape must allocate a new buffer every time. Then we would not need the strdup there at all. It only exists to satisfy the requirement to pass a pointer to heap in case val is a const or whatnot. -- Pete -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] lib/hstor.c: avoid an unconditional leak in append_qparam
On 09/27/2010 04:53 AM, Jim Meyering wrote:
Signed-off-by: Jim Meyeringmeyer...@redhat.com
---
I would have preferred to insert a single line right before the
huri_field_escape call:
char *v = strdup(val);
[would result in a more compact, single-hunk patch]
but it looks like hail uses the anachronistic (pre-C99)
declare all vars at outer scope style, so I conformed.
lib/hstor.c |5 -
1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/lib/hstor.c b/lib/hstor.c
index 6c67bfa..79e0420 100644
--- a/lib/hstor.c
+++ b/lib/hstor.c
@@ -676,6 +676,7 @@ static GString *append_qparam(GString *str, const char
*key, const char *val,
char *arg_char)
{
char *stmp;
+ char *v;
str = g_string_append(str, arg_char);
arg_char[0] = '';
@@ -683,9 +684,11 @@ static GString *append_qparam(GString *str, const char
*key, const char *val,
str = g_string_append(str, key);
str = g_string_append(str, =);
- stmp = huri_field_escape(strdup(val), QUERY_ESCAPE_MASK);
+ v = strdup(val);
+ stmp = huri_field_escape(v, QUERY_ESCAPE_MASK);
str = g_string_append(str, stmp);
free(stmp);
+ free(v);
applied
Yeah, I don't like C++ var decls; I think the code gets too
disorganized, making it really easy to miss a decl when reviewing.
Jeff
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] lib/hstor.c: avoid an unconditional leak in append_qparam
Pete Zaitcev wrote: On Mon, 27 Sep 2010 10:53:06 +0200 Jim Meyering j...@meyering.net wrote: -stmp = huri_field_escape(strdup(val), QUERY_ESCAPE_MASK); +v = strdup(val); +stmp = huri_field_escape(v, QUERY_ESCAPE_MASK); str = g_string_append(str, stmp); free(stmp); +free(v); I think you may be fooled by the ridiculous calling convention of huri_field_escape(). It takes a pointer to heap, then either returns its argument, or reallocates it, frees the argument, and returns the reallocated area. It frees with g_free, so it assumes its equivalence with free(), haha. The end result, it either returns what strdup returned of frees it. Therefore if you free what strudup returned, you double-free it. Oh! You're right. I missed the g_free (signed_str); at the end of huri_field_escape. Sorry about that. I honestly think this madness must stop and huri_field_escape must allocate a new buffer every time. Then we would not need the strdup there at all. It only exists to satisfy the requirement to pass a pointer to heap in case val is a const or whatnot. Making a function like huri_field_escape free a buffer allocated by the caller does seem to violate something fundamental. -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] lib/hstor.c: avoid an unconditional leak in append_qparam
On 09/27/2010 12:29 PM, Pete Zaitcev wrote: On Mon, 27 Sep 2010 10:53:06 +0200 Jim Meyeringj...@meyering.net wrote: - stmp = huri_field_escape(strdup(val), QUERY_ESCAPE_MASK); + v = strdup(val); + stmp = huri_field_escape(v, QUERY_ESCAPE_MASK); str = g_string_append(str, stmp); free(stmp); + free(v); I think you may be fooled by the ridiculous calling convention Doh, my memory and I were fooled, too. -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH hail] lib/hstor.c: avoid an unconditional leak in append_qparam
On Mon, 27 Sep 2010 12:53:48 -0400 Jeff Garzik j...@garzik.org wrote: - stmp = huri_field_escape(strdup(val), QUERY_ESCAPE_MASK); + v = strdup(val); + stmp = huri_field_escape(v, QUERY_ESCAPE_MASK); str = g_string_append(str, stmp); free(stmp); + free(v); applied I'm going to post a patch that undoes the damage. It's in testing. -- Pete -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [hail patch 1/1] Fix calling convention of huri_field_escape
On 09/27/2010 08:49 PM, Pete Zaitcev wrote: Premature optimization is the root of all evil. Use a sensible convention of not screwing with the argument, at the expense of extra strdup. Fortunately, all users are confined to Hail itself, even if huri_field_escape is exported. Signed-off-by: Pete Zaitcevzait...@redhat.com --- include/hstor.h |2 +- lib/hstor.c | 44 +--- lib/huri.c | 10 +- 3 files changed, 35 insertions(+), 21 deletions(-) applied -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [tabled patch 1/1] Add a test for hstor_keys
On 09/27/2010 08:52 PM, Pete Zaitcev wrote: Our current tests do not invoke hstor_keys at all, and so they did not catch a crash with double free in append_qparam. Add a very basic test which at least calls hstor_keys to verify that it does not crash right away. This test does not excercise complex modes such as S3 paging, but better this than nothing. Signed-off-by: Pete Zaitcevzait...@redhat.com --- test/Makefile.am |4 + test/list-keys.c | 102 + 2 files changed, 105 insertions(+), 1 deletion(-) applied.. FYI you forgot to update test/.gitignore. Fixed. -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH tabled 1/2] server/config.c: don't dereference NULL on OOM
Jeff Garzik wrote:
On 09/23/2010 04:43 AM, Jim Meyering wrote:
From fb7865d158b0d32907dde703c4d37c70a26e738c Mon Sep 17 00:00:00 2001
From: Jim Meyeringmeyer...@redhat.com
Date: Thu, 23 Sep 2010 10:11:44 +0200
Subject: [PATCH tabled 1/2] server/config.c: don't dereference NULL on OOM
(see other email for general response to these changes, comments on
GLib, OOM, etc.)
First off, I ACK (accept) all these changes. Technically they appear
correct, and I am interested in merging them.
But I request a few minor style and workflow adjustments, and a
resubmission. Specific comments:
[style]
1) the functional style of sizeof keyword, with parens, is preferred:
- snprintf(s, 64, get user '%s', user);
+ snprintf(s, sizeof s, get user '%s', user);
Sure. Adjusted.
2) it is preferred to omit optional braces for singleton test+stmt
style statements:
+ if (!pass) {
+ goto err_cmp;
+ }
Gladly. That's what I would have done in code I own, but there is a
braced single-line else block just above, so I presumed that the
style was always use braces. (I think we have the same preferences,
since I too would use braces around the single-line else in that case,
though not if the then block had also been a one-liner.
[patch submission administrivia]
3) I process patches similar to how Linus and others in the kernel do
it: git am /path/to/mbox_of_patches That tends to impose some
restrictions on the contents of each email.
In your case, while the patch descriptions and diffs themselves are
correct, you seem to be sending one-mbox-per-email, while I'm
expecting one-patch-per-email. If you could tweak your process to
make that change, that would reduce the manual labor on my part.
No problem.
4) While total number of patches is not really a problem, I would
request sweeping most of the one-and-two-liners in this series into a
single patch, leaving perhaps only the bucket.c and status.c changes
as standalone patches.
Will do.
You can tell that I'm too accustomed to posting FYI-patches
that I will shortly push -- or that I'll push upon review.
It's more an art and style preferences, than science, when deciding
how to separate out changes into patches. Trying to take my cues from
the kernel, it is preferred, for example, that bug fixes be separate
from new features, or whitespace and cosmetic changes separate from
functional changes. But it is also encouraged to group similar
changes together, if, for example, you're making a similar change
across a large number of files.
Mailing list review-ability, useful 'git bisect' boundaries, and a
coherent 'git shortlog' summary tend to be my guides when deciding
patch boundaries.
Preaching to the choir ;-)
Thanks for spelling out your guidelines.
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH tabled 1/2] server/config.c: don't dereference NULL on OOM
Jeff Garzik wrote:
On 09/23/2010 03:19 PM, Jeff Garzik wrote:
3) I process patches similar to how Linus and others in the kernel do
it: git am /path/to/mbox_of_patches That tends to impose some
restrictions on the contents of each email.
FWIW, 'git pull' submissions are welcome. Standard kernel-style pull
submission style applies[1].
Jeff
[1] public git pull URL including branch name, diffstat, shortlog or
full log of changeset summaries, and finally, the combined diff of all
changes.
Here you go.
You can pull from the oom branch here:
git://git.infradead.org/users/meyering/tabled.git
I think I've addressed all of your preferences, merging
most OOM fixes into one commit, but not the two you mentioned
that should stay separate. I also left the sizeof(s) one separate.
However, I did leave the copyright year updates in.
If they're a problem, let me know and I'll do another round.
Otherwise, I can send a patch to update all of the
remaining ones to include 2010 so this won't be an
issue for 3 more months.
$ git shortlog HEAD ^origin/master
Jim Meyering (4):
server/server.c: use sizeof(s) rather than equivalent 64
don't dereference NULL on OOM
server/status.c: don't deref NULL on failed strdup
server/bucket.c: don't deref NULL upon failed malloc
b/server/bucket.c | 25 -
b/server/config.c |7 +--
b/server/object.c |7 ++-
b/server/replica.c |7 +--
b/server/server.c |2 +-
b/server/status.c |8 +---
server/server.c| 13 +
7 files changed, 47 insertions(+), 22 deletions(-)
diff --git a/server/bucket.c b/server/bucket.c
index eb03e03..cf42d2d 100644
--- a/server/bucket.c
+++ b/server/bucket.c
@@ -1,6 +1,6 @@
/*
- * Copyright 2008-2009 Red Hat, Inc.
+ * Copyright 2008-2010 Red Hat, Inc.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -788,29 +788,36 @@ static GList *bucket_list_pfx(GList *content, GHashTable
*common_pfx,
s = malloc(cpfx_len);
p = s;
+#define append_const(buf, c) \
+ do { memcpy(buf, c, sizeof(c)-1); (buf) += sizeof(c)-1; } while (0)
+
tmpl = pfx_list;
while (tmpl) {
prefix = (char *) tmpl-data;
pfx_len = strlen(prefix);
- memcpy(p, optag, sizeof(optag)-1); p += sizeof(optag)-1;
- memcpy(p, pfoptag, sizeof(pfoptag)-1); p += sizeof(pfoptag)-1;
- memcpy(p, prefix, pfx_len); p += pfx_len;
- memcpy(p, delim, delim_len); p += delim_len;
- memcpy(p, pfedtag, sizeof(pfedtag)-1); p += sizeof(pfedtag)-1;
- memcpy(p, edtag, sizeof(edtag)-1); p += sizeof(edtag)-1;
+ if (p) {
+ append_const(p, optag);
+ append_const(p, pfoptag);
+ memcpy(p, prefix, pfx_len); p += pfx_len;
+ memcpy(p, delim, delim_len); p += delim_len;
+ append_const(p, pfedtag);
+ append_const(p, edtag);
+ }
free(prefix);
tmpl = tmpl-next;
}
- *p = 0;
+ if (p)
+ *p = 0;
free(delim);
g_list_free(pfx_list);
- return g_list_append(content, s);
+ return s ? g_list_append(content, s) : content;
}
+#undef append_const
struct bucket_list_info {
char *prefix;
diff --git a/server/config.c b/server/config.c
index f94886e..a58a0e6 100644
--- a/server/config.c
+++ b/server/config.c
@@ -1,6 +1,6 @@
/*
- * Copyright 2009 Red Hat, Inc.
+ * Copyright 2009, 2010 Red Hat, Inc.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -436,7 +436,10 @@ void read_config(void)
memset(ctx, 0, sizeof(struct config_context));
- tabled_srv.port = strdup(8080);
+ if (!(tabled_srv.port = strdup(8080))) {
+ applog(LOG_ERR, no core);
+ exit(1);
+ }
if (!g_file_get_contents(tabled_srv.config, text, len, NULL)) {
applog(LOG_ERR, failed to read config file %s,
diff --git a/server/object.c b/server/object.c
index 3801e94..1f2f68f 100644
--- a/server/object.c
+++ b/server/object.c
@@ -1,6 +1,6 @@
/*
- * Copyright 2008-2009 Red Hat, Inc.
+ * Copyright 2008-2010 Red Hat, Inc.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -801,6 +801,11 @@ static bool object_put_body(struct client *cli, const char
*user,
cli-out_objid = objid;
cli-out_user = strdup(user);
+ if (!cli-out_bucket || !cli-out_key || !cli-out_user) {
+ applog(LOG_ERR, OOM in object_put_body);
+ return cli_err(cli,
Re: [PATCH tabled 1/2] server/config.c: don't dereference NULL on OOM
On 09/24/2010 07:32 AM, Jim Meyering wrote: You can pull from the oom branch here: git://git.infradead.org/users/meyering/tabled.git Got nearly everything perfect. Need one more minor yet important change. As described in doc/contributions.txt, every changeset MUST have a Signed-off-by line at the end of a changeset's description. I was able to pull and build just fine, so your git repo setup and push appears correct. Also, in your pull request, please put the branch immediately following the repo URL on the same line, for easier cut-n-paste. Here's how Linus requests his pull-requests to look: ---SNIP- Please pull from 'upstream-linus' branch of git://git.kernel.org/pub/scm/git/jgarzik/libata-dev.git upstream-linus to receive the following updates: drivers/ata/ahci.c| 193 +++-- drivers/ata/libata-acpi.c | 40 +- drivers/ata/libata-core.c |3 + drivers/ata/libata.h |2 + drivers/ata/pata_ali.c|2 +- include/linux/ata.h |9 ++- include/linux/libata.h| 12 +++ 7 files changed, 178 insertions(+), 83 deletions(-) Dirk Hohndel (1): pata_ali: trivial fix of a very frequent spelling mistake Robert Hancock (1): ahci: display all AHCI 1.3 HBA capability flags (v2) Tejun Heo (5): ahci: disable 64bit DMA by default on SB600s libata: cosmetic updates libata: implement more acpi filtering options libata: make gtf_filter per-dev ahci: filter FPDMA non-zero offset enable for Aspire 3810T diff --git a/drivers/ata/ahci.c b/drivers/ata/ahci.c index acd1162..4edca6e 100644 --- a/drivers/ata/ahci.c +++ b/drivers/ata/ahci.c [COMBINED PATCH FOLLOWS...] ---SNIP- -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH tabled 1/2] server/config.c: don't dereference NULL on OOM
Jeff Garzik wrote: On 09/24/2010 07:32 AM, Jim Meyering wrote: You can pull from the oom branch here: git://git.infradead.org/users/meyering/tabled.git Got nearly everything perfect. Need one more minor yet important change. As described in doc/contributions.txt, every changeset MUST have a Signed-off-by line at the end of a changeset's description. I was able to pull and build just fine, so your git repo setup and push appears correct. Also, in your pull request, please put the branch immediately following the repo URL on the same line, for easier cut-n-paste. Here's how Linus requests his pull-requests to look: Ok. I've added those pesky S.O.B lines with this: git filter-branch --msg-filter \ 'cat printf \nSigned-off-by: Jim Meyering meyer...@redhat.com\n' \ HEAD~4..HEAD and pushed the result. Please pull from the 'oom' branch of git://git.infradead.org/users/meyering/tabled.git I presume there's no need to re-post the diffstat or diffs. -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH tabled 1/2] server/config.c: don't dereference NULL on OOM
On 09/24/2010 01:43 PM, Jim Meyering wrote: Jeff Garzik wrote: On 09/24/2010 07:32 AM, Jim Meyering wrote: You can pull from the oom branch here: git://git.infradead.org/users/meyering/tabled.git Got nearly everything perfect. Need one more minor yet important change. As described in doc/contributions.txt, every changeset MUST have a Signed-off-by line at the end of a changeset's description. I was able to pull and build just fine, so your git repo setup and push appears correct. Also, in your pull request, please put the branch immediately following the repo URL on the same line, for easier cut-n-paste. Here's how Linus requests his pull-requests to look: Ok. I've added those pesky S.O.B lines with this: git filter-branch --msg-filter \ 'cat printf \nSigned-off-by: Jim Meyeringmeyer...@redhat.com\n' \ HEAD~4..HEAD and pushed the result. Please pull from the 'oom' branch of git://git.infradead.org/users/meyering/tabled.git pulled from you pushed upstream, thanks! -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH tabled] server/server.c (net_write_port): Don't ignore write error.
On 09/23/2010 03:55 AM, Jim Meyering wrote: Better safe than sorry... Unreported write failures can be unpleasant. I fixed the one below so that a failure indication can propagate up the call tree. You might also want to report the failure to stderr. I let my editor automatically update the copyright date and remove trailing spaces. If you'd rather separate those from the fix, let me know and I can adjust and resend. Patch applied, thanks. The typical preference is to receive whitespace and other cosmetic changes in a separate patch, thereby highlighting the functional changes. But we're not so strict here that I would reject an otherwise useful patch... Also FWIW, we're not very strict about reproducing the GCC-ish (GNU-ish?) style of $FILENAME ($FUNCTION): in each changelog -- though you're certainly welcome to continue, if that's your preference. Given that git show $COMMIT will give you filename and per-diff-chunk function names, reproducing that in the git changelog entry seems somewhat redundant. A simple, English-language summary of the change is fine. Just a style tip, though, feel free to ignore! :) Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[tabled patch v2] abstract out TCP-write code
Changes from v1:
- avoid referencing dead struct client (grep for 'invalidate_cli'),
by changing FSM callback prototype.
- insert 'void *priv' member into struct atcp_wr_state, and replace
cb_data1/cb_data2 callback parameters with (struct atcp_wr_state *, void *).
struct client / struct session, or whatever, may be stored in
atcp_wr_state::priv.
- minor API polishing and further abstraction
server/Makefile.am |1
server/atcp.c | 238 +++
server/atcp.h | 100 +++
server/bucket.c|8 -
server/object.c| 56 +--
server/server.c| 268 +
server/status.c|3
server/tabled.h| 46 ++---
8 files changed, 436 insertions(+), 284 deletions(-)
diff --git a/server/Makefile.am b/server/Makefile.am
index 5b53a0a..5e0abd5 100644
--- a/server/Makefile.am
+++ b/server/Makefile.am
@@ -4,6 +4,7 @@ INCLUDES= -I$(top_srcdir)/include @GLIB_CFLAGS@
@HAIL_CFLAGS@
sbin_PROGRAMS = tabled tdbadm
tabled_SOURCES = tabled.h \
+ atcp.c atcp.h \
bucket.c cldu.c config.c metarep.c object.c replica.c \
server.c status.c storage.c storparse.c util.c
tabled_LDADD = ../lib/libtdb.a \
diff --git a/server/atcp.c b/server/atcp.c
new file mode 100644
index 000..0050a68
--- /dev/null
+++ b/server/atcp.c
@@ -0,0 +1,238 @@
+
+/*
+ * Copyright 2010 Red Hat, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; see the file COPYING. If not, write to
+ * the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA.
+ *
+ */
+
+#define _GNU_SOURCE
+#include tabled-config.h
+
+#include string.h
+#include stdlib.h
+#include errno.h
+#include sys/uio.h
+#include atcp.h
+
+bool atcp_cb_free(struct atcp_wr_state *wst, void *cb_data, bool done)
+{
+ free(cb_data);
+ return false;
+}
+
+static void atcp_write_complete(struct atcp_write *tmp)
+{
+ struct atcp_wr_state *wst = tmp-wst;
+
+ list_del(tmp-node);
+ list_add_tail(tmp-node, wst-write_compl_q);
+}
+
+static bool atcp_write_free(struct atcp_write *tmp, bool done)
+{
+ struct atcp_wr_state *wst = tmp-wst;
+ bool rcb = false;
+
+ wst-write_cnt -= tmp-length;
+ list_del_init(tmp-node);
+ if (tmp-cb)
+ rcb = tmp-cb(wst, tmp-cb_data, done);
+ free(tmp);
+
+ return rcb;
+}
+
+bool atcp_write_run_compl(struct atcp_wr_state *wst)
+{
+ struct atcp_write *wr;
+ bool do_loop;
+
+ do_loop = false;
+ while (!list_empty(wst-write_compl_q)) {
+ wr = list_entry(wst-write_compl_q.next,
+ struct atcp_write, node);
+ do_loop |= atcp_write_free(wr, true);
+ }
+ return do_loop;
+}
+
+void atcp_write_free_all(struct atcp_wr_state *wst)
+{
+ struct atcp_write *wr, *tmp;
+
+ atcp_write_run_compl(wst);
+ list_for_each_entry_safe(wr, tmp, wst-write_q, node) {
+ atcp_write_free(wr, false);
+ }
+}
+
+static bool atcp_writable(struct atcp_wr_state *wst)
+{
+ int n_iov;
+ struct atcp_write *tmp;
+ ssize_t rc;
+ struct iovec iov[ATCP_MAX_WR_IOV];
+
+ /* accumulate pending writes into iovec */
+ n_iov = 0;
+ list_for_each_entry(tmp, wst-write_q, node) {
+ if (n_iov == ATCP_MAX_WR_IOV)
+ break;
+ /* bleh, struct iovec should declare iov_base const */
+ iov[n_iov].iov_base = (void *) tmp-buf;
+ iov[n_iov].iov_len = tmp-togo;
+ n_iov++;
+ }
+
+ /* execute non-blocking write */
+do_write:
+ rc = writev(wst-fd, iov, n_iov);
+ if (rc 0) {
+ if (errno == EINTR)
+ goto do_write;
+ if (errno != EAGAIN)
+ goto err_out;
+ return true;
+ }
+
+ /* iterate through write queue, issuing completions based on
+* amount of data written
+*/
+ while (rc 0) {
+ int sz;
+
+ /* get pointer to first record on list */
+ tmp = list_entry(wst-write_q.next, struct atcp_write, node);
+
+ /* mark data consumed by decreasing tmp-len */
+ sz = (tmp-togo rc) ? tmp-togo : rc;
+ tmp-togo -= sz;
+ tmp-buf +=
Re: [tabled patch] abstract out TCP-write code
On 09/23/2010 11:28 AM, Jim Meyering wrote: Every developer should have MALLOC_PERTURB_=N (N in 1..255) set in his/her environment on glibc-based systems. Almost all the time. I heard about it a while ago, even submitted a bugzilla bug to have it documented adequately. But apparently its absent from my .bash_profile. Added. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [tabled patch] abstract out TCP-write code
On 09/22/2010 10:37 PM, Pete Zaitcev wrote: On Wed, 22 Sep 2010 21:26:13 -0400 Jeff Garzikj...@garzik.org wrote: It is a common idiom even in GLib that callbacks receive two anonymous pointers; witness the data type GFunc's 'data' and 'user_data' arguments: http://library.gnome.org/devel/glib/stable/glib-Doubly-Linked-Lists.html#GFunc There's a lot of retarged garbage in Glib, just look at their lists. If someone smarter wrote Glib, we would not need struct list_head. I use both list types, because there's a use case for both. You don't always have the luxury of having a struct in which to embed data+next pointers. Allocated strings are an excellent example. GFunc has two parameters for a reason :) See for example http://library.gnome.org/devel/glib/stable/glib-Doubly-Linked-Lists.html#g-list-foreach It really is a common idiom, based on a common need, not just my style preference. :) Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [tabled patch] abstract out TCP-write code
On 09/22/2010 10:37 PM, Pete Zaitcev wrote: On Wed, 22 Sep 2010 21:26:13 -0400 Jeff Garzikj...@garzik.org wrote: So, we go a longer route and re-hook the list of completions to a per-server global instead of a client. The patch is straight- forward. The only thing we need to be careful is to make sure that no outstanding completions are left in the queue before freeing a client struct. This is ensured by force-running completions. Looking at this change again, I don't see how this avoids use-after-free. If completions exist after state change function leads one to cli_evt_dispose() - cli_free(), then cli_write_run_compl() still calls cli_write_free() with the stale 'cli' pointer. We run completions before freeing in all cases. My patch was correct. Logically, if completions are run before freeing in all cases, there is no need to make write_compl_q global. That was a red herring, which by side effect avoided the bug with the stale 'cli' pointer. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Reconsidering libevent
On 09/21/2010 01:51 PM, Jeff Garzik wrote: Today's Project Hail hacking involves moving tcp_write_* code from itd.git to libhail. tcp_write_* is the well-debugged TCP async write queueing code from tabled, with cosmetic changes to provide a nice API encapsulation. I did this when moving the code originally from tabled - itd. By moving this code into libhail, with (yet more) cosmetic changes, the same TCP write queueing code may be used in both tabled and itd pkgs. chunkd'd async TCP write code is highly similar, and can be merged into libhail as well. (if you're curious why I'm doing this, it's one step in the road to updating itd to use chunkd's GET_PART) An interesting side effect: both tabled and itd use libevent, which means that libhail must now link with libevent. Is this a bad thing? Recently libevent was purged from cld (a simple use case, where poll sufficed) and chunkd (obscure concerns about multi-threading), but linking libevent into libhail is causing me to rethink the matter. My concerns about chunkd, libevent and M-T centered around libevent's use of a single, mutex-unprotected event_base into which events are queued etc. In one particular thread model, the set of listened-to events might be manipulated from multiple threads concurrently, thereby implying a commitment to libevent implies a commitment to libevent wrappers which add locks. However, it turns out that thread model is quite suboptimal compared to other methods. Assigning an incoming TCP connection to a thread, and having that thread manage that connection for its entire lifetime, is both much more efficient and quite compatible with libevent's API. One merely assigns an event_base to each thread. In Apache-land, this is called the prefork model (yes, I know, fork != thread; I didn't invent the terminology) So, I am reluctantly considering moving chunkd back to libevent. That will enable chunkd, tabled and itd to all share the same async TCP write code, and as a nice side effect, reduce our minor reinvention of libevent (c.f. libtimer and some of the network poll code). Comments welcome, as always. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html libevent version 2 has proper mutual exclusion, but the code needs some work. regards -steve -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Reconsidering libevent
On Tue, Sep 21, 2010 at 5:06 PM, Steven Dake sd...@redhat.com wrote: libevent version 2 has proper mutual exclusion, but the code needs some work. 1.x should work for chunkd at the moment. I need to resist my own urge to think too far ahead and overengineer for the future sometimes; I think this is one of those occasions. libevent 1.x seems solid for single-thread usage, and that's how we'll use libevent, even if multiple chunkd threads are in existence. Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: On licensing and libhail
On Tue, 21 Sep 2010 17:04:41 -0400 Jeff Garzik j...@garzik.org wrote: Occasionally I receive a ping about libhail licensing. GPL is probably less convenient than LGPL, so I was wondering what existing copyright holders thought about changing libhail to LGPL, while leaving the remaining bits GPL? Looks like Colin is the only one. I did my work on Red Hat's money, so it's their call. Just need to ask whoever replaced Mark Webbink. And we didn't have any other recorded contributors. -- Pete -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] don't expect inode name to be NUL-terminated (avoid read overrun)
Jeff Garzik wrote: On 09/10/2010 08:55 AM, Jim Meyering wrote: * server/msg.c (msg_get): Copy only name_len bytes, then NUL-terminate, rather than using snprintf to copy up to and including nonexistent NUL. --- valgrind exposed this. The use of snprintf would have been correct if the inode name buffer (following the struct raw_inode) were NUL-terminated, but it is not. applied -- good catch out of curiosity, what is your patch base? We combined cld and chunkd into a single 'hail' pkg, and from the pathname, your patch was generated from the older cld pkg. We'd like to find the source and replace cld/chunkd with 'hail'. F12? F13? rawhide? Hi Jeff, I was using the sources from here: git://git.kernel.org/pub/scm/daemon/cld/cld.git From your comment there must be a hail git repository. Found it: http://git.kernel.org/?p=daemon/distsrv/hail.git;a=summary FYI, when I searched for hail's git repository initially, https://hail.wiki.kernel.org/ was inaccessible, so I found the above in a presumably-old cache. -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [hail patch 0/3] chunkd: on-disk checksumming and get-partial operation
Just pushed this out to hail.git. -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] don't expect inode name to be NUL-terminated (avoid read overrun)
On 09/10/2010 08:55 AM, Jim Meyering wrote: * server/msg.c (msg_get): Copy only name_len bytes, then NUL-terminate, rather than using snprintf to copy up to and including nonexistent NUL. --- valgrind exposed this. The use of snprintf would have been correct if the inode name buffer (following the struct raw_inode) were NUL-terminated, but it is not. applied -- good catch out of curiosity, what is your patch base? We combined cld and chunkd into a single 'hail' pkg, and from the pathname, your patch was generated from the older cld pkg. We'd like to find the source and replace cld/chunkd with 'hail'. F12? F13? rawhide? Thanks, Jeff -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[hail patch 0/3] chunkd: on-disk checksumming and get-partial operation
This patchset is just about ready to go upstream. Just need to write a couple tests (familiar refrain eh?:)). These changes add a new Get-Partial-Object (GET_PART) chunkd operation. GET_PART permits partial retrieval of an object, by adding an (offset,length) pair to the standard Get-Object (GET) operation. length==0 is special-cased as meaning retrieve until end of object. The maximum number of bytes that may be requested in a single GET_PART request is 4 x 64k blocks (256k). Larger lengths will be truncated down to the maximum. Because we currently only store whole-object SHA1 checksums, we are left without an ability to verify on-disk data is valid, when retrieving a subset of an object. Thus, a necessary pre-req of GET_PART is changing the checksum scheme, which is done as follows: * objects are defined as runs of 64k logical blocks * checksums are stored on-disk for each 64k in an object * Rather than returning the stored SHA1 checksum, which serves to verify both on-disk and network integrity, we break this into two steps, * verify per-64k checksums at GET_PART time * generate on-the-fly SHA1 checksum for GET_PART returned data The chunkd network protocol supports any offset/length, including not-64k-aligned values. However, failure to align GET_PART requests on 64k boundaries will result in reduced performance, due to additional work chunkd must perform [and then throw away], because chunkd now works in 64k chunks internally. This is a major protocol milestone, and should immediately enable sane usage by nfs4d and itd (see wiki if unfamiliar), as well as hopefully providing useful benefits to tabled as well. -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[hail patch 1/3] chunkd: Add checksum table to on-disk format
commit f1de17a6e2b3afdbfbfa581228280b65a4a17e5f
Author: Jeff Garzik j...@garzik.org
Date: Thu Aug 5 17:47:03 2010 -0400
chunkd: Add checksum table to on-disk format, one sum per 64k of data
Signed-off-by: Jeff Garzik jgar...@redhat.com
chunkd/be-fs.c | 162 -
1 file changed, 137 insertions(+), 25 deletions(-)
diff --git a/chunkd/be-fs.c b/chunkd/be-fs.c
index 4b851a7..d714e7c 100644
--- a/chunkd/be-fs.c
+++ b/chunkd/be-fs.c
@@ -53,14 +53,23 @@ struct fs_obj {
int in_fd;
char*in_fn;
off_t sendfile_ofs;
+
+ size_t checked_bytes;
+ SHA_CTX checksum;
+ unsigned intcsum_idx;
+ void*csum_tbl;
+ size_t csum_tbl_sz;
+
+ unsigned intn_blk;
};
struct be_fs_obj_hdr {
charmagic[4];
uint32_tkey_len;
uint64_tvalue_len;
+ uint32_tn_blk;
- charreserved[16];
+ charreserved[12];
unsigned char hash[CHD_CSUM_SZ];
charowner[128];
@@ -208,6 +217,8 @@ static struct fs_obj *fs_obj_alloc(void)
obj-out_fd = -1;
obj-in_fd = -1;
+ SHA1_Init(obj-checksum);
+
return obj;
}
@@ -318,6 +329,17 @@ static bool key_valid(const void *key, size_t key_len)
return true;
}
+static unsigned int fs_blk_count(uint64_t data_len)
+{
+ uint64_t n_blk;
+
+ n_blk = data_len CHUNK_BLK_ORDER;
+ if (data_len (CHUNK_BLK_SZ - 1))
+ n_blk++;
+
+ return (unsigned int) n_blk;
+}
+
struct backend_obj *fs_obj_new(uint32_t table_id,
const void *key, size_t key_len,
uint64_t data_len,
@@ -325,6 +347,7 @@ struct backend_obj *fs_obj_new(uint32_t table_id,
{
struct fs_obj *obj;
char *fn = NULL;
+ size_t csum_bytes;
enum chunk_errcode erc = che_InternalError;
off_t skip_len;
@@ -339,6 +362,13 @@ struct backend_obj *fs_obj_new(uint32_t table_id,
return NULL;
}
+ obj-n_blk = fs_blk_count(data_len);
+ csum_bytes = obj-n_blk * CHD_CSUM_SZ;
+ obj-csum_tbl = malloc(csum_bytes);
+ if (!obj-csum_tbl)
+ goto err_out;
+ obj-csum_tbl_sz = csum_bytes;
+
/* build local fs pathname */
fn = fs_obj_pathname(table_id, key, key_len);
if (!fn)
@@ -359,7 +389,7 @@ struct backend_obj *fs_obj_new(uint32_t table_id,
obj-out_fn = fn;
/* calculate size of front-of-file metadata area */
- skip_len = sizeof(struct be_fs_obj_hdr) + key_len;
+ skip_len = sizeof(struct be_fs_obj_hdr) + key_len + csum_bytes;
/* position file pointer where object data (as in, not metadata)
* will begin
@@ -397,7 +427,10 @@ struct backend_obj *fs_obj_open(uint32_t table_id, const
char *user,
struct be_fs_obj_hdr hdr;
ssize_t rrc;
uint64_t value_len, tmp64;
+ size_t csum_bytes;
enum chunk_errcode erc = che_InternalError;
+ struct iovec iov[2];
+ size_t total_rd_len;
if (!key_valid(key, key_len)) {
*err_code = che_InvalidKey;
@@ -457,23 +490,45 @@ struct backend_obj *fs_obj_open(uint32_t table_id, const
char *user,
goto err_out;
value_len = GUINT64_FROM_LE(hdr.value_len);
+ obj-n_blk = GUINT32_FROM_LE(hdr.n_blk);
+ csum_bytes = obj-n_blk * CHD_CSUM_SZ;
/* verify file size large enough to contain value */
- tmp64 = value_len + sizeof(hdr) + key_len;
+ tmp64 = value_len + sizeof(hdr) + key_len + csum_bytes;
if (G_UNLIKELY(st.st_size tmp64)) {
applog(LOG_ERR, obj(%s) size error, too small, obj-in_fn);
goto err_out;
}
+ /* verify expected size of checksum table */
+ if (G_UNLIKELY(fs_blk_count(value_len) != obj-n_blk)) {
+ applog(LOG_ERR, obj(%s) unexpected blk count
+ (%u from val sz, %u from hdr),
+ obj-in_fn, fs_blk_count(value_len), obj-n_blk);
+ goto err_out;
+ }
+
+ obj-csum_tbl = malloc(csum_bytes);
+ if (!obj-csum_tbl)
+ goto err_out;
+ obj-csum_tbl_sz = csum_bytes;
+
obj-bo.key = malloc(key_len);
obj-bo.key_len = key_len;
if (!obj-bo.key)
goto err_out;
- /* read object variable-length header */
- rrc = read(obj-in_fd, obj-bo.key, key_len);
- if ((rrc != key_len) || (memcmp(key, obj-bo.key, key_len))) {
- applog(LOG_ERR, read hdr key obj(%s) failed: %s,
+ /* init additional header segment list */
+ iov[0].iov_base =
Re: [PATCH] don't expect inode name to be NUL-terminated (avoid read overrun)
The thing is, snprintf always NULL-terminates its output string, no
matter whether the input was NULL-terminated or not.
However, I looked at the snprintf man page again and found this
description for %s :
If no l modifier is present: The const char * argument is expected to be
a pointer to an array of character type (pointer to a string).
Characters from the array are written up to (but not including) a
terminating null byte ('\0'); if a precision is specified, no more than
the number specified are written. If a precision is given, no null byte
need be present; if the precision is not specified, or is greater than the
size of the array, the array must contain a terminating null byte.
So apparently the array must contain a terminating null byte unless
a precision is specified like %10s, etc.
So you're absolutely right, and the patch looks fine.
Colin
On Fri, Sep 10, 2010 at 5:55 AM, Jim Meyering j...@meyering.net wrote:
* server/msg.c (msg_get): Copy only name_len bytes, then NUL-terminate,
rather than using snprintf to copy up to and including nonexistent NUL.
---
valgrind exposed this. The use of snprintf would have been
correct if the inode name buffer (following the struct raw_inode)
were NUL-terminated, but it is not.
Invalid read of size 1
at 0x3502647FF7: vfprintf (vfprintf.c:1593)
by 0x350266EFB1: vsnprintf (vsnprintf.c:120)
by 0x350264F022: snprintf (snprintf.c:35)
by 0x4061D5: msg_get (msg.c:451)
by 0x407FD9: udp_rx_handle (server.c:164)
by 0x408244: udp_rx (server.c:233)
by 0x4091AA: udp_srv_event (server.c:640)
by 0x409DE6: main_loop (server.c:1026)
by 0x40A17E: main (server.c:1135)
Address 0x4d2afae is 0 bytes after a block of size 62 alloc'd
at 0x4A0515D: malloc (vg_replace_malloc.c:195)
by 0x3505F35527: __os_umalloc (os_alloc.c:68)
by 0x3505EF8F8D: __db_retcopy (db_ret.c:124)
by 0x3505EF90EB: __db_ret (db_ret.c:69)
by 0x3505ED93A7: __dbc_iget (db_cam.c:)
by 0x3505EE5CB3: __db_get (db_iface.c:779)
by 0x3505EE5FDA: __db_get_pp (db_iface.c:694)
by 0x4040F6: cldb_inode_get (cldb.c:537)
by 0x406069: msg_get (msg.c:430)
by 0x407FD9: udp_rx_handle (server.c:164)
by 0x408244: udp_rx (server.c:233)
by 0x4091AA: udp_srv_event (server.c:640)
by 0x409DE6: main_loop (server.c:1026)
by 0x40A17E: main (server.c:1135)
Here's a fix:
server/msg.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/server/msg.c b/server/msg.c
index f2dda59..8abb1e6 100644
--- a/server/msg.c
+++ b/server/msg.c
@@ -448,7 +448,8 @@ void msg_get(struct session *sess, const void *v)
name_len = le32_to_cpu(inode-ino_len);
inode_name = alloca(name_len + 1);
- snprintf(inode_name, name_len + 1, %s, (char *)(inode + 1));
+ memcpy (inode_name, inode + 1, name_len);
+ inode_name[name_len] = 0;
resp.inode_name = inode_name;
resp.data.data_len = 0;
@@ -1172,4 +1173,3 @@ err_out_noabort:
sess_sendresp_generic(sess, resp_rc);
free(h);
}
-
--
1.7.3.rc0.183.gb0497
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe hail-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Sorry! I made an error in my last email!
Hey, Sorry I made an error with the links in my last email. Here is how it should of been: Over the past few months I have taken a lot of my time to research and ask as many people as possible what the top 5 money making methods are. After weeks and weeks of different answers and even trying over 50 popular products and systems I have come to my conclusion and made the top 5 money making products online list. So here it goes: 1) The Mobile Monopoly - http://tiny.cc/ndonh 2) Auto Traffic Avalanche - http://tiny.cc/3wsuq 3) Auto Blog System - http://tiny.cc/ytf7r 4) Zero Cost Commissions - http://tiny.cc/ermdw 5) CPA Instruments - http://tiny.cc/ruh9b So there you have it. The reason I did this is because I am sick of gurus ripping people off, most of them are scammers! Be careful when buying online, only buy from trusted sources. I have checked the 5 sources above and so have thousands of other people just like you and me, and they do work. The problem with the internet is you don't know who to trust. The joke is you can check up reviews on some products and people would give it mixed reviews, some will be saying amazing products, worked for me others a scam don't buy it so sometimes you don't know who is telling the truth right? Well I let you in to a little secret here, most gurus will write reviews about their competitors saying how rubbish they are. They do this to wipe of competition. Therefore I decided I will take action and maybe get a some kind of Internet peace award for this :p so I tested and tested numerous products and came up with this list. So enjoy, it's worth checking these 5 products out, they do work especially the first two, the other three still work but are a bit over hyped. 1) The Mobile Monopoly - http://tiny.cc/ndonh 2) Auto Traffic Avalanche - http://tiny.cc/3wsuq 3) Auto Blog System - http://tiny.cc/ytf7r 4) Zero Cost Commissions - http://tiny.cc/ermdw 5) CPA Instruments - http://tiny.cc/ruh9b All the best, Sam L. Carl -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: stor_obj_test
On Wed, 07 Jul 2010 00:02:02 -0400 Jeff Garzik j...@garzik.org wrote: This function seems to be missing the meat. It retrieves then disposes of a keylist. bool stor_obj_test(struct open_chunk *cep, uint64_t key) I do not understand what kind of meat you expect stor_obj_test to have. In the filesystem backend this was a stat() that verified if the just-stored object existed and that was it. -- Pete -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [tabled patch 4/5] Support auto replicaton port
On 08/12/2010 03:22 PM, Pete Zaitcev wrote: Allow random ports for replication master to listen on. The patch is somewhat larger than expected, because before we had the MASTER file written right after locking. Now we may have it written without listening parameters, and the slaves must be ready to deal with it. Unlike the auto client port, we do not need to write any accessor files, because we already report the host and port through CLD. Listening on random ports has security implications. Signed-off-by: Pete Zaitcevzait...@redhat.com applied 1-4 of 5 -- To unsubscribe from this list: send the line unsubscribe hail-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
