On Tuesday, October 17, 2006 at 3:21 AM, Jimmy Mäkelä | Loopia AB wrote:
> We are setting up a cluster of clamd-servers which will be load-balanced
> by a TCP-loadbalancer in front of the servers.
>
> We want all clamdscan-clients to connect to the load-balancers IP and
> the connections to be evenly distributed across the cluster.
>
> Since we use stream-scanning and the clamd STREAM-command responds with
> only a port to which clamdscan then tries to connect for sending the
> data, this setup means that clamdscan will try to connect to the
> selected port on the load-balancer instead of on the selected server..
>
> We want to avoid having to use different port-ranges for the
> clamd-servers in the cluster and mapping them on the load-balancer to
> the corresponding server, so instead we have created a patch which makes
> clamd implement a new command called "NEWSTREAM" to which it responds
> with "PORT portnumber ip_of_control_connection" instead of just "PORT
> portnumber".
>
> With this patch clamdscan first connects to the load-balancer which
> selects a clamd-server to forward the connection to, then use NEWSTREAM
> for scanning, and connects directly to the chosen port on the selected
> server for sending the stream (since the stream should always go to the
> same server as the main clamd-connection). This lets us have the same
> clamd-config on all clamd-servers in the cluster (and on all clamdscan
> clients as well).
>
> The reason for using a new command instead of just changing STREAM is to
> allow old clamdscan-clients to connect to the new clamd without getting
> confused by the additional info in the PORT-response, but the patch
> could easily be modified to instead change STREAM if you consider that a
> better solution.
>
> What do you guys think of this method? Would it be possible to get the
> patch included in some future version of clamav (I'm more than happy to
> do any changes deemed neccesary for that to happen)?
>
> The patch is attached, and any comments would be greatly appreciated.
This is an interesting idea.
Clearly your suggestion works for your load balancing technology, but it may
not work for others, since the target IP addresses may not be reachable from
the client side.
Meanwhile.... A month ago, I proposed to Tomasz Kojm the addition of a
command which I called "CHUNKEDSTREAM", which would also solve your problem.
This command passes all the data to be scanned on the same TCP socket as the
clamd command and its response. The data is passed in size prefixed chunks
terminated with a zero sized chunk.
Your use case adds another reason to my list as to why it would be a good idea
to add:
I've occasionally had issues with:
1) firewall configuration protecting the clamd host,
2) issues tracing the network traffic relating to a Clamd stream request
This approach has the following advantages:
1) easier to manage a firewall configuration (no extra ports to transport
STREAM data)
2) easy to trace with network sniffing to debug since all traffic uses the
same TCP socket
3) uses fewer kernel resources (no extra socket along with listen() and
accept() calls)
4) has slightly simpler client implementation due to fewer networking calls.
The attached patch can be applied against the current CVS or clamav-0.90RC1 to
provide CHUNKEDSTREAM functionality.
- Mark Pizzolato
diff -c -r -X excluding.txt clamav-devel-20060917/clamd/scanner.c
clamav-devel-20060917.chunked-stream/clamd/scanner.c
*** clamav-devel-20060917/clamd/scanner.c Wed Sep 13 18:30:05 2006
--- clamav-devel-20060917.chunked-stream/clamd/scanner.c Wed Sep 20
11:56:57 2006
***************
*** 490,492 ****
--- 490,618 ----
return ret;
}
+
+ int scanchunkedstream(int odesc, unsigned long int *scanned, const struct
cl_node *root, const struct cl_limits *limits, int options, const struct
cfgstruct *copt)
+ {
+ int ret;
+ int tmpd, bread, bwrite, bwriting, retval = 1, timeout, btread, btdata;
+ long int size = 0, maxsize = 0;
+ const char *virname;
+ char buff[FILEBUFF];
+ struct cfgstruct *cpt;
+ char *tmpname;
+
+ timeout = cfgopt(copt, "ReadTimeout")->numarg;
+ if(timeout == 0)
+ timeout = -1;
+
+ maxsize = cfgopt(copt, "StreamMaxLength")->numarg;
+
+ if(mdprintf(odesc, "%d bytes of data will be scanned. PROCEED, send
counted chunks, end with a 0 sized chunk\n", maxsize) <= 0) {
+ logg("!ScanStream: error transmitting proceed.\n");
+ return -1;
+ }
+
+ logg("*Scanning Chunked Stream\n");
+
+ if ((tmpname = cli_gentempdesc(NULL, &tmpd)) == NULL) {
+ mdprintf(odesc, "tempfile() failed. ERROR\n");
+ logg("!ScanStream: Can't create temporary file.\n");
+ return -1;
+ }
+
+ bwriting = 1;
+
+ while (1) {
+ btread = readsock(odesc, buff, sizeof(buff)-1, '\n', timeout, 1, 0);
+ if (btread <= 0) {
+ mdprintf(odesc, "read data size failed. ERROR\n");
+ logg("!ScanStream: Can't read data size.\n");
+ close(tmpd);
+ if(!cfgopt(copt, "LeaveTemporaryFiles")->enabled)
+ unlink(tmpname);
+ free(tmpname);
+ return -1;
+ }
+ buff[btread] = '\0';
+ cli_chomp(buff);
+ btdata = atoi(buff);
+ if (0 == btdata)
+ break;
+
+ while((retval = poll_fd(odesc, timeout)) == 1) {
+ btread = sizeof(buff);
+ if (btread > btdata)
+ btread = btdata;
+ bread = recv(odesc, buff, btread, 0);
+ if(bread <= 0) {
+ mdprintf(odesc, "read stream data failed. ERROR\n");
+ logg("!ScanStream: Can't read stream data.\n");
+ close(tmpd);
+ if(!cfgopt(copt, "LeaveTemporaryFiles")->enabled)
+ unlink(tmpname);
+ free(tmpname);
+ return -1;
+ }
+ size += bread;
+ btdata -= bread;
+ if (bwriting) {
+ bwrite = bread;
+ if(maxsize < size) {
+ bwrite = maxsize-(size-bread);
+ bwriting = 0;
+ logg("^ScanStream: Size limit reached (max: %d)\n",
maxsize);
+ }
+ if(writen(tmpd, buff, bwrite) != bwrite) {
+ mdprintf(odesc, "Temporary file -> write ERROR\n");
+ logg("!ScanStream: Can't write to temporary file.\n");
+ close(tmpd);
+ if(!cfgopt(copt, "LeaveTemporaryFiles")->enabled)
+ unlink(tmpname);
+ free(tmpname);
+ return -1;
+ }
+ }
+ if (btdata == 0)
+ break;
+ }
+
+ switch(retval) {
+ case 0: /* timeout */
+ mdprintf(odesc, "read timeout ERROR\n");
+ logg("!ScanStream: read timeout.\n");
+ break;
+ case -1:
+ mdprintf(odesc, "read poll ERROR\n");
+ logg("!ScanStream: read poll failed.\n");
+ break;
+ }
+ }
+
+ if(retval == 1) {
+ lseek(tmpd, 0, SEEK_SET);
+ ret = cl_scandesc(tmpd, &virname, scanned, root, limits, options);
+ } else {
+ ret = -1;
+ }
+ close(tmpd);
+ if(!cfgopt(copt, "LeaveTemporaryFiles")->enabled)
+ unlink(tmpname);
+ free(tmpname);
+
+ if(ret == CL_VIRUS) {
+ mdprintf(odesc, "stream: %s FOUND\n", virname);
+ logg("stream: %s FOUND\n", virname);
+ virusaction("stream", virname, copt);
+ } else if(ret != CL_CLEAN) {
+ if(retval == 1) {
+ mdprintf(odesc, "stream: %s ERROR\n", cl_strerror(ret));
+ logg("stream: %s ERROR\n", cl_strerror(ret));
+ }
+ } else {
+ mdprintf(odesc, "stream: OK\n");
+ if(logok)
+ logg("stream: OK\n");
+ }
+
+ return ret;
+ }
diff -c -r -X excluding.txt clamav-devel-20060917/clamd/scanner.h
clamav-devel-20060917.chunked-stream/clamd/scanner.h
*** clamav-devel-20060917/clamd/scanner.h Mon Sep 11 18:30:04 2006
--- clamav-devel-20060917.chunked-stream/clamd/scanner.h Sun Sep 17
08:01:14 2006
***************
*** 33,36 ****
--- 33,38 ----
int checksymlink(const char *path);
+ int scanchunkedstream(int odesc, unsigned long int *scanned, const struct
cl_node *root, const struct cl_limits *limits, int options, const struct
cfgstruct *copt);
+
#endif
diff -c -r -X excluding.txt clamav-devel-20060917/clamd/session.c
clamav-devel-20060917.chunked-stream/clamd/session.c
*** clamav-devel-20060917/clamd/session.c Wed Sep 13 18:30:05 2006
--- clamav-devel-20060917.chunked-stream/clamd/session.c Sun Sep 17
08:07:51 2006
***************
*** 360,365 ****
--- 360,370 ----
}
}
+ } else if(!strncmp(buff, CMD14, strlen(CMD14))) { /* CHUNKEDSTREAM */
+ if(scanchunkedstream(desc, NULL, root, limits, options, copt) ==
CL_EMEM)
+ if(cfgopt(copt, "ExitOnOOM")->enabled)
+ return COMMAND_SHUTDOWN;
+
} else {
mdprintf(desc, "UNKNOWN COMMAND\n");
}
diff -c -r -X excluding.txt clamav-devel-20060917/clamd/session.h
clamav-devel-20060917.chunked-stream/clamd/session.h
*** clamav-devel-20060917/clamd/session.h Mon Sep 11 18:30:05 2006
--- clamav-devel-20060917.chunked-stream/clamd/session.h Sun Sep 17
08:03:11 2006
***************
*** 38,43 ****
--- 38,44 ----
#define CMD11 "SHUTDOWN"
#define CMD12 "FD"
#define CMD13 "MULTISCAN"
+ #define CMD14 "CHUNKEDSTREAM"
#include "libclamav/clamav.h"
#include "shared/cfgparser.h"
diff -c -r -X excluding.txt clamav-devel-20060917/clamdscan/client.c
clamav-devel-20060917.chunked-stream/clamdscan/client.c
*** clamav-devel-20060917/clamdscan/client.c Mon Sep 11 18:30:05 2006
--- clamav-devel-20060917.chunked-stream/clamdscan/client.c Thu Sep 21
10:40:15 2006
***************
*** 60,65 ****
--- 60,68 ----
int notremoved = 0, notmoved = 0;
static int hwaccel = 0;
+ static int sendn(int fd, void *buff, size_t count);
+ static int recvdelim(int fd, void *buff, size_t count, int delim);
+
static int dsresult(int sockd, const struct optstruct *opt)
{
int infected = 0, waserror = 0;
***************
*** 195,276 ****
static int dsstream(int sockd, const struct optstruct *opt)
{
! int wsockd, loopw = 60, bread, port, infected = 0;
! struct sockaddr_in server;
! struct sockaddr_in peer;
! int peer_size;
! char buff[4096], *pt;
!
! if(write(sockd, "STREAM", 6) <= 0) {
logg("^Can't write to the socket.\n");
return 2;
}
! memset(buff, 0, sizeof(buff));
! while(loopw) {
! read(sockd, buff, sizeof(buff));
! if((pt = strstr(buff, "PORT"))) {
! pt += 5;
! sscanf(pt, "%d", &port);
! break;
! }
! loopw--;
! }
! if(!loopw) {
! logg("^Daemon not ready for stream scanning.\n");
return -1;
}
! /* connect to clamd */
!
! if((wsockd = socket(SOCKET_INET, SOCK_STREAM, 0)) < 0) {
! perror("socket()");
! logg("^Can't create the socket.\n");
! return -1;
! }
!
! server.sin_family = AF_INET;
! server.sin_port = htons(port);
!
! peer_size = sizeof(peer);
! if(getpeername(sockd, (struct sockaddr *) &peer, &peer_size) < 0) {
! perror("getpeername()");
! logg("^Can't get socket peer name.\n");
! return -1;
! }
!
! switch (peer.sin_family) {
! case AF_UNIX:
! server.sin_addr.s_addr = inet_addr("127.0.0.1");
break;
! case AF_INET:
! server.sin_addr.s_addr = peer.sin_addr.s_addr;
! break;
! default:
! mprintf("^Unexpected socket type: %d.\n", peer.sin_family);
return -1;
! }
!
! if(connect(wsockd, (struct sockaddr *) &server, sizeof(struct
sockaddr_in)) < 0) {
! close(wsockd);
! perror("connect()");
! logg("^Can't connect to clamd [port: %d].\n", port);
! return -1;
! }
!
! while((bread = read(0, buff, sizeof(buff))) > 0) {
! if(write(wsockd, buff, bread) <= 0) {
logg("^Can't write to the socket.\n");
- close(wsockd);
return -1;
}
}
- close(wsockd);
memset(buff, 0, sizeof(buff));
! while((bread = read(sockd, buff, sizeof(buff))) > 0) {
logg("%s", buff);
if(strstr(buff, "FOUND\n")) {
infected++;
--- 198,244 ----
static int dsstream(int sockd, const struct optstruct *opt)
{
! int bread, readsize, datatosend, infected = 0;
! char buff[32768], sizebuf[64];
! if(sendn(sockd, "nCHUNKEDSTREAM\n", 15) <= 0) {
logg("^Can't write to the socket.\n");
return 2;
}
! buff[sizeof(buff)-1] = '\0';
! bread = recvdelim(sockd, buff, sizeof(buff)-1, '\n');
! if((bread <= 0) || (!strstr(buff, "PROCEED"))) {
! logg("^Daemon not ready for stream scanning: %s", buff);
return -1;
}
+ datatosend = atoi(buff);
! while(datatosend > 0) {
! readsize = (sizeof(buff) > datatosend) ? datatosend : sizeof(buff);
! bread = read(0, buff, sizeof(buff));
! if(bread <= 0)
break;
! snprintf(sizebuf, sizeof(sizebuf), "%d\n", bread);
! if(sendn(sockd, sizebuf, strlen(sizebuf)) <= 0) {
! logg("^Can't write to the socket.\n");
return -1;
! }
! if(sendn(sockd, buff, bread) <= 0) {
logg("^Can't write to the socket.\n");
return -1;
}
+ datatosend -= bread;
+ }
+ if(sendn(sockd, "0\n", 2) <= 0) {
+ logg("^Can't write to the socket.\n");
+ return -1;
}
memset(buff, 0, sizeof(buff));
! bread = recvdelim(sockd, buff, sizeof(buff)-1, '\n');
! if(bread > 0) {
logg("%s", buff);
if(strstr(buff, "FOUND\n")) {
infected++;
***************
*** 279,285 ****
logg("%s", buff);
return -1;
}
- memset(buff, 0, sizeof(buff));
}
return infected;
--- 247,252 ----
***************
*** 616,618 ****
--- 583,647 ----
free(movefilename);
}
+
+ /* Function: sendn
+ Try hard to send the specified number of bytes
+ */
+ static int sendn(int fd, void *buff, size_t count)
+ {
+ int retval;
+ unsigned int todo;
+ unsigned char *current;
+
+ todo = count;
+ current = (unsigned char *) buff;
+
+ do {
+ retval = send(fd, current, todo, 0);
+ if (retval < 0) {
+ if (errno == EINTR) {
+ continue;
+ }
+ return -1;
+ }
+ todo -= retval;
+ current += retval;
+ } while (todo > 0);
+
+ return count;
+ }
+
+ /* Function: recvdelim
+ recieve data up until a delimiter is reached (or the recieve buffer is
full)
+ */
+ static int recvdelim(int fd, void *buff, size_t count, int delim)
+ {
+ char *buf = (char *)buff;
+ int bread, breturn = 0;
+ char *loc;
+
+ while(1) {
+ bread = recv(fd, buf, count, MSG_PEEK);
+ if(bread < 0)
+ return bread;
+ if(bread == 0)
+ return breturn;
+ loc = memchr(buf, delim, bread);
+ if(loc) {
+ if(count > (loc-buf+1)) /* nul terminate the response if we have
room */
+ loc[1] = '\0';
+ return recv(fd, buf, loc-buf+1, 0);
+ }
+ if(bread == count)
+ return recv(fd, buf, count, 0);
+ bread = recv(fd, buf, bread, 0);
+ if(bread < 0)
+ return bread;
+ if(bread == 0)
+ return breturn;
+ buf += bread;
+ count -= bread;
+ breturn += bread;
+ }
+ }
+
diff -c -r -X excluding.txt clamav-devel-20060917/docs/man/clamd.8.in
clamav-devel-20060917.chunked-stream/docs/man/clamd.8.in
*** clamav-devel-20060917/docs/man/clamd.8.in Mon Sep 11 18:30:05 2006
--- clamav-devel-20060917.chunked-stream/docs/man/clamd.8.in Wed Sep 20
11:57:43 2006
***************
*** 41,46 ****
--- 41,49 ----
.TP
\fBSTREAM\fR
Scan stream \- on this command clamd will return "PORT number" and you can
connect to that port and send a data to scan.
+ .TP
+ \fBCHUNKEDSTREAM\fR
+ Scan chunkedstream \- on this command clamd will return "nnn bytes of data
will be scanned. PROCEED, send counted chunks, end with a 0 sized chunk". Data
is presented as a series of counted chunks on the same socket as the command.
A chunk is presented as an ascii decimal number terminated by a newline
followed by the indicated number of octets of data to scan. A chunk size of 0
indicates the end of the data to scan. Clamd indicates as nnn the maximum
number of bytes that will be processed. A client should send no more than nnn
bytes of data to scan since sending more will return the same scan result as
scanning nnn bytes.
.SH "OPTIONS"
.LP
_______________________________________________
http://lurker.clamav.net/list/clamav-devel.html
