On Thu, May 21, 2015 at 11:16:09PM -0400, Ted Unangst wrote:
> Sunil Nimmagadda wrote:
> > Hi,
> >
> > The idea is to start with the subset of ftp(1) functionality needed
> > by pkg_add(1):
> >
> > ftp [-o output] url ...
> >
> > i.e., should be able to download files over HTTP(S) and FTP.
> >
> > This implementation works as FETCH_CMD for pkg_add(1) over HTTP(S).
> >
> > FTP is not yet done, but thinking of implementing just PASV and
> > RETR commands and drop command interpreter compeletely.
> >
> > The SMALL variant drops HTTPS and FTP support.
> >
> > Comments?
>
> screw ftp. just make a new util http, that just does http.
Hi,
Here is a work in progress version 2. Changes from previous diff...
1. Drop ftp provision and rename the utility as http(1).
2. HTTP redirect handling.
3. Proxy support.
4. Resume interrupted file transfer.
5. Basic and Proxy Authorization.
Todo: progressmeter, cookies, RFC1738 URL encoding/decoding and
maybe some more.
Comments?
Index: Makefile
===================================================================
RCS file: Makefile
diff -N Makefile
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ Makefile 1 Jun 2015 09:02:59 -0000
@@ -0,0 +1,19 @@
+# Define SMALL to disable https support
+.if defined(SMALL)
+CFLAGS+= -DSMALL
+.endif
+
+PROG= http
+MAN= http.1
+DEBUG= -g -Wall -O0
+SRCS= main.c http.c
+LDADD+= -lutil
+DPADD+= ${LIBUTIL}
+
+.ifndef SMALL
+SRCS+= https.c
+LDADD+= -ltls -lssl -lcrypto
+DPADD+= ${LIBTLS} ${LIBSSL} ${LIBCRYPTO}
+.endif
+
+.include <bsd.prog.mk>
Index: http.1
===================================================================
RCS file: http.1
diff -N http.1
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ http.1 1 Jun 2015 09:02:59 -0000
@@ -0,0 +1,94 @@
+.\" Copyright (c) 1985, 1989, 1990, 1993
+.\" The Regents of the University of California. All rights reserved.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\" notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\" notice, this list of conditions and the following disclaimer in the
+.\" documentation and/or other materials provided with the distribution.
+.\" 3. Neither the name of the University nor the names of its contributors
+.\" may be used to endorse or promote products derived from this software
+.\" without specific prior written permission.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
+.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
+.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\" SUCH DAMAGE.
+.\"
+.\" @(#)ftp.1 8.3 (Berkeley) 10/9/94
+.\"
+.\" Copyright (c) 2015 Sunil Nimmagadda <su...@nimmagadda.net>
+.\"
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
+.\"
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\"
+.Dd $Mdocdate: June 1 2015 $
+.Dt HTTP 1
+.Os
+.Sh NAME
+.Nm http
+.Nd HTTP file transfer program
+.Sh SYNOPSIS
+.Nm
+.Op Fl C
+.Op Fl o Ar output
+.Op Fl P Ar port
+.Op Fl U Ar useragent
+.Op Fl V
+.Ar ...
+.Sh DESCRIPTION
+.Nm
+allows a user to transfer files from a remote HTTP server.
+.Pp
+The options are as follows:
+.Bl -tag -width Ds
+.It Fl C
+Continue a previously interrupted file transfer.
+.Nm
+will continue transferring from an offset equal to the length of file.
+.Pp
+Resuming HTTP(s) transfers are only supported
+if the remote server supports the
+.Dq Range
+header.
+.It Fl o Ar output
+When fetching a single file or URL, save the contents in
+.Ar output.
+.It Fl P Ar port
+Sets the port number to
+.Ar port .
+.It Fl U Ar useragent
+Set
+.Ar useragent
+as the User-Agent for HTTP(S) URL requests.
+If not specified, the default User-Agent is
+.Dq OpenBSD http .
+.It Fl V
+Disable verbose mode.
+.El
+.Sh ENVIRONMENT
+.Nm
+utilizes the following environment variables:
+.Bl -tag -width Ds
+.It Ev http_proxy
+URL of HTTP_PROXY to use when making HTTP(S) URL requests.
+.El
Index: http.c
===================================================================
RCS file: http.c
diff -N http.c
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ http.c 1 Jun 2015 09:02:59 -0000
@@ -0,0 +1,349 @@
+/*
+ * Copyright (c) 2015 Sunil Nimmagadda <su...@nimmagadda.net>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <sys/stat.h>
+
+#include <ctype.h>
+#include <err.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <netdb.h>
+#include <limits.h>
+#include <resolv.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <util.h>
+
+#include "http.h"
+
+int http_get(int, struct url *, const char *, int,
+ struct headers *);
+char *http_getline(FILE *, size_t *);
+int http_parse_headers(FILE *, struct headers *);
+void http_200(FILE *, int);
+int proxy_connect(int, struct url *, struct url *);
+const char *base64_encode(const char *, const char *);
+
+struct proto proto_http = {
+ http_connect,
+ http_get
+};
+
+extern const char *ua;
+
+int
+http_connect(struct url *url, struct url *proxy)
+{
+ struct addrinfo hints, *res, *res0;
+ char hbuf[NI_MAXHOST];
+ const char *cause = NULL, *host, *port;
+ int error, s = -1, save_errno;
+
+ host = (proxy) ? proxy->host : url->host;
+ port = (proxy) ? proxy->port : url->port;
+
+ if (port[0] == '\0')
+ port = "80";
+
+ memset(&hints, 0, sizeof(hints));
+ hints.ai_family = PF_UNSPEC;
+ hints.ai_socktype = SOCK_STREAM;
+ error = getaddrinfo(host, port, &hints, &res0);
+ if (error) {
+ warnx("%s: %s", host, gai_strerror(error));
+ return (-1);
+ }
+
+ for (res = res0; res; res = res->ai_next) {
+ if (getnameinfo(res->ai_addr, res->ai_addrlen, hbuf,
+ sizeof(hbuf), NULL, 0, NI_NUMERICHOST) != 0)
+ (void)strlcpy(hbuf, "(unknown)", sizeof(hbuf));
+
+ log_info("Trying %s...\n", hbuf);
+ s = socket(res->ai_family, res->ai_socktype, res->ai_protocol);
+ if (s == -1) {
+ cause = "socket";
+ continue;
+ }
+
+ if (connect(s, res->ai_addr, res->ai_addrlen) == -1) {
+ cause = "connect";
+ save_errno = errno;
+ close(s);
+ errno = save_errno;
+ s = -1;
+ continue;
+ }
+
+ break;
+ }
+
+ freeaddrinfo(res0);
+ if (s == -1) {
+ warn("%s", cause);
+ return (s);
+ }
+
+ if (proxy && proxy_connect(s, url, proxy) == -1)
+ return (-1);
+
+ return (s);
+}
+
+int
+proxy_connect(int s, struct url *url, struct url *proxy)
+{
+ FILE *fp;
+ char hostname[HOST_NAME_MAX+1], *buf;
+ const char *proxy_auth = NULL;
+ size_t len;
+ int code;
+
+ if ((fp = fdopen(s, "r+")) == NULL)
+ err(1, "http_connect: fdopen");
+
+ if (gethostname(hostname, sizeof(hostname)) != 0)
+ hostname[0] = '\0';
+
+ if (proxy->user[0] || proxy->pass[0])
+ proxy_auth = base64_encode(proxy->user, proxy->pass);
+
+ fprintf(fp,
+ "CONNECT %s:%s HTTP/1.0\r\n"
+ "Host: %s\r\n"
+ "User-Agent: %s\r\n"
+ "%s%s"
+ "\r\n",
+ url->host,
+ (url->port[0]) ? url->port : "80",
+ hostname,
+ ua,
+ (proxy_auth) ? "Proxy-Authorization: Basic " : "",
+ (proxy_auth) ? proxy_auth : "");
+
+ fflush(fp);
+ if ((buf = http_getline(fp, &len)) == NULL)
+ return (-1);
+
+ if ((code = http_response_code(buf)) == -1) {
+ free(buf);
+ return (-1);
+ }
+
+ free(buf);
+ if (code != 200)
+ errx(1, "Error retrieving file: %s", errstr(code));
+
+ return (0);
+}
+
+const char *
+base64_encode(const char *user, const char *pass)
+{
+ static char basic_auth[BUFSIZ];
+ char *creds, b64_creds[BUFSIZ];
+ int ret;
+
+ if ((ret = asprintf(&creds, "%s:%s", user, pass)) == -1)
+ errx(1, "base64_encode: asprintf failed");
+
+ if (b64_ntop(creds, ret, b64_creds, sizeof(b64_creds)) == -1)
+ errx(1, "error in base64 encoding");
+
+ free(creds);
+ ret = snprintf(basic_auth, sizeof(basic_auth), "%s\r\n", b64_creds);
+ if (ret == -1 || ret > sizeof(basic_auth))
+ errx(1, "base64_encode: basic_auth overflow");
+
+ return (basic_auth);
+}
+
+int
+http_get(int s, struct url *url, const char *fn, int resume,
+ struct headers *hdrs)
+{
+ struct stat sb;
+ char hostname[HOST_NAME_MAX+1], range[BUFSIZ];
+ const char *basic_auth = NULL;
+ FILE *fin;
+ char *buf;
+ size_t len;
+ int fd, flags, res = -1;
+
+ if ((fin = fdopen(s, "r+")) == NULL)
+ err(1, "http_get: fdopen");
+
+ if (gethostname(hostname, sizeof(hostname)) != 0)
+ hostname[0] = '\0';
+
+ if (stat(fn, &sb) == -1)
+ resume = 0;
+ else {
+ snprintf(range, sizeof(range),
+ "Range: bytes=%lld-\r\n", sb.st_size);
+ }
+
+ if (url->user[0] || url->pass[0])
+ basic_auth = base64_encode(url->user, url->pass);
+
+ fprintf(fin,
+ "GET %s HTTP/1.0\r\n"
+ "Host: %s\r\n"
+ "User-Agent: %s\r\n"
+ "%s"
+ "%s%s"
+ "\r\n",
+ (url->path[0]) ? url->path : "/",
+ hostname,
+ ua,
+ (resume) ? range : "",
+ (basic_auth) ? "Authorization: Basic " : "",
+ (basic_auth) ? basic_auth : "");
+
+ fflush(fin);
+ if ((buf = http_getline(fin, &len)) == NULL)
+ goto cleanup;
+
+ if ((res = http_response_code(buf)) == -1) {
+ free(buf);
+ goto cleanup;
+ }
+
+ free(buf);
+ if (http_parse_headers(fin, hdrs) != 0) {
+ res = -1;
+ goto cleanup;
+ }
+
+ flags = O_CREAT | O_WRONLY;
+ switch (res) {
+ case 206: /* Partial content */
+ flags |= O_APPEND;
+ /* FALLTHROUGH */
+ case 200: /* OK */
+ if (strcmp(fn, "-") == 0)
+ fd = STDOUT_FILENO;
+ else if ((fd = open(fn, flags, 0666)) == -1)
+ err(1, "open: %s", fn);
+
+ http_200(fin, fd);
+ break;
+ }
+
+cleanup:
+ fclose(fin);
+ return (res);
+}
+
+void
+http_200(FILE *fin, int out)
+{
+ size_t r, wlen;
+ ssize_t i;
+ char *cp;
+ static char *buf;
+
+ if (buf == NULL) {
+ buf = malloc(TMPBUF_LEN); /* allocate once */
+ if (buf == NULL)
+ err(1, "malloc");
+ }
+
+ /* XXX Content-Length or till EOF? */
+ while ((r = fread(buf, sizeof(char), TMPBUF_LEN, fin)) > 0) {
+ for (cp = buf, wlen = r; wlen > 0; wlen -= i, cp += i) {
+ if ((i = write(out, cp, wlen)) == -1) {
+ warn("http_200: write");
+ break;
+ } else if (i == 0)
+ break;
+ }
+ }
+}
+
+int
+http_response_code(char *buf)
+{
+ const char *errstr;
+ char *e;
+ int res;
+
+ if ((buf = strchr(buf, ' ')) == NULL) {
+ warnx("Malformed response code");
+ return (-1);
+ }
+
+ buf++;
+ if ((e = strchr(buf, ' ')) == NULL) {
+ warnx("Malformed response code");
+ return (-1);
+ }
+
+ *e = '\0';
+ res = strtonum(buf, 200, 503, &errstr);
+ if (errstr) {
+ warn("http_response_code: strtonum");
+ return (-1);
+ }
+
+ return (res);
+}
+
+int
+http_parse_headers(FILE *fin, struct headers *hdrs)
+{
+ char *buf;
+ size_t len;
+ int ret;
+
+ while ((buf = http_getline(fin, &len))) {
+ if (len == 0)
+ break; /* end of headers */
+
+ if (header_insert(hdrs, buf) != 0) {
+ ret = -1;
+ goto exit;
+ }
+ }
+
+ ret = 0;
+exit:
+ free(buf);
+ return (ret);
+}
+
+char *
+http_getline(FILE *fp, size_t *len)
+{
+ char *buf;
+ size_t ln;
+
+ if ((buf = fparseln(fp, &ln, NULL, "\0\0\0", 0)) == NULL) {
+ warn("http_getline");
+ return (NULL);
+ }
+
+ if (buf[ln - 1] == '\r')
+ buf[--ln] = '\0';
+
+ *len = ln;
+ return (buf);
+}
+
Index: http.h
===================================================================
RCS file: http.h
diff -N http.h
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ http.h 1 Jun 2015 09:02:59 -0000
@@ -0,0 +1,47 @@
+/*
+ * Copyright (c) 2015 Sunil Nimmagadda <su...@nimmagadda.net>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+
+#define TMPBUF_LEN 131072
+
+struct headers {
+ char location[BUFSIZ]; /* Redirect Location */
+ off_t c_len; /* Content-Length */
+};
+
+/* http://<user>:<pass>@<host>:<port>/<path> */
+struct url {
+ char host[HOST_NAME_MAX+1];
+ char port[16];
+ char user[LOGIN_NAME_MAX];
+ char pass[256];
+ char path[BUFSIZ];
+ int proto;
+};
+
+struct proto {
+ int (*connect)(struct url *, struct url *);
+ int (*get)(int, struct url *, const char *, int, struct headers *);
+};
+
+/* main.c */
+int header_insert(struct headers *, const char *);
+void log_info(const char *, ...);
+const char *errstr(int);
+
+/* http.c */
+int http_connect(struct url *, struct url *);
+int http_response_code(char *);
+
Index: https.c
===================================================================
RCS file: https.c
diff -N https.c
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ https.c 1 Jun 2015 09:02:59 -0000
@@ -0,0 +1,294 @@
+/*-
+ * Copyright (c) 1997 The NetBSD Foundation, Inc.
+ * All rights reserved.
+ *
+ * This code is derived from software contributed to The NetBSD Foundation
+ * by Jason Thorpe and Luke Mewburn.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
+ * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+/*
+ * Copyright (c) 2015 Sunil Nimmagadda <su...@nimmagadda.net>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+
+#include <sys/types.h>
+#include <sys/stat.h>
+
+#include <err.h>
+#include <fcntl.h>
+#include <limits.h>
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <tls.h>
+#include <unistd.h>
+
+#include "http.h"
+
+struct tls *ctx;
+struct tls_config *tls_config = NULL;
+
+int https_connect(struct url *, struct url *);
+int https_get(int, struct url *, const char *, int, struct headers *);
+int https_vprintf(struct tls *, const char *, ...);
+char *https_getline(struct tls *, size_t *);
+int https_parse_headers(struct tls *, struct headers *);
+void https_200(struct tls *, int);
+
+struct proto proto_https = {
+ https_connect,
+ https_get
+};
+
+extern const char *ua;
+
+int
+https_connect(struct url *url, struct url *proxy)
+{
+ int s;
+
+ if (tls_init() != 0) {
+ warnx("tls init failed");
+ return (-1);
+ }
+
+ if (tls_config == NULL) {
+ tls_config = tls_config_new();
+ if (tls_config == NULL)
+ errx(1, "tls config failed");
+
+ tls_config_set_protocols(tls_config, TLS_PROTOCOLS_ALL);
+ if (tls_config_set_ciphers(tls_config, "compat") != 0)
+ errx(1, "tls set ciphers failed");
+ }
+
+ if ((ctx = tls_client()) == NULL) {
+ warnx("failed to create tls client");
+ return (-1);
+ }
+
+ if (tls_configure(ctx, tls_config) != 0) {
+ warnx("tls_configure: %s", tls_error(ctx));
+ return (-1);
+ }
+
+ if (url->port[0] == '\0')
+ (void)strlcpy(url->port, "443", sizeof(url->port));
+
+ if ((s = http_connect(url, proxy)) == -1)
+ return (-1);
+
+ if (tls_connect_socket(ctx, s, url->host) != 0) {
+ warnx("tls_connect: %s", tls_error(ctx));
+ return (-1);
+ }
+
+ return (s);
+}
+
+int
+https_get(int s, struct url *url, const char *fn, int resume,
+ struct headers *hdrs)
+{
+ struct stat sb;
+ char hostname[HOST_NAME_MAX+1], range[BUFSIZ];
+ char *buf;
+ size_t len;
+ int fd, flags, res = -1;
+
+ if (gethostname(hostname, sizeof(hostname)) != 0)
+ hostname[0] = '\0';
+
+ if (stat(fn, &sb) == -1)
+ resume = 0;
+ else {
+ snprintf(range, sizeof(range),
+ "Range: bytes=%lld-\r\n", sb.st_size);
+ }
+
+ https_vprintf(ctx,
+ "GET https://%s%s%s%s HTTP/1.0\r\n"
+ "Host: %s\r\n"
+ "User-Agent: %s\r\n"
+ "%s"
+ "\r\n",
+ url->host,
+ (url->port[0]) ? ":" : "",
+ (url->port[0]) ? url->port : "",
+ url->path,
+ hostname,
+ ua,
+ (resume) ? range : "");
+
+ if ((buf = https_getline(ctx, &len)) == NULL)
+ goto cleanup;
+
+ if ((res = http_response_code(buf)) == -1)
+ goto cleanup;
+
+ if (https_parse_headers(ctx, hdrs) != 0) {
+ res = -1;
+ goto cleanup;
+ }
+
+ flags = O_CREAT | O_WRONLY;
+ switch (res) {
+ case 206: /* Partial content */
+ flags |= O_APPEND;
+ /* FALLTHROUGH */
+ case 200: /* OK */
+ if (strcmp(fn, "-") == 0)
+ fd = STDOUT_FILENO;
+ else if ((fd = open(fn, flags, 0666)) == -1)
+ err(1, "open: %s", fn);
+
+ https_200(ctx, fd);
+ break;
+ }
+
+cleanup:
+ free(buf);
+ tls_close(ctx);
+ tls_free(ctx);
+ return (res);
+}
+
+void
+https_200(struct tls *ctx, int out)
+{
+ size_t r, wlen;
+ ssize_t i;
+ char *cp;
+ static char *buf;
+
+ if (buf == NULL) {
+ buf = malloc(TMPBUF_LEN); /* allocate once */
+ if (buf == NULL)
+ err(1, "malloc");
+ }
+
+ /* XXX Content-Length or till EOF? */
+ while (tls_read(ctx, buf, TMPBUF_LEN, &r) == 0 && r > 0) {
+ for (cp = buf, wlen = r; wlen > 0; wlen -= i, cp += i) {
+ if ((i = write(out, cp, wlen)) == -1) {
+ warn("https_200: write");
+ break;
+ } else if (i == 0)
+ break;
+ }
+ }
+}
+
+int
+https_vprintf(struct tls *tls, const char *fmt, ...)
+{
+ va_list ap;
+ char *string;
+ size_t nw;
+ int ret;
+
+ va_start(ap, fmt);
+ if ((ret = vasprintf(&string, fmt, ap)) == -1)
+ return ret;
+
+ va_end(ap);
+ ret = tls_write(tls, string, ret, &nw);
+ free(string);
+ return ret;
+}
+
+char *
+https_getline(struct tls *tls, size_t *lenp)
+{
+ size_t i, len, nr;
+ char *buf, *q, c;
+ int ret;
+
+ len = 128;
+ if ((buf = malloc(len)) == NULL)
+ errx(1, "Can't allocate memory for transfer buffer");
+
+ for (i = 0; ; i++) {
+ if (i >= len - 1) {
+ if ((q = reallocarray(buf, len, 2)) == NULL)
+ errx(1, "Can't expand transfer buffer");
+
+ buf = q;
+ len *= 2;
+ }
+again:
+ ret = tls_read(tls, &c, 1, &nr);
+ if (ret == TLS_READ_AGAIN)
+ goto again;
+
+ if (ret != 0)
+ errx(1, "TLS read error: %u", ret);
+
+ buf[i] = c;
+ if (c == '\n')
+ break;
+ }
+
+ buf[i] = '\0';
+ if (buf[i - 1] == '\r')
+ buf[--i] = '\0';
+
+ *lenp = i;
+ return (buf);
+}
+
+int
+https_parse_headers(struct tls *ctx, struct headers *hdrs)
+{
+ char *buf;
+ size_t len;
+ int ret;
+
+ while ((buf = https_getline(ctx, &len))) {
+ if (len == 0)
+ break; /* end of headers */
+
+ if (header_insert(hdrs, buf) != 0) {
+ ret = -1;
+ goto exit;
+ }
+ }
+
+ ret = 0;
+exit:
+ free(buf);
+ return (ret);
+}
Index: main.c
===================================================================
RCS file: main.c
diff -N main.c
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ main.c 1 Jun 2015 09:02:59 -0000
@@ -0,0 +1,383 @@
+/*
+ * Copyright (c) 2015 Sunil Nimmagadda <su...@nimmagadda.net>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+
+#include <ctype.h>
+#include <err.h>
+#include <fcntl.h>
+#include <libgen.h>
+#include <limits.h>
+#include <stdarg.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "http.h"
+
+#define HTTP 1
+#define HTTPS 2
+#define UNDEF 3
+
+#define USER_AGENT "OpenBSD http"
+#define MAX_RETRIES 10
+
+struct proto *lookup(int);
+int proto_type(const char *);
+int url_parse(const char *, struct url *);
+void log_request(struct url *, struct url *, int);
+void usage(void);
+
+const char *ua = USER_AGENT;
+int verbose = 1;
+
+int
+main(int argc, char *argv[])
+{
+ struct url url, proxy, *pproxy = NULL;
+ struct headers res_hdrs;
+ struct proto *proto;
+ char *proxy_str;
+ const char *fn, *output = NULL, *port = NULL;
+ int ch, code, i, p, resume = 0, retries = 0, s = -1;
+
+ while ((ch = getopt(argc, argv, "Co:P:U:V")) != -1) {
+ switch (ch) {
+ case 'C':
+ resume = 1;
+ break;
+ case 'o':
+ output = optarg;
+ break;
+ case 'P':
+ port = optarg;
+ break;
+ case 'U':
+ ua = optarg;
+ break;
+ case 'V':
+ verbose = 0;
+ break;
+ default:
+ usage();
+ }
+ }
+
+ argc -= optind;
+ argv += optind;
+ if (argc == 0)
+ usage();
+
+ if ((proxy_str = getenv("http_proxy")) != NULL && *proxy_str == '\0')
+ proxy_str = NULL;
+
+ if (proxy_str) {
+ if (url_parse(proxy_str, &proxy) != 0)
+ errx(1, "Malformed proxy URL: %s", proxy_str);
+ else
+ pproxy = &proxy;
+ }
+
+ memset(&res_hdrs, 0, sizeof(res_hdrs));
+ for (i = 0; i < argc; i++) {
+retry:
+ if ((p = proto_type(argv[i])) == UNDEF)
+ errx(1, "Unknown protocol: %s\n", argv[i]);
+
+ fn = (output) ? output : basename(argv[i]);
+ if (url_parse(argv[i], &url) != 0)
+ errx(1, "Malformed URL: %s", argv[i]);
+
+ if (port)
+ (void)strlcpy(url.port, port, sizeof(url.port));
+
+ proto = lookup(p);
+ if ((s = proto->connect(&url, pproxy)) == -1)
+ return (-1);
+
+ log_request(&url, pproxy, p);
+ code = proto->get(s, &url, fn, resume, &res_hdrs);
+ switch (code) {
+ case 200: /* OK */
+ case 206: /* Partial Content */
+ break;
+ case 301: /* Move Permanently */
+ case 302: /* Found */
+ case 303: /* See Other */
+ case 307: /* Temporary Redirect */
+ if (++retries > MAX_RETRIES)
+ errx(1, "Too many redirections requested");
+
+ argv[i] = res_hdrs.location;
+ log_info("Redirected to %s\n", argv[i]);
+ goto retry;
+ case 416: /* Range not Satisfiable */
+ /* Ideally should check Content-Range header */
+ warnx("File is already fully retrieved");
+ break;
+ default:
+ errx(1, "Error retrieving file: %s", errstr(code));
+ }
+
+ retries = 0;
+ }
+
+ return (0);
+}
+
+struct proto *
+lookup(int p)
+{
+ extern struct proto proto_http;
+#ifndef SMALL
+ extern struct proto proto_https;
+#endif
+
+ switch (p) {
+ case HTTP:
+ return (&proto_http);
+#ifndef SMALL
+ case HTTPS:
+ return (&proto_https);
+#endif
+ }
+
+ errx(1, "invalid proto %d\n", p);
+ return (NULL);
+}
+
+int
+proto_type(const char *url)
+{
+ while (isblank((unsigned char)*url))
+ url++;
+
+ if (strncasecmp(url, "http://";, 7) == 0)
+ return (HTTP);
+#ifndef SMALL
+ else if (strncasecmp(url, "https://";, 8) == 0)
+ return (HTTPS);
+#endif
+ else
+ return (UNDEF);
+}
+
+/* parse http://<user>:<pass>@<host>:<port>/<path> */
+int
+url_parse(const char *url_str, struct url *url)
+{
+ char *curl, *e, *s, *t;
+ int ret = -1;
+
+ memset(url, 0, sizeof(struct url));
+ if ((curl = strdup(url_str)) == NULL)
+ err(1, "url_parse: strdup");
+
+ s = strstr(curl, "//");
+ s += 2;
+ /* extract user and password */
+ if ((e = strchr(s, '@'))) {
+ *e++ = '\0';
+ if ((t = strchr(s, ':'))) {
+ *t++ = '\0';
+ if (strlcpy(url->user, s,
+ sizeof(url->user)) >= sizeof(url->user)) {
+ warnx("url user overflow");
+ goto cleanup;
+ }
+ }
+
+ if (t) {
+ if (strlcpy(url->pass, t,
+ sizeof(url->pass)) >= sizeof(url->pass)) {
+ warnx("url pass overflow");
+ goto cleanup;
+ }
+ }
+
+ if ((s = e) == NULL)
+ goto cleanup;
+ }
+
+ /* extract url path */
+ if ((e = strchr(s, '/'))) {
+ if (strlcpy(url->path, e,
+ sizeof(url->path)) >= sizeof(url->path)) {
+ warnx("url path overflow");
+ goto cleanup;
+ }
+ *e = '\0';
+ }
+
+ /* extract url port */
+ if ((t = strchr(s, ':'))) {
+ *t++ = '\0';
+ if (strlcpy(url->port, t,
+ sizeof(url->port)) >= sizeof(url->port)) {
+ warnx("url port overflow");
+ goto cleanup;
+ }
+ }
+
+ /* finally extract host */
+ if (strlcpy(url->host, s, sizeof(url->host)) >= sizeof(url->host)) {
+ warnx("url host overflow");
+ goto cleanup;
+ }
+
+ ret = 0;
+cleanup:
+ free(curl);
+ return (ret);
+}
+
+int
+header_insert(struct headers *hdrs, const char *buf)
+{
+ const char *errstr;
+ size_t sz;
+
+ if (strncasecmp(buf, "Content-Length:", 15) == 0) {
+ buf = strchr(buf, ' ');
+ buf++;
+ hdrs->c_len = strtonum(buf, 0, LLONG_MAX, &errstr);
+ if (errstr) {
+ warn("insert_header: strtonum");
+ return (-1);
+ }
+ }
+
+ if (strncasecmp(buf, "Location:", 8) == 0) {
+ buf = strchr(buf, ' ');
+ buf++;
+ sz = strlcpy(hdrs->location, buf, sizeof(hdrs->location));
+ if (sz >= sizeof(hdrs->location)) {
+ warnx("header Location overflow");
+ return (-1);
+ }
+ }
+
+ return (0);
+}
+
+void
+log_info(const char *fmt, ...)
+{
+ va_list ap;
+
+ if (verbose == 0)
+ return;
+
+ va_start(ap, fmt);
+ vfprintf(stderr, fmt, ap);
+ va_end(ap);
+}
+
+void
+log_request(struct url *url, struct url *proxy, int p)
+{
+ if (proxy)
+ log_info("Requesting %s://%s%s%s%s%s%s%s"
+ " (via %s://%s%s%s%s%s%s)\n",
+ (p == HTTP) ? "http" : "https",
+ (url->user[0]) ? url->user : "",
+ (url->pass[0]) ? ":*****" : "",
+ (url->user[0] || url->pass[0]) ? "@" : "",
+ url->host,
+ (url->port[0]) ? ":" : "",
+ (url->port[0]) ? url->port : "",
+ url->path,
+
+ /* via proxy part */
+ "http",
+ (proxy->user[0]) ? proxy->user : "",
+ (proxy->pass[0]) ? ":*****" : "",
+ (proxy->user[0] || proxy->pass[0]) ? "@" : "",
+ proxy->host,
+ (proxy->port[0]) ? ":" : "",
+ (proxy->port[0]) ? proxy->port : "");
+ else
+ log_info("Requesting %s://%s%s%s%s%s%s%s\n",
+ (p == HTTP) ? "http" : "https",
+ (url->user[0]) ? url->user : "",
+ (url->pass[0]) ? ":*****" : "",
+ (url->user[0] || url->pass[0]) ? "@" : "",
+ url->host,
+ (url->port[0]) ? ":" : "",
+ (url->port[0]) ? url->port : "",
+ url->path);
+}
+
+void
+usage(void)
+{
+ fprintf(stderr, "usage: %s [-C] [-o output] "
+ "[-P port] [-U useragent] url ...\n", getprogname());
+ exit(0);
+}
+
+const char *
+errstr(int code)
+{
+ static char buf[32];
+
+ switch (code) {
+ case 400:
+ return ("400 Bad Request");
+ case 402:
+ return ("402 Payment Required");
+ case 403:
+ return ("403 Forbidden");
+ case 404:
+ return ("404 Not Found");
+ case 405:
+ return ("405 Method Not Allowed");
+ case 406:
+ return ("406 Not Acceptable");
+ case 408:
+ return ("408 Request Timeout");
+ case 409:
+ return ("409 Conflict");
+ case 410:
+ return ("410 Gone");
+ case 411:
+ return ("411 Length Required");
+ case 413:
+ return ("413 Payload Too Long");
+ case 414:
+ return ("414 URI Too Long");
+ case 415:
+ return ("415 Unsupported Media Type");
+ case 417:
+ return ("417 Expectation Failed");
+ case 426:
+ return ("426 Upgrade Required");
+ case 500:
+ return ("500 Internal Server Error");
+ case 501:
+ return ("501 Not Implemented");
+ case 502:
+ return ("502 Bad Gateway");
+ case 503:
+ return ("503 Service Unavailable");
+ case 504:
+ return ("504 Gateway Timeout");
+ case 505:
+ return ("505 HTTP Version Not Supported");
+ default:
+ (void)snprintf(buf, sizeof(buf), "%d ???", code);
+ return (buf);
+ }
+}
