Re: bridge(4) and if_input()
On 28/05/15(Thu) 11:23, Martin Pieuchot wrote:
Let's use if_input() rather than ether_input_mbuf(). This simplifies
the bpf(4) bits and removes some duplicate rcvif assignments.
Once ether_input_mbuf() is no more my plan is to start merging various
ifp counters and put an interface index in the packet header.
ok?
Anybody?
Index: net/if_bridge.c
===
RCS file: /cvs/src/sys/net/if_bridge.c,v
retrieving revision 1.239
diff -u -p -r1.239 if_bridge.c
--- net/if_bridge.c 18 May 2015 11:43:57 - 1.239
+++ net/if_bridge.c 22 May 2015 13:39:16 -
@@ -1364,6 +1364,7 @@ bridge_dispatch(struct bridge_iflist *if
struct bridge_iflist *srcifl;
struct ether_header *eh;
struct arpcom *ac;
+ struct mbuf_list ml = MBUF_LIST_INITIALIZER();
struct mbuf *mc;
int s;
@@ -1411,21 +1412,14 @@ bridge_dispatch(struct bridge_iflist *if
schednetisr(NETISR_BRIDGE);
if (ifp-if_type == IFT_GIF) {
TAILQ_FOREACH(ifl, sc-sc_iflist, next) {
- if (ifl-ifp-if_type == IFT_ETHER)
- break;
- }
- if (ifl != NULL) {
- m-m_pkthdr.rcvif = ifl-ifp;
- m-m_pkthdr.ph_rtableid = ifl-ifp-if_rdomain;
-#if NBPFILTER 0
- if (ifl-ifp-if_bpf)
- bpf_mtap_ether(ifl-ifp-if_bpf, m,
- BPF_DIRECTION_IN);
-#endif
+ if (ifl-ifp-if_type != IFT_ETHER)
+ continue;
+
m-m_flags |= M_PROTO1;
- ether_input_mbuf(ifl-ifp, m);
+ ml_enqueue(ml, m);
+ if_input(ifl-ifp, ml);
ifl-ifp-if_ipackets++;
- m = NULL;
+ return (NULL);
}
}
return (m);
@@ -1462,24 +1456,14 @@ bridge_dispatch(struct bridge_iflist *if
return (NULL);
}
- /* Make sure the real incoming interface
- * is aware */
-#if NBPFILTER 0
- if (ifl-ifp-if_bpf)
- bpf_mtap_ether(ifl-ifp-if_bpf, m,
- BPF_DIRECTION_IN);
-#endif
- /* Count for the interface we are going to */
- ifl-ifp-if_ipackets++;
-
/* Count for the bridge */
sc-sc_if.if_ipackets++;
sc-sc_if.if_ibytes += m-m_pkthdr.len;
- m-m_pkthdr.rcvif = ifl-ifp;
- m-m_pkthdr.ph_rtableid = ifl-ifp-if_rdomain;
m-m_flags |= M_PROTO1;
- ether_input_mbuf(ifl-ifp, m);
+ ml_enqueue(ml, m);
+ if_input(ifl-ifp, ml);
+ ifl-ifp-if_ipackets++;
return (NULL);
}
if (bcmp(ac-ac_enaddr, eh-ether_shost, ETHER_ADDR_LEN) == 0
@@ -1615,8 +1599,10 @@ void
bridge_localbroadcast(struct bridge_softc *sc, struct ifnet *ifp,
struct ether_header *eh, struct mbuf *m)
{
+ struct mbuf_list ml = MBUF_LIST_INITIALIZER();
struct mbuf *m1;
u_int16_t etype;
+ int s;
/*
* quick optimisation, don't send packets up the stack if no
@@ -1638,18 +1624,11 @@ bridge_localbroadcast(struct bridge_soft
sc-sc_if.if_oerrors++;
return;
}
- /* fixup header a bit */
- m1-m_pkthdr.rcvif = ifp;
- m1-m_pkthdr.ph_rtableid = ifp-if_rdomain;
m1-m_flags |= M_PROTO1;
-
-#if NBPFILTER 0
- if (ifp-if_bpf)
- bpf_mtap(ifp-if_bpf, m1,
- BPF_DIRECTION_IN);
-#endif
-
- ether_input_mbuf(ifp, m1);
+ ml_enqueue(ml, m1);
+ s = splnet();
+ if_input(ifp, ml);
+ splx(s);
ifp-if_ipackets++;
}
Re: [Patch] httpd - don't leak fcgi file descriptors
On Sun, 31 May 2015 19:25:22 -0400 Todd Mortimer t...@opennet.ca wrote: Hi tech@, Hi Joerg, Thanks for getting back to me. I cloned the server and upgraded it to the 31 May snapshot, did the sysmerge and upgraded the packages to the snapshot versions. The behaviour is still there. It actually appears to be somewhat more pronounced, and php-fpm hits max_children more quickly than it does under 5.7-stable. The same patch prevents the php-fpm processes from going idle on netio, and I have reproduced it against -current below. It also seems that httpd on -current has more parallel connections open to php-fpm at once compared to the same setup on 5.7-stable. I have been following this thread since the initial report. I'm also running owncloud (+ roundcube for mail) on an OpenBSD -current amd64 server (snapshot from May 20). I have been hitting the exact same issue but initially I accounted it for just not tweaking the settings. My server is really low on traffic (only two users rarely concurrent). I can confirm that after getting to max_children limit the server starts returning error 500 on each request (both on roundcube owncloud) until php-fpm or httpd is restarted. I did increase pm.max_children setting from 5 - 10 I still hit the max_children limit on roughly the same interval. I will try to find the time to upgrade the server to a newer snapshot and test the patch provided by Todd. # grep max_children /var/log/php-fpm.log [29-Mar-2015 17:25:31] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [29-Mar-2015 18:57:16] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [29-Mar-2015 19:22:12] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [30-Mar-2015 03:22:25] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [31-Mar-2015 21:47:08] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [31-Mar-2015 22:47:09] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [02-Apr-2015 11:39:22] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [02-Apr-2015 13:39:26] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [03-Apr-2015 15:44:39] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [05-Apr-2015 17:07:15] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [05-Apr-2015 17:12:54] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [08-Apr-2015 14:00:57] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [21-May-2015 13:32:37] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [25-May-2015 17:15:54] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [25-May-2015 17:42:39] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it [30-May-2015 15:27:55] WARNING: [pool www] server reached pm.max_children setting (10), consider raising it # After increasing the limit I also hit: [30-May-2015 15:27:51] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 6 total children [30-May-2015 15:27:52] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 7 total children [30-May-2015 15:27:53] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 8 total children [30-May-2015 15:27:54] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 9 total children I agree that my patch is more of a workaround, and it would be better to track down how it is that the client is being passed to server_fcgi with an open socket. I was going this way when I started looking at the source, but then I saw that clt-clt_srvevb and clt-clt_srvbev get the same treatment (free if not null, then reassign) at the same spot in server_fcgi(), and I figured if it was good enough for clt_srvevb and clt_srvbev, why not for clt_fd? I would be happy to look into a proper solution if that would be better. Thanks! Todd On May 31, 2015, at 2:23, Joerg Jung m...@umaxx.net wrote: Hi, Am 20.05.2015 um 02:06 schrieb Todd Mortimer t...@opennet.ca: The attached patch fixes a problem I’ve been having with httpd + php_fpm + owncloud on 5.7. The patch is against 5.7-release. Can you try with recent snapshot, and see if issue still occurs, please? Development happens in -current. After several days running
Re: ftp(1) rewrite
On Thu, May 21, 2015 at 11:16:09PM -0400, Ted Unangst wrote:
Sunil Nimmagadda wrote:
Hi,
The idea is to start with the subset of ftp(1) functionality needed
by pkg_add(1):
ftp [-o output] url ...
i.e., should be able to download files over HTTP(S) and FTP.
This implementation works as FETCH_CMD for pkg_add(1) over HTTP(S).
FTP is not yet done, but thinking of implementing just PASV and
RETR commands and drop command interpreter compeletely.
The SMALL variant drops HTTPS and FTP support.
Comments?
screw ftp. just make a new util http, that just does http.
Hi,
Here is a work in progress version 2. Changes from previous diff...
1. Drop ftp provision and rename the utility as http(1).
2. HTTP redirect handling.
3. Proxy support.
4. Resume interrupted file transfer.
5. Basic and Proxy Authorization.
Todo: progressmeter, cookies, RFC1738 URL encoding/decoding and
maybe some more.
Comments?
Index: Makefile
===
RCS file: Makefile
diff -N Makefile
--- /dev/null 1 Jan 1970 00:00:00 -
+++ Makefile1 Jun 2015 09:02:59 -
@@ -0,0 +1,19 @@
+# Define SMALL to disable https support
+.if defined(SMALL)
+CFLAGS+= -DSMALL
+.endif
+
+PROG= http
+MAN= http.1
+DEBUG= -g -Wall -O0
+SRCS= main.c http.c
+LDADD+=-lutil
+DPADD+=${LIBUTIL}
+
+.ifndef SMALL
+SRCS+= https.c
+LDADD+=-ltls -lssl -lcrypto
+DPADD+=${LIBTLS} ${LIBSSL} ${LIBCRYPTO}
+.endif
+
+.include bsd.prog.mk
Index: http.1
===
RCS file: http.1
diff -N http.1
--- /dev/null 1 Jan 1970 00:00:00 -
+++ http.1 1 Jun 2015 09:02:59 -
@@ -0,0 +1,94 @@
+.\ Copyright (c) 1985, 1989, 1990, 1993
+.\The Regents of the University of California. All rights reserved.
+.\
+.\ Redistribution and use in source and binary forms, with or without
+.\ modification, are permitted provided that the following conditions
+.\ are met:
+.\ 1. Redistributions of source code must retain the above copyright
+.\notice, this list of conditions and the following disclaimer.
+.\ 2. Redistributions in binary form must reproduce the above copyright
+.\notice, this list of conditions and the following disclaimer in the
+.\documentation and/or other materials provided with the distribution.
+.\ 3. Neither the name of the University nor the names of its contributors
+.\may be used to endorse or promote products derived from this software
+.\without specific prior written permission.
+.\
+.\ THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
+.\ ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\ ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
+.\ FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\ DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\ OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\ HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+.\ LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+.\ OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\ SUCH DAMAGE.
+.\
+.\@(#)ftp.1 8.3 (Berkeley) 10/9/94
+.\
+.\ Copyright (c) 2015 Sunil Nimmagadda su...@nimmagadda.net
+.\
+.\ Permission to use, copy, modify, and distribute this software for any
+.\ purpose with or without fee is hereby granted, provided that the above
+.\ copyright notice and this permission notice appear in all copies.
+.\
+.\ THE SOFTWARE IS PROVIDED AS IS AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\ WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\ MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\ ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\ WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\ ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\ OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+.\
+.Dd $Mdocdate: June 1 2015 $
+.Dt HTTP 1
+.Os
+.Sh NAME
+.Nm http
+.Nd HTTP file transfer program
+.Sh SYNOPSIS
+.Nm
+.Op Fl C
+.Op Fl o Ar output
+.Op Fl P Ar port
+.Op Fl U Ar useragent
+.Op Fl V
+.Ar ...
+.Sh DESCRIPTION
+.Nm
+allows a user to transfer files from a remote HTTP server.
+.Pp
+The options are as follows:
+.Bl -tag -width Ds
+.It Fl C
+Continue a previously interrupted file transfer.
+.Nm
+will continue transferring from an offset equal to the length of file.
+.Pp
+Resuming HTTP(s) transfers are only supported
+if the remote server supports the
+.Dq Range
+header.
+.It Fl o Ar output
+When fetching a single file or URL, save the contents in
+.Ar output.
+.It Fl P Ar port
Re: arpresolve: unresolved and rt_expire == 0
On 31/05/15(Sun) 22:05, mxb wrote: Hello, any ideas regarding subject ? Yes various. Any idea about how to reproduce it? What you're seeing is a side-effect of r1.151 of netinet/if_ether.c. I see this in ‘dmesg’. Also all local (on machine itself) arp entries are incomplete: Yes because we stopped copying the Ethernet address into the gateway field of RTF_LOCAL routes. But since RTF_LOCAL routes are bound to lo0 I don't know how your kernel ends up calling arpresolve() and trigger this code path.
Thinkpad E550
Hi, I got a Lenovo Thinkpad E550. I installed -current and everything looks fine except a few things: - it reboots on resume after suspend - graphics card is Intel HD 5500, it's supposed to be recognized bu I fall back into VESA driver (unable to get VGA screen working, nor watching videos for instance) - display a console and then go back to X lead to freeze (I still can reboot cleanly via ssh). $ dmesg|grep -i 'not configured' intagp at vga1 not configured Intel 9 Series MEI rev 0x03 at pci0 dev 22 function 0 not configured vendor ATI, unknown product 0x6604 (class display subclass miscellaneous, rev 0x00) at pci3 dev 0 function 0 not configured Intel 9 Series Thermal rev 0x03 at pci0 dev 31 function 6 not configured I don't know if some stuff are in progress or if I can help to get thinkgs better (particularly about the graphics part) Thanks anyway to all the good job already done complete dmesg: OpenBSD 5.7-current (GENERIC.MP) #0: Thu May 28 19:02:46 GMT 2015 root@boutxy:/usr/src/sys/arch/amd64/compile/GENERIC.MP RTC BIOS diagnostic error 80clock_battery real mem = 16884772864 (16102MB) avail mem = 16369209344 (15610MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xacdfd000 (61 entries) bios0: vendor LENOVO version J5ET43WW (1.14 ) date 02/10/2015 bios0: LENOVO 20DF004UFR acpi0 at bios0: rev 2 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP SLIC ASF! HPET ECDT APIC MCFG SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT PCCT SSDT UEFI MSDM BATB FPDT UEFI acpi0: wakeup devices LID_(S3) IGBE(S4) EXP3(S4) XHCI(S3) EHC1(S3) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 14318179 Hz acpiec0 at acpi0 acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz, 2295.08 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4, IBE cpu1 at mainbus0: apid 1 (application processor) cpu1: Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz, 2294.69 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 1, core 0, package 0 cpu2 at mainbus0: apid 2 (application processor) cpu2: Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz, 2294.69 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP cpu2: 256KB 64b/line 8-way L2 cache cpu2: smt 0, core 1, package 0 cpu3 at mainbus0: apid 3 (application processor) cpu3: Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz, 2294.69 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP cpu3: 256KB 64b/line 8-way L2 cache cpu3: smt 1, core 1, package 0 ioapic0 at mainbus0: apid 2 pa 0xfec0, version 20, 40 pins acpimcfg0 at acpi0 addr 0xf800, bus 0-63 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 5 (PEG_) acpiprt2 at acpi0: bus 2 (EXP1) acpiprt3 at acpi0: bus 4 (EXP3) acpiprt4 at acpi0: bus 6 (EXP6) acpicpu0 at acpi0: C3, C1, PSS acpicpu1 at acpi0: C3, C1, PSS acpicpu2 at acpi0: C3, C1, PSS acpicpu3 at acpi0: C3, C1, PSS acpipwrres0 at acpi0: PUBS, resource for XHCI, EHC1 acpipwrres1 at acpi0: AMD3, resource for PEG_ acpipwrres2 at acpi0: AMD2, resource for PEG_ acpitz0 at acpi0: critical temperature is 127 degC acpibtn0 at acpi0: LID_ acpibat0 at acpi0: BAT0 model LNV-45N1 serial 2079 type LION oem SANYO acpiac0 at acpi0: AC unit online acpithinkpad0 at acpi0 cpu0: Enhanced SpeedStep 2295 MHz: speeds: 2401, 2400, 2300, 2100, 2000, 1900, 1700, 1600, 1400, 1300, 1200, 1000, 900, 800, 600, 500 MHz pci0 at mainbus0 bus 0 pchb0 at pci0
Re: ftp(1) rewrite
Hi,
quick comments inline.
On 6/1/15, Sunil Nimmagadda su...@nimmagadda.net wrote:
On Thu, May 21, 2015 at 11:16:09PM -0400, Ted Unangst wrote:
Sunil Nimmagadda wrote:
Hi,
The idea is to start with the subset of ftp(1) functionality needed
by pkg_add(1):
ftp [-o output] url ...
i.e., should be able to download files over HTTP(S) and FTP.
This implementation works as FETCH_CMD for pkg_add(1) over HTTP(S).
FTP is not yet done, but thinking of implementing just PASV and
RETR commands and drop command interpreter compeletely.
The SMALL variant drops HTTPS and FTP support.
Comments?
screw ftp. just make a new util http, that just does http.
Hi,
Here is a work in progress version 2. Changes from previous diff...
1. Drop ftp provision and rename the utility as http(1).
2. HTTP redirect handling.
3. Proxy support.
4. Resume interrupted file transfer.
5. Basic and Proxy Authorization.
Todo: progressmeter, cookies, RFC1738 URL encoding/decoding and
maybe some more.
Comments?
[...]
Index: http.c
===
RCS file: http.c
diff -N http.c
--- /dev/null 1 Jan 1970 00:00:00 -
+++ http.c1 Jun 2015 09:02:59 -
[...]
+int
+proxy_connect(int s, struct url *url, struct url *proxy)
+{
+ FILE*fp;
+ char hostname[HOST_NAME_MAX+1], *buf;
+ const char *proxy_auth = NULL;
+ size_t len;
+ int code;
+
+ if ((fp = fdopen(s, r+)) == NULL)
+ err(1, http_connect: fdopen);
+
+ if (gethostname(hostname, sizeof(hostname)) != 0)
+ hostname[0] = '\0';
Pretty sure the Host: header should indicate the remote host,
not local.
+ if (proxy-user[0] || proxy-pass[0])
+ proxy_auth = base64_encode(proxy-user, proxy-pass);
+
+ fprintf(fp,
+ CONNECT %s:%s HTTP/1.0\r\n
+ Host: %s\r\n
+ User-Agent: %s\r\n
+ %s%s
+ \r\n,
+ url-host,
+ (url-port[0]) ? url-port : 80,
+ hostname,
+ ua,
+ (proxy_auth) ? Proxy-Authorization: Basic : ,
+ (proxy_auth) ? proxy_auth : );
+
+ fflush(fp);
+ if ((buf = http_getline(fp, len)) == NULL)
+ return (-1);
+
+ if ((code = http_response_code(buf)) == -1) {
+ free(buf);
+ return (-1);
+ }
+
+ free(buf);
+ if (code != 200)
+ errx(1, Error retrieving file: %s, errstr(code));
+
+ return (0);
+}
+
+const char *
+base64_encode(const char *user, const char *pass)
+{
+ static char basic_auth[BUFSIZ];
+ char*creds, b64_creds[BUFSIZ];
+ int ret;
+
+ if ((ret = asprintf(creds, %s:%s, user, pass)) == -1)
+ errx(1, base64_encode: asprintf failed);
+
+ if (b64_ntop(creds, ret, b64_creds, sizeof(b64_creds)) == -1)
+ errx(1, error in base64 encoding);
+
+ free(creds);
+ ret = snprintf(basic_auth, sizeof(basic_auth), %s\r\n, b64_creds);
+ if (ret == -1 || ret sizeof(basic_auth))
+ errx(1, base64_encode: basic_auth overflow);
+
+ return (basic_auth);
+}
+
+int
+http_get(int s, struct url *url, const char *fn, int resume,
+struct headers *hdrs)
+{
+ struct stat sb;
+ char hostname[HOST_NAME_MAX+1], range[BUFSIZ];
+ const char *basic_auth = NULL;
+ FILE*fin;
+ char*buf;
+ size_t len;
+ int fd, flags, res = -1;
+
+ if ((fin = fdopen(s, r+)) == NULL)
+ err(1, http_get: fdopen);
+
+ if (gethostname(hostname, sizeof(hostname)) != 0)
+ hostname[0] = '\0';
Same here.
But afaik HTTP/1.0 doesn't require that header.
--patrick
+ if (stat(fn, sb) == -1)
+ resume = 0;
+ else {
+ snprintf(range, sizeof(range),
+ Range: bytes=%lld-\r\n, sb.st_size);
+ }
+
+ if (url-user[0] || url-pass[0])
+ basic_auth = base64_encode(url-user, url-pass);
+
+ fprintf(fin,
+ GET %s HTTP/1.0\r\n
+ Host: %s\r\n
+ User-Agent: %s\r\n
+ %s
+ %s%s
+ \r\n,
+ (url-path[0]) ? url-path : /,
+ hostname,
+ ua,
+ (resume) ? range : ,
+ (basic_auth) ? Authorization: Basic : ,
+ (basic_auth) ? basic_auth : );
[...]
Re: ftp(1) rewrite
On 2015/06/01 10:20, patrick keshishian wrote: On 6/1/15, Sunil Nimmagadda su...@nimmagadda.net wrote: On Thu, May 21, 2015 at 11:16:09PM -0400, Ted Unangst wrote: screw ftp. just make a new util http, that just does http. Sorry, it's not good enough to replace ftp(1) for system use without ftp. Like it or not, ports fetches need FTP and can't really rely on installing something for ports to do that. + if (gethostname(hostname, sizeof(hostname)) != 0) + hostname[0] = '\0'; Pretty sure the Host: header should indicate the remote host, not local. Yep. And watch out to use the correct hostname for proxies. But afaik HTTP/1.0 doesn't require that header. Host: is pretty much required, one particular thing for OpenBSD is that fw_update mirrors need it.
Re: ftp(1) rewrite
On Mon, Jun 01, 2015 at 08:06:38PM +0100, Stuart Henderson wrote: On 2015/06/01 10:20, patrick keshishian wrote: On 6/1/15, Sunil Nimmagadda su...@nimmagadda.net wrote: On Thu, May 21, 2015 at 11:16:09PM -0400, Ted Unangst wrote: screw ftp. just make a new util http, that just does http. Sorry, it's not good enough to replace ftp(1) for system use without ftp. Like it or not, ports fetches need FTP and can't really rely on installing something for ports to do that. Yes, but splitting these protocols is good, right? IMHO, having a clean and simple http(1) and a (more) clean ftp(1) with the http bits removed makes sense to me. + if (gethostname(hostname, sizeof(hostname)) != 0) + hostname[0] = '\0'; Pretty sure the Host: header should indicate the remote host, not local. Yep. And watch out to use the correct hostname for proxies. But afaik HTTP/1.0 doesn't require that header. Host: is pretty much required, one particular thing for OpenBSD is that fw_update mirrors need it.
Re: ftp(1) rewrite
Sorry, it's not good enough to replace ftp(1) for system use without ftp. Like it or not, ports fetches need FTP and can't really rely on installing something for ports to do that. Yes, but splitting these protocols is good, right? IMHO, having a clean and simple http(1) and a (more) clean ftp(1) with the http bits removed makes sense to me. Maybe. But there is little point in showing a chunk of code which does not get close to satisfying the preconditions which have been described.
Re: arpresolve: unresolved and rt_expire == 0
I had an old kernel from 'Apr 28' laying around, as well as from 'May 15' . Both are OK. On 2015-06-01 23:04, mxb wrote: Well, this is a vmware setup, thus I have vmx(4). I just made a clean 'cvs co -P src' from anoncvs.eu.openbsd.org and still have the same result. //mxb On 2015-06-01 10:55, Martin Pieuchot wrote: Any idea about how to reproduce it?
Re: arpresolve: unresolved and rt_expire == 0
Well, this is a vmware setup, thus I have vmx(4). I just made a clean 'cvs co -P src' from anoncvs.eu.openbsd.org and still have the same result. //mxb On 2015-06-01 10:55, Martin Pieuchot wrote: Any idea about how to reproduce it?
Re: ftp(1) rewrite
Stuart Henderson wrote: On 2015/06/01 10:20, patrick keshishian wrote: On 6/1/15, Sunil Nimmagadda su...@nimmagadda.net wrote: On Thu, May 21, 2015 at 11:16:09PM -0400, Ted Unangst wrote: screw ftp. just make a new util http, that just does http. Sorry, it's not good enough to replace ftp(1) for system use without ftp. Like it or not, ports fetches need FTP and can't really rely on installing something for ports to do that. It's not obvious to me why this would be the case. The ports tree already depends on various external programs to extract xz, cab, etc.
Re: ftp(1) rewrite
Ted Unangst said: Stuart Henderson wrote: Sorry, it's not good enough to replace ftp(1) for system use without ftp. Like it or not, ports fetches need FTP and can't really rely on installing something for ports to do that. It's not obvious to me why this would be the case. The ports tree already depends on various external programs to extract xz, cab, etc. One important thing is missing from this discussion: what exactly is wrong with ftp(1)? -- Dmitrij D. Czarkoff
Re: ftp(1) rewrite
On 2015/06/02 00:09, Dmitrij D. Czarkoff wrote: Ted Unangst said: Stuart Henderson wrote: Sorry, it's not good enough to replace ftp(1) for system use without ftp. Like it or not, ports fetches need FTP and can't really rely on installing something for ports to do that. It's not obvious to me why this would be the case. The ports tree already depends on various external programs to extract xz, cab, etc. Fetching files happens really early, before dependencies are processed. And the program used to fetch is meant to be user-overridable. One important thing is missing from this discussion: what exactly is wrong with ftp(1)? The code isto put it nicelyrather dated ;-)
Re: ftp(1) rewrite
One important thing is missing from this discussion: what exactly is
wrong with ftp(1)?
The code isto put it nicelyrather dated ;-)
First off, let's show the result of a grep.
cmds.c: (void)setjmp(jabort);
cmds.c: (void)setjmp(jabort);
cmds.c: (void)setjmp(jabort);
cmds.c: if (setjmp(abortprox)) {
fetch.c:if (setjmp(httpabort)) {
fetch.c:if (setjmp(httpabort)) {
fetch.c:if (setjmp(toplevel)) {
ftp.c: if (setjmp(sendabort)) {
ftp.c: if (setjmp(sendabort))
ftp.c: if (setjmp(recvabort)) {
ftp.c: if (setjmp(recvabort))
ftp.c: if (setjmp(ptabort))
ftp.c: if (cout == NULL || setjmp(forceabort)) {
ftp_var.h:#include setjmp.h
main.c: if (setjmp(toplevel))
main.c: top = setjmp(toplevel) == 0;
small.c:(void)setjmp(jabort);
I believe it is one of the last programs in the tree using such a
fragile interface. Yes, setjmp can be used safely. All those
global variables could be used right, too
Re: ftp(1) rewrite
On 06/01/15 18:29, Stuart Henderson wrote: On 2015/06/02 00:09, Dmitrij D. Czarkoff wrote: Ted Unangst said: Stuart Henderson wrote: Sorry, it's not good enough to replace ftp(1) for system use without ftp. Like it or not, ports fetches need FTP and can't really rely on installing something for ports to do that. It's not obvious to me why this would be the case. The ports tree already depends on various external programs to extract xz, cab, etc. Fetching files happens really early, before dependencies are processed. And the program used to fetch is meant to be user-overridable. One important thing is missing from this discussion: what exactly is wrong with ftp(1)? The code isto put it nicelyrather dated ;-) Ever run into the fact that ftp(1) can get stuck during certain operations if you hit Ctrl-C and you have to kill -9 the process? That is absolutely ridiculous. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
