Re: libevent evutil.h

[Theo de Raadt]

 libevent has compatibilty wrappers in evutil.  Our tree does not
 use them anymore, but they are still part of libevent's interface.
 
 I don't want to include them automatically, so I suggest to remove
 evutil.h from event.h.  A version bump should not be necessary as
 the library itself does not change.
 
 Does my idea make sense?

Yes, it seems right.

 Is a full ports build needed with this diff?

I think so, just in case someone pulled it in without knowing the API
bounds.  Then those problems can be cleaned up, eventually make it
upstream, and everyone benefits.

Re: libevent evutil.h

[Nicholas Marriott]

I'd like to see evutil.h go so I'm happy with this idea but yes you will
need to make sure it doesn't break ports, there are still quite a few
ports that depend on the base libevent.


On Thu, Oct 30, 2014 at 02:43:32AM +0100, Alexander Bluhm wrote:
 Hi,
 
 libevent has compatibilty wrappers in evutil.  Our tree does not
 use them anymore, but they are still part of libevent's interface.
 
 I don't want to include them automatically, so I suggest to remove
 evutil.h from event.h.  A version bump should not be necessary as
 the library itself does not change.
 
 Does my idea make sense?
 Is a full ports build needed with this diff?
 
 bluhm
 
 Index: lib/libevent/event.h
 ===
 RCS file: /data/mirror/openbsd/cvs/src/lib/libevent/event.h,v
 retrieving revision 1.27
 diff -u -p -r1.27 event.h
 --- lib/libevent/event.h  8 Oct 2014 20:14:19 -   1.27
 +++ lib/libevent/event.h  29 Oct 2014 23:42:45 -
 @@ -168,8 +168,11 @@ extern C {
  #include stdarg.h
  #include stdint.h
  
 -/* For int types. */
 -#include evutil.h
 +#define ev_uint64_t uint64_t
 +#define ev_int64_t int64_t
 +#define ev_uint32_t uint32_t
 +#define ev_uint16_t uint16_t
 +#define ev_uint8_t uint8_t
  
  #define EVLIST_TIMEOUT   0x01
  #define EVLIST_INSERTED  0x02
 

Re: libevent evutil.h

[Landry Breuil]

On Thu, Oct 30, 2014 at 08:14:35AM +, Nicholas Marriott wrote:
 I'd like to see evutil.h go so I'm happy with this idea but yes you will
 need to make sure it doesn't break ports, there are still quite a few
 ports that depend on the base libevent.

I'll run a bulk build with that. Apply diff to event.h, remove
/usr/include/evutil.h, and that's enough, right ?

Landry

 On Thu, Oct 30, 2014 at 02:43:32AM +0100, Alexander Bluhm wrote:
  Hi,
  
  libevent has compatibilty wrappers in evutil.  Our tree does not
  use them anymore, but they are still part of libevent's interface.
  
  I don't want to include them automatically, so I suggest to remove
  evutil.h from event.h.  A version bump should not be necessary as
  the library itself does not change.
  
  Does my idea make sense?
  Is a full ports build needed with this diff?
  
  bluhm
  
  Index: lib/libevent/event.h
  ===
  RCS file: /data/mirror/openbsd/cvs/src/lib/libevent/event.h,v
  retrieving revision 1.27
  diff -u -p -r1.27 event.h
  --- lib/libevent/event.h8 Oct 2014 20:14:19 -   1.27
  +++ lib/libevent/event.h29 Oct 2014 23:42:45 -
  @@ -168,8 +168,11 @@ extern C {
   #include stdarg.h
   #include stdint.h
   
  -/* For int types. */
  -#include evutil.h
  +#define ev_uint64_t uint64_t
  +#define ev_int64_t int64_t
  +#define ev_uint32_t uint32_t
  +#define ev_uint16_t uint16_t
  +#define ev_uint8_t uint8_t
   
   #define EVLIST_TIMEOUT 0x01
   #define EVLIST_INSERTED0x02
  
 

Re: libevent evutil.h

[Landry Breuil]

On Thu, Oct 30, 2014 at 08:28:56AM +, Nicholas Marriott wrote:
 No I think we are keeping the evutil.h file for now, the idea is just to
 stop event.h including it.

OKay, bulk started with just the event.h diff.

Landry

 On Thu, Oct 30, 2014 at 09:20:49AM +0100, Landry Breuil wrote:
  On Thu, Oct 30, 2014 at 08:14:35AM +, Nicholas Marriott wrote:
   I'd like to see evutil.h go so I'm happy with this idea but yes you will
   need to make sure it doesn't break ports, there are still quite a few
   ports that depend on the base libevent.
  
  I'll run a bulk build with that. Apply diff to event.h, remove
  /usr/include/evutil.h, and that's enough, right ?
  
  Landry
  
   On Thu, Oct 30, 2014 at 02:43:32AM +0100, Alexander Bluhm wrote:
Hi,

libevent has compatibilty wrappers in evutil.  Our tree does not
use them anymore, but they are still part of libevent's interface.

I don't want to include them automatically, so I suggest to remove
evutil.h from event.h.  A version bump should not be necessary as
the library itself does not change.

Does my idea make sense?
Is a full ports build needed with this diff?

bluhm

Index: lib/libevent/event.h
===
RCS file: /data/mirror/openbsd/cvs/src/lib/libevent/event.h,v
retrieving revision 1.27
diff -u -p -r1.27 event.h
--- lib/libevent/event.h8 Oct 2014 20:14:19 -   1.27
+++ lib/libevent/event.h29 Oct 2014 23:42:45 -
@@ -168,8 +168,11 @@ extern C {
 #include stdarg.h
 #include stdint.h
 
-/* For int types. */
-#include evutil.h
+#define ev_uint64_t uint64_t
+#define ev_int64_t int64_t
+#define ev_uint32_t uint32_t
+#define ev_uint16_t uint16_t
+#define ev_uint8_t uint8_t
 
 #define EVLIST_TIMEOUT 0x01
 #define EVLIST_INSERTED0x02

   
  
 

Re: libevent evutil.h

[Nicholas Marriott]

Thanks Landry



On Thu, Oct 30, 2014 at 09:40:14AM +0100, Landry Breuil wrote:
 On Thu, Oct 30, 2014 at 08:28:56AM +, Nicholas Marriott wrote:
  No I think we are keeping the evutil.h file for now, the idea is just to
  stop event.h including it.
 
 OKay, bulk started with just the event.h diff.
 
 Landry
 
  On Thu, Oct 30, 2014 at 09:20:49AM +0100, Landry Breuil wrote:
   On Thu, Oct 30, 2014 at 08:14:35AM +, Nicholas Marriott wrote:
I'd like to see evutil.h go so I'm happy with this idea but yes you will
need to make sure it doesn't break ports, there are still quite a few
ports that depend on the base libevent.
   
   I'll run a bulk build with that. Apply diff to event.h, remove
   /usr/include/evutil.h, and that's enough, right ?
   
   Landry
   
On Thu, Oct 30, 2014 at 02:43:32AM +0100, Alexander Bluhm wrote:
 Hi,
 
 libevent has compatibilty wrappers in evutil.  Our tree does not
 use them anymore, but they are still part of libevent's interface.
 
 I don't want to include them automatically, so I suggest to remove
 evutil.h from event.h.  A version bump should not be necessary as
 the library itself does not change.
 
 Does my idea make sense?
 Is a full ports build needed with this diff?
 
 bluhm
 
 Index: lib/libevent/event.h
 ===
 RCS file: /data/mirror/openbsd/cvs/src/lib/libevent/event.h,v
 retrieving revision 1.27
 diff -u -p -r1.27 event.h
 --- lib/libevent/event.h  8 Oct 2014 20:14:19 -   1.27
 +++ lib/libevent/event.h  29 Oct 2014 23:42:45 -
 @@ -168,8 +168,11 @@ extern C {
  #include stdarg.h
  #include stdint.h
  
 -/* For int types. */
 -#include evutil.h
 +#define ev_uint64_t uint64_t
 +#define ev_int64_t int64_t
 +#define ev_uint32_t uint32_t
 +#define ev_uint16_t uint16_t
 +#define ev_uint8_t uint8_t
  
  #define EVLIST_TIMEOUT   0x01
  #define EVLIST_INSERTED  0x02
 

   
  
 

ftp: use constant instead of value

[Jan Klemkow]

Hi,

This diff just swaps -2 with the actual constant to increase
readability of the ftp source code.

bye,
Jan

Index: fetch.c
===
RCS file: /cvs/src/usr.bin/ftp/fetch.c,v
retrieving revision 1.132
diff -u -p -r1.132 fetch.c
--- fetch.c 8 Oct 2014 04:01:10 -   1.132
+++ fetch.c 30 Oct 2014 13:53:35 -
@@ -1519,7 +1519,7 @@ SSL_readline(struct ressl *ssl, size_t *
}
 again:
ret = ressl_read(ssl, c, 1, nr);
-   if (ret == -2)
+   if (ret == RESSL_READ_AGAIN)
goto again;
if (ret != 0)
errx(1, SSL read error: %u, ret);

Re: increase netcat's buffer...

[Ted Unangst]

On Mon, Oct 13, 2014 at 15:02, Arne Becker wrote:


 OK, no more fiddling with O_NONBLOCK.
 New diff below, tested with tcpbench and file transfers.

I think this is good. Thanks, committed. We'll let it sit for a while
and then see what if any changes need to take place on the buffer
side. Maybe we can revisit the original impetus and try 64k again.

Re: increase netcat's buffer...

[John-Mark Gurney]

Ted Unangst wrote this message on Thu, Oct 30, 2014 at 12:09 -0400:
 On Mon, Oct 13, 2014 at 15:02, Arne Becker wrote:
 
  OK, no more fiddling with O_NONBLOCK.
  New diff below, tested with tcpbench and file transfers.
 
 I think this is good. Thanks, committed. We'll let it sit for a while
 and then see what if any changes need to take place on the buffer
 side. Maybe we can revisit the original impetus and try 64k again.

I can't compile test this, but I think the vwrite cast could go away
if you changed the atomicio's f type from ssize_t (*f) (int, void *, size_t)
to ssize_t (*f) (int, const void *, size_t)...  There isn't any reason
why atomicio needs to have a non-const pointer to the buffer...

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 All that I will do, has been done, All that I have, has not.

add some sizes to free(9)

[Ted Unangst]

kern files, easy fixes.

Index: kern_sensors.c
===
RCS file: /cvs/src/sys/kern/kern_sensors.c,v
retrieving revision 1.29
diff -u -p -r1.29 kern_sensors.c
--- kern_sensors.c  14 Sep 2014 14:17:25 -  1.29
+++ kern_sensors.c  30 Oct 2014 17:33:29 -
@@ -247,7 +247,7 @@ sensor_task_work(void *xst, void *arg)
rw_exit_write(st-lock);
 
if (period == 0)
-   free(st, M_DEVBUF, 0);
+   free(st, M_DEVBUF, sizeof(struct sensor_task));
else 
timeout_add_sec(st-timeout, period);
 }
Index: kern_subr.c
===
RCS file: /cvs/src/sys/kern/kern_subr.c,v
retrieving revision 1.40
diff -u -p -r1.40 kern_subr.c
--- kern_subr.c 13 Jul 2014 15:48:41 -  1.40
+++ kern_subr.c 30 Oct 2014 17:32:56 -
@@ -221,7 +221,7 @@ hook_disestablish(struct hook_desc_head 
 #endif
hdp = vhook;
TAILQ_REMOVE(head, hdp, hd_list);
-   free(hdp, M_DEVBUF, 0);
+   free(hdp, M_DEVBUF, sizeof(*hdp));
 }
 
 /*
@@ -244,7 +244,7 @@ dohooks(struct hook_desc_head *head, int
TAILQ_REMOVE(head, hdp, hd_list);
(*hdp-hd_fn)(hdp-hd_arg);
if ((flags  HOOK_FREE) != 0)
-   free(hdp, M_DEVBUF, 0);
+   free(hdp, M_DEVBUF, sizeof(*hdp));
}
}
 }
Index: kern_sysctl.c
===
RCS file: /cvs/src/sys/kern/kern_sysctl.c,v
retrieving revision 1.267
diff -u -p -r1.267 kern_sysctl.c
--- kern_sysctl.c   17 Oct 2014 01:51:39 -  1.267
+++ kern_sysctl.c   30 Oct 2014 17:30:59 -
@@ -1288,7 +1288,7 @@ sysctl_file(int *name, u_int namelen, ch
error = EINVAL;
break;
}
-   free(kf, M_TEMP, 0);
+   free(kf, M_TEMP, sizeof(*kf));
 
if (!error) {
if (where == NULL)
@@ -1451,7 +1451,7 @@ again:
}
 err:
if (kproc)
-   free(kproc, M_TEMP, 0);
+   free(kproc, M_TEMP, sizeof(*kproc));
return (error);
 }
 
@@ -1732,7 +1732,7 @@ more:
 
 out:
uvmspace_free(vm);
-   free(buf, M_TEMP, 0);
+   free(buf, M_TEMP, PAGE_SIZE);
return (error);
 }
 
@@ -1796,7 +1796,7 @@ sysctl_proc_cwd(int *name, u_int namelen
}
 
vrele(vp);
-   free(path, M_TEMP, 0);
+   free(path, M_TEMP, len);
 
return (error);
 }
@@ -2095,7 +2095,7 @@ sysctl_sensors(int *name, u_int namelen,
ret = sysctl_rdstruct(oldp, oldlenp, newp, usd,
sizeof(struct sensordev));
 
-   free(usd, M_TEMP, 0);
+   free(usd, M_TEMP, sizeof(*usd));
return (ret);
}
 
@@ -2118,7 +2118,7 @@ sysctl_sensors(int *name, u_int namelen,
 
ret = sysctl_rdstruct(oldp, oldlenp, newp, us,
sizeof(struct sensor));
-   free(us, M_TEMP, 0);
+   free(us, M_TEMP, sizeof(*us));
return (ret);
 }
 
Index: kern_task.c
===
RCS file: /cvs/src/sys/kern/kern_task.c,v
retrieving revision 1.11
diff -u -p -r1.11 kern_task.c
--- kern_task.c 8 Oct 2014 15:28:39 -   1.11
+++ kern_task.c 30 Oct 2014 17:31:46 -
@@ -129,7 +129,7 @@ taskq_destroy(struct taskq *tq)
}
mtx_leave(tq-tq_mtx);
 
-   free(tq, M_DEVBUF, 0);
+   free(tq, M_DEVBUF, sizeof(*tq));
 }
 
 void
@@ -143,7 +143,7 @@ taskq_create_thread(void *arg)
switch (tq-tq_state) {
case TQ_S_DESTROYED:
mtx_leave(tq-tq_mtx);
-   free(tq, M_DEVBUF, 0);
+   free(tq, M_DEVBUF, sizeof(*tq));
return;
 
case TQ_S_CREATED:
Index: kern_tc.c
===
RCS file: /cvs/src/sys/kern/kern_tc.c,v
retrieving revision 1.26
diff -u -p -r1.26 kern_tc.c
--- kern_tc.c   14 Sep 2014 14:17:25 -  1.26
+++ kern_tc.c   30 Oct 2014 17:32:19 -
@@ -541,7 +541,7 @@ sysctl_tc_choice(void *oldp, size_t *old
strlcat(choices, buf, maxlen);
}
error = sysctl_rdstring(oldp, oldlenp, newp, choices);
-   free(choices, M_TEMP, 0);
+   free(choices, M_TEMP, maxlen);
return (error);
 }
 
Index: kern_workq.c
===
RCS file: /cvs/src/sys/kern/kern_workq.c,v
retrieving revision 1.16
diff -u -p -r1.16 kern_workq.c
--- kern_workq.c8 Oct 2014 15:28:39 -   1.16
+++ kern_workq.c30 Oct 2014 17:31:29 -
@@ -116,7 +116,7 @@ workq_destroy(struct workq *wq)
}
mtx_leave(wq-wq_mtx);
 
-   free(wq, M_DEVBUF, 0);
+   free(wq, M_DEVBUF, sizeof(*wq));
 }
 
 int
@@ -164,7 +164,7 @@ workq_create_thread(void *arg)
switch 

socket splicing thread

[Alexander Bluhm]

Hi,

Some performance measurements showed that socket splicing for TCP
can be made faster.  The main slowdown was that tcp_output() got
called for every incomming packet.  When copying through user-land
this cannot happen as the scheduler gets involved.

So my idea is to do the socket splicing for TCP in a special kernel
thread.  One drawback might be that the struct socket gets larger.
On amd64 that is from 472 to 520 bytes.  I could try to put the
splicing fields into a seperate struct that gets only allocated
when needed.

Does someone want to do some performance measurements with relayd?

ok?

bluhm

Index: sys/kern/uipc_socket.c
===
RCS file: /data/mirror/openbsd/cvs/src/sys/kern/uipc_socket.c,v
retrieving revision 1.133
diff -u -p -u -p -r1.133 uipc_socket.c
--- sys/kern/uipc_socket.c  9 Sep 2014 02:07:17 -   1.133
+++ sys/kern/uipc_socket.c  30 Oct 2014 18:56:28 -
@@ -56,6 +56,7 @@ void  sbsync(struct sockbuf *, struct mbu
 intsosplice(struct socket *, int, off_t, struct timeval *);
 void   sounsplice(struct socket *, struct socket *, int);
 void   soidle(void *);
+void   sotask(void *, void *);
 intsomove(struct socket *, int);
 
 void   filt_sordetach(struct knote *kn);
@@ -80,12 +81,18 @@ int somaxconn = SOMAXCONN;
 intsominconn = SOMINCONN;
 
 struct pool socket_pool;
+#ifdef SOCKET_SPLICE
+struct taskq *sosplice_taskq;
+#endif
 
 void
 soinit(void)
 {
 
pool_init(socket_pool, sizeof(struct socket), 0, 0, 0, sockpl, NULL);
+#ifdef SOCKET_SPLICE
+   sosplice_taskq = taskq_create(sosplice, 1, IPL_SOFTNET);
+#endif
 }
 
 /*
@@ -1101,6 +1108,7 @@ sosplice(struct socket *so, int fd, off_
else
timerclear(so-so_idletv);
timeout_set(so-so_idleto, soidle, so);
+   task_set(so-so_splicetask, sotask, so, NULL);
 
/*
 * To prevent softnet interrupt from calling somove() while
@@ -1124,6 +1132,7 @@ sounsplice(struct socket *so, struct soc
 {
splsoftassert(IPL_SOFTNET);
 
+   task_del(sosplice_taskq, so-so_splicetask);
timeout_del(so-so_idleto);
sosp-so_snd.sb_flagsintr = ~SB_SPLICE;
so-so_rcv.sb_flagsintr = ~SB_SPLICE;
@@ -1139,13 +1148,34 @@ soidle(void *arg)
int s;
 
s = splsoftnet();
-   if (so-so_splice) {
+   if (so-so_rcv.sb_flagsintr  SB_SPLICE) {
so-so_error = ETIMEDOUT;
sounsplice(so, so-so_splice, 1);
}
splx(s);
 }
 
+void
+sotask(void *arg1, void *arg2)
+{
+   struct socket *so = arg1;
+   int s;
+
+   s = splsoftnet();
+   if (so-so_rcv.sb_flagsintr  SB_SPLICE) {
+   /*
+* We may not sleep here as sofree() and unsplice() may be
+* called from softnet interrupt context.  This would remove
+* the socket during somove().
+*/
+   somove(so, M_DONTWAIT);
+   }
+   splx(s);
+
+   /* Avoid user land starvation. */
+   yield();
+}
+
 /*
  * Move data from receive buffer of spliced source socket to send
  * buffer of drain socket.  Try to move as much as possible in one
@@ -1414,8 +1444,20 @@ void
 sorwakeup(struct socket *so)
 {
 #ifdef SOCKET_SPLICE
-   if (so-so_rcv.sb_flagsintr  SB_SPLICE)
-   (void) somove(so, M_DONTWAIT);
+   if (so-so_rcv.sb_flagsintr  SB_SPLICE) {
+   /*
+* TCP has a sendbuffer that can handle multiple packets
+* at once.  So queue the stream a bit to accumulate data.
+* The sosplice thread will call somove() later and send
+* the packets calling tcp_output() only once.
+* In the UDP case, send out the packets immediately.
+* Using a thread would make things slower.
+*/
+   if (so-so_proto-pr_flags  PR_WANTRCVD)
+   task_add(sosplice_taskq, so-so_splicetask);
+   else
+   somove(so, M_DONTWAIT);
+   }
if (so-so_splice)
return;
 #endif
@@ -1429,7 +1471,7 @@ sowwakeup(struct socket *so)
 {
 #ifdef SOCKET_SPLICE
if (so-so_snd.sb_flagsintr  SB_SPLICE)
-   (void) somove(so-so_spliceback, M_DONTWAIT);
+   task_add(sosplice_taskq, so-so_spliceback-so_splicetask);
 #endif
sowakeup(so, so-so_snd);
 }
Index: sys/sys/socketvar.h
===
RCS file: /data/mirror/openbsd/cvs/src/sys/sys/socketvar.h,v
retrieving revision 1.56
diff -u -p -u -p -r1.56 socketvar.h
--- sys/sys/socketvar.h 9 Sep 2014 02:07:17 -   1.56
+++ sys/sys/socketvar.h 30 Oct 2014 18:32:16 -
@@ -34,6 +34,7 @@
 
 #include sys/selinfo.h   /* for struct selinfo */
 #include sys/queue.h
+#include sys/task.h
 #include sys/timeout.h
 
 #ifndef_SOCKLEN_T_DEFINED_
@@ -88,6 +89,7 @@ 

Remove USB locators

[Martin Pieuchot]

Since we cannot really guarantee that devices will be at the same place
on the USB tree and this stuff is unused, I don't see a reason to keep
it.

ok?

Index: usb_subr.c
===
RCS file: /cvs/src/sys/dev/usb/usb_subr.c,v
retrieving revision 1.110
diff -u -p -r1.110 usb_subr.c
--- usb_subr.c  5 Oct 2014 08:40:29 -   1.110
+++ usb_subr.c  30 Oct 2014 22:45:02 -
@@ -66,7 +66,6 @@ void  usbd_devinfo_vp(struct usbd_device
 char   *usbd_get_string(struct usbd_device *, int, char *, size_t);
 intusbd_getnewaddr(struct usbd_bus *);
 intusbd_print(void *, const char *);
-intusbd_submatch(struct device *, void *, void *);
 void   usbd_free_iface_data(struct usbd_device *, int);
 usbd_statususbd_probe_and_attach(struct device *,
struct usbd_device *, int, int);
@@ -873,7 +872,7 @@ usbd_probe_and_attach(struct device *par
 
/* First try with device specific drivers. */
DPRINTF((usbd_probe_and_attach trying device specific drivers\n));
-   dv = config_found_sm(parent, uaa, usbd_print, usbd_submatch);
+   dv = config_found(parent, uaa, usbd_print);
if (dv) {
dev-subdevs = malloc(2 * sizeof dv, M_USB, M_NOWAIT);
if (dev-subdevs == NULL) {
@@ -933,8 +932,7 @@ usbd_probe_and_attach(struct device *par
continue;
uaa.iface = ifaces[i];
uaa.ifaceno = ifaces[i]-idesc-bInterfaceNumber;
-   dv = config_found_sm(parent, uaa, usbd_print,
-  usbd_submatch);
+   dv = config_found(parent, uaa, usbd_print);
if (dv != NULL) {
dev-subdevs[dev-ndevs++] = dv;
usbd_claim_iface(dev, i);
@@ -970,7 +968,7 @@ generic:
uaa.configno = dev-ndevs == 0 ? UHUB_UNK_CONFIGURATION :
dev-cdesc-bConfigurationValue;
uaa.ifaceno = UHUB_UNK_INTERFACE;
-   dv = config_found_sm(parent, uaa, usbd_print, usbd_submatch);
+   dv = config_found(parent, uaa, usbd_print);
if (dv != NULL) {
if (dev-ndevs == 0) {
dev-subdevs = malloc(2 * sizeof dv, M_USB, M_NOWAIT);
@@ -1257,45 +1255,6 @@ usbd_print(void *aux, const char *pnp)
printf( %s\n, devinfop);
free(devinfop, M_TEMP, 0);
return (UNCONF);
-}
-
-int
-usbd_submatch(struct device *parent, void *match, void *aux)
-{
-   struct cfdata *cf = match;
-   struct usb_attach_arg *uaa = aux;
-
-   DPRINTFN(5,(usbd_submatch port=%d,%d configno=%d,%d 
-   ifaceno=%d,%d vendor=0x%x,0x%x product=0x%x,0x%x release=%d,%d\n,
-   uaa-port, cf-uhubcf_port,
-   uaa-configno, cf-uhubcf_configuration,
-   uaa-ifaceno, cf-uhubcf_interface,
-   uaa-vendor, cf-uhubcf_vendor,
-   uaa-product, cf-uhubcf_product,
-   uaa-release, cf-uhubcf_release));
-   if (uaa-port != 0/* root hub has port 0, it should match */
-   ((uaa-port != 0 
- cf-uhubcf_port != UHUB_UNK_PORT 
- cf-uhubcf_port != uaa-port) ||
-(uaa-configno != UHUB_UNK_CONFIGURATION 
- cf-uhubcf_configuration != UHUB_UNK_CONFIGURATION 
- cf-uhubcf_configuration != uaa-configno) ||
-(uaa-ifaceno != UHUB_UNK_INTERFACE 
- cf-uhubcf_interface != UHUB_UNK_INTERFACE 
- cf-uhubcf_interface != uaa-ifaceno) ||
-(uaa-vendor != UHUB_UNK_VENDOR 
- cf-uhubcf_vendor != UHUB_UNK_VENDOR 
- cf-uhubcf_vendor != uaa-vendor) ||
-(uaa-product != UHUB_UNK_PRODUCT 
- cf-uhubcf_product != UHUB_UNK_PRODUCT 
- cf-uhubcf_product != uaa-product) ||
-(uaa-release != UHUB_UNK_RELEASE 
- cf-uhubcf_release != UHUB_UNK_RELEASE 
- cf-uhubcf_release != uaa-release)
-)
-  )
-   return 0;
-   return ((*cf-cf_attach-ca_match)(parent, cf, aux));
 }
 
 void
Index: usbdivar.h
===
RCS file: /cvs/src/sys/dev/usb/usbdivar.h,v
retrieving revision 1.62
diff -u -p -r1.62 usbdivar.h
--- usbdivar.h  9 Aug 2014 09:45:14 -   1.62
+++ usbdivar.h  30 Oct 2014 22:47:06 -
@@ -253,39 +253,8 @@ void   usb_needs_explore(struct usbd_devi
 void   usb_needs_reattach(struct usbd_device *);
 void   usb_schedsoftintr(struct usbd_bus *);
 
-/* Locator stuff. */
-
-/* XXX these values are used to statically bind some elements in the USB tree
- * to specific driver instances. This should be somehow emulated in FreeBSD
- * but can be done later on.
- * The values are copied from the files.usb file in the NetBSD sources.
- */
-#define UHUBCF_PORT_DEFAULT -1

Re: socket splicing pool

[Alexander Bluhm]

On Thu, Oct 30, 2014 at 10:10:20PM +0100, Alexander Bluhm wrote:
 I could try to put the splicing fields into a seperate struct
 that gets only allocated when needed.

This way I can shrink struct socket from 472 to 392 bytes on amd64.
When splicing gets active, another 88 bytes are allocated for struct
sosplice.

before:
NameSize Requests FailInUse Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
sockpl   472749310  170   359   3302982 0 85

after:
sockpl   392 30580   8210 01010 0 81
sosppl88  36202 1 0 1 1 0 80

Is is worth it?

bluhm

Index: sys/kern/kern_sysctl.c
===
RCS file: /data/mirror/openbsd/cvs/src/sys/kern/kern_sysctl.c,v
retrieving revision 1.267
diff -u -p -u -p -r1.267 kern_sysctl.c
--- sys/kern/kern_sysctl.c  17 Oct 2014 01:51:39 -  1.267
+++ sys/kern/kern_sysctl.c  31 Oct 2014 01:04:46 -
@@ -1062,11 +1062,12 @@ fill_file(struct kinfo_file *kf, struct 
kf-so_family = so-so_proto-pr_domain-dom_family;
kf-so_rcv_cc = so-so_rcv.sb_cc;
kf-so_snd_cc = so-so_snd.sb_cc;
-   if (so-so_splice) {
+   if (isspliced(so)) {
if (show_pointers)
-   kf-so_splice = PTRTOINT64(so-so_splice);
-   kf-so_splicelen = so-so_splicelen;
-   } else if (so-so_spliceback)
+   kf-so_splice =
+   PTRTOINT64(so-so_sp-ssp_socket);
+   kf-so_splicelen = so-so_sp-ssp_len;
+   } else if (issplicedback(so))
kf-so_splicelen = -1;
if (!so-so_pcb)
break;
Index: sys/kern/uipc_socket.c
===
RCS file: /data/mirror/openbsd/cvs/src/sys/kern/uipc_socket.c,v
retrieving revision 1.133
diff -u -p -u -p -r1.133 uipc_socket.c
--- sys/kern/uipc_socket.c  9 Sep 2014 02:07:17 -   1.133
+++ sys/kern/uipc_socket.c  31 Oct 2014 01:04:19 -
@@ -80,12 +80,19 @@ int somaxconn = SOMAXCONN;
 intsominconn = SOMINCONN;
 
 struct pool socket_pool;
+#ifdef SOCKET_SPLICE
+struct pool sosplice_pool;
+#endif
 
 void
 soinit(void)
 {
 
pool_init(socket_pool, sizeof(struct socket), 0, 0, 0, sockpl, NULL);
+#ifdef SOCKET_SPLICE
+   pool_init(sosplice_pool, sizeof(struct sosplice), 0, 0, 0, sosppl,
+   NULL);
+#endif
 }
 
 /*
@@ -157,7 +164,7 @@ solisten(struct socket *so, int backlog)
if (so-so_state  (SS_ISCONNECTED|SS_ISCONNECTING|SS_ISDISCONNECTING))
return (EOPNOTSUPP);
 #ifdef SOCKET_SPLICE
-   if (so-so_splice || so-so_spliceback)
+   if (isspliced(so) || issplicedback(so))
return (EOPNOTSUPP);
 #endif /* SOCKET_SPLICE */
s = splsoftnet();
@@ -199,10 +206,15 @@ sofree(struct socket *so)
return;
}
 #ifdef SOCKET_SPLICE
-   if (so-so_spliceback)
-   sounsplice(so-so_spliceback, so, so-so_spliceback != so);
-   if (so-so_splice)
-   sounsplice(so, so-so_splice, 0);
+   if (so-so_sp) {
+   if (issplicedback(so))
+   sounsplice(so-so_sp-ssp_soback, so,
+   so-so_sp-ssp_soback != so);
+   if (isspliced(so))
+   sounsplice(so, so-so_sp-ssp_socket, 0);
+   pool_put(sosplice_pool, so-so_sp);
+   so-so_sp = NULL;
+   }
 #endif /* SOCKET_SPLICE */
sbrelease(so-so_snd);
sorflush(so);
@@ -647,7 +659,7 @@ restart:
 
m = so-so_rcv.sb_mb;
 #ifdef SOCKET_SPLICE
-   if (so-so_splice)
+   if (isspliced(so))
m = NULL;
 #endif /* SOCKET_SPLICE */
/*
@@ -669,7 +681,7 @@ restart:
 #ifdef DIAGNOSTIC
if (m == NULL  so-so_rcv.sb_cc)
 #ifdef SOCKET_SPLICE
-   if (so-so_splice == NULL)
+   if (!isspliced(so))
 #endif /* SOCKET_SPLICE */
panic(receive 1);
 #endif
@@ -1021,6 +1033,12 @@ sorflush(struct socket *so)
 }
 
 #ifdef SOCKET_SPLICE
+
+#define so_splicelen   so_sp-ssp_len
+#define so_splicemax   so_sp-ssp_max
+#define so_idletv  so_sp-ssp_idletv
+#define so_idleto  so_sp-ssp_idleto
+
 int
 sosplice(struct socket *so, int fd, off_t max, struct timeval *tv)
 {
@@ -1035,6 +1053,8 @@ sosplice(struct socket *so, int fd, off_
if ((so-so_state  (SS_ISCONNECTED|SS_ISCONNECTING)) == 0 
(so-so_proto-pr_flags  PR_CONNREQUIRED))
return (ENOTCONN);
+   if (so-so_sp == NULL)
+   so-so_sp = pool_get(sosplice_pool, PR_WAITOK | PR_ZERO);
 
/* If no fd is given, unsplice by removing existing link. */
if (fd  0) {
@@ -1043,8 

Re: socket splicing thread

[Ted Unangst]

On Thu, Oct 30, 2014 at 22:10, Alexander Bluhm wrote:
 +
 + /* Avoid user land starvation. */
 + yield();

I think this is the responsibility of the taskq thread, not the
individual task.

Regarding the second pool diff, that looks very reasonable. Quite a
savings for a feature many machines will never use.

EDNS0 and getrrsetbyname(3)

[Mike Burns]

It is my understanding of the code that ssh(1) no longer supports
DNSSEC. I say this because it calls getrrsetbyname(3) to check the
secure flag, which calls into the ASR machine. In asr.c, the `pass0'
function only parses two options, tcp and ndots:, ignoring edns0.
Indeed, the asr_run(3) man page lists a lack of EDNS0 support as a
caveat.

Until EDNS0 is supported, it would be helpful to change the documentation.

-Mike


Index: lib/libc/net/getrrsetbyname.3
===
RCS file: /cvs/src/lib/libc/net/getrrsetbyname.3,v
retrieving revision 1.19
diff -u -p -u -r1.19 getrrsetbyname.3
--- lib/libc/net/getrrsetbyname.3   23 Aug 2014 07:25:54 -  1.19
+++ lib/libc/net/getrrsetbyname.3   31 Oct 2014 02:00:29 -
@@ -115,10 +115,11 @@ is a pointer to a
 created by a call to
 .Fn getrrsetbyname .
 .Pp
-If the EDNS0 option is activated in
-.Xr resolv.conf 5 ,
+The EDNS0 option in
+.Xr resolv.conf 5
+is currently ignored, which means that
 .Fn getrrsetbyname
-will request DNSSEC authentication using the EDNS0 DNSSEC OK (DO) bit.
+will not request DNSSEC authentication using the EDNS0 DNSSEC OK (DO) bit.
 .Sh RETURN VALUES
 .Fn getrrsetbyname
 returns zero on success, and one of the following error
Index: share/man/man5/resolv.conf.5
===
RCS file: /cvs/src/share/man/man5/resolv.conf.5,v
retrieving revision 1.45
diff -u -p -u -r1.45 resolv.conf.5
--- share/man/man5/resolv.conf.523 Aug 2014 07:25:54 -  1.45
+++ share/man/man5/resolv.conf.531 Oct 2014 02:00:29 -
@@ -281,6 +281,9 @@ particularly if there is a reduced MTU,
 as is often the case with
 .Xr pppoe 4
 or with tunnels.
+.Pp
+This option is ignored on
+.Ox .
 .It Cm inet6
 Enables support for IPv6-only applications, by setting RES_USE_INET6 in
 _res.options (see

clear up libcrypto zlib

[Ted Unangst]

Remove a few bits of junk, mostly support for dso loading of zlib
which i suspect is fully dead.

Index: c_rle.c
===
RCS file: /cvs/src/lib/libssl/src/crypto/comp/c_rle.c,v
retrieving revision 1.7
diff -u -p -r1.7 c_rle.c
--- c_rle.c 12 Jun 2014 15:49:28 -  1.7
+++ c_rle.c 31 Oct 2014 04:03:32 -
@@ -27,10 +27,8 @@ static int
 rle_compress_block(COMP_CTX *ctx, unsigned char *out, unsigned int olen,
 unsigned char *in, unsigned int ilen)
 {
-   /* int i; */
 
if (ilen == 0 || olen  (ilen - 1)) {
-   /* ZZ */
return (-1);
}
 
@@ -46,7 +44,6 @@ rle_expand_block(COMP_CTX *ctx, unsigned
int i;
 
if (olen  (ilen - 1)) {
-   /* ZZ */
return (-1);
}
 
Index: c_zlib.c
===
RCS file: /cvs/src/lib/libssl/src/crypto/comp/c_zlib.c,v
retrieving revision 1.16
diff -u -p -r1.16 c_zlib.c
--- c_zlib.c12 Jun 2014 15:49:28 -  1.16
+++ c_zlib.c31 Oct 2014 04:07:43 -
@@ -13,9 +13,7 @@ static COMP_METHOD zlib_method_nozlib = 
.name = (undef)
 };
 
-#ifndef ZLIB
-#undef ZLIB_SHARED
-#else
+#ifdef ZLIB
 
 #include zlib.h
 
@@ -40,23 +38,6 @@ zlib_zfree(void* opaque, void* address)
free(address);
 }
 
-#if 0
-static int zlib_compress_block(COMP_CTX *ctx, unsigned char *out,
-unsigned int olen, unsigned char *in, unsigned int ilen);
-static int zlib_expand_block(COMP_CTX *ctx, unsigned char *out,
-unsigned int olen, unsigned char *in, unsigned int ilen);
-
-static int zz_uncompress(Bytef *dest, uLongf *destLen, const Bytef *source,
-uLong sourceLen);
-
-static COMP_METHOD zlib_stateless_method = {
-   .type = NID_zlib_compression,
-   .name = LN_zlib_compression,
-   .compress = zlib_compress_block,
-   .expand = zlib_expand_block
-};
-#endif
-
 static COMP_METHOD zlib_stateful_method = {
.type = NID_zlib_compression,
.name = LN_zlib_compression,
@@ -66,43 +47,6 @@ static COMP_METHOD zlib_stateful_method 
.expand = zlib_stateful_expand_block
 };
 
-#ifdef ZLIB_SHARED
-#include openssl/dso.h
-
-/* Function pointers */
-typedef int (*compress_ft)(Bytef *dest, uLongf *destLen, const Bytef *source,
-uLong sourceLen);
-typedef int (*inflateEnd_ft)(z_streamp strm);
-typedef int (*inflate_ft)(z_streamp strm, int flush);
-typedef int (*inflateInit__ft)(z_streamp strm, const char * version,
-int stream_size);
-typedef int (*deflateEnd_ft)(z_streamp strm);
-typedef int (*deflate_ft)(z_streamp strm, int flush);
-typedef int (*deflateInit__ft)(z_streamp strm, int level, const char * version,
-int stream_size);
-typedef const char * (*zError__ft)(int err);
-static compress_ft p_compress = NULL;
-static inflateEnd_ft   p_inflateEnd = NULL;
-static inflate_ft  p_inflate = NULL;
-static inflateInit__ft p_inflateInit_ = NULL;
-static deflateEnd_ft   p_deflateEnd = NULL;
-static deflate_ft  p_deflate = NULL;
-static deflateInit__ft p_deflateInit_ = NULL;
-static zError__ft  p_zError = NULL;
-
-static int zlib_loaded = 0; /* only attempt to init func pts once */
-static DSO *zlib_dso = NULL;
-
-#define compressp_compress
-#define inflateEnd  p_inflateEnd
-#define inflate p_inflate
-#define inflateInit_p_inflateInit_
-#define deflateEnd  p_deflateEnd
-#define deflate p_deflate
-#define deflateInit_p_deflateInit_
-#define zError p_zError
-#endif /* ZLIB_SHARED */
-
 struct zlib_state {
z_stream istream;
z_stream ostream;
@@ -225,102 +169,6 @@ zlib_stateful_expand_block(COMP_CTX *ctx
return olen - state-istream.avail_out;
 }
 
-#if 0
-static int
-zlib_compress_block(COMP_CTX *ctx, unsigned char *out,
-unsigned int olen, unsigned char *in, unsigned int ilen)
-{
-   unsigned long l;
-   int i;
-   int clear = 1;
-
-   if (ilen  128) {
-   out[0] = 1;
-   l = olen - 1;
-   i = compress((out[1]), l, in, (unsigned long)ilen);
-   if (i != Z_OK)
-   return (-1);
-   if (ilen  l) {
-   clear = 0;
-   l++;
-   }
-   }
-   if (clear) {
-   out[0] = 0;
-   memcpy((out[1]), in, ilen);
-   l = ilen + 1;
-   }
-
-#ifdef DEBUG_ZLIB
-   fprintf(stderr, compress(%4d)-%4d %s\n,
-   ilen,(int)l, (clear)?clear:zlib);
-#endif
-
-   return ((int)l);
-}
-
-static int
-zlib_expand_block(COMP_CTX *ctx, unsigned char *out, unsigned int olen,
-unsigned char *in, unsigned int ilen)
-{
-   unsigned long l;
-   int i;
-
-   if (in[0]) {
-   l = olen;
-   i = zz_uncompress(out, l, (in[1]), 

libcrypto: use libc string fns

[Ted Unangst]

Don't need BUF_ and its NULL arg handling here.

Index: x509/x509_trs.c
===
RCS file: /cvs/src/lib/libssl/src/crypto/x509/x509_trs.c,v
retrieving revision 1.16
diff -u -p -r1.16 x509_trs.c
--- x509/x509_trs.c 28 Sep 2014 10:52:59 -  1.16
+++ x509/x509_trs.c 31 Oct 2014 04:13:14 -
@@ -57,6 +57,7 @@
  */
 
 #include stdio.h
+#include string.h
 
 #include openssl/err.h
 #include openssl/x509v3.h
@@ -202,7 +203,7 @@ X509_TRUST_add(int id, int flags, int (*
if (trtmp-flags  X509_TRUST_DYNAMIC_NAME)
free(trtmp-name);
/* dup supplied name */
-   if ((trtmp-name = BUF_strdup(name)) == NULL)
+   if ((trtmp-name = strdup(name)) == NULL)
goto err;
/* Keep the dynamic flag of existing entry */
trtmp-flags = X509_TRUST_DYNAMIC;
Index: x509v3/v3_addr.c
===
RCS file: /cvs/src/lib/libssl/src/crypto/x509v3/v3_addr.c,v
retrieving revision 1.13
diff -u -p -r1.13 v3_addr.c
--- x509v3/v3_addr.c13 Jul 2014 16:03:10 -  1.13
+++ x509v3/v3_addr.c31 Oct 2014 04:12:45 -
@@ -1019,7 +1019,7 @@ v2i_IPAddrBlocks(const struct v3_ext_met
length = length_from_afi(afi);
 
/*
-* Handle SAFI, if any, and BUF_strdup() so we can 
null-terminate
+* Handle SAFI, if any, and strdup() so we can null-terminate
 * the other input values.
 */
if (safi != NULL) {
Index: store/str_lib.c
===
RCS file: /cvs/src/lib/libssl/src/crypto/store/str_lib.c,v
retrieving revision 1.10
diff -u -p -r1.10 str_lib.c
--- store/str_lib.c 10 Jul 2014 22:45:58 -  1.10
+++ store/str_lib.c 31 Oct 2014 04:12:30 -
@@ -1341,7 +1341,7 @@ STORE_ATTR_INFO_set_cstr(STORE_ATTR_INFO
return 0;
}
if (!ATTR_IS_SET(attrs, code)) {
-   if ((attrs-values[code].cstring = BUF_strndup(cstr, 
cstr_size)))
+   if ((attrs-values[code].cstring = strndup(cstr, cstr_size)))
return 1;
STOREerr(STORE_F_STORE_ATTR_INFO_SET_CSTR,
ERR_R_MALLOC_FAILURE);

Re: socket splicing thread

[David Gwynne]

 On 31 Oct 2014, at 07:10, Alexander Bluhm alexander.bl...@gmx.net wrote:
 
 Hi,
 
 Some performance measurements showed that socket splicing for TCP
 can be made faster.  The main slowdown was that tcp_output() got
 called for every incomming packet.  When copying through user-land
 this cannot happen as the scheduler gets involved.

so without splicing, the payloads from multiple tcp packets (at least all of 
the ones in a single softnet run?) get bundled up into a buffer that userland 
reads and then writes out again in a single go. right?

you're suggesting the taskq as a way to defer output till after the current 
softnet call has processed all its packets and queued all the tcp packet 
payloads onto the socket?

 
 So my idea is to do the socket splicing for TCP in a special kernel
 thread.  One drawback might be that the struct socket gets larger.
 On amd64 that is from 472 to 520 bytes.  I could try to put the
 splicing fields into a seperate struct that gets only allocated
 when needed.

its worth remembering there are other memory costs too. i think a kthread (the 
thing taskqs run on) is 5 pages amd64, so 20KB.

 Does someone want to do some performance measurements with relayd?
 
 ok?
 
 bluhm
 
 Index: sys/kern/uipc_socket.c
 ===
 RCS file: /data/mirror/openbsd/cvs/src/sys/kern/uipc_socket.c,v
 retrieving revision 1.133
 diff -u -p -u -p -r1.133 uipc_socket.c
 --- sys/kern/uipc_socket.c9 Sep 2014 02:07:17 -   1.133
 +++ sys/kern/uipc_socket.c30 Oct 2014 18:56:28 -
 @@ -56,6 +56,7 @@ voidsbsync(struct sockbuf *, struct mbu
 int   sosplice(struct socket *, int, off_t, struct timeval *);
 void  sounsplice(struct socket *, struct socket *, int);
 void  soidle(void *);
 +void sotask(void *, void *);
 int   somove(struct socket *, int);
 
 void  filt_sordetach(struct knote *kn);
 @@ -80,12 +81,18 @@ int   somaxconn = SOMAXCONN;
 int   sominconn = SOMINCONN;
 
 struct pool socket_pool;
 +#ifdef SOCKET_SPLICE
 +struct taskq *sosplice_taskq;
 +#endif
 
 void
 soinit(void)
 {
 
   pool_init(socket_pool, sizeof(struct socket), 0, 0, 0, sockpl, NULL);
 +#ifdef SOCKET_SPLICE
 + sosplice_taskq = taskq_create(sosplice, 1, IPL_SOFTNET);
 +#endif
 }
 
 /*
 @@ -1101,6 +1108,7 @@ sosplice(struct socket *so, int fd, off_
   else
   timerclear(so-so_idletv);
   timeout_set(so-so_idleto, soidle, so);
 + task_set(so-so_splicetask, sotask, so, NULL);
 
   /*
* To prevent softnet interrupt from calling somove() while
 @@ -1124,6 +1132,7 @@ sounsplice(struct socket *so, struct soc
 {
   splsoftassert(IPL_SOFTNET);
 
 + task_del(sosplice_taskq, so-so_splicetask);
   timeout_del(so-so_idleto);
   sosp-so_snd.sb_flagsintr = ~SB_SPLICE;
   so-so_rcv.sb_flagsintr = ~SB_SPLICE;
 @@ -1139,13 +1148,34 @@ soidle(void *arg)
   int s;
 
   s = splsoftnet();
 - if (so-so_splice) {
 + if (so-so_rcv.sb_flagsintr  SB_SPLICE) {
   so-so_error = ETIMEDOUT;
   sounsplice(so, so-so_splice, 1);
   }
   splx(s);
 }
 
 +void
 +sotask(void *arg1, void *arg2)
 +{
 + struct socket *so = arg1;
 + int s;
 +
 + s = splsoftnet();
 + if (so-so_rcv.sb_flagsintr  SB_SPLICE) {
 + /*
 +  * We may not sleep here as sofree() and unsplice() may be
 +  * called from softnet interrupt context.  This would remove
 +  * the socket during somove().
 +  */
 + somove(so, M_DONTWAIT);
 + }
 + splx(s);
 +
 + /* Avoid user land starvation. */
 + yield();
 +}
 +
 /*
  * Move data from receive buffer of spliced source socket to send
  * buffer of drain socket.  Try to move as much as possible in one
 @@ -1414,8 +1444,20 @@ void
 sorwakeup(struct socket *so)
 {
 #ifdef SOCKET_SPLICE
 - if (so-so_rcv.sb_flagsintr  SB_SPLICE)
 - (void) somove(so, M_DONTWAIT);
 + if (so-so_rcv.sb_flagsintr  SB_SPLICE) {
 + /*
 +  * TCP has a sendbuffer that can handle multiple packets
 +  * at once.  So queue the stream a bit to accumulate data.
 +  * The sosplice thread will call somove() later and send
 +  * the packets calling tcp_output() only once.
 +  * In the UDP case, send out the packets immediately.
 +  * Using a thread would make things slower.
 +  */
 + if (so-so_proto-pr_flags  PR_WANTRCVD)
 + task_add(sosplice_taskq, so-so_splicetask);
 + else
 + somove(so, M_DONTWAIT);
 + }
   if (so-so_splice)
   return;
 #endif
 @@ -1429,7 +1471,7 @@ sowwakeup(struct socket *so)
 {
 #ifdef SOCKET_SPLICE
   if (so-so_snd.sb_flagsintr  SB_SPLICE)
 - (void) somove(so-so_spliceback, M_DONTWAIT);
 + task_add(sosplice_taskq, so-so_spliceback-so_splicetask);