Re: remove some altq references from manuals

2013-11-01 Thread Stuart Henderson
On 2013/10/31 21:06, Arto Jonsson wrote:
 Index: usr.bin/systat/systat.1
 ===
 RCS file: /cvs/src/usr.bin/systat/systat.1,v
 retrieving revision 1.96
 diff -u -p -r1.96 systat.1
 --- usr.bin/systat/systat.1   7 Sep 2013 11:43:50 -   1.96
 +++ usr.bin/systat/systat.1   19 Oct 2013 16:34:56 -
 @@ -376,9 +376,8 @@ Available orderings are:
  and
  .Ic number of pages .
  .It Ic queues
 -Display statistics about the active
 -.Xr altq 9
 -queues, similar to the output of
 +Display statistics about the active queues,
 +similar to the output of
  .Cm pfctl Fl s Cm queue .
  .It Ic rules
  Display pf rules statistics, similar to the output of
 

Actually this one is incorrect. systat only handles altq queues.



Re: remove some altq references from manuals

2013-11-01 Thread Arto Jonsson
On Fri, Nov 01, 2013 at 10:45:16AM +, Stuart Henderson wrote:
 On 2013/10/31 21:06, Arto Jonsson wrote:
  Index: usr.bin/systat/systat.1
  ===
  RCS file: /cvs/src/usr.bin/systat/systat.1,v
  retrieving revision 1.96
  diff -u -p -r1.96 systat.1
  --- usr.bin/systat/systat.1 7 Sep 2013 11:43:50 -   1.96
  +++ usr.bin/systat/systat.1 19 Oct 2013 16:34:56 -
  @@ -376,9 +376,8 @@ Available orderings are:
   and
   .Ic number of pages .
   .It Ic queues
  -Display statistics about the active
  -.Xr altq 9
  -queues, similar to the output of
  +Display statistics about the active queues,
  +similar to the output of
   .Cm pfctl Fl s Cm queue .
   .It Ic rules
   Display pf rules statistics, similar to the output of
  
 
 Actually this one is incorrect. systat only handles altq queues.

I guess I just volunteered myself to add those bits to systat.



OpenBSD 5.4 released Nov 1, 2013

2013-11-01 Thread Nick Holland

- OpenBSD 5.4 RELEASED -

November 1, 2013.

We are pleased to announce the official release of OpenBSD 5.4.
This is our 34th release on CD-ROM (and 35th via FTP).  We remain
proud of OpenBSD's record of more than ten years with only two remote
holes in the default install.

As in our previous releases, 5.4 provides significant improvements,
including new features, in nearly all areas of the system:

 - New/extended platforms:
o OpenBSD/octeon
  New platform for systems based on the Cavium Octeon
  MIPS-compatible processors. Supported machines include:
  - Portwell CAM-0100
  - Ubiquiti Networks EdgeRouter LITE (no local storage)
o OpenBSD/beagle
  New platform for OMAP3/4 and AM335x systems using an ARM Cortex-A8
  or Cortex-A9 CPU. Supported boards include:
  - BeagleBoard C4 / xM
  - BeagleBone and BeagleBone Black
  - PandaBoard and PandaBoard ES

 - Improved hardware support, including:
o inteldrm(4) has been overhauled, including:
  - Now mostly in sync with Linux 3.8.13.
  - Support for Kernel Mode Setting (KMS) including support for
additional output types such as DisplayPort.
  - Sandy Bridge and newer parts which previously had only ShadowFB
acceleration now have full hardware acceleration including use
of the 3D rings.
  - wsdisplay(4) now attaches to inteldrm(4) and providers a
framebuffer console. 
o vgafb(4/macppc) now supports multiple virtual consoles.
o Support for Elantech touchpads version 4 (clickpad) added to
  pms(4).
o Fixed st(4) EOM handling, enabling much better Bacula support.
o Support for vdsk(4) disks larger than 2TB.

 - Generic network stack improvements:
o Reworked checksum handling for network protocols.
o divert(4) now recalculates the IP and protocol checksums of
  reinjected packets.
o No longer attempt to delete the undeletable RNF_ROOT route.

 - Routing daemons and other userland network improvements:
o Support SSL inspection in relayd(8).
o Added slowcgi(8), a libevent-based FastCGI implementation.
o Enabled ECDHE support in httpd(8).
o Do not start inetd(8) by default any more.
o Many ldpd(8) improvements, including a speed-up of the session
  establishment process, support for adjacencies and targeted
  hellos, support for multiple addresses per interface, and more.

 - dhcpd(8) improvements:
o Improved compliance with RFC 2131 strictures on
  client-identifiers.
o Fixed synchronization of leases.
o Replaced manual date parsing and printing with strftime and
  strptime.
o Explicitly label dates in leases files as being UTC dates. 

 - dhclient(8) improvements:
o Delete routes added by defunct dhclient processes.
o Improved handling of client-identifier option.
o Increased ip_ttl on packets to 128, allowing more distant servers
  to provide leases.
o Replaced manual date parsing and printing with strftime and
  strptime.
o Explicitly label dates in leases files as being UTC dates.
o Improved interactions between dhclient processes to make the most
  recent dhclient started the most likely to persist.
o Support for static routes and classless static routes options.
o Fixed log messages to print correct addresses.
o Reduced log verbosity by emitting debug messages only when
  debugging.
o Eliminated unnecessary address and route churn during lease
  renewal by not binding leases identical to the current one.

 - OpenSMTPD 5.3.3:
o New features:
  - Add support for LMTP local deliveries
  - Add SECURE and AUTH transmission types
  - Add support for transparent queue compression
  - helo names can now be looked up in a db(3) table
  - New error: alias kind allows aliasing a user-part to an error
  - Traces can be (de)activated at runtime
o Improvements:
  - More robust queue can cope with runtime errors
  - Improved routing strategies
  - Assorted minor bug fixes and cleanups

 - Performance improvements:
o Don't require the kernel lock when processing audio interrupts.
o Improved kernel bcopy/memmove/memcpy implementations and made more
  careful choices between them.
o Implemented symbol caching and RELCOUNT/RELACOUNT optimizations in
  ld.so(1).

 - Threading improvements:
o Closed various race conditions between exit/fork/execve/__tfork/
  __threxit/ptrace in both the kernel and libpthread.

 - Assorted improvements:
o Added a locale(1) utility.
o Added ltrace(1), a tool to trace PLT calls.
o Added a new implementation of cu(1).
o Added shm_open(3)/shm_unlink(3).
o Added getprogname(3)/setprogname(3).
o Added clock_getcpuclockid(3) and pthread_getcpuclockid(3).
o Added fmemopen(3).
o Added 

Re: pflow in rdomain

2013-11-01 Thread Florian Obser
commited, thanks!

On Thu, Oct 31, 2013 at 04:30:51PM +0100, Anders Berggren wrote:
 We tried to get pflow running in a non-default rdomain, and found this to get 
 it going. Make sense?
 
 --- sys/net/if_pflow.c.orig   Fri Sep 13 20:58:40 2013
 +++ sys/net/if_pflow.cMon Sep 16 13:25:54 2013
 @@ -1213,6 +1213,8 @@
   sc-sc_if.if_opackets++;
   sc-sc_if.if_obytes += m-m_pkthdr.len;
  
 + m-m_pkthdr.rdomain = sc-sc_if.if_rdomain;
 +
   if ((err = ip_output(m, NULL, NULL, IP_RAWOUTPUT, sc-sc_imo, NULL))) {
   pflowstats.pflow_oerrors++;
   sc-sc_if.if_oerrors++;
 

-- 
I'm not entirely sure you are real.



Re: update perl Module::Build in base

2013-11-01 Thread Ingo Schwarze
Hi Andrew,

Andrew Fresh wrote on Wed, Oct 30, 2013 at 01:50:56PM -0700:
 On Wed, Oct 30, 2013 at 08:31:58PM +0100, Ingo Schwarze wrote:

 -'DISTRIBUTION' = 'DAGOLDEN/Module-Build-0.39_01.tar.gz',
 +'DISTRIBUTION' = 'LEONT/Module-Build-0.4007.tar.gz',

 It's only 4.003 in 5.18, not sure if that makes a difference,

That would be sufficient to appease CPAN modules requiring =0.40,
but it wouldn't include three-argument-open(3p), and it lacks a few
minor bug fixes and enhancements.

 but this patch would need to be ported forward if we move to newer
 versions of perl.

I see your point.  Right now, i feel unable to judge how much
inconvenience that might cause, and which is the better way.

 https://github.com/Perl/perl5/tree/maint-5.18/cpan/Module-Build
 
 Module::Build will be deprecated as a core module in 5.20 (May 2014) and
 removed in 5.22 (I guess May 2015).
 http://www.dagolden.com/index.php/2140/paying-respect-to-modulebuild/

Oh well.  To paraphase (forgive me if i misunderstand), It ought to
be easy to write something better than Module::Build, and even though
nobody has so far volunteered to propose an actual design, and even
less to write some code, we already have a definite time schedule for
retiring Module::Build.  WT...?

However, whetever thay do, we are certainly able to cope, either just
keeping Module::Build in base or moving it to a port, whatever will be
more convenient when the time comes.

 Post vBSDcon I have most existing patches working with 5.18.1,

I have serious doubts that i'm experienced enough to manage a
complete Perl update; certainly not without help from more experienced
developers.  However, i'm currently having a look at your work
before making up my mind about what to propose for proceeding with
the update to Module::Build or even Perl as a whole.

 the only failing so far is that with threads enabled t/op/threads-dirh.t
 fails.
 (mv patches/{APPLIES,GOOD}/use_threads.patch)
 This test appears to point to a failure that needs fixing but I am not
 skilled enough to know how. 

I'll try to look at it - even though i know almost nothing about
threads so far.  But maybe we can find someone to help with this
specific detail, if that's really the main remaining hurdle.

 https://github.com/afresh1/OpenBSD-perl
 
 Would it be preferred to plan ahead for removal and to somehow do this
 update in the ports tree?

I don't think so.  Even when we update to 5.18, Module::Build is still
in base, and i don't see any reason to move it out even earlier than
upstream will (maybe) do it.  Which version of Module::Build we update
to is an orthogonal question, anyway.

If upstream sticks to their plan (even tough it still seems vague to me),
then right before merging perl-5.22 to OpenBSD base would be the right
time to decide whether we want to keep Module::Build in base, or to set
up a port at that point in time.

 Other than that, the changes that I saw in that patch didn't look scary
 to me and I would be happy to use the updated version, three argument
 open seems like a good improvement.

Good, thank you.  If i come to the conclusion that we cannot upgrade
to perl 1.18 right now, i will consider how to proceed with
Module::Build only, for now.

Yours,
  Ingo



Re: update perl Module::Build in base

2013-11-01 Thread Marc Espie
On Fri, Nov 01, 2013 at 04:56:04PM +0100, Ingo Schwarze wrote:
 Oh well.  To paraphase (forgive me if i misunderstand), It ought to
 be easy to write something better than Module::Build, and even though
 nobody has so far volunteered to propose an actual design, and even
 less to write some code, we already have a definite time schedule for
 retiring Module::Build.  WT...?
 
 However, whetever thay do, we are certainly able to cope, either just
 keeping Module::Build in base or moving it to a port, whatever will be
 more convenient when the time comes.

Go read Chromatics blog

http://modernperlbooks.com/mt/index.html

perl has been schizoid between practicality and moving forward for a few
years now.

One thing it has going for it is that there are very bright people who
understand most of the issues involved, and so I'm not too worried about
what's going to happen...



pool(9) for usbd_xfer

2013-11-01 Thread Martin Pieuchot
Diff below converts the three USB controllers to use a pool(9) for
allocating their transfer descriptors instead of maintaining their
own custom free list.

With it uhci(4) and ehci(4) no longer initialized the isdone value
to 1 for every new transfer descriptor (if DIAGNOSTIC is defined).
This is done on purpose so that these descriptors don't get ignored
by the abort function in case they were not properly initialized (and
that happens on various code paths).

Comments, ok?


Index: usb.c
===
RCS file: /cvs/src/sys/dev/usb/usb.c,v
retrieving revision 1.92
diff -u -p -r1.92 usb.c
--- usb.c   30 May 2013 16:15:02 -  1.92
+++ usb.c   1 Nov 2013 21:54:43 -
@@ -156,6 +156,8 @@ usb_attach(struct device *parent, struct
DPRINTF((usbd_attach\n));
 
if (usb_nbuses == 0) {
+   usbd_init();
+
rw_init(usbpalock, usbpalock);
TAILQ_INIT(usb_abort_tasks);
TAILQ_INIT(usb_explore_tasks);
Index: usbdi.c
===
RCS file: /cvs/src/sys/dev/usb/usbdi.c,v
retrieving revision 1.63
diff -u -p -r1.63 usbdi.c
--- usbdi.c 31 Oct 2013 20:06:59 -  1.63
+++ usbdi.c 1 Nov 2013 21:54:43 -
@@ -37,6 +37,7 @@
 #include sys/kernel.h
 #include sys/device.h
 #include sys/malloc.h
+#include sys/pool.h
 
 #include machine/bus.h
 
@@ -55,12 +56,20 @@ extern int usbdebug;
 #define DPRINTFN(n,x)
 #endif
 
-void usbd_do_request_async_cb(struct usbd_xfer *, void *,
-usbd_status);
+struct pool usbd_xfer_pool;
+
+void usbd_do_request_async_cb(struct usbd_xfer *, void *, usbd_status);
 void usbd_start_next(struct usbd_pipe *pipe);
 usbd_status usbd_open_pipe_ival(struct usbd_interface *, u_int8_t, u_int8_t,
 struct usbd_pipe **, int);
 
+void
+usbd_init(void)
+{
+   pool_init(usbd_xfer_pool, USBD_XFER_MAXSIZE, 0, 0, 0, usbdxfers,
+   NULL);
+}
+
 int
 usbd_is_dying(struct usbd_device *dev)
 {
@@ -402,11 +411,14 @@ usbd_alloc_xfer(struct usbd_device *dev)
 {
struct usbd_xfer *xfer;
 
-   xfer = dev-bus-methods-allocx(dev-bus);
+   xfer = pool_get(usbd_xfer_pool, PR_NOWAIT | PR_ZERO);
if (xfer == NULL)
return (NULL);
xfer-device = dev;
timeout_set(xfer-timeout_handle, NULL, NULL);
+#ifdef DIAGNOSTIC
+   xfer-busy_free = XFER_BUSY;
+#endif
DPRINTFN(5,(usbd_alloc_xfer() = %p\n, xfer));
return (xfer);
 }
@@ -417,7 +429,7 @@ usbd_free_xfer(struct usbd_xfer *xfer)
DPRINTFN(5,(usbd_free_xfer: %p\n, xfer));
if (xfer-rqflags  (URQ_DEV_DMABUF | URQ_AUTO_DMABUF))
usbd_free_buffer(xfer);
-   xfer-device-bus-methods-freex(xfer-device-bus, xfer);
+   pool_put(usbd_xfer_pool, xfer);
return (USBD_NORMAL_COMPLETION);
 }
 
Index: usbdi.h
===
RCS file: /cvs/src/sys/dev/usb/usbdi.h,v
retrieving revision 1.55
diff -u -p -r1.55 usbdi.h
--- usbdi.h 31 Oct 2013 10:12:19 -  1.55
+++ usbdi.h 1 Nov 2013 21:54:43 -
@@ -83,6 +83,16 @@ typedef void (*usbd_callback)(struct usb
 
 #define DEVINFOSIZE 1024
 
+/*
+ * Maximum controller xfer length (that is including the usbd_xfer
+ * structure).  Please make sure to update this value when changing the
+ * length of an existing controller xfer type or when adding a new one
+ * that has a larger size than the value below.
+ */
+#define USBD_XFER_MAXSIZE  288
+
+void usbd_init(void);
+
 usbd_status usbd_open_pipe(struct usbd_interface *iface, u_int8_t address,
 u_int8_t flags, struct usbd_pipe **pipe);
 usbd_status usbd_close_pipe(struct usbd_pipe *pipe);
Index: usbdivar.h
===
RCS file: /cvs/src/sys/dev/usb/usbdivar.h,v
retrieving revision 1.53
diff -u -p -r1.53 usbdivar.h
--- usbdivar.h  1 Nov 2013 12:00:54 -   1.53
+++ usbdivar.h  1 Nov 2013 21:54:43 -
@@ -54,8 +54,6 @@ struct usbd_bus_methods {
usbd_status   (*open_pipe)(struct usbd_pipe *pipe);
void  (*soft_intr)(void *);
void  (*do_poll)(struct usbd_bus *);
-   struct usbd_xfer *(*allocx)(struct usbd_bus *);
-   void  (*freex)(struct usbd_bus *, struct usbd_xfer *);
 };
 
 struct usbd_pipe_methods {
@@ -185,7 +183,6 @@ struct usbd_xfer {
__volatile char done;
 #ifdef DIAGNOSTIC
u_int32_t   busy_free;
-#define XFER_FREE 0x46524545
 #define XFER_BUSY 0x42555359
 #define XFER_ONQU 0x4f4e5155
 #endif
Index: ehci.c
===
RCS file: /cvs/src/sys/dev/usb/ehci.c,v
retrieving revision 1.135
diff -u -p -r1.135 ehci.c
--- ehci.c  1 Nov 2013 12:00:53 -   1.135
+++ ehci.c  1 Nov 2013 22:18:10 -
@@ -135,9 +135,6 @@ void

Improve routing functions

2013-11-01 Thread Loïc BLOT
Hello @tech
Congratulations for the 5.4 release.

I want to explain a draft to improve a little routing administration on
OpenBSD, maybe for 5.5.

There is a lack on routing daemon, the possibility to change routing
priorities for some protocols.
At this time routing priority is a dedicated constant in the kernel.
In some cases it's useful to change routing priority for a protocol to
make it prior on another.

To resolve this lack, two ways are possible:

1. Each routing daemon manage it's own priority itself, instead of using
kernel priorities, and limits the minimum and maximum value. When we
change the priority (for example bgpctl routing priority 10) all
priorities.
This is easy but a problem appears: the priority can be same as another
running daemon (ospfd for example). How can we know other routing
priorities ?

2. We need to change the utility of routing priority value in the
messages to another thing: routing type.
Then when a daemon register a route, he register route for its type
(BGP/OSPF/RIP/MPLS) and the kernel apply a variable value. 
This value could be modified by sysctl (example sysctl -w
net.routing.ospf_priority = 10)
When we change the priority for a protocol, kernel will search all
routes matching the protocol and apply the priority. Priority conflicts
must be detected by the kernel.

Do you know it's possible ? Is this interesting for future OpenBSD ?

If OpenBSD team is interest i can start a patch in next weeks.

Thanks for reading
-- 
Best regards,
Loïc BLOT, 
UNIX systems, security and network engineer
http://www.unix-experience.fr


signature.asc
Description: This is a digitally signed message part


Re: Improve routing functions

2013-11-01 Thread sven falempin
FreeBSD propose to have a specific routing table for a process, which is
even more powerful.
When the router has multiple gateway i guess when a source address is
choose the route should be chosen given that. Nothing more.

What use of this improvement do you imagine ?, of course you may want
this traffic over this network(low latency) and the other one on
another(high badnwith), put you may use pf for this, or specific route for
the services.

Writing about this make me think you want a route that select on the PORT
instead of the IP. Is this madness ???

route add smtp 1.2.3.4




On Fri, Nov 1, 2013 at 7:46 PM, Loïc BLOT loic.b...@unix-experience.frwrote:

 Hello @tech
 Congratulations for the 5.4 release.

 I want to explain a draft to improve a little routing administration on
 OpenBSD, maybe for 5.5.

 There is a lack on routing daemon, the possibility to change routing
 priorities for some protocols.
 At this time routing priority is a dedicated constant in the kernel.
 In some cases it's useful to change routing priority for a protocol to
 make it prior on another.

 To resolve this lack, two ways are possible:

 1. Each routing daemon manage it's own priority itself, instead of using
 kernel priorities, and limits the minimum and maximum value. When we
 change the priority (for example bgpctl routing priority 10) all
 priorities.
 This is easy but a problem appears: the priority can be same as another
 running daemon (ospfd for example). How can we know other routing
 priorities ?

 2. We need to change the utility of routing priority value in the
 messages to another thing: routing type.
 Then when a daemon register a route, he register route for its type
 (BGP/OSPF/RIP/MPLS) and the kernel apply a variable value.
 This value could be modified by sysctl (example sysctl -w
 net.routing.ospf_priority = 10)
 When we change the priority for a protocol, kernel will search all
 routes matching the protocol and apply the priority. Priority conflicts
 must be detected by the kernel.

 Do you know it's possible ? Is this interesting for future OpenBSD ?

 If OpenBSD team is interest i can start a patch in next weeks.

 Thanks for reading
 --
 Best regards,
 Loïc BLOT,
 UNIX systems, security and network engineer
 http://www.unix-experience.fr




-- 
-
() ascii ribbon campaign - against html e-mail
/\


Re: Improve routing functions

2013-11-01 Thread Loïc BLOT
Hello sven,
it's not a routing table problem, it's only a modification on route
priorities, it's not the same thing.
Here is my example at work:

I have BGP on the WAN, OSPF for my LAN (+ over GRE tunnels) and RIP to
my CISCO catalyst 45XX.

The problem is simple. I have two routers in this configuration. OSPF is
prior on RIP. routes obtained by RIP are redistributed on OSPF (because
my remote sites must know them). But OSPF is prior than RIP and then the
two border routers want to pass by the other instead of using the RIP
route.
I have the same problem with BGP. default route is prior on OSPF than
BGP. Then BGP must be prior on OSPF to don't loop default route between
the two routers.


-- 
Best regards,
Loïc BLOT, 
UNIX systems, security and network engineer
http://www.unix-experience.fr



Le vendredi 01 novembre 2013 à 19:57 -0400, sven falempin a écrit :
 FreeBSD propose to have a specific routing table for a process, which is
 even more powerful.
 When the router has multiple gateway i guess when a source address is
 choose the route should be chosen given that. Nothing more.
 
 What use of this improvement do you imagine ?, of course you may want
 this traffic over this network(low latency) and the other one on
 another(high badnwith), put you may use pf for this, or specific route for
 the services.
 
 Writing about this make me think you want a route that select on the PORT
 instead of the IP. Is this madness ???
 
 route add smtp 1.2.3.4
 
 
 
 
 On Fri, Nov 1, 2013 at 7:46 PM, Loïc BLOT loic.b...@unix-experience.frwrote:
 
  Hello @tech
  Congratulations for the 5.4 release.
 
  I want to explain a draft to improve a little routing administration on
  OpenBSD, maybe for 5.5.
 
  There is a lack on routing daemon, the possibility to change routing
  priorities for some protocols.
  At this time routing priority is a dedicated constant in the kernel.
  In some cases it's useful to change routing priority for a protocol to
  make it prior on another.
 
  To resolve this lack, two ways are possible:
 
  1. Each routing daemon manage it's own priority itself, instead of using
  kernel priorities, and limits the minimum and maximum value. When we
  change the priority (for example bgpctl routing priority 10) all
  priorities.
  This is easy but a problem appears: the priority can be same as another
  running daemon (ospfd for example). How can we know other routing
  priorities ?
 
  2. We need to change the utility of routing priority value in the
  messages to another thing: routing type.
  Then when a daemon register a route, he register route for its type
  (BGP/OSPF/RIP/MPLS) and the kernel apply a variable value.
  This value could be modified by sysctl (example sysctl -w
  net.routing.ospf_priority = 10)
  When we change the priority for a protocol, kernel will search all
  routes matching the protocol and apply the priority. Priority conflicts
  must be detected by the kernel.
 
  Do you know it's possible ? Is this interesting for future OpenBSD ?
 
  If OpenBSD team is interest i can start a patch in next weeks.
 
  Thanks for reading
  --
  Best regards,
  Loïc BLOT,
  UNIX systems, security and network engineer
  http://www.unix-experience.fr
 
 
 
 


signature.asc
Description: This is a digitally signed message part


Re: Improve routing functions

2013-11-01 Thread Chris Cappuccio
Lo?c BLOT [loic.b...@unix-experience.fr] wrote:
 Hello sven,
 it's not a routing table problem, it's only a modification on route
 priorities, it's not the same thing.

The two of you are solving totally different problems.

 Here is my example at work:
 
 I have BGP on the WAN, OSPF for my LAN (+ over GRE tunnels) and RIP to
 my CISCO catalyst 45XX.
 
 The problem is simple. I have two routers in this configuration. OSPF is
 prior on RIP. routes obtained by RIP are redistributed on OSPF (because
 my remote sites must know them). But OSPF is prior than RIP and then the
 two border routers want to pass by the other instead of using the RIP
 route.
 I have the same problem with BGP. default route is prior on OSPF than
 BGP. Then BGP must be prior on OSPF to don't loop default route between
 the two routers.

You don't need a new knob in the system to fix this.

You need to fix your configuration.



fix seekdir(3), was: update perl Module::Build

2013-11-01 Thread Ingo Schwarze
Hi,

Andrew Fresh wrote on Wed, Oct 30, 2013 at 01:50:56PM -0700:

 Post vBSDcon I have most existing patches working with 5.18.1, the only
 failing so far is that with threads enabled

If i understand correctly, we do not currently have threads enabled
in our -current base perl 5.16, so that is not a regression.

So, if that is all that is broken, you are basically claiming
that perl 5.18.1 is ready for commit?   ;-)

 t/op/threads-dirh.t fails.
 (mv patches/{APPLIES,GOOD}/use_threads.patch)
 This test appears to point to a failure that needs fixing but I am not
 skilled enough to know how. 

That problem is neither related to threads nor to 5.18 nor even to
Perl at all!  The problem is that the seekdir(3) in our C library
is badly broken: The readdir(3) function saves dirents in a buffer,
so the lseek(2) done by seekdir(3) only takes effect after that
buffer becomes exhausted.  Otto's nice regression test doesn't catch
that breakage because it only seeks far, far away, never in the
neighbourhood.

The following patch to the C library fixes the issue and lets
the perl 5.18.1 test succeed, even with threads enabled.

Some remarks to explain the patch:
 * To minimally fix the bug, just adding
 dirp-dd_loc = dirp-dd_size;
   to __seekdir() would be sufficient.
   But that would mean that each and every call to seekdir()
   would invalidate the buffer, so some programs might trigger
   getdents(2) after each seekdir(3) and re-read the buffer
   from kernel space over and over again.
   That's inefficient, we should really use the data which is
   already available in user space.
 * dd_loc = 0 is a bad choice to invalidate the buffer;
   it means that one cannot search back to the first entry
   in the buffer without invalidating the whole buffer.
 * While here, i'm garbage collecting some unused headers
   and even some unused functions (__seekdir, _telldir_unlocked),
   and i'm moving back seekdir(3) into the file where it belongs.
   The comment saying it must be in telldir.c to use some opaque
   data structures appears to be a untrue.

OK?
  Ingo


Index: opendir.c
===
RCS file: /cvs/src/lib/libc/gen/opendir.c,v
retrieving revision 1.25
diff -u -p -r1.25 opendir.c
--- opendir.c   6 Oct 2013 17:57:11 -   1.25
+++ opendir.c   2 Nov 2013 03:51:44 -
@@ -111,7 +111,7 @@ __fdopendir(int fd)
return (NULL);
}
 
-   dirp-dd_loc = 0;
+   dirp-dd_loc = -1;
dirp-dd_fd = fd;
dirp-dd_lock = NULL;
dirp-dd_curpos = 0;
Index: readdir.c
===
RCS file: /cvs/src/lib/libc/gen/readdir.c,v
retrieving revision 1.19
diff -u -p -r1.19 readdir.c
--- readdir.c   6 Oct 2013 17:57:54 -   1.19
+++ readdir.c   2 Nov 2013 03:51:44 -
@@ -44,14 +44,15 @@ _readdir_unlocked(DIR *dirp, struct dire
*result = NULL;
for (;;) {
if (dirp-dd_loc = dirp-dd_size)
-   dirp-dd_loc = 0;
-   if (dirp-dd_loc == 0) {
+   dirp-dd_loc = -1;
+   if (dirp-dd_loc == -1) {
dirp-dd_size = getdents(dirp-dd_fd, dirp-dd_buf,
dirp-dd_len);
if (dirp-dd_size == 0)
return (0);
if (dirp-dd_size  0)
return (-1);
+   dirp-dd_loc = 0;
}
dp = (struct dirent *)(dirp-dd_buf + dirp-dd_loc);
if ((long)dp  03 ||/* bogus pointer check */
Index: rewinddir.c
===
RCS file: /cvs/src/lib/libc/gen/rewinddir.c,v
retrieving revision 1.10
diff -u -p -r1.10 rewinddir.c
--- rewinddir.c 13 Aug 2013 05:52:12 -  1.10
+++ rewinddir.c 2 Nov 2013 03:51:44 -
@@ -1,5 +1,5 @@
 /* $OpenBSD: rewinddir.c,v 1.10 2013/08/13 05:52:12 guenther Exp $ */
-/*-
+/*
  * Copyright (c) 1990, 1993
  * The Regents of the University of California.  All rights reserved.
  *
@@ -28,16 +28,12 @@
  * SUCH DAMAGE.
  */
 
-#include sys/types.h
 #include dirent.h
-
 #include thread_private.h
 #include telldir.h
 
 void
 rewinddir(DIR *dirp)
 {
-   _MUTEX_LOCK(dirp-dd_lock);
-   __seekdir(dirp, 0);
-   _MUTEX_UNLOCK(dirp-dd_lock);
+   seekdir(dirp, 0);
 }
Index: seekdir.c
===
RCS file: /cvs/src/lib/libc/gen/seekdir.c,v
retrieving revision 1.9
diff -u -p -r1.9 seekdir.c
--- seekdir.c   5 Jun 2007 18:11:48 -   1.9
+++ seekdir.c   2 Nov 2013 03:51:45 -
@@ -1,46 +1,69 @@
 /* $OpenBSD: seekdir.c,v 1.9 2007/06/05 18:11:48 kurt Exp $ */
 /*
- * Copyright (c) 1983, 1993
- * The Regents of the University of California.  All rights reserved.
+ * Copyright (c) 2103 Ingo Schwarze schwa...@openbsd.org
  *
- * Redistribution 

Re: fix seekdir(3), was: update perl Module::Build

2013-11-01 Thread Andrew Fresh
On Sat, Nov 02, 2013 at 05:27:38AM +0100, Ingo Schwarze wrote:
 Andrew Fresh wrote on Wed, Oct 30, 2013 at 01:50:56PM -0700:
 
  Post vBSDcon I have most existing patches working with 5.18.1, the only
  failing so far is that with threads enabled
 
 If i understand correctly, we do not currently have threads enabled
 in our -current base perl 5.16, so that is not a regression.

You are correct, I had threads enabled in the github repo and forgot
that it hadn't actually made it in-tree.


 So, if that is all that is broken, you are basically claiming
 that perl 5.18.1 is ready for commit?   ;-)

It seems close, I'd like to see some test reports on other
architectures, but since the lack of threads isn't actually a problem, I
will set up some of the other arch's that I have and see about testing
it on those.  


I also do want to actually spend more time looking and trying to
understand the diff between 5.16.3 and 5.18.1 to make sure there wasn't
anything introduced that I had questions about.  Perhaps helping unload
a moving truck this weekend will take less time than I expect.


  t/op/threads-dirh.t fails.
  (mv patches/{APPLIES,GOOD}/use_threads.patch)
  This test appears to point to a failure that needs fixing but I am not
  skilled enough to know how. 
 
 That problem is neither related to threads nor to 5.18 nor even to
 Perl at all!  The problem is that the seekdir(3) in our C library
 is badly broken: The readdir(3) function saves dirents in a buffer,
 so the lseek(2) done by seekdir(3) only takes effect after that
 buffer becomes exhausted.  Otto's nice regression test doesn't catch
 that breakage because it only seeks far, far away, never in the
 neighbourhood.

This is amazing, thank you much, I will have to drink some extra coffee
and try to understand that as well. 

l8rZ,
-- 
andrew - http://afresh1.com

The 3 great virtues of a programmer: Laziness, Impatience, and Hubris.
  --Larry Wall