Re: malloc freelists

2014-05-01 Thread Marc Espie
Okay, the question is: why 4 ? why not 3 ? or 2 ? or 8 ?
Where do you stop ? how did you figure out that 4 was better ?

This looks a bit like hey, let's make our own crypto code, it ought to
work just fine, right ?



Re: malloc freelists

2014-05-01 Thread Marc Espie
Sorry, badly phrased reply. I didn't mean to imply it was a bad idea, but
you didn't explain at all why 4, and not 3 or 6, or 42 ?  If it's good with
4, it ought to be better with more, right ? any data point or rationale for
choosing 4 ?



Re: libc: #define to remove support for %n from printf(3)?

2014-05-03 Thread Marc Espie
On Sat, May 03, 2014 at 11:28:16AM +0200, Reyk Floeter wrote:
 On Fri, May 02, 2014 at 06:50:04PM -0600, Bob Beck wrote:
  What's their hangup with %n?  We normally don't like polluting the world
  with #ifdef OPENSSL_NO_PERCENT_N...  We normally nuke stuff like that
 
 Well, it is an evil thing that is rarely used and well-known for some
 format string vulnerabilities.  So either adding this #define or
 removing it from our libc wouldn't be a bad idea.  But it is in POSIX,
 right?
 
 At a first look, I only found it in a few places of our tree where it
 could be removed:
 
   gcc, texinfo, curses, libform, sqlite, less, tmux, unbound, nsd
 
 Not couting ports, but some software already seems to check if %n is
 available to do a fallback otherwise.
 
 Reyk

And you do expect the fallback to be of good quality ?

I think that the same rationale that lead to including some crypto code
in openssl applies there: if we don't provide it, people will roll their
own posixy-printf, and we know how well that turns out in general...


(including stuff that peeks and duplicates the internals of FILE, like
I'll be happy if I *never* get to fix this kind of code again).



Re: libc: #define to remove support for %n from printf(3)?

2014-05-03 Thread Marc Espie
On Sat, May 03, 2014 at 01:00:28PM +0200, Henning Brauer wrote:
 * Marc Espie es...@nerim.net [2014-05-03 11:50]:
  On Sat, May 03, 2014 at 11:28:16AM +0200, Reyk Floeter wrote:
   On Fri, May 02, 2014 at 06:50:04PM -0600, Bob Beck wrote:
What's their hangup with %n?  We normally don't like polluting the world
with #ifdef OPENSSL_NO_PERCENT_N...  We normally nuke stuff like that
   Well, it is an evil thing that is rarely used and well-known for some
   format string vulnerabilities.  So either adding this #define or
   removing it from our libc wouldn't be a bad idea.  But it is in POSIX,
   right?
   At a first look, I only found it in a few places of our tree where it
   could be removed:
   
 gcc, texinfo, curses, libform, sqlite, less, tmux, unbound, nsd
   
   Not couting ports, but some software already seems to check if %n is
   available to do a fallback otherwise.
  And you do expect the fallback to be of good quality ?
  
  I think that the same rationale that lead to including some crypto code
  in openssl applies there: if we don't provide it, people will roll their
  own posixy-printf, and we know how well that turns out in general...
  
  (including stuff that peeks and duplicates the internals of FILE, like
  I'll be happy if I *never* get to fix this kind of code again).

 both of you are kinda missing the point. there is no argument here to
 remove %n support from our libc, the point is wether we can add a
 #define to allow people compiling themselves (probably not as part of
 OpenBSD) to remove it without having to change the code.
 
 And since that's not intrusive and doesn't create a portability mess
 like the one we're dealing with in libssl right now, I don't see a
 problem with that.

No, I was just responding to Reyk, who apparently wanted to take it further.

As long as the #ifdef doesn't turn into yet another quagmire, it's obviously
okay to have it (coming after the commit, of course).



Re: malloc in libssl/src/apps

2014-05-04 Thread Marc Espie
On Sun, May 04, 2014 at 12:21:04AM -0700, patrick keshishian wrote:
 why not use calloc(2)?

Because it doesn't exist ?

(hint: the 2 in calloc(2) is the manual section. There is no calloc system
call, therefore you can't be right. See guenther(2) for a more serious answer).



Re: Support for LC_TIME

2014-05-08 Thread Marc Espie
On Thu, May 08, 2014 at 12:07:30PM +0200, Stefan Sperling wrote:
 On Wed, May 07, 2014 at 07:44:51PM +0200, Ingo Schwarze wrote:
  While LC_CTYPE and LC_COLLATE make some sense, LC_MONETARY, LC_NUMERIC,
  and LC_TIME are badly overengineered, pointless bloat, causing nothing
  but surprising, erratic behaviour and portability problems when trying
  to parse output from programs.  I think this should be rejected outright
  and you should stop wasting your time on it.
 
 They make sense for systems that try to provide full i18n.
 Of course, we don't try to provide i18n, at least not for the base system
 which is English only. So they don't really make sense *for OpenBSD*.

???

Basic support for that stuff makes sense, as part of a *full* libc.
Not surprisingly, Antoine is for providing LC_* support. So am I.

This has little to do with base OpenBSD, everything to do with enough
stuff to be able to compile reasonable portable software on OpenBSD 
without needing to patch left and right.

As for portability issues: programs stay with the C locale *in any case*
unless they do setlocale()   right at the start, in which case they
explicitly say yes, I want to be localized. So, from that point of view
the portability issues are minimal (yes, I'm aware of the can of worms
that threads+locale may open).


 That said, I don't have a general problem with adding other locale categories.
 I believe LC_TIME would provide a useful testbed for eventually switching all 
 our
 locales to the localedef format (including LC_CTYPE). Alas, the proposed diff
 does something else, and unfortunately I don't have enough time for a detailed
 rabbit hole discussion and review with a lot of back-and-forth that we had 
 when
 discussing similar diffs in the past.

THAT on the other hand is the issue at hand... chronic time shortage to be
certain that what we do for locales isn't dangerous...



pkg_add: extract out-of-order archives

2014-06-12 Thread Marc Espie
I already sent a preliminary version of this patch to selected people.
Now that packages use long names when needed, we can pinpoint every file
in the archive based on its archived name.

This patch changes pkg_add to be able to extract archives that contain
elements out-of-order, e.g., match package contents to plist in whatever
order. Once the temporary elements are extracted, the actual extract
routine will put everything in place, run scripts, etc.

This is really important. Test/read/report failures to me.

Once this is in, there is another patch that will allow pkg_create to
create out-of-order archives.

The neat effect will be to speed up updates, by relegating unchanged files
to the end of the archive, so they don't need to be fetched at all.

the effect should be drastic on monsters like texlive, where most of 
the files do not change at all from version to version.

Index: OpenBSD/Add.pm
===
RCS file: /cvs/src/usr.sbin/pkg_add/OpenBSD/Add.pm,v
retrieving revision 1.153
diff -u -p -r1.153 Add.pm
--- OpenBSD/Add.pm  20 May 2014 05:55:43 -  1.153
+++ OpenBSD/Add.pm  12 Jun 2014 09:50:44 -
@@ -113,7 +113,33 @@ sub perform_extraction
$state-{partial} = $handle-{partial};
$state-{archive} = $handle-{location};
$state-{check_digest} = $handle-{plist}{check_digest};
-   $state-progress-visit_with_size($handle-{plist}, 'extract', $state);
+   my ($wanted, $tied) = {};
+   $handle-{plist}-find_extractible($state, $wanted, $tied);
+   my $p = $state-progress-new_sizer($handle-{plist}, $state);
+   while (my $file = $state-{archive}-next) {
+   my $e = $tied-{$file-name};
+   if (defined $e) {
+   delete $tied-{$file-name};
+   $p-advance($e);
+   $state-{archive}-skip;
+   # skip to next;
+   }
+   $e = $wanted-{$file-name};
+   if (!defined $e) {
+   $state-Fatal(archive member not found #1,
+   $file-name);
+   }
+   delete $wanted-{$file-name};
+   $e-prepare_to_extract($state, $file);
+   $e-extract($state, $file);
+   $p-advance($e);
+   if (keys %$wanted == 0) {
+   return;
+   }
+   }
+   if (keys %$wanted  0) {
+   $state-fatal(Truncated archive);
+   }
 }
 
 my $user_tagged = {};
@@ -181,6 +207,10 @@ sub prepare_for_addition
 {
 }
 
+sub find_extractible
+{
+}
+
 sub extract
 {
my ($self, $state) = @_;
@@ -236,6 +266,16 @@ sub set_modes
}
 }
 
+package OpenBSD::PackingElement::Meta;
+
+# XXX stuff that's invisible to find_extractible should be considered extracted
+# for the most part, otherwise we create broken partial packages
+sub find_extractible
+{
+   my ($self, $state, $wanted, $tied) = @_;
+   $state-{partial}{$self} = 1;
+}
+
 package OpenBSD::PackingElement::ExtraInfo;
 use OpenBSD::Error;
 
@@ -360,6 +400,18 @@ use File::Basename;
 use File::Path;
 use OpenBSD::Temp;
 
+sub find_extractible
+{
+   my ($self, $state, $wanted, $tied) = @_;
+   $wanted-{$self-name} = $self;
+   return;
+   if ($self-{tied}) {
+   $tied-{$self-name} = $self;
+   } else {
+   $wanted-{$self-name} = $self;
+   }
+}
+
 sub prepare_for_addition
 {
my ($self, $state, $pkgname) = @_;
@@ -384,19 +436,11 @@ sub prepare_for_addition
 
 sub prepare_to_extract
 {
-   my ($self, $state) = @_;
+   my ($self, $state, $file) = @_;
my $fullname = $self-fullname;
my $destdir = $state-{destdir};
 
-   my $file=$state-{archive}-next;
-   if (!defined $file) {
-   $state-fatal(truncated archive);
-   }
$file-{cwd} = $self-cwd;
-   if (!$file-check_name($self)) {
-   $state-fatal(archive does not match #1 != #2,
-   $file-name, $self-name);
-   }
if (defined $self-{symlink} || $file-isSymLink) {
unless (defined $self-{symlink}  $file-isSymLink) {
$state-fatal(bogus symlink #1, $self-name);
@@ -427,14 +471,11 @@ sub prepare_to_extract
if (defined $self-{symlink}  $state-{do_faked}) {
$file-{linkname} = $destdir.$file-{linkname};
}
-   return $file;
 }
 
 sub extract
 {
-   my ($self, $state) = @_;
-
-   my $file = $self-prepare_to_extract($state);
+   my ($self, $state, $file) = @_;
 
if (defined $self-{link} || defined $self-{symlink}) {
$state-{archive}-skip;
@@ -483,6 +524,7 @@ sub extract
 
$state-say(extracting #1, $tempname) if $state-verbose = 3;
 
+
if (!$file-isFile) {
$state-fatal(can't extract #1, it's not a file, 

newer patch Re: pkg_add: extract out-of-order archives

2014-06-14 Thread Marc Espie
Tweaked version. The previous version was working just fine, but this
one also marks hardlinks and symlinks as skippable, as should be.
It also adjusts the pace of progress slightly (making the meter jump AFTER
skipping the entry), and also it does prepare_to_extract systematically
(which will recheck links vs their packing-list entry for consistency)

I haven't seen any negative report from the previous version.

If this one does work too, I intend to commit it sometimes during the week.

Index: OpenBSD/Add.pm
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/pkg_add/OpenBSD/Add.pm,v
retrieving revision 1.153
diff -u -p -r1.153 Add.pm
--- OpenBSD/Add.pm  20 May 2014 05:55:43 -  1.153
+++ OpenBSD/Add.pm  13 Jun 2014 17:14:05 -
@@ -113,7 +113,34 @@ sub perform_extraction
$state-{partial} = $handle-{partial};
$state-{archive} = $handle-{location};
$state-{check_digest} = $handle-{plist}{check_digest};
-   $state-progress-visit_with_size($handle-{plist}, 'extract', $state);
+   my ($wanted, $tied) = {};
+   $handle-{plist}-find_extractible($state, $wanted, $tied);
+   my $p = $state-progress-new_sizer($handle-{plist}, $state);
+   while (my $file = $state-{archive}-next) {
+   my $e = $tied-{$file-name};
+   if (defined $e) {
+   delete $tied-{$file-name};
+   $e-prepare_to_extract($state, $file);
+   $state-{archive}-skip;
+   $p-advance($e);
+   # skip to next;
+   }
+   $e = $wanted-{$file-name};
+   if (!defined $e) {
+   $state-Fatal(archive member not found #1,
+   $file-name);
+   }
+   delete $wanted-{$file-name};
+   $e-prepare_to_extract($state, $file);
+   $e-extract($state, $file);
+   $p-advance($e);
+   if (keys %$wanted == 0) {
+   return;
+   }
+   }
+   if (keys %$wanted  0) {
+   $state-fatal(Truncated archive);
+   }
 }
 
 my $user_tagged = {};
@@ -181,6 +208,10 @@ sub prepare_for_addition
 {
 }
 
+sub find_extractible
+{
+}
+
 sub extract
 {
my ($self, $state) = @_;
@@ -236,6 +267,16 @@ sub set_modes
}
 }
 
+package OpenBSD::PackingElement::Meta;
+
+# XXX stuff that's invisible to find_extractible should be considered extracted
+# for the most part, otherwise we create broken partial packages
+sub find_extractible
+{
+   my ($self, $state, $wanted, $tied) = @_;
+   $state-{partial}{$self} = 1;
+}
+
 package OpenBSD::PackingElement::ExtraInfo;
 use OpenBSD::Error;
 
@@ -360,6 +401,18 @@ use File::Basename;
 use File::Path;
 use OpenBSD::Temp;
 
+sub find_extractible
+{
+   my ($self, $state, $wanted, $tied) = @_;
+   $wanted-{$self-name} = $self;
+   return;
+   if ($self-{tied} || $self-{link} || $self-{symlink}) {
+   $tied-{$self-name} = $self;
+   } else {
+   $wanted-{$self-name} = $self;
+   }
+}
+
 sub prepare_for_addition
 {
my ($self, $state, $pkgname) = @_;
@@ -384,19 +437,11 @@ sub prepare_for_addition
 
 sub prepare_to_extract
 {
-   my ($self, $state) = @_;
+   my ($self, $state, $file) = @_;
my $fullname = $self-fullname;
my $destdir = $state-{destdir};
 
-   my $file=$state-{archive}-next;
-   if (!defined $file) {
-   $state-fatal(truncated archive);
-   }
$file-{cwd} = $self-cwd;
-   if (!$file-check_name($self)) {
-   $state-fatal(archive does not match #1 != #2,
-   $file-name, $self-name);
-   }
if (defined $self-{symlink} || $file-isSymLink) {
unless (defined $self-{symlink}  $file-isSymLink) {
$state-fatal(bogus symlink #1, $self-name);
@@ -427,14 +472,11 @@ sub prepare_to_extract
if (defined $self-{symlink}  $state-{do_faked}) {
$file-{linkname} = $destdir.$file-{linkname};
}
-   return $file;
 }
 
 sub extract
 {
-   my ($self, $state) = @_;
-
-   my $file = $self-prepare_to_extract($state);
+   my ($self, $state, $file) = @_;
 
if (defined $self-{link} || defined $self-{symlink}) {
$state-{archive}-skip;
@@ -483,6 +525,7 @@ sub extract
 
$state-say(extracting #1, $tempname) if $state-verbose = 3;
 
+
if (!$file-isFile) {
$state-fatal(can't extract #1, it's not a file, 
$self-stringize);
@@ -558,6 +601,10 @@ sub prepare_for_addition
if ($s-avail  0) {
$s-report_overflow($state, $fname);
}
+}
+
+sub find_extractible
+{
 }
 
 sub extract
Index: OpenBSD/ProgressMeter.pm

new package format

2014-07-01 Thread Marc Espie
I've just switched on out-of-order packages, after much testing.

What this means:

new packages won't be compatible with older pkg_add. Most specifically,
the plist order may no longer match the packing-list.

- if you see strange pkg_add errors, and your base system is not uptodate,
that's your fault.

New bulks will exploit that feature: build machines log an history of
files in a package. With out-of-order archives, they can take advantage
of that to produce packages where the most recently changed files are
at the front.

For some packages, this can be a drastic performance improvement: some
measurements show that as much as half of some packages do not change
over a 2 months period.

Note that this will speed up dependency, maintainer and other meta info
updates even more, as the package extraction may stop right after the meta
information...



Re: using -Werror-implicit-function-declaration

2014-07-11 Thread Marc Espie
On Fri, Jul 11, 2014 at 10:27:57AM -0400, Ted Unangst wrote:
 On Fri, Jul 11, 2014 at 14:09, Florian Obser wrote:
  usr.sbin, make sure subdirs of subdirs see usr.sbin/Makefile.inc:
  
  Does the same thing as lpr/pac and pppd/pppstats. I have no idea if
  this is the right way, but it seems to work. Cluebats welcome.
 
 I think maybe it would be nicer to create a Makefile.inc in .. and
 have that include ../../Makefile.inc in some cases? That would work
 more automagically?

Avoiding multiple inclusions is somewhat hard...



Re: PATCH: misc mkstemp and fdopen fixes

2014-07-11 Thread Marc Espie
On Fri, Jul 11, 2014 at 04:55:36PM +, Doug Hogan wrote:
 Index: usr.bin/m4/eval.c
 ===
 RCS file: /cvs/src/usr.bin/m4/eval.c,v
 retrieving revision 1.72
 diff -u -p -d -r1.72 eval.c
 --- usr.bin/m4/eval.c 28 Apr 2014 12:34:11 -  1.72
 +++ usr.bin/m4/eval.c 11 Jul 2014 16:20:25 -
 @@ -818,8 +818,12 @@ dodiv(int n)
   char fname[] = _PATH_DIVNAME;
  
   if ((fd = mkstemp(fname))  0 || 
 - (outfile[n] = fdopen(fd, w+)) == NULL)
 - err(1, %s: cannot divert, fname);
 + (outfile[n] = fdopen(fd, w+)) == NULL) {
 + int saved_errno = errno;
 + if (fd != -1)
 + unlink(fname);
 + errc(1, saved_errno, %s: cannot divert, 
 fname);
 + }
   if (unlink(fname) == -1)
   err(1, %s: cannot unlink, fname);
   }
I don't like that part. The logic is a bit wrong. Especially since 
unlink(fname) is always called for fd != -1, so I feel there should be one
single call.



Re: First release of LibreSSL portable is available.

2014-07-12 Thread Marc Espie
On Fri, Jul 11, 2014 at 11:07:10PM +, Miod Vallat wrote:
  it. As expected, OPENSSL does the opposite and makes life harder for
  everyone.
 
 Hasn't this been the OpenSSL roadmap since the very beginning?

Jury is still out as whether they did it on purpose, or whether it was
just a side-effect of the lack of funding, or whether they did it on
purpose to later complain about the lack of funding.



HEADS-UP: faster mirrors synching coming your way soon

2014-09-22 Thread Marc Espie
I've just committed changes to pkg_create that will help mirrors
synch by using much less bandwidth.

I just ran a final test. Rsynching a full amd64 snapshot now says something
like:
sent 7,315,796,510 bytes  received 40,292,721 bytes  4,517,095.01 bytes/sec
total size is 28,752,806,019  speedup is 3.91

A few months ago, after the reorder files in packages, Stuart
Henderson commented that this would not help mirrors, but just the
end user, which got me thinking...

(Reminder: archives are compressed files. rsync does not peek inside the
compressed data, so its comparison algorithms don't work so well with them,
as the first different byte will change everything for the rest of the
archive, so no speed-up for compressed files).


I looked at the --rsyncable patch for zlib/gzip, and talked it over
with sthen@ and millert@, but pretty soon we discarded that idea.
That patch is brittle (every zlib version has got its own flavor of it,
with wild differences) and a nightmare to maintain. Plus it won't work
at all with other compression formats.

The solution was low-tech: simply cut the archive into more gzip chunks
(signatures already split the package into two parts, so we know the
tools work).  I chose 16 files as a simple guideline to experiment with.
There were still some discrepancies, such as tar timestamps metadata, which
is why those migrated to the plist a few weeks ago (side-effect: the tarball
effectively says everything dates back to the epoch... not so bad).

I was pleasantly surprised: the size increase is minimal (very much 
under 1%).

I also wacked on gzip timestamps, which don't serve any useful purpose 
either, especially since the plist signature also contains a timestamp (and
that one is signed, so it's ways more trustworthy).

Obviously, the first snapshot out will still copy everything. But from the
second one, mirror owners should see a difference.

To benefit:
- mirror owners must now use rsync algorithms. Turn off -W / --whole-file
if you were using it.
- turn on -y / --fuzzy, as this will track minor package version changes.

Note that this only applies to the package snapshots part of OpenBSD.

My test was a bit extreme: I did build two snaps with the exact same ports
tree, so the similarities are maximal. Nevertheless, there are lots of
*huge* packages in the ports tree.  So I expect the bandwidth gain to
be very significant anyway, especially for fast architectures which turn
up one snapshot a week or more. e.g., bandwidth use should be more than
halved, I expect.



Re: HEADS-UP: faster mirrors synching coming your way soon

2014-09-23 Thread Marc Espie
On Mon, Sep 22, 2014 at 09:42:58AM +0200, Marc Espie wrote:
 I've just committed changes to pkg_create that will help mirrors
 synch by using much less bandwidth.
 
 I just ran a final test. Rsynching a full amd64 snapshot now says something
 like:
 sent 7,315,796,510 bytes  received 40,292,721 bytes  4,517,095.01 bytes/sec
 total size is 28,752,806,019  speedup is 3.91
 My test was a bit extreme: I did build two snaps with the exact same ports
 tree, so the similarities are maximal. Nevertheless, there are lots of
 *huge* packages in the ports tree.  So I expect the bandwidth gain to
 be very significant anyway, especially for fast architectures which turn
 up one snapshot a week or more. e.g., bandwidth use should be more than
 halved, I expect.

I got a newer snaps, so an actual test on real data. Snaps apart by about
1 week.

sent 7,502,610,665 bytes received 40,313,605 bytes 4,561,792.72 bytes/sec
total size is 28,788,092,671 speedup is 3.82


Still very good. 



Re: Conditional include in make(1)

2014-10-19 Thread Marc Espie
On Sun, Oct 19, 2014 at 09:25:51AM +0200, Matthieu Herrb wrote:
 On Thu, Oct 16, 2014 at 08:14:16PM +0200, Mark Kettenis wrote:
  Hi Marc,
  
  Is there a reason why conditional includes (sinclude/-include) aren't
  enabled in OpenBSD?
  
  I'm asking because the Xorg people now use it in one of the xserver
  Makefile.  We could of course try to convince them to revert the
  change they made.  But it is a somewhat useful feature.
 
 So, now that espie@ has committed the bit, ok to remove the local
 change that made the inclusion unconditionnal (and slightly broke make
 clean in xserver) ?
 
 Index: Makefile.am
 ===
 RCS file: /cvs/OpenBSD/xenocara/xserver/hw/xfree86/Makefile.am,v
 retrieving revision 1.9
 diff -u -r1.9 Makefile.am
 --- Makefile.am   27 Sep 2014 17:53:01 -  1.9
 +++ Makefile.am   19 Oct 2014 07:23:22 -
 @@ -138,7 +138,7 @@
   $(AM_V_GEN)CPP='$(CPP)' AWK='$(AWK)' $(SHELL) $(srcdir)/sdksyms.sh 
 $(top_srcdir) $(CFLAGS) $(AM_CFLAGS) $(AM_CPPFLAGS)
  
  SDKSYMS_DEP = sdksyms.dep
 -include $(SDKSYMS_DEP)
 +-include $(SDKSYMS_DEP)
  
  i2c/libi2c.la:
   $(AM_V_at)cd i2c  $(MAKE) libi2c.la
 Index: Makefile.in
 ===
 RCS file: /cvs/OpenBSD/xenocara/xserver/hw/xfree86/Makefile.in,v
 retrieving revision 1.29
 diff -u -r1.29 Makefile.in
 --- Makefile.in   27 Sep 2014 17:53:01 -  1.29
 +++ Makefile.in   19 Oct 2014 07:23:22 -
 @@ -1116,7 +1116,7 @@
  
  sdksyms.dep sdksyms.c: sdksyms.sh
   $(AM_V_GEN)CPP='$(CPP)' AWK='$(AWK)' $(SHELL) $(srcdir)/sdksyms.sh 
 $(top_srcdir) $(CFLAGS) $(AM_CFLAGS) $(AM_CPPFLAGS)
 -include $(SDKSYMS_DEP)
 +-include $(SDKSYMS_DEP)
  
  i2c/libi2c.la:
   $(AM_V_at)cd i2c  $(MAKE) libi2c.la
 
 -- 
Wait for a bit until new snaps have the new make, then sure.



Re: help X11 performance: make sigprocmask(2) SY_NOLOCK

2013-06-19 Thread Marc Espie
On Wed, Jun 19, 2013 at 01:40:19PM +0200, Martin Pelikan wrote:
  If you're right that atomic_{clear,set}bits_int is correct and
  sufficient and actually faster, then all dynamic executables would
  benefit from this speedup (sigprocmask is used in ld.so(1)).
 
 Since on i386 GENERIC these atomic_* things don't emit the LOCK prefix,
 performance shouldn't be an issue; I'm actually more worried about this bit:
 
   - p-p_sigmask = mask ~ sigcantmask;
   + p-p_sigmask = mask;
 
 On the right architecture where a word store isn't atomic enough and
 with the right compiler that decides to put p_sigmask on an address
 ending with 0xFFF with 4k-sized pages, we have two problems already.

If I read the code correctly, just adding an alignment constraint
to pool_init for the proc_pool would fix that 2nd part.



Re: binary integer constants in gcc

2013-06-21 Thread Marc Espie
On Fri, Jun 21, 2013 at 11:03:16AM +0200, Mark Kettenis wrote:
  Date: Fri, 21 Jun 2013 10:50:42 +0200
  From: Landry Breuil lan...@rhaalovely.net
  
  On Fri, Jun 21, 2013 at 10:20:01AM +0200, Mark Kettenis wrote:
   
   Well, lots of ports stuff is compiled with newer gcc versions anyway.
  
  Actually, not so many:
  
  $echo select count(*) from modules where value='gcc4'; | 
  sqlite3/usr/local/share/sqlports 
  34
  
  And if you rip out all the subpackages, the actual list is:
  
  audio/mscore
  editors/libreoffice
  lang/classpath
  lang/luajit
  net/rtorrent
  print/cups-filters
  textproc/pdftk
  www/mozilla-firefox
  www/seamonkey
  www/squid
 
 Don't libreoffice and mozilla-firefox account for about half the lines
 of code that's in ports? ;)
 
 Seriously though; I was under the impression that it was a lot more.
 Thanks for enlightening me Landry.

Some of the big ones have moved to llvm (chromium).
In actuality, as you know, we're playing linking games, since we mix
both libstdc++ from base and libstdc++ from the ports gcc4.

The only clean option would be to move all the ports that want C++ to
using a single compiler... or to a single libstdc++/libcxxrt...



Re: [Fwd: HTTP2 script problem]

2013-06-22 Thread Marc Espie
On Sat, Jun 22, 2013 at 12:06:51PM -0700, Ryan Freeman wrote:
 On Sat, Jun 22, 2013 at 07:00:09PM +0200, Max Power wrote:
  Hi guys!
  
  OpenBSD 5.3/amd64:
  
  pkg_add apache-httpd [ok.]
  
  next step
  /etc/rc.d/httpd2 start
  returns:
  httpd2(failed)
  
  Instead
  /usr/local/sbin/apachectl2 start
  It works and load Apache2.
  
  Why?
 
 Because you keep wasting time emailing a technical list for
 improvements and patches, VERY SPECIFICALLY NOTED on the page where
 this address resides.
 
 If you spent any time at all reading manual pages, you would have all
 the questions you have asked answered.  STOP POSTING THESE QUESTIONS
 TO THIS LIST.  m...@openbsd.org is the ONLY list you should be posting
 to, based on all the questions I have seen from you so far.
 
 -ryan

I will also say that, eventually, we will ask the list caretaker to BAN
YOU FROM TECH@.

GO TO MISC@

You've been warned three times.

NOOB QUESTIONS *DO NOT BELONG* ON TECH@.



Re: Update elf(5)

2013-07-09 Thread Marc Espie
On Mon, Jul 08, 2013 at 11:36:14PM -0700, Matthew Dempsky wrote:
 Might be nice to mention that 5.4 'was' the first release that used
 ELF on all platforms?


Starting
.Ox 5.4 ,
all supported platforms use it as the native binary file format.

 On Mon, Jul 8, 2013 at 11:16 PM, Simon Kuhnle si...@blarzwurst.de wrote:
  With VAX being an ELF platform, this is no longer true, is it?
 
  Regards
  Simon
 
  Index: elf.5
  ===
  RCS file: /cvs/src/share/man/man5/elf.5,v
  retrieving revision 1.20
  diff -u -r1.20 elf.5
  --- elf.5   17 Jan 2013 21:54:18 -  1.20
  +++ elf.5   9 Jul 2013 06:14:54 -
  @@ -1355,9 +1355,7 @@
   .Sh HISTORY
   .Ox
   ELF support first appeared in
  -.Ox 1.2 ,
  -although not all supported platforms use it as the native
  -binary file format.
  +.Ox 1.2 .
   ELF in itself first appeared in
   .At V .
   The ELF format is an adopted standard.
 



Re: Update elf(5)

2013-07-09 Thread Marc Espie
On Tue, Jul 09, 2013 at 06:09:02PM +0200, Simon Kuhnle wrote:
 On Tue, Jul 09, 2013 at 12:29:10AM -0700, Matthew Dempsky wrote:
  On Tue, Jul 9, 2013 at 12:22 AM, Marc Espie es...@nerim.net wrote:
   Starting
   .Ox 5.4 ,
   all supported platforms use it as the native binary file format.
  
  I think Starting *with* .Ox 5.4, ... but otherwise ok matthew.
 
 Updated with your suggestions.
 
 Regards
   Simon
 
 Index: elf.5
 ===
 RCS file: /cvs/src/share/man/man5/elf.5,v
 retrieving revision 1.20
 diff -u -r1.20 elf.5
 --- elf.5 17 Jan 2013 21:54:18 -  1.20
 +++ elf.5 9 Jul 2013 16:07:27 -
 @@ -1355,9 +1355,10 @@
  .Sh HISTORY
  .Ox
  ELF support first appeared in
 -.Ox 1.2 ,
 -although not all supported platforms use it as the native
 -binary file format.
 +.Ox 1.2 .
 +Starting with
 +.Ox 5.4 ,
 +all supported platforms use it as the native binary file format.
  ELF in itself first appeared in
  .At V .
  The ELF format is an adopted standard.
Okay



Re: arp rarpd ndp warnings

2013-07-19 Thread Marc Espie
On Sat, Jul 20, 2013 at 12:40:47AM +0200, Alexander Bluhm wrote:
 Hi,
 
 Enable gcc warnings for arp, rarpd, ndp and fix findings.
 
 ok?
 
 bluhm
 
 Index: usr.sbin/arp/Makefile
 ===
 RCS file: /data/mirror/openbsd/cvs/src/usr.sbin/arp/Makefile,v
 retrieving revision 1.3
 diff -u -p -u -p -r1.3 Makefile
 --- usr.sbin/arp/Makefile 29 Mar 2005 23:56:30 -  1.3
 +++ usr.sbin/arp/Makefile 19 Jul 2013 22:35:30 -
 @@ -2,6 +2,7 @@
  
  PROG=arp
  MAN= arp.4 arp.8
 -CFLAGS+= -Wall
 +
 +WARNINGS= Yes
  
I may be wrong, but I think the consensus was that WARNINGS was something
you turned on when you wanted to check things, but that was too expensive
in terms of compiler speed, especially on slow slow architectures ?



Re: Inline assembly is hard...

2013-08-06 Thread Marc Espie
On Tue, Aug 06, 2013 at 12:37:49AM +0200, Mark Kettenis wrote:
 The atomic_setbits_int() and atomic_clearbits_int() implementations
 use stwcx., which touches the condition code register.  This means we
 need to tell GCC that it gets clobbered.  Fixes issues with
 uvm_page_physload() which currently gets miscompiled.
 
 ok?
 
 
 Index: atomic.h
 ===
 RCS file: /cvs/src/sys/arch/powerpc/include/atomic.h,v
 retrieving revision 1.4
 diff -u -p -r1.4 atomic.h
 --- atomic.h  23 Mar 2011 16:54:36 -  1.4
 +++ atomic.h  5 Aug 2013 22:34:00 -
 @@ -17,7 +17,7 @@ atomic_setbits_int(__volatile unsigned i
  or  %0, %1, %0  \n
  stwcx.  %0, 0, %2   \n
  bne-1b  \n
 -sync : =r (tmp) : r (v), r (uip) : memory);
 +sync : =r (tmp) : r (v), r (uip) : cc, memory);
  }
  
  static __inline void
 @@ -30,7 +30,7 @@ atomic_clearbits_int(__volatile unsigned
  andc%0, %0, %1  \n
  stwcx.  %0, 0, %2   \n
  bne-1b  \n
 -sync : =r (tmp) : r (v), r (uip) : memory);
 +sync : =r (tmp) : r (v), r (uip) : cc, memory);
  }
  
  #endif /* defined(_KERNEL) */
Obviously okay.



Re: Iso image integrity verification

2013-09-11 Thread Marc Espie
On Wed, Sep 11, 2013 at 05:36:45PM +0300, Valentin Zagura wrote:
 Thanks for the suggestion, we will probably order the CD.
 
 But on the other hand, I hope that you realize that people in some
 countries (Iran, China, Egypt, Syria) would not have this possibility and
 they could be more affected by a compromise than we would be (they might
 probably pay with their lives) and I hope you guys are also thinking of
 them.
 
 Thanks,
 Valentin Zagura

Do your homework. There are specifically companies that deal with OpenBSD
in such countries, most specially the ones who can't deal with the US
because of embargoes...



Re: more /dev/ugen*

2013-09-13 Thread Marc Espie
In general, when we don't have enough of a device, we end up with clonable
shit or something don't we ?



Re: more /dev/ugen*

2013-09-13 Thread Marc Espie
On Fri, Sep 13, 2013 at 07:24:27PM +0200, Mark Kettenis wrote:
  Date: Fri, 13 Sep 2013 17:55:17 +0200
  From: Marc Espie es...@nerim.net
  
  In general, when we don't have enough of a device, we end up with clonable
  shit or something don't we ?
 
 Not really.  Cloneable devices are used to create per-open context.

well, it seems to me that having a limit on the number of devices like usb
stuff or something is somewhat wasteful. It would probably be nicer if it
could adjust automatically (note that I have no idea how much more work this
involves besides allocating the structures in the right location... but since
usb hot-attaches after boot, I figure most of the work is already there).



Re: /etc/rc.d/rc.subr; prefix ${pexp} with script interpretor path

2013-09-16 Thread Marc Espie
On Mon, Sep 16, 2013 at 08:57:11PM +0200, Antoine Jacoutot wrote:
  Any other thoughts?
 
 Is there any way we could use something else than file(1) ?

Well, you can look for a script using dd and matching it against
the two possible signatures. e.g.,

dd 2/dev/null if=file bs=4 count=1|grep -qe '#!/./ -e '#! /'

I'm not sure that's too much of a progress.

There might be another way to grab the first 4 bytes of a file.
Off-hand I don't see any.



Re: /etc/rc.d/rc.subr; prefix ${pexp} with script interpretor path

2013-09-16 Thread Marc Espie
On Mon, Sep 16, 2013 at 11:28:06PM +0200, Alexander Hall wrote:
 sed can do it all. Really. Notes:
 
 - I separate re_quote() cause I think it can be useful in other places.
 - I think re_quote() is (basic) regex complete.
 - I don't care if the interpreter is (or seems) nonexistant, as that
   shouldn't be a runtime error.
 - I'm sure sed may die horribly if you try to feed it a 9GB oneline
   file. However, if so, it should not produce any output anyway. ;)
   If this would ever be considered a real problem, dd(1) would help
   (as espie already mentioned).
 
   re_quote() { sed 's/\([]^$*.\\[]\)/\\\1/g'; }
 
   interpreter=$(
   sed -n 's/^#![[:space:]]*\(.*\)/\1 /p;q' ${daemon} |
   re_quote)
   pexp=$interpreter$pexp

Ah, but what happens when you run that thru a pure binary file that
spans a single line ?...



Re: SQLite 3.8.0.2 diff

2013-09-18 Thread Marc Espie
On Thu, Sep 12, 2013 at 02:35:02PM -0400, James Turner wrote:
 Attached is a diff to update our in tree version of SQLite to the
 recently released 3.8.0.2. SQLite 3.8.0 is needed for a fossil update
 I'm working on.
 
 I've tested this diff against my fossil update and everything appears to
 be good, the recent arc4random addition should have been maintained
 across the diff.
 
 This should probably go through a round of bulk builds. I'll express my
 thanks now for anyone who can assist by running a bulk build on a couple
 platforms.
 

I'm also okay with this.



HEADS UP: tmux protocol change

2013-10-12 Thread Marc Espie
be sure to not have any old sessions hanging around
before you rebuild source, as you will be unable
to reattach these.

(quick fix: recompile tmux from -D2013-09-01,
don't install the binary, but use it to attach
to your old session. Just saved my ass).



config: make reconfig target

2013-10-28 Thread Marc Espie
The following patch expands config(8) slightly to be able to say
make reconfig from the build directory.

Yeah, it's sugar, but it got tiresome having to change directories
all the time.

I've put the reconfig target at end, where I'm sure it can't be the
first target.

An alternative approach would be to add an extra keyword for
boilerplate stuff that would always be emitted by config, but I
don't see the point of the extra knob in that particular case.

Index: config.h
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/config/config.h,v
retrieving revision 1.25
diff -u -p -r1.25 config.h
--- config.h5 Dec 2012 23:20:26 -   1.25
+++ config.h28 Oct 2013 07:44:56 -
@@ -339,6 +339,10 @@ void   defoption(const char *name);
 intdevbase_has_instances(struct devbase *, int);
 intdeva_has_instances(struct deva *, int);
 void   setupdirs(void);
+intpflag;
+char   *sflag;
+char   *bflag;
+char   *startdir;
 
 /* mkheaders.c */
 intmkheaders(void);
Index: main.c
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/config/main.c,v
retrieving revision 1.44
diff -u -p -r1.44 main.c
--- main.c  22 Jun 2012 22:02:29 -  1.44
+++ main.c  28 Oct 2013 07:47:13 -
@@ -93,15 +93,24 @@ usage(void)
exit(1);
 }
 
+int pflag = 0;
+char *sflag = NULL;
+char *bflag = NULL;
+char *startdir;
+
 int
 main(int argc, char *argv[])
 {
char *p;
const char *last_component;
char *outfile = NULL;
-   int pflag, ch, eflag, uflag, fflag;
+   int ch, eflag, uflag, fflag;
+   char dirbuffer[PATH_MAX];
 
pflag = eflag = uflag = fflag = 0;
+   startdir = getcwd(dirbuffer, sizeof dirbuffer);
+   if (startdir == NULL)
+   warn(Can't getcwd, no make reconfig);
while ((ch = getopt(argc, argv, egpfb:s:o:u)) != -1) {
switch (ch) {
 
@@ -148,10 +157,12 @@ main(int argc, char *argv[])
break;
 
case 'b':
+   bflag = optarg;
builddir = optarg;
break;
 
case 's':
+   sflag = optarg;
srcdir = optarg;
break;
 
Index: mkmakefile.c
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/config/mkmakefile.c,v
retrieving revision 1.37
diff -u -p -r1.37 mkmakefile.c
--- mkmakefile.c17 Sep 2012 17:36:13 -  1.37
+++ mkmakefile.c28 Oct 2013 07:47:03 -
@@ -60,6 +60,7 @@
 static const char *srcpath(struct files *);
 
 static int emitdefs(FILE *);
+static int emitreconfig(FILE *);
 static int emitfiles(FILE *, int);
 
 static int emitobjs(FILE *);
@@ -120,6 +121,10 @@ mkmakefile(void)
if ((*fn)(ofp))
goto wrerror;
}
+   if (startdir != NULL) {
+   if (emitreconfig(ofp) != 0)
+   goto wrerror;
+   }
if (ferror(ifp)) {
(void)fprintf(stderr,
config: error reading %s (at line %d): %s\n,
@@ -271,6 +276,33 @@ emitdefs(FILE *fp)
for (nv = mkoptions; nv != NULL; nv = nv-nv_next)
if (fprintf(fp, %s=%s\n, nv-nv_name, nv-nv_str)  0)
return (1);
+   return (0);
+}
+
+static int
+emitreconfig(FILE *fp)
+{
+   if (fputs(\n
+   .PHONY: reconfig\n
+   reconfig:\n, fp)  0)
+   return (1);
+   if (fprintf(fp, \tcd %s  config , startdir)  0)
+   return (1);
+   if (pflag) {
+   if (fputs(-p , fp)  0)
+   return (1);
+   }
+   if (sflag) {
+   if (fprintf(fp, -s %s , sflag)  0)
+   return (1);
+   }
+   if (bflag) {
+   if (fprintf(fp, -b %s , bflag)  0)
+   return (1);
+   }
+   /* other options */
+   if (fprintf(fp, %s\n, conffile)  0)
+   return (1);
return (0);
 }
 



config(8) 'make config' v2

2013-10-28 Thread Marc Espie
Theo says no need for reconfig, make config will do just fine.

okays ?


Index: config.h
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/config/config.h,v
retrieving revision 1.25
diff -u -p -r1.25 config.h
--- config.h5 Dec 2012 23:20:26 -   1.25
+++ config.h28 Oct 2013 07:44:56 -
@@ -339,6 +339,10 @@ void   defoption(const char *name);
 intdevbase_has_instances(struct devbase *, int);
 intdeva_has_instances(struct deva *, int);
 void   setupdirs(void);
+intpflag;
+char   *sflag;
+char   *bflag;
+char   *startdir;
 
 /* mkheaders.c */
 intmkheaders(void);
Index: main.c
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/config/main.c,v
retrieving revision 1.44
diff -u -p -r1.44 main.c
--- main.c  22 Jun 2012 22:02:29 -  1.44
+++ main.c  28 Oct 2013 07:47:13 -
@@ -93,15 +93,24 @@ usage(void)
exit(1);
 }
 
+int pflag = 0;
+char *sflag = NULL;
+char *bflag = NULL;
+char *startdir;
+
 int
 main(int argc, char *argv[])
 {
char *p;
const char *last_component;
char *outfile = NULL;
-   int pflag, ch, eflag, uflag, fflag;
+   int ch, eflag, uflag, fflag;
+   char dirbuffer[PATH_MAX];
 
pflag = eflag = uflag = fflag = 0;
+   startdir = getcwd(dirbuffer, sizeof dirbuffer);
+   if (startdir == NULL)
+   warn(Can't getcwd, no make reconfig);
while ((ch = getopt(argc, argv, egpfb:s:o:u)) != -1) {
switch (ch) {
 
@@ -148,10 +157,12 @@ main(int argc, char *argv[])
break;
 
case 'b':
+   bflag = optarg;
builddir = optarg;
break;
 
case 's':
+   sflag = optarg;
srcdir = optarg;
break;
 
Index: mkmakefile.c
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/config/mkmakefile.c,v
retrieving revision 1.37
diff -u -p -r1.37 mkmakefile.c
--- mkmakefile.c17 Sep 2012 17:36:13 -  1.37
+++ mkmakefile.c28 Oct 2013 17:57:38 -
@@ -60,6 +60,7 @@
 static const char *srcpath(struct files *);
 
 static int emitdefs(FILE *);
+static int emitreconfig(FILE *);
 static int emitfiles(FILE *, int);
 
 static int emitobjs(FILE *);
@@ -120,6 +121,10 @@ mkmakefile(void)
if ((*fn)(ofp))
goto wrerror;
}
+   if (startdir != NULL) {
+   if (emitreconfig(ofp) != 0)
+   goto wrerror;
+   }
if (ferror(ifp)) {
(void)fprintf(stderr,
config: error reading %s (at line %d): %s\n,
@@ -271,6 +276,33 @@ emitdefs(FILE *fp)
for (nv = mkoptions; nv != NULL; nv = nv-nv_next)
if (fprintf(fp, %s=%s\n, nv-nv_name, nv-nv_str)  0)
return (1);
+   return (0);
+}
+
+static int
+emitreconfig(FILE *fp)
+{
+   if (fputs(\n
+   .PHONY: config\n
+   config:\n, fp)  0)
+   return (1);
+   if (fprintf(fp, \tcd %s  config , startdir)  0)
+   return (1);
+   if (pflag) {
+   if (fputs(-p , fp)  0)
+   return (1);
+   }
+   if (sflag) {
+   if (fprintf(fp, -s %s , sflag)  0)
+   return (1);
+   }
+   if (bflag) {
+   if (fprintf(fp, -b %s , bflag)  0)
+   return (1);
+   }
+   /* other options */
+   if (fprintf(fp, %s\n, conffile)  0)
+   return (1);
return (0);
 }
 



V3 Re: config(8) 'make config' v2

2013-10-28 Thread Marc Espie
Backforth with Theo. A bit more logic.
Most trees are moveable, and it's possible to
do so when -b is not in use...

Index: config.h
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/config/config.h,v
retrieving revision 1.25
diff -u -p -r1.25 config.h
--- config.h5 Dec 2012 23:20:26 -   1.25
+++ config.h28 Oct 2013 07:44:56 -
@@ -339,6 +339,10 @@ void   defoption(const char *name);
 intdevbase_has_instances(struct devbase *, int);
 intdeva_has_instances(struct deva *, int);
 void   setupdirs(void);
+intpflag;
+char   *sflag;
+char   *bflag;
+char   *startdir;
 
 /* mkheaders.c */
 intmkheaders(void);
Index: main.c
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/config/main.c,v
retrieving revision 1.44
diff -u -p -r1.44 main.c
--- main.c  22 Jun 2012 22:02:29 -  1.44
+++ main.c  28 Oct 2013 18:36:33 -
@@ -93,13 +93,19 @@ usage(void)
exit(1);
 }
 
+int pflag = 0;
+char *sflag = NULL;
+char *bflag = NULL;
+char *startdir;
+
 int
 main(int argc, char *argv[])
 {
char *p;
const char *last_component;
char *outfile = NULL;
-   int pflag, ch, eflag, uflag, fflag;
+   int ch, eflag, uflag, fflag;
+   char dirbuffer[PATH_MAX];
 
pflag = eflag = uflag = fflag = 0;
while ((ch = getopt(argc, argv, egpfb:s:o:u)) != -1) {
@@ -148,10 +154,12 @@ main(int argc, char *argv[])
break;
 
case 'b':
+   bflag = optarg;
builddir = optarg;
break;
 
case 's':
+   sflag = optarg;
srcdir = optarg;
break;
 
@@ -163,7 +171,15 @@ main(int argc, char *argv[])
argc -= optind;
argv += optind;
if (argc  1 || (eflag  argv[0] == NULL))
+
usage();
+   if (bflag) {
+   startdir = getcwd(dirbuffer, sizeof dirbuffer);
+   if (startdir == NULL)
+   warn(Use of -b and can't getcwd, no make config);
+   } else {
+   startdir = ../../conf;
+   }
 
if (eflag) {
 #ifdef MAKE_BOOTSTRAP
Index: mkmakefile.c
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/config/mkmakefile.c,v
retrieving revision 1.37
diff -u -p -r1.37 mkmakefile.c
--- mkmakefile.c17 Sep 2012 17:36:13 -  1.37
+++ mkmakefile.c28 Oct 2013 18:36:46 -
@@ -60,6 +60,7 @@
 static const char *srcpath(struct files *);
 
 static int emitdefs(FILE *);
+static int emitreconfig(FILE *);
 static int emitfiles(FILE *, int);
 
 static int emitobjs(FILE *);
@@ -120,6 +121,10 @@ mkmakefile(void)
if ((*fn)(ofp))
goto wrerror;
}
+   if (startdir != NULL) {
+   if (emitreconfig(ofp) != 0)
+   goto wrerror;
+   }
if (ferror(ifp)) {
(void)fprintf(stderr,
config: error reading %s (at line %d): %s\n,
@@ -271,6 +276,33 @@ emitdefs(FILE *fp)
for (nv = mkoptions; nv != NULL; nv = nv-nv_next)
if (fprintf(fp, %s=%s\n, nv-nv_name, nv-nv_str)  0)
return (1);
+   return (0);
+}
+
+static int
+emitreconfig(FILE *fp)
+{
+   if (fputs(\n
+   .PHONY: config\n
+   config:\n, fp)  0)
+   return (1);
+   if (fprintf(fp, \tcd %s  config , startdir)  0)
+   return (1);
+   if (pflag) {
+   if (fputs(-p , fp)  0)
+   return (1);
+   }
+   if (sflag) {
+   if (fprintf(fp, -s %s , sflag)  0)
+   return (1);
+   }
+   if (bflag) {
+   if (fprintf(fp, -b %s , bflag)  0)
+   return (1);
+   }
+   /* other options */
+   if (fprintf(fp, %s\n, conffile)  0)
+   return (1);
return (0);
 }
 



Re: update perl Module::Build in base

2013-11-01 Thread Marc Espie
On Fri, Nov 01, 2013 at 04:56:04PM +0100, Ingo Schwarze wrote:
 Oh well.  To paraphase (forgive me if i misunderstand), It ought to
 be easy to write something better than Module::Build, and even though
 nobody has so far volunteered to propose an actual design, and even
 less to write some code, we already have a definite time schedule for
 retiring Module::Build.  WT...?
 
 However, whetever thay do, we are certainly able to cope, either just
 keeping Module::Build in base or moving it to a port, whatever will be
 more convenient when the time comes.

Go read Chromatics blog

http://modernperlbooks.com/mt/index.html

perl has been schizoid between practicality and moving forward for a few
years now.

One thing it has going for it is that there are very bright people who
understand most of the issues involved, and so I'm not too worried about
what's going to happen...



Re: CVS: cvs.openbsd.org: src

2013-11-18 Thread Marc Espie
On Tue, Nov 19, 2013 at 12:11:17AM -0500, Ted Unangst wrote:
 On Mon, Nov 18, 2013 at 18:49, Philip Guenther wrote:
 
  btw, no library version change because the function stubs already
  existed.
  
  Hmm, since this is actually offering new functionality (by sem_open()
  and friends no longer always failing), I think it a minor bump would
  be appropriate.  Consider that a program with an autoconf test of
  sem_open() will now return a different answer, just as if sem_open()
  was completely new.  no?
 
 I hear you, but disagree. We fix program disabling bugs in libraries
 frequently without bumping. I have always thought of library
 versioning being more about program integrity, as in all the pieces
 you expect to find are all there, but it doesn't say anything about
 the inner workings of the pieces.

As theo says, there are other library bumps later, but you're wrong.

Use-case: new packages, slightly older snapshots. New packages actually
make use of sem_open, because of said added functionality. Without a bump,
pkg_add will allow to add them, and they won't work, because the functionality
wasn't there...

It is added functionality. It's not a minor bugfix.



Re: mention dpb in release(8)

2013-11-26 Thread Marc Espie
On Tue, Nov 26, 2013 at 02:38:43PM +, Stuart Henderson wrote:
 Is this OK, or should we remove bits about building packages
 individually too?

I'd rather send people to ports(7) directly... since that explains both
single port and bulk building.

 Index: share/man/man8/release.8
 ===
 RCS file: /cvs/src/share/man/man8/release.8,v
 retrieving revision 1.66
 diff -u -p -u -1 -5 -r1.66 release.8
 --- share/man/man8/release.8  2 Sep 2012 13:47:25 -   1.66
 +++ share/man/man8/release.8  26 Nov 2013 14:36:59 -
 @@ -338,25 +338,29 @@ The
  subsystem of contributed applications is described in
  .Xr ports 7 .
  For ease of installation ports can be pre-compiled into
  .Sq packages
  which can then be installed on multiple machines using
  .Xr pkg_add 1 .
  Packages are created by selecting an application to build
  (we'll call this one CATEGORY/PORT) and then running the following:
  as root:
  .Bd -literal -offset indent
  $ cd /usr/ports/CATEGORY/PORT
  $ su
  # make package
  .Ed
  .Pp
 -That's all there is to it.
 +To build a larger set of packages,
 +.Xr dpb 1
 +has various facilities to manage the build,
 +either on a single machine or a cluster.
  .Sh SEE ALSO
  .Xr cvs 1 ,
 +.Xr dpb 1 ,
  .Xr pkg_add 1 ,
  .Xr ports 7 ,
  .Xr sudo 8 ,
  .Xr sysmerge 8
  .Sh HISTORY
  This document first appeared in
  .Ox 2.8 .



Re: mention dpb in release(8)

2013-11-26 Thread Marc Espie
On Tue, Nov 26, 2013 at 03:38:49PM +, Stuart Henderson wrote:
 On 2013/11/26 16:28, Marc Espie wrote:
  On Tue, Nov 26, 2013 at 02:38:43PM +, Stuart Henderson wrote:
   Is this OK, or should we remove bits about building packages
   individually too?
  
  I'd rather send people to ports(7) directly... since that explains both
  single port and bulk building.
 
 Something like this?
 
 Includes a bonus diff for ports(7), the mirror-maker target was removed.
 
 Index: man8/release.8
 ===
 RCS file: /cvs/src/share/man/man8/release.8,v
 retrieving revision 1.66
 diff -u -p -r1.66 release.8
 --- man8/release.82 Sep 2012 13:47:25 -   1.66
 +++ man8/release.826 Nov 2013 15:37:29 -
 @@ -335,22 +335,11 @@ in your release directory.
  .Ss 7. Make the third party packages
  The
  .Sq ports
 -subsystem of contributed applications is described in
 -.Xr ports 7 .
 -For ease of installation ports can be pre-compiled into
 +subsystem of contributed applications is capable of producing
  .Sq packages
 -which can then be installed on multiple machines using
 -.Xr pkg_add 1 .
 -Packages are created by selecting an application to build
 -(we'll call this one CATEGORY/PORT) and then running the following:
 -as root:
 -.Bd -literal -offset indent
 -$ cd /usr/ports/CATEGORY/PORT
 -$ su
 -# make package
 -.Ed
 -.Pp
 -That's all there is to it.
 +for installation, either individually or in bulk.
 +This is described in
 +.Xr ports 7 .
  .Sh SEE ALSO
  .Xr cvs 1 ,
  .Xr pkg_add 1 ,
 
 Index: man7/ports.7
 ===
 RCS file: /cvs/src/share/man/man7/ports.7,v
 retrieving revision 1.96
 diff -u -p -r1.96 ports.7
 --- man7/ports.7  11 Nov 2013 21:14:29 -  1.96
 +++ man7/ports.7  26 Nov 2013 15:37:29 -
 @@ -83,9 +83,6 @@ offers a few useful targets.
  .It Ar index
  rebuild the ports complete index,
  .Pa /usr/ports/INDEX
 -.It Ar mirror-maker
 -see
 -.Xr mirroring-ports 7 ,
  .It Ar pkglocatedb
  build a
  .Xr pkg_mklocatedb 1
 @@ -740,6 +737,7 @@ List of users and groups created by port
  .Xr pkg_info 1 ,
  .Xr bsd.port.mk 5 ,
  .Xr port-modules 5 ,
 +.Xr mirroring-ports 7 ,
  .Xr packages 7
  .Pp
  The
okay for me.



Re: perlre(1) and substitution evaluations

2013-11-30 Thread Marc Espie
On Sat, Nov 30, 2013 at 12:59:54PM +0200, Lars Nooden wrote:
 perlre(1) seems to be missing information about substitution evaluations 
 with the /e option.  The functionality is present in perl:
 
   perl -e '$_=2; s/2/1+3/e; print'
 
 But it is not listed in the base documentation.  The modifier /e is 
 described in Programming Perl, 4th ed, pp 186, 254-255.  Could something 
 like the following be added?
 
 Regards,
 /Lars
 
 ? pod2htmd.tmp
 Index: perlre.pod
 ===
 RCS file: /cvs/src/gnu/usr.bin/perl/pod/perlre.pod,v
 retrieving revision 1.16
 diff -u -p -u -p -r1.16 perlre.pod
 --- perlre.pod25 Mar 2013 20:40:59 -  1.16
 +++ perlre.pod30 Nov 2013 10:58:38 -
 @@ -105,6 +105,12 @@ X/p Xregex, preserve Xregexp, prese
  Preserve the string matched such that ${^PREMATCH}, ${^MATCH}, and
  ${^POSTMATCH} are available for use after matching.
  
 +=item e
 +X/e
 +
 +Treat the replacement portion as an interpreted expression.  
 +Each additional C/e modifier after the first functions as an eval() around 
 the code to execute.  
 +
  =item g and c
  X/g X/c
  
For this kind of thing, you're much better off talking to upstream
as this is totally openbsd-independent.



Re: Remove 4.3BSD tty compatibility, take 2

2013-12-10 Thread Marc Espie
On Tue, Dec 10, 2013 at 10:31:31PM +0100, Mark Kettenis wrote:
  Date: Tue, 10 Dec 2013 21:48:41 +0100
  From: Christian Weisgerber na...@mips.inka.de
  
  New attempt, now that the userland has been cleaned up.  This diff
  kills the remaining parts of the COMPAT_43 tty handling in the
  kernel.  (There are some further network-related COMPAT_43 fragments
  that are not touched by this.)
  
  Comments?  ok?
 
 Noticed TIOCGSID in that list.  Don't think that is a 4.3 compat
 ioctl.  Rather a System V compat ioctl.  Might be useful if we ever
 find the need to implement tcgetsid(3).  There is code in gnulib (the
 GNU portability library) that uses this ioctl to implement
 tcgetsid(3):
 
 https://github.com/gagern/gnulib/blob/master/lib/tcgetsid.c
 
 Not sure if there actually is anything out there that uses it.

Note that tcgetsid(3) is part of posix 2008... I don't see this as
a noxious interface, so I don't see any reason not to keep it...

In ports, devel/ddd uses it as a fallback on systems without setsid.

lang/sbcl checks for it, but it definitely is just wanting to create
a wrapper for it.

(apparently, vala has similar code, just looking thru google).

Oh, hey, and it's in Stevens... so we MUST have it :)



Re: Remove 4.3BSD tty compatibility, take 2

2013-12-10 Thread Marc Espie
On Tue, Dec 10, 2013 at 11:05:01PM +0100, Marc Espie wrote:
 
 Note that tcgetsid(3) is part of posix 2008... I don't see this as
 a noxious interface, so I don't see any reason not to keep it...

My bad, it's XSI in posix 2008, but I was reading posix 2013, which has:

Issue 7

The tcgetsid() function is moved from the XSI option to the Base.



more predictable package installations

2013-12-28 Thread Marc Espie
I'm thinking I'd like to impose a PATH for running @exec/@unexec
style lines in packing-lists.

Some people have their default path ignoring /usr/local, which is just
fine, but the packages expect things to run in a correct way, so currently
any local command requires /usr/local/bin/cmd to be run (and does that).

setting PATH explicitly to
/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin:${LOCALBASE}/bin:${LOCALBASE}/sbin
before running @exec/@unexec would allow for reproducibility in package 
installations, and simplify some of the command lines.



tedu old PKG_PREFIX

2013-12-31 Thread Marc Espie
I think I put this in for compatibility with the old tools years ago.
I don't think anyone still uses this, so this can probably die.

(I have run a grep thru the ports tree and the man pages, no reference at all)

okay ?

Index: OpenBSD/Delete.pm
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/pkg_add/OpenBSD/Delete.pm,v
retrieving revision 1.122
diff -u -p -r1.122 Delete.pm
--- OpenBSD/Delete.pm   30 Dec 2013 09:02:37 -  1.122
+++ OpenBSD/Delete.pm   31 Dec 2013 12:12:52 -
@@ -147,7 +147,6 @@ sub delete_plist
 
my $pkgname = $plist-pkgname;
$state-{pkgname} = $pkgname;
-   $ENV{'PKG_PREFIX'} = $plist-localbase;
if (!$state-{size_only}) {
$plist-register_manpage($state);
manpages_unindex($state);
Index: OpenBSD/PkgAdd.pm
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/pkg_add/OpenBSD/PkgAdd.pm,v
retrieving revision 1.41
diff -u -p -r1.41 PkgAdd.pm
--- OpenBSD/PkgAdd.pm   31 Dec 2013 11:24:55 -  1.41
+++ OpenBSD/PkgAdd.pm   31 Dec 2013 12:13:07 -
@@ -1,7 +1,7 @@
 #! /usr/bin/perl
 
 # ex:ts=8 sw=4:
-# $OpenBSD: PkgAdd.pm,v 1.41 2013/12/31 11:24:55 espie Exp $
+# $OpenBSD: PkgAdd.pm,v 1.40 2013/12/25 14:38:15 espie Exp $
 #
 # Copyright (c) 2003-2010 Marc Espie es...@openbsd.org
 #
@@ -134,15 +134,11 @@ sub handle_options
$state-{do_faked} = 1;
} elsif (defined $state-opt('B')) {
$state-{destdir} = $state-opt('B');
-   } elsif (defined $ENV{'PKG_PREFIX'}) {
-   $state-{destdir} = $ENV{'PKG_PREFIX'};
}
if (defined $state-{destdir}) {
$state-{destdir}.='/';
-   $ENV{'PKG_DESTDIR'} = $state-{destdir};
} else {
$state-{destdir} = '';
-   delete $ENV{'PKG_DESTDIR'};
}
 
 
@@ -729,8 +729,6 @@ sub really_add
 #  $replacing = 1;
 #  }
$state-{replacing} = $replacing;
-
-   $ENV{'PKG_PREFIX'} = $state-{localbase};
 
my $handler = sub {
$state-{interrupted} = shift;
Index: OpenBSD/PkgCheck.pm
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/pkg_add/OpenBSD/PkgCheck.pm,v
retrieving revision 1.42
diff -u -p -r1.42 PkgCheck.pm
--- OpenBSD/PkgCheck.pm 25 Dec 2013 14:38:15 -  1.42
+++ OpenBSD/PkgCheck.pm 31 Dec 2013 12:13:16 -
@@ -316,15 +316,11 @@ sub handle_options
$self-{quick} = $self-opt('q');
if (defined $self-opt('B')) {
$self-{destdir} = $self-opt('B');
-   } elsif (defined $ENV{'PKG_PREFIX'}) {
-   $self-{destdir} = $ENV{'PKG_PREFIX'};
-   }
+   } 
if (defined $self-{destdir}) {
$self-{destdir} .= '/';
-   $ENV{'PKG_DESTDIR'} = $self-{destdir};
} else {
$self-{destdir} = '';
-   delete $ENV{'PKG_DESTDIR'};
}
 }
 
Index: OpenBSD/PkgCreate.pm
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/pkg_add/OpenBSD/PkgCreate.pm,v
retrieving revision 1.76
diff -u -p -r1.76 PkgCreate.pm
--- OpenBSD/PkgCreate.pm31 Dec 2013 11:21:10 -  1.76
+++ OpenBSD/PkgCreate.pm31 Dec 2013 12:13:24 -
@@ -198,9 +198,7 @@ sub handle_options
my $base = '/';
if (defined $state-opt('B')) {
$base = $state-opt('B');
-   } elsif (defined $ENV{'PKG_PREFIX'}) {
-   $base = $ENV{'PKG_PREFIX'};
-   }
+   } 
 
$state-{base} = $base;
 



Re: redo signfify options

2014-01-01 Thread Marc Espie
Something like this for the program itself:

Index: signify.c
===
RCS file: /build/data/openbsd/cvs/src/usr.bin/signify/signify.c,v
retrieving revision 1.5
diff -u -p -r1.5 signify.c
--- signify.c   31 Dec 2013 17:33:17 -  1.5
+++ signify.c   1 Jan 2014 15:50:11 -
@@ -64,8 +64,8 @@ extern char *__progname;
 static void
 usage(void)
 {
-   fprintf(stderr, usage: %s [-N] [-I input] [-O output] [-P pubkey] [-S 
seckey] 
-   -V generate | sign | verify\n, __progname);
+   fprintf(stderr, usage: %s -G | -S | -V [-n] [-o output] 
+   [-p pubkey] [-s seckey] [file...]\n, __progname);
exit(1);
 }
 
@@ -316,64 +316,91 @@ verify(const char *pubkeyfile, const cha
 int
 main(int argc, char **argv)
 {
-   const char *verb = NULL;
-   const char *pubkeyfile = NULL, *seckeyfile = NULL, *inputfile = NULL,
-   *sigfile = NULL;
-   char sigfilebuf[1024];
-   int ch, rounds;
+   const char *pubkeyfile = NULL, *seckeyfile = NULL, 
+   *sigfilefmt = %s.sig;
+   int ch, rounds, needfmt, i;
+   enum {
+   NONE,
+   GENERATE,
+   SIGN,
+   VERIFY
+   } verb = NONE;
+
 
rounds = 42;
 
-   while ((ch = getopt(argc, argv, I:NO:P:S:V:)) != -1) {
+   while ((ch = getopt(argc, argv, GSVi:no:p:s:)) != -1) {
switch (ch) {
-   case 'I':
-   inputfile = optarg;
+   case 'G':
+   if (verb)
+   usage();
+   verb = GENERATE;
+   break;
+   case 'S':
+   if (verb)
+   usage();
+   verb = SIGN;
+   break;
+   case 'V':
+   if (verb)
+   usage();
+   verb = VERIFY;
break;
-   case 'N':
+   case 'n':
rounds = 0;
break;
-   case 'O':
-   sigfile = optarg;
+   case 'o':
+   sigfilefmt = optarg;
break;
-   case 'P':
+   case 'p':
pubkeyfile = optarg;
break;
-   case 'S':
+   case 's':
seckeyfile = optarg;
break;
-   case 'V':
-   verb = optarg;
-   break;
default:
usage();
break;
}
}
argc -= optind;
-   if (argc != 0 || verb == NULL)
-   usage();
-
-   if (inputfile  !sigfile) {
-   if (snprintf(sigfilebuf, sizeof(sigfilebuf), %s.sig,
-   inputfile) = sizeof(sigfilebuf))
-   errx(1, path too long);
-   sigfile = sigfilebuf;
-   }
 
-   if (streq(verb, generate)) {
-   if (!pubkeyfile || !seckeyfile)
+   if (verb == GENERATE) {
+   if (argc != 0 || !pubkeyfile || !seckeyfile)
usage();
generate(pubkeyfile, seckeyfile, rounds);
-   } else if (streq(verb, sign)) {
-   if (!seckeyfile || !inputfile)
-   usage();
-   sign(seckeyfile, inputfile, sigfile);
-   } else if (streq(verb, verify)) {
-   if (!pubkeyfile || !inputfile)
-   usage();
-   verify(pubkeyfile, inputfile, sigfile);
-   } else {
+   } else if (verb == NONE || argc == 0) {
usage();
+   }
+
+   needfmt = strstr(sigfilefmt, %s) != NULL;
+
+   if (!needfmt  argc != 1) {
+   errx(1, -o file for several inputs);
+   }
+
+   for (i = 0; i != argc; i++) {
+   char sigfilebuf[1024];
+   const char *inputfile = argv[i];
+   const char *sigfile = NULL;
+
+   if (needfmt) {
+   if (snprintf(sigfilebuf, sizeof(sigfilebuf),
+   sigfilefmt, inputfile) = sizeof(sigfilebuf))
+   errx(1, path too long);
+   sigfile = sigfilebuf;
+   } else {
+   sigfile = sigfilefmt;
+   }
+   if (verb == SIGN) {
+   if (!seckeyfile)
+   usage();
+   sign(seckeyfile, inputfile, sigfile);
+   } else if (verb == VERIFY) {
+   if (!pubkeyfile || !inputfile)
+   usage();
+   verify(pubkeyfile, inputfile, sigfile);
+   }
}
return 0;
 }



Re: redo signfify options

2014-01-01 Thread Marc Espie
On Wed, Jan 01, 2014 at 12:53:09PM -0500, Ted Unangst wrote:
 On Wed, Jan 01, 2014 at 16:25, Marc Espie wrote:
  Now that you're fixing it, I think you've still got it wrong.
  
  Instead of
  signify -V -p pubkey -i input
  you want to be able to do
  signify -V -p pubkey  input ...
  
  e.g., at this point, input would not need to be an option, so that you
  can verify several files in one go.
 
 we can do that too if we need batch verification, but I left it out
 of this commit.

Well, at least make input not an option but an actual parameter.

Otherwise, you have a command with no parms.

And considering the command name, it makes little sense. As the name
says, that command is used to sign or verify a file. The *object* of that
command is the file. It's not an option !



some error fixes in signify

2014-01-02 Thread Marc Espie
Revisiting the error messages:
- pass the filenames to the low-level functions, so they can tell you
what's going on.

- FIX readall and writeall. The logic *is* wrong. Just because they
return something != len, doesn't mean they return -1.

Okay ?

Index: signify.c
===
RCS file: /build/data/openbsd/cvs/src/usr.bin/signify/signify.c,v
retrieving revision 1.6
diff -u -p -r1.6 signify.c
--- signify.c   1 Jan 2014 17:50:33 -   1.6
+++ signify.c   2 Jan 2014 13:37:38 -
@@ -92,10 +92,14 @@ xmalloc(size_t len)
 }
 
 static void
-readall(int fd, void *buf, size_t len)
+readall(int fd, void *buf, size_t len, const char *filename)
 {
-   if (read(fd, buf, len) != len)
-   err(1, read);
+   ssize_t x = read(fd, buf, len);
+   if (x == -1) {
+   err(1, read from %s, filename);
+   } else if (x != len) {
+   errx(1, short read from %s, filename);
+   }
 }
 
 static void
@@ -108,7 +112,7 @@ readb64file(const char *filename, void *
memset(b64, 0, sizeof(b64));
rv = read(fd, b64, sizeof(b64) - 1);
if (rv == -1)
-   err(1, read in %s, filename);
+   err(1, read from %s, filename);
for (i = 0; i  rv; i++)
if (b64[i] == '\n')
break;
@@ -137,7 +141,7 @@ readmsg(const char *filename, unsigned l
if (msglen  (1UL  30))
errx(1, msg too large in %s, filename);
msg = xmalloc(msglen);
-   readall(fd, msg, msglen);
+   readall(fd, msg, msglen, filename);
close(fd);
 
*msglenp = msglen;
@@ -145,10 +149,14 @@ readmsg(const char *filename, unsigned l
 }
 
 static void
-writeall(int fd, const void *buf, size_t len)
+writeall(int fd, const void *buf, size_t len, const char *filename)
 {
-   if (write(fd, buf, len) != len)
-   err(1, write);
+   ssize_t x = write(fd, buf, len);
+   if (x == -1) {
+   err(1, write to %s, filename);
+   } else if (x != len) {
+   errx(1, short write to %s, filename);
+   }
 }
 
 static void
@@ -161,10 +169,10 @@ writeb64file(const char *filename, const
 
fd = xopen(filename, O_CREAT|O_EXCL|O_NOFOLLOW|O_RDWR, mode);
snprintf(header, sizeof(header), signify -- %s\n, comment);
-   writeall(fd, header, strlen(header));
+   writeall(fd, header, strlen(header), filename);
if ((rv = b64_ntop(buf, len, b64, sizeof(b64))) == -1)
errx(1, b64 encode failed);
-   writeall(fd, b64, rv);
+   writeall(fd, b64, rv, filename);
memset(b64, 0, sizeof(b64));
close(fd);
 }



Re: some error fixes in signify

2014-01-02 Thread Marc Espie
On Thu, Jan 02, 2014 at 10:36:53AM -0500, Ted Unangst wrote:
 On Thu, Jan 02, 2014 at 14:40, Marc Espie wrote:
  Revisiting the error messages:
  - pass the filenames to the low-level functions, so they can tell you
  what's going on.
  
  - FIX readall and writeall. The logic *is* wrong. Just because they
  return something != len, doesn't mean they return -1.
 
 hmm. the intention was to keep the code as small as possible.
 a detailed dissection of every error that will never happen runs
 somewhat contrary to that goal. it's not like the difference between a
 failed write and a short write is meaningful to the user.

Oh come on, that's a bug, and really poor style.

You're going to call err() when there's no meaningful value in errno ?
That's really a bad example, if nothing else.


There's a difference between small and obnoxiously wrong.
Even if it's a wee bit larger, it doesn't make it less easy to read.



signify API change and manpage

2014-01-03 Thread Marc Espie
I really, really want to have an actual parameter for sign and verify,
this makes more sense.

Also, the set of modes of signify means it's much clearer to have separate
synopsis in the manpage...

Also fix some .Ar which should be .Fl while we're at it.

To *keep things simple*, let's only sign a single file... ;)
(note that this actually fixes the code logic, again, since you could
specify an input file option, and generate a sigfile name, and even error
out in cases things were too long if -G got used)...

Note  that I haven't changed the option order in the synopsis, though I
would tend to put -G/-S/-V first, personally...

Index: signify.1
===
RCS file: /build/data/openbsd/cvs/src/usr.bin/signify/signify.1,v
retrieving revision 1.6
diff -u -p -r1.6 signify.1
--- signify.1   1 Jan 2014 17:50:33 -   1.6
+++ signify.1   3 Jan 2014 13:47:54 -
@@ -23,15 +23,25 @@
 .Sh SYNOPSIS
 .Nm signify
 .Op Fl n
-.Op Fl i Ar input
+.Fl p Ar pubkey
+.Fl s Ar seckey
+.Fl G
+.Nm signify
 .Op Fl o Ar output
-.Op Fl p Ar pubkey
-.Op Fl s Ar seckey
-.Fl G | S | V
+.Fl s Ar seckey
+.Fl S
+.Ar input
+.Nm signify
+.Op Fl o Ar output
+.Fl p Ar pubkey
+.Fl V
+.Ar input
 .Sh DESCRIPTION
 The
 .Nm
-utility creates and verifies cryptographic signatures.
+utility creates and verifies cryptographic signatures for
+an input file
+.Ar input .
 The mode of operation is selected by the
 .Fl G ,
 .Fl S ,
@@ -43,8 +53,6 @@ The options are as follows:
 .Bl -tag -width Ds
 .It Fl G
 Generate a new keypair.
-.It Fl i Ar input
-Input file to sign or verify.
 .It Fl n
 Do not ask for a passphrase during key generation.
 Otherwise,
@@ -56,17 +64,17 @@ The default is
 .Ar input Ns .sig .
 .It Fl p Ar pubkey
 Public key produced by
-.Ar G ,
+.Fl G ,
 and used by
-.Ar V
+.Fl V
 to check a signature.
 .It Fl S
 Sign the input file.
 .It Fl s Ar seckey
 Secret (private) key produced by
-.Ar G ,
+.Fl G ,
 and used by
-.Ar S
+.Fl S
 to sign a message.
 .It Fl V
 Verify the input file and signature match.
@@ -94,13 +102,13 @@ The message file is too large.
 .El
 .Sh EXAMPLES
 Create a new keypair:
-.Dl $ signify -p newkey.pub -s newkey.sec -G
+.Dl $ signify -G -p newkey.pub -s newkey.sec
 .Pp
 Sign a file, specifying a signature name:
-.Dl $ signify -s key.sec -i message.txt -o msg.sig -S
+.Dl $ signify -S -s key.sec -o msg.sig message.txt
 .Pp
 Verify a signature, using the default signature name:
-.Dl $ signify -p key.pub -i generalsorders.txt -V
+.Dl $ signify -V -p key.pub generalsorders.txt
 .Sh SEE ALSO
 .Xr cmp 1 ,
 .Xr sha256 1 ,
@@ -109,4 +117,4 @@ Verify a signature, using the default si
 The
 .Nm
 command first appeared in
-.Ox 5.5
+.Ox 5.5 .
Index: signify.c
===
RCS file: /build/data/openbsd/cvs/src/usr.bin/signify/signify.c,v
retrieving revision 1.7
diff -u -p -r1.7 signify.c
--- signify.c   2 Jan 2014 16:34:02 -   1.7
+++ signify.c   3 Jan 2014 13:57:23 -
@@ -64,8 +64,8 @@ extern char *__progname;
 static void
 usage(void)
 {
-   fprintf(stderr, usage: %s [-n] [-i input] [-o output] [-p pubkey] [-s 
seckey] 
-   -G | -S | -V\n, __progname);
+   fprintf(stderr, usage: %s [-n] [-o output] [-p pubkey] [-s seckey] 
+   -G | -S | -V [input]\n, __progname);
exit(1);
 }
 
@@ -338,7 +338,7 @@ main(int argc, char **argv)
 
rounds = 42;
 
-   while ((ch = getopt(argc, argv, GSVi:no:p:s:)) != -1) {
+   while ((ch = getopt(argc, argv, GSVno:p:s:)) != -1) {
switch (ch) {
case 'G':
if (verb)
@@ -355,9 +355,6 @@ main(int argc, char **argv)
usage();
verb = VERIFY;
break;
-   case 'i':
-   inputfile = optarg;
-   break;
case 'n':
rounds = 0;
break;
@@ -376,21 +373,29 @@ main(int argc, char **argv)
}
}
argc -= optind;
-   if (argc != 0)
+   argv += optind;
+
+   if (verb == GENERATE) {
+   if (!pubkeyfile || !seckeyfile || argc != 0)
+   usage();
+   generate(pubkeyfile, seckeyfile, rounds);
+   } else if (verb == NONE) {
+   usage();
+   }
+
+   if (argc != 1)
usage();
 
-   if (inputfile  !sigfile) {
+   inputfile = argv[0];
+
+   if (!sigfile) {
if (snprintf(sigfilebuf, sizeof(sigfilebuf), %s.sig,
inputfile) = sizeof(sigfilebuf))
errx(1, path too long);
sigfile = sigfilebuf;
}
 
-   if (verb == GENERATE) {
-   if (!pubkeyfile || !seckeyfile)
-   usage();
-   generate(pubkeyfile, seckeyfile, rounds);
-   } else if (verb == SIGN) {
+   if 

redux: signify diff

2014-01-03 Thread Marc Espie
Rechecked that -G was working (broken) and jmc wants identical SYNOPSIS/usage.
Together with the \n for base64...

Index: signify.1
===
RCS file: /build/data/openbsd/cvs/src/usr.bin/signify/signify.1,v
retrieving revision 1.6
diff -u -p -r1.6 signify.1
--- signify.1   1 Jan 2014 17:50:33 -   1.6
+++ signify.1   3 Jan 2014 13:47:54 -
@@ -23,15 +23,25 @@
 .Sh SYNOPSIS
 .Nm signify
 .Op Fl n
-.Op Fl i Ar input
+.Fl p Ar pubkey
+.Fl s Ar seckey
+.Fl G
+.Nm signify
 .Op Fl o Ar output
-.Op Fl p Ar pubkey
-.Op Fl s Ar seckey
-.Fl G | S | V
+.Fl s Ar seckey
+.Fl S
+.Ar input
+.Nm signify
+.Op Fl o Ar output
+.Fl p Ar pubkey
+.Fl V
+.Ar input
 .Sh DESCRIPTION
 The
 .Nm
-utility creates and verifies cryptographic signatures.
+utility creates and verifies cryptographic signatures for
+an input file
+.Ar input .
 The mode of operation is selected by the
 .Fl G ,
 .Fl S ,
@@ -43,8 +53,6 @@ The options are as follows:
 .Bl -tag -width Ds
 .It Fl G
 Generate a new keypair.
-.It Fl i Ar input
-Input file to sign or verify.
 .It Fl n
 Do not ask for a passphrase during key generation.
 Otherwise,
@@ -56,17 +64,17 @@ The default is
 .Ar input Ns .sig .
 .It Fl p Ar pubkey
 Public key produced by
-.Ar G ,
+.Fl G ,
 and used by
-.Ar V
+.Fl V
 to check a signature.
 .It Fl S
 Sign the input file.
 .It Fl s Ar seckey
 Secret (private) key produced by
-.Ar G ,
+.Fl G ,
 and used by
-.Ar S
+.Fl S
 to sign a message.
 .It Fl V
 Verify the input file and signature match.
@@ -94,13 +102,13 @@ The message file is too large.
 .El
 .Sh EXAMPLES
 Create a new keypair:
-.Dl $ signify -p newkey.pub -s newkey.sec -G
+.Dl $ signify -G -p newkey.pub -s newkey.sec
 .Pp
 Sign a file, specifying a signature name:
-.Dl $ signify -s key.sec -i message.txt -o msg.sig -S
+.Dl $ signify -S -s key.sec -o msg.sig message.txt
 .Pp
 Verify a signature, using the default signature name:
-.Dl $ signify -p key.pub -i generalsorders.txt -V
+.Dl $ signify -V -p key.pub generalsorders.txt
 .Sh SEE ALSO
 .Xr cmp 1 ,
 .Xr sha256 1 ,
@@ -109,4 +117,4 @@ Verify a signature, using the default si
 The
 .Nm
 command first appeared in
-.Ox 5.5
+.Ox 5.5 .
Index: signify.c
===
RCS file: /build/data/openbsd/cvs/src/usr.bin/signify/signify.c,v
retrieving revision 1.7
diff -u -p -r1.7 signify.c
--- signify.c   2 Jan 2014 16:34:02 -   1.7
+++ signify.c   3 Jan 2014 15:37:57 -
@@ -64,8 +64,11 @@ extern char *__progname;
 static void
 usage(void)
 {
-   fprintf(stderr, usage: %s [-n] [-i input] [-o output] [-p pubkey] [-s 
seckey] 
-   -G | -S | -V\n, __progname);
+   fprintf(stderr, usage:
+   \t%s [-n] -p pubkey -s seckey -G\n
+   \t%s [-o output] -s seckey -S input\n
+   \t%s [-o output] -p pubkey -V input\n,
+   __progname, __progname, __progname);
exit(1);
 }
 
@@ -170,8 +173,9 @@ writeb64file(const char *filename, const
fd = xopen(filename, O_CREAT|O_EXCL|O_NOFOLLOW|O_RDWR, mode);
snprintf(header, sizeof(header), signify -- %s\n, comment);
writeall(fd, header, strlen(header), filename);
-   if ((rv = b64_ntop(buf, len, b64, sizeof(b64))) == -1)
+   if ((rv = b64_ntop(buf, len, b64, sizeof(b64)-1)) == -1)
errx(1, b64 encode failed);
+   b64[rv++] = '\n';
writeall(fd, b64, rv, filename);
memset(b64, 0, sizeof(b64));
close(fd);
@@ -338,7 +342,7 @@ main(int argc, char **argv)
 
rounds = 42;
 
-   while ((ch = getopt(argc, argv, GSVi:no:p:s:)) != -1) {
+   while ((ch = getopt(argc, argv, GSVno:p:s:)) != -1) {
switch (ch) {
case 'G':
if (verb)
@@ -355,9 +359,6 @@ main(int argc, char **argv)
usage();
verb = VERIFY;
break;
-   case 'i':
-   inputfile = optarg;
-   break;
case 'n':
rounds = 0;
break;
@@ -376,30 +377,37 @@ main(int argc, char **argv)
}
}
argc -= optind;
-   if (argc != 0)
-   usage();
-
-   if (inputfile  !sigfile) {
-   if (snprintf(sigfilebuf, sizeof(sigfilebuf), %s.sig,
-   inputfile) = sizeof(sigfilebuf))
-   errx(1, path too long);
-   sigfile = sigfilebuf;
-   }
+   argv += optind;
 
if (verb == GENERATE) {
-   if (!pubkeyfile || !seckeyfile)
+   if (!pubkeyfile || !seckeyfile || argc != 0)
usage();
generate(pubkeyfile, seckeyfile, rounds);
-   } else if (verb == SIGN) {
-   if (!seckeyfile || !inputfile)
-   usage();
-   sign(seckeyfile, inputfile, sigfile);
-   } else if 

Re: redux: signify diff

2014-01-03 Thread Marc Espie
On Fri, Jan 03, 2014 at 10:46:45AM -0500, Ted Unangst wrote:
 On Fri, Jan 03, 2014 at 16:39, Marc Espie wrote:
  Rechecked that -G was working (broken) and jmc wants identical
  SYNOPSIS/usage.
  Together with the \n for base64...
 
 
 
  +   } else if (verb == NONE) {
  usage();
 
 this is weird, mixing that into the middle.
Well, do you prefer that ?
Index: signify.1
===
RCS file: /cvs/src/usr.bin/signify/signify.1,v
retrieving revision 1.6
diff -u -p -r1.6 signify.1
--- signify.1   1 Jan 2014 17:50:33 -   1.6
+++ signify.1   3 Jan 2014 15:53:58 -
@@ -23,15 +23,25 @@
 .Sh SYNOPSIS
 .Nm signify
 .Op Fl n
-.Op Fl i Ar input
+.Fl p Ar pubkey
+.Fl s Ar seckey
+.Fl G
+.Nm signify
 .Op Fl o Ar output
-.Op Fl p Ar pubkey
-.Op Fl s Ar seckey
-.Fl G | S | V
+.Fl s Ar seckey
+.Fl S
+.Ar input
+.Nm signify
+.Op Fl o Ar output
+.Fl p Ar pubkey
+.Fl V
+.Ar input
 .Sh DESCRIPTION
 The
 .Nm
-utility creates and verifies cryptographic signatures.
+utility creates and verifies cryptographic signatures for
+an input file
+.Ar input .
 The mode of operation is selected by the
 .Fl G ,
 .Fl S ,
@@ -40,11 +50,9 @@ or
 options.
 .Pp
 The options are as follows:
-.Bl -tag -width Ds
+.Bl -tag -width Dssoutput
 .It Fl G
 Generate a new keypair.
-.It Fl i Ar input
-Input file to sign or verify.
 .It Fl n
 Do not ask for a passphrase during key generation.
 Otherwise,
@@ -56,17 +64,17 @@ The default is
 .Ar input Ns .sig .
 .It Fl p Ar pubkey
 Public key produced by
-.Ar G ,
+.Fl G ,
 and used by
-.Ar V
+.Fl V
 to check a signature.
 .It Fl S
 Sign the input file.
 .It Fl s Ar seckey
 Secret (private) key produced by
-.Ar G ,
+.Fl G ,
 and used by
-.Ar S
+.Fl S
 to sign a message.
 .It Fl V
 Verify the input file and signature match.
@@ -94,13 +102,13 @@ The message file is too large.
 .El
 .Sh EXAMPLES
 Create a new keypair:
-.Dl $ signify -p newkey.pub -s newkey.sec -G
+.Dl $ signify -G -p newkey.pub -s newkey.sec
 .Pp
 Sign a file, specifying a signature name:
-.Dl $ signify -s key.sec -i message.txt -o msg.sig -S
+.Dl $ signify -S -s key.sec -o msg.sig message.txt
 .Pp
 Verify a signature, using the default signature name:
-.Dl $ signify -p key.pub -i generalsorders.txt -V
+.Dl $ signify -V -p key.pub generalsorders.txt
 .Sh SEE ALSO
 .Xr cmp 1 ,
 .Xr sha256 1 ,
@@ -109,4 +117,4 @@ Verify a signature, using the default si
 The
 .Nm
 command first appeared in
-.Ox 5.5
+.Ox 5.5 .
Index: signify.c
===
RCS file: /cvs/src/usr.bin/signify/signify.c,v
retrieving revision 1.8
diff -u -p -r1.8 signify.c
--- signify.c   3 Jan 2014 15:42:22 -   1.8
+++ signify.c   3 Jan 2014 15:53:58 -
@@ -64,8 +64,11 @@ extern char *__progname;
 static void
 usage(void)
 {
-   fprintf(stderr, usage: %s [-n] [-i input] [-o output] [-p pubkey] [-s 
seckey] 
-   -G | -S | -V\n, __progname);
+   fprintf(stderr, usage:
+   \t%s [-n] -p pubkey -s seckey -G\n
+   \t%s [-o output] -s seckey -S input\n
+   \t%s [-o output] -p pubkey -V input\n,
+   __progname, __progname, __progname);
exit(1);
 }
 
@@ -339,7 +342,7 @@ main(int argc, char **argv)
 
rounds = 42;
 
-   while ((ch = getopt(argc, argv, GSVi:no:p:s:)) != -1) {
+   while ((ch = getopt(argc, argv, GSVno:p:s:)) != -1) {
switch (ch) {
case 'G':
if (verb)
@@ -356,9 +359,6 @@ main(int argc, char **argv)
usage();
verb = VERIFY;
break;
-   case 'i':
-   inputfile = optarg;
-   break;
case 'n':
rounds = 0;
break;
@@ -377,30 +377,37 @@ main(int argc, char **argv)
}
}
argc -= optind;
-   if (argc != 0)
-   usage();
-
-   if (inputfile  !sigfile) {
-   if (snprintf(sigfilebuf, sizeof(sigfilebuf), %s.sig,
-   inputfile) = sizeof(sigfilebuf))
-   errx(1, path too long);
-   sigfile = sigfilebuf;
-   }
+   argv += optind;
 
if (verb == GENERATE) {
-   if (!pubkeyfile || !seckeyfile)
+   if (!pubkeyfile || !seckeyfile || argc != 0)
usage();
generate(pubkeyfile, seckeyfile, rounds);
-   } else if (verb == SIGN) {
-   if (!seckeyfile || !inputfile)
+   } else if (verb == SIGN || verb == VERIFY) {
+   if (argc != 1)
usage();
-   sign(seckeyfile, inputfile, sigfile);
-   } else if (verb == VERIFY) {
-   if (!pubkeyfile || !inputfile)
-   usage();
-   verify(pubkeyfile, inputfile, sigfile);
+
+   inputfile = argv[0

properly terminate lines in signify

2014-01-03 Thread Marc Espie
Looking more closely at the generated files, they don't have terminating
newlines...

The base64 routines will properly ignore newlines, since they match isspace().

okay ?

Index: signify.c
===
RCS file: /build/data/openbsd/cvs/src/usr.bin/signify/signify.c,v
retrieving revision 1.7
diff -u -p -r1.7 signify.c
--- signify.c   2 Jan 2014 16:34:02 -   1.7
+++ signify.c   3 Jan 2014 15:27:16 -
@@ -170,8 +170,9 @@ writeb64file(const char *filename, const
fd = xopen(filename, O_CREAT|O_EXCL|O_NOFOLLOW|O_RDWR, mode);
snprintf(header, sizeof(header), signify -- %s\n, comment);
writeall(fd, header, strlen(header), filename);
-   if ((rv = b64_ntop(buf, len, b64, sizeof(b64))) == -1)
+   if ((rv = b64_ntop(buf, len, b64, sizeof(b64)-1)) == -1)
errx(1, b64 encode failed);
+   b64[rv++] = '\n';
writeall(fd, b64, rv, filename);
memset(b64, 0, sizeof(b64));
close(fd);



patch to test: simplify pkg addition

2014-01-05 Thread Marc Espie
The following patch removes pure package additions in pkg_add:
instead of having separate code paths for normal installs and for updates
(with extract then install), everything follows the extract then install road.

There are several reasons behind this patch:
first, it simplifies the code, and makes it simpler to fix some details
later on (in particular, I think there are now conditions where you may end
up with a slightly suboptimal /var/db/pkg if you interrupt at the wrong time).
Second, it should allow me to do some interesting things with extraction,
namely reorder archives wrt packing-lists, which is a desireable property
to speed up incremental updates.

Index: OpenBSD/Add.pm
===
RCS file: /build/data/openbsd/cvs/src/usr.sbin/pkg_add/OpenBSD/Add.pm,v
retrieving revision 1.135
diff -u -p -r1.135 Add.pm
--- OpenBSD/Add.pm  5 Jan 2014 10:24:30 -   1.135
+++ OpenBSD/Add.pm  5 Jan 2014 10:25:11 -
@@ -102,14 +102,21 @@ sub perform_installation
 {
my ($handle, $state) = @_;
 
-   $state-{archive} = $handle-{location};
$state-{end_faked} = 0;
-   $handle-{partial} //= {};
-   $state-{partial} = $handle-{partial};
$state-progress-visit_with_size($handle-{plist}, 'install', $state);
$handle-{location}-finish_and_close;
 }
 
+sub perform_extraction
+{
+   my ($handle, $state) = @_;
+
+   $handle-{partial} = {};
+   $state-{partial} = $handle-{partial};
+   $state-{archive} = $handle-{location};
+   $state-progress-visit_with_size($handle-{plist}, 'extract', $state);
+}
+
 my $user_tagged = {};
 
 sub extract_pkgname
@@ -174,13 +181,21 @@ sub prepare_for_addition
 {
 }
 
+sub extract
+{
+   my ($self, $state) = @_;
+   $state-{partial}-{$self} = 1;
+   if ($state-{interrupted}) {
+   die Interrupted;
+   }
+}
+
 sub install
 {
my ($self, $state) = @_;
if ($state-{interrupted}) {
die Interrupted;
}
-   $state-{partial}-{$self} = 1;
 }
 
 sub copy_info
@@ -352,6 +367,7 @@ package OpenBSD::PackingElement::FileBas
 use OpenBSD::Error;
 use File::Basename;
 use File::Path;
+use OpenBSD::Temp;
 
 sub prepare_for_addition
 {
@@ -375,53 +391,6 @@ sub prepare_for_addition
}
 }
 
-sub install
-{
-   my ($self, $state) = @_;
-   $self-SUPER::install($state);
-   my $fullname = $self-fullname;
-   my $destdir = $state-{destdir};
-   if ($fullname =~ m,^$state-{localbase}/share/doc/pkg-readmes/,) {
-   $state-{readmes}++;
-   }
-
-   if ($state-{extracted_first}) {
-   if ($state-{not}) {
-   $state-say(moving tempfile - #1,
-   $destdir.$fullname) if $state-verbose = 5;
-   return;
-   }
-   File::Path::mkpath(dirname($destdir.$fullname));
-   if (defined $self-{link}) {
-   link($destdir.$self-{link}, $destdir.$fullname);
-   } elsif (defined $self-{symlink}) {
-   symlink($self-{symlink}, $destdir.$fullname);
-   } else {
-   rename($self-{tempname}, $destdir.$fullname) or
-   $state-fatal(can't move #1 to #2: #3,
-   $self-{tempname}, $fullname, $!);
-   $state-say(moving #1 - #2,
-   $self-{tempname}, $destdir.$fullname)
-   if $state-verbose = 5;
-   undef $self-{tempname};
-   }
-   } else {
-   my $file = $self-prepare_to_extract($state);
-
-   $state-say(extracting #1, $destdir.$fullname)
-   if $state-verbose = 5;
-   if ($state-{not}) {
-   $state-{archive}-skip;
-   return;
-   } else {
-   $file-create;
-   $self-may_check_digest($file, $state);
-
-   }
-   }
-   $self-set_modes($state, $destdir.$fullname);
-}
-
 sub prepare_to_extract
 {
my ($self, $state) = @_;
@@ -467,6 +436,96 @@ sub prepare_to_extract
return $file;
 }
 
+sub extract
+{
+   my ($self, $state) = @_;
+
+   my $file = $self-prepare_to_extract($state);
+
+   if (defined $self-{link} || defined $self-{symlink}) {
+   $state-{archive}-skip;
+   return;
+   }
+
+   $self-SUPER::extract($state);
+
+   # figure out a safe directory where to put the temp file
+   my $d = dirname($file-{destdir}.$file-name);
+   # we go back up until we find an existing directory.
+   # hopefully this will be on the same file system.
+   while (!-d $d  -e _ || defined $state-{noshadow}-{$d}) {
+   $d = dirname($d);
+   }
+   if ($state-{not}) {
+   $state-say(extracting tempfile 

Re: signify untrusted comments

2014-01-07 Thread Marc Espie
On Tue, Jan 07, 2014 at 04:54:59PM +, Christian Weisgerber wrote:
  (I'm also open to reconsidering whether keys should include
  identifiers. Perhaps a random id created during key generation? Just
  enough to say you're using the wrong key.)
 
 I'm in favor.

Yeah, looks like a good idea.
64 bits random number ?



Re: signify untrusted comments

2014-01-07 Thread Marc Espie
On Tue, Jan 07, 2014 at 01:17:12PM -0500, Ted Unangst wrote:
 On Tue, Jan 07, 2014 at 16:54, Christian Weisgerber wrote:
  Ted Unangst t...@tedunangst.com wrote:
  
  To that end, I think the comment should be marked as untrusted, and
  signify should even check that it says untrusted. Hopefully this makes
  it a little harder to con somebody into believing the comment actually
  should be trusted.
  
  I think somebody who can be conned into accepting a key will not
  understand or think about untrusted comment.
 
 hmm.
 1. people who don't trust strangers. we don't need to protect them.
 2. people who do trust strangers. we can't protect them.
 3. people who are initially suspicious, but have a mistaken belief
 that they can spot mischief. maybe they will be fooled regardless, but
 I don't want to provide anything that can be used as leverage against
 them.
 
  
  (I'm also open to reconsidering whether keys should include
  identifiers. Perhaps a random id created during key generation? Just
  enough to say you're using the wrong key.)
  
  I'm in favor.
 
 Here's a diff with that too. I print out the fingerprints after a
 mismatch, but the fingerprints are in the opaque part of the file, so
 I'm not sure how useful that is. At least it provides a hint to check
 the command line arguments.

I think the fingerprint mismatch error message is too technical.
errx(1, Signature failed: checked against the wrong key);
looks like enough for me.

Apart from that, I'm okay with that code.

If you want to display fingerprints, well, you have to make it possible
to display fingerprints for any key-like message.
Not that it's much code, but is it really necessary ?



Re: signify embedded sigs

2014-01-08 Thread Marc Espie
On Wed, Jan 08, 2014 at 03:53:32PM -0500, Ted Unangst wrote:
 This may or may not be useful. We can embed the message after the
 signature and reduce the number of files in half. It's not a lot of
 code. (This is how the crypto_sign API originally worked, but we've
 since added some metadata, so we still need to destruct and
 reconstruct the crypto_sign inputs and outputs.)
 
 The -e option is added. It works a lot like gzip/gunzip by default.
 You sign a message, creating message.txt.sig:
   signify -e -S message.txt
 You verify message.txt.sig, creating message.txt:
   signify -e -V message.txt
 Note you always specify the message name. I've cleaned up the source a
 bit to disambiguate between input and message. And of course, -o
 works as before.

I'm not fond of embedded as a global variable, especially since it's
only used by two functions.

Can you pass it as a parameter, so that it's more explicit where it's used ?



fw_update experiments

2014-01-10 Thread Marc Espie
We're currently trying out some new stuff, purely as an experiment (the
files in /etc/signify all say so explicitly, don't put any confidence in
them)

if you experience trouble running fw_update, make sure you have
a new snapshot.

if you're trying to use pkg_add directly to grab/update firmwares, make
sure to use -DFW_UPDATE on those.



Re: md5/sha256 -c option

2014-01-10 Thread Marc Espie
On Fri, Jan 10, 2014 at 08:05:44AM +0001, Jason McIntyre wrote:
 On Fri, Jan 10, 2014 at 01:01:46AM -0700, Theo de Raadt wrote:
   well then this means the description of -c is very poor. i would look
   for a fix there, not in SYNOPSIS.
  
  But look closer, the synopsis is wrong:
  
   sha256 [-bpqrtx] [-c [checklist ...]] [-s string] [file ...]
  
  It is not regular.  When does checklist ... stop and file ... start?
  
  No matter what, that line needs a change.  At minimum,
  
   sha256 [-bcpqrtx] [-s string] [file ...]
  
  With wording changes below.
 
 right, so i support the idea of making a single synopsis as clear as
 possible, and improving the text.

Sorry, that's not always possible.

I think you're completely wrong and pig-headed about that one.

(You probably have a natural bias towards full fledged text, whereas
coder-type people will tend to trust the quick summary that looks like
code, and won't have a problem parsing more information from there.)

The main reason we have several synopsis is that those are actually
related *commands* that happen to share the same *filename*.

When I'm in a hurry, I check the SYNOPSIS.

When I type wrong options, the first thing I see is the SYNOPSIS, not
the manpage.

Oh, hey, let's read the full manpage... hum, lots of text. Oh, hey, let's
check what option goes with what.

One-line synopsis for signify(1):
 signify [-neGVI] [-o sigfile] [-p pubkey] [-s seckey] [message]

What's currently in there:
 signify -G [-n] -p pubkey -s seckey
 signify -I [-o sigfile] [-p pubkey] [-s seckey]
 signify -S [-e] [-o sigfile] -s seckey message
 signify -V [-e] [-o sigfile] -p pubkey message

Sorry, the current version is *ways* more useful.

In case options are completely separated by some kind of mode, it's
much much clearer to put things as separate SYNOPSIS.



signed packages

2014-01-17 Thread Marc Espie
It's probably time to talk about it.

Yes, we are now distributing signed packages.  A lot of people have probably
noticed because there was a key mismatch on at least one batch of signed
packages.

Obviously, we haven't finished testing yet.

Don't read too much into that.  Signed packages just mean you can use
an insecure medium, such as ftp, to download packages: if the key matches,
it means the package hasn't been tampered with since it was signed.

The cryptographic framework used to sign packages is called signify(1),
mostly written by Ted Unangst, with a lot of feedback from (mostly) Theo
and I.

The signing framework in pkg_add/pkg_create is much older than that, if
was written for x509 a few years ago, but signify(1) will probably be more
robust and ways simpler.  In particular, there's no chain-of-trust, so
you keep complete control on the sources YOU trust.

Signatures should be transparent in use: the package is opened, the 
packing-list signature is checked, and then files are checksummed while
extracted against the packing-list embedded checksums (there are provisions
to ensure any dangerous meta-data is also encoded in the packing-list as
@mode/@user/@group annotations.

So, barring problems, you shouldn't even notice signatures.



Re: pkg_add (pkg.conf): option to require signed packages

2014-01-17 Thread Marc Espie
On Fri, Jan 17, 2014 at 09:59:03AM +0100, Sébastien Marie wrote:
 On Thu, Jan 16, 2014 at 10:02:22AM +, Stuart Henderson wrote:
  On 2014/01/16 08:53, Sébastien Marie wrote:
   Hi,
   
   Does it make sens to have an option to require package to be signed ?
  
  It makes more sense to just enable that by default, when we are happy
  with the infrastructure and usage.
  
 
 I saw enable by default more as long term purpose. The patch would
 permit to easily test it...

Enable by default is trivial to do. Look around the code that says
check_signature in OpenBSD/PkgAdd.pm, I'm sure you can figure out the
change.



Re: signed packages

2014-01-17 Thread Marc Espie
On Fri, Jan 17, 2014 at 12:09:31PM -0500, sven falempin wrote:
 
Awesome.
 * the public one on the client openBSD
 * the private one on the builder
is there a new make command in ports to sign ? like make sign ? make
resign ?

See signify(1), pkg_add(1), pkg_create(1), bsd.port.mk(5) (look for
SIGNING_PARAMETERS).

Packages can be signed during build, or later.
There's no new command, pkg_create(1) is used for creating signed packages.



Re: signed packages

2014-01-17 Thread Marc Espie
On Fri, Jan 17, 2014 at 06:23:53PM +0100, Marc Espie wrote:
 On Fri, Jan 17, 2014 at 12:09:31PM -0500, sven falempin wrote:
  
 Awesome.
 Â * the public one on the client openBSD
 Â * the private one on the builder
 is there a new make command in ports to sign ? like make sign ? make
 resign ?
 
 See signify(1), pkg_add(1), pkg_create(1), bsd.port.mk(5) (look for
 SIGNING_PARAMETERS).
 
 Packages can be signed during build, or later.
 There's no new command, pkg_create(1) is used for creating signed packages.

Note that things are still WILDLY changing.  I assume that by now,
lots of people have noticed the signed stuff.   This is still a moving
target (working quite well IMO).



Re: signed packages

2014-01-18 Thread Marc Espie
On Fri, Jan 17, 2014 at 12:39:49PM -0500, sven falempin wrote:
 i read the manuals , and well , i am still unsure,
 
 if i put SIGNER=bob in the package configuration
 
 then it will be signed with
 
 /etc/signify/bob.sec
 
 having to read 4 different manual page to get this is strange :p

No, that part got simpler.

Keys are currently under /etc/signify
They *must* be there for the public keys.

Keys for signed packages should match *pkg.sec /  *pkg.pub
(distinguished by function: firmware keys end in fw.sec / fw.pub)

Read signify(1) to generate the keys.

Say:
signify -G -n -s sven-pkg.sec -p sven-pkg.pub

For signing while building, just set
SIGNING_PARAMETERS = -s signify -s /etc/signify/sven-pkg.sec

for signing after building, do something like:
cd /usr/ports/packages/${ARCH}
mkdir signed
pkg_create -j4 -v -s signify -s /etc/signify/sven-pkg.sec -o signed -S all

That's all there is to it.  If both the pubkey and privkey are present, the 
first signed package written out will be checked (signify keys don't 
carry any identity, they just have a fingerprint, so key mismatches are 
easy to create if you're not careful - the signed package carries a 
@signer sven-pkg  annotation to select the correct key).

pkg_add will trust keys under /etc/signify   that match *pkg.pub

If you really want to trust a specific key *only*,
pkg_add -DSIGNER=sven-pkg ...

If some booboo happened, pkg_add -Dnosig   will not check sigs at all...



Re: signed packages

2014-01-22 Thread Marc Espie
On Wed, Jan 22, 2014 at 01:46:33PM +0400, Loganaden Velvindron wrote:
  The signing framework in pkg_add/pkg_create is much older than that, if
  was written for x509 a few years ago, but signify(1) will probably be more
  robust and ways simpler.  In particular, there's no chain-of-trust, so
  you keep complete control on the sources YOU trust.
 
 Can you please elborate more on the trusting part ?
 
 Both DNSSEC and RPKI have a root anchor that we're all supposed to trust,
 and your model is different.

There's no chain of trust.

pkg_add trusts pub keys under /etc/signify that end in *pkg.pub
(respectively *fw.pub for firmwares).

Put shit there - get shit out.

the only way to get keys in there is:
- base install,
- explicitly putting keys there as root.

There's nothing more automated. Keys are not certified nor anything.



Re: signed packages

2014-01-23 Thread Marc Espie
A huge swath of clean-up has just hit the trees.

Most specifically, now that it works, the signing-only code has been
moved into a separate pkg_sign command.

This is partly for documentation purpose: it's much simpler to document
the parameters to that command separately, instead of as additions to
pkg_create(1) proper.

pkg_create(1) still retains the ability to create signed packages on the
fly, if people want to create their own signed packages (not recommanded
for really paranoid people, since the build process can be dirty),
but signing existing packages is really a mostly independent process
(the only common part is signing packing-lists) so it makes more sense
as a separate command.



pkg_add tests

2014-01-24 Thread Marc Espie
You should realize that the pkg tools have gone thru *major internal changes*
over the last month or so.

If you see weird things happening, now is a really good time to report them,
before it's too late...


Also, the 5.4 - 5.5 update is going to be fun: there was a kernel flag day.

The proper way to update packages will be along the lines of:
in 5.4:

pkg_info -mq list
pkg_delete *

(update to 5.5)
in 5.5, install new packages:
pkg_add -z -l list


Again, there might be issues.   Now is a very good time to start looking...



Re: pkg_add tests

2014-01-25 Thread Marc Espie
On Fri, Jan 24, 2014 at 11:15:21PM -0600, Shawn K. Quinn wrote:
 On Fri, Jan 24, 2014, at 06:50 AM, Marc Espie wrote:
  You should realize that the pkg tools have gone thru *major internal
  changes*
  over the last month or so.
  
  If you see weird things happening, now is a really good time to report
  them,
  before it's too late...
  
  
  Also, the 5.4 - 5.5 update is going to be fun: there was a kernel flag
  day.
  
  The proper way to update packages will be along the lines of:
  in 5.4:
  
  pkg_info -mq list
  pkg_delete *
  
  (update to 5.5)
  in 5.5, install new packages:
  pkg_add -z -l list
  
  
  Again, there might be issues.   Now is a very good time to start
  looking...

 Do we have to do any of this if we have been following snapshots or
 -current? Are there any other gotchas related to these for those of us
 who are running snapshots or -current? I haven't had any obvious Bad
 Things jump up and bite me yet, want to keep it that way.
 (Software-related, at least; having my onboard SATA controller fall down
 and go boom kind of falls outside the realm of that.)

Nope.  You already went over the hurdle of timestamps.
But if you have some 5.4 installs, or if people want to check how 5.4-5.5
goes, well, it will soon be the time to do so.


Basically, the big change for them is that pkg_add now always  installs the
quirks package first, and it uses it to hunt for package names on fuzzy
installs. e.g., the above scenario that I outlined should now get people
over the 5.4 - 5.5 hurdle   *while handling package renames correctly*.

Obviously, this hasn't been tested too much yet...



As for the usefulness of testing, for instance, I made a large 
booboo in an extra check in pkg_sign.  Fortunately rpe@ caught it 
fairly early, so a fixed amd64 snap should be on its way out soon 
(and it was just garbled archives, so rsync should chew on that one 
really fast compared to the usual slooow trickle of snapshots out).



Re: your mail

2014-01-25 Thread Marc Espie
On Fri, Jan 24, 2014 at 08:08:29PM -0500, Ted Unangst wrote:
 I generally associate negative connotations with so-called, as in
 the so-called free world. I wouldn't use it just to name something,
 as in the kernel is written in the so-called C language. so-called
 implies it's called this, but it's not.
 
 Two imo dubious occurrences in the install notes. It's not a so-called
 MBR partition; it is an MBR partition. Similarly with hppa LIF.
 There's one other use in loongson about initrd which seems ok.
 

I agree with so-called having negative connotations.

I think both those instances are using it intentionally, namely there
are nasty surprises in some MBR blocks that are not covered by
the so-called MBR standard.

Probably likewise for hppa LIF...



Re: your mail

2014-01-26 Thread Marc Espie
On Sat, Jan 25, 2014 at 11:04:21PM -0600, Shawn K. Quinn wrote:
 On Sat, Jan 25, 2014, at 08:53 AM, Marc Espie wrote:
  I agree with so-called having negative connotations.
  
  I think both those instances are using it intentionally, namely there
  are nasty surprises in some MBR blocks that are not covered by
  the so-called MBR standard.
 
 There's an actual MBR standard? If so, maintained by whom?

Precisely.

MBR is a dark art.

We know we produce valid MBR blocks because machines don't blow up on us...



churn in pkg_add code

2014-02-09 Thread Marc Espie
hopefully, things should be back to normal.
last commit was

revision 1.37 of PackageLocation.pm
date: 2014/02/09 11:13:59;  author: espie;  state: Exp;  lines: +4 -3;
do a better job in wipe_info: close files and reap children.
also wipe every state variable... fixes reget update_info in fringe
cases (this should fix nigel@'s problems...)


pkg_add memory usage is back down to reasonable levels, after two large
fixes, and a heck of a lot of small changes... with tidying up and minor
bug fixes along the way (well, some of them probably not so minor).

Just to put things in perspective, my big testcase backed down from
480M to 280M... and in more normal cases, usage stops growing.

The two large fixes were to prevent ProgressMeter/Term.pm from capturing
full plists (with their sha values and all that long post the peremption
date) and to limit the number of locally opened IO::Uncompress::Gunzip handles
to 3 (issue was already there for fork()d gunzip, but as extra processes
hanging around...).  Small fixes include shrinking conflict representation,
bringing full packing-lists in only AFTER solving dependencies, not storing
full objects for simple shared dirs, and a lot lot of work on caching
disk access better (PackageLocation/Repository cleanup fixes).

Kuddoes to pirofti@  for suggesting some awesome introspection tools
(Devel::Leak::Object, Devel::BackRef, Devel::Gladiator. That stuff rocks).


So if you notice suspicious things and you have the current version, report
them asap... there's precious little time to fix them before the release.



Re: churn in pkg_add code

2014-02-09 Thread Marc Espie
On Sun, Feb 09, 2014 at 01:16:38PM +0100, Marc Espie wrote:
 hopefully, things should be back to normal.
 last commit was
 
 revision 1.37 of PackageLocation.pm
 date: 2014/02/09 11:13:59;  author: espie;  state: Exp;  lines: +4 -3;
 do a better job in wipe_info: close files and reap children.
 also wipe every state variable... fixes reget update_info in fringe
 cases (this should fix nigel@'s problems...)

And of course, I fucked something obvious up...

So the good commit is revision 1.38



Re: churn in pkg_add code

2014-02-09 Thread Marc Espie
On Sun, Feb 09, 2014 at 01:16:38PM +0100, Marc Espie wrote:
 hopefully, things should be back to normal.
 last commit was
 
 revision 1.37 of PackageLocation.pm
 date: 2014/02/09 11:13:59;  author: espie;  state: Exp;  lines: +4 -3;
 do a better job in wipe_info: close files and reap children.
 also wipe every state variable... fixes reget update_info in fringe
 cases (this should fix nigel@'s problems...)

And a second fix... revision 1.39 is the right one...

Sending an mail to say  problem fixed always works. If you don't do it,
the bugs stay hidden in the woodwork :)



Re: release(8) patch

2014-02-12 Thread Marc Espie
On Wed, Feb 12, 2014 at 08:16:26AM +0100, Mark Kettenis wrote:
 
 Just make sure you set $ARCH in your .profile.

If you feel like it, how about a patch for /etc/skel ?

It's not set by default !



Re: man.conf mandoc -Tlocale

2014-02-16 Thread Marc Espie
On Sun, Feb 16, 2014 at 03:11:07PM +0100, Ingo Schwarze wrote:
 Few, probably, because mandoc(1) is fast enough that we run in on
 demand whenever possible and usually avoid preformatting anything
 during builds, or where we do preformat, we use groff for that, anyway.
 But even missing one single instance that is hiding somewhere
 would just be pointless disruption in a release.

We can try specifically poisoning mandoc in ports land, so that you would
see whenever it's invoked.

Don't remember if you're familiar with that, but it's fairly trivial to do:
the ports tree runs all the builds with PATH starting with
${WRKDIR}/bin

you can poison mandoc very easily around line 2477 of bsd.port.mk.



Re: yacc stdc output

2014-02-18 Thread Marc Espie
On Tue, Feb 18, 2014 at 02:50:53PM -0700, Todd C. Miller wrote:
 On Tue, 18 Feb 2014 16:38:07 -0500, Ted Unangst wrote:
 
  True, I phrased that poorly. What I'm assuming is that the code will
  be built by a compiler that supports the const keyword. Or in other
  words, if you're using a 25 year old cross compiler, I don't think
  it's unreasonable to expect you to dig up a 25 year old cross yacc
  either (or the yacc from openbsd 5.4).
  
  Who is actually using yacc to generate code for a compiler that
  doesn't accept const?
 
 Sudo 1.7.x still compiles on KR, mostly for the benefit of the
 bundled HP-UX compiler that has C89 support disabled.
 
  - todd
Come on Jim, it's dead.



Re: missing ports.tar.gz in snapshot

2014-03-06 Thread Marc Espie
On Thu, Mar 06, 2014 at 03:18:38PM +, Devin Ceartas wrote:
 
 Are all the ports in the packages in snapshots? I seem to remember needing to 
 build some of the things I use to test, but maybe that has changed. 

There are a few ports not in snapshots, stuff that is not authorized for
ftp redistribution.

Accidents happen. Sometimes, stuff does not build. That's more frequent
on less used architectures.

Most builds on amd64 give 100%. All packages referenced in the ports tree
that should be on amd64 do build.



using dpb for chroot builds

2014-03-14 Thread Marc Espie
-current work, again.

Doing chroot on distant hosts involves chroot, obviously... This doesn't
work too well with sudo (the processes tends to be hard to kill, so if
you've got to interrupt dpb, ouch).

There are no exact instructions for using dpb in chroot settings.

Obviously, you need a fairly complete chroot install (duh, including
devices (so get rid of nosuid,nodev on that partition).

For local host, put everything in the chroot, do the chroot manually, THEN
build.

For distant hosts, it's slightly simpler to connect as root, because then
dpb can do chroot -u user /build *without* needing sudo, and hence
resulting in distant makes that are trivial to ^Z/^C if the need arises...

(-current gained changes to support that: initing a core runs whoami, thus
it knows whether you're root on the distant machine, and if so, chroot
runs directly without sudo).



HEADS UP: librt revert

2014-03-23 Thread Marc Espie
kili@  just committed a revert of the librt addition in src and corresponding
patches in ports.

If you've built a tree with librt, you want to
# rm -f /usr/lib/librt.a

This lib was added to facilitate porting software, as posix asks for it.
but since it's only a stub, it was only added as a static library. No-one
would approve a shared library, as that would waste space.

Unfortunately, libtool (ours and gnu's) don't cope well with static-only
libraries.  The untested commit of librt in source broke the ports tree.

Specifically, programs such as x11/vlc, multimedia/xine-lib, or lang/php
would no longer build (all 3 are using mutant versions of gnu libtool).

It's possible further breakage would happen, but those 3 were broken.

After almost a week (!), there has been exactly zero activity to fix the
breakage.  No-one volunteered to do the requisite patches, and well,
these ports are not exactly low profile, we need a ports tree in working
condition to be able to conduct other work (such as the pending update to
perl, or some other clean-up work).

So for now, the librt experiment got reverted. Maybe temporarily ().
If people really want it in, they had better be willing to figure out
how to fix the libtool breakage first...



Re: HEADS UP: librt revert

2014-03-23 Thread Marc Espie
On Sun, Mar 23, 2014 at 10:46:08PM +0400, Loganaden Velvindron wrote:
 On Sun, Mar 23, 2014 at 10:34 PM, Marc Espie es...@nerim.net wrote:
  kili@  just committed a revert of the librt addition in src and 
  corresponding
  patches in ports.
 
  If you've built a tree with librt, you want to
  # rm -f /usr/lib/librt.a
 
  This lib was added to facilitate porting software, as posix asks for it.
  but since it's only a stub, it was only added as a static library. No-one
  would approve a shared library, as that would waste space.
 
  Unfortunately, libtool (ours and gnu's) don't cope well with static-only
  libraries.  The untested commit of librt in source broke the ports tree.
 
  Specifically, programs such as x11/vlc, multimedia/xine-lib, or lang/php
  would no longer build (all 3 are using mutant versions of gnu libtool).
 
  It's possible further breakage would happen, but those 3 were broken.
 
  After almost a week (!), there has been exactly zero activity to fix the
  breakage.  No-one volunteered to do the requisite patches, and well,
  these ports are not exactly low profile, we need a ports tree in working
  condition to be able to conduct other work (such as the pending update to
  perl, or some other clean-up work).
 
 That's very sad. I get the impression that there aren't many active
 developers (?)

No, it's more a question of people being busy with other things at times.

There is a flurry of important activity in preparation for various things.

It's one of those times when having a tree that works is important.

A week is an fairly long time for the tree to stay broken, in OpenBSD 
terms.

The revert is a pain, but much less pain than figuring out what to do to
get the affected ports to build again in a clean fashion.

And again, the concerned people were not available to do timely fixes.

As for myself, for instance, I'm busy with perl 5.18, and there are some
interesting new features (randomization of the hash function) that have
exposed other interesting problems in our libtool...

you can't wage a war on several fronts.  The librt breakage was 
fairly distracting for a group of people that are working on other stuff.

After a while, you just have to do something that lets the tree build.



Re: HEADS UP: librt revert

2014-03-23 Thread Marc Espie
On Sun, Mar 23, 2014 at 05:05:06PM -0400, STeve Andre' wrote:
 On 03/23/14 14:34, Marc Espie wrote:
 kili@  just committed a revert of the librt addition in src and corresponding
 patches in ports.
 
 If you've built a tree with librt, you want to
 # rm -f /usr/lib/librt.a
 
 Shouldn't that be librt*a to get rid of librt_p.a too?

Yep, it should.  Though it's less likely to cause issues...



xucred change

2014-03-24 Thread Marc Espie
I found out, after a bit of head-scratching, that you will have to
make sure you're running the newly compiled mountd if you want cred-mapping
to make sense.

e.g., the usual build kernel, reboot, make build is not enough. You
also have to kill mountd and start the new one, otherwise 
-maproot=root won't work.



heads-up: your ports *will break*

2014-04-08 Thread Marc Espie
Got fed up of the nth report flavors are broken.

The cause is always the same: for packages to build correctly, your setup
*must* propagate some env variables thru sudo during the fake/packaging stage.

So, I finally added up a variable that's there just to trigger an error in
case your setup is not squeaky clean.

Accordingly, be sure to integrate the latest sudoers modification (coming
to a snapshot + sysmerge near you soon), and you get a fairly explicit
clue from bsd.port.mk now...

Error: sudo does not let env variables through



Re: Check syntax of Perl stuff at build time

2014-04-16 Thread Marc Espie
On Wed, Apr 16, 2014 at 08:07:55PM +0400, Vadim Zhukov wrote:
 This is a PoC I'm currently using while hacking at our libtool. What
 I dream about is putting such code in /usr/share/bsd.perl.mk or such,
 allowing to run checks for all Perl-based tools in base.It's better to
 know something is broken at build time rather after install, isn't it?
 
 If idea and actual implementation is approved, I'll prepare and mail
 out a patch for /usr/share/bsd.perl.mk (or whatever) and actual
 Makefiles.
 
 Any recommendations or other comments are welcome. Maybe, this should
 be put in some other file or done differently... Thanks in advance.
 --
 WBR,
   Vadim Zhukov
 
 
 Index: Makefile
 ===
 RCS file: /cvs/src/usr.bin/libtool/Makefile,v
 retrieving revision 1.7
 diff -u -p -r1.7 Makefile
 --- Makefile  16 Apr 2014 10:31:27 -  1.7
 +++ Makefile  16 Apr 2014 15:58:30 -
 @@ -37,4 +37,12 @@ realinstall: 
   ${INSTALL} ${INSTALL_COPY} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \
   ${.CURDIR}/libtool ${DESTDIR}${BINDIR}/libtool
  
 +PERLCHECK=   libtool ${PACKAGES}
 +.for pm in libtool ${PERLCHECK}
 +CLEANFILES +=${pm:C@/@_@g}.perlcheck
 +all: ${pm:C@/@_@g}.perlcheck
 +${pm:C@/@_@g}.perlcheck: ${pm}
 + rm -f $@; perl -c ${.CURDIR}/${pm}  touch $@
 +.endfor
 +
  .include bsd.prog.mk

Nope, doesn't always work. use needs to actually eval part of the code,
and so for complicated code, -c will just check the syntax and say there
are issues, because there is some code which really really needs the use
to be evaluated to be syntaxicallt correct.



Re: ffs2 boot

2014-04-17 Thread Marc Espie
On Thu, Apr 17, 2014 at 12:22:44PM -0500, Adam Thompson wrote:
 Given the single-threaded nature of much of the kernel, what applications do 
 you run where multiple CPUs makes much of a difference to OpenBSD?
 

You're a narrow-minded troll.



Re: openssl's *strlcy

2014-04-19 Thread Marc Espie
On Fri, Apr 18, 2014 at 09:41:47PM -0400, Jacob L. Leifman wrote:
 I'm guessing that openssl was incorporated into OpenBSD base without 
 prior sufficient audit by the OBSD devs because it was presumed to have 
 better auditing / quality control upstream given its security critical 
 nature and function. (A number of devs have commented in the past about 
 the [lack of] code style, but I get the impression no-one expected the 
 degree of *sloppiness* now being uncovered.)  So here's a question, are 
 there any other chunks of code that have been imported en-mass from an 
 upstream source that could/should use an audit? especially, something 
 that some of us non-developers might be able to assist with?

No, you're mistaken. We've known for a while it was on the very dirty
side, but there are obvious human reasons because of which people were 
reluctant to dive in.

Note that it's on a par with a lot of opensource code out there, 
unfortunately.  At some point, you have to make choices, as the
amount of time and manpower we have is limited.



openssl fallout, dealing with it

2014-04-19 Thread Marc Espie
Since the tree wants to move fast, here's a sketch of how we deal
with ports breakage.

- someone makes sure everyone is aware there's breakage. ports and src
should communicate so we assert what got removed intentionally, what
got removed by accident.

- if the removal is not accidental, fix critical ports asap. By critical,
I mean stuff such as, oh say, ruby, python, openldap, kdelibs, since that
tends to take out a large chunk of the ports tree, thus effectively hiding
further breakage.

- iterate until most of the turds are gone.



.libs handling

2011-02-19 Thread Marc Espie
At least jeremy@ and landry@ got confused about this, so I expect more people
to ask.

The algorithms for package updates are a bit complicated, to say the least.
When you update a package with shared libraries, those can get split off
into a separate package with a .lib name (which is well known).

This package can be removed manually later, once a full upgrade has been done
and nothing depends on it (it's not done automatically because user-compiled
programs may depend on it... I intend to have a pkg_create -u mode (user) to
be able to quickly register user programs so that we CAN remove them
automatically). Anyways...

In the meantime, those .libs* will participate in updates.
In modern pkg_add, pkg_add -u will organize packages to update as updatesets:
the smallest set of packages that need to be updated together because of
interdependencies/conflicts.

Guess what ? a .libs-foo-*   will always get pulled in when you update foo,
because those two packages belong to the same software and contain files
that may conflict. That's why you'll see lines like
.libs-foo-1.0 + foo-2.0 - foo-3.0

That's because pkg_add needs to know what's going on.
What you're not going to see, though, is the creation of new .libs (unless
you use -vvv).

But that is necessary. That way, if you need to downgrade  back from
foo-2.0 to foo-1.0, pkg_add knows enough about it to be able to merge
back .libs-foo-1.0 into foo-1.0 instead of complaining about a conflict and
stopping.

I should probably show the creation of new .libs? that might make things more
simple to understand.  What happened is that this algorithm evolved, it's
one of those details I hadn't figured out yet when we embarked on correct
safe updates...



Re: horribly slow fsck_ffs pass1 performance

2011-04-10 Thread Marc Espie
On Sat, Apr 02, 2011 at 01:45:36PM -0500, Amit Kulkarni wrote:
 Hi,
 
 I am replying in a single email.
 
 I do a fsck once in a while, not regular. In the last 6-8 months I
 might have done it about 5 times. And I did it multi-user the few
 times I did it, but plan on doing it single user in future and I do
 plan to do it monthly. After seeing the messages when you fsck, it is
 better to do it monthly. FreeBSD which is the origin of FFS does a
 background fsck, and if Kirk McCusick feels so strongly I will do it
 too. (I remember somebody talking about having background fsck here on
 a openbsd list, but I forgot who it was).

This is completely stupid.

What do you trust more ? your file system, or fsck ?

oth have bugs ! I'm sure of it ! 

so, if you run fsck, it's likely
you're going to run into fsck bugs eventually (and trying fsck on a mounted
partition was really, really stupid). Whereas, if you don't run fsck, you're
going to run into fs bugs eventually.

Now, consider this: the fs code is very heavily tested. People use it 24 hours
a day, 365 days a year.

Compared to THAT, the fsck code is very lightly tested. It's run only once
in a while, when the power shuts down, or when you update your machines.

What is more likely ? corrupting a perfectly sane filesystem by running
fsck on it (which has MORE code paths to correct problems and is usually
run on corrupted filesystems), or having an unseen bug in the fs code that
affects only you and that fsck would be able to see ?



Re: horribly slow fsck_ffs pass1 performance

2011-04-10 Thread Marc Espie
On Sun, Apr 10, 2011 at 11:27:41AM +, Miod Vallat wrote:
  Now, consider this: the fs code is very heavily tested. People use it 24 
  hours
  a day, 365 days a year.
 
 Except on leap years, of course. Those years see even more real-life
 testing happening!

Good point. Maybe we should go to single user and run fsck in a loop on
february 29th.



Re: horribly slow fsck_ffs pass1 performance

2011-04-10 Thread Marc Espie
On Sun, Apr 10, 2011 at 02:40:09PM +0200, David Vasek wrote:
 On Sun, 10 Apr 2011, Marc Espie wrote:
 
 This is completely stupid.
 
 What do you trust more ? your file system, or fsck ?
 
 oth have bugs ! I'm sure of it !
 
 so, if you run fsck, it's likely
 you're going to run into fsck bugs eventually (and trying fsck on a mounted
 partition was really, really stupid).
 
 There is an optional fsck -n of (mostly) mounted filesystems in
 /etc/daily. In what cases is this automated check intended to be
 used?

It's code we got from BSD 4.4 lite2. This specific chunk was brought
by millert@ in rev 1.17.

I would venture that apart from file system hackers, or people with really
flaky hardware, no-one should set that specific test...



the importance of writing documentation

2011-04-10 Thread Marc Espie
Like most people, I have my pet peeves. I consider that documentation
is very important to code quality.

I ran into a school-book example a week ago, so I decided to share what I
mean.

There was this problem with dpb where some packages in a MULTI_PACKAGES show 
up as Errors when they don't build on a given architecture, even though 
they're marked as NOT_FOR_ARCHS-sub = ...

I had fixed that particular way of building things by getting those to
vanish when building all packages, so that the build would not error out,
but dpb was still getting them as errors (and the remaining subpackages were
built correclty)

Looking more closely, dpb actually dumps information for all subpackages,
including the non-building ones. Not including the non-building ones would
not work, because the listing job would retry and retry. Nope, what was needed
was the same mechanism that lets dpb ignore regular packages, that is the
IGNORE variable.

So, all that was needed was to set the IGNORE variable on a subpackage
dependent basis, and dpb would happily trudge along.

Thus, a patch was born, tested and committed that would make IGNORE subpackage
dependent. I tried to make it as simple as could be:

(the relevant part is setting all
IGNORE-sub, then having
IGNORE = ${IGNORE-sub} and testing IGNORE,
plus marking IGNORE as subpackage dependent for dump-vars)


Index: bsd.port.mk
===
RCS file: /home/openbsd/cvs/ports/infrastructure/mk/bsd.port.mk,v
retrieving revision 1.1074
retrieving revision 1.1075
diff -u -p -r1.1074 -r1.1075
--- bsd.port.mk 28 Mar 2011 00:16:13 -  1.1074
+++ bsd.port.mk 3 Apr 2011 07:19:05 -   1.1075
@@ -92,11 +92,11 @@ _SHSCRIPT = sh ${PORTSDIR}/infrastructur
 _PERLSCRIPT = perl ${PORTSDIR}/infrastructure/bin
 
 # All variables relevant to the port's description
-_ALL_VARIABLES = BUILD_DEPENDS IGNORE IS_INTERACTIVE \
+_ALL_VARIABLES = BUILD_DEPENDS IS_INTERACTIVE \
SUBPACKAGE MULTI_PACKAGES
 # and stuff needing to be MULTI_PACKAGE'd
 _ALL_VARIABLES_INDEXED = FULLPKGNAME RUN_DEPENDS LIB_DEPENDS \
-   PKG_ARCH
+   PKG_ARCH IGNORE 
 _ALL_VARIABLES_PER_ARCH =
 
 .if ${DPB:L:Mfetch}
@@ -1320,7 +1320,6 @@ IS_INTERACTIVE = Yes
 #
 # Don't build a port if it comes with the base system.
 
-IGNORE ?=
 TRY_BROKEN ?= No
 _IGNORE_REGRESS ?=
 .if defined(REGRESS_IS_INTERACTIVE)  defined(BATCH)
@@ -1328,44 +1327,49 @@ _IGNORE_REGRESS += has interactive test
 .elif !defined(REGRESS_IS_INTERACTIVE)  defined(INTERACTIVE)
 _IGNORE_REGRESS += does not have interactive tests
 .endif
-.if defined(IS_INTERACTIVE)  defined(BATCH)
-IGNORE += is an interactive port
-.elif !defined(IS_INTERACTIVE)  defined(INTERACTIVE)
-IGNORE += is not an interactive port
-.endif
-.if !exists(${X11BASE})
-IGNORE += building ports requires X11 but ${X11BASE} not found
-.endif
-.if !defined(_ARCH_OK${SUBPACKAGE}) || ${_ARCH_OK${SUBPACKAGE}} == 0
-.  if defined(ONLY_FOR_ARCHS${SUBPACKAGE})
-.if ${MACHINE_ARCH} == ${ARCH}
-IGNORE += is only for ${ONLY_FOR_ARCHS${SUBPACKAGE}}, not ${MACHINE_ARCH}
+
+.for _s in ${MULTI_PACKAGES}
+IGNORE${_s} ?=
+.  if defined(IS_INTERACTIVE)  defined(BATCH)
+IGNORE${_s} += is an interactive port
+.  elif !defined(IS_INTERACTIVE)  defined(INTERACTIVE)
+IGNORE${_s} += is not an interactive port
+.  endif
+.  if !exists(${X11BASE})
+IGNORE${_s} += building ports requires X11 but ${X11BASE} not found
+.  endif
+.  if !defined(_ARCH_OK${_s}) || ${_ARCH_OK${_s}} == 0
+.if defined(ONLY_FOR_ARCHS${_s})
+.  if ${MACHINE_ARCH} == ${ARCH}
+IGNORE${_s} += is only for ${ONLY_FOR_ARCHS${_s}}, not ${MACHINE_ARCH}
+.  else
+IGNORE${_s} += is only for ${ONLY_FOR_ARCHS${_s}}, not ${MACHINE_ARCH} 
\(${ARCH}\)
+.  endif
 .else
-IGNORE += is only for ${ONLY_FOR_ARCHS${SUBPACKAGE}}, not ${MACHINE_ARCH} 
\(${ARCH}\)
+IGNORE${_s} += is not for ${NOT_FOR_ARCHS}
 .endif
-.  else
-IGNORE += is not for ${NOT_FOR_ARCHS}
 .  endif
-.endif
-.if ${SHARED_ONLY:L} == yes  ${NO_SHARED_LIBS:L} == yes
-IGNORE += requires shared libraries
-.endif
-
-.if ${TRY_BROKEN:L} != yes
-.  if defined(BROKEN-${ARCH})
-IGNORE += is marked as broken for ${ARCH}: ${BROKEN-${ARCH}:Q}
+.  if ${SHARED_ONLY:L} == yes  ${NO_SHARED_LIBS:L} == yes
+IGNORE${_s} += requires shared libraries
 .  endif
-.  if ${MACHINE_ARCH} != ${ARCH}  defined(BROKEN-${MACHINE_ARCH})
-IGNORE += is marked as broken for ${MACHINE_ARCH}: 
${BROKEN-${MACHINE_ARCH}:Q}
+
+.  if ${TRY_BROKEN:L} != yes
+.if defined(BROKEN-${ARCH})
+IGNORE${_s} += is marked as broken for ${ARCH}: ${BROKEN-${ARCH}:Q}
+.endif
+.if ${MACHINE_ARCH} != ${ARCH}  defined(BROKEN-${MACHINE_ARCH})
+IGNORE${_s} += is marked as broken for ${MACHINE_ARCH}: 
${BROKEN-${MACHINE_ARCH}:Q}
+.endif
+.if defined(BROKEN) 
+IGNORE${_s} += is marked as broken: ${BROKEN:Q}
+.endif
 .  endif
-.  if defined(BROKEN) 
-IGNORE += is marked as 

Re: wprintf and friends

2011-04-22 Thread Marc Espie
I'm happy with this as well.

I think the libstdc++ is a good way to track i18n progress. When we implement
enough of it, more code will activate.

Even if we  don't use it much directly  wfscanf would be nice.

My guess is that having both wfprintf and wfscanf will lead to working wide
streams, for instance.



Re: aucat: suggest using -v100 ?

2011-05-27 Thread Marc Espie
On Fri, May 27, 2011 at 07:01:28AM +, Jacob Meuser wrote:
 can we please just accept that a 100% perfect for everyone solution
 is not really possible?

Of course not, that way lies mediocrity. ;-)



Re: En/disabling power button shutdown

2011-06-11 Thread Marc Espie
On Sat, Jun 11, 2011 at 09:06:05PM +0200, gilbert.fernan...@orange.fr wrote:
 One knob to be able to turn it on so when you're working
 on a machine, and console dies, to be able to shutdown
 properly and avoid a fsck.

One knob for the dwarfen guys that work in the IT mines, to bind the machines
and rule them all.



Re: grep \words\

2011-07-18 Thread Marc Espie
If we start adding this to some tools, it would start to make sense to
support it at the source.

Namely, conversion functions are a bit annoying, and it's always possible
to fuck it up. Plus, same code twice = same bugs.

Is there anything preventing us from adding non-standard flags to regcomp(3)
and do it directly there ?



kill old stuff in pkg tools

2011-08-20 Thread Marc Espie
INSTALL and DEINSTALL were deprecated in June 2007, and removed from
pkg_create in January 2010.

I think it's time to remove all traces of +MTREE_DIRS, +REQ, +INSTALL, 
+DEINSTALL
from all commands.

Index: pkg_add.1
===
RCS file: /home/openbsd/cvs/src/usr.sbin/pkg_add/pkg_add.1,v
retrieving revision 1.110
diff -u -p -r1.110 pkg_add.1
--- pkg_add.1   17 Aug 2011 10:48:27 -  1.110
+++ pkg_add.1   20 Aug 2011 10:11:57 -
@@ -282,8 +282,6 @@ external scripts may fail.
 .It Ar updatedepends
 force update even if forward dependencies no longer match.
 .El
-.It Fl I
-If scripts exist for a given package, do not execute them.
 .It Fl i
 Switch on interactive mode.
 .Nm
Index: pkg_delete.1
===
RCS file: /home/openbsd/cvs/src/usr.sbin/pkg_add/pkg_delete.1,v
retrieving revision 1.41
diff -u -p -r1.41 pkg_delete.1
--- pkg_delete.117 Aug 2011 10:48:27 -  1.41
+++ pkg_delete.120 Aug 2011 10:12:26 -
@@ -92,8 +92,6 @@ uninstall even if not running as root.
 .It Ar scripts
 external scripts may fail.
 .El
-.It Fl I
-If scripts exist for a given package, do not execute them.
 .It Fl i
 Switch on interactive mode.
 .Nm
Index: OpenBSD/Add.pm
===
RCS file: /home/openbsd/cvs/src/usr.sbin/pkg_add/OpenBSD/Add.pm,v
retrieving revision 1.124
diff -u -p -r1.124 Add.pm
--- OpenBSD/Add.pm  12 Jul 2011 10:09:52 -  1.124
+++ OpenBSD/Add.pm  20 Aug 2011 10:08:36 -
@@ -707,13 +707,6 @@ sub copy_info
$self-fullname, $dest, $!);
 }
 
-package OpenBSD::PackingElement::FINSTALL;
-sub install
-{
-   my ($self, $state) = @_;
-   $self-run($state, 'PRE-INSTALL');
-}
-
 package OpenBSD::PackingElement::FCONTENTS;
 sub copy_info
 {
Index: OpenBSD/Delete.pm
===
RCS file: /home/openbsd/cvs/src/usr.sbin/pkg_add/OpenBSD/Delete.pm,v
retrieving revision 1.115
diff -u -p -r1.115 Delete.pm
--- OpenBSD/Delete.pm   12 Jul 2011 10:30:29 -  1.115
+++ OpenBSD/Delete.pm   20 Aug 2011 10:09:16 -
@@ -667,14 +667,6 @@ sub delete
$self-mark_ldconfig_directory($state);
 }
 
-package OpenBSD::PackingElement::FDEINSTALL;
-sub delete
-{
-   my ($self, $state) = @_;
-
-   $self-run($state, DEINSTALL);
-}
-
 package OpenBSD::PackingElement::Depend;
 sub copy_old_stuff
 {
Index: OpenBSD/PackageInfo.pm
===
RCS file: /home/openbsd/cvs/src/usr.sbin/pkg_add/OpenBSD/PackageInfo.pm,v
retrieving revision 1.55
diff -u -p -r1.55 PackageInfo.pm
--- OpenBSD/PackageInfo.pm  18 Aug 2011 07:53:59 -  1.55
+++ OpenBSD/PackageInfo.pm  20 Aug 2011 10:14:30 -
@@ -23,8 +23,8 @@ require Exporter;
 our @ISA=qw(Exporter);
 our @EXPORT=qw(installed_packages installed_info installed_name info_names 
is_info_name installed_stems
 lock_db unlock_db
-add_installed delete_installed is_installed borked_package CONTENTS 
COMMENT DESC INSTALL DEINSTALL REQUIRE
-REQUIRED_BY REQUIRING DISPLAY UNDISPLAY MTREE_DIRS);
+add_installed delete_installed is_installed borked_package CONTENTS 
COMMENT DESC
+REQUIRED_BY REQUIRING DISPLAY UNDISPLAY);
 
 use OpenBSD::PackageName;
 use OpenBSD::Paths;
@@ -32,21 +32,17 @@ use constant {
CONTENTS = '+CONTENTS',
COMMENT = '+COMMENT',
DESC = '+DESC',
-   INSTALL = '+INSTALL',
-   DEINSTALL = '+DEINSTALL',
-   REQUIRE = '+REQUIRE',
REQUIRED_BY = '+REQUIRED_BY',
REQUIRING = '+REQUIRING',
DISPLAY = '+DISPLAY',
-   UNDISPLAY = '+UNDISPLAY',
-   MTREE_DIRS = '+MTREE_DIRS' };
+   UNDISPLAY = '+UNDISPLAY'};
 
 use Fcntl qw/:flock/;
 my $pkg_db = $ENV{PKG_DBDIR} || OpenBSD::Paths-pkgdb;
 
 my ($list, $stemlist);
 
-our @info = (CONTENTS, COMMENT, DESC, REQUIRE, INSTALL, DEINSTALL, 
REQUIRED_BY, REQUIRING, DISPLAY, UNDISPLAY, MTREE_DIRS);
+our @info = (CONTENTS, COMMENT, DESC, REQUIRED_BY, REQUIRING, DISPLAY, 
UNDISPLAY);
 
 our %info = ();
 for my $i (@info) {
Index: OpenBSD/PackingElement.pm
===
RCS file: /home/openbsd/cvs/src/usr.sbin/pkg_add/OpenBSD/PackingElement.pm,v
retrieving revision 1.199
diff -u -p -r1.199 PackingElement.pm
--- OpenBSD/PackingElement.pm   24 Jun 2011 22:43:58 -  1.199
+++ OpenBSD/PackingElement.pm   20 Aug 2011 10:10:10 -
@@ -1631,35 +1631,6 @@ sub category() { OpenBSD::PackageInfo::C
 sub write
 {}
 
-package OpenBSD::PackingElement::ScriptFile;
-our @ISA=qw(OpenBSD::PackingElement::SpecialFile);
-use OpenBSD::Error;
-
-sub exec_on_add { 1 }
-sub exec_on_delete { 1 }
-
-sub run
-{
-   my ($self, $state, @args) = @_;
-
-   my $pkgname = $state-{pkgname};
-   my $name = $self-fullname;
-
-   return if $state-{dont_run_scripts};
-

more old stuff

2011-08-24 Thread Marc Espie
Apparently, nobody cares about fat packages.

Not surprisingly, killing that code simplifies a few things.
Especially since the necessity of passing arch around was only due to
the possibility of fat packages...

Index: package.5
===
RCS file: /cvs/src/usr.sbin/pkg_add/package.5,v
retrieving revision 1.9
diff -u -p -r1.9 package.5
--- package.5   11 Jan 2010 10:16:20 -  1.9
+++ package.5   24 Aug 2011 10:35:28 -
@@ -29,7 +29,6 @@ and
 .Xr pkg_merge 1 ,
 and are usually manipulated using
 .Xr pkg_add 1 ,
-.Xr pkg_merge 1 ,
 .Xr pkg_mklocatedb 1 ,
 or
 .Xr pkg_info 1 .
@@ -57,9 +56,6 @@ All types of archive contents can be pre
 including files, directories, hardlinks, symlinks, fifos, block and character
 devices.
 .Pp
-A special extension to the format, dubbed fat packages, is described in
-the next section.
-.Pp
 In order to allow just-in-time extraction,
 packages always begin with a table of contents, named
 .Pa +CONTENTS .
@@ -74,8 +70,8 @@ See
 for annotation details.
 .Pp
 This table of contents is always followed by a few special files, some of
-which are optional: the package description (+DESC), an installation script
-(+INSTALL), a display message (+DISPLAY), etc.
+which are optional: the package description (+DESC),
+a display message (+DISPLAY), etc.
 .Pp
 The ustar format has some limitations with respect to file names.
 Accordingly, the package tools will replace very long names with
@@ -103,38 +99,17 @@ Once the packing-list signature is check
 will be checksummed, resulting in a
 .Sq just-in-time
 signature checking.
-.Sh FAT PACKAGES DESCRIPTION
-The
-.Xr pkg_merge 1
-command can create fat packages, which coalesce several normal packages in
-a single ustar archive, by interleaving their contents.
-.Pp
-Other tools, such as
-.Xr pkg_add 1 ,
-are aware of fat packages and can handle them transparently.
 .Pp
-In a fat package, every item has a small prefix that identifies the
-original package.
-For instance, after merging two packages, the package will usually
-begin with
-.Pa a/+CONTENTS
-and
-.Pa b/+CONTENTS .
-Individual items will then begin with
-.Pa ab/file ,
-for a file common to both packages;
-.Pa a/file
-for a file belonging to the first package;
-and
-.Pa b/file
-for a file belonging to the second package.
+Fat packages were removed in
+.Ox 5.1 ,
+since no practical application was found.
 .Sh SEE ALSO
 .Xr pkg_add 1 ,
 .Xr pkg_create 1 ,
 .Xr pkg_info 1 ,
-.Xr pkg_merge 1 ,
 .Xr packages 7 ,
 .Xr packages-specs 7
+.Sh HISTORY
 .Sh STANDARDS
 Packages are valid gzip'ed ustar archives that can be extracted using
 .Xr tar 1 .
Index: pkg_merge
===
RCS file: pkg_merge
diff -N pkg_merge
--- pkg_merge   28 Jul 2010 12:19:54 -  1.21
+++ /dev/null   1 Jan 1970 00:00:00 -
@@ -1,257 +0,0 @@
-#! /usr/bin/perl
-# Copyright (c) 2005-2007 Marc Espie es...@openbsd.org
-# $OpenBSD: pkg_merge,v 1.21 2010/07/28 12:19:54 espie Exp $
-#
-# Permission to use, copy, modify, and distribute this software for any
-# purpose with or without fee is hereby granted, provided that the above
-# copyright notice and this permission notice appear in all copies.
-#
-# THE SOFTWARE IS PROVIDED AS IS AND THE AUTHOR DISCLAIMS ALL WARRANTIES
-# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
-# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
-# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
-# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
-# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
-# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
-
-use strict;
-use warnings;
-
-use OpenBSD::PackageLocator;
-use OpenBSD::PackageInfo;
-use OpenBSD::PackingList;
-use OpenBSD::Getopt;
-use OpenBSD::Error;
-use OpenBSD::Ustar;
-use OpenBSD::ArcCheck;
-use OpenBSD::Paths;
-use OpenBSD::State;
-use File::Copy;
-use File::Path;
-
-package OpenBSD::PackingElement;
-sub copy_over {}
-
-sub mark_tocopy {}
-
-sub make_alias {}
-package OpenBSD::PackingElement::FileBase;
-sub mark_tocopy
-{
-   my ($self, $list) = @_;
-   push(@$list, $self);
-}
-
-sub copy_over
-{
-   my ($self, $wrarc, $prefix, $pkg, $state) = @_;
-   my $e = $pkg-{pkg}-next;
-   if (!$e-check_name($self)) {
-   $state-fatal(Names don't match: #1 vs #2, 
-   $e-{name}, $self-{name});
-   }
-   $e-{name} = $prefix./.$e-{name};
-   $e-copy_long($wrarc);
-}
-
-sub make_alias
-{
-   my ($self, $wrarc, $prefix, $pkg, $alias, $state) = @_;
-   my $e = $pkg-{pkg}-next;
-   if (!$e-check_name($self)) {
-   $state-fatal(Names don't match: #1 vs #2, 
-   $e-{name}, $self-{name});
-   }
-   $e-{name} = $prefix./.$e-{name};
-   $e-alias($wrarc, $prefix/$alias);
-}
-
-package OpenBSD

Re: more old stuff

2011-08-25 Thread Marc Espie
On Wed, Aug 24, 2011 at 07:22:47PM +0200, Landry Breuil wrote:
 On Wed, Aug 24, 2011 at 12:39:07PM +0200, Marc Espie wrote:
  Apparently, nobody cares about fat packages.
  
  Not surprisingly, killing that code simplifies a few things.
  Especially since the necessity of passing arch around was only due to
  the possibility of fat packages...
 
 I don't see the relation between fat package and arch.. was one of the
 planned usecase 'build a single package containing a pkg for each archs' ?

Read the removed code. This is most of it.



Re: diff to PackageLocator.pm to support multiple installpath entries

2011-09-07 Thread Marc Espie
On Wed, Sep 07, 2011 at 12:26:30PM +0200, Marian Hettwer wrote:
 Hi there,
 
 see attached diff to PackageLocator.pm.
 It seems like in it's current version it doesn't support multiple
 installpath entries, like:
 
 installpath = ftp://mymirror/localrepo
 installpath += ftp://mymirror/officialrepo
 
 Before it was only honoring the first entry, no matter whether += or
 = for the first entry.
 
 However, now it works as documented in pkg.conf(5)
 
 best regards,
 Marian
 
 http://crivens.terrorteam.de/~rabauke/OpenBSD/PackageLocator.pm.diff
 
 or
 
 Index: usr.sbin/pkg_add/OpenBSD/PackageLocator.pm
 ===
 RCS file: /cvs/src/usr.sbin/pkg_add/OpenBSD/PackageLocator.pm,v
 retrieving revision 1.99
 diff -p -u -r1.99 PackageLocator.pm
 --- usr.sbin/pkg_add/OpenBSD/PackageLocator.pm26 Aug 2011 08:46:10
 - 1.99
 +++ usr.sbin/pkg_add/OpenBSD/PackageLocator.pm7 Sep 2011 10:19:24
 -
 @@ -40,7 +40,7 @@ sub build_default_path
   return;
   }
   $default_path-add(OpenBSD::PackageRepository-new(./,
 $state)-can_be_empty);
 - if (my $i = $state-config-value(installpath)) {
 + for my $i ($state-config-value(installpath)) {
   $default_path-add(OpenBSD::PackageRepository-new($i, $state));
   }
 }
2 points:

1/ pkg.conf(5) does not actually say that all variables are lists. And it was
not really intended for installpath (in particular, if you make installpath
into a list, your next binary upgrade of the base system is going to give
really funny results).

2/ the whole repository/locator/handle mess is under reconstruction, as it is
a total mess, and doesn't even do what I want it to do correctly.  Not exactly
the right time to tweak minor bits in code that's directly headed to the
trashcan...



the newer changes in the ports tree (normalization of pkgpath)

2011-09-17 Thread Marc Espie
I promised yesterday I was going to write an email explaining what this
is about.

This has got everything to do with reproducible package builds, and the
complexity that's package version numbers.

First, here's the situation before the change
-
Take minitube-1.5, for instance.
Among other things, it has

RUN_DEPENDS = multimedia/gstreamer-0.10/plugins-base

this ends up in the package as something like this:

@depend 
multimedia/gstreamer-0.10/plugins-base:gstreamer-plugins-base-*:gstreamer-plugins-base-0.10.35p0


when you want to update a package, very often, the package name itself won't
change. So when do we update ?
pkg_add actually looks at the depend lines in the package and compares them.
That is, the
@depend multimedia/gstreamer-0.10/plugins-base:gstreamer-plugins-base-*:
part is used as key, and the gstreamer-plugins-base-0.10.35p0 part is used
for further comparison.

If the set of keys change, the packages won't be comparable based on depend
line, and normally, you're supposed to bump the REVISION.  Otherwise, pkg_add
will complain it doesn't know which is the newest package.


Prior to the most recent change, the RUN_DEPENDS was used as-is. 
Let's look at net/avahi, it contains:
RUN_DEPENDS-gtk = net/avahi,-ui,no_mono,no_qt3,no_qt4

thus, the plist for avahi-gtk-0.6.30p2 contains:
@depend net/avahi,-ui,no_mono,no_qt3,no_qt4:avahi-ui-*:avahi-ui-0.6.30

However, this is somewhat strange, because the no_mono,no_qt3,no_qt4 part
*is* a pseudo-flavor: it is just pertinent to the build, but not at all
for a binary package. avahi-gtk would be happy with an avahi-ui that
would come as
@depend net/avahi,-ui,no_qt4:avahi-ui-*:avahi-ui-0.6.30

But this doesn't work, because the binary packages don't know about
pseudo-flavors, and thus net/avahi,-ui,no_qt4 and 
net/avahi,-ui,no_mono,no_qt3,no_qt4  are completely different things,
thus the packages will be different, and non comparable.

The change itself
-
Some time ago, I came to the realization that some stuff were better left
to the depending port.  We always went there to get the build package name
anyways (avahi-ui-0.6.30), but we also decided to get there to grab the
pkgspec (the avahi-ui-* part). That's how ports depending on python magically
got their python-=2.7,2.8   spec without having to say anything.
For instance, youtube-dl-2011.08.04 has:
@depend lang/python/2.7:python-=2.7,2.8:python-2.7.1p11



So I realized I could go to the depending port for the pkgpath *as well*.
This has the neat effect of putting back the path in a standard order,
and of removing pseudo-flavors entirely... it also creates full pkgpaths
every time.

There was some fallout: all of a sudden, *a lot* of @depend lines in packages
would change, and I would need to bump all the affected packages (1980 ports
to be exact). Because alongside pseudo-flavors, all multi-package dependencies 
would have a subpackage appended if it wasn't already there.

The result
--
Let's go back to our examples. minitube got bumped to minitube-1.5p0
it now sports
@depend 
multimedia/gstreamer-0.10/plugins-base,-main:gstreamer-plugins-base-*:gstreamer-plugins-base-0.10.35p1
because of the subpackage part.

youtube-dl-2011-08-04p0 sports:
@depend lang/python/2.7,-main:python-=2.7,2.8:python-2.7.1p12
for the same reason.

More interestingly, 

avahi-gtk-0.6.30p3 now says:
@depend net/avahi,-ui:avahi-ui-*:avahi-ui-0.6.30p0


That's right, the pseudo-flavors are completely gone.


This triggered a small bug in pkg_create (net/avahi,-ui, in that context,
means the -ui subpackage of net/avahi with an empty flavor, not a default
flavor) which  is now fixed.

Why this is cool

well, the previous way pkgpaths were stored, you had to be really careful
how you presented stuff, because equivalent paths would end up as different
strings, and thus divergent paths stored in a package, with ill effect.

For instance
some/dir,flavor1,flavor2
and
some/dir,flavor2,flavor1
are now completely equivalent for package building.

Also, mentioning a subpackage explicitly (or not mentioning it) won't change
the string recorded in the package. 

And finally, the new version *doesn't care at all* about pseudo-flavors.
So, if you want to depend on
print/poppler,no_qt,no_qt4
or
print/poppler
the resulting @depend line will be the same: it will be something like:
@depend print/poppler,-main:poppler-*:poppler-0.6.17p1

This can solve some nightmarish situations between subpackages in the
same port with pseudo-flavors, where you had to be very careful about 
which pseudo-flavor you were depending on if you wanted to have reproducible
results. Thus removing a lot of the interest of pseudo-flavors to avoid
building things (looking at such cases, the current situation is slightly
wrong in a lot of directories).

There's also the very real possibility to tweak pseudo-flavors in dependencies
between individual builds and bulk builds thru dpb. So, 

Re: tar -J for xz

2011-10-05 Thread Marc Espie
On Wed, Oct 05, 2011 at 12:05:18AM +0100, Nicholas Marriott wrote:
 And from the look of it they also have -J, but .xz is so rarely used
 right now I'm not sure it is something we really need to copy...

That's not gonna be the case from gnome3 (for instance). As I've been
told by Antoine, they're going to drop bz2 tarballs entirely.



Re: terminal emulators using /usr/local/share/terminfo and a bug in perl's Term::Cap

2011-10-06 Thread Marc Espie
On Thu, Oct 06, 2011 at 04:00:21PM +0100, Nicholas Marriott wrote:
 I can't really follow what it is trying to do. Won't your change always
 call infocmp if TERMCAP is not set instead of searching through the
 files?
 
 It looks like currently infocmp is only used if there are no termcap
 files at all, not if the entry is missing from the files it does
 find. So shouldn't the fallback infocmp call happen much later?
 Somewhere around the line:
 
 croak Can't find $term if $entry eq '';
 
 
 On Thu, Oct 06, 2011 at 04:46:45PM +0200, David Coppa wrote:
  On Sun, Oct 2, 2011 at 8:22 AM, Nicholas Marriott
  nicholas.marri...@gmail.com wrote:
   This seems fine to me, but I'm not a perl guru. Have you talked to
   upstream?
  
  I'm having big difficulties trying to report the bug on rt.cpan.org.
  It seems they do all the best to make you feel uncomfortable :(
  
  Can this be fixed locally at least for now if my patch is correct?
  
  ciao
  David
  
   On Thu, Sep 29, 2011 at 10:49:36AM +0200, David Coppa wrote:
   Hi,
  
   The patch to ncurses nicm@ commited some days ago, exposes a bug
   in perl's cpan/Term-Cap/Cap.pm. So, when you use rxvt-unicode on
   a recent -current, you will hit this bug with pkg_* tools:
  
   # pkg_delete -v xclip
   failed termcap lookup on rxvt-unicode-256color at 
   /usr/libdata/perl5/OpenBSD/ProgressMeter/Term.pm line 125
  
   If you use a terminal emulator that has a terminfo entry but not
   a termcap one (just like x11/rxvt-unicode or x11/st), you should
   fall back to the case where perl uses infocmp to fake up a
   termcap entry from terminfo, but this never happens because the
   logic is flawed.
  
   Here's a diff:
  
   Index: cpan/Term-Cap/Cap.pm
   ===
   RCS file: /cvs/src/gnu/usr.bin/perl/cpan/Term-Cap/Cap.pm,v
   retrieving revision 1.1.1.1
   diff -u -p -r1.1.1.1 Cap.pm
   --- cpan/Term-Cap/Cap.pm ? ? ?24 Sep 2010 14:49:05 - ? ? ?1.1.1.1
   +++ cpan/Term-Cap/Cap.pm ? ? ?29 Sep 2011 08:14:46 -
   @@ -273,7 +273,7 @@ sub Tgetent
  
   ? ? ?my @termcap_path = termcap_path();
  
   - ? ?unless ( @termcap_path || $entry )
   + ? ?if ( !@termcap_path || !$entry )
   ? ? ?{
  
   ? ? ? ? ?# last resort--fake up a termcap from terminfo
  
  
   ---
   cheers,
   David
  
  
I want to test it, which I haven't had the time to do yet.



Re: enable aucat by default

2011-10-07 Thread Marc Espie
On Thu, Oct 06, 2011 at 05:04:03PM -0700, patrick keshishian wrote:
 yuck... sounds like linux to me. please don't go down that route of
 enable everything, it can't hurt even if you don't use it mentality.

In a wide difference to linux, we try to avoid running buggy shit on the
system by default.

I mean, if aucat was a large piece of junk like gnome's tracker, or firefox,
I would understand objections to running it by default.

Heck, even on servers. What's the actual footprint of an idle aucat ?
It's not even accessible from outside, so there's no possibility of a
remote hole.



  1   2   3   4   5   6   7   >