Re: I/O semantics of pipe and FIFO.

[Alfred Perlstein]

I've seen that bug, but I think our bug was more that dd would exit if it got 
back a short read from input. So if you did something like this (maybe not 
exactly, but close):

dd bs=1m | dd bs=10m > f.out

You might get only 1mb in f.out. This had to do with skipping disklabels and 
other such schenanigans. 

Sent from my iPhone

> On Mar 8, 2017, at 4:09 PM, Devin Teske <dte...@freebsd.org> wrote:
> 
> Problem we had found was:
> 
> Executing dd with a closed stdout and stderr would cause the summary messages 
> printed at the end to go into the destination output file.
> 
> For example,
> 
> dd if=/dev/zero of=/tmp/foo bs=1m count=1
> 
> Works fine, but the following:
> 
> dd if=/dev/zero of=/tmp/foo bs=1m count=1 >&- 2>&-
> 
> Will cause the summary statistics of dd to appear in /tmp/foo instead of on 
> the console.
> 
> The issue is that the summary statistics are send to fd1, which if you close 
> down stdout and stdin, fd1 is actually the output file since it got the 
> lowest file descriptor available when open(2) was called on the output file.
> 
> This was never fixed because it was deemed “silly developer, don’t close 
> stdout and stderr before invoking dd”.
> 
> The argument has been made by Jilles T. that it is generally a bad idea to 
> close down any of the standard file descriptors because it cannot be 
> predicted how a particular UNIX utility will react (e.g., in the case of dd, 
> causing a simple printf(3) to go to an unexpected location).
> — 
> Devin
> 
> 
>> On Mar 4, 2017, at 8:12 PM, Alfred Perlstein <alf...@freebsd.org> wrote:
>> 
>> Devin and I found this when we worked together.  I think it was due to some 
>> situation in dd(1) where short reads would exit pre-maturely, however I may 
>> be mis-remembering.  Devin, do you recall the specifics?
>> 
>> 
>>> On 3/4/17 7:44 PM, Julian Elischer wrote:
>>> 
>>> an interesting point to discuss? is our behaviour in this test right?
>>>  from: "austin-group mailng list (posix standard discussion)"
>>> 
>>> -- rest of email is quoted ---
>>> On 5/3/17 5:48 am, Stephane Chazelas wrote:
>>> 
>>> 2017-03-04 13:14:08 +, Danny Niu:
>>>> Hi all.
>>>> 
>>>> I couldn't remember where I saw it saying, that when reading
>>>> from a pipe or a FIFO, the read syscall returns the content of
>>>> at most one write call. It's a bit similar to the
>>>> message-nondiscard semantics of dear old STREAM.
>>>> 
>>>> Currently, I'm reading through the text to find out a bit
>>>> more, and I appreciate a bit of pointer on this.
>>> [...]
>>> 
>>> (echo x; echo y) | (sleep 1; dd count=1 2> /dev/null)
>>> 
>>> outputs both x and y in all of Linux, FreeBSD and Solaris in my
>>> tests.
>>> 
>>> That a read wouldn't read what's currently in the pipe would be
>>> quite surprising.
>>> 
>>> I also wouldn't expect pipes to store the writes as individual
>>> separate message but use one buffer.
>>> 
>>> In:
>>> 
>>> (
>>> dd bs=4 count=1 if=/dev/zero 2> /dev/null
>>> echo first through >&2
>>> dd bs=4 count=1 if=/dev/zero 2> /dev/null
>>> echo second through >&2
>>> ) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c
>>> 
>>> That is where the second write blocks because the pipe is full,
>>> the reading dd still reads both writes in Linux and Solaris in
>>> my tests (on Solaris (10 on amd64 at least), reduce to 2
>>> instead of 4 or both writes would block).
>>> 
>>> On FreeBSD, I get only the first write (using 8000 followed by
>>> 1 for instance).
>>> 
>>> FreeBSD is also the only one of the three where
>>> 
>>> dd bs=100 count=1 if=/dev/zero | dd bs=100 count=1 | wc -c
>>> 
>>> Doesn't output 100. The others schedule both processes back
>>> and forth during their write() and read() system call while the
>>> pipe is being filled and emptied several times.
>>> 
>> 
> 

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Fwd: Re: I/O semantics of pipe and FIFO.

[Alfred Perlstein]

Devin and I found this when we worked together.  I think it was due to 
some situation in dd(1) where short reads would exit pre-maturely, 
however I may be mis-remembering.  Devin, do you recall the specifics?



On 3/4/17 7:44 PM, Julian Elischer wrote:


an interesting point to discuss? is our behaviour in this test right?
   from: "austin-group mailng list (posix standard discussion)"

-- rest of email is quoted ---
On 5/3/17 5:48 am, Stephane Chazelas wrote:

2017-03-04 13:14:08 +, Danny Niu:

Hi all.

I couldn't remember where I saw it saying, that when reading
from a pipe or a FIFO, the read syscall returns the content of
at most one write call. It's a bit similar to the
message-nondiscard semantics of dear old STREAM.

Currently, I'm reading through the text to find out a bit
more, and I appreciate a bit of pointer on this.

[...]

(echo x; echo y) | (sleep 1; dd count=1 2> /dev/null)

outputs both x and y in all of Linux, FreeBSD and Solaris in my
tests.

That a read wouldn't read what's currently in the pipe would be
quite surprising.

I also wouldn't expect pipes to store the writes as individual
separate message but use one buffer.

In:

(
  dd bs=4 count=1 if=/dev/zero 2> /dev/null
  echo first through >&2
  dd bs=4 count=1 if=/dev/zero 2> /dev/null
  echo second through >&2
) | (sleep 1; dd bs=10 count=1 2> /dev/null) | wc -c

That is where the second write blocks because the pipe is full,
the reading dd still reads both writes in Linux and Solaris in
my tests (on Solaris (10 on amd64 at least), reduce to 2
instead of 4 or both writes would block).

On FreeBSD, I get only the first write (using 8000 followed by
1 for instance).

FreeBSD is also the only one of the three where

dd bs=100 count=1 if=/dev/zero | dd bs=100 count=1 | wc -c

Doesn't output 100. The others schedule both processes back
and forth during their write() and read() system call while the
pipe is being filled and emptied several times.



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: effect of strip(1) on du(1)

[Alfred Perlstein]

On 3/2/17 5:30 PM, Alan Somers wrote:

On Thu, Mar 2, 2017 at 6:12 PM, Ngie Cooper  wrote:

On Thu, Mar 2, 2017 at 4:31 PM, Rodney W. Grimes
 wrote:
...

Even if that is the case file system cache effects should NOT be
visible to a userland process.   This is NOT as if your running
2 different processing beating on a file.  Your test cases are
serialially syncronous shell invoked commands seperated with
&& the results should be exact and predictable.

When strip returns the operation from the userland perspecive
is completed and any and all processeses started after that
should have the view of the completed strip command.

This IS a bug.

Would the same statement necessarily apply if the filesystem was
writing things asynchronously to the backing storage?
Thanks,
-Ngie

du(1) is using fts_read(3), which is based on the stat(2) information.
The OpenGroup defines st_blocksize as "Number of blocks allocated for
this object."  In the case of ZFS, a write(2) may return before any
blocks are actually allocated.  And thanks to compression, gang
blocks, and deduplication, at this point it's not even possible for
ZFS to know how many blocks it will need to allocate.  I think
st_blocksize should be interpreted as a "best effort" output.  Just
like df(1), you can't rely on du's output to be mathematically precise
in any way.  I certainly don't see any way to fix it besides doing
something like an fsync(2) before getting stat information, and we
certainly don't want to do that.



Try adding an fsync(1) to the file before running du(1) on it.

-Alfred
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Is there possible run a MacOS X binary

[Alfred Perlstein]

On 12/7/16 10:57 AM, K. Macy wrote:




A MachO activator is indeed not useful without an OSX install.

But let's be honest, Mach IPC is a loadable kernel module requiring no real
kernel changes. It's not upstreamable because of a general poor
understanding of IPC by noisy commentators and a religious aversion to a
technology perceived as having failed in the marketplace of ideas.

I'd be happy to upstream it.  Are there diffs relative to -current?

-Alfred




On Wed, Dec 7, 2016 at 10:45 Warner Losh  wrote:


On Mon, Dec 5, 2016 at 12:31 PM, Kevin P. Neal 
wrote:


On Mon, Dec 05, 2016 at 02:49:07PM -0300, Nilton Jose Rizzo wrote:

  Sorry for cross posting (-current and -ports)
Is there any emulator like linuxator to run Mac OS X binaries, or
is ther any licensing problem?

It may be possible to make an emulator for Darwin (the OS that Mac OS

sits


on top of), but an emulator for Mac OS would probably require a legal

copy


of Mac OS.
So, no, there is no Mac OS emulator for FreeBSD. And I'd be surprised if
it ever happened.



NetBSD has (or had) a macho image activator, which is the first step

in this process. But Kevin is right that most of the functionality of

MacOS isn't in the kernel, and you'd need a copy of MacOS to run it in

emulation. Plus there's a lot of Mach code that MacOS depends on that

has no simple counterparts in FreeBSD, and that would be a lot of work

to make happen. It's one of the things that's a barrier to entry for a

simple, straight forward launchd port, for example.








___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [RFC] remove GNU rcs from FreeBSD 12

[Alfred Perlstein]

On 9/11/16 6:38 AM, Baptiste Daroussin wrote:

hi,

For long we are planning to remove GNU rcs from base, after a failed attempt
before FreeBSD 10.0. Let see where we are to be able to remove it from FreeBSD
12.

GNU rcs is a GPLv2 software with newer version being GPLv3 preventing any
updates/fixes.

 From previous discussions there were issues that has been raised in previous
attempts:
- ident(1) is still useful given we still have Keywords in our sources. It has
   been replaced by a BSD Licensed version (enhanced to improve compatibility
   with Subversion Keyword) for FreeBSD 11. So that tool will remain in base
   after removal of GNU rcs.
- etc-update uses merge(1) from GNU rcs, this has been changed in head to use
   diff3 instead.
- rc.subr allows to use rcs for the backup file functionality. This
   functionality is off by default as such I plan to make a warning if rcs is 
not
   installed and recommand to install rcs from base (or if noone claim using the
   feature I will just remove the functionality and only keep the default
   behaviour aka keep one backup copy).
- people uses rcs to handle configuration files in /etc for example. for those
   multiple compatible alternatives are available in ports:
   * rcs57: a copy of the latest version of GNU rcs in base before removal
 (GPLv2)
   * rcs: latest GNU rcs version (GPLv3)

I haven't gone the direction of importing OpenRCS (BSD licensed version from
OpenBSD) as it needs way more work to be 100% compatible with latest version of
GNU rcs.

How to proceed:
- First turn off GNU rcs by default for a couple of month.
- Totally remove GNU rcs if no blockers has been raised.

Best regards,
Bapt

\o/

-Alfred
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [PATCH] randomized delay in locking primitives, take 2

[Alfred Perlstein]

Why is

+struct lock_delay_config {
+u_int initial;
+u_int step;
+u_int min;
+u_int max;
+};

missing comments for its members?  Are they documented anywhere else?

-Alfred


On 7/31/16 5:41 AM, Mateusz Guzik wrote:

On Sun, Jul 31, 2016 at 01:49:28PM +0300, Konstantin Belousov wrote:
[snip]

After an irc discussion, the following was produced (also available at:
https://people.freebsd.org/~mjg/lock_backoff_complete4.diff):

Differences:
- uint64_t usage was converted to u_int (also see r303584)
- currently unused features (cap limit and return value) were removed
- lock_delay args got packed into a dedicated structure

Note this patch requires the tree to be at least at r303584.

diff --git a/sys/kern/kern_mutex.c b/sys/kern/kern_mutex.c
index 0555a78..9b07b8b 100644
--- a/sys/kern/kern_mutex.c
+++ b/sys/kern/kern_mutex.c
@@ -55,6 +55,7 @@ __FBSDID("$FreeBSD$");
  #include 
  #include 
  #include 
+#include 
  #include 
  #include 
  #include 
@@ -138,6 +139,36 @@ struct lock_class lock_class_mtx_spin = {
  #endif
  };
  
+#ifdef ADAPTIVE_MUTEXES

+static SYSCTL_NODE(_debug, OID_AUTO, mtx, CTLFLAG_RD, NULL, "mtx debugging");
+
+static struct lock_delay_config mtx_delay = {
+   .initial= 1000,
+   .step   = 500,
+   .min= 100,
+   .max= 5000,
+};
+
+SYSCTL_INT(_debug_mtx, OID_AUTO, delay_initial, CTLFLAG_RW, _delay.initial,
+0, "");
+SYSCTL_INT(_debug_mtx, OID_AUTO, delay_step, CTLFLAG_RW, _delay.step,
+0, "");
+SYSCTL_INT(_debug_mtx, OID_AUTO, delay_min, CTLFLAG_RW, _delay.min,
+0, "");
+SYSCTL_INT(_debug_mtx, OID_AUTO, delay_max, CTLFLAG_RW, _delay.max,
+0, "");
+
+static void
+mtx_delay_sysinit(void *dummy)
+{
+
+   mtx_delay.initial = mp_ncpus * 25;
+   mtx_delay.min = mp_ncpus * 5;
+   mtx_delay.max = mp_ncpus * 25 * 10;
+}
+LOCK_DELAY_SYSINIT(mtx_delay_sysinit);
+#endif
+
  /*
   * System-wide mutexes
   */
@@ -408,8 +439,10 @@ __mtx_lock_sleep(volatile uintptr_t *c, uintptr_t tid, int 
opts,
int contested = 0;
uint64_t waittime = 0;
  #endif
+#if defined(ADAPTIVE_MUTEXES) || defined(KDTRACE_HOOKS)
+   struct lock_delay_arg lda;
+#endif
  #ifdef KDTRACE_HOOKS
-   u_int spin_cnt = 0;
u_int sleep_cnt = 0;
int64_t sleep_time = 0;
int64_t all_time = 0;
@@ -418,6 +451,9 @@ __mtx_lock_sleep(volatile uintptr_t *c, uintptr_t tid, int 
opts,
if (SCHEDULER_STOPPED())
return;
  
+#if defined(ADAPTIVE_MUTEXES) || defined(KDTRACE_HOOKS)

+   lock_delay_arg_init(, _delay);
+#endif
m = mtxlock2mtx(c);
  
  	if (mtx_owned(m)) {

@@ -451,7 +487,7 @@ __mtx_lock_sleep(volatile uintptr_t *c, uintptr_t tid, int 
opts,
if (m->mtx_lock == MTX_UNOWNED && _mtx_obtain_lock(m, tid))
break;
  #ifdef KDTRACE_HOOKS
-   spin_cnt++;
+   lda.spin_cnt++;
  #endif
  #ifdef ADAPTIVE_MUTEXES
/*
@@ -471,12 +507,8 @@ __mtx_lock_sleep(volatile uintptr_t *c, uintptr_t tid, int 
opts,
"spinning", "lockname:\"%s\"",
m->lock_object.lo_name);
while (mtx_owner(m) == owner &&
-   TD_IS_RUNNING(owner)) {
-   cpu_spinwait();
-#ifdef KDTRACE_HOOKS
-   spin_cnt++;
-#endif
-   }
+   TD_IS_RUNNING(owner))
+   lock_delay();
KTR_STATE0(KTR_SCHED, "thread",
sched_tdname((struct thread *)tid),
"running");
@@ -570,7 +602,7 @@ __mtx_lock_sleep(volatile uintptr_t *c, uintptr_t tid, int 
opts,
/*
 * Only record the loops spinning and not sleeping.
 */
-   if (spin_cnt > sleep_cnt)
+   if (lda.spin_cnt > sleep_cnt)
LOCKSTAT_RECORD1(adaptive__spin, m, all_time - sleep_time);
  #endif
  }
diff --git a/sys/kern/kern_rwlock.c b/sys/kern/kern_rwlock.c
index d4cae61..363b042 100644
--- a/sys/kern/kern_rwlock.c
+++ b/sys/kern/kern_rwlock.c
@@ -44,6 +44,7 @@ __FBSDID("$FreeBSD$");
  #include 
  #include 
  #include 
+#include 
  #include 
  #include 
  #include 
@@ -65,15 +66,6 @@ PMC_SOFT_DECLARE( , , lock, failed);
   */
  #define   rwlock2rw(c)(__containerof(c, struct rwlock, rw_lock))
  
-#ifdef ADAPTIVE_RWLOCKS

-static int rowner_retries = 10;
-static int rowner_loops = 1;
-static SYSCTL_NODE(_debug, OID_AUTO, rwlock, CTLFLAG_RD, NULL,
-"rwlock debugging");
-SYSCTL_INT(_debug_rwlock, OID_AUTO, retry, CTLFLAG_RW, _retries, 0, "");
-SYSCTL_INT(_debug_rwlock, OID_AUTO, loops, CTLFLAG_RW, _loops, 0, "");
-#endif
-
  #ifdef DDB
  #include 
  
@@ -100,6 +92,41 @@ struct lock_class lock_class_rw = {

  #endif
  };
  
+#ifdef ADAPTIVE_RWLOCKS

Re: difference in SIGCHLD behavior between Linux and FreeBSD breaks apt

[Alfred Perlstein]

On 7/18/16 4:15 PM, Matthew Macy wrote:



   On Mon, 18 Jul 2016 16:11:53 -0700 Alfred Perlstein <alf...@freebsd.org> 
wrote 
  > I believe the
  >
  >
  > On 7/6/16 9:34 PM, Matthew Macy wrote:
  > > As a first step towards managing linux user space in a chrooted /compat/linux, 
initially for i915 testing with intel gpu tools, later on to get widevine and steam to work 
I'm trying to get apt to work. I've fixed a number of issues to date in pseudofs/linprocfs 
but now I'm running in to a bug caused by differences in SIGCHLD handling between Linux and 
FreeBSD. The situation is that apt will spawn dpkg and wait on a pipe read. On Linux when 
dpkg exits the  SIGCHLD to apt causes a short read on the pipe which lets apt then 
continue. On FreeBSD a SIGCHLD is silently ignored. I've even experimented with doing a 
kill -20  to no effect.
  > >
  > > It would be easy enough to check sysvec against linux in pipe_read and 
break out of the loop when it's awakened from msleep (assuming there aren't deeper 
issues with signal propagation for anything other than SIGINT/SIGKILL) and then do a 
short read. However, I'm assuming that anyone who has worked in this area probably 
has a cleaner solution.
  > >
  > > Thanks in advance.
  >
  > Are you sure you need a hack in pipe_read and not one of the following
  > possibilities:
  > 1) a setting for the default signal disposition for linux processes
  > needs to be fixed.
  > 2) a flag set in p_flag2 that says set this behavior properly in a
  > generic manner.
  >
  > Again not sure why you need to hack pipe_read and not just make sure
  > that SIGCHLD is generated...
  >
  > Finally that sure is oddball behavior, dpkg probably has a bug where the
  > parent is keeping the write side of the pipe open, you might be able to
  > get them to take a patch upstream to fix that.
  >

If you read my final mail it turns out I was holding a reference to the pipe in 
question in linprocfs. Maintaining the reference kept apt from getting the EOF 
on the pipe. I've since fixed this.

-M


Ah that makes sense. :)

-Alfred
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: difference in SIGCHLD behavior between Linux and FreeBSD breaks apt

[Alfred Perlstein]

I believe the


On 7/6/16 9:34 PM, Matthew Macy wrote:

As a first step towards managing linux user space in a chrooted /compat/linux, 
initially for i915 testing with intel gpu tools, later on to get widevine and steam 
to work I'm trying to get apt to work. I've fixed a number of issues to date in 
pseudofs/linprocfs but now I'm running in to a bug caused by differences in SIGCHLD 
handling between Linux and FreeBSD. The situation is that apt will spawn dpkg and 
wait on a pipe read. On Linux when dpkg exits the  SIGCHLD to apt causes a short read 
on the pipe which lets apt then continue. On FreeBSD a SIGCHLD is silently ignored. 
I've even experimented with doing a kill -20  to no effect.
  
It would be easy enough to check sysvec against linux in pipe_read and break out of the loop when it's awakened from msleep (assuming there aren't deeper issues with signal propagation for anything other than SIGINT/SIGKILL) and then do a short read. However, I'm assuming that anyone who has worked in this area probably has a cleaner solution.


Thanks in advance.


Are you sure you need a hack in pipe_read and not one of the following 
possibilities:
1) a setting for the default signal disposition for linux processes 
needs to be fixed.
2) a flag set in p_flag2 that says set this behavior properly in a 
generic manner.


Again not sure why you need to hack pipe_read and not just make sure 
that SIGCHLD is generated...


Finally that sure is oddball behavior, dpkg probably has a bug where the 
parent is keeping the write side of the pipe open, you might be able to 
get them to take a patch upstream to fix that.


-Alfred

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: r300951 make buildworld duration 18h

[Alfred Perlstein]

It's too bad make(1) can't have a "max parallel jobs" for a particular 
directory.


I believe that some of the llvm build winds up using nearly a gig of RAM 
per compiled .c file, so when you are running 2 jobs at once, LLVM will 
swap like mad.


If we could add something under the llvm directory "max jobs = RAM/1GB" 
we could have much faster build worlds.


-Alfred

On 5/30/16 7:43 AM, Kurt Jaeger wrote:

Hi!


Yesterday I pulled via svn co ... a fresh r300951 and started the
buildworld as

# make -j2 buildworld

it ended today at lunchtime after 18 hours. The laptop has
2x Intel Atom 1.6GHz and 1 GByte RAM and was otherwise unused.
Is this the normal buildworld time of today?

Yes, this sounds in line with similar setups.


I think it spent most of the time in building the llvm...

Yes, llvm is a real hog.

That's why I normally build on amd64 boxes...



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] packaging the base system with pkg(8)

[Alfred Perlstein]

On 4/19/16 1:09 PM, Poul-Henning Kamp wrote:

As far as I know, nobody is taking the source code or the Makefiles
away, so if somebody doesn't like the system being distributed with
pkg, they can very well roll their own.

It's nice to see the level of enthusiasm the FreeBSD project can
muster, I just wish it wasn't always enthusiasm for stopping progress.


+1000


-Alfred
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] packaging the base system with pkg(8)

[Alfred Perlstein]

I don't think we need 100% consensus to proceed on anything and if I've 
learned anything from 20 years in this community is that forcing that 
issue does the community a huge disservice as well as turn off the code 
submitters.   See my thread on the missed opportunities in threads, or 
if you want I can paint the picture of what caused SMP to lag half a 
decade behind Linux as well.


I would say that if someone submitted a patch for /dev/givemeroot, sure 
that would be righteously shot down but to force the whole, entire, 
"right" solution the first time around is remarkably blocking and unfair 
to the community and submitters as well.


Why is this even happening in email?  If folks want "the right solution" 
then why aren't they submitting patches or pull requests to the pkg repo 
(or where ever this is stored?).  This seems counter-intuitive, but 
really actually should be how it works. Specifically: if you like where 
an idea is going, then don't block the code, submit improvements on top 
of it.  Stone soup it if you will.


-Alfred

On 4/19/16 9:28 AM, Nathan Whitehorn wrote:
Well, this discussion has gone pretty far off of the rails. I am of 
course happy to make a patch that cuts this down to 10 packages, but 
that's not something that should be committed without agreement -- 
which we obviously don't have. It would have been good to have had 
meaningful discussion of this before.


There are basically three workable options:
1. Have fewer packages. This is easy to implement and preserves the 
integrity of the base system (as well as unified versioning so that a 
system at some particular patch level will have the same global 
state). I have not seen any meaningful downside suggested to this so 
far except for marginally higher load on update servers.
2. Have 755 packages. This makes it harder to version the system and 
makes the user interface significantly worse (my opinion, but shared 
by others). This is the easiest to implement since it is already 
implemented.
3. Have ~10 meta packages that just depend on sets of the 755 packages 
and hide the internal details. This gives the user experience of (1) 
with the implementation of (2), and is marginally more complex than 
either.


Other things (the overlapping packages idea, for instance) are way too 
complex and will just lead to breakage. Can anyone provide an argument 
against (1) or, alternatively, for (2)? (2) seems to add a lot of 
complexity for no clear gain and I remain pretty confused about why it 
was chosen.

-Nathan

On 04/18/16 20:17, Alfred Perlstein wrote:
Maybe what the "too many packages" folks need to do is write some 
code to hide that it's so many packages.


:)

I think the rule of two feet should be applied here.

What we have is people that have worked quite hard to bring us 
something that we can easily work with, and on the other hand some 
folks that want something they consider even better. Personally I 
can't see how having the system less granular is better, since having 
it MORE granular is actually harder work.


Can someone on the "too many packages" campaign here explain to me 
how having too fine a granularity stops you from making macro 
packages containing packages?


Because honestly I can't see how having granularity hurts at all when 
if someone wanted to make it less granular all they would have to do 
is make some meta-packages.


-Alfred

On 4/18/16 7:23 PM, Lyndon Nerenberg wrote:

On 2016-04-18 7:01 PM, Roger Marquis wrote:

Can you explain what would be accomplished by testing all or even a
fraction of the possible permutations of base package 
combinations?  We

don't do that for ports.


The ports tree isn't a mandatory part of the system. And by 
definition it could not be tested that way, since it offers so many 
alternative implementations of specific functionality.



Other operating systems don't do that for
their base packages.


I'm pretty sure Solaris had some fairly hard-core regression tests 
to ensure basic system functionality wouldn't be compromised by 
'oddball' selections of packages offered up at install time.


> Honestly, some of us are wondering what exactly is
> behind some of these concerns regarding base packages.

The concern is from all of us UNIX dinosaurs who predate the 
fine-grained packaging environment, which just worked, and who now 
rip our (little remaining) hair out due to unsolvable package 
dependency loops in the Linux machines we are forced to administer 
in order to pay rent.  For me, as a sysadmin, I derive a negative 
benefit from this optimization.


I guess what I'm really asking is: where is the peer reviewed 
research that shows this actually improves things for the not-1% of 
FreeBSD users?


--lyndon

P.S.  Don't turn this into a pissing match.  I really want to know 
how this is of net benefit to everyone.  But I don't want 
hyperbole.  I have looked at a lot of, e.g., USENIX and ACM, 
bibliographie

Re: [CFT] packaging the base system with pkg(8)

[Alfred Perlstein]

On 4/19/16 7:47 AM, dan_partelly wrote:



Look, take a look at history and the Linux kernel threads story and its
impact on FreeBSD.  If you'd like I can talk about it.


Please, yes, I would love to hear about it.
Sure, so back in late 90s, ~1999 sometime after Solaris released kernel 
threads Linux did as well.


It was pretty laughable, it was basically a hacked up version of fork 
that shared address space, files and a few other things and was horribly 
unstable and you couldn't use most of libc with it as there were many 
single-threaded deps in it.


It was so "laughably bad" that if you logged into a Linux system with 
threads and ran "top", it would look very strange as each thread had its 
own pid.  So if you were running something like mysql it would look like 
you were forkbombed.


Now the point of all of this however was to get threads, and also to get 
threads that did well with disk IO.


FreeBSD at the time had threads, but they were "green threads" or 
"userland threads" which means they were super good for network IO, but 
nearly useless for disk IO.


This made Linux a much better platform for mysql, as mysql performed an 
order of magnitude better on Linux.  In fact not only was FreeBSD 
slower, but in my experience it was buggier because the green threads 
implementation, while very cool, was complex and had its own share of bugs.


Some years passed and many people migrated to Linux to get the high 
performance mysql that was available on the platform.


Now during those years, as more and more people migrated away, FreeBSD 
had many, many, many, oh gosh, so many discussions on what to do about 
threads.  So many opinions were had by so many people during which more 
people moved to Linux as a web platform...  Not only were there 
opinions, but also implementations suggested, but those implementations 
were shot down in the search for the "one true threading system".  Of 
course there were some loud folks that even insisted that threads were 
bad (and probably a phase) and we shouldn't even support them).


Luckily during that time (~2002) someone made this crazy "Linuxthreads" 
port which made """Linuxthreads"""  available on FreeBSD, it had many of 
the same problems, but at least mysql was fast.  Spoiler: The funny part 
about calling it "Linuxthreads" was that "Linuxthreads" wound up being 
the defacto way to do threads across *UNIX* because it was just simple 
and worked, should have just called it "realthreads" in my opinion.  
Luckily with the Linuxthreads port the die hards in the FreeBSD 
community who needed to run mysql for performance (or other software 
requiring real threads) could stay using FreeBSD while the community 
sorted out what exactly was to be done.


Places like Yahoo moved off of FreeBSD because they "just wanted 
threads" and if you tell someone to get "good performance with threads, 
then use Linuxthreads" (and of course make a grumpy face because "it's 
Linux") then they reason, "geez, why I am not just using Linux anyhow?" 
which they then migrate to.


So some years later (~2005) we went with a far more complex solution 
than needed called libkse.  Libkse offered N:M threads, which didn't 
work very well for years due to complexity (hats off to those that 
*tried* to get it working in FreeBSD's political climate) to which 
finally Sun abandoned its N:M threads because they themselves could 
never get it right, and finally FreeBSD switched to 1:1 threads 
following Solaris.  That was probably in 2007 timeline.


Now rewind to ~2001, about 2 years after the introduction of the sloppy, 
weird and strange kernel threads in Linux, and they had a pretty solid 
implementation.  It wouldn't be until nearly 2009 when FreeBSD had a 
truly solid threads library and even now it suffers due to added 
complexity because we did things "smarter" than Linux.


Moral of the story?   My takeaways from this story (and there are many 
similar ones in our recent history) are:


1) Graciously and rapidly accept steps forward and then contribute to 
them.  Anything else leaves you stagnant and worse for wear.

2) Simple over complex.
3) If something someone else did is working for someone, then copy it 
and move on, don't waste a huge amount of your customer's time trying to 
make something better until you are sure that just copying it will not 
suffice.


-Alfred
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] packaging the base system with pkg(8)

[Alfred Perlstein]

On 4/19/16 7:39 AM, dan_partelly wrote:

What should not happen is that this incremental step forward be blocked
by those unwilling to hash out the next steps.

-Alfred



While incremental steps forward are great, how do you avoid situations
like VNET, where a "good enough" enough implementation, usable in some
scenarios lingered for years in kernel, but to this day it suffers from
leaks and bugs. Once you go down the path of enabling it in this state,
chances are that it will stay that way for more than half a decade.





We happened to use VNET at our last company with great success.  Had it 
not existed we would have been much further away from our goals. Maybe 
you picked a bad example? :)


Look, take a look at history and the Linux kernel threads story and its 
impact on FreeBSD.  If you'd like I can talk about it.


-Alfred

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] packaging the base system with pkg(8)

[Alfred Perlstein]

It is very important to understand that a packaged base is extremely 
useful for those building any sort of distro or appliance distro.


So although the concept of "user serviceable" is important, it's not 
just that.  Such a change makes it easy for a distro or appliance making 
to cherry pick updates to the system without having to pull the entire 
system forward.


One of the huge pains about using FreeBSD at my last work was that the 
base system as a whole was a bit challenging to pry apart so that an 
incremental update could happen.  Let's say I needed a patch to openssl, 
well that was HARD even for me.  My choices were to update the whole 
system (which broke things), pull in patches and apply them (hard and 
scary), figure out a way to pull it from ports instead... super hard as 
"base" built before ports in "nanobsd".


Quite frankly I didn't have the time for it.  As someone who laid the 
foundation for an FreeBSD appliance, I wholeheartedly welcome this, it 
will be HUGE for appliance builders.


I am also confident that we will very easily sort out how to make 
"micropackages" or some such mechanism within at most 3 months after the 
code lands.  The reason why is because I already see some excellent 
proposals for such mechanisms in this thread.


-Alfred

On 4/19/16 12:54 AM, David Chisnall wrote:

On 19 Apr 2016, at 08:44, Julian Elischer  wrote:

All this can be done by meta-packages which depend on larger package groups.

Currently Metapackage is a way to make 10 packages look like 11 packages.  The 
framework needs to understand to hide the 10 internal packages if they are part 
of a metapackage.

I agree, and patches to do this are very welcome.  Currently, pkg is short of 
contributors.

I see basically three use cases for a packaged base:

1) People wanting a FreeBSD install to use as a server or workstation.  These 
people will install the FreeBSD 11 metapackage and not care that it is a few 
hundred MBs.  It would be nice if the pkg tool could present this as a single 
package in list views, but that’s a UI issue with pkg, not an issue with the 
number of packages in the base system.

2) People wanting to install embedded systems.  Anyone who has tried to run 
FreeBSD on a system with a small amount of flash storage will have encountered 
the pain of having to use some kind of ad-hoc update.  Being able to manage 
updates to these systems with the same packaging tool as you manage big systems 
is a big improvement.

3) People wanting to install service jails (sorry, containerised applications). 
 These want the smallest possible attack surface and so want the smallest 
amount of the base system that they can have.  Here, small packages are an 
advantage.  It will take a little while for ports to learn enough about the 
granularity of the base system for this to really be useful, but it would be 
great to be able to install nginx, for example, in a jail and have only the 
handful of libraries that it needs.

The big advantage of going with small packages initially, however, is that it 
will allow us to get some data on what the correct groupings are.  If we have 
large packages, then it’s very hard to tell which subsets of the packages 
people want.  That’s exactly the situation that we’re in now: we know some 
people don’t want docs or games, but that’s about all that we know.  It’s easy 
to move to a model where we have *fewer* packages in the future, but it’s 
harder to split them.  That also applies to dependencies.  If I know that a 
port depends on the shell, then it’s easy to update it from depending on a sh 
package to depending on a core system utilities package automatically, but it’s 
very hard to do an automatic update in the other direction.

David



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] packaging the base system with pkg(8)

[Alfred Perlstein]

Again, the point is that those objecting should put aside the time to 
implement what you (and I) are suggesting:



I could live with:

base-utils11.1
 - ktrace  uninstalled
 - tcpdump uninstalled
 + dd  11.1.1   (CVE-123412 fix)



but not
{700 packages )
dd  11.1.1 dd with CVE fix
{29 more packages}
[tcpdump line is not present but you don't notice that]
{10 more packages}
[ktrace line would be here but you don't notice that either]
{15 more packages}


What should not happen is that this incremental step forward be blocked 
by those unwilling to hash out the next steps.


-Alfred


On 4/19/16 12:44 AM, Julian Elischer wrote:

On 19/04/2016 5:29 AM, Alfred Perlstein wrote:
Guys please stop arguing about the number of packages.  The high 
granularity is VERY useful!



it's going to make us a laughing stock
"look FreeBSD just split into 1.43 million packages" (effectively the 
same number.. it's bigger than 10)



Managing large groups of small packages is much easier than just 
having large packages.
err, Alfred, what planet do you live on? when they get out of sync it 
becomes a nightmare.
If you also had a packaging system that was smart enough to manage a 
hierarchy of packages then "maybe"..




All this can be done by meta-packages which depend on larger package 
groups.
Currently Metapackage is a way to make 10 packages look like 11 
packages.  The framework needs to understand to hide the 10 internal 
packages if they are part of a metapackage.


Later pkg can be augmented to "remove packages not explicitly 
installed" which would remove leaf packages.


Example: you installed "base-debug" which pulls in let's say 50 small 
packages, later you want all of those removed, you can do something 
like:  "pkg delete --leafs base-debug" which should delete 
"base-debug" and any dangling packages it pulled in not required by 
other pkgs.


Huge thanks to the team that implemented this!


I'm sure the work was large and will be useful in the future but we 
are not ready to have the system installed like this.
no-one wants to see 750 packages show up when you do an enquiry on a 
newly installed system.

I could live with:

base-utils11.1
 - ktrace  uninstalled
 - tcpdump uninstalled
 + dd  11.1.1   (CVE-123412 fix)



but not
{700 packages )
dd  11.1.1 dd with CVE fix
{29 more packages}
[tcpdump line is not present but you don't notice that]
{10 more packages}
[ktrace line would be here but you don't notice that either]
{15 more packages}


In other words, I have no objection to all the utilities coming in the 
form of little packages.
but I have a major objection if that fact is at all obvious to the end 
user,
and certainly if we need to wade through 750 packages to see what's 
going on.




thanks.
-Alfred

On 4/18/16 1:07 PM, Lev Serebryakov wrote:

On 18.04.2016 22:40, Glen Barber wrote:


This granularity allows easy removal of things that may not be wanted
(such as *-debug*, *-profile*, etc.) on systems with little 
storage.  On

one of my testing systems, I removed the tests packages and all debug
and profiling, and the number of base system packages is 383.

  IMHO, granularity like "all base debug", "all base profile" is enough
for this. Really, I hardly could imagine why I will need only 1 
debug or

profile package, say, for csh. On resource-constrained systems NanoBSD
is much better anyway (for example, my typical NanoBSD installation is
37MB base system, 12MB /boot and 10 packages), and on developer system
where you need profiled libraries it is Ok to install all of them and
don't think about 100 packages for them.

  Idea of "Roles" from old FreeBSD installers looks much better. Again,
here are some "contrib" software which have one-to-one replacements in
ports, like sendmail, ssh/sshd, ntpd, but split all other
FreeBSD-specific code? Yes, debug. Yes, profile. Yes, static libraries.
Yes, lib32 on 64 bit system.

   It seems that it is ideological ("holy war") discussion more than
technical one...



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"






___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] packaging the base system with pkg(8)

[Alfred Perlstein]

Maybe what the "too many packages" folks need to do is write some code 
to hide that it's so many packages.


:)

I think the rule of two feet should be applied here.

What we have is people that have worked quite hard to bring us something 
that we can easily work with, and on the other hand some folks that want 
something they consider even better.  Personally I can't see how having 
the system less granular is better, since having it MORE granular is 
actually harder work.


Can someone on the "too many packages" campaign here explain to me how 
having too fine a granularity stops you from making macro packages 
containing packages?


Because honestly I can't see how having granularity hurts at all when if 
someone wanted to make it less granular all they would have to do is 
make some meta-packages.


-Alfred

On 4/18/16 7:23 PM, Lyndon Nerenberg wrote:

On 2016-04-18 7:01 PM, Roger Marquis wrote:

Can you explain what would be accomplished by testing all or even a
fraction of the possible permutations of base package combinations?  We
don't do that for ports.


The ports tree isn't a mandatory part of the system. And by definition 
it could not be tested that way, since it offers so many alternative 
implementations of specific functionality.



Other operating systems don't do that for
their base packages.


I'm pretty sure Solaris had some fairly hard-core regression tests to 
ensure basic system functionality wouldn't be compromised by 'oddball' 
selections of packages offered up at install time.


> Honestly, some of us are wondering what exactly is
> behind some of these concerns regarding base packages.

The concern is from all of us UNIX dinosaurs who predate the 
fine-grained packaging environment, which just worked, and who now rip 
our (little remaining) hair out due to unsolvable package dependency 
loops in the Linux machines we are forced to administer in order to 
pay rent.  For me, as a sysadmin, I derive a negative benefit from 
this optimization.


I guess what I'm really asking is: where is the peer reviewed research 
that shows this actually improves things for the not-1% of FreeBSD users?


--lyndon

P.S.  Don't turn this into a pissing match.  I really want to know how 
this is of net benefit to everyone.  But I don't want hyperbole.  I 
have looked at a lot of, e.g., USENIX and ACM, bibliographies and 
papers for justification for this, and I can't find it.  It would 
really help (me, at least) if someone could take a moment to point me 
at demonstrable evidence of the benefits of this model.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to 
"freebsd-current-unsubscr...@freebsd.org"




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] packaging the base system with pkg(8)

[Alfred Perlstein]

Guys please stop arguing about the number of packages.  The high 
granularity is VERY useful!


Managing large groups of small packages is much easier than just having 
large packages.


All this can be done by meta-packages which depend on larger package groups.

Later pkg can be augmented to "remove packages not explicitly installed" 
which would remove leaf packages.


Example: you installed "base-debug" which pulls in let's say 50 small 
packages, later you want all of those removed, you can do something 
like:  "pkg delete --leafs base-debug" which should delete "base-debug" 
and any dangling packages it pulled in not required by other pkgs.


Huge thanks to the team that implemented this!

thanks.
-Alfred

On 4/18/16 1:07 PM, Lev Serebryakov wrote:

On 18.04.2016 22:40, Glen Barber wrote:


This granularity allows easy removal of things that may not be wanted
(such as *-debug*, *-profile*, etc.) on systems with little storage.  On
one of my testing systems, I removed the tests packages and all debug
and profiling, and the number of base system packages is 383.

  IMHO, granularity like "all base debug", "all base profile" is enough
for this. Really, I hardly could imagine why I will need only 1 debug or
profile package, say, for csh. On resource-constrained systems NanoBSD
is much better anyway (for example, my typical NanoBSD installation is
37MB base system, 12MB /boot and 10 packages), and on developer system
where you need profiled libraries it is Ok to install all of them and
don't think about 100 packages for them.

  Idea of "Roles" from old FreeBSD installers looks much better. Again,
here are some "contrib" software which have one-to-one replacements in
ports, like sendmail, ssh/sshd, ntpd, but split all other
FreeBSD-specific code? Yes, debug. Yes, profile. Yes, static libraries.
Yes, lib32 on 64 bit system.

   It seems that it is ideological ("holy war") discussion more than
technical one...



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Heads up

[Alfred Perlstein]

Warner thank you very much. 

Sent from my iPhone

> On Apr 14, 2016, at 8:17 PM, Warner Losh <i...@bsdimp.com> wrote:
> 
>> On Thu, Apr 14, 2016 at 8:01 PM, Warner Losh <i...@bsdimp.com> wrote:
>> 
>> 
>> 
>>> On Thu, Apr 14, 2016 at 7:22 PM, Alfred Perlstein <bri...@mu.org> wrote:
>>> 
>>> 
>>> 
>>>> On 4/14/16 3:42 PM, Warner Losh wrote:
>>>> 
>>>> The CAM I/O scheduler has been committed to current. This work is
>>>> described
>>>> in https://people.freebsd.org/~imp/bsdcan2015/iosched-v3.pdf though the
>>>> default scheduler doesn't change the default (old) behavior.
>>>> 
>>>> One possible issue, however, is that it also enables NCQ Trims on ada
>>>> SSDs.
>>>> There are a few rogue drives that claim support for this feature, but
>>>> actually implement data corrupt instead of queued trims. The list of
>>>> known
>>>> rogues is believed to be complete, but some caution is in order.
>>>> 
>>>> Yowch...
>>> 
>>> With data at stake wouldn't a whitelist be better along with a tool for
>>> testing it?
>>> 
>>> Example, you have whitelist and blacklist, if the device isn't on either
>>> list you output a kernel message and suggest they run a tool to "test" the
>>> controller and report back the findings?
>> 
>> 
>> The only way to test it is to enable it. Run it for a day or six. If your
>> data goes away, the drive is a lying sack. There's no tool to detect this
>> that I've seen. You run the NCQ trim, it works. You do it again, it works
>> again. After a while, if you have a bad drive model, bad things happen that
>> are drive model specific.
>> 
>> Did I mention that the black list matches Linux's black list and that only
>> a tiny number of drive models lie. I guess I didn't.
> 
> I just sync it back up to the Linux list. This list has been stable for the
> past year or so, with only one entry added back in August. All the other
> entries came early in Linux's tables. I did add a quirk to allow it on the
> Micron/Crucial M500 with MU07 firmware, but only because I've personally
> tested that on dozens of drives over the past 6 months under Netflix
> streaming video loads after getting the new firmware.
> 
> 
>> I am thinking of adding a tunable to turn it off though for people that
>> are paranoid.
> 
> Actually, since it is already  a quirk, you can use the tunable
> 
> kern.cam.ada.X.quirks=0x2
> 
> to disable NCQ trim. You can change it to 0x3 if you need 4k sectors as
> well. So there's no need to change anything to be able to disable it. Given
> how long Linux has been in the wild with NCQ enabled (approximately 18
> months), I'm guessing their quirk list is going to be fairly complete. I
> have no other systems to cross check this with, but would welcome pointers
> if I've overlooked something. I did this bit of code about 15 months ago,
> but it wasn't until 6 months ago that I had working SSD firmware to test it
> on.
> 
> Warner
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
> 

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Heads up

[Alfred Perlstein]

On 4/14/16 3:42 PM, Warner Losh wrote:

The CAM I/O scheduler has been committed to current. This work is described
in https://people.freebsd.org/~imp/bsdcan2015/iosched-v3.pdf though the
default scheduler doesn't change the default (old) behavior.

One possible issue, however, is that it also enables NCQ Trims on ada SSDs.
There are a few rogue drives that claim support for this feature, but
actually implement data corrupt instead of queued trims. The list of known
rogues is believed to be complete, but some caution is in order.


Yowch...

With data at stake wouldn't a whitelist be better along with a tool for 
testing it?


Example, you have whitelist and blacklist, if the device isn't on either 
list you output a kernel message and suggest they run a tool to "test" 
the controller and report back the findings?


-Alfred
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: New LOR ?

[Alfred Perlstein]

On 2/12/16 12:50 AM, Poul-Henning Kamp wrote:

I don't recall seeing this one before:

FreeBSD critter.freebsd.dk 11.0-CURRENT FreeBSD 11.0-CURRENT #31 r293468: Sat 
Jan  9 11:50:09 UTC 2016 
r...@critter.freebsd.dk:/freebsd/obj/freebsd/svn_src/head/sys/GENERIC  amd64

+taskqueue_drain with the following non-sleepable locks held:
+exclusive sleep mutex bpf global lock (bpf global lock) r = 0 (0x81c53
fe8) locked @ /freebsd/svn_src/head/sys/net/bpf.c:772
+stack backtrace:
+#0 0x80a79ee0 at witness_debugger+0x70
+#1 0x80a7b1f7 at witness_warn+0x3d7
+#2 0x80a6daeb at taskqueue_drain+0x3b
+#3 0x80b4bb0b at ieee80211_waitfor_parent+0x3b
+#4 0x80b32d57 at ieee80211_ioctl+0x2f7
+#5 0x80afe025 at if_setflag+0xd5
+#6 0x80afdefc at ifpromisc+0x2c
+#7 0x80af41e8 at bpf_detachd_locked+0x1e8
+#8 0x80af6cfe at bpf_dtor+0x8e
+#9 0x808f7312 at devfs_destroy_cdevpriv+0x82
+#10 0x808faa85 at devfs_close_f+0x65
+#11 0x809d0e6a at _fdrop+0x1a
+#12 0x809d3fb1 at closef+0x1e1
+#13 0x809d3afd at fdescfree_fds+0x9d
+#14 0x809d361c at fdescfree+0x46c
+#15 0x809e1676 at exit1+0x4e6
+#16 0x809e118d at sys_sys_exit+0xd
+#17 0x80e6b13b at amd64_syscall+0x2db



Yeah, "ifpromisc(ifp, 0);" shouldn't really be called with the global 
bpf lock, it is unneeded and extends the length of the lock held to an 
unacceptable depth.


However it looks like dropping the lock inside of bpf_detachd_locked() 
isn't safe... at least at a glance bpfdetach() relies on BPF_LOCK() 
being held through its entire run, and bpfdetach() calls 
bpf_detachd_locked(), so temporarily unlocking in bpf_detachd_locked() 
would break things.


It may be possible to fix the ieee80211 stack not to call taskqueue 
drain, however in reality the BPF_LOCK is just being held over too much 
code.


Can the call to ifpromisc() be deferred into a taskqueue without causing 
too much grief and/or a race condition?


-Alfred


-Alfred
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Packaging the FreeBSD base system with pkg(8)

[Alfred Perlstein]

This is going to be huge for FreeBSD.  Thank you Glen, Bapt and I 
believe Peter Wemm as well.


Having been engineering lead on multiple appliances based on FreeBSD 
this is going to revolutionize and make life so much easier for future 
appliance endeavors and general manageability of FreeBSD.


Thanks again!

On 1/27/16 2:33 PM, Glen Barber wrote:

As many know, work has been in progress for quite some time to provide
the ability to package and upgrade the FreeBSD base system using pkg(8).
The majority of the initial implementation has provided much of the core
functionality to make this possible, however much work still needs to be
done.

Over the past few weeks, there have been several inquiries on if this
work is still targeted for the 11.0-RELEASE, as well as the status of
the project branch (base/projects/release-pkg).

The answer to the first question is: Yes.  This is still targeted for
11.0-RELEASE, which was one of the requirements during discussion of the
new support model announced early last year [1].

The status of the in progress work is a bit more complex to answer in
a short email, but work on packaging the FreeBSD base system is indeed
ongoing, and has been my primary focus over the past several weeks.

I am finishing an initial list of outstanding items that need to be
resolved before the project branch can feasibly merged back to head,
which I will send to the new freebsd-pkgbase@ mailing list.  People
interested in discussion surrounding this topic are urged to subscribe:

 https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase

Finally, I want to personally thank Baptiste Daroussin for all of his
tireless efforts to get us to the point we are at now.  Without his
ideas and insights, as well as ensuring pkg(8) contained the necessary
functionality, we would not be anywhere close to completing this work
for the 11.0-RELEASE.

[1]
https://lists.freebsd.org/pipermail/freebsd-announce/2015-February/001624.html

Thanks.

Glen



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: libXO-ification - Why - and is it a symptom of deeper issues?

[Alfred Perlstein]

On 11/15/15 10:09 AM, Allan Jude wrote:

On 2015-11-15 13:06, Garrett Cooper wrote:

On Nov 15, 2015, at 09:51, Andrey Chernov  wrote:


On 15.11.2015 20:37, Adrian Chadd wrote:

On 15 November 2015 at 09:10, Dan Partelly  wrote:
Meaning, is that simple to push things in head , if somone does the work, even 
with with no proper review of the problem at hand , and the proposed solutions ?

Nope and yup. The juniper folk had a solution to a problem multiple
people had requested work on, and their proposal was by far the
furthest along code and use wise.

It's all fine and good making technical decisions based on drawings
and handwaving and philosophizing, but at some point someone has to do
the code. Juniper's libxo was the furthest along in implementation and
production.

It seems it is the only and final argument for libXO existence. I
remember 2 or 3 discussions against libXO spontaneously happens in the
FreeBSD lists, all ended with that, approximately: "we already have the
code and you have just speculations". Alternative and more architecture
clean ideas, like making standalone template-oriented parser probably
based on liXO, are never seriously considered, because nobody will code
it, not for other reasons.

We lack a [dtd/json] spec for tools, so programming for xo'ification doesn't 
seems like the best idea in the world to me from a end-user sysadmin/developer 
perspective.

I could just as easily use standard tools like awk, grep, sed, and more 
advanced languages like perl or Python to parse output, and assuming output 
doesn't get a major rewrite, I'd just go with that method that's worked pretty 
well for me over the last 10 years of my career..



The big difference is, a json parser isn't going to blow up if a new
field gets added in the middle, and your awk/grep/sed script probably will.



Alan is exactly correct.  This is EXACTLY why it is important.  It 
allows us to focus on the text UI without worrying about the 5000 tools 
built to parse the output of the utilities.  It allows us to grow a 
useful text UI.


Second is that parsing is as simple as doing "json2struct()" in whatever 
language you're using and then selecting a field as opposed to writing 
terrible regex.


It's basically "libification" of the program without actual libification.

Next step is to actually libify the programs.  Libifying the programs 
would be VERY, VERY cool.  But "libification" doesn't make json output 
not useful.


-Alfred

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: libXO-ification - Why - and is it a symptom of deeper issues?

[Alfred Perlstein]

On 11/16/15 9:39 AM, Dan Partelly wrote:

How big of a donor you are to the FreeBSD Foundation does not affect the
committable of your code. Having code ready to commit, vs just a vague
plan, does help your solution win out over another proposed solution though


Then surely you will salvage something from a lot of GsoCs where people wrote 
code with various degrees of success, only
to never hear again of anintegration, or an  evaluation of that code, and 
possible integration.

One which directly interests me: what happened to the BSD libctf  code from 
GSoc ? Was the resulted code evaluated ? If
it falls short, where it does ? Can it be salvaged ?

Libxo might be a fine facility to have for some corner cases, but it doesnt 
solve the problem of binary code reuse in general. Might have solved it fast 
for Juniper.  It is yet another stick into a scaffolding of shell scripts which 
should have been replaced years ago with proper libraries, services and IPC, 
opening the road towards modern service management, service frameworks , fault 
management , fault response and transactional OS databases
   
I continue to believe this is or will become shortly an issue of utmost importance , one which is worthy of the status of a FreeBSD

initiated and sponsored object.


Yes.  We will get there.

-Alfred (who pushed for libxo as well)
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: libXO-ification - Why - and is it a symptom of deeper issues?

[Alfred Perlstein]

On 11/16/15 9:16 AM, Allan Jude wrote:

On 2015-11-16 12:09, Elizabeth Myers wrote:

On 15/11/15 06:54, Dan Partelly wrote:

Hi all,

I was looking at the new facility of dumping JSON,XML from many utils in base 
and after some funny minutes, I couldn't stop ask myself “ Ok, this is funny , 
but why ? “ And I couldn't find a real answer. Ill outline what I think:


1. Undoubtedly, it makes base code slightly harder to understand and maintain.
2. I have seen the idea that this makes the information dumped by utilities in 
the base easily accessible programatically. OK, maybe it does , but
it doesn't fit with the current paradigm of "tool | filter | tool” at all. 
There are no tools able to accept JSON and filter it in any meaningful way, and I
dont see too many ppl changing their code to read JSON instead of text.  I 
don't even see the base tools changing. This output may be useful in corner 
cases only.
3. The integration of libxo IMO only points at a much deeper issue IMO. It is 
only an expression of the need of a mechanism aimed at binary code reuse. But 
it does not solve the problem, it only adds yet another possibility in a world 
where too much choices already result in too much splits and incompatible APIs.
4. This whole effort would have been IMO much better served  by porting the 
bulk of ifconfig(8) , route(8) and wpaclient(8) to a library API, much like the 
libs for geom, zfs , etc , ready for reuse of 3rd party code. Eventually 
writing network control daemons in time over it , much like solaris does.

5. A port of partial OS config data to UCL …. would induce yet induce another 
orthogonality violation. What makes UCL better than the bestiary of ad hoc 
databases already existing in BSDs ? Programatic readability, yes. but it does 
not add any real much needed functionality such as *transactional databases* 
for system tools.   Why not research a proper solution - easily accessible by 
other programs ,orthogonal , transactional, and ACL protected   solution  which 
can be used all over the place , from OS boot, to ABI management, service 
management, network management, user management.  I hope this day will come, a 
day when I will not have to edit a single config file manually, yet I would 
have access to all the config and system state  easy with wrapper APIs. In the 
light of this point, why go with UCL ? It is not orthogonal, it is not 
transnational, and editing the config files directly would result in the same 
old human errors which bite as all from time to time.

5. It is my opinion that Solaris addressed some of those issue. Solaris FMRI 
and SMF are lightyears ahead of the very tired models we keep using on BSDs. 
Why not build on the insight offered by those (or even on the insight offered 
by Windows :P) , then inventing more adhoc solutions and ad-hoc databases, 
which do not address the real issues we have , like binary code reuse, service 
management issues,  lack of a system wide published -subscriber bus ( not kdbus 
:P ) fault detection and reaction, fault reporting, all much needed parts of a 
modern OS.

And now thee questions

1. Why lib XO ? Why burden the OS for some corner cases where it may be useful ?

2. Was there any real talk on how to bring FreeBSD up to speed regarding those 
issues ?  A period of research on what exists, on what can be done , and ensure 
important things are not showed in background and replaced with yet another 
ad-hoc config database which lacks modern features ?
 From where I am standing, this could be a project spawning multiple years , 
but it would be well worth it, and in my opinion it would be also worthy of
the freeSBD foundation sponsorship for several years in a row. The features I 
touched upon became very important parts of oder OSes, and rightly so.

Note:

this message is serious and it is not intended to start flame wars, religious 
crusades, or offend anyone.
  
___

freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

It seems to boil down to the golden rule: he who has the gold, makes the
rules. Juniper wanted it, they're a non-trivial donor to the FreeBSD
foundation and employ many devs, so they got their way.

That's all there is to it.


As Simon pointed out, many more people than Juniper wanted it. Juniper
had a less generalized version of this in their own tree, and would have
happily kept it to them selves, but they went to the extra effort of
generalizing it and cleaning it up to commit it to base, because at a
FreeBSD Vendor Summit, a number of other companies wished for the same
functionality.

Even my small 3 person company greatly desires this functionality, and
this is why I have been helping to convert additional utilities to use
libxo.

How big of a donor you are to the FreeBSD Foundation does not affect the
committable of your code. Having 

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Alfred Perlstein]

Lars,

Try to remove .git/gc.log then re-run fetch.

If that doesn't work then move ".git/refs/remotes/origin/HEAD" to backup 
location outside of your .git directory and try again.


-Alfred

On 11/11/15 4:03 AM, Eggert, Lars wrote:

Hi,

I just got this error when fetching from remote; related?

[elars@laurel: ~/src] git fetch --all
Fetching origin
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
Fetching upstream
remote: Counting objects: 557, done.
remote: Compressing objects: 100% (543/543), done.
remote: Total 557 (delta 213), reused 2 (delta 2), pack-reused 0
Receiving objects: 100% (557/557), 1.15 MiB | 433.00 KiB/s, done.
Resolving deltas: 100% (213/213), completed with 2 local objects.
 From github.com:/freebsd/freebsd
b4eb11a..3eb0ea4  master -> upstream/master
f147893..9c319c0  stable/10  -> upstream/stable/10
e901edd..b3c9fd2  stable/8   -> upstream/stable/8
81ab2b1..2fc7a9a  stable/9   -> upstream/stable/9
c2c933c..cc76737  svn_head   -> upstream/svn_head
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
error: The last gc run reported the following. Please correct the root cause
and remove .git/gc.log.
Automatic cleanup will not be performed until the file is removed.

fatal: bad object refs/remotes/origin/HEAD
error: failed to run repack

Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
error: The last gc run reported the following. Please correct the root cause
and remove .git/gc.log.
Automatic cleanup will not be performed until the file is removed.

fatal: bad object refs/remotes/origin/HEAD
error: failed to run repack

Lars


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: SVN to GIT converter currently broken, github is falling behind

[Alfred Perlstein]

On 11/5/15 6:46 AM, Ulrich Spörlein wrote:

2015-11-04 18:57 GMT+01:00 Ulrich Spörlein :

The recent SVN update on the cluster broke svn2git in certain circumstances.

To fix this and make sure no content was dropped, the converter has
been stopped and we're working on bringing a fixed version online, as
well as vetting the correctness of the published git repositories.

ETA is currently unknown, expect an update to this thread within 24h.
Sorry for the inconvenience!

Uli, on behalf of git-admin

An independent run of the converter produces a different git
repository starting at the commit following this one:
https://github.com/freebsd/freebsd/commit/bf66c97c4a64e64410bf0223d221a54ca9555f52

This is from 9d ago and will likely require a force push to github
that will necessitate people to rebase or merge there work (a
fast-forward merge will fail).

This is the preliminary inspection and a third verification run is
currently underway. Expect another update within 24h.


Uli,

One of the biggest concerns I've heard from folks using FreeBSD's git 
mirror is that the hashes can change.


I have a question about this.   Is it possible to keep track of what the 
"official" git mirror (on github) is doing and keep that as a log.  Then 
that log can be used to replay commits when there is a divergence problem.


What I'm basically saying is that let's take this small example:

importer is working fine @rev 1
imports 1
imports 10001
imports 10002
something happens to importer to give indeterminate shas.
imports 10003 - sha is "unstable" sha3
imports 10004 - sha is "unstable" sha4
imports 10005 - sha is "unstable" sha5
imports 10006 - sha is "unstable" sha6
importer is fixed


At this point normally we'd rewind the importer to 10002 and then force 
update the affected branches.


My question is... can the imports of 10003, 10004, 10005 and 10006 be 
put into the importer such that any "mirror site" that re-does the 
import using the most up to date importer will get the same shas.


That would allow to proceed with 10007, etc without force pushing.

This should be possible based on querying "git" for the meta data 
associated with sha3..sha6 and then forcing those commits to have the 
same meta data.


This would eliminate the concern about shas in the mirror changing that 
I've heard.


-Alfred








___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Segmentation fault running ntpd

[Alfred Perlstein]

Adrian the crash we are seeing here is very easily reproducible. Grab 
our private ports repo and revert my most recent revert and build.  It 
appears to show up multiple times per day somehow in our configuration.


On 7/28/15 7:25 PM, Adrian Chadd wrote:

On 28 July 2015 at 16:09, David Wolfskill da...@catwhisker.org wrote:

On Tue, Jul 28, 2015 at 04:05:33PM -0700, Adrian Chadd wrote:

Is this still happening for you?


g1-245(10.2-P)[4] ls -lT /S4/ntpd.core
-rw-r--r--  1 root  wheel  13783040 Jul 28 04:56:28 2015 /S4/ntpd.core

Apparently so, yes.

(/S4 is where I have the head root file system mounted when I'm not
running from slice 4.)

Hm, is there any way you can get symbols for it?

I don't think I can easily get symbols for the crash we have, but my
crash is also deep in malloc code..


-a
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org



___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Increase BUFSIZ to 8192

[Alfred Perlstein]

On 5/14/15 2:23 AM, Garrett Cooper wrote:

On May 14, 2015, at 1:06, Poul-Henning Kamp p...@phk.freebsd.dk wrote:



In message 20150514075316.gy37...@funkthat.com, John-Mark Gurney writes:

Poul-Henning Kamp wrote this message on Thu, May 14, 2015 at 07:42 +:


In message 20150514072155.gt37...@funkthat.com, John-Mark Gurney writes:


Since you apprently missed my original reply, I said that we shouldn't
abuse BUFSIZ for this work, and that it should be changed in mdXhl.c...

Say what ?

BUFSIZ is used entirely appropriately in MDXFileChunk():  For reading
a file into an algorithm.

In fact, posix-2008 references LINE_MAX because:

MDXFileChunk() does not read lines, it reads an entire file.

Being pedantic, technically it’s a portion of a file, which can be the whole thing, 
and it reads it in “sizeof(buffer)” chunks (of which buffer is “hardcoded to 
BUFSIZ right now).
Cheers!

Shouldn't most of these be using st.st_blksize ?

I recall being part of the move to get rid of PAGE_SIZE, perhaps many 
places should be rid of BUFSIZE as well and BUFSIZE should be something 
we query the system for.


-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Merging GitHub Pull Requests into Subversion using git-svn

[Alfred Perlstein]

[[ reply private ]]

On 4/25/15 12:30 AM, David Chisnall wrote:

On 23 Apr 2015, at 00:12, Craig Rodrigues rodr...@freebsd.org wrote:

While not as smooth as clicking a merge button in GitHub,
this is a valid way to accept patches submitted via GitHub pull requests,
and integrate them in our FreeBSD Subversion repo.

The merge button on GitHub does the wrong thing anyway (merges without 
fast-forward, so you end up with a tangled history), so (after the initial 
setup) the steps that I use for merging pull requests from GitHub projects are 
very similar (locally pull the branch with fast-fordward, test, push).
Not to bikeshed this, but you really almost never want a fast-forward 
commit.  The reason is that it becomes challenging to git-bisect things 
to sort out where a bad commit was.


In addition then the merge is actually one atomic commit.

Getting over viewing merge commits as messy was the final hurdle I 
faced going towards git-nirvana.


-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Merging GitHub Pull Requests into Subversion using git-svn

[Alfred Perlstein]

Very cool.  Glad it worked and thanks for the shout-out.

Hoping this can be automated some day.

On 4/22/15 4:12 PM, Craig Rodrigues wrote:

Hi,

Alfred Perlstein recently wrote this document for how to use
git-svn for interacting between the FreeBSD Subversion repo,
and the GitHub mirror of this repo:

https://wiki.freebsd.org/GitWorkflow/GitSvn

By following the steps in that article, step-by-step,
I was able to:

(1)  take these three GitHub pull requests from Steve Kiernan:

https://github.com/freebsd/freebsd/pull/26
https://github.com/freebsd/freebsd/pull/27
https://github.com/freebsd/freebsd/pull/28

(2)  Pull them into my own git checkout of the FreeBSD src tree

(3)  Modify the commit message slightly

(4)  Use git svn dcommit to push these changes directly from my Git tree
back to the
   FreeBSD svn repo:

https://svnweb.freebsd.org/changeset/base/281844
https://svnweb.freebsd.org/changeset/base/281845
https://svnweb.freebsd.org/changeset/base/281855

While there were multiple steps involved, I just followed the steps in the
wiki article, and it *just worked*!  Thanks for writing this article,
Alfred!

While not as smooth as clicking a merge button in GitHub,
this is a valid way to accept patches submitted via GitHub pull requests,
and integrate them in our FreeBSD Subversion repo.

--
Craig
___
freebsd-po...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to freebsd-ports-unsubscr...@freebsd.org


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Bazaaring the cathedral (Lowering the Barrier to Entry)

[Alfred Perlstein]

On 4/2/15 6:53 AM, Julian H. Stacey wrote:

Hans Petter Selasky wrote:

I hope this is not one more of those April fools :-)

I've been thinking that since Eitan's first post of
Wed, 1 Apr 2015 09:55:11 -0700 (18:55 CEST)
self-serve commit access
I kept wondering what would keep looneys out ? :-)

Your experience feeding back to Linux was interesting, I suppose
we assume the grass is greener till we hear someone tried it :-)



Agreed.  Contributing to the git project itself was quite eye opening.   
They are very particular about the patch format and a few processes 
involved were very long.  They also don't use github. That said, the 
community was pretty open to the patches (once I got the format correct) 
and it took about average amount of work to get my code submitted.


-Alfred


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Massive libxo-zation that breaks everything

[Alfred Perlstein]

On 3/4/15 8:21 AM, John Baldwin wrote:

On Tuesday, March 03, 2015 09:09:43 AM David Chisnall wrote:

Hopefully there's a lesson here that we can learn from: human-readable
formats do not make good intermediate representations when communicating
between tools.


I think this is actually an argument against libxo-ification in the one case
where I've cringed a bit at the diffs: pciconf.  The current pciconf code is
tailored to outputting something human readable.  For non-human output I would
probably generate different output (not just put tags on the human output)
because I would want the non-human output to be both more verbose and more
raw.  I think some other cases like 'netstat -s' are far more straightforward
as the current output maps fairly well to the backing structure, but in
general I would want machine-readable output that is closer to the structures
than to the human-readable formatting of them.

For example, for something like 'mfiutil show drives', I would want the human
readable format to stay as it is (it only highlights certain fields in the PD
structures) but I would want the machine-readable format to basically output
tagged versions of the backing structures from sys/dev/mfi/mfireg.h.  That way
the machine-readable format has all of the data instead of only the subset
that is presented in the human-readable output.

So while I am in general a big fan of having machine-readable output from
tools (and I think it belongs in the base system, and I don't think you want a
post-processing tool), I think there is a bit of a flawed assumption that says
that I want the same data in the human-readable format that I want in the
machine-readable format.  I, for one, don't.  I want the human-readable form
more condensed.


If your argument is about maintainability of these changes, then please
point to concrete instances where the changes are complex and difficult to
maintain.


When I've looked at the xo diffs for pciconf, my reaction has been ugh, I
guess I'm not going to work on pciconf again in the future because that's
super ugly.  I don't object to the idea, I think I would just rather have a
very different schema for machine-readable output.


+1


I would probably want
pciconf -l in that case to dump the entire PCI header (right now the human-
readable pciconf -l only dumps a subset), and I would want it to dump fields
in capabilities that we don't currently bother printing (and that I don't
think the human-readable output should print due to it being too obscure,
etc.)



I actually agree on this and this is why I was upset that the more 
straightforward GSoC code was shot down in favor of this.  That said 
don't we all need to look at the greater good when making software and 
we have some consensus on this so its worth going forward imo.


We can agree on something even though it might make hairs stand up or we 
just stagnate.

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Massive libxo-zation that breaks everything

[Alfred Perlstein]

Sent from my iPhone

 On Mar 3, 2015, at 9:32 AM, hiren panchasara hi...@strugglingcoder.info 
 wrote:
 
 On 03/02/15 at 07:33P, Alfred Perlstein wrote:
 
 
 
 Actually I want to shame third party ports into adopting libxo (or at least 
 providing machine readable output). 
 
 I know it's scary to try to lead the pack, after all we could be wrong, but 
 maybe it's time to try something new and see what happens. 
 
 And no, your idea doesn't make sense it just will lead to those files bit 
 rotting.  
 
 Bedsides that you don't even have a real spec other than it should be done 
 differently. 
 
 Again, show the code.
 
 Wow. He told you want he didn't like and he moved on. I hope/wish we as
 a project can take criticism more positively than this.
 
 Answer to every criticism should not be show me your code. We (should)
 know better than that.

Maybe I am too old school but the motd on freefall is still now shut up and 
code. I still believe that's right.  

We ALL need to know when to step back from an issue we really do not have the 
bandwidth to constructively contribute to. 

This doesn't mean ideas or concerns aren't valid, but that unless you truly 
have the bandwidth to contribute, taking a step back is a good idea. 

-Alfred. 
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Massive libxo-zation that breaks everything

[Alfred Perlstein]

 On Mar 3, 2015, at 11:07 AM, Justin Hibbits jr...@alumni.cwru.edu wrote:
 
 On Tue, Mar 3, 2015 at 10:39 AM, Alfred Perlstein bri...@mu.org wrote:
 
 
 Sent from my iPhone
 
 On Mar 3, 2015, at 9:32 AM, hiren panchasara hi...@strugglingcoder.info 
 wrote:
 
 On 03/02/15 at 07:33P, Alfred Perlstein wrote:
 
 
 Actually I want to shame third party ports into adopting libxo (or at 
 least providing machine readable output).
 
 I know it's scary to try to lead the pack, after all we could be wrong, 
 but maybe it's time to try something new and see what happens.
 
 And no, your idea doesn't make sense it just will lead to those files bit 
 rotting.
 
 Bedsides that you don't even have a real spec other than it should be 
 done differently.
 
 Again, show the code.
 
 Wow. He told you want he didn't like and he moved on. I hope/wish we as
 a project can take criticism more positively than this.
 
 Answer to every criticism should not be show me your code. We (should)
 know better than that.
 
 Maybe I am too old school but the motd on freefall is still now shut up and 
 code. I still believe that's right.
 
 We ALL need to know when to step back from an issue we really do not have 
 the bandwidth to constructively contribute to.
 
 This doesn't mean ideas or concerns aren't valid, but that unless you truly 
 have the bandwidth to contribute, taking a step back is a good idea.
 
 -Alfred.
 
 You're both right.  Maybe we need a hackathon on this.  Is BSDCan too late?

Hackathon on what exactly?

-Alfred 
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Massive libxo-zation that breaks everything

[Alfred Perlstein]

On 3/2/15 4:57 PM, Adrian Chadd wrote:

.. we can/should do both.

Just make sure the json/html/xml output is versioned, otherwise you're
going to end up with /exactly the same problems/ you have with the
current format.


+1


-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Massive libxo-zation that breaks everything

[Alfred Perlstein]

 On Mar 2, 2015, at 4:22 PM, Andrey Chernov a...@freebsd.org wrote:
 
 On 02.03.2015 22:55, Julian Elischer wrote:
 On 3/2/15 5:27 AM, Alfred Perlstein wrote:
 
 
 On 3/2/15 4:14 AM, Julian Elischer wrote:
 On 3/1/15 10:49 AM, Harrison Grundy wrote:
 Thanks!
 
 That does seem useful, but I'm not sure I see the reasoning behind
 putting into base, over a port or package, since processing XML in base
 is a pain, and it can't serve up JSON or HTML without additional
 utilities anyway.
 
 (If I'm reviving a long-settled thing, let me know and I'll drop it.
 I'm
 trying to understand the use case for this.)
 
 To me it would almost seem more useful to have a programmable filter
 for which you could produce
 parse grammars to parse the output of various programs..
 thus
 
 ifconfig -a | xmlize -g ifconfig | your-favourite-xml-parser
 with a set of grammars in /usr/share/xmlize/
 then we could use it for out-of-tree programs as well if we wrote
 grammars for them..
 
 The sentiment of machine-readable output is nice, but I think it's
 slightly off target.
 we shouldn't have to change all out utilities, and it isn't going to
 help at all with 3rd party apps,
 e.g. samba stuff. A generally easy to program output grammar parser
 would be truely useful.
 and not just for FreeBSD.
 
 I've been watching with an uncomfortable feeling, but it's taken me a
 while to put my
 finger on what it was..
 Are you sure it's not the hairs on the back of your neck standing up
 due to NIH?
 
 Juniper has been doing this for years and it's very useful for them.
 I'm not saying the ability to generate machine readable output is wrong,
 but that the 'unix way' would be to make a filter for it. It seems that
 the noisy people don't
 agree with me so I will not stand in the way of progress..
 
 I agree. Even if someone starts with json and xml only, it will need
 some 3rd format soon, and adding any new format have real possibility to
 break all already existent (like adding json+xml breaks plain text in
 pipes). Moreover, it violates Unix principle 'one tool == one general
 function' and lots of other rules like Eric Raymond ones, making each
 program looks like systemd. It makes harder to merge changes from other
 BSDs too.
 Proper way to do this thing is to back out all changes and write
 completely separate templates-based parser - xml/json writer.


Read the library. It doesn't care what output format it needs. It is up to the 
translation layer to do it. You could even do a csv format or most any other 
structured output format without changing the userland utils. 





 
 -- 
 http://ache.vniz.net/
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
 
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Massive libxo-zation that breaks everything

[Alfred Perlstein]

 On Mar 2, 2015, at 6:27 PM, Andrey Chernov a...@freebsd.org wrote:
 
 
 The responsibility is on you to provide something better, both the 
 architecture AND code. So if you want it backed out, then write something 
 better. Otherwise step back and let progress happen.
 
 As it seems you know a lot about my responsibility and duty, I am very
 surprised by your broad interests. If you let me to speak for myself a
 bit, I can tell what I feel. In this particular case my responsibility
 is just to give good advice at the road fork with one way leads to
 obvious hell, nothing more. I already express my opinion from the
 technical point of view and don't want to participate in the flame war,
 so continue this thread without me, please.
 

Other bad ideas (in the past) were support for threads, kernel threads, SMP, 
etc in the project's past. So maybe trying something new might be a good idea. 
At the end of the day, if in a year we think it's really terrible then it can 
be replaced, ripped out, or better yet can be fixed.  

What's worse than a bad idea is not moving on any ideas. 

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Massive libxo-zation that breaks everything

[Alfred Perlstein]

 On Mar 2, 2015, at 7:14 PM, Julian Elischer jul...@freebsd.org wrote:
 
 On 3/2/15 5:30 PM, Alfred Perlstein wrote:
 
 On Mar 2, 2015, at 4:22 PM, Andrey Chernov a...@freebsd.org wrote:
 
 On 02.03.2015 22:55, Julian Elischer wrote:
 On 3/2/15 5:27 AM, Alfred Perlstein wrote:
 
 
 On 3/2/15 4:14 AM, Julian Elischer wrote:
 On 3/1/15 10:49 AM, Harrison Grundy wrote:
 Thanks!
 
 That does seem useful, but I'm not sure I see the reasoning behind
 putting into base, over a port or package, since processing XML in base
 is a pain, and it can't serve up JSON or HTML without additional
 utilities anyway.
 
 (If I'm reviving a long-settled thing, let me know and I'll drop it.
 I'm
 trying to understand the use case for this.)
 To me it would almost seem more useful to have a programmable filter
 for which you could produce
 parse grammars to parse the output of various programs..
 thus
 
 ifconfig -a | xmlize -g ifconfig | your-favourite-xml-parser
 with a set of grammars in /usr/share/xmlize/
 then we could use it for out-of-tree programs as well if we wrote
 grammars for them..
 
 The sentiment of machine-readable output is nice, but I think it's
 slightly off target.
 we shouldn't have to change all out utilities, and it isn't going to
 help at all with 3rd party apps,
 e.g. samba stuff. A generally easy to program output grammar parser
 would be truely useful.
 and not just for FreeBSD.
 
 I've been watching with an uncomfortable feeling, but it's taken me a
 while to put my
 finger on what it was..
 Are you sure it's not the hairs on the back of your neck standing up
 due to NIH?
 
 Juniper has been doing this for years and it's very useful for them.
 I'm not saying the ability to generate machine readable output is wrong,
 but that the 'unix way' would be to make a filter for it. It seems that
 the noisy people don't
 agree with me so I will not stand in the way of progress..
 I agree. Even if someone starts with json and xml only, it will need
 some 3rd format soon, and adding any new format have real possibility to
 break all already existent (like adding json+xml breaks plain text in
 pipes). Moreover, it violates Unix principle 'one tool == one general
 function' and lots of other rules like Eric Raymond ones, making each
 program looks like systemd. It makes harder to merge changes from other
 BSDs too.
 Proper way to do this thing is to back out all changes and write
 completely separate templates-based parser - xml/json writer.
 
 Read the library. It doesn't care what output format it needs. It is up to 
 the translation layer to do it. You could even do a csv format or most any 
 other structured output format without changing the userland utils.
 As far as I can see that's not an argument either way.
 I just think it makes more sense to spend more time writing one generic 
 converter and grammar files than to mess up the insides of every utility in 
 the system. If we had a tool, we could have grammar templates for 3rd party 
 tools easily.. do YOU want to make libxo changes to 3rd party ports? of 
 course not. so you are going off here solving a half of the problem.

Actually I want to shame third party ports into adopting libxo (or at least 
providing machine readable output). 

I know it's scary to try to lead the pack, after all we could be wrong, but 
maybe it's time to try something new and see what happens. 

And no, your idea doesn't make sense it just will lead to those files bit 
rotting.  

Bedsides that you don't even have a real spec other than it should be done 
differently. 

Again, show the code. 


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Massive libxo-zation that breaks everything

[Alfred Perlstein]

 On Mar 2, 2015, at 5:37 PM, Andrey Chernov a...@freebsd.org wrote:
 
 On 03.03.2015 4:30, Alfred Perlstein wrote:
 
 
 On Mar 2, 2015, at 4:22 PM, Andrey Chernov a...@freebsd.org wrote:
 
 On 02.03.2015 22:55, Julian Elischer wrote:
 On 3/2/15 5:27 AM, Alfred Perlstein wrote:
 
 
 On 3/2/15 4:14 AM, Julian Elischer wrote:
 On 3/1/15 10:49 AM, Harrison Grundy wrote:
 Thanks!
 
 That does seem useful, but I'm not sure I see the reasoning behind
 putting into base, over a port or package, since processing XML in base
 is a pain, and it can't serve up JSON or HTML without additional
 utilities anyway.
 
 (If I'm reviving a long-settled thing, let me know and I'll drop it.
 I'm
 trying to understand the use case for this.)
 
 To me it would almost seem more useful to have a programmable filter
 for which you could produce
 parse grammars to parse the output of various programs..
 thus
 
 ifconfig -a | xmlize -g ifconfig | your-favourite-xml-parser
 with a set of grammars in /usr/share/xmlize/
 then we could use it for out-of-tree programs as well if we wrote
 grammars for them..
 
 The sentiment of machine-readable output is nice, but I think it's
 slightly off target.
 we shouldn't have to change all out utilities, and it isn't going to
 help at all with 3rd party apps,
 e.g. samba stuff. A generally easy to program output grammar parser
 would be truely useful.
 and not just for FreeBSD.
 
 I've been watching with an uncomfortable feeling, but it's taken me a
 while to put my
 finger on what it was..
 Are you sure it's not the hairs on the back of your neck standing up
 due to NIH?
 
 Juniper has been doing this for years and it's very useful for them.
 I'm not saying the ability to generate machine readable output is wrong,
 but that the 'unix way' would be to make a filter for it. It seems that
 the noisy people don't
 agree with me so I will not stand in the way of progress..
 
 I agree. Even if someone starts with json and xml only, it will need
 some 3rd format soon, and adding any new format have real possibility to
 break all already existent (like adding json+xml breaks plain text in
 pipes). Moreover, it violates Unix principle 'one tool == one general
 function' and lots of other rules like Eric Raymond ones, making each
 program looks like systemd. It makes harder to merge changes from other
 BSDs too.
 Proper way to do this thing is to back out all changes and write
 completely separate templates-based parser - xml/json writer.
 
 
 Read the library. It doesn't care what output format it needs. It is up to 
 the translation layer to do it. You could even do a csv format or most any 
 other structured output format without changing the userland utils.
 
 I am happy the library can do it. So please stop to change userland
 utils and back out all libxo changes from them. My concern is userland
 utils, feel free to implement anything you need/want without changing
 them in this ugly way.


The responsibility is on you to provide something better, both the architecture 
AND code. So if you want it backed out, then write something better. Otherwise 
step back and let progress happen. 

-Alfred. 

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Massive libxo-zation that breaks everything

[Alfred Perlstein]

On 3/2/15 4:25 AM, David Chisnall wrote:

On 2 Mar 2015, at 09:16, Julian Elischer jul...@freebsd.org wrote:

if we develop a suitable post processor with pluggable grammars, we save a lot 
of work.
given enough examples you could almost have automatically generated grammars.

This decoupled approach is problematic.  A large part of the point of libxo is 
to allow changing the human-readable output without breaking tools that consume 
the output.  Now I need to keep the tool that consumes it and the tool that 
produces it in sync, so that's an extra set of moving parts.  When you throw 
jails with multiple versions of world into the mix, it becomes a recipe for 
disaster.


+1
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Massive libxo-zation that breaks everything

[Alfred Perlstein]

On 3/2/15 4:14 AM, Julian Elischer wrote:

On 3/1/15 10:49 AM, Harrison Grundy wrote:

Thanks!

That does seem useful, but I'm not sure I see the reasoning behind
putting into base, over a port or package, since processing XML in base
is a pain, and it can't serve up JSON or HTML without additional
utilities anyway.

(If I'm reviving a long-settled thing, let me know and I'll drop it. I'm
trying to understand the use case for this.)


To me it would almost seem more useful to have a programmable filter 
for which you could produce

parse grammars to parse the output of various programs..
thus

ifconfig -a | xmlize -g ifconfig | your-favourite-xml-parser
with a set of grammars in /usr/share/xmlize/
then we could use it for out-of-tree programs as well if we wrote 
grammars for them..


The sentiment of machine-readable output is nice, but I think it's 
slightly off target.
we shouldn't have to change all out utilities, and it isn't going to 
help at all with 3rd party apps,
e.g. samba stuff. A generally easy to program output grammar parser 
would be truely useful.

and not just for FreeBSD.

I've been watching with an uncomfortable feeling, but it's taken me a 
while to put my

finger on what it was..


Are you sure it's not the hairs on the back of your neck standing up due 
to NIH?


Juniper has been doing this for years and it's very useful for them.

-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Massive libxo-zation that breaks everything

[Alfred Perlstein]

On 3/2/15 2:53 PM, Julian Elischer wrote:

On 3/2/15 5:25 AM, Alfred Perlstein wrote:



On 3/2/15 4:25 AM, David Chisnall wrote:

On 2 Mar 2015, at 09:16, Julian Elischer jul...@freebsd.org wrote:
if we develop a suitable post processor with pluggable grammars, we 
save a lot of work.
given enough examples you could almost have automatically generated 
grammars.
This decoupled approach is problematic.  A large part of the point 
of libxo is to allow changing the human-readable output without 
breaking tools that consume the output.  Now I need to keep the tool 
that consumes it and the tool that produces it in sync, so that's an 
extra set of moving parts.  When you throw jails with multiple 
versions of world into the mix, it becomes a recipe for disaster.

why? the jail has it own /usr/share?




+1


 I think the risk is exactly opposite.  That the human readable output 
will change subtly with bugs in the xo implementation.
and people will not update the two output paths in exactly the same 
way, leading bugs. I'm not going to fight on it, but I am 
uncomfortable with it. 
So you mean that we're going to have to act like mature software devs 
and have regression tests (atf) and such?  I welcome such a change.



You are increasing the complexity of every program you touch.

And its utility as well.  Worth it.

-Alfred


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Massive libxo-zation that breaks everything

[Alfred Perlstein]

On 3/1/15 4:29 PM, Rui Paulo wrote:

On Mar 1, 2015, at 11:11, David Chisnall thera...@freebsd.org wrote:

How would it be in a port?  It involves modifying core utilities (some of 
which, like ifconfig, rely on kernel APIs that change between releases) to emit 
structured output.  Maintaining two copies of each utility, one in the base 
system with plain-text output only and another in ports with XML/JSON output 
would be very painful.

It would work fine if we had *libraries* for ifconfig/netstat/route/etc.  Obviously 
that's not the case and no one has stepped up to implement them.  I've also seen FreeBSD 
committers expressing their distaste for libraries for trivial command line 
utilities, which implies they are unaware of another world beyond the CLI.  :-)


+1.

Next step is making those utils into libraries.  Should be easy, just 
make all globals into TLS and don't munge stdin/stdout in dumb ways and 
we're there.


Glad you are a big thinker Rui. :)

-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Massive libxo-zation that breaks everything

[Alfred Perlstein]

On 3/1/15 4:57 PM, Harrison Grundy wrote:

I like the idea behind this... where I'm running into difficulty is why
these bits of functionality need to be combined. What someone does with
ifconfig on the command line, versus what someone wants to know about
their network interfaces in an XML dump may be very distinct bits of
information. (For instance, an XML dump of network information may want
to incorporate data from netstat, route, and ifconfig, while ifconfig is
really only designed to deal with the interfaces themselves.)
Part of this is putting the burden of machine readable output into the 
utilities themselves.  Meaning that people have complained against libxo 
because well you can get the same information by combining data points 
X, Y and Z from sysctl whereas ifconfig just does that for you.  So if 
we split the code then those interfaces would wither and die from 
neglect because honestly very few core freebsd devs care about XML and JSON.


So in order to prevent bitotting and make this a feature of FreeBSD, 
we decided to put it into the base utils so that we would be sure to 
carry the burden and keep it working.



I know it involves some notable reworking, but would there be any
interest in seeing what libifying a handful of binaries and splitting
the libxo and CLI functionality looks like? It seems like this may give
FreeBSD the best of both worlds, so the XML output isn't logically
limited by how the CLI needs to work, and the CLI won't have to deal
with the complexity XML output can require.

I'd be happy to get this set up for a few binaries so people can see how
it might work and what it could look like if there's any interest in
going that route.


Please do.  That would be great.  Please make sure they are thread safe. :)

-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: default pager (csh)

[Alfred Perlstein]

On Feb 22, 2015, at 8:41 AM, RW wrote:

 On Fri, 20 Feb 2015 14:18:24 +1030
 Shane Ambler wrote:
 
 On 20/02/2015 01:22, RW wrote:  
 On Wed, 18 Feb 2015 19:16:59 -0600
 Mike Karels wrote:
 
 
 Trivia: the version of more on BSD systems actually is derived from
 less, not the original version of more.  
 
 Actually, more is less
 
 $ md5 -r  `which less ` `which more `
 50404f1beaa4e1261407190a88494b59 /usr/bin/less
 50404f1beaa4e1261407190a88494b59 /usr/bin/more  
 
 Yes more has been less for some time, trouble is it is programmed to
 behave differently based on the name used to call it so they aren't
 the same.  
 
 I'm aware of that.
 
 
 Like everything, if you don't like the way it works you find a setting
 to make it work the way you want. Whether you ask on a mailing list or
 google it you will find a way to configure it your way.
 
 As long as it works, the default is of little concern to most.  
 
 Whatever the choice of default pager, I think the defaults should be
 set-up so man doesn't exit at the bottom of a page. The fact that it's
 easy to change make it all the more obtuse have a default that's awkward
 for new users who want to read the documentation.

+1

It's funny, our less doesn't bother telling people to hit 'q' to quit, so it's 
poisoned on both sides unfortunately.

-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: default pager (csh)

[Alfred Perlstein]

On 2/19/15 9:04 AM, Chris H wrote:

On Thu, 19 Feb 2015 00:16:55 -0800 Julian Elischer jul...@freebsd.org wrote


On 2/18/15 3:41 PM, Xin Li wrote:

The _only_ reason that I can think of is that more(1) does not clear
screen for certain terminals (done with 'ti' and 'te' sequences),
while less(1) when running as less does.

The less(1) behavior can be annoying to some people (sometimes even
myself when using less to show contents of a file and ^Z to paste
them), and unfortunately quite a few of them also happen to be the
more vocal ones when it comes to a change.

I find that behaviour infuriating

I page down to a place to get some text on the screen to use as  a
reference, then exit to run a command with that information, and *bam*
the info I wanted to use has gone away.

Indeed. Quite annoying.
I just had an interesting thought after reading most of this
thread/bikeshed;
Theoretical, mind you. But would it be remotely possible to
[in (t)csh(1)] choose a pager on-the-fly (dynamically)?
In other words; bind keys to a chosen pager;

^M switches to more(1)
^L switched to less(1)

I've done no *actual* research [yet] but just thought
I'd bring it up. As it seemed worth a shot, as it would
satisfy *anyone's* need. :)


Not as annoying as people who don't search the web for an answer for things.

~ % export PAGER=less -X

everyone please stop crying over small changes, it really sucks.

-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: HEADS-UP: Enabling WITH_DEBUG_FILES by default

[Alfred Perlstein]

On 2/11/15 8:28 PM, Glen Barber wrote:

On Wed, Feb 11, 2015 at 09:23:01PM -0700, Ian Lepore wrote:

On Thu, 2015-02-12 at 04:11 +, Glen Barber wrote:

On Wed, Feb 11, 2015 at 08:56:00PM -0700, Ian Lepore wrote:

On Wed, 2015-02-11 at 22:21 -0500, Ed Maste wrote:

On 11 February 2015 at 21:39, Glen Barber g...@freebsd.org wrote:


Within the next 24 hours, I will merge the release-install-debug branch
into head, which will enable building and installing stripped debugging
files by default.

In general, this should have no significant impact, but any fallout will
be addressed as soon as possible after the merge.

Those that do not want debugging files built/installed by default should
add 'WITHOUT_DEBUG_FILES=1' to src.conf(5).  This will also be noted in
UPDATING.


Note that the debug files do consume a reasonably large amount of disk
space in both the OBJDIR and in the installed location under
/usr/lib/debug. Users with limited disk space will probably want to
disable them.  As an example, the installed debug data on my laptop is
about 2GB.


Seriously?  2GB is bigger than the entire filesystem on many ARM boards
that do useful work.  Not to mention how long it will take to write all
that to an sdcard (it already takes a long time to installworld/kernel
to an sdcard and it's only 400MB).

Just what are these files, and what use will the average user make of
them?

What use will I make of them, that is going to justify that every one of
my couple-dozen build sandboxes will now be 4gb larger (a copy in obj
and a copy in the nfs root that things install to)?

How much time will this add to a build?

Yeah yeah, I can update a couple dozen src.conf files to eliminate them,
and that's not the biggest hassle in the world (but it's also not
nothing).  It seems like this is a heavy enough load that it needs to
justify its existance.



The major benefit is that all debugging data that we need to properly
debug application crashes in the base system will be available
out-of-box.

There is a trade-off here, in both directions.  For arm, for example,
the trade-off is that the default installed userland would grow, however
when there is a PR regarding an application crash, the tools to diagnose
the issue are there by default (we do not need to ask that the utility
is rebuilt with debugging options enabled, and then recreate the crash).

I considered making this an opt-in thing for arm, but given the above
rationale, thought it would be more beneficial for the opposite route.
If you feel necessary, however, we can turn this off by default for now
for arm.

Glen



I can't imagine that anybody is going to be happy with an installed
system size increase from 520 to 2520 MB no matter how much it helps
debugging, especially considering the the typical installation media is
in the 2-8 GB range (with lots of 2 and 4 GB cards out there because
that's what vendors bundle with the boards).



Absolutely understood.  As mentioned in a recent reply (before seeing
this email), I'll provide a closer estimate of what to expect soon.
Again, if you want this turned off for arm, that's fine.  I would,
however, like to see it enabled by default across the board eventually
(even for -RELEASE builds).




YES

Thanks Glen, this is a great step forward for Intel platform.

Agreed, SMALL platforms should turn off at decision of those platform 
maintainers.  Won't fight to keep debug default on for arm, that's for sure.


Our Arm stuff runs Linux anyhow. :(

-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: PSA: If you run -current, beware!

[Alfred Perlstein]

On 2/5/15 11:00 AM, Peter Wemm wrote:

On Thursday, February 05, 2015 10:48:54 AM John Baldwin wrote:

On Thursday, February 05, 2015 04:22:23 PM Luigi Rizzo wrote:

On Thu, Feb 05, 2015 at 08:21:45AM -0500, John Baldwin wrote:

On Thursday, February 05, 2015 08:48:33 AM Luigi Rizzo wrote:

...


It is fixed (in the proper meaning of the word, not like worked
around,
covered by paper) by the patch at the end of the mail.

We already have a story trying to enable much less ambitious
option
-fno-strict-overflow, see r259045 and the revert in r259422.  I do
not
see other way than try one more time.  Too many places in kernel
depend on the correctly wrapping 2-complement arithmetic, among
others
are callweel and scheduler.


Rather than depending on a compiler option, wouldn't it be better/more
robust to change ticks to unsigned, which has specified wrapping
behavior?


Yes, but non-trivial.  It's also not limited to ticks.  Since the
compiler
knows when it would apply these optimizations, it would be nice if it
could
warn instead (GCC apparently has a warning, but clang does not).  Having
people do a manual audit of every signed integer expression in the tree
will take a long time.


I think I misunderstood the problem as being limited to ticks,
which is probably only one symptom of a fundamental change in behaviour
of the compiler.
Still, it might be worthwhile start looking at ints that ought to be
implemented as u_int


I actually agree, I just think we are stuck with -fwrapv in the interval,
but it's probably not a short interval.  I think converting ticks to
unsigned would be a good first start.


For the record, I agree.  However, I suspect that attempts to do so will have
a non trivial number of bugs introduced.  We have a track record of recurring
problems with tcp sequence number space arithmetic and tcp timing, partly
because the wraparounds happens infrequently.

In the mean time, I feel that telling the compiler that it's OK to let it
behave the way we expect (vs actively sabotaging it) is a viable stopgap.



Seems like it would make sense to move these functions into files that 
can be easily compiled outside of kernel and then adding unit tests.


I've done this before, to prove that larger pcb hashes help performance 
on large workloads.


-Alfred


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Weird behavior writing to SSD on 2013 MacBook

[Alfred Perlstein]

On 2/5/15 12:30 AM, Konstantin Belousov wrote:

On Thu, Feb 05, 2015 at 08:56:59AM +0100, Dimitry Andric wrote:

If you let bsdtar continue, and press control-T a few times, does the
user time (u) increase at all?  Does it ever go any further, if you let
it run for a very long time?

I believe a problem may have been introduced by r277922, leading to
filesystem hangs in some scenarios.  It looks like this commit is also
in dumbbell's github fork:

https://github.com/dumbbell/freebsd/commit/83723416a6bb8695d60c6573722a81086899f521



Would be nice if you mailed me with your findings.

Please try this.

diff --git a/sys/ufs/ffs/ffs_softdep.c b/sys/ufs/ffs/ffs_softdep.c
index 79783c8..700854e 100644
--- a/sys/ufs/ffs/ffs_softdep.c
+++ b/sys/ufs/ffs/ffs_softdep.c
@@ -1393,7 +1393,7 @@ softdep_flush(addr)
VFSTOUFS(mp)-softdep_jblocks-jb_suspended))
kthread_suspend_check();
ACQUIRE_LOCK(ump);
-   while ((ump-softdep_flags  (FLUSH_CLEANUP | FLUSH_EXIT)) == 0)
+   if ((ump-softdep_flags  (FLUSH_CLEANUP | FLUSH_EXIT)) == 0)
msleep(ump-softdep_flushtd, LOCK_PTR(ump), PVM,
sdflush, hz / 2);
ump-softdep_flags = ~FLUSH_CLEANUP;
@@ -1423,10 +1423,9 @@ worklist_speedup(mp)

ump = VFSTOUFS(mp);
LOCK_OWNED(ump);
-   if ((ump-softdep_flags  (FLUSH_CLEANUP | FLUSH_EXIT)) == 0) {
+   if ((ump-softdep_flags  (FLUSH_CLEANUP | FLUSH_EXIT)) == 0)
ump-softdep_flags |= FLUSH_CLEANUP;
-   wakeup(ump-softdep_flushtd);
-   }
+   wakeup(ump-softdep_flushtd);
  }

  static int
@@ -1471,11 +1470,10 @@ softdep_speedup(ump)
TAILQ_INSERT_TAIL(softdepmounts, sdp, sd_next);
FREE_GBLLOCK(lk);
if ((altump-softdep_flags 
-   (FLUSH_CLEANUP | FLUSH_EXIT)) == 0) {
+   (FLUSH_CLEANUP | FLUSH_EXIT)) == 0)
altump-softdep_flags |= FLUSH_CLEANUP;
-   altump-um_softdep-sd_cleanups++;
-   wakeup(altump-softdep_flushtd);
-   }
+   altump-um_softdep-sd_cleanups++;
+   wakeup(altump-softdep_flushtd);
FREE_LOCK(altump);
}
}
___


Why the conversion of while() to if()?


The reason for a while() when doing msleep/wakeup is typically to 
prevent superfluous wakeups from signalling early.


-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Weird behavior writing to SSD on 2013 MacBook

[Alfred Perlstein]

It's possible original intent of that construct was just a pause/throttle if it 
used to be an if(). Makes sense although should investigate further. 

Sent from my iPhone

 On Feb 5, 2015, at 1:03 AM, Konstantin Belousov kostik...@gmail.com wrote:
 
 On Thu, Feb 05, 2015 at 12:45:55AM -0800, Alfred Perlstein wrote:
 
 
 On 2/5/15 12:30 AM, Konstantin Belousov wrote:
 On Thu, Feb 05, 2015 at 08:56:59AM +0100, Dimitry Andric wrote:
 If you let bsdtar continue, and press control-T a few times, does the
 user time (u) increase at all?  Does it ever go any further, if you let
 it run for a very long time?
 
 I believe a problem may have been introduced by r277922, leading to
 filesystem hangs in some scenarios.  It looks like this commit is also
 in dumbbell's github fork:
 
 https://github.com/dumbbell/freebsd/commit/83723416a6bb8695d60c6573722a81086899f521
 
 Would be nice if you mailed me with your findings.
 
 Please try this.
 
 diff --git a/sys/ufs/ffs/ffs_softdep.c b/sys/ufs/ffs/ffs_softdep.c
 index 79783c8..700854e 100644
 --- a/sys/ufs/ffs/ffs_softdep.c
 +++ b/sys/ufs/ffs/ffs_softdep.c
 @@ -1393,7 +1393,7 @@ softdep_flush(addr)
  VFSTOUFS(mp)-softdep_jblocks-jb_suspended))
  kthread_suspend_check();
  ACQUIRE_LOCK(ump);
 -while ((ump-softdep_flags  (FLUSH_CLEANUP | FLUSH_EXIT)) == 0)
 +if ((ump-softdep_flags  (FLUSH_CLEANUP | FLUSH_EXIT)) == 0)
  msleep(ump-softdep_flushtd, LOCK_PTR(ump), PVM,
  sdflush, hz / 2);
  ump-softdep_flags = ~FLUSH_CLEANUP;
 @@ -1423,10 +1423,9 @@ worklist_speedup(mp)
 
  ump = VFSTOUFS(mp);
  LOCK_OWNED(ump);
 -if ((ump-softdep_flags  (FLUSH_CLEANUP | FLUSH_EXIT)) == 0) {
 +if ((ump-softdep_flags  (FLUSH_CLEANUP | FLUSH_EXIT)) == 0)
  ump-softdep_flags |= FLUSH_CLEANUP;
 -wakeup(ump-softdep_flushtd);
 -}
 +wakeup(ump-softdep_flushtd);
  }
 
  static int
 @@ -1471,11 +1470,10 @@ softdep_speedup(ump)
  TAILQ_INSERT_TAIL(softdepmounts, sdp, sd_next);
  FREE_GBLLOCK(lk);
  if ((altump-softdep_flags 
 -(FLUSH_CLEANUP | FLUSH_EXIT)) == 0) {
 +(FLUSH_CLEANUP | FLUSH_EXIT)) == 0)
  altump-softdep_flags |= FLUSH_CLEANUP;
 -altump-um_softdep-sd_cleanups++;
 -wakeup(altump-softdep_flushtd);
 -}
 +altump-um_softdep-sd_cleanups++;
 +wakeup(altump-softdep_flushtd);
  FREE_LOCK(altump);
  }
  }
 ___
 
 Why the conversion of while() to if()?
 
 
 The reason for a while() when doing msleep/wakeup is typically to 
 prevent superfluous wakeups from signalling early.
 
 if()-while() was one of the changes in r277922, and I suspect that it
 is the cause of the issue.  I.e. my thought right now is that
 softdep_process_worklist() does not keep up with the requests.
 
 If this is true, then real fix is somewhere else, but restoring
 pre-r277922 behaviour should get rid of immediate pain.
 
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: simple task to speed up booting

[Alfred Perlstein]

On Dec 14, 2014, at 6:52 AM, Ian Lepore wrote:

 On Sun, 2014-12-14 at 10:32 +, Poul-Henning Kamp wrote:
 The rotating swirlie ('-/|\') in the loader accounts for a surprisingly
 large part of our boot time on systems with slow-ish serial consoles.
 
 I think right now it takes a step for each 512 byte read, reducing that
 to once every 64kB or even 1MB would be an improvement with the kind of
 kernel sizes we have today.
 
 
 I experimented with that a while ago using the attached patch and was
 disappointed with the results.  As I vaguely remember it, a divisor of 8
 looked fine, but had no significant speedup.  With a divisor of 32 the
 difference was measureable (only like 1.5 seconds or so faster), but it
 gave the impression that something was wrong, and the overall perception
 was that it was slower rather than faster, despite what a stopwatch
 said.
 
 I was testing at 115kbps, maybe at 9600 it would be significant.  I
 don't understand why anything these days is still defaulting to 9600.
 It's the 21st century, but we never got the George Jetson flying cars we
 were promised, and apparently we're never going to break loose from the
 standards set by accoustic-coupled modems.
 
 -- Ian
 
 Index: lib/libstand/twiddle.c
 ===
 --- lib/libstand/twiddle.c(revision 274850)
 +++ lib/libstand/twiddle.c(working copy)
 @@ -46,7 +46,11 @@ void
 twiddle()
 {
   static int pos;
 + static int divisor;
 
 - putchar(|/-\\[pos++  3]);
 - putchar('\b');
 + if (divisor-- == 0) {
 + divisor = 32;
 + putchar(|/-\\[pos++  3]);
 + putchar('\b');
 + }
 }

Ian, can divisor be exposed so that it can be set based on the loader's 
output device?  

That was we can preserve it for video consoles, but other things such as serial 
at = 9600 could throttle it (or even shut it off… twiddle_divisor = 
TWIDDLE_SHUT_OFF).

-Alfred

-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

console screen saver broken? -current

[Alfred Perlstein]

Up until a few months ago the following added to /etc/rc.conf would 
blank the screen:


# /etc/rc.conf
saver=green
allscreens_flags=-t 60

Now that no longer works.

Can someone explain how to restore the power saving screen saver please?

It doesn't seem obvious from the docs or anything else.

Are the current green_saver.ko and other screen saver modules still 
able to blank the screen?


Do splash screen work?

-Alfred


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: External toolchain support

[Alfred Perlstein]

On 11/29/14, 7:04 AM, Baptiste Daroussin wrote:

Hi all,

It is now possible to use an external toolchain to build the kernel and base
(tested with gcc 4.9.1 and latest binutils)

Of course a lot of work is needed to make it build cleanly (aka lots of warning
to fix).

What have been tested so far:
- sparc64 kernel + world
- amd64 kernel + world
- powerpc64 kernel + world

mips cannot be tested because upstream gcc never heard of FreeBSD running on
mips, and I did not receive any patches for mips.

for amd64, in the kernel two things had to be removed from the build:
- aesni: (it request a header which is compiler specific and on recent gcc
will end up including stdlib.h which gives errors because kernel version of free
and malloc are not compatible with the version defined in stdlib.h)
- hptmv: I had to remove it from GENERIC and kernel building.

The result is:

$ sysctl kern.ostype kern.osrelease kern.osrevision kern.compiler_version

kern.ostype: FreeBSD
kern.osrelease: 11.0-CURRENT
kern.osrevision: 199506
kern.compiler_version: gcc version 4.9.1 (FreeBSD Ports Collection for amd64)

so yes it boots and runs

How to do you own testing:
in the ports tree/packages (the amd64 version will appear in packages next week)
install:
amd64-xtoolchain-gcc or powerpc64-xtoolchain-gcc or sparc64-xtoolchain-gcc

if your source tree:
make CROSS_TOOLCHAIN=amd64-gcc -j8 buildkernel
or
make CROSS_TOOLCHAIN=powerpc64-gcc -j8 buildkernel
or
make CROSS_TOOLCHAIN=sparc64-gcc -j8 buildkernel

To build world:
same operation with buildworld. Please note that for world you will need to add
define NO_WERROR (world will also require a change in share/mk/bsd.lib.mk:
s/--fatal-warnings/--no-fatal-warnings/)

also notes that for the kernel a lots of warnings are disabled in
share/sys/kern.mk so do not hesitate to remove yourself those -Wno-error= and
fix the issue they are hidding!

Best regards,
Bapt

This is amazing work, thank you Bapt!

-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: RFC: Remove pty(4)

[Alfred Perlstein]

On Nov 27, 2014, at 1:52 AM, Konstantin Belousov wrote:

 On Wed, Nov 26, 2014 at 04:41:27PM -0800, Davide Italiano wrote:
 On Mon, Aug 25, 2014 at 12:37 PM, John Baldwin j...@freebsd.org wrote:
 On Wednesday, August 20, 2014 11:00:14 AM Davide Italiano wrote:
 One of my personal goals for 11 is to get rid of cloning mechanism
 entirely, and pty(4) is one of the few in-kernel drivers still relying
 on such mechanism.
 Why this is good thing to do ?
 
 It's not possible, at least to my understanding, converting pty(4) to
 cdevpriv(9) as happened with other drivers. This is mainly because we
 always need a pair of devices (/dev/ptyXX and /dev/ttyXX) and
 userspace loops over ptyXX and after it successfully opens it tries to
 open the other one with the same suffix. So, having a single device is
 not really enough.
 My option, instead, is that of removing pty(4), which is nothing more
 than a compatibility driver, and move pmtx(4) code somewhere else.
 The main drawback of the removal of this is that it makes impossible
 to run FreeBSD = 7 jails and SSH into them. I personally don't
 consider this a huge issue, in light of the fact that FreeBSD-7 has
 been EOL for a long time, but I would like to hear other people
 comments.
 You don't, but other people care about ABI.
 
 Besides older jails which you do not care about, there is significant
 set of programs which were coded to use Berkley' pty directly.  Even
 high-profile applications like Emacs automatically selected pty(4)
 up to its previous version on FreeBSD.
 
 
 The code review for the proposed change can be found here:
 https://reviews.freebsd.org/D659
 
 If I won't get any objection I'll commit this in one week time, i.e.
 August 27th.
 
 Why not just statically create the pairs in /dev?  Use some loader tunable
 (kern.ptymax) to set a count on the number of pre-created device pairs to
 create and then just explicitly create them in the mod_event handler?  It
 could default to 100 or so.
 
 
 Done, thank you for the suggestion, John.
 
 root@maxwell:/home/davide # kldload pty
 root@maxwell/home/davide # sysctl -a |grep pty
 kern.tty_pty_warningcnt: 1
 kern.npty: 32
 debug.softdep.emptyjblocks: 0
 
 root@maxwell:/home/davide # ls /dev/pty*
 /dev/ptyl0 /dev/ptyl2 /dev/ptyl4 /dev/ptyl6 /dev/ptyl8 /dev/ptyla
 /dev/ptylc /dev/ptyle /dev/ptylg /dev/ptyli /dev/ptylk /dev/ptylm
 /dev/ptylo /dev/ptylq /dev/ptyls /dev/ptylu
 /dev/ptyl1 /dev/ptyl3 /dev/ptyl5 /dev/ptyl7 /dev/ptyl9 /dev/ptylb
 /dev/ptyld /dev/ptylf /dev/ptylh /dev/ptylj /dev/ptyll /dev/ptyln
 /dev/ptylp /dev/ptylr /dev/ptylt /dev/ptylv
 
 https://reviews.freebsd.org/D1238 for review.
 I hope anybody that raised concerns about the previous patch can try
 this new one.
 
 I do not see why dev_clone event makes your so unhappy.  I object against
 removal of it (and this is what you are aiming at, it seems).  It provides
 useful functionality, which is not substituted by anything cdevpriv(9)
 can provide.
 
 My only hope is that you are confusing dev_clone event and a library of
 clone_create(9)/clone_cleanup(9)/dev_stdclone(9) functions.  The former
 is needed and cannot be replaced by cdevpriv(9).
 
 The later is clumsy and never was used properly. My opinion is that it
 is impossible to use properly. There are five uses of that in tree left,
 and it seems that removing them worth cleaning of buggy by design and
 undocumented KPI.
 
 Really big and complicated target is the hand-written timeout-based (?!)
 custom cloning code in snd(4).  I believe it _can_ be converted to
 cdevpriv(9).

Thank you kib, I feel the same about leaving the pty system as it is.

-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: comments on code-in tasks for FreeBSD (Re: FreeBSD + Google Code-In 2014 = we need ideas.)

[Alfred Perlstein]

On 11/17/14, 6:55 AM, Chris H wrote:

On Thu, 13 Nov 2014 19:55:16 -0800 Rui Paulo rpa...@me.com wrote


On Nov 13, 2014, at 17:40, Luigi Rizzo ri...@iet.unipi.it wrote:

But please nuke the current list -- it is completely inadequate
for the code-in candidates and misleading for whoever wants to
suggest new tasks. Again i am not saying that the projects
suggested there are not important, just belong somewhere else
e.g. gsoc.

I refrained from voicing my opinion while the call for help was going on, but
I completely agree that the target age of this Google initiative is
inadequate to FreeBSD.  The target population is 13 years to 17 years old and
I cannot even imagine a 13 year old knowing what FreeBSD is.

I'm not sure we should participate in Code In ever again and perhaps we
should focus our efforts on Summer of Code only.

@Luigi
Very insightful. Thanks for taking the time to share your thoughts. :)

@Rui
Would tasks in re, or Mk/ be too far off base?




[[ cc'd Kris ]]

Maybe PCBSD has some ideas for code-in?  It's a bit easier on newbies 
than FreeBSD.


Mind you, I was using FreeBSD at 19 years old (would have used it 
earlier if I'd know about it), coding in assembler at 14... so there are 
people out there.  I just think for it to be something people will want 
to do it's going to be less edit these text files to improve process 
and more make dancing daemons happen on the console.


-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Finding a rogue src/sys commit with bisection?

[Alfred Perlstein]

On 11/15/14, 10:43 AM, Steve Kargl wrote:

Before I totally hose by /usr/src directory, does anyone
have some guidelines on doing a binary search for a rogue
commit in /usr/src/sys?.  Either cam or usb (or acpi?) has
broken the ability to remove a external USB device once it
is plugged into a usb port on my Dell Latitude D530 laptop.
I know that a good kernel can be built with r271273 and
a bad kernel comes from (nearly) top of tree at r274456.

I assume I need to do somthing along the lines

% cd /usr/src/sys
% svn merge -r 274456:272864(half way point between good and bad)
(build kernel and test)
% cd /usr/src/sys
% svn revert -R .
(assume 272864 builds working kernel)
% svn merge -r 274456:273660   (1/2 point between 272864 and 274456).

Rinse and repeat.


Use git, it has a built in bisector to shake this sort of thing out:

git clone --config remote.origin.fetch='+refs/notes/*:refs/notes/*' 
https://github.com/freebsd/freebsd.git


cd freebsd
git log # find the hash of the commit for r271273
HASH=the git hash you found
# then:
git bisect start
git bisect bad # Current version is bad
git bisect good $HASH

Now test compile / etc...

Then as things work or don't work you keep running:
git bisect good
-or-
git bisect bad

Then compile and test.. you should converge on the problem.

-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Finding a rogue src/sys commit with bisection?

[Alfred Perlstein]

On 11/15/14, 10:52 AM, Steve Kargl wrote:

On Sat, Nov 15, 2014 at 07:46:09PM +0100, Mateusz Guzik wrote:

On Sat, Nov 15, 2014 at 10:43:32AM -0800, Steve Kargl wrote:

Before I totally hose by /usr/src directory, does anyone
have some guidelines on doing a binary search for a rogue
commit in /usr/src/sys?.  Either cam or usb (or acpi?) has
broken the ability to remove a external USB device once it
is plugged into a usb port on my Dell Latitude D530 laptop.
I know that a good kernel can be built with r271273 and
a bad kernel comes from (nearly) top of tree at r274456.

I assume I need to do somthing along the lines

% cd /usr/src/sys
% svn merge -r 274456:272864(half way point between good and bad)
(build kernel and test)
% cd /usr/src/sys
% svn revert -R .
(assume 272864 builds working kernel)
% svn merge -r 274456:273660   (1/2 point between 272864 and 274456).


http://search.cpan.org/dist/App-SVN-Bisect/bin/svn-bisect


There's no instructions on how to install it.


In absolutely worst case yu can check out git tree and bisect that :-


Nope.  Learning how to use git and github is well beyond any interest
I have.


I just sent you a workflow that will take you about 10 minutes to sort 
out for git.


If you want reply in private and I can send you my phone number to show 
how if you get stuck.


-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Finding a rogue src/sys commit with bisection?

[Alfred Perlstein]

On 11/15/14, 11:01 AM, Steve Kargl wrote:

On Sat, Nov 15, 2014 at 10:56:50AM -0800, Alfred Perlstein wrote:

On 11/15/14, 10:43 AM, Steve Kargl wrote:

Before I totally hose by /usr/src directory, does anyone
have some guidelines on doing a binary search for a rogue
commit in /usr/src/sys?.  Either cam or usb (or acpi?) has
broken the ability to remove a external USB device once it
is plugged into a usb port on my Dell Latitude D530 laptop.
I know that a good kernel can be built with r271273 and
a bad kernel comes from (nearly) top of tree at r274456.

I assume I need to do somthing along the lines

% cd /usr/src/sys
% svn merge -r 274456:272864(half way point between good and bad)
(build kernel and test)
% cd /usr/src/sys
% svn revert -R .
(assume 272864 builds working kernel)
% svn merge -r 274456:273660   (1/2 point between 272864 and 274456).

Rinse and repeat.


Use git, it has a built in bisector to shake this sort of thing out:


I won't be drawn into the git debate.

OK, so we don't want to use a tool purposefully built for the problem 
you are facing?  Doesn't seem like a git debate more like hammering in 
screws...


-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Finding a rogue src/sys commit with bisection?

[Alfred Perlstein]

On 11/15/14, 11:32 AM, Ian Lepore wrote:

On Sat, 2014-11-15 at 11:03 -0800, Alfred Perlstein wrote:

On 11/15/14, 11:01 AM, Steve Kargl wrote:

On Sat, Nov 15, 2014 at 10:56:50AM -0800, Alfred Perlstein wrote:

On 11/15/14, 10:43 AM, Steve Kargl wrote:

Before I totally hose by /usr/src directory, does anyone
have some guidelines on doing a binary search for a rogue
commit in /usr/src/sys?.  Either cam or usb (or acpi?) has
broken the ability to remove a external USB device once it
is plugged into a usb port on my Dell Latitude D530 laptop.
I know that a good kernel can be built with r271273 and
a bad kernel comes from (nearly) top of tree at r274456.

I assume I need to do somthing along the lines

% cd /usr/src/sys
% svn merge -r 274456:272864(half way point between good and bad)
(build kernel and test)
% cd /usr/src/sys
% svn revert -R .
(assume 272864 builds working kernel)
% svn merge -r 274456:273660   (1/2 point between 272864 and 274456).

Rinse and repeat.


Use git, it has a built in bisector to shake this sort of thing out:


I won't be drawn into the git debate.


OK, so we don't want to use a tool purposefully built for the problem
you are facing?  Doesn't seem like a git debate more like hammering in
screws...


This in-your-face git evangelism is getting REALLY OLD, REALLY FAST.
Please stop it.

I have nothing in particular against git, I just have no interest in it.
But that's rapidly transforming into active dislike in exact proportion
to being repeatedly talked down to by someone with a different opinion,
and apparently the belief that folks with other opinions just need more
repetitious condescention to see the light.


This is really over the top.

It's not evangelism, the guy asked how do I do X?  I showed him how to 
do it in a few simple steps.


If you take git out of the equation it would have been a simple thank 
you.


I could have just as simply replied with the steps needed, however 
s/git/frob/g and it wouldn't have triggered any reaction from people.



-Alfred


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Finding a rogue src/sys commit with bisection?

[Alfred Perlstein]

On 11/15/14, 11:42 AM, Steve Kargl wrote:

On Sat, Nov 15, 2014 at 11:39:33AM -0800, Alfred Perlstein wrote:

This is really over the top.

It's not evangelism, the guy asked how do I do X?  I showed him how to
do it in a few simple steps.

The guy, that would be me, asked how to do it with svn.



Nope.  You showed some svn commands to not do it, you weren't explicit 
in asking for ways to do it in svn, go ahead, look:



~ % grep -i svn  foo.out
Before I totally hose by /usr/src directory, does anyone
have some guidelines on doing a binary search for a rogue
commit in /usr/src/sys?.  Either cam or usb (or acpi?) has
broken the ability to remove a external USB device once it
is plugged into a usb port on my Dell Latitude D530 laptop.
I know that a good kernel can be built with r271273 and
a bad kernel comes from (nearly) top of tree at r274456.

I assume I need to do somthing along the lines

% cd /usr/src/sys
% svn merge -r 274456:272864(half way point between good and bad)
(build kernel and test)
% cd /usr/src/sys
% svn revert -R .
(assume 272864 builds working kernel)
% svn merge -r 274456:273660   (1/2 point between 272864 and 274456).

Rinse and repeat.

.(11:46:36)(alfred@AlfredMacbookAir.local)
~ % cat foo.out
% svn merge -r 274456:272864(half way point between good and bad)
% svn revert -R .
% svn merge -r 274456:273660   (1/2 point between 272864 and 274456).
.(11:46:38)(alfred@AlfredMacbookAir.local)
~ %


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Finding a rogue src/sys commit with bisection?

[Alfred Perlstein]

On 11/15/14, 11:51 AM, Poul-Henning Kamp wrote:


In message 5467adf7.1020...@mu.org, Alfred Perlstein writes:


Nope.  You showed some svn commands to not do it, you weren't explicit
in asking for ways to do it in svn, go ahead, look:

I didn't realize that the git-zealots also wanted us to adopt the
petty and childish behaviour of the linux email-culture ?

Can everybody please just shut up and code now ?

Thankyou!


I resent your implications.  Seriously I do.

There was no intent to be childish or anything as such.

He asked: how do i bisect code.

I gave a tool that can do it in a few easy steps?

I didn't realize that git has become a trigger word for some people in 
the project.  I will proceed to warn people going forward.


-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Finding a rogue src/sys commit with bisection?

[Alfred Perlstein]

On 11/15/14, 11:53 AM, Steve Kargl wrote:

On Sat, Nov 15, 2014 at 11:48:07AM -0800, Alfred Perlstein wrote:

On 11/15/14, 11:42 AM, Steve Kargl wrote:

On Sat, Nov 15, 2014 at 11:39:33AM -0800, Alfred Perlstein wrote:

This is really over the top.

It's not evangelism, the guy asked how do I do X?  I showed him how to
do it in a few simple steps.

The guy, that would be me, asked how to do it with svn.


Nope.  You showed some svn commands to not do it, you weren't explicit
in asking for ways to do it in svn, go ahead, look:

I won't insult your intelligence.  Oh what the hell, it is
clear from the list of commands I meant how to do it with svn.
Go ahead twist it however suits your needs.

Ian's suggestion on using 'svn update -r#' was exactly what
I was looking for.

PS: 'svn up -r271000' has already shown to provide a good kernel.
PPS: I currently building a -r272728 kernel.



There was no intention to twist anything.  Just intended to lend a hand.

Steve, if there are other technologies you never want discussed, please 
just let me know I'll not reply with help if it involves those tools.


My goal is to help, not to traumatize.

-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Finding a rogue src/sys commit with bisection?

[Alfred Perlstein]

On 11/15/14, 11:56 AM, Poul-Henning Kamp wrote:


In message 5467af7a.2080...@mu.org, Alfred Perlstein writes:


I resent your implications.  Seriously I do.

There was no intent to be childish or anything as such.

Well, you were, and intentional or not, you're wasting
a hell of a lot of peoples time right now.

See also: www.bikeshed.org

I do not believe it wasteful of people's time to give them to tools to 
do the job they asked for.


Poul, be careful, there comes a time that when everything minor seems 
childish, and you start slinging it as an insult left and right, that it 
is quite possible that you've gone too far in the opposite of the 
spectrum, that being a cantankerous old fart.  Take inventory.


-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Jenkins, Kyua, and Bhyve used for FreeBSD OS testing

[Alfred Perlstein]

Craig this is really great.  Thanks for doing this and thanks for the Jenkins 
guys on giving a shout out!

On Oct 20, 2014, at 9:42 AM, Craig Rodrigues wrote:

 Hi,
 
 FYI, Kohsuke Kawaguchi, the lead developer of Jenkins,
 accepted my posting on the Jenkins blog, which describes
 how the FreeBSD project is using Jenkins, Kyua, and Bhyve
 for FreeBSD OS testing:
 
 http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing
 
 --
 Craig
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
 

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [PATCH] Lock scd(4): test or the driver will be removed

[Alfred Perlstein]

On 10/8/14 11:53 AM, John Baldwin wrote:

This patch adds locking to scd(4) and marks it MPSAFE.  It also uses bus_*()
instead of bus_space_*().  The patch is against HEAD but probably applies to 9
and 10 as well.

http://people.freebsd.org/~jhb/patches/scd_locking.patch

Note that this driver is using a deprecated API that will be removed in 11.
If no one tests updates to this driver then it is not feasible to continue
maintaining it in the tree.  In that case, it will be removed from HEAD one
month from today.


Hey John, which APIs are being used?
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [PATCH] Various fixes to wl(4)

[Alfred Perlstein]

On 9/22/14 3:05 PM, Adrian Chadd wrote:

On 22 September 2014 13:46, Warner Losh i...@bsdimp.com wrote:

I have a dozen I can bring to the next bafug I go to :)

But, do you have the pentium class machines with ISA slots that we'd
need to use em? :P


I have the pci card to pccard bridge cards.

I'll bring those in, they'll be on your desk tomorrow AM.

-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: shells/bash port, add a knob which symlinks to /bin/bash ?

[Alfred Perlstein]

On 9/13/14, 11:39 AM, Nathan Whitehorn wrote:

On 09/13/14 11:32, Craig Rodrigues wrote:


If adding an optional knob to the bash port which is OFF by default 
to do

this is a no-go,
would having an optional port like what Brooks Davis mentioned be 
allowed

which creates
the symlink and updates /etc/shells?

--


I'd point out that the perl ports have exactly such an option already 
(putting links in /usr/bin, in this case). The CUPS port does too.

-Nathan


Should really be a standalone package.

-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: shells/bash port, add a knob which symlinks to /bin/bash ?

[Alfred Perlstein]

On 9/14/14, 3:22 PM, Craig Rodrigues wrote:



On Sun, Sep 14, 2014 at 10:44 AM, Alfred Perlstein bri...@mu.org 
mailto:bri...@mu.org wrote:



On 9/13/14, 11:39 AM, Nathan Whitehorn wrote:

Should really be a standalone package.


It's not exactly the same, but the lang/python2 port for example is a 
meta-port which creates

symlinks such as  /usr/local/bin/python2 - python2.7.

So the precedent of having a metaport which creates symlinks is there.
What folks  have been complaining about in this thread is having symlinks
in the base system directories such as /bin.

Why do you care what people are complaining about?  Just make the port. :)



My personal feeling is that we should look at this on a case-by-case 
basis and allow these types

of ports, such as for /bin/bash but I'm sure others will disagree.


We already allow such ports, for example a bunch of kmods install into 
/boot/kernel or /boot/modules.


There are always exceptions to rules and for good reasons!

find . -name pkg-plist | xargs grep -A 3 '^@cwd /$'









--
Craig


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: shells/bash port, add a knob which symlinks to /bin/bash ?

[Alfred Perlstein]

The correct thing is to make a port/pkg that installs the symlink and 
/etc/shells this for the user.


There is no need for changes to 'base' nor do we need a change to the 
system port.


-Alfred

On 9/12/14 2:40 PM, Baptiste Daroussin wrote:

On Fri, Sep 12, 2014 at 02:12:45PM -0700, Craig Rodrigues wrote:

Hi,

In the last 3 jobs that I have worked at, there have been
a mix of Linux machines and FreeBSD machines.
When using an NIS or LDAP environment where
there is a single login across multiple machines, it is useful to
have a single shell setting.

Since Linux and MacOS X have /bin/bash as the shell,
in order to get the FreeBSD boxes to play in this environment,
I have seen admins do the following on FreeBSD setups:
ln -s /usr/local/bin/bash /bin/bash

or

ln /usr/local/bin/bash /bin/bash

and then make sure that /etc/shells as:
/usr/local/bin/bash
/bin/bash

Can we add an optional knob (turned off by default) which creates this
symlink
and updates /etc/shells?

This would help with interoperability of FreeBSD hosts in environments mixed
with Linux and MacOS X.


Please no, no and no!

We are fighting for a very long time to prevent the ports to pollute base.

We have added the shebangfix USES to be able to catch with up with cleanup this
properly as well as a qa test to discover it automatically.

no interpreters at all have a symlink in base but perl and this one is going to
be removed.

If you want interoperability just use /usr/bin/env bash as a shebang. Btw you
cannot get interoprability with OS-X in there because the bash they do provide
is the last GPL-2 recent bash have many incompatiblities with this old version.

regards,
Bapt


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: shells/bash port, add a knob which symlinks to /bin/bash ?

[Alfred Perlstein]

On 9/12/14 3:23 PM, Craig Rodrigues wrote:

Hi,

I could live with this solution of additional port outside of the main
bash port, which creates the symlink and updates /etc/shells.

One other thing I am seeing is that many, many shell scripts are 
written assuming #!/bin/bash.

Forcing all upstream script writers to switch to #!/usr/bin/env bash, or
to convert their scripts to #!/bin/sh and remove all bash-specific 
behaviors, is getting harder and harder,

since many people are exposed to MacOS X and Linux on desktops.



Lol, or we could hack the image activator.  :)

-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: RFC: Remove pty(4)

[Alfred Perlstein]

On 8/20/14 11:00 AM, Davide Italiano wrote:

One of my personal goals for 11 is to get rid of cloning mechanism
entirely, and pty(4) is one of the few in-kernel drivers still relying
on such mechanism.
It's not possible, at least to my understanding, converting pty(4) to
cdevpriv(9) as happened with other drivers. This is mainly because we
always need a pair of devices (/dev/ptyXX and /dev/ttyXX) and
userspace loops over ptyXX and after it successfully opens it tries to
open the other one with the same suffix. So, having a single device is
not really enough.
My option, instead, is that of removing pty(4), which is nothing more
than a compatibility driver, and move pmtx(4) code somewhere else.
The main drawback of the removal of this is that it makes impossible
to run FreeBSD = 7 jails and SSH into them. I personally don't
consider this a huge issue, in light of the fact that FreeBSD-7 has
been EOL for a long time, but I would like to hear other people
comments.

The code review for the proposed change can be found here:
https://reviews.freebsd.org/D659

If I won't get any objection I'll commit this in one week time, i.e.
August 27th.

I don't think that we want to break userland apps pre-7.x.  Do you mean 
just jails are broken?  Or is all pre-7.x compat?  I believe either is 
dicey.  What is the reason for getting rid of cloning? What is the 
difficulty in maintaining the old interface?


-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: HOWTO articles for migrating from Linux to FreeBSD, especially for pkg?

[Alfred Perlstein]

On 7/18/14, 6:28 AM, Allan Jude wrote:

On 2014-07-17 16:12, Adrian Chadd wrote:

On 17 July 2014 13:03, Alberto Mijares amijar...@gmail.com wrote:

On Thu, Jul 17, 2014 at 2:58 PM, Adrian Chadd adr...@freebsd.org wrote:

Hi!

3) The binary packages need to work out of the box
4) .. which means, when you do things like pkg install apache, it
can't just be installed and not be enabled, because that's a bit of a
problem;


No. Please NEVER do that! The user must be able to edit the files and
start the service by himself.

Cool, so what's the single line command needed to type in to start a
given package service?



-a
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


We could make 'service apache22 enable'

which can run: sysrc -f /etc/rc.conf apache22_enable=YES

and 'service apache22 disable'

that can use sysrc -x

And then ports can individually extend the functionality if they require.


I like this a lot.

That said, if other distros are setting up apache in 2 steps and we 
require 3 then we require 50% MORE STEPs!


Or they require 33% LESS steps than us.

Just to put it into perspective.  Should FreeBSD be 50% more difficult 
or time consuming to configure?


-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: tmpfs panic

[Alfred Perlstein]

On 7/6/14 11:12 AM, Konstantin Belousov wrote:

On Sun, Jul 06, 2014 at 05:25:12PM +, Steve Wills wrote:

On Sun, Jul 06, 2014 at 12:28:07PM -0400, Ryan Stone wrote:

On Sun, Jul 6, 2014 at 11:46 AM, Steve Wills swi...@freebsd.org wrote:

I should have noted this system is running in bhyve. Also I'm told this panic
may be related to the fact that the system is running in bhyve.

Looking at it a little more closely:

(kgdb) list *__mtx_lock_sleep+0xb1
0x809638d1 is in __mtx_lock_sleep (/usr/src/sys/kern/kern_mutex.c:431).
426  * owner stops running or the state of the lock changes.
427  */
428 v = m-mtx_lock;
429 if (v != MTX_UNOWNED) {
430 owner = (struct thread *)(v  ~MTX_FLAGMASK);
431 if (TD_IS_RUNNING(owner)) {
432 if (LOCK_LOG_TEST(m-lock_object, 0))
433 CTR3(KTR_LOCK,
434 %s: spinning on %p held by 
%p,
435 __func__, m, owner);
(kgdb)

I'm told that MTX_CONTESTED was set on the unlocked mtx and that MTX_CONTENDED
is spuriously left behind, and to ask how lock prefix is handled in bhyve. Any
of that make sense to anyone?

The mutex has both MTX_CONTESTED and MTX_UNOWNED set on it?  That is a
special sentinel value that is set on a mutex when it is destroyed
(see MTX_DESTROYED in sys/mutex.h).  If that is the case it looks like
you've stumbled upon some kind of use-after-free in tmpfs.  I doubt
that bhyve is responsible (other than perhaps changing the timing
around making the panic more likely to happen).

Given the first thing seen was:

Freed UMA keg (TMPFS node) was not empty (16 items).  Lost 1 pages of memory.

this sounds reasonable to me.

What can I do to help find and elliminate the source of the error?

The most worrying fact there is that the mutex which is creating trouble
cannot be anything other but allnode_lock, from the backtrace.  For this
mutex to be destroyed, the unmount of the corresponding mount point must
run to completion.

In particular, it must get past the vflush(9) call in tmpfs_unmount().
This call reclaims all vnodes belonging to the unmounted mount point.
New vnodes cannot be instantiated meantime, since insmntque(9) is
blocked by the MNTK_UNMOUNT flag.

That said, the backtrace indicates that we have live vnode, which is
reclaimed, and also we have the mutex which is in the destroyed (?)
state.  My basic claim is that the two events cannot co-exist, at least,
this code path was heavily exercised and most issues were fixed during
several years.

I cannot exclude the possibility of tmpfs/VFS screwing things up,
but given the above reasoning, and the fact that this is the first
appearance of the MTX_DESTROED problem for the tmpfs unmounting code,
which was not changed for long time, I would at least ask some things
about bhyve.  I.e., I would rather first look at the locked prefix
emulation then at the tmpfs.
What about running the code with INVARIANTS + DEBUG_VFS_LOCKS and see if 
anything shakes out?


-Alfred

--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: ipmi patch for review

[Alfred Perlstein]

On 5/30/14, 10:44 AM, Doug Ambrisko wrote:

On Thu, Sep 19, 2013 at 03:04:46PM -0400, John Baldwin wrote:
| On Tuesday, September 17, 2013 6:21:10 am Gleb Smirnoff wrote:
|Hi!
| 
|When system is writing a kernel core dump, it issues watchdog
|  pat wdog_kern_pat(WD_LASTVAL). If ipmi is in action, it registers
|  ipmi_wd_event() as event for watchdog. Thus ipmi_wd_event() is
|  called in dumping context.
| 
|  The problem is that ipmi_wd_event() calls into ipmi_set_watchdog(),
|  that calls into ipmi_alloc_request(), which uses M_WAITOK and
|  thus sleeps. This is a smaller problem, since can be converted to
|  M_NOWAIT. But ipmi_set_watchdog() then calls into
|  ipmi_submit_driver_request(), which calls msleep() any time.
| 
|The attached patch allows me to successfully write cores in
|  presence of IPMI.
|
| Of course, the watchdog might go off during your dump. :)
|
| The real fix is more complicated, which is that we should not use
| a worker thread for at least SMIC and KCS.

I haven't looked at this patch but I have local code that switches
KCS into polling direct mode when the kernel goes into panic mode.
I use this to write Linux compatible back traces into the system
event logs.  This could allow the watchdogd to continue to work.
This should be easily extended to SMIC mode.  SMBUS would be
harder but at a prior company I made the SMBIO driver work in polled
mode.

If someone wants to look at this I can post the changes for KCS and
the kernel backtrace to the system event log.  We find this useful
when looking at customer machines.

IPMI gets upset if things get intermixed/interrupted so there needs
to be serialization and cancellation if being interrupted.

These patches would be really nice to have in base.  I noticed this 
problem too, you can't really touch watchdogs some of the time when in a 
panic(9) situation and it leaves you in a bad state to stop them or 
reset them while you are dumping.


Thank you for looking at this.

-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Make variables to force non default libraries and includes?

[Alfred Perlstein]

On 4/28/14 12:48 AM, Julian Elischer wrote:
I need to do the equivalent of  cd /usr/src/cddl/usr.sbin/dtrace; 
make DESTDIR=/mumble all install


but it pulls in libraries from the base system, which differ slightly 
from those in the source tree.


How can I force it to use /mumble2/include and /mumble2/lib instead of 
/ ?


I can pre-populate /mumble2 using make buildworld, make libraries, 
and make includes but
I need to be able to do selective builds of just subdirectories after 
that..  I haven't spotted the right way of forcing the use of the 
--system_root /mumble2 option in the compiles.


I know we do it in 'buildworld' is there a more generic way?

I have been looking in the .mk files but I haven't spotted it so far.

There may be a way to use bsd.*.mk to do this, however we just use 
chroots + nullfs mounts.


Basically we buildworld into a directory and then nullfs mount our other 
sources under it, then we chroot to that build.


I recommend doing this (or even using vms) as it's way too easy to 
introduce contamination from the host build environment otherwise.


-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: OpenSSL vs. LibreSSL (OpenBSD)

[Alfred Perlstein]

On 4/24/14, 1:35 PM, O. Hartmann wrote:

It seems that OpenBSD is now forking their own SSL implementation, called 
LibreSSL. As
OpenBSD speaks for many similar opinion regarding the state of the code of 
OpenSSL, I'd
like to hear what the plans are in FreeBSD for this critical portion of 
software.

Is FreeBSD going to support the effords taken by OpenBSD and participating in 
the
LibreSSL development (http://www.libressl.org/)?

oh
We need to discuss the use of comic sans font across our web properties 
first.


-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: warning: filename ends with '.', which is not allowed on Windows: 'tools/test/sort/bigtest/q-1.024.003.'

[Alfred Perlstein]

On 3/7/14 8:41 AM, Rui Paulo wrote:

On 6 Mar 2014, at 23:30, David Xu davi...@freebsd.org wrote:

it seems filename ended with a dot is illegal on Windows, if someone
wants to check out freebsd source code on Windows, it will be a problem.

Is this something we want to support?  NetBSD made some invasive changes on 
their source tree to be able to support case-insensitive filesystems (like 
renaming 'cvs' to 'xcvs' to avoid clashing with the 'CVS' metadata directory), 
but they support building NetBSD on many different platforms.  We don't support 
that yet, though.

The file in question can be easily renamed, I think.


We definitely want to support untaring/zipping the code on Windows. Even 
if not for cross compiling, just for convenience of use.


--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [rfc] bind per-cpu timeout threads to each CPU

[Alfred Perlstein]

On 2/19/14, 12:04 PM, Adrian Chadd wrote:

On 19 February 2014 11:59, Alexander Motin m...@freebsd.org wrote:


So if we're moving towards supporting (among others) a pcbgroup / RSS
hash style work load distribution across CPUs to minimise
per-connection lock contention, we really don't want the scheduler to
decide it can schedule things on other CPUs under enough pressure.
That'll just make things worse.

True, though it is also not obvious that putting second thread on CPU run
queue is better then executing it right now on another core.

Well, it depends if you're trying to optimise for run all runnable
tasks as quickly as possible or run all runnable tasks in contexts
that minimise lock contention.

The former sounds great as long as there's no real lock contention
going on. But as you add more chances for contention (something like
100,000 concurrent TCP flows) then you may end up having your TCP
timer firing stuff interfere with more TXing or RXing on the same
connection.

Chasing this stuff down is a pain, because it only really shows up
when you're doing lots of concurrency.

I'm happy to make this a boot-time option and leave it off for the
time being. How's that?


options THROUGHPUT

Yes, looks like a latency vs throughput issue.  One giant switch might 
be a starting point so that it doesn't become death of 1000 switches to 
get throughput or latency sensitive work done.






-a
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org



___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: kasserts behind invariants

[Alfred Perlstein]

On 12/13/13 1:50 PM, Sean Bruno wrote:

I guess this may have been argued before, but I don't see why we would
want to hide specific things like:  sys/kern/subr_lock.c

/* Check for double-init and zero object. */
KASSERT(!lock_initalized(lock), (lock \%s\ %p already initialized,
 name, lock));

If I hadn't completely missed the fact that I had INVARIANTS activated,
I'd never have found out why this vendor driver was being so completely
stupid and crashing my machine.

If I find things like this that I want old KASSERT behavior on (panic if
true) and I don't want to run INVARIANTS, is that possible?


I don't understand the question, do you want to move it from INVARIANTS 
to under just a plain if(condition)?


-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Automated submission of kernel panic reports: sysutils/panicmail

[Alfred Perlstein]

On 11/4/13, 2:41 AM, Colin Percival wrote:

Hi all,

After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have
now added sysutils/panicmail to the FreeBSD ports tree.  If you install this
and add
panicmail_enable=YES
to your /etc/rc.conf, a panic report will be generated and sent to root@ for
you to review and submit (via email).  You can skip the reviewing step and
submit panics automatically by setting panicmail_autosubmit=YES.

The panics submitted are encrypted to an RSA key which I hold in order to keep
them secure in transit; and I intend to keep the raw panic reports confidential
except to the minimum extent necessary for other developers to help me process
the incoming reports.

If I receive enough panic reports to be useful, I hope to provide developers
with aggregate statistics.  This may include:

* regular email reports listing the top panics, to help guide developers
towards the most fertile areas for stability improvements;

* email to specific developers alerting them to recurring panics in code they
maintain (especially if it becomes clear that the panic has been recently
introduced); and

* guidance to re@ and secteam@ about how often a particular panic occurs if
an errata notice is being considered

as well as other yet-to-be-imagined reports of a similarly aggregate and
anonymized nature.

So please install the sysutils/panicmail port and enable it in rc.conf!  This
all depends on getting useful data, and I can't do that without your help.

Colin, have you had a few minutes to check out the crash reporting 
facilities in FreeNAS?


The reason I ask is that:

1) we would like to share code.
2) we have this running for a few months now and have a huge corpus of 
information.
3) we are building a nice UI (screenshots attached) over it, we have a 
couple of thousands of lines of code we can share for this.


Our scripts can be found here:

1) A startup script that sends us the crashes on system start:
https://github.com/freenas/freenas/blob/master/nanobsd/Files/etc/rc.d/ix_textdump 

2) A script to submit data at boot OR from command line that sends more 
comprehensive system information ixdiagnose:
https://github.com/freenas/freenas/blob/master/nanobsd/Files/usr/local/bin/ixdiagnose 


3) A very simple script to upload that report:
https://github.com/freenas/freenas/blob/master/nanobsd/Files/usr/local/bin/crashuploader 



We send a minimal set of information: kernel stack trace, ddb buffer and 
hardware.  Just enough to get some very, very handy stuff.


I can share with you offline the crash server code, it's django and 
relatively straight forward.


The screenshots can also be seen at:
http://people.freebsd.org/~alfred/crashreporter/

We could modify our framework for FreeBSD to do so by checking for a 
sentinel file depending on the host type and only auto-sending if we see 
that.



-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [rfc] removing the NDISulator

[Alfred Perlstein]

On 10/23/13 7:23 AM, John Baldwin wrote:

On Monday, October 21, 2013 6:29:24 pm Adrian Chadd wrote:

The NDISulator is a crutch from a time when there wasn't _any_ real
alternative.

There are plenty of alternatives now. What's lacking is desire and
person-power. But the datasheets are there, or the vendor code has been
released, or there's linux/otherbsd drivers.

Leaving it in there is just delaying the inevitable - drivers need to be
fixed, ported, or reverse engineered.

This is going to upset users in the same way that eliminating any other
transition/sideways compatibility layer upsets users. But as I said, the
path forward is fixing up the lack of stable drivers, not simply supporting
some crutch.

If there are drivers that people absolutely need fixed then they should
stand up and say hey, I really would like X to work better! and then
follow it up with some encouraging incentives. Right now the NDISulator
lets people work _around_ this by having something that kind of works for
them but it doesn't improve our general driver / stack ecosystems.

Eh, having taken a stab at porting the bwl blob already, I would strongly
oppose removing NDIS.  If you do that I will just stop using my netbook
with a Broadcom part altogether as I wouldn't be able to use it to try to
test bwl changes.  The NDIS thing is a bit hackish, but it is quite useful
for a lot of folks.


I have to agree.  Deprecation != motivation.

--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [rfc] removing the NDISulator

[Alfred Perlstein]

On 10/23/13 11:11 AM, Adrian Chadd wrote:

On 23 October 2013 11:09, Alfred Perlstein bri...@mu.org wrote:



Eh, having taken a stab at porting the bwl blob already, I would strongly

oppose removing NDIS.  If you do that I will just stop using my netbook
with a Broadcom part altogether as I wouldn't be able to use it to try to
test bwl changes.  The NDIS thing is a bit hackish, but it is quite useful
for a lot of folks.

  I have to agree.  Deprecation != motivation.


I can pull out examples of this not holding true:

* all the giant locking in drivers
* all the giant locking in VFS

People did pop up and claim ownership of things they cared about. Some
stuff died, some stuff didn't. There was enough of a motivation by us to
kill giant off in these pathways so things could continue to evolve. We
didn't leave the GIANT crutch in forever.


Sure, however those drivers and vfs systems were not sustainable and 
holding the kernel back.


What part of the NDISulator actually holds the system back?  I'm saying 
that it seems as if it was conjecture rather than a need.  Is the 
NDISulator giant locked?


Also why the interest in writing drivers so much?  Being able to 
leverage other platform drivers is pretty neat and saves us a ton of work.


--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [rfc] removing the NDISulator

[Alfred Perlstein]

On 10/18/13 2:04 PM, Nathan Whitehorn wrote:

On 10/18/13 16:01, Steve Kargl wrote:

On Fri, Oct 18, 2013 at 08:53:54PM +, Steve Wills wrote:

I would love to have a native driver for this:

none2@pci0:2:0:0:
 class=0x028000 card=0x00101028 chip=0x472714e4 rev=0x01 hdr=0x00
 vendor = 'Broadcom Corporation'
 device = 'BCM4313 802.11b/g/n Wireless LAN Controller'
 class  = network

Are there docs or other drivers available that we could look at?


Please, don't top post as it loses context.

http://www.broadcom.com/support/802.11/linux_sta.php



Have you looked at bwn(4)? It might just need an additional PCI ID.
-Nathan


I'm having no love with if_bwn.  Any tips on making it work better?

I have -current as of ~2 weeks ago.

--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: urtwn driver for Edimax EW-7811U WLAN nano USB Adapter

[Alfred Perlstein]

On 10/15/13 11:13 PM, Rui Paulo wrote:

On 8 Oct 2013, at 10:41, Julian H. Stacey j...@berklix.com wrote:


I too am seeing
urtwn0: timeout waiting for checksum report

Sorry, this is a know problem that I haven't been able to figure out... It 
probably exists in the OpenBSD driver as well. Usually retrying works.

--
Rui Paulo



I have a device timeout problem with urtwn as well, but I haven't had 
time to hack the driver to self-reset itself.  boo :(


--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: rcs

[Alfred Perlstein]

On 10/9/13 1:59 AM, Daniel Nebdal wrote:

On Tue, Oct 8, 2013 at 11:34 PM, Lev Serebryakov l...@freebsd.org wrote:

Hello, Daniel.
You wrote 8 октября 2013 г., 19:40:23:

DN If they get the package repositories back up - which I assume will
DN happen before any official releases from 10 - it should just be pkg
DN install rcs. As challenges go, that doesn't seem too bad?
   Topic starter mentioned, that assumption that everybody has online
  connection is completely wrong! As far as I understand, it was his main
  objection -- they have a lot of offline computers at work (something
  related to Security Theatre by DHS, but who am I to judge them?).

   And you again says about pkg install rcs which needs internet
  connection!

--
// Black Lion AKA Lev Serebryakov l...@freebsd.org



I was answering Alfred Perlsteins claim that FreeBSD is a chore to
install packages into. I agree that it won't work without an internet
connection (or a local package repository), but that's hardly a
problem limited to FreeBSD.

Anyway, it seems likely that we'll get OpenRCS in base, hopefully
making everyone happy.



Only a few years ago you could take a dvd or memstick of FreeBSD and 
have 1000s of packages to choose from during your install.  That is 
broken now?


Bummer.

--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: rcs

[Alfred Perlstein]

On 10/9/13 7:20 AM, Glen Barber wrote:

On Wed, Oct 09, 2013 at 07:17:11AM -0700, Alfred Perlstein wrote:

Only a few years ago you could take a dvd or memstick of FreeBSD and
have 1000s of packages to choose from during your install.  That is
broken now?


memstick.img always had a minimal set of packages (mostly documentation
sets).

dvd1.iso has packages.

Glen


Thanks Glen!

I'm just confused as to why this is such a big deal if the packages are 
available.


Do we need a package memstick now that memsticks are typically 4 or 
even 8GB?


--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: rcs

[Alfred Perlstein]

On 10/7/13 9:49 PM, Julian Elischer wrote:

On 10/8/13 11:19 AM, Lyndon Nerenberg wrote:
On 2013-10-07, at 8:15 PM, Steve Kargl 
s...@troutmask.apl.washington.edu wrote:



Maybe there was no development for 15 years.  However, the 7364
lines in ChangeLog after 2010-02-04 suggests that there may
be few bugs to worry about.


I think the fact is that most direct users of RCS use it in a very 
simple way, and
it works just fine for that.  with no real need for any updates or any 
change.


With all due respect Julian, The more we discuss this more this really 
points to the problem that FreeBSD appears to be a challenge to install 
packages into such that a package moving out of base is such a big deal.


Can we fix that instead?

I mean, this change should really not be a big deal, but yet it is and 
this speaks to the core of FreeBSD utility.


So again, is there a way to make it so adding RCS after install is not a 
big deal at all?



--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: rcs

[Alfred Perlstein]

On 10/8/13 8:04 AM, sth...@nethelp.no wrote:

I think the fact is that most direct users of RCS use it in a very
simple way, and
it works just fine for that.  with no real need for any updates or any
change.

With all due respect Julian, The more we discuss this more this really
points to the problem that FreeBSD appears to be a challenge to install
packages into such that a package moving out of base is such a big deal.

Can we fix that instead?

I mean, this change should really not be a big deal, but yet it is and
this speaks to the core of FreeBSD utility.

Not commenting on RCS here, but on the concept of moving packages out
of the base:

- For some of us, the attraction of FreeBSD is that it is a tightly
integrated system, and the base contains enough useful functionality
that we don't *have* to add a lot of packages.

- Each package that is moved out of the base system means less useful
functionality in the base system - and for me: Less reason to use
FreeBSD instead of Linux.

I absolutely see the problem of maintaining out-of-date packages in
the base system, and the desirability of making the base system less
reliant on GPL. I'm mostly troubled by the fact that there seems to
be a rather strong tendency the last few years of having steadily
less functionality in the base system - and I'm not at all convinced
that the right balance has been found here.

This discussion is not new, and I don't expect to convince any new
persons...


I'm sure other devs will disagree, but with ~15 years of FreeBSD 
experience and ~13 years as a dev, my very strong opinion is that this 
tightly coupled system is actually a boat anchor sinking us.


Just because no one else does it a certain way, does not mean that a 
unique way of doing something is correct and/or sustainable.  Maybe in 
1995, 1999, or 2005 even, but not today.  Especially in the context of 
add-on tools like rcs.


What we need to discuss is lowering the bar to making custom installs.

I personally find that installing FreeBSD is useless until I install 
screen, zsh, vim-lite, git why is that so manual for me?  Why can't I 
just register a package set somewhere so that all I have to type in is 
alfred.perlstein.devel into a box during the installer and I get all 
my packages by default?


--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: rcs

[Alfred Perlstein]

On 10/8/13 8:26 AM, Andreas Nilsson wrote:




On Tue, Oct 8, 2013 at 5:20 PM, Alfred Perlstein bri...@mu.org 
mailto:bri...@mu.org wrote:


On 10/8/13 8:04 AM, sth...@nethelp.no mailto:sth...@nethelp.no
wrote:

I think the fact is that most direct users of RCS use
it in a very
simple way, and
it works just fine for that.  with no real need for
any updates or any
change.

With all due respect Julian, The more we discuss this more
this really
points to the problem that FreeBSD appears to be a
challenge to install
packages into such that a package moving out of base is
such a big deal.

Can we fix that instead?

I mean, this change should really not be a big deal, but
yet it is and
this speaks to the core of FreeBSD utility.

Not commenting on RCS here, but on the concept of moving
packages out
of the base:

- For some of us, the attraction of FreeBSD is that it is a
tightly
integrated system, and the base contains enough useful
functionality
that we don't *have* to add a lot of packages.

- Each package that is moved out of the base system means less
useful
functionality in the base system - and for me: Less reason to use
FreeBSD instead of Linux.

I absolutely see the problem of maintaining out-of-date
packages in
the base system, and the desirability of making the base
system less
reliant on GPL. I'm mostly troubled by the fact that there
seems to
be a rather strong tendency the last few years of having steadily
less functionality in the base system - and I'm not at all
convinced
that the right balance has been found here.

This discussion is not new, and I don't expect to convince any new
persons...


I'm sure other devs will disagree, but with ~15 years of FreeBSD
experience and ~13 years as a dev, my very strong opinion is that
this tightly coupled system is actually a boat anchor sinking us.

Just because no one else does it a certain way, does not mean that
a unique way of doing something is correct and/or sustainable.
 Maybe in 1995, 1999, or 2005 even, but not today.  Especially in
the context of add-on tools like rcs.

What we need to discuss is lowering the bar to making custom installs.

I personally find that installing FreeBSD is useless until I
install screen, zsh, vim-lite, git why is that so manual for me?
 Why can't I just register a package set somewhere so that all I
have to type in is alfred.perlstein.devel into a box during the
installer and I get all my packages by default?

-- 
Alfred Perlstein


You technically can. Make your own meta-port which depends on the 
stuff you want. Build package-set with for example poudriere, ship 
those packages on your install-media. Done.




Oh I have done that in the past, but why the editing, the makefiles, the 
etc, etc, etc.  Why isn't there a customize.freebsd.org where I just hit 
a few checkboxes, save and then hit download?



--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: rcs

[Alfred Perlstein]

On 10/8/13 8:59 AM, Jonathan Anderson wrote:

On 8 October 2013 16:04, sth...@nethelp.no wrote:
 - For some of us, the attraction of FreeBSD is that it is a tightly
 integrated system, and the base contains enough useful functionality
 that we don't *have* to add a lot of packages.

 - Each package that is moved out of the base system means less useful
 functionality in the base system - and for me: Less reason to use
 FreeBSD instead of Linux.

 I absolutely see the problem of maintaining out-of-date packages in
 the base system, and the desirability of making the base system less
 reliant on GPL. I'm mostly troubled by the fact that there seems to
 be a rather strong tendency the last few years of having steadily
 less functionality in the base system - and I'm not at all convinced
 that the right balance has been found here.

I think this is the core problem at the root of many discussions
besides this one. What is the base system?

FreeBSD users tend to agree that we like a self-contained wad of stuff
called The Base System but disagree quite strongly about what should
be in it. There are several approaches to the problem, ranging from
concrete and specific (exactly what shipped with 4.4BSD, a.k.a.
Originalism) to principled but open to interpretation (what 4.4BSD
would ship if it were released today, a.k.a. Founders' Intent).

We will never all agree on exactly what should be in base vs
ports/packages, but can we perhaps build consensus around principles?

When you first take it out of the box, does The Base System need to be:

 - self-bootstrapping
 - POSIX-compliant
 - administerable
   - with local shell
   - with local tools (e.g. RCS, vim, git...)
   - with remote shell (SSH)
   - with remote tools (e.g. Puppet)
   - with enterprise integration (e.g. Kerberos, LDAP, 802.1x, SMB...)
 - useful for end-user workloads:
   - [cross-]building FreeBSD
   - [cross-]building {program X in language Y}
   - file server
   - DNS server
   - Kerberos server
   - SVN server
   - Postgres server
   - Web server
   - Hadoop node
   - X server
   - desktop
 - able to install packages / build ports to do the above
 - able to run Linux binaries

?

I think we all agree with the first two items, but where should we
draw the line?

Suppose we distributed install media with The Base System + some
packages tailored to a particular environment; would that change what
needs to be in The Base System? If FreeBSD Enterprise Edition or
FreeBSD Hacker Edition shipped with The Base System plus whatever
packages you need for that environment/workload, and if the installer
knew how to install those packages, could The Base System itself be
smaller, e.g. just what we need to bootstrap FreeBSD itself?




Jon,

You're right on the money, to be honest this is one of the reasons why 
I've switched to using OSX as my desktop OS.


zsh, vim, screen by default.  and upgrades work.  At the end of the day 
I'm spending time doing work, not mucking about my workspace to make it 
usable for development.


I think this was brought up at BSDCan in the discussion about making 
FreeBSD a more featured development platform.


Speaking of... has anyone tried PCBSD?



--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: rcs

[Alfred Perlstein]

oops, sent from my non-subscribed email..

On 10/8/13 11:17 AM, Freddie Cash wrote:
On Tue, Oct 8, 2013 at 7:09 AM, Alfred Perlstein bri...@mu.org 
mailto:bri...@mu.orgwrote:


On 10/7/13 9:49 PM, Julian Elischer wrote:

On 10/8/13 11:19 AM, Lyndon Nerenberg wrote:

On 2013-10-07, at 8:15 PM, Steve Kargl
s...@troutmask.apl.washington.edu
mailto:s...@troutmask.apl.washington.edu wrote:

Maybe there was no development for 15 years.  However,
the 7364
lines in ChangeLog after 2010-02-04 suggests that
there may
be few bugs to worry about.


I think the fact is that most direct users of RCS use it in a
very simple way, and
it works just fine for that.  with no real need for any
updates or any change.


With all due respect Julian, The more we discuss this more this
really points to the problem that FreeBSD appears to be a
challenge to install packages into such that a package moving out
of base is such a big deal.

Can we fix that instead?

I mean, this change should really not be a big deal, but yet it is
and this speaks to the core of FreeBSD utility.

So again, is there a way to make it so adding RCS after install is
not a big deal at all?


​I haven't kept up-to-date with all the developments, but isn't this 
part of the bsdinstall/pkgng plan?  Once the pkgng repos are all 
available and populated, then bsdinstall will be able to install 
packages from there during the install.  And, isn't that part of the 
plan for the DVD installers, to include an installer repo for 
off-line installs?


IOW, theoretically, one could just download the 10.0 DVD, boot, 
install the base, browse the repo on the DVD, select items to install, 
install, reboot, and be finished.  Without ever needing to touch an 
Internet connection until after rebooting into FreeBSD, if it's even 
needed at all.


Or was that one of the pie-in-the-sky ideas that hasn't materialised 
yet?​



I thought sysinstall could do this 13 years ago?

-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: urtwn driver for Edimax EW-7811U WLAN nano USB Adapter

[Alfred Perlstein]

On 10/8/13 10:41 AM, Julian H. Stacey wrote:

Hi Alfred  cc current.

Alfred Perlstein wrote:

On 10/6/13 8:21 AM, Julian H. Stacey wrote:

I wrote Thu, 03 Oct 2013 23:28:43 +0200

Rui Paulo wrote:

On 2 Oct 2013, at 16:57, Julian H. Stacey j...@berklix.com wrote:


Hi current@,
It seems I need if_urtwn driver for a really miniature WLAN USB stick,
 if_urtwn is only in current ?
man 4 if_urtwn refers to ports/net/urtwn-firmware-kmod which is missing ?

This driver was never merged to FreeBSD 9.

OK, Thanks for confirmation.



Can you use FreeBSD 10 instead?

Yes, easier than building from 9.X I guess ( helps test alpha :-).
I'll fetch from local mirror, per
http://lists.freebsd.org/pipermail/freebsd-current/2013-September/044951.html



The port reference in the man page is wrong. The firmware is now shipped as 
part of the base system.

Oh nice, easier :-)

I'm happy to report with 10.0-ALPHA4 /boot/loader.conf if_urtwn_load=YES
`ifconfig wlan0 scan` works OK. Thanks :-)

Cheers,
Julian

Cool!

I have a g4 tibook 12in with an if_bwn that doesn't really work at all.

I got one of these (if_urtwn) and it works enough to download about a
meg or so before the watchdog kicks in and I have to ifconfig down/up it
to get it to respond again.

I even have a patch pending to add the usb identifier for this.

Is there someone I can provide debug information for to help resolve this?

I too am seeing
urtwn0: timeout waiting for checksum report
sometimes I can scan  sometimes not,
(whereas with a run0: stick I have no problem)
I've not got as far as trying to move data.
I'd appreciate any patches you have Alfred
Unfortunately all my time right now is being spent syncing changes to 
FreeNAS from our commercial product TrueNAS to improve both products.  I 
tried to look at the code but my eyes quickly went cross and I had to 
get back to build stuff.


-Alfred

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: rcs

[Alfred Perlstein]

On 10/8/13 11:17 AM, Freddie Cash wrote:
On Tue, Oct 8, 2013 at 7:09 AM, Alfred Perlstein bri...@mu.org 
mailto:bri...@mu.orgwrote:


On 10/7/13 9:49 PM, Julian Elischer wrote:

On 10/8/13 11:19 AM, Lyndon Nerenberg wrote:

On 2013-10-07, at 8:15 PM, Steve Kargl
s...@troutmask.apl.washington.edu
mailto:s...@troutmask.apl.washington.edu wrote:

Maybe there was no development for 15 years.  However,
the 7364
lines in ChangeLog after 2010-02-04 suggests that
there may
be few bugs to worry about.


I think the fact is that most direct users of RCS use it in a
very simple way, and
it works just fine for that.  with no real need for any
updates or any change.


With all due respect Julian, The more we discuss this more this
really points to the problem that FreeBSD appears to be a
challenge to install packages into such that a package moving out
of base is such a big deal.

Can we fix that instead?

I mean, this change should really not be a big deal, but yet it is
and this speaks to the core of FreeBSD utility.

So again, is there a way to make it so adding RCS after install is
not a big deal at all?


​I haven't kept up-to-date with all the developments, but isn't this 
part of the bsdinstall/pkgng plan?  Once the pkgng repos are all 
available and populated, then bsdinstall will be able to install 
packages from there during the install.  And, isn't that part of the 
plan for the DVD installers, to include an installer repo for 
off-line installs?


IOW, theoretically, one could just download the 10.0 DVD, boot, 
install the base, browse the repo on the DVD, select items to install, 
install, reboot, and be finished.  Without ever needing to touch an 
Internet connection until after rebooting into FreeBSD, if it's even 
needed at all.


Or was that one of the pie-in-the-sky ideas that hasn't materialised 
yet?​




Man, we used to be able to do that with sysinstall 10 years ago?  Is 
that broken?



--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: rcs is gone?

[Alfred Perlstein]

On 10/7/13 2:14 PM, Lyndon Nerenberg wrote:

On 2013-10-07, at 2:08 PM, Lyndon Nerenberg lyn...@orthanc.ca wrote:


And sorry, what I left out was how having ci/co in the base is immensely 
helpful with the installer scripts I write.  The server installation scripts 
I've cooked up use ci(1) to keep a record of changes made during the (possibly 
customized) installation process.  This is impossible if there isn't a basic 
RCS in the base system.

Finally, an issue with missing SCCS in the base is for those of us who work in 
shops behind an air-gapped firewall.

Install from ports is a non-starter.  Our development systems will never be 
connected to the internet for a ports upgrade.  In this environment, in-base RCS is a 
very useful tool.
___



How do you do your initial install?

--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [Heads Up] RCS removed from base

[Alfred Perlstein]

On 10/7/13 6:30 PM, Steve Kargl wrote:

On Tue, Oct 08, 2013 at 08:01:34AM +0800, Julian Elischer wrote:

On 10/7/13 11:06 PM, Steve Kargl wrote:

On Sun, Oct 06, 2013 at 10:43:21PM -0400, Eitan Adler wrote:

Hey all,

RCS was removed from the base system in r256095.  If you still want to
use RCS please install either devel/rcs or devel/rcs57.  If not, be
sure to check out the alternatives (pun stolen and intended).


Perhaps, a note in src/UPDATING is appropriate?


ok so what is this, the secret cabal to make FreeBSD useless?

I'm unsure why you are using the word 'cabal' here.  I have
no dog in this fight.  I happen to read svn commit emails.
I simply noticed that Eitan had removed RCS, and has a service
to those that may use RCS I suggested that a note be added to
src/UPDATING.

PS: As noted, the code is GPL.  There has been an effort
to remove GPL code from FreeBSD (whether prudent or not).


That plus the age of the code is good enough reason to ditch it! huzzah!

Plus we can make RCSBSD along with it.

--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: urtwn driver for Edimax EW-7811U WLAN nano USB Adapter

[Alfred Perlstein]

On 10/6/13 8:21 AM, Julian H. Stacey wrote:

I wrote Thu, 03 Oct 2013 23:28:43 +0200

Rui Paulo wrote:

On 2 Oct 2013, at 16:57, Julian H. Stacey j...@berklix.com wrote:


Hi current@,
It seems I need if_urtwn driver for a really miniature WLAN USB stick,
 if_urtwn is only in current ?
man 4 if_urtwn refers to ports/net/urtwn-firmware-kmod which is missing ?


This driver was never merged to FreeBSD 9.

OK, Thanks for confirmation.



Can you use FreeBSD 10 instead?

Yes, easier than building from 9.X I guess ( helps test alpha :-).
I'll fetch from local mirror, per
http://lists.freebsd.org/pipermail/freebsd-current/2013-September/044951.html



The port reference in the man page is wrong. The firmware is now shipped as 
part of the base system.

Oh nice, easier :-)

I'm happy to report with 10.0-ALPHA4 /boot/loader.conf if_urtwn_load=YES
`ifconfig wlan0 scan` works OK. Thanks :-)

Cheers,
Julian


Cool!

I have a g4 tibook 12in with an if_bwn that doesn't really work at all.

I got one of these (if_urtwn) and it works enough to download about a 
meg or so before the watchdog kicks in and I have to ifconfig down/up it 
to get it to respond again.


I even have a patch pending to add the usb identifier for this.

Is there someone I can provide debug information for to help resolve this?

--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: New iSCSI stack.

[Alfred Perlstein]

On 9/5/13 3:27 AM, Edward Tomasz Napierała wrote:

Hello.  At http://people.freebsd.org/~trasz/cfiscsi-20130904.diff you'll find
a patch which adds the new iSCSI initiator and target, against 10-CURRENT.
To use the new initiator, start with man iscsictl.  For the target - man
ctld.

All feedback is welcome.  If nothing unexpected comes up, I'll commit it
in a few days from now.  Note that it's still not optimized; at this point
I'm focusing more on reliability and interoperability.

This work is being sponsored by FreeBSD Foundation.

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Edward, this is really exciting!

Is there an easy way to use the userland iscsi configuration files?

We would love to quickly backport and ship this with FreeNAS as an 
option for our users, having the config files be the same OR having a 
very good converter would really make that much easier for us.




--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: 2013 MacBook Air Project

[Alfred Perlstein]

This is so cool!

Do you have a repo on github with these patches?

On 8/30/13 9:25 AM, Lundberg, Johannes wrote:

Hi

I thought I'd give a progress report on running FreeBSD 10 on a MacBook Air
11 2013 model.

PCI-E SSD DRIVE
- Added device ID to device list. Should be committed to head already.
- Failed to write partition table due to weird characters at the end in the
SSD's identifier key. Solved by ugly hack (cutting off the ident string in
the middle), fix not committed.

SMP
- No problem when booting from usb memory stick. However, have to disable
smp in /boot/loader.conf with kern.smp.disabled=1 to boot from the SSD.

USB
- Reverted sys/dev/usb/controller/xhci* to 243780 to make it work.

WIFI
- Seems like this one is gonna be difficult due to Broadcom's proprietary
driver

ETHERNET
- Thunderbolt adapter works fine but hot-plugging not supported so you need
to connect it before booting.

BLUETOOTH
- Added device to usbdevs and ng_ubt.c. Device is recognised and ubt0
created but when running service bluetooth start ubt0 I get Unable to
setup Bluetooth stack for device ubt0. Works fine with other generic
bluetooth 4.0 usb dongle. No debugging done.

Will install Xorg next week so report about that coming later.

Best Regards

Johannes Lundberg
BRILLIANTSERVICE CO., LTD. http://www.brilliantservice.co.jp
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org




--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [rfc] migrate lagg to an rmlock

[Alfred Perlstein]

On 8/24/13 7:16 AM, Robert Watson wrote:

On Sat, 24 Aug 2013, Alexander V. Chernikov wrote:


On 24.08.2013 00:54, Adrian Chadd wrote:


I'd like to commit this to -10. It migrates the if_lagg locking
from a rw lock to a rm lock. We see a bit of contention between the
transmit and


We're running lagg with rmlock on several hundred heavily loaded 
machines, it really works better. However, there should not be any 
contention between receive and transmit side since there is actually 
no _real_ need to lock RX (and even use lagg receive code at all):


http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html


We should distinguish lock contention from line contention. When 
acquiring a rwlock on multiple CPUs concurrently, the cache lines used 
to implement the lock are contended, as they must bounce between 
caches via the cache coherence protocol, also referred to as 
contention.  In the if_lagg code, I assume that the read-only 
acquire of the rwlock (and perhaps now rmlock) is for data stability 
rather than mutual exclusion -- e.g., to allow processing to 
completion against a stable version of the lagg configuration. As 
such, indeed, there should be no lock contention unless a 
configuration update takes place, and any line contention is a 
property of the locking primitive rather than data model.


There are a number of other places in the kernel where migration to an 
rmlock makes sense -- however, some care must be taken for four 
reasons: (1) while read locks don't experience line contention, write 
locking becomes observably e.g., rmlocks might not be suitable for 
tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not 
suitable for all rwlock line contention spots -- implement reader 
priority propagation, so you must reason about; and (3) historically, 
rmlocks have not fully implemented WITNESS so you may get less good 
debugging output.  if_lagg is a nice place to use rmlocks, as 
reconfigurations are very rare, and it's really all about long-term 
data stability.


Robert
___
freebsd-...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to freebsd-net-unsubscr...@freebsd.org



Robert, what do you think about a quick swap of the ifnet structures to 
counter before 10.x?


-Alfred

--
Alfred Perlstein

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: patch to add AES intrinsics to gcc

[Alfred Perlstein]

On 8/24/13 9:33 AM, Adrian Chadd wrote:

You know, I could be a total jerk and say:

If you push gcc out to a port, and you have the 'external compiler'
toolchain support working correctly enough to build with this, why don't we
just push clang out to a port, and be done with it?

... just saying.



-adrian



Are you calling me a total jerk?  :)

I'm fine with that because I agree.

-Alfred
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org