Re: Unable to boot -CURRENT on Thinkpad P16s G2

[Doug Ambrisko]

On Thu, Mar 07, 2024 at 07:15:48PM +0100, Philipp Ost wrote:
| On 2/28/24 21:10, Philipp Ost wrote:
| [boot log stripped]
| > Does anyone have any suggestions on how to proceed at this point? [...]
| 
| Short follow-up: disabling uart0 and uart1 at the loader prompt allowed us
| to boot and install FreeBSD (the -CURRENT snapshot from 2024-02-29 in case
| it matters).

UARTS on AMD can be a bit different.  Some BIOS implementations seem
to set them up to work like legacy ports others do not.  On a Naples
platform I helped add support for them since they were not setup
in the legacy configuration.  The AMD servers I'm using now have them
setup in legacy mode and just work like on other systems.

If I remember right those UARTS were defined in ACPI.  On a laptop they
probably don't have serial ports and the probe is getting stuck on
something.  It might be good to instrument it to see what.

Thanks,

Doug A.

Re: 15 & 14: ram_attach vs. its using regions_to_avail vs. "bus_alloc_resource" can lead to: panic("ram_attach: resource %d failed to attach", rid)

[Doug Rabson]

On Sat, 30 Sept 2023 at 08:47, Mark Millard  wrote:

> ram_attach is based on regions_to_avail but that is a problem for
> its later bus_alloc_resource use --and that can lead to:
>
> panic("ram_attach: resource %d failed to attach", rid);
>
> Unfortunately, the known example is use of EDK2 on RPi4B
> class systems, not what is considered the supported way.
> The panic happens for main [so: 15] and will happen once
> the cortex-a72 handling in 14.0-* is in a build fixed by:
>
> • git: 906bcc44641d - releng/14.0 - arm64: Fix errata workarounds that
> depend on smccc Andrew Turner
>
> The lack of the fix leads to an earlier panic as stands.
>
>
> sys/kern/subr_physmem.c 's regions_to_avail is based on ignoring
> phys_avail and using only hwregions and exregions. In other words,
> in part:
>
>  * Initially dump_avail and phys_avail are identical.  Boot time memory
>  * allocations remove extents from phys_avail that may still be included
>  * in dumps.
>
> This means that early, dedicated memory allocations are treated
> as available for general use by regions_to_avail . The distinction
> is visible in the  boot -v output in that:
>
> real memory  = 3138154496 (2992 MB)
> Physical memory chunk(s):
> 0x20 - 0x002b7f, 727711744 bytes (177664 pages)
> 0x002ce3a000 - 0x003385, 111304704 bytes (27174 pages)
> 0x00338c - 0x00338c6fff, 28672 bytes (7 pages)
> 0x0033a3 - 0x0036ef, 55377920 bytes (13520 pages)
> 0x00372e - 0x003b2f, 67239936 bytes (16416 pages)
> 0x004000 - 0x00bb3dcfff, 2067648512 bytes (504797 pages)
> avail memory = 3027378176 (2887 MB)
>
> does not list the wider:
>
> 0x004000 - 0x00bfff
>
> because of phys_avail . But the earlier dump based on hwregions and
> exregions shows:
>
> Physical memory chunk(s):
>   0x001d - 0x001e, 0 MB ( 32 pages)
>   0x0020 - 0x338c6fff,   822 MB ( 210631 pages)
>   0x3392 - 0x3b2f,   121 MB (  31200 pages)
>   0x4000 - 0xbfff,  2048 MB ( 524288 pages)
> Excluded memory regions:
>   0x001d - 0x001e, 0 MB ( 32 pages) NoAlloc
>   0x2b80 - 0x2ce39fff,22 MB (   5690 pages) NoAlloc
>   0x3386 - 0x338b, 0 MB ( 96 pages) NoAlloc
>   0x3392 - 0x33a2, 1 MB (272 pages) NoAlloc
>   0x36f0 - 0x372d, 3 MB (992 pages) NoAlloc
>
> which indicates:
>
>   0x4000 - 0xbfff
>
> is available as far as it is concerned.
>
> (Note some code works/displays in terms of: 0x4000 - 0xc000
> instead.)
>
> For aarch64 , sys/arm64/arm64/nexus.c has a nexus_alloc_resource
> that is used as bus_alloc_resource . It ends up rejecting the
> RPi4B boot via using the result of the call in ram_attach:
>
> if (bus_alloc_resource(dev, SYS_RES_MEMORY, , start,
> end,
> end - start, 0) == NULL)
> panic("ram_attach: resource %d failed to attach",
> rid);
>
> as shown by the just-prior start/end pair sequence messages:
>
> ram0: reserving memory region:   20-2b80
> ram0: reserving memory region:   2ce3a000-3386
> ram0: reserving memory region:   338c-338c7000
> ram0: reserving memory region:   33a3-36f0
> ram0: reserving memory region:   372e-3b30
> ram0: reserving memory region:   4000-c000
> panic: ram_attach: resource 5 failed to attach
>
> I do not see anything about this that looks inherently RPi*
> specific for possibly ending up with an analogous panic. So
> I expect the example is sufficient context to identify a
> problem is present, despite EDK2 use not being normal for
> RPi4B's and the like as far as FreeBSD is concerned.
>

I'm not quite clear why phys_avail changes and why that is triggered by the
906bcc44641d commit. I'm wondering if it makes sense to arrange for
ram_attach to happen before acpi, e.g. using BUS_PASS_ORDER_FIRST?

Doug.

Re: mount_nullfs: /var/run/log: must be either a file or directory

[Doug Rabson]

On Fri, 7 Jul 2023 at 13:11, Mina Galić  wrote:

> Hi folks,
>
> "recently", we added support for null-mounting single files:
>
>
> https://freshbsd.org/freebsd/src/commit/521fbb722c33663cf00a83bca70ad7cb790687b3
>
> This code restricts the mountable … thing to:
>
> if ((lowerrootvp->v_type != VDIR && lowerrootvp->v_type != VREG)
> || …
>
>
> As the author of the abandoned https://reviews.freebsd.org/D27411
> which attempted to add facility to syslog's rc to provide (selected)
> jails with a log socket, it was pointed out to me that this is a big
> security risk: https://reviews.freebsd.org/D27411#882100
>
> so I was wondering if null mounts are the same kind of security
> hazard, or if not allowing sockets is just the oversight of a
> first approximation of this patch?
>

Mounting anything into a jail needs to be done carefully. Clearly null
mounting /sbin into an untrusted jail allows all kinds of shenanigans to
happen but I don't see a huge problem with mounting e.g. a data volume or a
config file into a jail. Care needs to be taken at the point when the
object is mounted to defend against symlinks in the jail's chroot causing
the mount point to change to a surprising location outside the chroot. In
ocijail, I added code to resolve symlinks in the context of the jail's
chroot to avoid this.

I also think it's important to perform any mounts or other configuration
strictly before the jail is started - for OCI containers under podman or
containerd, this may happen after the jail is created but strictly before
anything in the container image is executed. Conversely, unmounting happens
strictly after the jail is removed.

In principle, I don't think it's a problem to mount sockets or fifos into a
jail but one of the points made in your diff that allowing jails to connect
to the host syslogd is a potential risk is a good one.

Doug.

Re: kldunload kernel: How should the kernel behave when it is requested to unload itself

[Doug Rabson]

I think your intuition is correct - it never makes sense to unload the
kernel (IMO). I approved the review.

Doug.


On Thu, 9 Nov 2023 at 16:10, Zhenlei Huang  wrote:

> Hi,
>
> This is *NOT* joking.
>
> While working on https://reviews.freebsd.org/D42527 I realized the
> module kernel also has userrefs, that is to say, userland can request
> to unload kernel, aka `kldunload kernel`.
>
> This is interesting. Well no doubt that the loader can unload kernel.
> Then after the kernel is loaded and has been initialized (SYSINIT), how
> should it behave when it get an unload request?
>
> I'm proposing https://reviews.freebsd.org/D42530 to do not allow unloading
> the kernel. It is by intuition.
>
> What do you think ?
>
>
> Best regards,
> Zhenlei
>
>

Re: MegaCLI port is ports-only -- how would you deploy it?

[Doug Ambrisko]

On Tue, Aug 16, 2022 at 03:33:57PM -0400, Dan Mahoney wrote:
| > On Aug 16, 2022, at 14:04, Doug Ambrisko  wrote:
| > 
| > On Tue, Aug 16, 2022 at 11:58:40AM -0600, Warner Losh wrote:
| > |On Tue, Aug 16, 2022 at 11:40 AM Ed Maste <[1]ema...@freebsd.org>
| > |wrote:
| > | 
| > |  On Mon, 15 Aug 2022 at 20:03, Doug Ambrisko
| > |  <[2]ambri...@ambrisko.com> wrote:
| > |  > | > | > I'd have to put in -current first then look at MFC later
| > |  on.  If looks
| > |  > | > | > good for you then I'll put it up for review.  I just don't
| > |  use this
| > |  > | > | > stuff day to day anymore.
| > |  I think it would be good to put this into review, perhaps separate
| > |  ones for the kernel and userland parts. Feel free to put me on as a
| > |  reviewer or subscriber.
| > | 
| > |I can review as well. I like this plan.
| > 
| > I'll add John as well since I think he wrote mfiutil originally.
| 
| I haven't attached the makefile patch to the bug I opened yet.  (And the
| existing patch is a link, not an attach).

That isn't critical since I'm the one that made it and plan to be
the committer to deal with it.  I grabbed ownership of the bug.
I updated my personal branch with the fix so I shouldn't drop it
again.  I had factored out some other local changes I had in the
driver.  It had issue with various things that the team I used be
part of would hit since that SW sent lots of concurrent I/O to the
card.  It was made worse on the low end system without cache.
Broadcom didn't seem interested in addressing it when I used to work
with them more closely.  I had patches in that teams driver.
 
| If further diagnostics are needed (I'm not going to like, try to use it
| to blow away and re-create an array, sorry), I can provide SOME of that.
| Just tell me how I can help.

That shouldn't really be needed since the MFI commands are consistent
being MegaRAID SAS controllers.  The main thing that has changed is
the driver.  I've heard they are changing for their next controller that
would require new tools and driver.  In a few months I might have
a machine with one to play with.
 
| In the mean time, since the system that I'm testing this on is one of
| the dayjob's console servers, and we still want to be able to run puppet
| on it periodically, y'all have motivated me to get puppet to fix their
| "ports" package provider, which still depends on pkg-classic binaries
| being present.  Not relevant to this problem, other than noting the cool
| ripple effect.

Thanks,

Doug A.

Re: MegaCLI port is ports-only -- how would you deploy it?

[Doug Ambrisko]

On Tue, Aug 16, 2022 at 11:58:40AM -0600, Warner Losh wrote:
|On Tue, Aug 16, 2022 at 11:40 AM Ed Maste <[1]ema...@freebsd.org>
|wrote:
| 
|  On Mon, 15 Aug 2022 at 20:03, Doug Ambrisko
|  <[2]ambri...@ambrisko.com> wrote:
|  > | > | > I'd have to put in -current first then look at MFC later
|  on.  If looks
|  > | > | > good for you then I'll put it up for review.  I just don't
|  use this
|  > | > | > stuff day to day anymore.
|  I think it would be good to put this into review, perhaps separate
|  ones for the kernel and userland parts. Feel free to put me on as a
|  reviewer or subscriber.
| 
|I can review as well. I like this plan.

I'll add John as well since I think he wrote mfiutil originally.

Thanks,

Doug A.

Re: MegaCLI port is ports-only -- how would you deploy it?

[Doug Ambrisko]

On Sat, Aug 13, 2022 at 10:41:33PM -0400, Dan Mahoney wrote:
| 
| 
| > On Aug 12, 2022, at 12:35, Doug Ambrisko  wrote:
| > 
| > On Fri, Aug 12, 2022 at 12:32:56PM -0400, Dan Mahoney wrote:
| > | 
| > | 
| > | > On Aug 12, 2022, at 12:31, Doug Ambrisko  wrote:
| > | > 
| > | > On Fri, Aug 12, 2022 at 12:21:36PM -0400, Dan Mahoney wrote:
| > | > | 
| > | > | > On Aug 8, 2022, at 16:45, Doug Ambrisko  
wrote:
| > | > | > 
| > | > | > On Mon, Aug 08, 2022 at 04:10:10PM -0400, Dan Mahoney wrote:
| > | > | > | 
| > | > | > | 
| > | > | > | > On Aug 8, 2022, at 15:57, Doug Ambrisko  
wrote:
| > | > | > | > 
| > | > | > | > On Thu, Aug 04, 2022 at 05:22:29PM +0300, Ruslan Makhmatkhanov 
wrote:
| > | > | > | > |03.08.2022, 02:07, "Dan Mahoney" :
| > | > | > | > |  Hey there all,
| > | > | > | > |  At the dayjob we have a fleet of Dell Poweredge servers 
that can use
| > | > | > | > |  either mptsas or mrsas -- if you use mptsas, you use 
mptutil (in
| > | > | > | > |  base) to check the state of the card.
| > | > | > | > |  If you use mrsas, you need megacli, which is only in 
ports, and the
| > | > | > | > |  port hasn't translated to pkg probably because of license
| > | > | > | > |  restrictions. ( _LICENSE_RESTRICTED = delete-package
| > | > | > | > |  delete-distfiles), but the license listed is just 
"megacli".
| > | > | > | > |  * We want to deploy a cron job to periodically check the 
raid status
| > | > | > | > |  (we're writing a wrapper, also having it check mfiutil, 
zpool, etc).
| > | > | > | > |  * We do not want to install and manage a whole ports 
tree on every
| > | > | > | > |  machine in our fleet, just to install a raid utlity.
| > | > | > | > |  Option A:
| > | > | > | > |  Make a local package somehow.
| > | > | > | > |  The port just downloads a static binary, there's nothing 
to build
| > | > | > | > |  here, but we want to do this the "right" way. Is there 
some way to
| > | > | > | > |  have pkg deploy a single local package for this that 
will, for
| > | > | > | > |  example, report the right package ownership, without 
moving every
| > | > | > | > |  other package to our poudriere install (we're just using 
base
| > | > | > | > |  packages, we keep poudriere around for testing in case 
we need to
| > | > | > | > |  hot-patch something).
| > | > | > | > |  For what it's worth, we use puppet for config 
management, so pushing
| > | > | > | > |  out the static binary is not the worst answer, but it 
also feels
| > | > | > | > |  "dirty".
| > | > | > | > |  Option B:
| > | > | > | > |  Figure out how to fix the license. I have no idea what 
this would
| > | > | > | > |  involve.
| > | > | > | > |  Option C:
| > | > | > | > |  Also, apparently MegaCLI is no longer maintained 
(replaced by
| > | > | > | > |  StorCLI), but there's no port for StorCLI, and...there 
are multiple
| > | > | > | > |  raid-card specific versions? Jeez.
| > | > | > | > |  Feels even more dirty.
| > | > | > | > |  
[1]https://support.siliconmechanics.com/portal/en/kb/articles/storcl
| > | > | > | > |  i-for-freebsd-and-other-operating-systems
| > | > | > | > |  Ideas welcome?
| > | > | > | > |  -Dan Mahoney
| > | > | > | > 
| > | > | > | > Although the path to get to StorCli goes through various cards 
the
| > | > | > | > latest greatest seem to work on all earlier cards.  It works on
| > | > | > | > HBAs and not just RAID cards.  At work I did a Linux/FreeBSD
| > | > | > | > POC for FW management and found the FreeBSD version could flash 
the HBA
| > | > | > | > and drive FW.  I've moved to StorCli from MegaCli.  I would 
suggest
| > | > | > | > we drop the MegaCli port and move to StorCli.
| > | > | > | > 
| > | > | > | > I have code to make mfiutil into mrsasutil and added the MFI 
ioctl
| > | > | > | > handler to mrsas.  I'm not sure how much value that has.  I 
don't
| > | > | > | > deal with supporting FreeBSD and RAID much anymore.  If 
interested
| > | > | > | > I could send patches.
| > | > | > | 
| > | > | > | This feels like it should be in base, regardless.  Just 
*something* to 
| > | > | > | q

Re: SAS/SATA controllers: 8 port that support 8TB Drives

[Doug Ambrisko]

On Fri, Jun 17, 2022 at 05:55:50PM -0500, Larry Rosenman wrote:
| On 06/17/2022 5:48 pm, Michael Gmelin wrote:
| >> On 18. Jun 2022, at 00:31, Alexander Motin  wrote:
| >> 
| >> 
| >> 
| >>> On 17.06.2022 18:24, Alexander Motin wrote:
| >>>> On 17.06.2022 18:16, Larry Rosenman wrote:
| >>>> On 06/17/2022 5:08 pm, Alexander Motin wrote:
| >>>>> On 17.06.2022 11:59, Larry Rosenman wrote:
| >>>>>> I'm looking to upgrade the controllers in my TrueNAS box to 
| >>>>>> something that will
| >>>>>> support 8TB drives because apparently my LSI 2108 controllers do 
| >>>>>> not support 8TB drives.
| >>>>>> 
| >>>>>> What's the communities recommendation?
| >>>>>> needs to support SFF connectors for a total of 4 SFF connectors, 
| >>>>>> as I have 16 slots.
| >>>>> 
| >>>>> We at iX are still using LSI/Broadcom HBAs, just moved from long
| >>>>> discontinued mps(4) to newer mpr(4).  And I don't believe the 
| >>>>> problem
| >>>>> is directly related to capacity.  According to my observations it 
| >>>>> may
| >>>>> be Seagate HDDs of/above certain (8TB) generation.  We do not use
| >>>>> Seagate HDDs in our products, so about that instability I only 
| >>>>> heard
| >>>>> from forums and TrueNAS community user reports.
| >>>> 
| >>>> This is a mfi(4) set of controllers, and a ST8Nm0045 8TB (CMR) 
| >>>> drive.
| >>>> 
| >>>> Is this a bad combo?
| >>>> 
| >>>> mfi0: 9973 (708793330s/0x0002/WARN) - PD 00(e0xfc/s3) is not 
| >>>> supported
| >>>> (probe0:mfi0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
| >>>> (probe0:mfi0:0:0:0): CAM status: CCB request completed with an error
| >>>> (probe0:mfi0:0:0:0): Retrying command, 3 more tries remain
| >>>> (probe0:mfi0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
| >>>> (probe0:mfi0:0:0:0): CAM status: CCB request completed with an error
| >>>> (probe0:mfi0:0:0:0): Retrying command, 2 more tries remain
| >>>> (probe0:mfi0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
| >>>> (probe0:mfi0:0:0:0): CAM status: CCB request completed with an error
| >>>> (probe0:mfi0:0:0:0): Retrying command, 1 more tries remain
| >>>> (probe0:mfi0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
| >>>> (probe0:mfi0:0:0:0): CAM status: CCB request completed with an error
| >>>> (probe0:mfi0:0:0:0): Retrying command, 0 more tries remain
| >>>> (probe0:mfi0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
| >>>> (probe0:mfi0:0:0:0): CAM status: CCB request completed with an error
| >>>> (probe0:mfi0:0:0:0): Error 5, Retries exhausted
| >>>> 
| >>>> mfi0 Physical Drives:
| >>>>   0 (  932G) UNCONFIGURED GOOD >>> serial=ZA1AC912> SATA E1:S3
| >>> mfi(4) are RAIDs, not HBAs.  We do not recommend RAIDs with TrueNAS 
| >>> due to problems with hot-plug, disk identification, etc. and so have 
| >>> limited experience with them.  But I know some of LSI RAIDs can be 
| >>> reflashed into equivalent HBAs, so if they share the hardware, I can 
| >>> speculate that they may share some issues.
| >> 
| >> I've just noticed "932G" instead of "8000G".  It is obviously a bigger 
| >> problem than what we heard for HBAs.  It looks like a kind of problems 
| >> that should not happen to HBAs, since they should not care about disk 
| >> capacity.
| >> 
| > 
| > What does `smartctl -a ` report (especially sector sizes)?
| > 
| > -m
| > 
| > 
| >> --
| >> Alexander Motin
| >> 
| It's not even making a mfid* node (it is a 4Kn disk)

FYI, mfi never got support for 4Kn drives and is hard coded to 512.  mrsas
does have support so that is better to use for 4Kn and newer controllers.
My day job isn't in that area anymore.  I looked a bit as what it would
take to support 4Kn in mfi and that required a bunch of things and structures
to figure it out.

Doug A.

Re: nullfs and ZFS issues

[Doug Ambrisko]

On Fri, Apr 22, 2022 at 09:04:39AM +0200, Alexander Leidinger wrote:
| Quoting Doug Ambrisko  (from Thu, 21 Apr 2022  
| 09:38:35 -0700):
| 
| > On Thu, Apr 21, 2022 at 03:44:02PM +0200, Alexander Leidinger wrote:
| > | Quoting Mateusz Guzik  (from Thu, 21 Apr 2022
| > | 14:50:42 +0200):
| > |
| > | > On 4/21/22, Alexander Leidinger  wrote:
| > | >> I tried nocache on a system with a lot of jails which use nullfs,
| > | >> which showed very slow behavior in the daily periodic runs (12h runs
| > | >> in the night after boot, 24h or more in subsequent nights). Now the
| > | >> first nightly run after boot was finished after 4h.
| > | >>
| > | >> What is the benefit of not disabling the cache in nullfs? I would
| > | >> expect zfs (or ufs) to cache the (meta)data anyway.
| > | >>
| > | >
| > | > does the poor performance show up with
| > | > https://people.freebsd.org/~mjg/vnlru_free_pick.diff ?
| > |
| > | I would like to have all the 22 jails run the periodic scripts a
| > | second night in a row before trying this.
| > |
| > | > if the long runs are still there, can you get some profiling from it?
| > | > sysctl -a before and after would be a start.
| > | >
| > | > My guess is that you are the vnode limit and bumping into the 1  
| > second sleep.
| > |
| > | That would explain the behavior I see since I added the last jail
| > | which seems to have crossed a threshold which triggers the slow
| > | behavior.
| > |
| > | Current status (with the 112 nullfs mounts with nocache):
| > | kern.maxvnodes:   10485760
| > | kern.numvnodes:3791064
| > | kern.freevnodes:   3613694
| > | kern.cache.stats.heldvnodes:151707
| > | kern.vnodes_created: 260288639
| > |
| > | The maxvnodes value is already increased by 10 times compared to the
| > | default value on this system.
| >
| > I've attached mount.patch that when doing mount -v should
| > show the vnode usage per filesystem.  Note that the problem I was
| > running into was after some operations arc_prune and arc_evict would
| > consume 100% of 2 cores and make ZFS really slow.  If you are not
| > running into that issue then nocache etc. shouldn't be needed.
| 
| I don't run into this issue, but I have a huge perf difference when  
| using nocache in the nightly periodic runs. 4h instead of 12-24h (22  
| jails on this system).

I wouldn't do the nocache then!  It would be good to see what
Mateusz patch does without nocache for your env.
 
| > On my laptop I set ARC to 1G since I don't use swap and in the past
| > ARC would consume to much memory and things would die.  When the
| > nullfs holds a bunch of vnodes then ZFS couldn't release them.
| >
| > FYI, on my laptop with nocache and limited vnodes I haven't run
| > into this problem.  I haven't tried the patch to let ZFS free
| > it's and nullfs vnodes on my laptop.  I have only tried it via
| 
| I have this patch and your mount patch installed now, without nocache  
| and reduced arc reclaim settings (100, 1). I will check the runtime  
| for the next 2 days.
| 
| Your mount patch to show the per mount vnodes count looks useful, not  
| only for this particular case. Do you intend to commit it?

I should since it doesn't change the size of the structure etc.  I need
to put it up for review.

Thanks,

Doug A.

Re: nullfs and ZFS issues

[Doug Ambrisko]

On Thu, Apr 21, 2022 at 03:44:02PM +0200, Alexander Leidinger wrote:
| Quoting Mateusz Guzik  (from Thu, 21 Apr 2022  
| 14:50:42 +0200):
| 
| > On 4/21/22, Alexander Leidinger  wrote:
| >> I tried nocache on a system with a lot of jails which use nullfs,
| >> which showed very slow behavior in the daily periodic runs (12h runs
| >> in the night after boot, 24h or more in subsequent nights). Now the
| >> first nightly run after boot was finished after 4h.
| >>
| >> What is the benefit of not disabling the cache in nullfs? I would
| >> expect zfs (or ufs) to cache the (meta)data anyway.
| >>
| >
| > does the poor performance show up with
| > https://people.freebsd.org/~mjg/vnlru_free_pick.diff ?
| 
| I would like to have all the 22 jails run the periodic scripts a  
| second night in a row before trying this.
| 
| > if the long runs are still there, can you get some profiling from it?
| > sysctl -a before and after would be a start.
| >
| > My guess is that you are the vnode limit and bumping into the 1 second 
sleep.
| 
| That would explain the behavior I see since I added the last jail  
| which seems to have crossed a threshold which triggers the slow  
| behavior.
| 
| Current status (with the 112 nullfs mounts with nocache):
| kern.maxvnodes:   10485760
| kern.numvnodes:3791064
| kern.freevnodes:   3613694
| kern.cache.stats.heldvnodes:151707
| kern.vnodes_created: 260288639
| 
| The maxvnodes value is already increased by 10 times compared to the  
| default value on this system.

With the patch, you shouldn't mount with nocache!  However, you might
want to tune:
vfs.zfs.arc.meta_prune
vfs.zfs.arc.meta_adjust_restarts

Since the code on restart will increment the prune amount by
vfs.zfs.arc.meta_prune and submit that amount to the vnode reclaim
code.  So then it will end up reclaiming a lot of vnodes.  The
defaults of 1 * 4096 and submitting it each loop can most of
the cache to be freed.  With relative small values of them, then
the cache didn't shrink to much.

Doug A.

Re: nullfs and ZFS issues

[Doug Ambrisko]

On Thu, Apr 21, 2022 at 03:44:02PM +0200, Alexander Leidinger wrote:
| Quoting Mateusz Guzik  (from Thu, 21 Apr 2022  
| 14:50:42 +0200):
| 
| > On 4/21/22, Alexander Leidinger  wrote:
| >> I tried nocache on a system with a lot of jails which use nullfs,
| >> which showed very slow behavior in the daily periodic runs (12h runs
| >> in the night after boot, 24h or more in subsequent nights). Now the
| >> first nightly run after boot was finished after 4h.
| >>
| >> What is the benefit of not disabling the cache in nullfs? I would
| >> expect zfs (or ufs) to cache the (meta)data anyway.
| >>
| >
| > does the poor performance show up with
| > https://people.freebsd.org/~mjg/vnlru_free_pick.diff ?
| 
| I would like to have all the 22 jails run the periodic scripts a  
| second night in a row before trying this.
| 
| > if the long runs are still there, can you get some profiling from it?
| > sysctl -a before and after would be a start.
| >
| > My guess is that you are the vnode limit and bumping into the 1 second 
sleep.
| 
| That would explain the behavior I see since I added the last jail  
| which seems to have crossed a threshold which triggers the slow  
| behavior.
| 
| Current status (with the 112 nullfs mounts with nocache):
| kern.maxvnodes:   10485760
| kern.numvnodes:3791064
| kern.freevnodes:   3613694
| kern.cache.stats.heldvnodes:151707
| kern.vnodes_created: 260288639
| 
| The maxvnodes value is already increased by 10 times compared to the  
| default value on this system.

I've attached mount.patch that when doing mount -v should
show the vnode usage per filesystem.  Note that the problem I was
running into was after some operations arc_prune and arc_evict would
consume 100% of 2 cores and make ZFS really slow.  If you are not
running into that issue then nocache etc. shouldn't be needed.
On my laptop I set ARC to 1G since I don't use swap and in the past
ARC would consume to much memory and things would die.  When the
nullfs holds a bunch of vnodes then ZFS couldn't release them.

FYI, on my laptop with nocache and limited vnodes I haven't run
into this problem.  I haven't tried the patch to let ZFS free
it's and nullfs vnodes on my laptop.  I have only tried it via
bhyve test.  I use bhyve and a md drive to avoid wearing
out my SSD and it's faster to test.  I have found the git, tar,
make world etc. could trigger the issue before but haven't had
any issues with nocache and capping vnodes.

Thanks,

Doug A.
diff --git a/sbin/mount/mount.c b/sbin/mount/mount.c
index 79d9d6cb0ca..00eefb3a5e0 100644
--- a/sbin/mount/mount.c
+++ b/sbin/mount/mount.c
@@ -692,6 +692,13 @@ prmount(struct statfs *sfp)
 			xo_emit("{D:, }{Lw:fsid}{:fsid}", fsidbuf);
 			free(fsidbuf);
 		}
+		if (sfp->f_nvnodelistsize != 0 || sfp->f_lazyvnodelistsize != 0) {
+			xo_open_container("vnodes");
+xo_emit("{D:, }{Lwc:vnodes}{Lw:count}{w:count/%ju}{Lw:lazy}{:lazy/%ju}",
+(uintmax_t)sfp->f_nvnodelistsize,
+(uintmax_t)sfp->f_lazyvnodelistsize);
+			xo_close_container("vnodes");
+		}
 	}
 	xo_emit("{D:)}\n");
 }
diff --git a/sys/kern/vfs_mount.c b/sys/kern/vfs_mount.c
index a495ad86ac4..3648ef8d080 100644
--- a/sys/kern/vfs_mount.c
+++ b/sys/kern/vfs_mount.c
@@ -2625,6 +2626,8 @@ __vfs_statfs(struct mount *mp, struct statfs *sbp)
 	sbp->f_version = STATFS_VERSION;
 	sbp->f_namemax = NAME_MAX;
 	sbp->f_flags = mp->mnt_flag & MNT_VISFLAGMASK;
+	sbp->f_nvnodelistsize = mp->mnt_nvnodelistsize;
+	sbp->f_lazyvnodelistsize = mp->mnt_lazyvnodelistsize;
 
 	return (mp->mnt_op->vfs_statfs(mp, sbp));
 }
diff --git a/sys/sys/mount.h b/sys/sys/mount.h
index 3383bfe8f43..95dd3c76ae5 100644
--- a/sys/sys/mount.h
+++ b/sys/sys/mount.h
@@ -91,7 +91,9 @@ struct statfs {
 	uint64_t f_asyncwrites;		/* count of async writes since mount */
 	uint64_t f_syncreads;		/* count of sync reads since mount */
 	uint64_t f_asyncreads;		/* count of async reads since mount */
-	uint64_t f_spare[10];		/* unused spare */
+	uint32_t f_nvnodelistsize;	/* (i) # of vnodes */
+	uint32_t f_lazyvnodelistsize;/* (l) # of lazy vnodes */
+	uint64_t f_spare[9];		/* unused spare */
 	uint32_t f_namemax;		/* maximum filename length */
 	uid_t	  f_owner;		/* user that mounted the filesystem */
 	fsid_t	  f_fsid;		/* filesystem id */

Re: nullfs and ZFS issues

[Doug Ambrisko]

On Wed, Apr 20, 2022 at 11:39:44AM +0200, Alexander Leidinger wrote:
| Quoting Doug Ambrisko  (from Mon, 18 Apr 2022  
| 16:32:38 -0700):
| 
| > With nullfs, nocache and settings max vnodes to a low number I can
| 
| Where is nocache documented? I don't see it in mount_nullfs(8),  
| mount(8) or nullfs(5).

I didn't find it but it is in:
src/sys/fs/nullfs/null_vfsops.c:  if (vfs_getopt(mp->mnt_optnew, 
"nocache", NULL, NULL) == 0 ||

Also some file systems disable it via MNTK_NULL_NOCACHE

| I tried a nullfs mount with nocache and it doesn't show up in the  
| output of "mount".

Yep, I saw that as well.  I could tell by dropping into ddb and then
do a show mount on the FS and look at the count.  That is why I added
the vnode count to mount -v so I could see the usage without dropping
into ddb.

Doug A.

Re: nullfs and ZFS issues

[Doug Ambrisko]

On Wed, Apr 20, 2022 at 11:43:10AM +0200, Mateusz Guzik wrote:
| On 4/19/22, Doug Ambrisko  wrote:
| > On Tue, Apr 19, 2022 at 11:47:22AM +0200, Mateusz Guzik wrote:
| > | Try this: https://people.freebsd.org/~mjg/vnlru_free_pick.diff
| > |
| > | this is not committable but should validate whether it works fine
| >
| > As a POC it's working.  I see the vnode count for the nullfs and
| > ZFS go up.  The ARC cache also goes up until it exceeds the ARC max.
| > size tten the vnodes for nullfs and ZFS goes down.  The ARC cache goes
| > down as well.  This all repeats over and over.  The systems seems
| > healthy.  No excessive running of arc_prune or arc_evict.
| >
| > My only comment is that the vnode freeing seems a bit agressive.
| > Going from ~15,000 to ~200 vnode for nullfs and the same for ZFS.
| > The ARC drops from 70M to 7M (max is set at 64M) for this unit
| > test.
| >
| 
| Can you check what kind of shrinking is requested by arc to begin
| with? I imagine encountering a nullfs vnode may end up recycling 2
| instead of 1, but even repeated a lot it does not explain the above.

I dug it into a bit more and think there could be a bug in:
module/zfs/arc.c
arc_evict_meta_balanced(uint64_t meta_used)
prune += zfs_arc_meta_prune;
//arc_prune_async(prune);
arc_prune_async(zfs_arc_meta_prune);

Since arc_prune_async, is queuing up a run of arc_prune_task for each
call it is actually already accumulating the zfs_arc_meta_prune
amount.  It makes the count to vnlru_free_impl get really big quickly
since it is looping via restart.

   1 HELLO arc_prune_task 164   ticks 2147465958 count 2048

dmesg | grep arc_prune_task | uniq -c
  14 HELLO arc_prune_task 164   ticks -2147343772 count 100
  50 HELLO arc_prune_task 164   ticks -2147343771 count 100
  46 HELLO arc_prune_task 164   ticks -2147343770 count 100
  49 HELLO arc_prune_task 164   ticks -2147343769 count 100
  44 HELLO arc_prune_task 164   ticks -2147343768 count 100
 116 HELLO arc_prune_task 164   ticks -2147343767 count 100
1541 HELLO arc_prune_task 164   ticks -2147343766 count 100
  53 HELLO arc_prune_task 164   ticks -2147343101 count 100
 100 HELLO arc_prune_task 164   ticks -2147343100 count 100
  75 HELLO arc_prune_task 164   ticks -2147343099 count 100
  52 HELLO arc_prune_task 164   ticks -2147343098 count 100
  50 HELLO arc_prune_task 164   ticks -2147343097 count 100
  51 HELLO arc_prune_task 164   ticks -2147343096 count 100
 783 HELLO arc_prune_task 164   ticks -2147343095 count 100
 884 HELLO arc_prune_task 164   ticks -2147343094 count 100

Note I shrunk vfs.zfs.arc.meta_prune=100 to see how that might
help.  Changing it to 1, helps more!  I see less agressive
swings.

I added
printf("HELLO %s %d   ticks %d count 
%ld\n",__FUNCTION__,__LINE__,ticks,nr_scan);

to arc_prune_task.

Adjusting both
sysctl vfs.zfs.arc.meta_adjust_restarts=1
sysctl vfs.zfs.arc.meta_prune=100

without changing arc_prune_async(prune) helps avoid excessive swings.

Thanks,

Doug A.

| > | On 4/19/22, Mateusz Guzik  wrote:
| > | > On 4/19/22, Mateusz Guzik  wrote:
| > | >> On 4/19/22, Doug Ambrisko  wrote:
| > | >>> I've switched my laptop to use nullfs and ZFS.  Previously, I used
| > | >>> localhost NFS mounts instead of nullfs when nullfs would complain
| > | >>> that it couldn't mount.  Since that check has been removed, I've
| > | >>> switched to nullfs only.  However, every so often my laptop would
| > | >>> get slow and the the ARC evict and prune thread would consume two
| > | >>> cores 100% until I rebooted.  I had a 1G max. ARC and have increased
| > | >>> it to 2G now.  Looking into this has uncovered some issues:
| > | >>>  -nullfs would prevent vnlru_free_vfsops from doing 
anything
| > | >>>   when called from ZFS arc_prune_task
| > | >>>  -nullfs would hang onto a bunch of vnodes unless mounted 
with
| > | >>>   nocache
| > | >>>  -nullfs and nocache would break untar.  This has been 
fixed
| > now.
| > | >>>
| > | >>> With nullfs, nocache and settings max vnodes to a low number I can
| > | >>> keep the ARC around the max. without evict and prune consuming
| > | >>> 100% of 2 cores.  This doesn't seem like the best solution but it
| > | >>> better then when the ARC starts spinning.
| > | >>>
| > | >>> Looking into this issue with bhyve and a md drive for testing I
| > create
| > | >>> a brand new zpool mounted as /test and then nullfs mount /test to
| > /mnt.
| > | >>> I loop through untaring the Linux kernel into the nullfs mount, rm
|

Re: nullfs and ZFS issues

[Doug Ambrisko]

On Tue, Apr 19, 2022 at 11:47:22AM +0200, Mateusz Guzik wrote:
| Try this: https://people.freebsd.org/~mjg/vnlru_free_pick.diff
| 
| this is not committable but should validate whether it works fine

As a POC it's working.  I see the vnode count for the nullfs and
ZFS go up.  The ARC cache also goes up until it exceeds the ARC max.
size tten the vnodes for nullfs and ZFS goes down.  The ARC cache goes
down as well.  This all repeats over and over.  The systems seems
healthy.  No excessive running of arc_prune or arc_evict.

My only comment is that the vnode freeing seems a bit agressive.
Going from ~15,000 to ~200 vnode for nullfs and the same for ZFS.
The ARC drops from 70M to 7M (max is set at 64M) for this unit
test.

Thanks,

Doug A.
 
| On 4/19/22, Mateusz Guzik  wrote:
| > On 4/19/22, Mateusz Guzik  wrote:
| >> On 4/19/22, Doug Ambrisko  wrote:
| >>> I've switched my laptop to use nullfs and ZFS.  Previously, I used
| >>> localhost NFS mounts instead of nullfs when nullfs would complain
| >>> that it couldn't mount.  Since that check has been removed, I've
| >>> switched to nullfs only.  However, every so often my laptop would
| >>> get slow and the the ARC evict and prune thread would consume two
| >>> cores 100% until I rebooted.  I had a 1G max. ARC and have increased
| >>> it to 2G now.  Looking into this has uncovered some issues:
| >>>  -nullfs would prevent vnlru_free_vfsops from doing anything
| >>>   when called from ZFS arc_prune_task
| >>>  -nullfs would hang onto a bunch of vnodes unless mounted with
| >>>   nocache
| >>>  -nullfs and nocache would break untar.  This has been fixed now.
| >>>
| >>> With nullfs, nocache and settings max vnodes to a low number I can
| >>> keep the ARC around the max. without evict and prune consuming
| >>> 100% of 2 cores.  This doesn't seem like the best solution but it
| >>> better then when the ARC starts spinning.
| >>>
| >>> Looking into this issue with bhyve and a md drive for testing I create
| >>> a brand new zpool mounted as /test and then nullfs mount /test to /mnt.
| >>> I loop through untaring the Linux kernel into the nullfs mount, rm -rf
| >>> it
| >>> and repeat.  I set the ARC to the smallest value I can.  Untarring the
| >>> Linux kernel was enough to get the ARC evict and prune to spin since
| >>> they couldn't evict/prune anything.
| >>>
| >>> Looking at vnlru_free_vfsops called from ZFS arc_prune_task I see it
| >>>   static int
| >>>   vnlru_free_impl(int count, struct vfsops *mnt_op, struct vnode *mvp)
| >>>   {
| >>>   ...
| >>>
| >>> for (;;) {
| >>>   ...
| >>> vp = TAILQ_NEXT(vp, v_vnodelist);
| >>>   ...
| >>>
| >>> /*
| >>>  * Don't recycle if our vnode is from different type
| >>>  * of mount point.  Note that mp is type-safe, the
| >>>  * check does not reach unmapped address even if
| >>>  * vnode is reclaimed.
| >>>  */
| >>> if (mnt_op != NULL && (mp = vp->v_mount) != NULL &&
| >>> mp->mnt_op != mnt_op) {
| >>> continue;
| >>> }
| >>>   ...
| >>>
| >>> The vp ends up being the nulfs mount and then hits the continue
| >>> even though the passed in mvp is on ZFS.  If I do a hack to
| >>> comment out the continue then I see the ARC, nullfs vnodes and
| >>> ZFS vnodes grow.  When the ARC calls arc_prune_task that calls
| >>> vnlru_free_vfsops and now the vnodes go down for nullfs and ZFS.
| >>> The ARC cache usage also goes down.  Then they increase again until
| >>> the ARC gets full and then they go down again.  So with this hack
| >>> I don't need nocache passed to nullfs and I don't need to limit
| >>> the max vnodes.  Doing multiple untars in parallel over and over
| >>> doesn't seem to cause any issues for this test.  I'm not saying
| >>> commenting out continue is the fix but a simple POC test.
| >>>
| >>
| >> I don't see an easy way to say "this is a nullfs vnode holding onto a
| >> zfs vnode". Perhaps the routine can be extrended with issuing a nullfs
| >> callback, if the module is loaded.
| >>
| >> In the meantime I think a good enough(tm) fix would be to check that
| >> nothing was freed and fallback to good old regular clean up without
| >> filtering by vfsops. This would 

nullfs and ZFS issues

[Doug Ambrisko]

I've switched my laptop to use nullfs and ZFS.  Previously, I used
localhost NFS mounts instead of nullfs when nullfs would complain
that it couldn't mount.  Since that check has been removed, I've
switched to nullfs only.  However, every so often my laptop would
get slow and the the ARC evict and prune thread would consume two
cores 100% until I rebooted.  I had a 1G max. ARC and have increased
it to 2G now.  Looking into this has uncovered some issues:
 -  nullfs would prevent vnlru_free_vfsops from doing anything
when called from ZFS arc_prune_task
 -  nullfs would hang onto a bunch of vnodes unless mounted with
nocache
 -  nullfs and nocache would break untar.  This has been fixed now.

With nullfs, nocache and settings max vnodes to a low number I can
keep the ARC around the max. without evict and prune consuming
100% of 2 cores.  This doesn't seem like the best solution but it
better then when the ARC starts spinning.

Looking into this issue with bhyve and a md drive for testing I create
a brand new zpool mounted as /test and then nullfs mount /test to /mnt.
I loop through untaring the Linux kernel into the nullfs mount, rm -rf it
and repeat.  I set the ARC to the smallest value I can.  Untarring the
Linux kernel was enough to get the ARC evict and prune to spin since
they couldn't evict/prune anything.

Looking at vnlru_free_vfsops called from ZFS arc_prune_task I see it
  static int
  vnlru_free_impl(int count, struct vfsops *mnt_op, struct vnode *mvp)
  {
...

for (;;) {
...
vp = TAILQ_NEXT(vp, v_vnodelist);
...

/*
 * Don't recycle if our vnode is from different type
 * of mount point.  Note that mp is type-safe, the
 * check does not reach unmapped address even if
 * vnode is reclaimed.
 */
if (mnt_op != NULL && (mp = vp->v_mount) != NULL &&
mp->mnt_op != mnt_op) {
continue;
}
...

The vp ends up being the nulfs mount and then hits the continue
even though the passed in mvp is on ZFS.  If I do a hack to
comment out the continue then I see the ARC, nullfs vnodes and 
ZFS vnodes grow.  When the ARC calls arc_prune_task that calls
vnlru_free_vfsops and now the vnodes go down for nullfs and ZFS.
The ARC cache usage also goes down.  Then they increase again until
the ARC gets full and then they go down again.  So with this hack
I don't need nocache passed to nullfs and I don't need to limit
the max vnodes.  Doing multiple untars in parallel over and over
doesn't seem to cause any issues for this test.  I'm not saying
commenting out continue is the fix but a simple POC test.

It appears that when ZFS is asking for cached vnodes to be
free'd nullfs also needs to free some up as well so that
they are free'd on the VFS level.  It seems that vnlru_free_impl
should allow some of the related nullfs vnodes to be free'd so
the ZFS ones can be free'd and reduce the size of the ARC.

BTW, I also hacked the kernel and mount to show the vnodes used
per mount ie. mount -v:
  test on /test (zfs, NFS exported, local, nfsv4acls, fsid 2b23b2a1de21ed66, 
vnodes: count 13846 lazy 0)
  /test on /mnt (nullfs, NFS exported, local, nfsv4acls, fsid 11ff00292900, 
vnodes: count 13846 lazy 0)

Now I can easily see how the vnodes are used without going into ddb.
On my laptop I have various vnet jails and nullfs mount my homedir into
them so pretty much everything goes through nullfs to ZFS.  I'm limping
along with the nullfs nocache and small number of vnodes but it would be
nice to not need that.

Thanks,

Doug A.

A proposal to redefine RB trees

[Doug Moore]

I have a change ready to commit at https://reviews.freebsd.org/D25480
that would redefine the tree-balancing criteria for the RB tree macros,
changing them from red-black trees to the weak-AVL trees described in
the paper "Rank-balanced trees" by Haeupler, Sen and Tarjan.  By happy
coincidence (or the authors' deliberate scheme), the letters RB can
represent "Rank-balanced" as easily as "red-black", so no global
renaming is required.

This change does not add or remove fields.  It does keep a tighter
balance constraint than red-black trees, so that in conditions where
balancing really matters (for example, when inserting tree nodes in
sorted order), weak-avl trees produce a better balanced tree and faster
lookups.  That same tighter balance constraint means that an insert
operation is more likely to lead to restructuring of the tree than
before, for which there is a small performance cost.

The original paper at
http://sidsen.azurewebsites.net/papers/rb-trees-talg.pdf provides more
analysis of the relative benefits of weak-avl trees.

Are there any objections?

Doug Moore (do...@freebsd.org)

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: frequent panics from current built today

[Doug Moore]

This problem is almost certainly my fault, due to r349393.  Once I
learned of the problem and verified that I was responsible, I reverted
that change with r349405.

I apologize for the multiple inconveniences I have caused.

Doug Moore


> - Forwarded message from Pete Wright  -
>
> Date: Tue, 25 Jun 2019 19:09:07 -0700
> From: Pete Wright 
> To: FreeBSD Current 
> Subject: frequent panics from current built today
>
> hello - i've had a system panic multiple times after rebuilding world/kernel
> today.  here's my info:
> FreeBSD topanga 13.0-CURRENT FreeBSD 13.0-CURRENT 2474a68216f(master)
> GENERIC-NODEBUG  amd64
>
> Looking at two of the panic texts I see this:
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 1; apic id = 01
> fault virtual address   = 0x52
> fault code  = supervisor read data, page not present
> instruction pointer = 0x20:0x80f47f54
> stack pointer   = 0x0:0xfe0122e4a6d0
> frame pointer   = 0x0:0xfe0122e4a7b0
> code segment    = base 0x0, limit 0xf, type 0x1b
>     = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags    = interrupt enabled, resume, IOPL = 0
> current process = 69399 (java)
> trap number = 12
> panic: page fault
> cpuid = 1
> time = 1561514025
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe0122e4a380
> vpanic() at vpanic+0x19d/frame 0xfe0122e4a3d0
> panic() at panic+0x43/frame 0xfe0122e4a430
> trap_fatal() at trap_fatal+0x39c/frame 0xfe0122e4a490
> trap_pfault() at trap_pfault+0x49/frame 0xfe0122e4a4f0
> trap() at trap+0x29f/frame 0xfe0122e4a600
> calltrap() at calltrap+0x8/frame 0xfe0122e4a600
> --- trap 0xc, rip = 0x80f47f54, rsp = 0xfe0122e4a6d0, rbp =
> 0xfe0122e4a7b0 ---
> vm_map_lookup() at vm_map_lookup+0x2a4/frame 0xfe0122e4a7b0
> vm_fault_hold() at vm_fault_hold+0x72/frame 0xfe0122e4a900
> vm_fault() at vm_fault+0x60/frame 0xfe0122e4a940
> trap_pfault() at trap_pfault+0x164/frame 0xfe0122e4a9a0
> trap() at trap+0x42b/frame 0xfe0122e4aab0
> calltrap() at calltrap+0x8/frame 0xfe0122e4aab0
> --- trap 0xc, rip = 0x41f695e0, rsp = 0x7fffdfff0738, rbp = 0x7fffdfff07b0 ---
> KDB: enter: panic
>
> __curthread () at /usr/home/pete/git/freebsd/sys/amd64/include/pcpu.h:246
> 246 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n"
> (OFFSETOF_CURTHREAD));
>
>
> I have usable core files and can post any additional debug info needed.  The
> first crash happened when launching chrome, the second (which is what the
> above text is from) happened when I was running a java process in a jail.  Has
> anyone else seen this?
>
> Cheers,
> -pete
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: panic: mutex pmap not owned at ... efirt_machdep.c:255

[Doug Ambrisko]

On Wed, Aug 08, 2018 at 01:42:07AM +0300, Konstantin Belousov wrote:
| On Tue, Aug 07, 2018 at 02:49:10PM -0700, Doug Ambrisko wrote:
| > On Tue, Aug 07, 2018 at 08:29:49PM +0300, Konstantin Belousov wrote:
| > | On Tue, Aug 07, 2018 at 11:50:44AM -0500, Kyle Evans wrote:
| > | > On Tue, Aug 7, 2018 at 12:09 AM, Eitan Adler  wrote:
| > | > > On Mon, 6 Aug 2018 at 11:27, Kyle Evans  wrote:
| > | > >>
| > | > >> On Sun, Aug 5, 2018 at 5:43 AM, Konstantin Belousov 
 wrote:
| > | > >> > On Sat, Aug 04, 2018 at 09:46:39PM -0500, Kyle Evans wrote:
| > | > >> >>
| > | > >> >> He now gets a little further, but ends up with the same panic due 
to
| > | > >> >> efirtc_probe trying to get time to verify the rtc's actually
| > | > >> >> implemented. What kind of approach must we take to ensure curcpu 
is
| > | > >> >> synced?
| > | > >> >
| > | > >> > It does not panic for me, when I load efirt.ko from the loader 
prompt.
| > | > >> > Anyway, try this
| > | > >>
| > | > >> Right, I also don't get a panic on any of my machines from this.
| > | > >> Hopefully he'll have a chance to try this soon.
| > | > >
| > | > > This change has no impact: it still panics in the same way as without 
the patch.
| > | > >
| > | > 
| > | > That seems indicative of a bigger problem, since we use proc0
| > | > throughout all these bits so we should still be dealing with the same
| > | > pmap that got passed to pmap_pinit0 when we grab
| > | > curthread->td_proc->p_vmspace->vm_pmap.
| > | 
| > | Can you confirm that you get the early efi_enter() call from rtc code,
| > | when you preload the module or compile it into the kernel ?
| > 
| > When I ran into this, I did this change:
| > 
| > Index: dev/efidev/efirt.c
| > ===
| > --- dev/efidev/efirt.c  (revision 337264)
| > +++ dev/efidev/efirt.c  (working copy)
| > @@ -257,7 +257,8 @@
| > if (efi_runtime == NULL)
| > return (ENXIO);
| > td = curthread;
| > -   curpmap = >td_proc->p_vmspace->vm_pmap;
| > +// curpmap = >td_proc->p_vmspace->vm_pmap;
| > +   curpmap = PCPU_GET(curpmap);
| > PMAP_LOCK(curpmap);
| > mtx_lock(_lock);
| > fpu_kern_enter(td, NULL, FPU_KERN_NOCTX);
| > @@ -272,7 +273,8 @@
| >  
| > efi_arch_leave();
| >  
| > -   curpmap = >p_vmspace->vm_pmap;
| > +// curpmap = >p_vmspace->vm_pmap;
| > +   curpmap = PCPU_GET(curpmap);
| > td = curthread;
| > fpu_kern_leave(td, NULL);
| > mtx_unlock(_lock);
| > 
| > Don't know if it is right.  Some previous code used both
| > curpmap = PCPU_GET(curpmap);
| > and
| > curpmap = >td_proc->p_vmspace->vm_pmap;
| > recently it was changes to only use
| > curpmap = >td_proc->p_vmspace->vm_pmap;
| > 
| > Things seem to work after that.  I was able to repro. it with 
| > qemu-system-x86_64 in UEFI mode.  I think it also failed in
| > bhyve UEFI mode.
| 
| The pcpu curpmap and curproc vmspace pmap should be synced.  Esp. since
| there is code relying on this early.  I do not want to paper it over.
| 
| In fact, try this please.  Ignore my previous change.
| 
| diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c
| index 572b2197453..4bce36cc0e5 100644
| --- a/sys/amd64/amd64/pmap.c
| +++ b/sys/amd64/amd64/pmap.c
| @@ -7536,7 +7536,8 @@ pmap_activate_sw(struct thread *td)
|   PCPU_SET(kcr3, pmap->pm_cr3);
|   PCPU_SET(ucr3, pmap->pm_ucr3);
|   }
| - }
| + } else
| + PCPU_SET(curpmap, pmap);
|   if (pmap->pm_ucr3 != PMAP_NO_CR3) {
|   rsp0 = ((vm_offset_t)PCPU_PTR(pti_stack) +
|   PC_PTI_STACK_SZ * sizeof(uint64_t)) & ~0xful;

That works for qemu and bhyve booting in UEFI PXE mode.  I backed
out my other change and synced to head.

Thanks,

Doug A.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: panic: mutex pmap not owned at ... efirt_machdep.c:255

[Doug Ambrisko]

On Tue, Aug 07, 2018 at 08:29:49PM +0300, Konstantin Belousov wrote:
| On Tue, Aug 07, 2018 at 11:50:44AM -0500, Kyle Evans wrote:
| > On Tue, Aug 7, 2018 at 12:09 AM, Eitan Adler  wrote:
| > > On Mon, 6 Aug 2018 at 11:27, Kyle Evans  wrote:
| > >>
| > >> On Sun, Aug 5, 2018 at 5:43 AM, Konstantin Belousov 
 wrote:
| > >> > On Sat, Aug 04, 2018 at 09:46:39PM -0500, Kyle Evans wrote:
| > >> >>
| > >> >> He now gets a little further, but ends up with the same panic due to
| > >> >> efirtc_probe trying to get time to verify the rtc's actually
| > >> >> implemented. What kind of approach must we take to ensure curcpu is
| > >> >> synced?
| > >> >
| > >> > It does not panic for me, when I load efirt.ko from the loader prompt.
| > >> > Anyway, try this
| > >>
| > >> Right, I also don't get a panic on any of my machines from this.
| > >> Hopefully he'll have a chance to try this soon.
| > >
| > > This change has no impact: it still panics in the same way as without the 
patch.
| > >
| > 
| > That seems indicative of a bigger problem, since we use proc0
| > throughout all these bits so we should still be dealing with the same
| > pmap that got passed to pmap_pinit0 when we grab
| > curthread->td_proc->p_vmspace->vm_pmap.
| 
| Can you confirm that you get the early efi_enter() call from rtc code,
| when you preload the module or compile it into the kernel ?

When I ran into this, I did this change:

Index: dev/efidev/efirt.c
===
--- dev/efidev/efirt.c  (revision 337264)
+++ dev/efidev/efirt.c  (working copy)
@@ -257,7 +257,8 @@
if (efi_runtime == NULL)
return (ENXIO);
td = curthread;
-   curpmap = >td_proc->p_vmspace->vm_pmap;
+// curpmap = >td_proc->p_vmspace->vm_pmap;
+   curpmap = PCPU_GET(curpmap);
PMAP_LOCK(curpmap);
mtx_lock(_lock);
fpu_kern_enter(td, NULL, FPU_KERN_NOCTX);
@@ -272,7 +273,8 @@
 
efi_arch_leave();
 
-   curpmap = >p_vmspace->vm_pmap;
+// curpmap = >p_vmspace->vm_pmap;
+   curpmap = PCPU_GET(curpmap);
td = curthread;
fpu_kern_leave(td, NULL);
mtx_unlock(_lock);

Don't know if it is right.  Some previous code used both
curpmap = PCPU_GET(curpmap);
and
curpmap = >td_proc->p_vmspace->vm_pmap;
recently it was changes to only use
curpmap = >td_proc->p_vmspace->vm_pmap;

Things seem to work after that.  I was able to repro. it with 
qemu-system-x86_64 in UEFI mode.  I think it also failed in
bhyve UEFI mode.

Thanks,

Doug A.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: HWPMC on SkyLake

[Doug Rabson]

Did you get any feedback on this? I would like to be able to use hwpmc but
my desktop is a recent skylake and I also get the 'hwpc_core: unknown PMC
architecture: 4' error.

CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz (4008.14-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x506e3  Family=0x6  Model=0x5e  Stepping=3

Features=0xbfebfbff

Features2=0x7ffafbbf
  AMD Features=0x2c100800
  AMD Features2=0x121
  Structured Extended
Features=0x29c6fbb
  XSAVE Features=0xf
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics

hwpc_core: unknown PMC architecture: 4
hwpmc: SOFT/16/64/0x67



On 20 February 2016 at 12:49, Larry Rosenman  wrote:

> Is there anything I can do to help:
>
> hwpc_core: unknown PMC architecture: 4
> hwpmc: SOFT/16/64/0x67
>
> CPU: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz (2592.13-MHz K8-class CPU)
>   Origin="GenuineIntel"  Id=0x506e3  Family=0x6  Model=0x5e  Stepping=3
>
> Features=0xbfebfbff
>
> Features2=0x7ffafbbf
>   AMD Features=0x2c100800
>   AMD Features2=0x121
>   Structured Extended
> Features=0x29c6fbf
>   XSAVE Features=0xf
>   VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
>   TSC: P-state invariant, performance statistics
>
>
> --
> Larry Rosenman http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 E-Mail: l...@lerctr.org
> US Mail: 7011 W Parmer Ln, Apt 1115, Austin, TX 78729-6961
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

plockstat

[Doug Rabson]

Is plockstat supposed to work on FreeBSD? I'm running FreeBSD-current and
when I try it, I get:

plockstat: failed to compile program: probe description
plockstat65047:::rw-block does not match any probes

Any ideas what to try next?
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: ZFSROOT UEFI boot

[Doug Rabson]

On 28 January 2016 at 15:03, Tomoaki AOKI  wrote:

> It's exactly the NO GOOD point. The disk where boot1 is read from
> should be where loader.efi and loader.conf are first read.
>

I just wanted to note that gptzfsboot and zfsboot behaves this way. Boot1
looks for loader in the pool which contains the disk that the BIOS booted.
It passes through the ID of that pool to loader which uses that pool as the
default for loading kernel and modules. I believe this is the correct
behaviour. For gptzfsboot and zfsboot, it is possible to override by
pressing space at the point where it is about to load loader.
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: RPC request sent to 127.0.0.1 becomes from other IP on machine

[Doug Rabson]

I think a local socket is probably the best solution long term. Using a
local socket also allows using filesystem permissions to control access
which is required for gssd but not necessarily for nfsuserd.


On 10 December 2015 at 13:37, Rick Macklem  wrote:

> Hi,
>
> Mark has reported a problem via email where the nfsuserd daemon sees
> requests coming from an IP# assigned to the machine instead of 127.0.0.1.
> Here's a snippet from his message:
>   Ok, I have Plex in a jail and when I scan the remote NFS file share the
>   *local* server's nfsuserd spams the logs.
> Spamming the logs refers to the messages nfsuserd generates when it gets
> a request from an address other than 127.0.0.1.
>
> I think the best solution is to switch nfsuserd over to using an AF_LOCAL
> socket like the gssd uses, but that will take a little coding and probably
> won't be MFCable.
>
> I've sent him the attached patch to try as a workaround.
>
> Does anyone happen to know under what circumstances the address 127.0.0.1
> gets replaced?
>
> And do you know if it will always be replaced with the same
> address?
> (I'm basically wondering if the workaround needs to be a list of IP
> addresses
>  instead of a single address?)
>
> Thanks in advance for any help with this, rick
>
>
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Segmentation fault running ntpd

[Doug Hardie]

> On 4 November 2015, at 08:15, Mark Martinec <mark.martinec+free...@ijs.si> 
> wrote:
> 
> Upgrading 10.2-RELEASE-p6 to 10.2-RELEASE-p7 now solved ntpd crashes
> (apparently fixed by: FreeBSD Errata Notice FreeBSD-EN-15:20.vm).
> 
> Thanks!!!
> 
>  Mark
> 

ntpdc hangs when you do a peers command on 9.3.  Eventually it returns a no 
response from the server.  However, ntpq works just fine and nagios is able to 
get the status without problems.  Both of those did not work properly before.

— Doug

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: gettimeofday((void *)-1, NULL) implicates core dump on recent FreeBSD 11-CURRENT

[Doug Rabson]

On Thursday, July 9, 2015, Garrett Wollman woll...@hergotha.csail.mit.edu
wrote:

 In article
 capq4ffvem2uzi-qptm_6msneelwft814g1ke4rgxu6mrzwy...@mail.gmail.com
 javascript:;,
 oliver.pin...@hardenedbsd.org javascript:; writes:

 Btw, I have found this is atf's documantation:
 atf_tc_expect_signal(SIGSEGV, reaseon), with this, we could mark the
 specific test case could fail / or expect to coredump.

 No.

 I'm not sure why people are having trouble understanding this.

 The test in question is not valid C.  It is entirely erroneous, and
 should be deleted.  Merely computing the value (void *)-1 is allowed
 to perform LITERALLY ANY ACTION AT ALL, including turning your
 computer into a frog.  The compiler is free to implement this as a
 call to abort() if it chooses.  Testing this is nonsensical.

 -GAWollman
 ___
 freebsd-current@freebsd.org javascript:; mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
 javascript:;


+1
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: gettimeofday((void *)-1, NULL) implicates core dump on recent FreeBSD 11-CURRENT

[Doug Rabson]

As far as I can tell, POSIX doesn't require either EFAULT or any other
behaviour - the text in http://www.open-std.org/jtc1/sc22/open/n4217.pdf
just says, No errors are defined. Our man page is wrong and any real
program which relies on gettimeofday not faulting when given bad inputs is
broken.

On 8 July 2015 at 17:16, Jamie Landeg-Jones ja...@dyslexicfish.net wrote:

 Oliver Pinter oliver.pin...@hardenedbsd.org wrote:

  On 7/8/15, O'Connor, Daniel dar...@dons.net.au wrote:
  
   In defence of the test, the man page says it can return EFAULT.
 
  That's fine, but why changed the behaviour since 2015. May 27.? I have
  an older FreeBSD/HardenedBSD install, where this test passing. See
  some previous email in this thread.

 That's why it's called 'undefined' :-)

 Pedantically, being 'undefined', it's behaviour hasn't changed at all!
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: nmount, mountd drops high order MNT_xx flags during a mount update

[Doug Rabson]

You could add a single integer-valued vfsopt which holds the high-order
bits of f_flags?

On 7 May 2015 at 02:10, Rick Macklem rmack...@uoguelph.ca wrote:

 David Boyd reported a problem to freebsd-current@ w.r.t. the
 MNT_AUTOMOUNTED
 flag getting cleared by mountd.
   http://docs.FreeBSD.org/cgi/mid.cgi?1429477202.29812.38.camel
 I just took a look at this and it's kinda ugly...

 mountd acquires a list of mounts via getmntinfo() and then does an
 nmount() for each of them to get rid of any exports. It uses
 f_flags from the statfs returned by getmntinfo() as the 3rd
 argument to nmount().
 -- Since nmount()s 3rd argument is a int, it silently drops any
 MNT_xx flag in the high order 32bits of f_flags. At this time,
 it happens that MNT_AUTOMOUNTED is the only one, but...

 I can think of a couple of ways to fix this, but I don't like any of
 them;-)

 1) I suppose mountd could check for each flag set in f_flags and generate
 a vfsopts string for it, but that means that every time a new flag that
 can be updated is added to MNT_xx, this mountd.c code would have to
 updated.
 -- Ugly!!

 2) Require that all flags in MNT_UPDATEMASK be in the low order 32bits.
I wouldn`t mind this except in means that the MNT_xx flags must be
 redefined
and I don`t think that could get MFC`d.

 3) Add a new newernmount(2) which has a 64bit flags argument instead of
 int.

 Hopefully someone can think of a nice way to fix this, rick
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: What to do about RCS/OpenRCS

[Doug Brewer]

 On Thu, May 07, 2015 at 04:18:38PM -0700, NGie Cooper wrote:

 On Thu, May 7, 2015 at 1:38 PM, Pedro Giffuni p...@freebsd.org wrote:
  Hello;
 
  On 05/07/15 14:56, Lyndon Nerenberg wrote:
 
  On Thu, 7 May 2015, Pedro Giffuni wrote:
 
  Unfortunately I don't use RCS enough (it looks like I should though)
so
  I am not in a good position to take the next step and deal with any
  fallout it may produce.
 
 
  If we can have a build-knob to disable GNU RCS and enable the new one
I
  will happily twist up the new version and hammer on it.
 
 
  Yes, that's usually the next step in the process. It is a little bit
messy
  because
  there is a WITHOUT_RCS option and openrcs doesn't have rcsfreeze (and
  perhaps something else that we don't use).
 
  I really want to check out first if there is some strong opinion
against
  OpenRCS. Perhaps someone that has used it before and thinks it is a
  bad idea.
 
  It looks like there are voices against it, so those have to be
addressed
  first.

 Setting WITHOUT_RCS also breaks etcupdate (the tool requires rcs
 bits); check with jhb first to make sure that OpenRCS works with
 etcupdate.

Confirmed.  Pedro, are you also willing to fix fallout as Xin Li pointed
out?
If not, please revert, thanks.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: old bug: mount_nfs path/name is limited to 88 chars

[Doug Ambrisko]

On Tue, Jan 20, 2015 at 12:45:32PM +0100, Willem Jan Withagen wrote:
| On 2015-01-20 2:05, Xin Li wrote:
| Doing it in 11 makes sense since there is a compat layer for 10
| now? if I knew all of the steps I would happily do them as annoys
| me from time to time as well with the path length issue.
| 
| Compat layer may break applications in other funny ways and we
| probably have to return e.g. ENAMETOOLONG for legacy applications if
| the names are too long to fit into the buffer, but I don't see any
| easy solution either: I wish we have bumped it in 2003 when the struct
| receives its first big revamp by bumping all statistic fields to 64-bit.
| 
| On 2015-01-20 1:23, Allan Jude wrote: On 2015-01-19 16:20, Garrett 
|  Especially with ZFS, I find I have a lot more mounts now, under longer
|  and longer path names, and then I have
|  .zfs/snapshots/snapshotnamehere/path/to/file
| 
|  etc.
| 
|  Definitely a +1 for this is something we need for 11
| 
| +1
| 
| Well to be honest: Things are already broken for me ATM.
| I do have paths that are too long, and tools trip over it.
| 
| Things like building the locate database just don't have everything.
| Which is a pain, especially for finding things fast in backups of ZFS, 
| where the path is even more convoluted that what Allan suggests
| 
| So whether being bitten by the cat of the dog: it still hurts.
| 
| Perhaps it is an intermediate solution to add new settings which are 
| going to be used for userspace programs, like USER_MNAMELEN and 
| USER_PATH_MAX. It will certainly give ENAMETOOLONG back when it involves 
| systemcalls. But then a lot of the other tool stuff would just function.

I have a patch at:
http://www.ambrisko.com/doug/mount/latest.patch
for -current (Nov).  It should work with most file systems, that is I
tried to cover them all.  I haven't tested with ZFS but it should work.
I left some debug HELLO's in there just to make sure mounting looked
right.  They can be safely deleted.  If someone wants to test it and
report issues (with a simple test case) I can fix it and add it to
my test script.

This code won't be going into FreeBSD now since the 64 bit inode will
negate the need for it since the size will be a lot larger.

Thanks,

Doug A.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: SMBus controller

[Doug Ambrisko]

On Sat, Jun 14, 2014 at 04:48:30PM -0700, Sean Bruno wrote:
| On Sat, 2014-06-14 at 17:25 -0600, Warner Losh wrote:
|  On Jun 14, 2014, at 4:43 PM, Sean Bruno sbr...@ignoranthack.me wrote:
|  
|   On Sat, 2014-06-14 at 12:08 -0700, Sean Bruno wrote: 
|   I note that my TLenovo 61 has one of these:
|   
|   ichsmb0@pci0:0:31:3: class=0x0c0500 card=0x20a917aa chip=0x283e8086
|   rev=0x03 hdr=0x00
|  vendor = 'Intel Corporation'
|  device = '82801H (ICH8 Family) SMBus Controller'
|  class  = serial bus
|  subclass   = SMBus
|   
|   
|   
|   So, there's something broken in the initialization of the driver and the
|   driver dependencies here.
|   
|   If I load smb.ko by itself, no other modules are loaded (ichsmb,
|   smbus).  Should it?
|  
|  I don't think so. If I kldload pci, would you expect most of the 
|  drivers in the system to be loaded?
| 
| Heck if I know.  :-)  
| 
| I would suspect that of the three, ichsmb, smbus or smb, one of these
| should cause the other two to be loaded.  This is not the case.

... catching up on email

I wouldn't since these act at various levels.  ichsmb is the HW interface
of which there are different HW interfaces.  Then you want the
generic stack on that.  You might want a user land interface.  There are
kernel consumers like IPMI SSIF and other things that have been done
at companies.  So wouldn't expect loading smb to load a i2c printer
port bit banger driver of which is wrong for that HW.

One word of warning is that it appears ACPI has methods to talk to the
i2c bus.  I've been concerned that there could be fights between ACPI
accessing stuff and FreeBSD native drivers.  On the HW that I have
used this stuff, it wasn't a problem.  I'd be a little concerned on
a laptop since a lot of things can be tied to a i2c bus that ACPI
wants to talk to.  On a server/desktop I haven't run into that problem.
I've noticed there seem to be i2c buses for displays which seems to be
separate.

I've made i2c interface to machines via PCI slots or soldered 
wires onto DIMMs.  Some motherboard have i2c headers.  It's easily
hackable to add widgets to HW that way since there are so many i2c
widgets out there.

Doug A.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: ipmi patch for review

[Doug Ambrisko]

On Thu, Sep 19, 2013 at 03:04:46PM -0400, John Baldwin wrote:
| On Tuesday, September 17, 2013 6:21:10 am Gleb Smirnoff wrote:
|Hi!
|  
|When system is writing a kernel core dump, it issues watchdog
|  pat wdog_kern_pat(WD_LASTVAL). If ipmi is in action, it registers
|  ipmi_wd_event() as event for watchdog. Thus ipmi_wd_event() is
|  called in dumping context.
|  
|  The problem is that ipmi_wd_event() calls into ipmi_set_watchdog(),
|  that calls into ipmi_alloc_request(), which uses M_WAITOK and
|  thus sleeps. This is a smaller problem, since can be converted to
|  M_NOWAIT. But ipmi_set_watchdog() then calls into
|  ipmi_submit_driver_request(), which calls msleep() any time.
|  
|The attached patch allows me to successfully write cores in
|  presence of IPMI.
| 
| Of course, the watchdog might go off during your dump. :)
| 
| The real fix is more complicated, which is that we should not use
| a worker thread for at least SMIC and KCS.

I haven't looked at this patch but I have local code that switches
KCS into polling direct mode when the kernel goes into panic mode.
I use this to write Linux compatible back traces into the system
event logs.  This could allow the watchdogd to continue to work.
This should be easily extended to SMIC mode.  SMBUS would be 
harder but at a prior company I made the SMBIO driver work in polled
mode.

If someone wants to look at this I can post the changes for KCS and
the kernel backtrace to the system event log.  We find this useful
when looking at customer machines.

IPMI gets upset if things get intermixed/interrupted so there needs
to be serialization and cancellation if being interrupted.

Thanks,

Doug A.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: newcons comming

[Doug Ambrisko]

On Sun, Nov 03, 2013 at 11:16:53PM -0800, Kevin Oberman wrote:
|On Thu, Oct 31, 2013 at 2:01 PM, Doug Ambrisko ambri...@ambrisko.com
|wrote:
| 
|  On Fri, Oct 25, 2013 at 03:18:47PM +0300, Aleksandr Rybalko wrote:
|  | Hello fellow hackers!
|  |
|  | I finally reach the point when I can work with newcons instead of
|  | syscons on my laptop. Yes, I know it still buggy and have a lot of
|  | style(9) problems. But we really have to get it into HEAD and 10.0 to
|  | enable shiny new Xorg features, drivers, etc.
| 
|  I built the kernel from:
|  A  A  A  A  base/user/ed/newcons/
|  and installed it on my new ThinkPad T530 with Intel graphics chip.
|  In general it works better since now syspend and resume works with
|  and without X. A However, as with my prior ThinkPads I need to switch
|  out of X to suspend and resume. A So I have vidcontrol -s 1 in my
|  /etc/rc.suspend and vidcontrol -s 9 in /etc/rc.resume. A If I don't
|  the display ends up corrupted but somewhat working.
| 
|  I had to kldload i915kms in /etc/rc.local since having it there at
|  boot via /boot/loader.conf resulted in the system not booting. A I need
|  i915kms for X so I don't need to use vesa mode.
| 
|  The FreeBSD logo on boot is interesting but I'd prefer seeing
|  FreeBSD boot messages to see what is happening. A I'm not sure if
|  there is a flag to disable that since I have played with it much.
| 
|  However, on a whole it is much improved since with i915kms and
|  newcons when in X via vesa I couldn't switch to a non-X display
|  since it was blank.
| 
|Excellent news. I'm really looking forward to newcons. Guess it's time to
|move my T520 to 10-Stable (or Beta)
|.
|Since I'm not running 10, I may simply be clueless. If so, please ignore
|the rest of this.

Newcons isn't in 10 yet but in its own branch.
 
|Are you booting directly to X or using startx from the console? In either
|case, I think that i915kms should auto-load at the start of X, so you
|should not need to pre-load it.

I use XDM to start X.  Auto load might not quite work for me since I
run X in a vimage jail.  I find it easier to keep my base system a
minimal install to start up a bounch of vimage jails.  I have two
mains that are FreeBSD -current that I leap frog and then have a couple of
-stable release (ie. 9.0 and 9.2 now) that I use to build ports.  I
nulls mount the /var/db/pkg, /usr/local and /usr/ports of that release
between those so my ports tend to be fixed in time.  I have some patches
to jail to allow any sysctls and to set what the OS reports for uname etc.
Then pkg_add, ports builds thinks it is running on that version of FreeBSD
(ie. 9.2 release).  This makes it easier for me to run -current and let me
add things without issue later on trying to add something new.  I do have
to build modules outside so it is in sync. with my base OS.
 
|Does newcons require VESA? If not, you should be able to remove it from
|your kernel to get suspend/resume working from X.

As soon as I went to newcons the suspend and resume started to work.
I don't have vesa kldload or static in the kernel.  I auto switch out
of X on suspend and switch back to X on resume to ensure that on resume
the X display isn't messed up (lines shifted etc.).  If I don't do this
then the X gets messed up.  I can see stuff working on the LCD it's just
messed up.

Brightness seems messed up, I patched acpi_video to also set the 2nd
LCD (SB.PCI0.PEG.VID.LCD0._BCM) so that works since the keys don't work.
However when X comes back from resume or dpms then it is full bridgtness.
So I need to figure that out.  I also run into the N compatibility issue
with the iwn 6205 that Sean did so I #ifdef'ed that out like he did.  Now
the 6205 works fine.  My other laptop with a 5300 didn't have this issue.

My only real problem right now is that this new laptop came with a defective
key so I'm waiting to get that replaced and then I'll switch to it.

Thanks,

Doug A.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: newcons comming

[Doug Ambrisko]

On Mon, Nov 04, 2013 at 10:30:39AM -0500, Ed Maste wrote:
| On 4 November 2013 02:16, Kevin Oberman rkober...@gmail.com wrote:
|  Excellent news. I'm really looking forward to newcons. Guess it's time to
|  move my T520 to 10-Stable (or Beta)
| 
| I'm running a Newcons kernel on my Thinkpad X220 now and it's working
| well, with a few minor quirks.  This is with the X-related packages
| rebuilt with WITH_NEW_XORG= and WITH_KMS= in /etc/make.conf.
| 
|  Are you booting directly to X or using startx from the console? In either
|  case, I think that i915kms should auto-load at the start of X, so you
|  should not need to pre-load it.
| 
| I log in and running startx.  i915kms does auto-load.
| 
|  Does newcons require VESA? If not, you should be able to remove it from
|  your kernel to get suspend/resume working from X.
| 
| It does not use VESA - it's removed from GENERIC on the newcons
| branch.  Suspend and resume generally works on my X220 now, from X or
| console.  There are some outstanding issues, for example the X display
| sometimes ends up corrupted after resume; stopping and restarting X
| fixes that.

Adding
vidcontrol -s 1  /dev/ttyv0
to /etc/rc.suspend and
vidcontrol -s 9  /dev/ttyv0
to /etc/rc.resume fixed that for me.

Doug A.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: newcons comming

[Doug Ambrisko]

On Fri, Oct 25, 2013 at 03:18:47PM +0300, Aleksandr Rybalko wrote:
| Hello fellow hackers!
| 
| I finally reach the point when I can work with newcons instead of
| syscons on my laptop. Yes, I know it still buggy and have a lot of
| style(9) problems. But we really have to get it into HEAD and 10.0 to
| enable shiny new Xorg features, drivers, etc.

I built the kernel from:
base/user/ed/newcons/
and installed it on my new ThinkPad T530 with Intel graphics chip.
In general it works better since now syspend and resume works with
and without X.  However, as with my prior ThinkPads I need to switch
out of X to suspend and resume.  So I have vidcontrol -s 1 in my
/etc/rc.suspend and vidcontrol -s 9 in /etc/rc.resume.  If I don't
the display ends up corrupted but somewhat working.

I had to kldload i915kms in /etc/rc.local since having it there at
boot via /boot/loader.conf resulted in the system not booting.  I need
i915kms for X so I don't need to use vesa mode.

The FreeBSD logo on boot is interesting but I'd prefer seeing 
FreeBSD boot messages to see what is happening.  I'm not sure if
there is a flag to disable that since I have played with it much.

However, on a whole it is much improved since with i915kms and
newcons when in X via vesa I couldn't switch to a non-X display
since it was blank.

Thanks,

Doug A.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Problem with r255775 include/mk-osreldate.sh

[Doug Ambrisko]

I don't know if others have run into this but I hit a problem with
include/mk-osreldate.sh.  It does a set -e to exit on commands failing
and sources in sys/conf/newvers.sh to get various things set.
In newvers.sh it does a bunch of
commmand
if [ $? -eq 0 ]; then
to decide what to do when it passes or fails.  Unfortunately, when
it fails due to the set -e it just exits and doesn't do the
else clause.  For me I check out a svn tree then build in a chroot.
In the chroot svn was failing then not creating a osreldate.h
resulting in the build dying.  This happened on two different machines
of which I use this method.

Removing the set -e in mk-osreldate.sh fixed my problem.  It should
probably be reworked to not depend on set -e and print errors when things
fail.  I guess newvers.sh could be reworked to do
if command ; then
which should pass set -e.

What do folks think?  It would be good to get this fixed before MFC
and before 10 is released.

Doug A.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Problem with r255775 include/mk-osreldate.sh

[Doug Ambrisko]

On Wed, Sep 25, 2013 at 12:16:47PM -0600, Ian Lepore wrote:
| On Wed, 2013-09-25 at 10:52 -0700, Doug Ambrisko wrote:
|  I don't know if others have run into this but I hit a problem with
|  include/mk-osreldate.sh.  It does a set -e to exit on commands failing
|  and sources in sys/conf/newvers.sh to get various things set.
|  In newvers.sh it does a bunch of
|  commmand
|  if [ $? -eq 0 ]; then
|  to decide what to do when it passes or fails.  Unfortunately, when
|  it fails due to the set -e it just exits and doesn't do the
|  else clause.  For me I check out a svn tree then build in a chroot.
|  In the chroot svn was failing then not creating a osreldate.h
|  resulting in the build dying.  This happened on two different machines
|  of which I use this method.
|  
|  Removing the set -e in mk-osreldate.sh fixed my problem.  It should
|  probably be reworked to not depend on set -e and print errors when things
|  fail.  I guess newvers.sh could be reworked to do
|  if command ; then
|  which should pass set -e.
|  
|  What do folks think?  It would be good to get this fixed before MFC
|  and before 10 is released.
| 
| For such a simple little change, this sure has been problematic.
| There are as many ways for it to fail as there are ways to arrange
| checkout-and-build workflows, apparently.
| 
| I've been mostly inclined to stay away from any big changes in
| newvers.sh for fear of breaking it when it's used in some way I'm not
| familiar with (such as building a release).  Sticking with that theory,
| I'd be inclined to leave it alone again, and not push the 'set -e'
| problem into its world, and instead do something like the attached.  

Yes, I'd be nervous to touch newvers.sh as well.
 
| My thinking is that newvers.sh does a variety of things, only some of
| which are germane to the needs of mk-osreldate.h, so have mk-osreldate
| check for just what it needs, and let newvers.sh take care of its
| internal errors however it likes.

Index: include/mk-osreldate.sh
===
--- include/mk-osreldate.sh (revision 255775)
+++ include/mk-osreldate.sh (working copy)
@@ -25,8 +25,6 @@
 #
 # $FreeBSD$
 
-set -e
-
 CURDIR=$(pwd)
 ECHO=${ECHO:=echo}
 
@@ -37,6 +35,12 @@ ${ECHO} creating osreldate.h from newvers.sh
 
 export PARAMFILE=${PARAM_H:=$CURDIR/../sys/sys/param.h}
 . ${NEWVERS_SH:=$CURDIR/../sys/conf/newvers.sh}
+
+if [ -z ${COPYRIGHT} -o -z ${RELDATE} ] ; then
+${ECHO} newvers.sh did not generate required information
+exit 1
+fi
+
 cat  $tmpfile EOF
 $COPYRIGHT
 #ifdef _KERNEL


Your patch worked for me.

Thanks,

Doug A.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Kernel output interleaved on boot

[Doug Ambrisko]

On Sun, Apr 07, 2013 at 09:52:37PM -0700, Jeremy Chadwick wrote:
| I have discussed this problem for years now -- over 5 years, to be
| exact.  As if I haven't sounded like a broken record before, I surely do
| now.  Start here, under section Kernel, item Scrambled or garbled
| kernel output:
| 
| https://wiki.freebsd.org/BugBusting/Commonly_reported_issues
| 
| The problem has not gone away.  It has not been solved.  It has not been
| worked around.  PRINTF_BUFR_SIZE does not solve the problem, and rarely
| helps relieve it.
| 
| I have discussed this issue more recently (2010) with John Baldwin as
| well:
| 
| http://lists.freebsd.org/pipermail/freebsd-questions/2010-March/214412.html
| http://lists.freebsd.org/pipermail/freebsd-questions/2010-March/214423.html
| 
| And in December 2011 too -- particularly an important read if you think
| increasing the number is a wise idea:
| 
| http://lists.freebsd.org/pipermail/freebsd-stable/2011-December/065158.html
| 
| Bottom line: there is no solution other than to switch OSes.

I have tweaked some kernel message generaters to be more line based.
We had issues with CAM attached devices print out parts separately
and then get intermixed a lot.  So I modified them to print them as
one line.  Then they didn't seem to get intermixed.  This doesn't solve
the overall problem, but if messages are done more this way then 
it helps.

Thanks,

Doug A.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: serial console not accepting input?

[Doug Ambrisko]

On Thu, Jan 24, 2013 at 01:23:50PM +, Eggert, Lars wrote:
| Hi,
| 
| On Jan 23, 2013, at 17:04, Dimitry Andric d...@freebsd.org wrote:
|  CTS/RTS hardware flow control, maybe?  E.g. add :hw to the default
|  settings in /etc/gettytab, or make a specific entry with an added :hw
|  setting.
| 
| nope, I don't even get a login prompt if I do that.
| 
|  If it is a physical serial console, you could also simply have a bad
|  cable.  Try swapping it with working system. :)
| 
| Spent the last few hours fiddling with the cabling and the various BIOS 
| serial redirection options (it's a Dell 2950). My best guess is that 
| the serial port on the box is physically broken.

Try to do a {Ctrl}D to see if works.  We've seen that the TX on reset
hangs but input works fine.  I'm not sure if we ran into this with
uart(4) but had a problem with sio(4).

Doug A.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Possible bug in NFSv4 with krb5p security?

[Doug Rabson]

This may be a stupid question but does the user 'elias' exist in the local
password database?

If you are using heimdal from the base distribution and you have source,
you should be able to build them with debug information which may help.
When I was writing gssd, I mostly ran it under gdb to debug problems like
this. To build something in the base with debug information, go to the
directory in the source tree for that component and type something like
'make DEBUG_FLAGS=-g clean all install'.


On 16 February 2013 09:24, Elias Mårtenson loke...@gmail.com wrote:

 OK, here I am replying to my own email. I just want to mention that I
 removed the ports version of Heimdal, but with no change in behaviour.


 On 16 February 2013 09:38, Elias Mårtenson loke...@gmail.com wrote:

 
  On 16 Feb, 2013 1:42 AM, Benjamin Kaduk ka...@mit.edu wrote:
  
   And yet one more thing: Heimdal ships with its own version of
  libgssapi. I
   can link gssd to it, but it won't run properly (it hangs pretty
 early).
  
   I have forgotten: you are using Heimdal from ports, not from the base
  system?  I remember it being easy to get into subtly-broken
 configurations
  when both a ports and a base version are present.
 
  I am indeed using Heimdal from ports. This machine is also the KDC. I
  wasn't aware that there was a non-ports version available.
 
  What do you suggest I do? Simply remove the one from ports? Do I have to
  do something to activate the other one?
 
  (I have a hard time checking this as I am nowhere near the computers now)
 
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Possible bug in NFSv4 with krb5p security?

[Doug Rabson]

On 16 February 2013 13:57, Elias Mårtenson loke...@gmail.com wrote:

 On 16 February 2013 18:58, Doug Rabson d...@rabson.org wrote:

 This may be a stupid question but does the user 'elias' exist in the
 local password database?

 If you are using heimdal from the base distribution and you have source,
 you should be able to build them with debug information which may help.
 When I was writing gssd, I mostly ran it under gdb to debug problems like
 this. To build something in the base with debug information, go to the
 directory in the source tree for that component and type something like
 'make DEBUG_FLAGS=-g clean all install'.


 No worries. I do have that user (and everything else, specifically single
 sign-on ssh) works with it. I do agree that if I had not that user, the
 behaviour I see would be neatly explained.

 When it comes to gssd, I've got its behaviour pretty well nailed down. It
 does what it's supposed to do.

 However, when I tried rebuilding libgssapi.so.10, I ended up with gssd
 hanging when it used the new library. I have no idea why.

 Would it be wise to upgrade from 9.1-RELEASE to something newer? I've seen
 references to 10-CURRENT. I'd like to be debugging the latest version of
 everything, but this machine also needs to serve as a fileserver for my
 home office, so some degree of stability is needed.


I don't think much (if anything) has changed with gssd between 9.1 and
current. When your gssd hangs, you can try to get a stack trace using gdb's
attach command.



 Regards,
 Elias

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Possible bug in NFSv4 with krb5p security?

[Doug Rabson]

On 16 February 2013 16:18, Elias Mårtenson loke...@gmail.com wrote:

 On 17 February 2013 00:03, Doug Rabson d...@rabson.org wrote:

 I don't think much (if anything) has changed with gssd between 9.1 and
 current. When your gssd hangs, you can try to get a stack trace using gdb's
 attach command.

 Fair enough. However, when it hangs, I have at least a 50% chance of
 hitting the gssd-realted kerberl panic. Should I apply the patch you gave
 me, or would you suggest an upgrade?


I think it was Rick that mentioned the patch. I would apply the patch and
rebuild your kernel in the interests of changing as little as possible
while debugging the original issue.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [head tinderbox] failure on arm/arm

[Doug Brewer]

Chuck Burns wrote:
On 11/9/2012 4:16 PM, Brett wrote:
  Message: 11
  Date: Fri, 9 Nov 2012 23:34:58 +0800
  From: Doug Brewer brewer.d...@gmail.com
  To: Adrian Chadd adr...@freebsd.org
  Cc: a...@freebsd.org, FreeBSD Tinderbox tinder...@freebsd.org,
 curr...@freebsd.org
  Subject: Re: [head tinderbox] failure on arm/arm
  Message-ID:
 cag0v13tpalmdpg-8rifcjjroxz948mqzjnn1yvqz4teybjz...@mail.gmail.com
  Content-Type: text/plain; charset=ISO-8859-1
 
  No offence, but how many times did you break the build? Could you please
  compile your code before committing next time? Thanks a lot!
 
 
  Just an observation: a few years ago when I got sick of Linux's headlong 
  rush development model, I subscribed to various BSD mailing lists to see 
  what else was out there. I considered FreeBSD at the time - there was a 
  neverending avalanche of [head tinderbox] failure messages. This told me 
  that I would be more likely to be running code written by people who knew 
  what they were doing if I went with Open, Net, or DragonflyBSD.
 
  I safely run OpenBSD-current on my main computer and it always works (I 
  think I have had 2-3 build problems in about 3 years, and they were all my 
  fault). At the moment, I only feel confident enough with FreeBSD-current to 
  run it on my unimportant torrent computer. This is 80% due to constant 
  build failures, and 20% due to invasive changes being introduced with 
  documentation/instructions scattered over many different pages and mailing 
  lists, e.g:
 
  http://wiki.freebsd.org/FrontPage?action=fullsearchcontext=180value=xorgtitlesearch=Titles
 
  http://wiki.freebsd.org/FrontPage?action=fullsearchcontext=180value=pkgngtitlesearch=Titles
 
  Hypothetical user: Is it WITHOUT_PKGNG= or WITHOUT_PKGNG=yes or 
  WITH_PKGNG=no today?
 
  I wonder how many other people that you never hear from feel the same, and 
  if some sort of x weeks commit freezeout should apply to the build 
  breakers. Cute pointy hats or whatever obviously have no effect.
 
  Rant over!
  ___
  freebsd-current@freebsd.org mailing list
  http://lists.freebsd.org/mailman/listinfo/freebsd-current
  To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
 
 Quite honestly, the head/current branch is going to have build
 failures.. It's the test bed..

YOU'VE GOTTA BE KIDDING ME.

 Stick with the release system unless you
 want cutting edge.. just remember.. cutting edge cuts sometimes...
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [head tinderbox] failure on arm/arm

[Doug Brewer]

Adrian Chadd wrote:
On 9 November 2012 14:37, Chuck Burns brea...@gmail.com wrote:

  Adrian. diskspace and cpu cycles are things I can spare, drop me a line
  outside of the ML and we can discuss particulars. It's just a personal
  box.. on a residential internet service, I have an amd64 box with 600G free
  on my pool.. 8G ram.. and I have a smaller i386 box... 100G or so free, 512M
  ram..  just drop me a line..

 Hi,

 Those I do have - I have access to all of the ref* boxes in the
 cluster. I'm just typically hacking on this stuff on the train or at a
 cafe, and I don't have a workflow setup for pushing out potential
 diffs to build machines that have all the grunt/disk space for each
 little change that I do.

Wait wait wait. It makes me wonder if you get the patch tested well
on the train or at a cafe before being committed.

 I'm sorry about breaking things from time to time, but besides a small
 handful of what was I thinking?! things, the build breaks are just
 that - build breaks. They're easily fixed.

I do not care how things are easily fixed. Remember, when Sam Leffler
was the maintainer of ath and CAMBRIA board (it's an embedded device, right?),
he had never broke the build.

 Adrian

BR,
Doug.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [head tinderbox] failure on arm/arm

[Doug Brewer]

No offence, but how many times did you break the build? Could you please
compile your code before committing next time? Thanks a lot!

On Fri, Nov 9, 2012 at 7:13 AM, Adrian Chadd adr...@freebsd.org wrote:
 Fixed, sorry!




 adrian

 On 8 November 2012 15:03, FreeBSD Tinderbox tinder...@freebsd.org wrote:
 TB --- 2012-11-08 21:30:00 - tinderbox 2.9 running on 
 freebsd-current.sentex.ca
 TB --- 2012-11-08 21:30:00 - FreeBSD freebsd-current.sentex.ca 
 8.3-PRERELEASE FreeBSD 8.3-PRERELEASE #0: Mon Mar 26 13:54:12 EDT 2012 
 d...@freebsd-current.sentex.ca:/usr/obj/usr/src/sys/GENERIC  amd64
 TB --- 2012-11-08 21:30:00 - starting HEAD tinderbox run for arm/arm
 TB --- 2012-11-08 21:30:00 - cleaning the object tree
 TB --- 2012-11-08 21:30:00 - checking out /src from 
 svn://svn.freebsd.org/base/head
 TB --- 2012-11-08 21:30:00 - cd /tinderbox/HEAD/arm/arm
 TB --- 2012-11-08 21:30:00 - /usr/local/bin/svn cleanup /src
 TB --- 2012-11-08 21:31:52 - /usr/local/bin/svn update /src
 TB --- 2012-11-08 21:32:01 - At svn revision 242793
 TB --- 2012-11-08 21:32:02 - building world
 TB --- 2012-11-08 21:32:02 - CROSS_BUILD_TESTING=YES
 TB --- 2012-11-08 21:32:02 - MAKEOBJDIRPREFIX=/obj
 TB --- 2012-11-08 21:32:02 - PATH=/usr/bin:/usr/sbin:/bin:/sbin
 TB --- 2012-11-08 21:32:02 - SRCCONF=/dev/null
 TB --- 2012-11-08 21:32:02 - TARGET=arm
 TB --- 2012-11-08 21:32:02 - TARGET_ARCH=arm
 TB --- 2012-11-08 21:32:02 - TZ=UTC
 TB --- 2012-11-08 21:32:02 - __MAKE_CONF=/dev/null
 TB --- 2012-11-08 21:32:02 - cd /src
 TB --- 2012-11-08 21:32:02 - /usr/bin/make -B buildworld
 Building an up-to-date make(1)
 World build started on Thu Nov  8 21:32:09 UTC 2012
 Rebuilding the temporary build tree
 stage 1.1: legacy release compatibility shims
 stage 1.2: bootstrap tools
 stage 2.1: cleaning up the object tree
 stage 2.2: rebuilding the object tree
 stage 2.3: build tools
 stage 3: cross tools
 stage 4.1: building includes
 stage 4.2: building libraries
 stage 4.3: make dependencies
 stage 4.4: building everything
 World build completed on Thu Nov  8 22:32:33 UTC 2012
 TB --- 2012-11-08 22:32:33 - cd /src/sys/arm/conf
 TB --- 2012-11-08 22:32:33 - /usr/sbin/config -m AC100
 TB --- 2012-11-08 22:32:33 - skipping AC100 kernel
 TB --- 2012-11-08 22:32:33 - cd /src/sys/arm/conf
 TB --- 2012-11-08 22:32:33 - /usr/sbin/config -m ARMADAXP
 TB --- 2012-11-08 22:32:33 - skipping ARMADAXP kernel
 TB --- 2012-11-08 22:32:33 - cd /src/sys/arm/conf
 TB --- 2012-11-08 22:32:33 - /usr/sbin/config -m ATMEL
 TB --- 2012-11-08 22:32:34 - building ATMEL kernel
 TB --- 2012-11-08 22:32:34 - CROSS_BUILD_TESTING=YES
 TB --- 2012-11-08 22:32:34 - MAKEOBJDIRPREFIX=/obj
 TB --- 2012-11-08 22:32:34 - PATH=/usr/bin:/usr/sbin:/bin:/sbin
 TB --- 2012-11-08 22:32:34 - SRCCONF=/dev/null
 TB --- 2012-11-08 22:32:34 - TARGET=arm
 TB --- 2012-11-08 22:32:34 - TARGET_ARCH=arm
 TB --- 2012-11-08 22:32:34 - TZ=UTC
 TB --- 2012-11-08 22:32:34 - __MAKE_CONF=/dev/null
 TB --- 2012-11-08 22:32:34 - cd /src
 TB --- 2012-11-08 22:32:34 - /usr/bin/make -B buildkernel KERNCONF=ATMEL
 Kernel build for ATMEL started on Thu Nov  8 22:32:34 UTC 2012
 stage 1: configuring the kernel
 stage 2.1: cleaning up the object tree
 stage 2.2: rebuilding the object tree
 stage 2.3: build tools
 stage 3.1: making dependencies
 stage 3.2: building everything
 Kernel build for ATMEL completed on Thu Nov  8 22:36:09 UTC 2012
 TB --- 2012-11-08 22:36:09 - cd /src/sys/arm/conf
 TB --- 2012-11-08 22:36:09 - /usr/sbin/config -m AVILA
 TB --- 2012-11-08 22:36:09 - skipping AVILA kernel
 TB --- 2012-11-08 22:36:09 - cd /src/sys/arm/conf
 TB --- 2012-11-08 22:36:09 - /usr/sbin/config -m BEAGLEBONE
 TB --- 2012-11-08 22:36:09 - skipping BEAGLEBONE kernel
 TB --- 2012-11-08 22:36:09 - cd /src/sys/arm/conf
 TB --- 2012-11-08 22:36:09 - /usr/sbin/config -m BWCT
 TB --- 2012-11-08 22:36:09 - building BWCT kernel
 TB --- 2012-11-08 22:36:09 - CROSS_BUILD_TESTING=YES
 TB --- 2012-11-08 22:36:09 - MAKEOBJDIRPREFIX=/obj
 TB --- 2012-11-08 22:36:09 - PATH=/usr/bin:/usr/sbin:/bin:/sbin
 TB --- 2012-11-08 22:36:09 - SRCCONF=/dev/null
 TB --- 2012-11-08 22:36:09 - TARGET=arm
 TB --- 2012-11-08 22:36:09 - TARGET_ARCH=arm
 TB --- 2012-11-08 22:36:09 - TZ=UTC
 TB --- 2012-11-08 22:36:09 - __MAKE_CONF=/dev/null
 TB --- 2012-11-08 22:36:09 - cd /src
 TB --- 2012-11-08 22:36:09 - /usr/bin/make -B buildkernel KERNCONF=BWCT
 Kernel build for BWCT started on Thu Nov  8 22:36:09 UTC 2012
 stage 1: configuring the kernel
 stage 2.1: cleaning up the object tree
 stage 2.2: rebuilding the object tree
 stage 2.3: build tools
 stage 3.1: making dependencies
 stage 3.2: building everything
 Kernel build for BWCT completed on Thu Nov  8 22:38:19 UTC 2012
 TB --- 2012-11-08 22:38:19 - cd /src/sys/arm/conf
 TB --- 2012-11-08 22:38:19 - /usr/sbin/config -m CAMBRIA
 TB --- 2012-11-08 22:38:19 - skipping CAMBRIA kernel
 TB --- 2012-11-08 22:38:19 - cd /src/sys/arm/conf
 TB --- 2012-11-08 22:38:19 - /usr/sbin/config -m CNS11XXNAS
 

Re: boot2/loader: serial port handling

[Doug Ambrisko]

On Fri, Oct 12, 2012 at 05:13:47PM -0700, Garrett Cooper wrote:
| On Fri, Oct 12, 2012 at 5:09 PM, Xin Li delp...@delphij.net wrote:
|  -BEGIN PGP SIGNED MESSAGE-
|  Hash: SHA256
| 
| ...
| 
|  Ah I wish I am not this far behind my email backlog.  Yes I think
|  these (241300 and 241301) will solve the problem.
| 
| Yeah -- forgot about the other one. There's another enhancement
| that would make this even better (apart from maybe having multiple
| primary consoles): setting the primary console if present and having
| fallbacks in the event that the original primary wasn't set or
| configurable; it was a thing that was present in another project I
| worked on with sio that was pretty slick (and I think that there would
| be some parties who wouldn't mind if the same was done with uart(4)).

This concept was objected to when I checked it into sio(4) so I had
to back it out.  Some liked it.  I have ported it to uart(4) since we
need that functionality when we moved to a newer FreeBSD.

Doug A.
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: squealing/whistling audio

[Doug Barton]

On 09/18/2012 23:17, matt wrote:
 On 09/18/12 23:00, Doug Barton wrote:
 On 9/18/2012 10:56 PM, matt wrote:
 On 09/18/12 18:01, Doug Barton wrote:
 Sometime in the last couple of months an old problem has resurfaced on
 HEAD, a sort of squealing/whistling sound in the audio, even without
 anything playing. The sound is similar to the wind whistling through
 something.

 Before I blindly go off on a bisecting spree, does anyone have a
 suggestion as to where I might look?

 Doug


 Electrically that's usually oscillation (too high gain or too much
 electrical feedback) or an unshielded input.
 Sorry, I should have been more clear. This problem doesn't occur in
 windows, or linux, and last time I checked, didn't occur in freebsd 8. I
 have everything irrlevant zeroed out in mixer.

 Doug

 It was worth a shot. It is possible that each OS (and version) is
 setting the associations up properly though except HEAD on your machine.
 Is it a laptop or a desktop board?

Desktop.

 Any difference in the nid configuration between freebsd 8 and HEAD?

No.

 Does changing any of the hw.snd sysctls (latency, exact rate, vpc_0db)
 have any effect on the sound?

It doesn't affect the squeal.

 http://freshbsd.org/commit/freebsd/r230551 might be worth a look.

Didn't help.

 I
 couldn't find anything newer that looked like it would have an effect.
 Their are some earlier commits around January that are dealing with
 signal gain that could also have an effect.
 
 Otherwise, I'm not sure where else to look.

Thanks anyway.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: squealing/whistling audio

[Doug Barton]

On 9/18/2012 10:56 PM, matt wrote:
 On 09/18/12 18:01, Doug Barton wrote:
 Sometime in the last couple of months an old problem has resurfaced on
 HEAD, a sort of squealing/whistling sound in the audio, even without
 anything playing. The sound is similar to the wind whistling through
 something.

 Before I blindly go off on a bisecting spree, does anyone have a
 suggestion as to where I might look?

 Doug


 Electrically that's usually oscillation (too high gain or too much
 electrical feedback) or an unshielded input.

Sorry, I should have been more clear. This problem doesn't occur in
windows, or linux, and last time I checked, didn't occur in freebsd 8. I
have everything irrlevant zeroed out in mixer.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

squealing/whistling audio

[Doug Barton]

Sometime in the last couple of months an old problem has resurfaced on
HEAD, a sort of squealing/whistling sound in the audio, even without
anything playing. The sound is similar to the wind whistling through
something.

Before I blindly go off on a bisecting spree, does anyone have a
suggestion as to where I might look?

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Shouldn't world be able to build without /usr/include?

[Doug Barton]

=== tools/build (obj,includes,depend,all,install)
grep: /usr/include/stdio.h: No such file or directory
/usr/obj/frontier/svn/head/tmp/frontier/svn/head/tools/build created for
/frontier/svn/head/tools/build
grep: /usr/include/stdio.h: No such file or directory
cd /frontier/svn/head/tools/build; make buildincludes; make installincludes
grep: /usr/include/stdio.h: No such file or directory
grep: /usr/include/stdio.h: No such file or directory
grep: /usr/include/stdio.h: No such file or directory
grep -v HAVE_GETLINE
/frontier/svn/head/tools/build/../../lib/libmagic/config.h  config.h
rm -f .depend
mkdep -f .depend -a-DHAVE_CONFIG_H -I.
-I/usr/obj/frontier/svn/head/tmp/legacy/usr/include -std=gnu99
/frontier/svn/head/tools/build/../../contrib/file/getline.c
In file included from
/frontier/svn/head/tools/build/../../contrib/file/getline.c:32:
/frontier/svn/head/tools/build/../../contrib/file/file.h:52:74: error:
stdio.h: No such file or directory
/frontier/svn/head/tools/build/../../contrib/file/file.h:53:19: error:
errno.h: No such file or directory
/frontier/svn/head/tools/build/../../contrib/file/file.h:54:44: error:
fcntl.h: No such file or directory
/frontier/svn/head/tools/build/../../contrib/file/file.h:59:20: error:
stdint.h: No such file or directory
/frontier/svn/head/tools/build/../../contrib/file/file.h:62:22: error:
inttypes.h: No such file or directory
/frontier/svn/head/tools/build/../../contrib/file/file.h:64:19: error:
regex.h: No such file or directory
/frontier/svn/head/tools/build/../../contrib/file/file.h:65:23: error:
sys/types.h: No such file or directory
/frontier/svn/head/tools/build/../../contrib/file/file.h:66:23: error:
sys/param.h: No such file or directory
/frontier/svn/head/tools/build/../../contrib/file/file.h:68:22: error:
sys/stat.h: No such file or directory
/frontier/svn/head/tools/build/../../contrib/file/file.h:69:20: error:
stdarg.h: No such file or directory
/frontier/svn/head/tools/build/../../contrib/file/getline.c:34:20:
error: stdlib.h: No such file or directory
/frontier/svn/head/tools/build/../../contrib/file/getline.c:36:20:
error: unistd.h: No such file or directory
/frontier/svn/head/tools/build/../../contrib/file/getline.c:38:20:
error: string.h: No such file or directory
mkdep: compile failed
*** [.depend] Error code 1



-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: underexposed snapshots

[Doug Barton]

On 09/14/2012 23:43, Randy Bush wrote:
 ftp://ftp.FreeBSD.org/pub/FreeBSD/ISO-IMAGES-arch is a bit empty.  i
 guess things are moving around.  any idea where i can get the latest
 tag=.

I and others have brought up this issue repeatedly over the last couple
of years, and the PTB have decided that since allbsd is doing it for us,
we don't need to put any effort into making it happen ourselves. That in
spite of the fact that numerous volunteers have come forward willing to
help.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Clang as default compiler November 4th

[Doug Barton]

On 09/11/2012 02:52 AM, Erik Cederstrand wrote:
 So can we do a sweep on the ports tree and mark the 2232 ports with 
 USE_GCC=4.2 until they can actually build with clang?

Unfortunately it isn't that simple. We already have a statistically
significant number of ports that don't even compile with gcc 4.2.1. How
many compilers do we expect the users to install? :)

What we need to do is what I and others have been asking to do for
years. We need to designate a modern version of gcc (no less than 4.6)
as the official default ports compiler, and rework whatever is needed to
support this. Fortunately, that goal is much more easily achieved than
fixing ports to build and run with clang. (It's harder than it sounds
because there are certain key libs that define some paths depending on
what compiler they were built with, but still easier than dealing with
clang in the short term.)

Once that is done, the compiler in the base is an afterthought, and we
can do away with gcc in the base altogether much more easily. Users who
want to help support building ports with clang can continue to do so.

Doug
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Clang as default compiler November 4th

[Doug Barton]

On 09/11/2012 11:15 PM, Mark Linimon wrote:
 On Tue, Sep 11, 2012 at 11:27:50AM +0200, Lars Engels wrote:
 At the moment the ports maintainers don't give much about if their ports
 build with CLANG or not because they're not forced to.
 
 I think this is a mis-representation.
 
 Adding the requirement your ports must work on clang is adding an
 ex-post-facto requirement.  This creates the following matrix of what
 we are implicitly asking maintainers to do:
 
 (FreeBSD 7|8|9|10) * (amd64|arm|i386|powerpc|sparc64) * (base gcc|base clang)
 
 It is completely insane to expect anyone to be able to test in all of those
 environments, or even a tiny subset of them.  This isn't what most people
 sign up for when they sign up to maintain ports.
 
 Those who don't run CURRENT won't notice, but those who do will have to
 get their butts up and fix the ports
 
 I think it's foolish to assume that maintainres don't have their butts in
 gear as it is.  Please note, we have nearly 1300 PRs, hundreds of ports with
 build errors and/or PRs, and hundreds that fail on -current only.  I try to
 advertise all these things the best I know how.  Adding the hundreds that
 fail on -clang only and then blaming the maintainers is simply going to be
 counter-productive.

Write the day on your calendars folks, I completely agree with what Mark
said above. :) This is a big part of what I meant with some of my more
colorful comments in my original post on this topic.

Doug

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Clang as default compiler November 4th

[Doug Barton]

On 09/11/2012 05:03 AM, Steve Kargl wrote:
 On Tue, Sep 11, 2012 at 04:10:13PM +0200, Dimitry Andric wrote:

 However, I think the majority of users can get by just fine using clang,
 right now.  Doug Barton even confirmed in this thread that 80% of our
 ports already work with it!
 
 He stated that 80% build with clang.  I doubt that he actually
 tested the functionality of some 17000 ports.

Correct.

Also, users who actually are helping with testing clang for ports
continue to report runtime problems, even with things that build fine.

Doug

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Clang as default compiler November 4th

[Doug Barton]

On 9/12/2012 12:40 AM, Erik Cederstrand wrote:
 Den 12/09/2012 kl. 11.29 skrev Doug Barton do...@freebsd.org:
 
 On 09/11/2012 02:52 AM, Erik Cederstrand wrote:
 So can we do a sweep on the ports tree and mark the 2232 ports
 with USE_GCC=4.2 until they can actually build with clang?
 
 Unfortunately it isn't that simple. We already have a
 statistically significant number of ports that don't even compile
 with gcc 4.2.1. How many compilers do we expect the users to
 install? :)
 
 If a port doesn't compile with the default compiler in base, I expect
 that port to add a build dependency on the compiler that it actually
 does compiles with.

Yes, they do this now. The problem is that the set is growing, and the
rate of growth is increasing.

 Of course, I hope to not have 6 different
 compilers installed on my system, but the list of build or runtime
 dependencies are at the discretion of the port (maintainer). As you
 (I think) said, we can't force port maintainers to patch their ports
 to support clang.

Those are unrelated issues. Please re-read the bits of my post that you
snipped. The overwhelming majority of problems we have with compiling
ports now would be fixed by having a modern version of gcc as the
official (i.e., supported) ports compiler. The clang efforts would be
a parallel track.

Doug

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Clang as default compiler November 4th

[Doug Barton]

On 9/12/2012 1:49 AM, David Chisnall wrote:
 On 12 Sep 2012, at 10:09, Doug Barton wrote:
 
 Also, users who actually are helping with testing clang for ports
 continue to report runtime problems, even with things that build fine.
 
 I hope that you are encouraging maintainers of ports that don't work as 
 expected with clang to submit bug reports upstream.  We can't fix bugs if we 
 aren't made aware of them.

I personally am not directly involved in this effort (other than for my
own ports), but from what I've seen the classical emphasis on pushing
bug reports upstream has been applied in this area as well.

hth,

Doug
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Clang as default compiler November 4th

[Doug Barton]

On 09/11/2012 02:27 AM, Lars Engels wrote:
 On Mon, Sep 10, 2012 at 10:54:04PM -0700, Doug Barton wrote:
 As of last week, 4,680 ports out of 23,857 failed to build with clang on
 9-amd64. That's almost a 20% failure rate. Until we have better support
 for either building ports with clang, or have better support for the
 idea of a ports compiler, this change is premature. The ports are an
 important part of the FreeBSD Operating _System_, and pulling the
 trigger on the default compiler before the ports problems are addressed
 robustly seems like a big fat FU.

 That said, I agree that this issue needs to be addressed. In fact, 9
 months before the release of 9.0 I said on the internal committers list
 that there was no point in making a new release until we had thoroughly
 addressed both the default compiler for the base, and resolving the
 ports compiler issue. While there has been some movement on the
 former, there has been nothing done on the latter for years now, even
 though everyone agrees that it is an important issue.

 I'd like to request that rather than moving the default compiler
 prematurely that you call for volunteers to address the problems with
 the ports. Both the issues of fixing more ports to build correctly with
 clang, and the issue of defining a ports compiler version of gcc (and
 appropriate infrastructure) for those that can't. Once those issues are
 resolved there would not be any further obstacles to moving the default.
 Until they are, the change is premature.

 Doug
 
 Doug, as you can already use CLANG instead of GCC now, you will be able
 to use GCC instead of CLANG after November 4th.

There's lots of things I _can_ do, what we're discussing is what the
defaults should be.

 At the moment the ports maintainers don't give much about if their ports
 build with CLANG or not

Do you follow ports development? At all? There have been extensive
efforts over the last several years to get more ports compiling with
clang. The problem is that things like the c89 issue don't percolate
down, and we don't have a concerted effort from all of the relevant
parties to improve the issue.

Fixing the problem of getting the right eyeballs on the things that need
fixing won't be improved by switching the default before they are fixed.
In fact, it's likely to make the people who are src-centric now even
less likely to help because their work will be done.

 Those who don't run CURRENT won't notice, but those who do will have to
 get their butts up and fix the ports, so 10.0 can have 99% of all ports
 build with CLANG and even 8.x and 9.x can already profit from having the
 broken ports fixed now.

Yeah, and I'm going to get a pony out of this deal, right? :)

You completely misunderstand the nature of the problem, therefore your
proposed solution isn't going to solve it.

Doug

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Clang as default compiler November 4th

[Doug Barton]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

As of last week, 4,680 ports out of 23,857 failed to build with clang on
9-amd64. That's almost a 20% failure rate. Until we have better support
for either building ports with clang, or have better support for the
idea of a ports compiler, this change is premature. The ports are an
important part of the FreeBSD Operating _System_, and pulling the
trigger on the default compiler before the ports problems are addressed
robustly seems like a big fat FU.

That said, I agree that this issue needs to be addressed. In fact, 9
months before the release of 9.0 I said on the internal committers list
that there was no point in making a new release until we had thoroughly
addressed both the default compiler for the base, and resolving the
ports compiler issue. While there has been some movement on the
former, there has been nothing done on the latter for years now, even
though everyone agrees that it is an important issue.

I'd like to request that rather than moving the default compiler
prematurely that you call for volunteers to address the problems with
the ports. Both the issues of fixing more ports to build correctly with
clang, and the issue of defining a ports compiler version of gcc (and
appropriate infrastructure) for those that can't. Once those issues are
resolved there would not be any further obstacles to moving the default.
Until they are, the change is premature.

Doug


On 09/10/2012 14:12, Brooks Davis wrote:
 [Please confine your replies to toolch...@freebsd.org to keep the thread
 on the most relevant list.]
 
 For the past several years we've been working towards migrating from
 GCC to Clang/LLVM as our default compiler.  We intend to ship FreeBSD
 10.0 with Clang as the default compiler on i386 and amd64 platforms.  To
 this end, we will make WITH_CLANG_IS_CC the default on i386 and amd64
 platforms on November 4th.
 
 What does the mean to you?
 
  * When you build world after the default is changed /usr/bin/cc, cpp, and
c++ will be links to clang.
 
  * This means the initial phase of buildworld and old style kernel
compilation will use clang instead of gcc.  This is known to work.
 
  * It also means that ports will build with clang by default.  A major
of ports work, but a significant number are broken or blocked by
broken ports. For more information see:
  http://wiki.freebsd.org/PortsAndClang
 
 What issues remain?
 
  * The gcc-clang transition currently requires setting CC, CXX, and CPP
in addition to WITH_CLANG_IS_CC.  I will post a patch to toolchain@
to address this shortly.
 
  * Ports compiler selection infrastructure is still under development.
 
  * Some ports could build with clang with appropriate tweaks.
 
 What can you do to help?
 
  * Switch (some of) your systems.  Early adoption can help us find bugs.
 
  * Fix ports to build with clang.  If you don't have a clang system, you
can use the CLANG/amd64 or CLANG/i386 build environments on
redports.org.
 
 tl;dr: Clang will become the default compiler for x86 architectures on 
 2012-11-04
 
 -- Brooks
 


- -- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.19 (FreeBSD)

iQEcBAEBCAAGBQJQTtH8AAoJEFzGhvEaGryEU3gIAJ3X2EHDCVnkC/CYTMOkceho
KS6qVcQK4OCbbG+8TKkjrHNdiBO7ZuJKxfvr/TZC1zNKc8wYBlWo3s07wCHmu8Nj
OP8UwTMKumnljnYlRanQiLO9iAZKwGfI2gdxJTb5YABN2StRMXnD17Yyic6pw090
7l+cQw3iJAI8vbO4su33HJOhru0o4XLodbazHXFc6RjabAfXfuk1W6V0PfAodVC9
ZUGbF4WA7F0sJOEVuohmSk8ICHQRzTWofpdvCTlhHc1XYTaQ9u/dLGUp1C8g/BUG
CJQua7wsBdf4VgsvlYBxTAOEpURqot0Ild7zQL+9vZtf7cGCsfalpwBWzQ9J/Wk=
=gRkN
-END PGP SIGNATURE-
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

On 09/01/2012 23:01, Matthew Seaman wrote:
 As rebuilding the repo database is something you'ld do routinely anyhow
 as part of normal maintenance

Errr ... what? Why would this be true? Doesn't pkg keep the repo
database up to date as it's making changes?

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

On 09/01/2012 12:59, Garrett Cooper wrote:
 Again, this is part of the reason why I suggested multiple release
 trains. Although it's more painful for bapt@, et all, it's ultimately
 what would need to be done in order for pkgng to be packaged with a
 release or set of releases.

Garrett,

I think you're seriously misunderstanding how pkg is going to work.

One thing we desperately want to get away from is tying ports stuff to
specific base releases. Can you please be more clear as to why you keep
trying to pull things back in the wrong direction?

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Can't build FreeBSD-head with CLANG

[Doug Barton]

On 08/30/2012 09:16, Dimitry Andric wrote:
 [Note that linking GPL-contaminated code into your
 kernel proper is, shall we say, ideologically impure ;-)  But that is
 not the issue here.]

Can you keep this kind of stuff to -chat please? The more we deal with
the technical aspects the better off we will all be.

I for one put ext2fs in my kernel config because I have critical stuff
on those file systems and I both do want the speed boost and don't want
to worry about what's going to happen when I boot a new kernel.

Tools, not policy.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

On 08/30/2012 07:32 AM, John Baldwin wrote:
 On Thursday, August 30, 2012 1:10:24 pm Chris Rees wrote:
 On 30 Aug 2012 18:03, John Baldwin j...@freebsd.org wrote:

 On Thursday, August 30, 2012 10:39:17 am Tijl Coosemans wrote:
 On 27-08-2012 18:24, John Baldwin wrote:
 On Sunday, August 26, 2012 4:37:53 pm Doug Barton wrote:
 The problem is that we don't really support the idea of things in the
 base magically deleting themselves.

 As I have said in previous messages, the bootstrapping problem is being
 overblown by several orders of magnitude. For newly installed systems
 where pkg is the default, /usr/local/bin/pkg will be installed. So there
 is no bootstrapping problem.

 For already-installed systems who wish to switch to pkg, they can
 install from /usr/ports, or use the pkg bootstrap tool in the base.
 Given that they will be intentionally making this change, and there will
 be instructions written up on how to do this which include the
 bootstrapping step, once again this is a non-issue.

 The whole idea of having every call to /usr/local/sbin/pkg pass through
 /usr/sbin/pkg in order to help a tiny minority of users with a one-time
 bootstrapping issue is just plain ludicrous.

 I agree.  Even if we keep /usr/sbin/pkg, we will presumably want to remove
 it from the base in a year or so via 'make delete-old', etc.  Given that,
 I'm not sure we need it there in the first place.

 What if you pkg_delete \* or rm -rf /usr/local? Do you have to reboot
 pkg then?

 Yes, if we've decided it (pkgng) should not be part of the base.  This
 doesn't strike me as that weird.  It seems similar to how one has to
 bootstrap, say, MacPorts.

 Difference is, MacPorts isn't the official Mac distribution centre.

 Leaving out pkg-bootstrap (or whatever) is marginalising ports as a
 non-integral part of the OS.
 
 *sigh*  I sadly expected an emotional red herring reply such as this.
 
 This has nothing to do with marginalising ports.  Prior to this it has been
 a key argument and point that pkg* should _not_ be tied to the base system as
 that limits flexibility in the pkg tools.  I completely agree with that
 argument and having /usr/sbin/pkg doesn't appear to be consistent with that.
 
 For example, we've already shipped a binary in 9.1 release that has a
 hardcoded URL of http://pkgbeta.FreeBSD.org;.  So now you are stuck keeping
 that URL around for the next N years, albeit pointing to the production
 (not-beta) repository.  You can't safely reuse pkgbeta.FreeBSD.org for 
 anything
 until 9.1 is EOL'd.  And you'd have to change that before 9.2 and 10.0 if you
 want to avoid being in the same boat for even longer.  That is directly 
 contrary
 to the goal of having pkg* not being tied to the base.  A much more flexible
 and scalable approach would be for each pkg repository to include a 
 binary/script
 whatever that you can make available at a URL (which is easily changeable in
 documentation on our website) that when you run self-extracts and bootstraps
 pkgng.  (The pkg-static stuff is already basically this AFAICT.)
 
 If you wish to support existing users of, say, 8.2 or 8.3 release then you 
 need
 something like this anyway.  Also, as a downstream consumer who plans to use
 a custom pkgng repository on top of a modified FreeBSD distribution, this 
 approach
 is less failure prone (i.e. if someone runs 'pkg' and it tries to download 
 something
 from some hardcoded URL that's completely wrong).

I agree with John on all counts here. Further, the idea of a
self-installing package, at least for the pkg stuff itself, addresses
the issue that someone else brought up about how to handle installation
of pkg by the installer for a new system.

For example it's pretty common in the Linux world to have a package
which is wrapped in a shell script which unpacks the tarball, verifies
it with a digital signature, then installs the bits from the tarball
where they need to go. Since pkg brings a lot of the pieces of this to
the party already, it wouldn't be hard to add the rest.

... and please feel free to insert your favorite version of my We have
to get away from the idea that something is only good/cool/really part
of FreeBSD if it's in the base rant here. :)

Doug

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

On 08/25/2012 02:49, Julien Laffaye wrote:
 True. But when you create jails without the installer, you have to
 install pkgng by hand. 

Just like all the other ports you have to install in a jail.


-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

On 08/26/2012 11:37, Baptiste Daroussin wrote:
 On Sun, Aug 26, 2012 at 11:34:08AM -0700, Doug Barton wrote:
 On 08/25/2012 02:49, Julien Laffaye wrote:
 True. But when you create jails without the installer, you have
 to install pkgng by hand.
 
 Just like all the other ports you have to install in a jail.
 
 We are speaking about binary only packages, not ports.

Um, duh. I have a bad habit of using the terms interchangeably, sorry
if I caused confusion. Doesn't change my actual point though.


-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

On 08/26/2012 05:58, Baptiste Daroussin wrote:

 The is the longer plan but this with also true with pkg_add -r, and the pkg
 bootstrap may it be pkg-bootstrap or /usr/sbin/pkg. We have been discussing 
 with
 Security officers and we are waiting for the plan being written and setup by
 them, so we can improved security in both pkgng and the bootstrap. This should
 have happen in BSDCan, but lack of time from everyone, didn't made it happen, 
 we
 are now aiming at Cambridge DevSummit for that.

It would be nice if this were in place before 10-current shifted to pkg
by default in order to limit the number of times that we have to start
testing over from scratch.

 Given that such a security issue is already in with the current pkg_* tools, 
 it
 was accepting that we can still go that way until the policy is written, given
 that the final goal is to have the pkgng package checked against a signature.

This isn't the security issue I was talking about by having sbin/pkg
pass every command line to local/sbin/pkg.

You keep saying that you have no objections to changing the name. I am
asking you to do that. I don't care if it is pkg-bootstrap or something
else you like better. But please change the name to not be pkg, and
limit the functionality of the tool to bootstrapping the pkg package.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

On 08/26/2012 11:58, Baptiste Daroussin wrote:
 On Sun, Aug 26, 2012 at 11:39:07AM -0700, Doug Barton wrote:
 On 08/26/2012 05:58, Baptiste Daroussin wrote:

 The is the longer plan but this with also true with pkg_add -r, and the pkg
 bootstrap may it be pkg-bootstrap or /usr/sbin/pkg. We have been discussing 
 with
 Security officers and we are waiting for the plan being written and setup by
 them, so we can improved security in both pkgng and the bootstrap. This 
 should
 have happen in BSDCan, but lack of time from everyone, didn't made it 
 happen, we
 are now aiming at Cambridge DevSummit for that.

 It would be nice if this were in place before 10-current shifted to pkg
 by default in order to limit the number of times that we have to start
 testing over from scratch.

 Given that such a security issue is already in with the current pkg_* 
 tools, it
 was accepting that we can still go that way until the policy is written, 
 given
 that the final goal is to have the pkgng package checked against a 
 signature.

 This isn't the security issue I was talking about by having sbin/pkg
 pass every command line to local/sbin/pkg.

 You keep saying that you have no objections to changing the name. I am
 asking you to do that. I don't care if it is pkg-bootstrap or something
 else you like better. But please change the name to not be pkg, and
 limit the functionality of the tool to bootstrapping the pkg package.

 
 I received more feedback about keep pkg

As far as I could tell the people who responded that way don't seem to
be aware that every command to /usr/local/sbin/pkg is going to pass
through /usr/sbin/pkg. On its face, that is a bad idea for many reasons,
not the least of which is that it adds complexity where that complexity
does not need to be. The larger problem with that approach is that gives
an attacker 2 places to compromise the package installation process
instead of just 1. This becomes even more important if the pkg bootstrap
tool is the place that the public key for the digital signature is located.

 and changing it to
 pkg-bootstrap, so what should I do, changing it because you are asking for it?

A) You said you had no objections to changing it
B) I'm not the only one asking

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

On 08/26/2012 12:08, Ian Lepore wrote:
 Would this get better if the bootstrap tool were named pkg and were
 installed on a fresh system at /usr/local/sbin, so that it in effect
 replaces itself with the real thing, and has no need to leave a
 forwarding stub in /usr/sbin ?
 
 Maybe it could rename itself to /usr/local/sbin/pkg-bootstrap as part of
 replacing itself, so that you could re-bootstrap your way out of a
 problem later.

That's certainly creative thinking, but I'm still queasy about 2
commands with the same name that do 2 different things. And having it
rename itself adds to the confusion down the road.

Having a simple pkg bootstrapping tool in the base is a good idea. But
the functionality needs to be extremely limited so that we don't
increase the security exposure; and so that we don't end up in a
situation where a bug fix for something in the base limits our ability
to innovate with pkg in the ports tree.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

On 08/26/2012 13:02, namor wrote:
 On Thu, Aug 23, 2012 at 03:28:27PM -0700, Doug Barton wrote:
 On 8/23/2012 3:19 PM, Steve Wills wrote:
 Hi,

 It seems to me that renaming the pkg binary in /usr/sbin/pkg to 
 /usr/sbin/pkg-bootstrap would make sense. From a user standpoint, it is 
 confusing that running the command gets different results the second time 
 it is run vs. the first time. I can imagine a user saying I ran pkg, but 
 it didn't do what they said it would.  Now I run it again, and it does do 
 what it is supposed to. Also, it would enable setting up a pkg-bootstrap 
 man page separate from the pkg man page, without confusion about which one 
 you're looking at.

 So, opinions? There may still be time to fix it for 9.1 if we can decide 
 quickly.

 Yes please.

 Every time in the past that we have talked about moving the pkg_* tools
 to the ports the corresponding change for the base was to have a
 pkg_bootstrap tool that was a use once and forget kind of thing. I was
 quite surprised when sbin/pkg was added, but since people tell me I
 already comment on too much, I decided to wait and see what others thought.
 
 If I understand correctly, the main concern of the pkg-name fraction
 is to not confuse newbies. All you write is pkg install foo and pkg
 will bootstrap itself if not installed. You don't have to call
 pkg-bootstrap first (how would you know about it anyways? read pkg(8)?)
 
  - How about his: stick with /usr/sbin/pkg-boostrap
  - cat  /usr/sbin/pkg  EOF
 #!/bin/sh
 echo To use pkg you have to bootstrap the pkgng installation first,
 please call /usr/sbin/pkg-bootstrap
 EOF
 
  - pkg-debootstrap replaces/removes /usr/sbin/pkg messenger (above) after
successful installation

Again, creative thinking, so you get points for that. :)

The problem is that we don't really support the idea of things in the
base magically deleting themselves.

As I have said in previous messages, the bootstrapping problem is being
overblown by several orders of magnitude. For newly installed systems
where pkg is the default, /usr/local/bin/pkg will be installed. So there
is no bootstrapping problem.

For already-installed systems who wish to switch to pkg, they can
install from /usr/ports, or use the pkg bootstrap tool in the base.
Given that they will be intentionally making this change, and there will
be instructions written up on how to do this which include the
bootstrapping step, once again this is a non-issue.

The whole idea of having every call to /usr/local/sbin/pkg pass through
/usr/sbin/pkg in order to help a tiny minority of users with a one-time
bootstrapping issue is just plain ludicrous.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

On 08/26/2012 13:35, Warren Block wrote:
 On Sun, 26 Aug 2012, Ian Lepore wrote:
 
 On Sun, 2012-08-26 at 20:58 +0200, Baptiste Daroussin wrote:
 On Sun, Aug 26, 2012 at 11:39:07AM -0700, Doug Barton wrote:
 On 08/26/2012 05:58, Baptiste Daroussin wrote:
 This isn't the security issue I was talking about by having sbin/pkg
 pass every command line to local/sbin/pkg.

 You keep saying that you have no objections to changing the name. I am
 asking you to do that. I don't care if it is pkg-bootstrap or something
 else you like better. But please change the name to not be pkg, and
 limit the functionality of the tool to bootstrapping the pkg package.


 I received more feedback about keep pkg and changing it to
 pkg-bootstrap, so what should I do, changing it because you are
 asking for it?

 Would this get better if the bootstrap tool were named pkg and were
 installed on a fresh system at /usr/local/sbin, so that it in effect
 replaces itself with the real thing, and has no need to leave a
 forwarding stub in /usr/sbin ?

 Maybe it could rename itself to /usr/local/sbin/pkg-bootstrap as part of
 replacing itself, so that you could re-bootstrap your way out of a
 problem later.
 
 Ew.  But on a similar note, an idea I just had in IRC is to have pkgng
 overwrite the base /usr/bin/pkg with a link to /usr/local/bin/pkg.
 That effectively removes that binary.  We do have precedent for ports
 overwriting base with sendmail and openssl.

... and bind, but that's a whole different category of problems.

 Hmmm, might have to be careful that future updates don't replace the
 real thing with a newer bootstrap program.
 
 Yes.  A link could be detected by installworld and not overwritten...
 although that's a hack.

Like you said above, Ew. :)

There really is no need to be so clever here. The bootstrapping issue is
going to be a minor annoyance that affects a small percentage of our users.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

On 8/23/2012 8:03 PM, Eitan Adler wrote:
 On 23 August 2012 22:59, Doug Barton do...@freebsd.org wrote:

 I tend to agree with Steve here ... we can't be responsible for other
 people's poorly written docs.
 
 This isn't about poorly written docs. This is the user expecting a
 tool to exist, which doesn't. Take another example of a sysadmin which
 rarely installs new systems, installs FreeBSD for the third time, and
 then gets confused when pkg install vim fails.

Aren't we going to install the pkg package on new systems when they are
installed? Isn't that (shouldn't that be?) part of the project plan? It
would be insane for us not to do that, at least for the releases where
pkg is the default.

 You bring up a valid point that we should
 keep in mind for our own however. The bootstrapping issue will be the
 smallest possible annoyance on a long road of the migration process.
 
 The bootstrapping issue is a factor even after the migration :)

I think that the point I'm trying to make is that it shouldn't be.

 note that I'm not talking about the mechanism here, I'm trying to
 avoid pkg doesn't seem to be installed on my fresh system becoming a
 FAQ.

The way that we avoid that problem is not to create it for ourselves in
the first place. :)

The role of pkg-bootstrap is for those users who have already-installed
systems that need to do the conversion, or if somehow pkg becomes
corrupted on the user's system and needs to be reinstalled. That's it.

I like that you're thinking through the related issues, but in this
particular case I think you're overthinking it.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 8/24/2012 1:15 AM, Baptiste Daroussin wrote:
 BTW for people who haven't tested and want to share their opinion,
 here is how work /usr/sbin/pkg:
 
 it first checks if ${LOCALBASE}/sbin/pkg is there - if yes it
 directly execute ${LOCALBASE}/sbin/pkg with arguments passed to
 /usr/sbin/pkg

As others have already pointed out, this is a bad idea for a variety
of reasons, not the least of which is security related. It also
removes one of the primary benefits of pkg, that it be (fully) hosted
in the ports tree.

The bootstrap procedure does not need to be simple or transparent
because it's only going to exist for a very short period while users
are bringing pkg into already-installed systems where pkg is not
already the default; and they don't have an existing ports tree. The
way that you solve the bootstrap problem for systems where pkg IS the
default is to install the pkg package at system install time.

Let me rephrase that more simply ... very few users are ever going to
need the bootstrapping tool that will be in the base. Making it
mandatory for *every* user is therefore not only a bad idea, it's
contrary to one of the primary goals of the project.

Doug

- -- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (MingW32)

iQEcBAEBCAAGBQJQOAJpAAoJEFzGhvEaGryEY2QH/2Hv+cW20htODBTrFNScd7qS
NxBc7YHr3ddoyoui+Qwtl3ErjMOJr+kA3nRSsZ5ewGN5HVQ4gCAWp76bJn75BM71
q2Qmgx7HCnBMJKrmRTvDAA1nJcTKAgXFKK8hYQgiTOhWFaIjJxhDlln5llFcXwBa
VN0ErF421FkD8oyHpcQLga1BHLtldOL5itt/4Tp9cKaC5l2P9dBNbyCTxVa4XBiZ
tsK+x7XJqwj0NvXLk+b3icLIEeyopa3TJAoAtXZ27igg65norCoh2EPo7aJqP2zY
T75PKdbRJprwCpeJXulC02oZu/UERoIMLeaH1JbCZRcLAqaCJQuEGSP95as3hSY=
=BEya
-END PGP SIGNATURE-
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 8/24/2012 5:33 PM, Glen Barber wrote:
 On Sat, Aug 25, 2012 at 01:25:15AM +0100, Jonathan Anderson wrote:
 On 24 Aug 2012, at 23:38, Doug Barton do...@freebsd.org wrote:
 Let me rephrase that more simply ... very few users are ever going to
 need the bootstrapping tool that will be in the base.

 
 So, then they won't use it.  I fail to see the problem here.

That's because you're not paying attention. :)

Which comes first in your PATH, /usr/sbin, or /usr/local/sbin? Which
comes first in the default PATH?

What Baptiste said is that the way /usr/sbin/pkg works is to take
arguments handed to it and pass them through to /usr/local/sbin/pkg.
That means that every user who has their PATH configured in the default
manner (which is what every security text on Unix has recommended for 30
years) will be using /usr/sbin/pkg every time they type the pkg command.

 But surely the whole point of pkgng is that people *will* use pkg
 as the default method of acquiring third-party software, so they'll
 want to pkg install foo and have it Just Work. To say either you
 must download the ports tree in order to use binary packages or
 you must use pkg_add to install pkg seems to miss the point...

 
 /usr/sbin/pkg installs /usr/local/sbin/pkg without requiring the Ports
 Collection to be available locally.

It does much more than that. Go read the code.

As to the security related problems, they should be obvious. Having 1
binary that is always executed to pass arguments to another binary at
minimum doubles your attack surface. Given what /usr/sbin/pkg does, it
more than doubles it. Not to mention the flat out wrong-headed design of
having a binary that will be run as root whose primary purpose is to
pass arguments to another binary.

The reason this defeats the purpose of putting pkg in the ports tree is
that if there is a bug in /usr/sbin/pkg (which of course, there will be)
then it has to be fixed in the base, with all of the consequent drama
and delays that this will entail. If there is a bug in
/usr/local/bin/pkg, it gets fixed in the ports tree and instantly
updated, which is part of the virtue of having it in the ports tree in
the first place.

Given that if we do the rollout properly the bootstrap function will be
limited to a very small percentage of users, it makes sense to split
that function out into a separate, limited binary; and not pollute the
pkg stream with extra cruft it does not need.

 What I would like to know, is why all the anti-progress emails[1] have
 to wait until the Last Minute(tm) when information on pkgng availability
 has been available for quite some time now.

First off, I resent being told that because I'm raising legitimate
issues with something that I am being obstructionist, or
anti-progress. And my concerns are certainly not last minute. I've
been raising concerns about pkg since day 1, and given that there is
still no coherent, comprehensive project plan about the migration it's
not at all surprising that others are also starting to discover daemons
in the details.

It's also part and parcel of life in an open source project. Most people
don't pay attention about most things until they feel that it will be
affecting them. This is doubly true in open source. Given how well-known
this issue is, it should be planned for in any kind of big project such
as this. It's probably also worth mentioning that there are only so many
hours in the day, so one has to prioritize.

Doug

- -- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (MingW32)

iQEcBAEBCAAGBQJQOCwUAAoJEFzGhvEaGryEcpgH/2CAPBCldr7HlTjIzErqtbTO
S0ZaI0RabwEk85+HuFCmBLTbdKqVjGYcLqIbz7l6wOa20N1rPARtBDy5DkrMrL6s
5YAgWiZ43FyKQ4826VDVBvhPqxXMD0O+sETs2kskFUkV73u/r1/8EpfZgwCDk9F9
G8hqMVTRyoWgoh1HIaBba5/m4D7+UGPYE2w8M3QAGSULePYJLgaRdu+jd2aNBrJD
NFjY4lyLbitbIH1/fYHDR90KqlBVP6vr+bWUvdoHFJQ0W0HQw7wMtamo418SlORI
qfTaoHL4sA1sggHrlUVvxjgWbAtIcYT2F3K+u34yTaWAoqxoN9pzRy3GWXyFRzM=
=PNr3
-END PGP SIGNATURE-
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng 1.0 release schedule, and HEAD switch to pkgng by default schedule

[Doug Barton]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 8/22/2012 5:27 AM, Ivan Voras wrote:
 On 21/08/2012 22:15, Doug Barton wrote:
 
 And in this case, it doesn't matter how awesome the new tools
 are, they are a MAJOR paradigm shift for how users interact with
 ports, and we are
 
 Unless I've missed something,

Yes, you've missed quite a lot actually. You really need to follow the
discussion on ports@ if you want to stay up to speed.

 pkgng is actually *zero* paradigm shift for users familiar with
 *ports*, and here's why: people using ports can and will continue
 to use ports the way they are used to. AFAIK, the infrastructure
 which registers port installation is already there and there are
 also patches for portupgrade and portmaster which make them 
 interact nicely with the new package database.

For users who only have very limited interaction with the ports tree
this is probably true. But what we're seeing is that a lot of users
(especially those with larger installations, and re-packagers like
PC-BSD) have more than simple/limited ports interaction.

For those users the change is going to take time, sometimes
significant time to adjust to.

 The only important aspect of this is that the actual package
 database format changed (IMO, immensely for the better) and there
 are several other port management utilities which may need to be
 changed. People who got used to manually altering the old
 text-based package database will learn either not to do it anymore,
 since whole classes of errors have now become impossible to have,
 or learn how to do it with the new format.
 
 Can you explain what you mean as the paradigm shift for ports
 users here?

You just described it. And I certainly hope that the change is indeed
for the better, however that has yet to be demonstrated on a large
scale. I think shifting the default for 10 is going to give us more
data on this point, which is a good thing. But making it mandatory in
10 is premature.

 OTOH, people using *binary packages* (the very few and miserable
 users that they are since the old binary package infrastructure has
 sucked for the last decade or so), will get their world turned
 upside down, but for the better, and hopefully grow in numbers.

No argument from me on the sucking, but the number of users using the
existing packages is not few. There are more consumers of the
FreeBSD-distributed packages than you probably realize. But more
importantly there are a LOT of enterprise users who roll their own
package infrastructure.

I have been trying to get across to some of our src-centric Illuminati
for years just how valuable/important the ports are to the FreeBSD
Operating _System_. For better or worse I think that this change is
going to bear out the truth of what I (and others of course) have been
saying.

Doug

- -- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (MingW32)

iQEcBAEBCAAGBQJQNp/WAAoJEFzGhvEaGryEpIkH/AqfqLIugyLDWv6ehzaKhne8
pGCIGL6bS6naRzpvMu+hzA+eEg/ZnAp5tOjC2e0qowi50e5fF8CKEt11eZKOkyXA
FPQX00kX3KTKMyHd6SEsp6AL5FAihBASN9rVs3BGqBXge/ViI9HIDRBKpQW+11Yd
tH3wdCSfflI3UpteyJFFumIxITuTvAhYPBzSFEoThNAmf58qJWTNx8zW5jS3/lis
OnCWApouUfYOKdimbpRbguYiAnuX7o/Vrwvc9XQ6awsATDWNSPgf4kgaPvwnp9HH
eUlFtsNInlFMT9pwQhS2oQtIccx0BYsCQIXkCNQFIjddvRuUeVNjB5Vdqq7NuLk=
=kUKF
-END PGP SIGNATURE-
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

On 8/23/2012 3:19 PM, Steve Wills wrote:
 Hi,
 
 It seems to me that renaming the pkg binary in /usr/sbin/pkg to 
 /usr/sbin/pkg-bootstrap would make sense. From a user standpoint, it is 
 confusing that running the command gets different results the second time it 
 is run vs. the first time. I can imagine a user saying I ran pkg, but it 
 didn't do what they said it would.  Now I run it again, and it does do what 
 it is supposed to. Also, it would enable setting up a pkg-bootstrap man page 
 separate from the pkg man page, without confusion about which one you're 
 looking at.
 
 So, opinions? There may still be time to fix it for 9.1 if we can decide 
 quickly.

Yes please.

Every time in the past that we have talked about moving the pkg_* tools
to the ports the corresponding change for the base was to have a
pkg_bootstrap tool that was a use once and forget kind of thing. I was
quite surprised when sbin/pkg was added, but since people tell me I
already comment on too much, I decided to wait and see what others thought.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng suggestion: renaming /usr/sbin/pkg to /usr/sbin/pkg-bootstrap

[Doug Barton]

On 8/23/2012 7:23 PM, Eitan Adler wrote:
 On 23 August 2012 22:15, Steve Wills swi...@freebsd.org wrote:
 On Aug 23, 2012, at 10:08 PM, Eitan Adler wrote:

 On 23 August 2012 22:05, Steve Wills swi...@freebsd.org wrote:

 Why can't one of those steps be to run pkg-bootstrap?

 Because the how-to may not be for a new system ;)

 The possibility of bad docs somewhere outside of our control, when we can 
 (and I am actively working on) document(ing) pkgng for the handbook seems 
 kinda thin. It's not even Something's wrong on the Internet! 
 (http://xkcd.com/386/), it's Something might some day be wrong on the 
 Internet!
 
 It isn't a problem of bad docs. Its a problem of the user following
 some not-for-new-systems documentation and getting very confused when
 they see command not found.. It is practically the definition of
 POLA.

No, POLA refers to not changing long-established practices out from
under the user.

I tend to agree with Steve here ... we can't be responsible for other
people's poorly written docs. You bring up a valid point that we should
keep in mind for our own however. The bootstrapping issue will be the
smallest possible annoyance on a long road of the migration process.

OTOH this is a good use case for the prompt the user when they type a
command for something that can be installed from ports idea. :)

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng 1.0 release schedule, and HEAD switch to pkgng by default schedule

[Doug Barton]

On 8/21/2012 6:58 PM, Bjoern A. Zeeb wrote:
 On Tue, 21 Aug 2012, Doug Barton wrote:
 
 I don't think we have ever done a complete replacement of major
 infrastructure in one release.
 
 You mean like sysinstall can be used as an installer on 9 that would
 do something meaningful with the current infrastructure we provide?

Given the number of users who complain when sysinstall breaks in 9, I'd
say yes. Not to mention that sysinstall is a good example of something
that we deprecated in one release and removed in the following release.

Furthermore, I don't think of the installer as nearly as critical as the
ports collection. Yes, it is important, clearly.  But it's something
that is likely to happen only once in the lifetime of a system, as
opposed to the numerous times that users will interact with the ports.
Not to mention all of the enterprise users who bypass it altogether.

Aside from the installer part of sysinstall, the post-install config
portion has been taken over by bsdconfig. So in HEAD you have 2 new
tools that are mandatory that fulfill sysinstall's old role; and in 9
you have those same 2 new tools which are the defaults, but optional.
That's exactly how it is supposed to work.

Finally, the thing that we have to keep in mind is how different the
ports tree is from anything else in the base. The infrastructure of the
ports has to support all versions of FreeBSD. So we have to be extra
cautious about deprecating things. Of course the upside of pkg is that
it (properly) lives in the ports tree itself, which will make innovation
much easier in a few years.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: r239356: does it mean, that synchronous dhcp and dhcplcinet with disabled devd gone?

[Doug Barton]

On 8/21/2012 6:34 AM, John Baldwin wrote:

 Humm.  devd is the more common case, and we explicitly don't use devd to 
 start 
 dhclient on boot even when devd is enabled (so out of the box dhcp would 
 first 
 be started by rc, but would be restarted by devd).

That sounds reasonable. People who choose not to run devd can be
responsible for restarting dhclient themselves.

 Another option is to rework dhclient to work like it does on OpenBSD where it
 renews its lease if the link bounces, but to not exit when the link goes down.

That would be preferable.

 That case would fix the currently broken case that you unplug your cable, 
 take 
 your laptop over to another network (e.g. take it home if suspend/resume 
 works), then plug it back in and are still stuck with your old IP.

I do think it's important to fix this case. However I agree with the
chorus of responders that it is more important to maintain our historic
resilience to temporary loss of connectivity.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng 1.0 release schedule, and HEAD switch to pkgng by default schedule

[Doug Barton]

On 8/21/2012 6:46 AM, Baptiste Daroussin wrote:
 I would also like to just remove pkg_* tools from RELENG_10 if that fits the
 schedule.

Um, no?

Until pkg becomes mandatory (which can't happen for several years) the
pkg_* tools can't be removed altogether.

What _would_ be useful is what should have been done many years ago when
it was first suggested: Move the pkg_* tools to ports.

It's too late for 9.1 already, but if you made that change today in
HEAD, and after 9.1 (but before 8.4) you MFC it to stable/[89], then you
could theoretically make pkg mandatory after 9.1 EOLs.

To make my point more clear, the ports tree has to support the last
release to ship with pkg_* tools in the base throughout its lifetime. To
do anything else would be be a massive POLA violation.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng 1.0 release schedule, and HEAD switch to pkgng by default schedule

[Doug Barton]

On 8/21/2012 11:47 AM, Garrett Cooper wrote:
 On Tue, Aug 21, 2012 at 11:17 AM, Doug Barton do...@freebsd.org wrote:
 On 8/21/2012 6:46 AM, Baptiste Daroussin wrote:
 I would also like to just remove pkg_* tools from RELENG_10 if that fits the
 schedule.

 Um, no?
 
 ...
 
 What _would_ be useful is what should have been done many years ago when
 it was first suggested: Move the pkg_* tools to ports.
 
 It already exists -- it's just out of date / crufty:

Right ... I was using move as shorthand for several different ideas,
including but not limited to the latest version of the code itself,
robust support for the code going forward, the primary supported way of
using pkg_*, etc. All of these ideas have been discussed in the past, so
I was hoping to avoid having to re-discuss them. :)

 It's too late for 9.1 already, but if you made that change today in
 HEAD, and after 9.1 (but before 8.4) you MFC it to stable/[89], then you
 could theoretically make pkg mandatory after 9.1 EOLs.

 To make my point more clear, the ports tree has to support the last
 release to ship with pkg_* tools in the base throughout its lifetime. To
 do anything else would be be a massive POLA violation.
 
 Agreed.

Great (and I saw Baptiste's response on this as well). Glad to hear that
we're on the same page about something at least. :)


-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng 1.0 release schedule, and HEAD switch to pkgng by default schedule

[Doug Barton]

On 8/21/2012 12:05 PM, Baptiste Daroussin wrote:
 1/ if it fits the schedule: get rid of pkg_* tools in current to be
 able to have a fully pkgng only 10-RELEASE

I think it would fit better with historic precedents to make pkg
optional (but default on) in 10, and mandatory in 11. As stated
before, I'm fine with removing pkg_* tools from 10 if there is robust
support for them in the ports tree.

I know you're excited about this project, but let's not lose sight of
how big a change this is, and how important ports are to the project.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng 1.0 release schedule, and HEAD switch to pkgng by default schedule

[Doug Barton]

On 8/21/2012 12:42 PM, Baptiste Daroussin wrote:
 On Tue, Aug 21, 2012 at 12:38:04PM -0700, Doug Barton wrote:
 On 8/21/2012 12:05 PM, Baptiste Daroussin wrote:
 1/ if it fits the schedule: get rid of pkg_* tools in current
 to be able to have a fully pkgng only 10-RELEASE
 
 I think it would fit better with historic precedents to make pkg 
 optional (but default on) in 10, and mandatory in 11. As stated 
 before, I'm fine with removing pkg_* tools from 10 if there is
 robust support for them in the ports tree.
 
 I know you're excited about this project, but let's not lose
 sight of how big a change this is, and how important ports are to
 the project.
 
 That was what if it fits the schedule was about.

I think what I'm trying to say, ever so politely, is that what you're
suggesting isn't even an option, so it shouldn't be discussed.


-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: pkgng 1.0 release schedule, and HEAD switch to pkgng by default schedule

[Doug Barton]

On 8/21/2012 1:08 PM, Warner Losh wrote:
 
 On Aug 21, 2012, at 1:51 PM, Doug Barton wrote:
 
 On 8/21/2012 12:42 PM, Baptiste Daroussin wrote:
 On Tue, Aug 21, 2012 at 12:38:04PM -0700, Doug Barton wrote:
 On 8/21/2012 12:05 PM, Baptiste Daroussin wrote:
 1/ if it fits the schedule: get rid of pkg_* tools in
 current to be able to have a fully pkgng only 10-RELEASE
 
 I think it would fit better with historic precedents to make
 pkg optional (but default on) in 10, and mandatory in 11. As
 stated before, I'm fine with removing pkg_* tools from 10 if
 there is robust support for them in the ports tree.
 
 I know you're excited about this project, but let's not lose 
 sight of how big a change this is, and how important ports are
 to the project.
 
 That was what if it fits the schedule was about.
 
 I think what I'm trying to say, ever so politely, is that what
 you're suggesting isn't even an option, so it shouldn't be
 discussed.
 
 If you are fine with removing them if there's robust support, how can
 you also be suggesting that it is impossible and shouldn't be talked
 about?

Those address different parts of the problem. Making pkg mandatory in 10
is different from where the old pkg_* tools end up. The command line
tools are just the tip of the iceberg, there are a lot of interactions
behind the scenes.

 Personally, I think we should handle this the same way that other
 replacement tools have been done, which is close to what Baptiste has
 proposed.  If the new tools are totally awesome, we have replaced old
 tools. 

I don't think we have ever done a complete replacement of major
infrastructure in one release. The traditional model has been to
deprecate in one release, remove in the next.

And in this case, it doesn't matter how awesome the new tools are, they
are a MAJOR paradigm shift for how users interact with ports, and we are
going to have a lot of users who take years to transition their
installed base. No matter how much we may want to move fast on this, it
just isn't going to be possible.

 If the new tools are good, but don't cover the older users,
 we develop along size. 

Yes, this is precisely what I'm saying. Sorry if I wasn't clear.

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: CURRENT as gateway on not-so-fast hardware: where is a bottlneck?

[Doug Barton]

On 08/15/2012 03:18, Alexander Motin wrote:
 On 15.08.2012 03:09, Doug Barton wrote:
 On 08/14/2012 12:20 PM, Adrian Chadd wrote:
 Would you be willing to compile a kernel with KTR so you can capture
 some KTR scheduler dumps?

 That way the scheduler peeps can feed this into schedgraph.py (and you
 can too!) to figure out what's going on.

 Maybe things aren't being scheduled correctly and the added latency is
 killing performance?

 You might also try switching to SCHED_ULE to see if it helps.

 Although, in the last few months as mav has been converging the 2 I've
 started to see the same problems I saw on my desktop systems previously
 re-appear even using ULE. For example, if I'm watching an AVI with VLC
 and start doing anything that generates a lot of interrupts (like moving
 large quantities of data from one disk to another) the video and sound
 start to skip. Also, various other desktop features (like menus, window
 switching, etc.) start to take measurable time to happen, sometimes
 seconds.

 ... and lest you think this is just a desktop problem, I've seen the
 same scenario on 8.x systems used as web servers. With ULE they were
 frequently getting into peak load situations that created what I called
 mini thundering herd problems where they could never quite get caught
 up. Whereas switching to 4BSD the same servers got into high-load
 situations less often, and they recovered on their own in minutes.
 
 It is quite pointless to speculate without real info like mentioned
 above KTR_SCHED traces.

I'm sorry, you're quite wrong about that. In the cases I mentioned, and
in about 2 out of 3 of the cases where users reported problems and I
suggested that they try 4BSD, the results were clear. This obviously
points out that there is a serious problem with ULE, and if I were the
one who was responsible for that code I would be looking at ways of
helping users figure out where the problems are. But that's just me.

 Main thing I've learned about schedulers, things
 there never work as you expect. There are two many factors are relations
 to predict behavior in every case.

In the web hosting case that I mentioned, I purposely kept every other
factor consistent; and changed only s/ULE/4BSD/. The results were both
clear and consistent.

 What's about playing AVIs and using other GUIs, key word here and for
 ULE in general is interactivity. ULE gives huge boost to threads it
 counts interactive.

I'm not using ULE. I haven't for over a year. Sorry if I wasn't clear.

 If somebody still wish area for experiments, there is always some:
  - if you want video player to not lag, set negative nice for it (ULE is
 not a magician to guess user wishes);

At the same time, I don't have these problems on my Linux systems, and I
don't need to adjust anything. Not to mention that given how web servers
are one of our main server implementations, the fact that we have what
seems to be a serious performance problem with out default scheduler in
that use case seems like an issue that we would want to address.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: CURRENT as gateway on not-so-fast hardware: where is a bottlneck?

[Doug Barton]

On 08/20/2012 02:59, Alexander Motin wrote:
 On 20.08.2012 11:32, Doug Barton wrote:
 On 08/15/2012 03:18, Alexander Motin wrote:
 On 15.08.2012 03:09, Doug Barton wrote:
 On 08/14/2012 12:20 PM, Adrian Chadd wrote:
 Would you be willing to compile a kernel with KTR so you can capture
 some KTR scheduler dumps?

 That way the scheduler peeps can feed this into schedgraph.py (and you
 can too!) to figure out what's going on.

 Maybe things aren't being scheduled correctly and the added latency is
 killing performance?

 You might also try switching to SCHED_ULE to see if it helps.

 Although, in the last few months as mav has been converging the 2 I've
 started to see the same problems I saw on my desktop systems previously
 re-appear even using ULE. For example, if I'm watching an AVI with VLC
 and start doing anything that generates a lot of interrupts (like
 moving
 large quantities of data from one disk to another) the video and sound
 start to skip. Also, various other desktop features (like menus, window
 switching, etc.) start to take measurable time to happen, sometimes
 seconds.

 ... and lest you think this is just a desktop problem, I've seen the
 same scenario on 8.x systems used as web servers. With ULE they were
 frequently getting into peak load situations that created what I called
 mini thundering herd problems where they could never quite get caught
 up. Whereas switching to 4BSD the same servers got into high-load
 situations less often, and they recovered on their own in minutes.

 It is quite pointless to speculate without real info like mentioned
 above KTR_SCHED traces.

 I'm sorry, you're quite wrong about that. In the cases I mentioned, and
 in about 2 out of 3 of the cases where users reported problems and I
 suggested that they try 4BSD, the results were clear. This obviously
 points out that there is a serious problem with ULE, and if I were the
 one who was responsible for that code I would be looking at ways of
 helping users figure out where the problems are. But that's just me.
 
 I am not telling anything bad about 4BSD.

Yes, I get that, but thanks for making it clear.

 Choice is provided because
 they are indeed different and none is perfect.

... which is why I'm asking you to stop making them more the same until
we get a better idea of what the issues are.

 What I would like to say is that if we want to improve situation, we
 need more detailed info then just verbal description.

And what I'm saying is that the only realistic way that you're going to
get that information that you need is to make it easier for users to
give it to you. I don't know what form that is going to need to take, I
don't know anything about schedulers.

 I am not telling
 that ULE is perfect. I went there because I've seen problems, and I am
 still fixing some pieces. I am just trying to explain described behavior
 from the point of my knowledge about it, hoping that it may help
 somebody to set up some new experiments or try some tuning/fixing.

Yes, I think it's great that you're doing this work. I'm glad to see
that someone is improving ULE. It clearly needs it. :)

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: CURRENT as gateway on not-so-fast hardware: where is a bottlneck?

[Doug Barton]

On 08/20/2012 06:32, Alexander Motin wrote:
 I have no plans to converge them. I've just found problem in ULE, that
 was replicated into 4BSD and it would be strange to fix one without
 another. But fixing it exposed another old problem specific to 4BSD,
 which I fixed reusing logically equivalent code from ULE. I saw no
 reason to reinvent a wheel there, same as to not fix obvious bug. Sure,
 it can change behavior in some way, but ULE is not guilty.

Thank you for that explanation.

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: CURRENT as gateway on not-so-fast hardware: where is a bottlneck?

[Doug Barton]

On 08/14/2012 09:18 PM, Dimitry Andric wrote:
 On 2012-08-15 02:09, Doug Barton wrote:
 On 08/14/2012 12:20 PM, Adrian Chadd wrote:
 ...
 Maybe things aren't being scheduled correctly and the added latency is
 killing performance?

 You might also try switching to SCHED_ULE to see if it helps.
 
 Most likely, s/ULE/4BSD/ here, and in the rest of your mail? :)
 

yes, thanks
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: CURRENT as gateway on not-so-fast hardware: where is a bottlneck?

[Doug Barton]

On 08/14/2012 12:20 PM, Adrian Chadd wrote:
 Hi,
 
 Would you be willing to compile a kernel with KTR so you can capture
 some KTR scheduler dumps?
 
 That way the scheduler peeps can feed this into schedgraph.py (and you
 can too!) to figure out what's going on.
 
 Maybe things aren't being scheduled correctly and the added latency is
 killing performance?

You might also try switching to SCHED_ULE to see if it helps.

Although, in the last few months as mav has been converging the 2 I've
started to see the same problems I saw on my desktop systems previously
re-appear even using ULE. For example, if I'm watching an AVI with VLC
and start doing anything that generates a lot of interrupts (like moving
large quantities of data from one disk to another) the video and sound
start to skip. Also, various other desktop features (like menus, window
switching, etc.) start to take measurable time to happen, sometimes
seconds.

... and lest you think this is just a desktop problem, I've seen the
same scenario on 8.x systems used as web servers. With ULE they were
frequently getting into peak load situations that created what I called
mini thundering herd problems where they could never quite get caught
up. Whereas switching to 4BSD the same servers got into high-load
situations less often, and they recovered on their own in minutes.

Doug

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: VirtualBox: Eating up 100% CPU, freezing Windows 7

[Doug Barton]

On 08/04/2012 17:56, Kevin Oberman wrote:
 On Sat, Aug 4, 2012 at 2:30 PM, Doug Barton do...@freebsd.org wrote:
 On 08/04/2012 14:26, Garrett Cooper wrote:
 On Sat, Aug 4, 2012 at 1:26 PM, Doug Barton do...@freebsd.org wrote:

 On 08/04/2012 00:40, O. Hartmann wrote:
 No, also in my case. I build world and the VBox software with each
 kernel - usually.

 You can ensure that by putting this in src.conf:

 PORTS_MODULES=  emulators/virtualbox-ose-kmod

 You can place other modules in that list as well. I use vbox so you can
 be pretty confident that this is going to keep working. :)

 That doesn't work

 I assure you that it does. I have put a non-zero amount of work into
 fixing it, I use this method, and the resulting kernel module works just
 fine.

 If you actually try it and find something is not as it should be, then
 yes; please do file a PR and feel free to cc me.

 Doug
 
 I am only aware of this because of your posts. No reference to it at
 all in src.conf(5). It would be nice to see it there.

It's in make.conf(5) for historical reasons, along with a lot of other
options that should be moved.

 It is mentioned in build(7), but only as a MAKE option and a lot of
 people are not going to realize it can be placed in src.conf.

It can also go in make.conf, so that's not a total loss I suppose.

 Also,
 for those not fairly conversant in make, it is not clear whether
 multiple ports should be space, comma, colon, or otherwise delimited
 
 This is a very nice option as it is very easy to overlook rebuilding
 kernel modules in ports when building the kernel. Thanks or working on
 it.

My pleasure ... it was one of those things on the list and I finally
got around to it.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: On cooperative work [Was: Re: newbus' ivar's limitation..]

[Doug Barton]

On 08/02/2012 12:18, David Chisnall wrote:
 Thank you for your thoughtful reply,

You too ... I let some time go by to see what others had to say. I think
it's disappointing that more people aren't concerned about this issue.

 On 2 Aug 2012, at 19:33, Doug Barton wrote:
 
 However, my point is that in spite of the fact that it's
 non-trivial, the mindset on this topic needs to change if the dev
 summits are going to continue to be significant focii of both work
 being done and decisions being made (which of course, they are).
 
 I believe that, before that decision can be made, there needs to be
 some consensus on what the purpose of the DevSummits is.  In my view,
 DevSummits do not exist to set project policy. 

And yet, that's exactly what happens. It's easy to understand why ...
you have a bunch of people together in the same place, they all agree on
a topic, and after all, since they are the ones who are there, they
should be the ones to make the decision, right?

It's not necessarily that they are trying to do something malicious,
it's just human nature. I know, I used to travel to conferences for a
living.

 They are places where:
 
 - People can talk face to face about their current and planned
 projects. - Developers can meet on a social basis, making remote
 working easier.
 
 The latter is very important - I've found in other projects that it's
 far easier to work with someone on the other side of the world when
 you've chatted with them over a few beverages-of-choice.

I agree with these points as well. (Again, been there, done that.) In
fact I'm quite confident that a lot of the issues people have with me
are related to this deficiency. :)

 Any official conversations happen on the mailing lists. 

This should be true, but it isn't. This is my point.

 DevSummits
 are for people working on similar things to meet and discuss their
 plans, and for people to have a chance to get to know what everyone
 else is doing,

... so far, so good ..

 for a limited set of 'everyone else'. 

... and this is where I call shenanigans. This is an open project.
Anything that is going to be done is going to be seen. If there are
problems with a proposal it's better to see them early. If it's a good
proposal, there is no reason not to share it.

 Slides and
 summaries so on from the more structured parts of this are available
 afterwards, which helps people who can't attend get the same benefit
 - they know what other people are working on.

In the IETF context proposals often benefit from the real-time focus of
attention, from both local and remote participants, that the meetings
provide. There is no reason to believe that this would not be true in
FreeBSD as well.

 The original complaint that spawned this long discussion was that
 decisions about the project are made behind closed doors.  This is
 obviously true in the literal sense, as code always wins over chatter
 in an open source project, and the person willing to sit down and
 write the code gets the final say on how it should work,

That's an oft-repeated maxim, especially around here. But we've already
demonstrated that it isn't true. The only time that this is true is if
the work being done is in line with what the PTB want. I've demonstrated
this time after time by volunteering to do various projects my way and
being told that my efforts weren't welcome. Not to mention having my
finished, working code reverted by a core team member for an inferior,
broken result.

So can we please stop repeating that BS and focus on the real issues?

 although
 ideally with code review, design feedback and so on from others.
 Even if we broadcast everything that happens in the official parts of
 the DevSummits, that won't necessarily fix anything because a lot of
 the most productive conversations happen over dinner or in the pub.

The community needs to not be accepting of the status quo, and demand
that things be publicized on the lists first before decisions are made.

Once again, if they are good decisions, they won't be harmed by sharing
them in advance. If they are less-good, we could be saving someone
efforts that would be otherwise wasted.

 If there is a real problem to address, then it is people making
 policy decisions at DevSummits, without adequate consultation.  I
 have not observed this happening, but I would regard it as no
 different to people making policy decisions via private email, and
 something that should be subject to the same response: revisit the
 decisions in public if there are legitimate concerns raised about it,
 subject to the usual open source rule that the person actually
 willing to do the work gets to make the final call.

Exactly.

 Finance is not the only obstacle.  In some venues, bandwidth is a
 problem (not at Cambridge hopefully - people will have stopped using
 it all to stream the olympics by then), but in other venues we only
 have WiFi, which is shared with a room full of developers.  If we buy
 some equipment (decent

Re: VirtualBox: Eating up 100% CPU, freezing Windows 7

[Doug Barton]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 08/04/2012 00:40, O. Hartmann wrote:
 No, also in my case. I build world and the VBox software with each
 kernel - usually.

You can ensure that by putting this in src.conf:

PORTS_MODULES=  emulators/virtualbox-ose-kmod

You can place other modules in that list as well. I use vbox so you can
be pretty confident that this is going to keep working. :)

Doug

- -- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.19 (FreeBSD)

iQEcBAEBCAAGBQJQHYWJAAoJEFzGhvEaGryEyLUIALgi1n65I8oBaFYJEcIkDB6P
W3f5PMZa72DN4r2lI3A3XXdPUJsNRmNy/X0HYyrrIwvfD3Z3m8bReYCd7DHAKOX4
pBsLA/73cwns9c3+zsUe4i9TZOsJM8fNVsRp/BSvdtBgv61ZZUurvt43H+Ek0E0B
h5ttGaIanxLqrkwP2FC/q30t0pmauJYu3jDTGiugOh98o/3oNT+25etyJBNgvg4c
VxBs/5aCSc5VHAcLXRN6Y0BGGbeimpPqEFlq3FEFGLkC7LGjqoSBUaJVz1cgDP+t
RIK9g0V+XIfyirgZ2VMeK3tfQ0Q17zfyl0+Iyzl2IxZptU67OBV/9LMqyhRaBOc=
=KbES
-END PGP SIGNATURE-
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: VirtualBox: Eating up 100% CPU, freezing Windows 7

[Doug Barton]

On 08/04/2012 14:26, Garrett Cooper wrote:
 On Sat, Aug 4, 2012 at 1:26 PM, Doug Barton do...@freebsd.org wrote:

 On 08/04/2012 00:40, O. Hartmann wrote:
 No, also in my case. I build world and the VBox software with each
 kernel - usually.

 You can ensure that by putting this in src.conf:

 PORTS_MODULES=  emulators/virtualbox-ose-kmod

 You can place other modules in that list as well. I use vbox so you can
 be pretty confident that this is going to keep working. :)
 
 That doesn't work

I assure you that it does. I have put a non-zero amount of work into
fixing it, I use this method, and the resulting kernel module works just
fine.

If you actually try it and find something is not as it should be, then
yes; please do file a PR and feel free to cc me.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: On cooperative work [Was: Re: newbus' ivar's limitation..]

[Doug Barton]

On 08/02/2012 09:20, Scott Long wrote:
 
 On Aug 2, 2012, at 12:23 AM, Kevin Oberman kob6...@gmail.com
 wrote:
 
 Doug makes some good points.
 
 No, he doesn't. 

Yes I do! (So there)

 He and Arnould being argumentative and accusatory
 where none of that is warranted.
 
 I used to run the devsummits, and we did tele-conference lines for
 remote people to participate. 

I singled out BSDCAN specifically since that's where the action is for
the last several years. I do recall your efforts in this regard, but it
so happened that I was able to attend most of them in person back then.
No slight towards what you did was intended.

 After I stepped down, others took it
 up and did the same thing.  Usually, the lines were unused.  I
 suspect that organizers simply stopped thinking about them after a
 while because of poor interest.  There is no conspiracy of exclusion
 here, just simple human apathy.

Here I have to disagree with you. Once again, speaking specifically
about BSDCAN dev summits, I repeatedly asked the organizers to provide
some sort of audio stream (phone, Internet, anything) and was repeatedly
told it wasn't possible. This was not a case of lack of interest. This
was a case of We understand that it is something people want, but it
isn't going to happen.

 The invite system for the devsummit was, and still is, purely about
 providing some order to the process.  It ensures that people
 attending are willing to demonstrate a minimum amount of interest,
 more than just wondering by a room one day and dropping in for free
 food and wifi. 

I specifically made allowances for this issue in my post.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: On cooperative work [Was: Re: newbus' ivar's limitation..]

[Doug Barton]

On 08/02/2012 05:54, David Chisnall wrote:
 On 2 Aug 2012, at 05:30, Doug Barton wrote:
 
 I used to ask the PTB to provide *some* form of remote
 participation for even a fraction of the events at the dev summit.
 I don't bother asking anymore because year after year my requests
 were met with any of: indifference, hostility, shrugged shoulders
 (that's a hard problem that we can't solve), or embarrassment.
 Since if the right people around here want something to happen, it
 happens; I finally came to the conclusion that they didn't want
 remote participation to happen, so it won't. That's a shame.
 
 You haven't asked for this for the Cambridge DevSummit,

You did read the part where I gave up, right?

 but others
 have and so we have arranged for cameras and microphones to be
 available for two of the sessions (the DocSummit and the ARM working
 group) to allow those who can not attend in person for various
 reasons to participate.

Well that's a start. :) And where was this availability announced? If I
missed it, that's on me. But providing remote access that you don't tell
people about isn't really any better than not providing it at all.

 I don't know how useful it will be (hopefully everything will work,
 but my experience with video conferencing is that it stops working as
 soon as you try to do something important with it),

If I can offer some advice from the trenches ... focus on making the
audio robust, and put efforts into the video as resources permit. The
combination of solid audio, making presentations available on line, and
a chat room (IRC, jabber, whatever) allows for a great deal of remote
participation. Video is nice, but if the video going down takes the
audio with it, you're no better off than when you started.

 but there is
 certainly no active attempt to exclude people who can't attend.

... and here is where I need to push back. No active attempt to exclude
people is not the same thing as actively encouraging remote
participation. It's the latter that we're after.

 After each DevSummit, the results seem to appear on the wiki quite
 promptly - often during the sessions.  At BSDCan this year, two of
 the working groups that I attended used OpenEtherPad to take minutes,
 so they were available in real time for non-attendees and people
 outside of the room were able to add things to them.  There are
 usually people in the room on IRC as well, who are willing to relay
 things from people outside.

Those all sound like nice steps forward, thank you for pointing them
out. Nothing would make me happier than to be proven wrong in this area.
What would be nice I think would be if these steps were formalized, and
shared more openly. Having things on the wiki is nice, but reporting
things in detail on the mailing lists puts it in the archives for future
reference, as well as making it more broadly available to start with.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: On cooperative work [Was: Re: newbus' ivar's limitation..]

[Doug Barton]

On 08/02/2012 09:44, Garrett Cooper wrote:

 The Watson/Losh connection worked really well in BSDCan 2010 :). 

I wasn't going to mention that, since I didn't want to tell tales out of
school. But the fact that remote participation actually was provided for
the right people, even though I was told repeatedly that it wasn't
possible, actually highlights a big part of the problem.

 Advertising the teleconferencing lines might be an issue (I would
 have loved to have joined in some of the remote conferences, if for
 nothing more than be a fly on the wall, this year), but that's a
 separate thing aside.

At various points when I was asking for remote participation at BSDCAN
different people offered to provide this through their company's
teleconferencing solutions, providing that the organizers could put a
phone line in the room(s). They were told that it wasn't possible to do
that.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: On cooperative work [Was: Re: newbus' ivar's limitation..]

[Doug Barton]

On 08/02/2012 10:13, David Chisnall wrote:
 On 2 Aug 2012, at 17:46, Doug Barton wrote:
 
 Well that's a start. :) And where was this availability announced?
 If I missed it, that's on me. But providing remote access that you
 don't tell people about isn't really any better than not providing
 it at all.
 
 It's not widely advertised, because we're likely to be able to
 support a limited number of remote participants (10 seems like the
 upper limit for the technology that we're looking at, and I wouldn't
 be surprised if it degrades before then). 

Welcome to the 21st Century. :) There are widely available audio and
video conferencing solutions that easily scale into the thousands of
users, at minimal cost.

 As with all other things
 in the project, we welcome people who are willing to make an effort
 to engage.  We provide it when people ask, not spontaneously, because
 organising cameras and decent microphones requires effort on the bart
 of the organisers. 

Yes, It takes effort. I get that. I've been part of the effort to
provide remote participation for other groups, on a much larger scale
than anything FreeBSD can dream of.

My point, and I cannot emphasize this highly enough, is that your entire
mindset about this is all wrong. It needs to shift from We'll do this
on a small scale, for those who ask to We'll make providing robust
remote participation a top priority, built into the planning from day
1. It's as simple as that.

 The FreeBSD Foundation has also offered to fund new contributors who
 want to attend but are unable to afford to do so on their own.  In
 spite of the fact that I spent some effort encouraging people to
 apply for this, only one person actually did.

It isn't just the financial cost of attending the summit. Often (as in
my case) it's the lack of ability to take time away from personal, work,
and/or family commitments. For others it may be the difficulty of doing
the traveling at all. The fact that only 1 person took you up on this
offer (and IIRC there have been similar results in the past) pretty
clearly indicates that you're trying to solve the wrong problem.

Given that the foundation has money to spend here, why not put 2 and 2
together and have the foundation invest in providing remote
participation? That would benefit far more people, and almost certainly
at less cost.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: On cooperative work [Was: Re: newbus' ivar's limitation..]

[Doug Barton]

BTW, for those who'd like to get a flavor of what the IETF model looks
like, the Vancouver meeting is in process now:

https://datatracker.ietf.org/meeting/84/agenda.html

Feel free to join in as a lurker.

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: On cooperative work [Was: Re: newbus' ivar's limitation..]

[Doug Barton]

On 08/02/2012 10:34, Doug Barton wrote:
 BTW, for those who'd like to get a flavor of what the IETF model looks
 like, the Vancouver meeting is in process now:
 
 https://datatracker.ietf.org/meeting/84/agenda.html
 
 Feel free to join in as a lurker.

Sorry, this agenda makes it easier to see the remote participation stuff:

https://tools.ietf.org/agenda/84/


-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: On cooperative work [Was: Re: newbus' ivar's limitation..]

[Doug Barton]

On 08/02/2012 10:37, David Chisnall wrote:

 Thank you for volunteering to organise this.  It's good to see people with 
 both the motivation and experience required to do something well actively 
 contributing to the project.

Cheap copout. And quite sad, especially coming from a newly elected core
team member.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: On cooperative work [Was: Re: newbus' ivar's limitation..]

[Doug Barton]

On 08/02/2012 10:40, Warner Losh wrote:
 One thing to remember about the IETF.  There's many vendors that devote 
 significant resources to the IETF.  While I was at Cisco, for example, I know 
 that we provided audio and video bridges to IEFT meetings to facilitate 
 remote attendance at the meetings.  Cisco dedicated several engineers to 
 ensure that the audio and video quality remained good during the meetings and 
 were able to use facilities cisco normally used for things like WebEx and 
 MeetingPlace.  With a global presence and infrastructure, they were able to 
 pull it off.  I'm not aware of similar resources within the project.

I'm not suggesting we need anything at the full WebEx
audio/video/presentation/chat level. And apparently the Foundation has
money to spend on dev summits. As I suggested in a previous message,
this would be a good long-term investment that would benefit a lot of
people, especially in comparison to the money set aside for travel
grants which is now going begging.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: On cooperative work [Was: Re: newbus' ivar's limitation..]

[Doug Barton]

On 08/02/2012 05:39, John Baldwin wrote:
 I find this a bit ironic from you given that I've met you in person at
 USENIX ATC which is an order of magnitude more expensive than BSDCan (and
 in fact, one of the reasons the US-based BSDCon died and was effectively
 supplanted by BSDCan was that BSDCan is far cheaper).

Yep, back in 2004 when traveling to conferences was part of my job, and
before my daughter was born. My life now is quite different.

... not to mention that this thread isn't about me. It's about the
importance of remote participation to the FreeBSD community as a whole.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: On cooperative work [Was: Re: newbus' ivar's limitation..]

[Doug Barton]

On 08/02/2012 11:12, David Chisnall wrote:
 FreeBSD is a volunteer project.

Yeah, I get that. I've been around quite a bit longer than you have, in
case you didn't notice. :)

I understand what you're saying, it's going to take work to change this
mindset, and to provide these resources. If you read my posts on a
factual basis, I'm not suggesting that the dev summits provide remote
participation at the same level as groups like the IETF or ICANN do, and
your point (and Warner's) that these groups have significant financial
backing is well taken.

However, my point is that in spite of the fact that it's non-trivial,
the mindset on this topic needs to change if the dev summits are going
to continue to be significant focii of both work being done and
decisions being made (which of course, they are).

What I'm *not* doing is demanding that any one person, or even any one
group take responsibility for solving the whole problem on their own.
Unfortunately, due to my inability to actually attend these meetings, I
won't be able to provide the kind of hands-on assistance that I'd like
to be able to. However it sounds like there may be financial resources
available through the foundation, which would go a long way towards
making a solution possible.

The next step would be for people to agree that this is a problem that
*needs* to be solved, followed by willingness on the part of dev summit
organizers to support these efforts, which will hopefully lead to people
who are willing and interested to step up and actually provide
solutions. It's already been true in the past that various companies
have volunteered to do this. There is no reason to believe that it
wouldn't happen again if organizers are willing to be supportive.

What I'm hearing so far is defensiveness, and an attempt to focus the
discussion on me. Neither is helpful. :) Acknowledging that this is a
problem that needs to be solved does not imply that by not solving it
you personally have failed in some way. I apologize if anything I've
written so far has implied otherwise.

Doug

-- 

I am only one, but I am one.  I cannot do everything, but I can do
something.  And I will not let what I cannot do interfere with what
I can do.
-- Edward Everett Hale, (1822 - 1909)
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org