Re: Stop installing /usr/bin/clang

2019-08-16 Thread David Chisnall

On 16/08/2019 10:10, Konstantin Belousov wrote:

You did not read neither review summary nor followups.  


This is needlessly insulting and this kind of attitude from you towards 
people on the mailing lists is one of the main reasons that my 
engagement with the FreeBSD project tends to be in brief bursts.  If 
this were a one-off, then I would be happy to assume that you were 
unusually stressed, but this is a long-term repeated pattern of behaviour.


I was not aware that devel/llvm was anything other than a meta-port that 
installed the latest devel/llvm{version} (I have only ever installed the 
packages when I need a specific version and so do not have the 
devel/llvm port installed).  You could have clarified that.  Instead, 
you chose to launch a personal attack.


You are not Linus and the FreeBSD project does not need a Linus.

David
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Stop installing /usr/bin/clang

2019-08-16 Thread David Chisnall

On 15/08/2019 17:48, Konstantin Belousov wrote:
> Please look at https://reviews.freebsd.org/D21060
> I propose to stop installing /usr/bin/clang, clang++, clang-cpp.
>
> It probably does not matter when all your software comes from ports or
> packages, but is actually very annoying when developing on FreeBSD.
> In particular, you never know which `clang' is called in the user
> environment, because it depends on the $PATH elements ordering.

What is the confusion here?  The binary that is invoked as clang is from 
the base system.  The binary that is invoked as clang{version number} is 
from ports.  If the user has built clang from source and has set up 
their path to put that first, then they will get a different clang, but 
there's no way we can stop that kind of behaviour.


For reference, on my machine, I have:

clang <- this one is from the base system
clang60 <- this one if from ports
clang70 <- this one if from ports
clang80 <- this one if from ports
clang-devel <- this one if from ports

Nothing in my PATH order affects this.

The only source of confusion that I regularly encounter comes from the 
fact that FreeBSD packages install clang80, when every other system 
installs clang-8, so I end up having to have a special case in CMake 
logic for finding specific versions of tools like clang-format on FreeBSD.


That said, I don't know what the impact would be on configure scripts if 
we didn't have a clang binary.  CMake seems to run ${CC} -v and parse 
the output, so it's quite happy finding that cc is clang (and the 
specific version).  How do most autoconf things handle this?  Apple 
shipped a gcc symlink to clang for years because, in the absence of a 
gcc binary, a load of programs detected /usr/bin/cc and decided not to 
enable any GNU extensions.  We've managed to avoid having to do that, 
but how many things look for clang, gcc, and cc in the path and enable 
features based on which one they find?


David
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: FreeBSD Core Team Response to Controversial Social Media Posts

2019-05-20 Thread David Chisnall
On 19 May 2019, at 20:43, Igor Mozolevsky  wrote:
> 
> the best
> explanation of democracy I have ever heard was: "two wolves and a
> sheep deciding what to have for dinner!"

If you believe that this quote in any way supports your argument, then I would 
suggest that you work through the game theoretic implications of this structure.

(Hint: if the sheep can abstain, the sheep is never eaten and even without 
abstention the sheep isn’t going to be eaten today)

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: CFT: FreeBSD Package Base

2019-04-30 Thread David Chisnall

On 29/04/2019 21:12, Joe Maloney wrote:

With CFT version you chose to build, and package individual components such as 
sendmail with a port option.  That does entirely solve the problem of being 
able to reinstall sendmail after the fact without a rebuild of the userland 
(base) port but perhaps base flavors could solve that problem assuming flavors 
could extend beyond python.


This sounds very much like local optimisation. It's now easy to create a 
custom base image.  Great.  But how do I express dependencies in ports 
on a specific base configuration? This is easy if I depend on a specific 
base package, but how does this work in your model?  For example, if I 
have a package that depends on a library that is an optional part of the 
base system, how do I express that pkg needs to either refuse to install 
it, or install a userland pkg that includes that library in place of my 
existing version as part of the install process?


More importantly for the container use case, if I want to take a 
completely empty jail and do pkg ins nginx (for example), what does the 
maintainer of the nginx port need to do to express the minimum set of 
the base system that needs to be installed to allow nginx to work?


One of the goals for the pkg base concept was to allow this kind of use 
case, easily creating a minimal environment required to run a single 
service. With a monolithic base package set, you're going to need some 
mechanism other than packages to express the specific base subset 
package that you need and I think that you need to justify why this 
mechanism is better than using small individual packages.


David
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: CFT: FreeBSD Package Base

2019-04-29 Thread David Chisnall

On 29/04/2019 14:19, Lev Serebryakov wrote:

  I'm not very interested in packetized base for "big servers" which
contains full FreeBSd installation


'Big servers' may have a full FreeBSD installation in the base system, 
but they may also have hundreds of jails that want the absolute minimum 
required for the service that they're exporting.


FreeBSD is currently suffering quite a lot from the lack of any solid 
story here.  The vast majority of cloud deployments are now using some 
combination of Docker and Kubernetes or equivalents to spin up a large 
number of VMs and an even larger number of microservice containers 
within them.  This should be something that FreeBSD is ideal for - jails 
preform better and provide a more coherent interface than the mess of 
cgroups and seccomp-bpf that Linux containers use.


It *ought* to be trivial to create a jail that has basically nothing 
other than the core libraries (and maybe a shell) and is managed from 
the outside.  Even the few FreeBSD core utilities that support jails 
don't really work like this (for example, I can use pkg to install 
something in a jail, but doing so implicitly installs a copy of the pkg 
tool inside the jail and invokes that).


David
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Building freebsd on another OS

2019-03-19 Thread David Chisnall

On 19/03/2019 00:01, Eric Joyner wrote:

On Sun, Mar 17, 2019 at 6:35 AM Hans Petter Selasky  wrote:


See the freebsd-build utils package for Linux.

--HPS



Is there anything for Windows?


Your best bet on Windows is to use the Windows Subsystem for Linux 
(WSL).  This lets you install a Linux distro's userland on top of the NT 
kernel.  If you install vcxsrv (available in chocolatey) then you can 
also run graphical applications.


That said, FreeBSD also runs very well under Hyper-V, so if you have 
enough RAM then you may find that a better option.  In my experience, 
compilers that spawn a new process for every file (e.g. gcc, clang) are 
noticeably faster in a FreeBSD VM on Windows than in WSL or native in 
Windows (and a *lot* faster than their cygwin versions).


David
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: GNU binutils 2.17.50 retirement planning

2018-11-25 Thread David Chisnall
On 23 Nov 2018, at 16:23, Ed Maste  wrote:
> 
> For some time we have been incrementally working to retire the use of
> obsolete GNU Binutils 2.17.50 tools. At present we still install three
> binutils by default:
> 
> as
> ld.bfd
> objdump

We probably need to kill ld.bfd before 12.0.  It predates ifunc and so 
interprets anything with an ifunc as requiring a copy relocation.  This means 
that if you use it to link against any shared library (like, say, libc.so.7 in 
FreeBSD 12.0) that uses ifuncs then it will insert a relocation so that the 
ifunc resolver (which contains PC-relative addresses of other functions) will 
be copied into the main binary.  This then causes your program to crash the 
first time anything calls memcpy, in a very difficult-to-debug way (it jumps 
into a random bit of your main binary, runs for a bit, and then dies).

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Warnings about dlclose before thread exit. __cxa_thread_call_dtors

2018-08-26 Thread David Chisnall
The FreeBSD implementation here looks racy.  If one thread dlcloses an object 
while another thread is exiting, we can end up calling a function at an invalid 
memory address.  It also looks as if it may be possible to unload one library, 
load another at the same address, and end up executing entirely the wrong code, 
which would have some serious security implications.

The GNU/Linux equivalent of this function locks the DSO in memory until all 
references to it have gone away.  A call to dlclose() on GNU/Linux will not 
actually unload the library until all threads with destructors in that library 
have been unloaded.  I believe that this reuses the same reference counting 
mechanism that allows the same library to be dlopened and dlclosed multiple 
times.

It would be nice if the FreeBSD version had the same behaviour, because this is 
almost certainly expected in code written on other platforms.

David

> On 18 Aug 2018, at 14:18, Willem Jan Withagen  wrote:
> 
> Hi,
> 
> I've sent the question below to the Ceph-devel list, asking if any recent 
> changes would be able to cause this.
> 
> But then of course this could stem from FreeBSD libs, and of ports
> So the question here is if anybody has gotten these "warnings" in other tools.
> 
> --WjW
> 
> 
>  Forwarded Message 
> Subject: Warnings about dlclose before thread exit
> Date: Sat, 18 Aug 2018 14:46:35 +0200
> From: Willem Jan Withagen 
> To: Ceph Development 
> 
> Hi,
> 
> I've have upgraded to FreeBSD ALPHA 12.0, but I don't think the errors them 
> from there. Although they could be in one of the libs that came along with 
> the upgrade.
> 
> I'm getting these warnings during rbd and ceph (maybe even more) invocations 
> that indicate that indicate a possible problem because:
> ===
>   It could be possible that a dynamically loaded library, use
>   thread_local variable but is dlclose()'d before thread exit.  The
>   destructor of this variable will then try to access the address,
>   for calling it but it's unloaded, so it'll crash.  We're using
>   __elf_phdr_match_addr() to detect and prevent such cases and so
>   prevent the crash.
> ===
> this is from : 
> https://github.com/freebsd/freebsd/blob/master/lib/libc/stdlib/cxa_thread_atexit_impl.c
> 
> Now it could be that dlclose() and thread exit are just closed to one 
> another. But still this is hard core embedded in libc already since 2017, so 
> I'm sort of expecting that a recent change has caused this.
> 
> And as indicated it is a possible cause for crashed, because thread_exit is 
> going to clean up things that are no longer there.
> 
> Now the 20 dollar question is:
>   Where was this introduced??
> 
> Otherwise I'll have to try and throw my best gdb capabilities at it, and try 
> to invoke an rbd call and see where it activates this warning.
> 
> --WjW
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: clang manual page?

2018-04-05 Thread David Chisnall
On 6 Apr 2018, at 01:30, Pete Wright  wrote:
> 
> 
> On 04/05/2018 17:15, Steve Kargl wrote:
>> This assumes that a gcc(1) is available on the system.
>> 
>> % man gcc
>> No manual entry for gcc
>> 
>> If the system compiler is clang/clang++, then it ought to be
>> documented better than it currently is.  Ian's suggests for
>> 'clang --help' is even worse
>> 
>> %  clang --help | grep -- -std
>>   -cl-std= OpenCL language standard to compile for.
>>   -std=Language standard to compile for
>>   -stdlib= C++ standard library to use
>> 
>> Does  == ?
>> 
> a quick google search turns up the following additional information:
> 
> "clang supports the -std option, which changes what language mode clang uses. 
> The supported modes for C are c89, gnu89, c99, gnu99, c11, gnu11, c17, gnu17, 
> and various aliases for those modes. If no -std option is specified, clang 
> defaults to gnu11 mode. Many C99 and C11 features are supported in earlier 
> modes as a conforming extension, with a warning. Use |-pedantic-errors| to 
> request an error if a feature from a later standard revision is used in an 
> earlier mode."
> 
> https://clang.llvm.org/docs/UsersManual.html

I believe that the clang user manual referenced here is written in Sphynx, 
which is able to generate mandoc output as well as HTML.  Perhaps we should 
incorporate the generated file in the next import?

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: atomic in i386 Current after CLANG 6 upgrade

2018-01-15 Thread David Chisnall
On 15 Jan 2018, at 17:00, Jan Beich  wrote:
> 
> It wouldn't help (see below). Clang 6 accidentally made __atomic* work
> enough to satisfy configure check but not for the port to build. I guess,
> it also confuses configure in net/librdkafka and net-mgmt/netdata.
> 

Can we (by which I probably mean emaste@) push out an EN that adds the atomic.c 
from compiler-rt to our libgcc_s?  That should provide all of these helper 
functions.  Clang assumes that they exist because both compiler-rt and vaguely 
recent libgcc_s provide them.  Recent GCC will also assume that they exist and 
so the correct fix is probably for us to make them to exist.

If this is difficult, then we can perhaps provide a port that compiles atomic.c 
into libatomic_fudge.so or similar and provides a libgcc_s.so that’s a linker 
script that forces linking to libatomic_fudge.so and libgcc_s.so.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: inconsistent for() and while() behavior when using floating point

2018-01-15 Thread David Chisnall
On 15 Jan 2018, at 14:49, Hans Petter Selasky  wrote:
> 
> The "seq" utility should use two 64-bit integers to represent the 10-base 
> decimal number instead of float/double. And then you need to step this pair 
> of integers.

As the saying goes:

> Sometimes, people think 'I have a problem and I will solve it with floating 
> point values' and then they have 1. problems.


David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [self base packages] pkg: packages for wrong OS version: FreeBSD:12:amd64

2018-01-11 Thread David Chisnall
On 10 Jan 2018, at 18:53, Baptiste Daroussin  wrote:
> 
> I need to figure out a mechanism to make this simpler to handle to upgrade of
> base system while keeping this safety belt for users.
> 
> Any idea is welcome

I believe the apt approach to this is to have a different verb (distupgrade vs 
upgrade) to perform complete version upgrades.  Ideally, the proper fix would 
probably be to depend on a base package version, rather than OSVERSION, and if 
the base packages are not being used to synthesise a phantom set of base 
package metadata based on OSVERSION.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Programmatically cache line

2018-01-05 Thread David Chisnall
On 5 Jan 2018, at 02:46, Jon Brawn  wrote:
> This idea of Arm big.LITTLE systems having cache lines of different lengths 
> really, really bothers me - how on earth is the cache coherency supposed to 
> work in such a system? I doubt the usual cache coherency protocols would work 
> - probably need a really MESSY protocol to deal with this config :-)

I believe that the systems that have different cache line sizes (which ARM 
explicitly tells partners not to do) don’t allow cores from both the big and 
little clusters to be active at the same time - the OS is supposed to migrate 
everything entirely from one cluster to the other.  The more complex designs, 
that allow mixes of cores from two or three different clusters that I’m aware 
of all have the same cache line size.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Programmatically cache line

2018-01-04 Thread David Chisnall
On 3 Jan 2018, at 22:12, Nathan Whitehorn  wrote:
> 
> On 01/03/18 13:37, Ed Schouten wrote:
>> 2018-01-01 11:36 GMT+01:00 Konstantin Belousov :
>> On x86, the CPUID instruction leaf 0x1 returns the information in
>> %ebx register.
> Hm, weird. Why don't we extend sysctl to include this info?
>>> For the same reason we do not provide a sysctl to add two integers.
>> I strongly agree with Kostik on this one. Why add stuff to the kernel,
>> if userspace is already capable of extracting this? Adding that stuff
>> to sysctl has the downside that it will effectively introduce yet
>> another FreeBSDism, whereas something generic already exists.
>> 
> 
> Well, kind of. The userspace version is platform-dependent and not always 
> available: for example, on PPC, you can't do this from userland and we 
> provide a sysctl machdep.cacheline_size to userland. It would be nice to have 
> an MI API.

On ARMv8, similarly, sometimes the kernel needs to advertise the wrong size.  A 
few big.LITTLE cores have 64-byte cache lines on one cluster and 32-byte on the 
other.  If you query the size from userspace while running on a 64-byte 
cluster, then issue the zero-cache-line instruction while migrated to the 
32-byte cluster, you only clear half the size.  Linux works around this by 
trapping and emulating the instruction to query the cache size and always 
reporting the size for the smallest cache lines.  ARM tells people not to build 
systems like this, but it doesn’t always stop them.  Trapping and emulating is 
much slower than just providing the information in a shared page, elf aux args 
vector, or even (often) a system call.

To give another example, Linux provides a very cheap way for a userspace 
process to enquire which core it’s running on.  Some more recent 
high-performance mallocs use this to have a second-layer per-core cache after 
the per-thread cache for free blocks.  Unlike the per-thread cache, the 
per-core cache does need a lock, but it’s very unlikely to be contended (it 
will only be contended if either a thread is migrated in between checking and 
locking, so acquires the wrong CPU’s lock, or if a thread is preempted in the 
middle of middle of the very brief fill operation).  The author of the 
SuperMalloc paper tried doing this with CPUID and found that it was slower by a 
sufficient margin to almost entirely offset the benefits of the extra layer of 
caching.  

Just because userspace can get at the information directly from the hardware 
doesn’t mean that this is the most efficient or best way for userspace to get 
at it.

Oh, and some of these things are useful in portable code, so having to write 
some assembly for every target to get information that the kernel already knows 
is wasteful.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Programmatically cache line

2018-01-01 Thread David Chisnall
On 1 Jan 2018, at 05:09, Adrian Chadd  wrote:
> 
> On 30 December 2017 at 00:28, Konstantin Belousov  wrote:
>> On Sat, Dec 30, 2017 at 07:50:19AM +, blubee blubeeme wrote:
>>> Is there some way to programmatically get the CPU cache line sizes on
>>> FreeBSD?
>> 
>> There are, all of them are MD.
>> 
>> On x86, the CPUID instruction leaf 0x1 returns the information in
>> %ebx register.
> 
> Hm, weird. Why don't we extend sysctl to include this info?

It would be nice to expose this kind of information via VDSO or similar.  There 
are a lot of similar bits of info that people want to use for ifunc and, SVE is 
going to have a bunch of similar requirements.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: libfuzzer support

2017-10-24 Thread David Chisnall
On 24 Oct 2017, at 10:29, Kurt Jaeger  wrote:
> 
>> 
>> is libfuzzer (see https://llvm.org/docs/LibFuzzer.html) supported on FreeBSD 
>> head?
>> It seems that it is not supported by /usr/bin/clang...
>> 
>> Am I wrong and it is supported or is someone working on it?
> 
> I searched in the Port devel/llvm50 and it looks like there's no support
> for libfuzzer right now.
> 
> But that would probably(?) be the best place to add it first.

It’s been a while, but I’m pretty sure I had libFuzzer working from the 
upstream sources on FreeBSD.  I found AFL (which also works on FreeBSD out of 
the box) more useful, so I didn’t spend much time with it.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: There is *NO* abi stability in -head

2017-10-24 Thread David Chisnall
On 23 Oct 2017, at 21:35, Mateusz Guzik  wrote:
> 
> Instead, the same can be reshuffled:
> struct crap2 {
>int i1;
>int i2;
>void *p1;
>void *p2;
> };
> 
> With offsets:
> 
> 0x1000 i1
> 0x1004 i2
> 0x1008 p1
> 0x1010 p2
> 
> This is only 24 bytes. 2 ints can be placed together and since they add
> up to 8 the p1 pointer gets the right alignment without extra padding.

If you are making changes of this nature, please consider sorting in the other 
order.  When we start seeing 128-bit pointers (which, with CHERI-like systems, 
may be sooner than you think) then this ordering will give you lots of padding, 
whereas putting the pointers first will not.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [RFC] future of drm1 in base

2017-09-03 Thread David Chisnall
On 3 Sep 2017, at 06:31, Cy Schubert  wrote:
> 
> Thanks for the heads up Johannes. I currently have three machines that each 
> run ATI r128, mach64 and the last one an mga card. I normally use my i945 
> and i915 laptops (mostly the former) but on occasion I may fire up X on one 
> of the other three. Having a drm-legacy port in the tree would benefit to 
> me.

It’s been quite a while since I used any of these (though much of the list is 
very familiar), but the last machine I ran with a mach64 card was faster with 
the vesa driver than with the ‘accelerated’ driver (as I recall, it was a 
500MHz Pentium III).  I suspect the real question is not whether people have 
machines that use these cards, but whether they do anything with them where 
they’d notice the lack of acceleration.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r322875 - head/sys/dev/nvme

2017-08-25 Thread David Chisnall
On 25 Aug 2017, at 07:32, Mark Millard  wrote:
> 
> As I remember _Static_assert is from C11, not
> the older C99.

In pre-C11 dialects of C, _Static_assert is an identifier reserved for the 
implementation.  sys/cdefs.h defines it to generate a zero-length array if the 
condition is true or a negative-length array if it is false, emulating the 
behaviour (though giving less helpful error messages)

> 
> As I understand head/sys/dev/nvme/nvme.h use by
> C++ code could now reject attempts to use
> _Static_assert .

In C++, _Static_assert is an identifier reserved for the implementation, but in 
C++11 or newer static_assert is a keyword.  sys/cdefs.h defines _Static_assert 
to static_assert for newer versions of C++ and defines it to the 
C-before-11-compatible version for C++-before-11.

TL;DR: We have gone to a lot of effort to ensure that these keywords work in 
all C/C++ dialects, please use them, please report bugs if you find a case 
where they don’t work.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: swapfile query

2017-08-19 Thread David Chisnall
On 19 Aug 2017, at 17:54, Cy Schubert  wrote:
> 
>> 3. should total swap be 1x 2x or some other multiple of RAM these days?
> 
> Depends. If you're running some kind of database server or OLTP 
> application. Some vendors recommend no swap whatsoever while others 
> recommend some. What does your application vendor recommend?

The main advantage of swap these days (on machines with that sort of amount of 
RAM) is to allow you to keep some file-backed memory objects in memory in 
preference to leaked (or very cold) heap memory.  

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [libelftc] internal library or not?

2017-08-01 Thread David Chisnall
On 1 Aug 2017, at 12:36, Michael Zhilin  wrote:
> 
> Hi Ed, freebsd-current,
> 
> I want to add C++ demangling into sysutils/pstack. In man pages I've found
> eltfc_demangle, exact what I need. This function belongs to libelftc. "make
> installworld" installs man pages and include files, but there is no
> installed library. As results compilation error is occuried.

Is pstack written in C++ or linking anything that is?  If so, you will get 
__cxa_demangle[1] provided by the C++ ABI library (libcxxrt on FreeBSD, which 
currently uses the libelftc implementation, though might switch soon).  If not, 
adding -lcxxrt will provide it.

David

[1] https://itanium-cxx-abi.github.io/cxx-abi/abi.html#demangler
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: libstdc++ build failures on MIPS, PowerPC, Sparc

2017-07-24 Thread David Chisnall
On 23 Jul 2017, at 23:54, Mark Millard  wrote:
> 
>>c++ -isystem ${OUTDIR}/tmp/usr/include/c++/v1 -std=c++11  -nostdinc++ 
>> -isystem ${OUTDIR}/tmp/usr/include -L${OUTDIR}/tmp/usr/lib 
>> -B${OUTDIR}/tmp/usr/lib --sysroot=${OUTDIR}/tmp -B${OUTDIR}/tmp/usr/bin  -O 
>> -pipe -G0 -EB -mabi=32 -msoft-float  -DIN_GLIBCPP_V3 -DHAVE_CONFIG_H 
>> -I${SRCDIR}/gnu/lib/libstdc++ -I${SRCDIR}/contrib/libstdc++/libsupc++ 
>> -I${SRCDIR}/contrib/gcc -I${SRCDIR}/contrib/libstdc++/include 
>> -I${SRCDIR}/contrib/gcclibs/include -I${SRCDIR}/contrib/libstdc++/include 
>> -I. -frandom-seed=RepeatabilityConsideredGood -fno-implicit-templates 
>> -ffunction-sections -fdata-sections -Wno-deprecated   -c 
>> ${SRCDIR}/contrib/libstdc++/src/bitmap_allocator.cc -o bitmap_allocator.o

This is quite a surprising build command.  It’s using usr/include/c++/v1 for 
system includes, but usr/include/c++/v1 is the libc++ header directory.  
libstdc++ shouldn’t need to be built with C++11 support, but libc++ does, so 
this command looks like a combination of both libc++ and libstdc++ build flags 
all mashed together.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Getting PID of socket client

2017-07-09 Thread David Chisnall
On 9 Jul 2017, at 14:25, Stefan Ehmann  wrote:
> 
> Don't why the structs are not compatible, maybe because:
> "The process ID cmcred_pid should not be looked up (such as via the
> KERN_PROC_PID sysctl) for making security decisions.  The sending process 
> could have exited and its process ID already been reused for a new process."

Note that having the kernel provide a process descriptor instead of a PID would 
allow the userspace process to have race-free access to the PID.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [libltdl] removal from gnu ports

2017-06-07 Thread David Chisnall
On 7 Jun 2017, at 10:33, blubee blubeeme  wrote:
> 
> Hi
> 
> I'm sure I was reading yesterday on a different machine about the linker
> flag -ld which has something to do with gnu dlopen and how it's ok to
> remove those from your Makefile since FreeBSD handles dlopen and a few
> other things from that header in the standard libc.
> 
> Is that correct?

Do you mean -ldl?  If so, then yes.  On Linux, the dl* symbols are only 
exported from ld-linux.so if you link against libdl.  On FreeBSD, they are 
exported from rtld regardless.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: build src with colored output?

2017-05-16 Thread David Chisnall
On 16 May 2017, at 07:42, Johannes Lundberg  wrote:
> 
> Gonna answer myself here. Think I found a way.
> 
> Add CFLAGS=-fcolor-diagnostics to env or /etc/src.conf
> Do clean build, that is no -DNO_CLEAN,KERNFAST, etc.
> 
> Makes it a lot easier to find the errors in a 16 threads build output...
> 
> The mystery still remains though, why is color disabled for parallel
> builds?

It’s disabled for two reasons.  The first is aesthetic - some people don’t like 
coloured output.  I’m not going to debate that one.  The other is technical.  
Unlike modern build tools, such as Ninja, bmake’s handling of multithreaded 
output is very bad.  It simply allows each task the same output device, whereas 
ninja gives each parallel job a pipe back to the build process and then merges 
the output itself.  This means that you periodically encounter the case where 
one child process has sent a colour escape sequence to the output and then 
another process sends the next line, giving weird visual effects and reducing 
the utility of colour outputs.

Ideally, we’d solve this by fixing bmake to behave more like a modern build 
tool and:

 - Giving each sub-process its own pipe.
 - Emitting the full compile command for all failed tasks.
 - Displaying only a summary for successful commands

Or we could find someone with the time to spend giving FreeBSD a modern build 
system, which would probably save us 1-2 man years of developer time each year 
overall.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: libstd++ missing in drm-next-4.7

2017-01-23 Thread David Chisnall
On 20 Jan 2017, at 14:06, Christian Schwarz  wrote:
> 
> I know clang is no longer built and installed as part of buildworld  in
> the FreeBSDDesktop repo,
> but why isn't libstd++ present?
> 
> …
> 
>  clang39 # now run clang, won't work, see below
>  Shared object "libc++.so.1" not found, required by "clang"

For clarification, are you missing libc++ or libstdc++?  libstdc++ hasn’t been 
a default part of the base system since 10.0 for x86 architectures.

David



smime.p7s
Description: S/MIME cryptographic signature


Re: vt(4) chops off the leftmost three columns

2017-01-14 Thread David Chisnall
On 13 Jan 2017, at 01:00, Ernie Luzar  wrote:
> 
> VT should have had better testing before becoming the default in 11.0.

The choice was VT or no acceleration in X11, because all of the new DRI drivers 
depend on KMS, which requires VT.  We only got VT in a useable state (and 
therefore useable accelerated X11 for most of the 10.x series) because the 
Foundation funded the work.  More contributions to VT are always welcome, but 
the choice was not between VT or something better tested, it was between VT and 
black screens.

David



smime.p7s
Description: S/MIME cryptographic signature


Re: Is there possible run a MacOS X binary

2016-12-07 Thread David Chisnall
On 5 Dec 2016, at 19:31, Kevin P. Neal  wrote:
> 
>> Is there any emulator like linuxator to run Mac OS X binaries, or 
>> is ther any licensing problem?
> 
> It may be possible to make an emulator for Darwin (the OS that Mac OS sits
> on top of), but an emulator for Mac OS would probably require a legal copy
> of Mac OS.
> 

NetBSD started working on one, and had it at a state where it could run the 
Darwin version of XFree86 (which should give you some idea of how old it was), 
but it couldn’t run the Mac Window Server and I don’t think it’s maintained. 
It’s not very interesting, because you need all of the frameworks and programs 
from OS X for it to be useful, and the license prohibits using them in such a 
way (and even if you did get it to work, you wouldn’t be able to run Mac apps 
at the same time as X apps without running XQuartz, and at that point you may 
as well just run macOS).

There is a more recent project called Darling (https://www.darlinghq.org) that 
tries to run Mac apps on Linux using a custom Mach-O loader, bits of GNUstep, 
and some of their own stuff.  No one has ever tried using it on FreeBSD, to the 
best of my knowledge.  It’s GPLv3, so I’m not motivated to contribute to it 
(though it does incorporate some of my code in various places).

David



smime.p7s
Description: S/MIME cryptographic signature


Re: Optimising generated rules for SAT solving (5/12 are duplicates)

2016-11-29 Thread David Chisnall
On 23 Nov 2016, at 18:11, A. Wilcox  wrote:
> 
> Or you could just, I don't know, email the diff as a patch using git
> send-email like normal people instead of using GitHub's walled garden.
> That way, people without GitHub accounts can still comment on it.

GitHub pull requests are branches in the recipient’s git repo.  Anyone can see 
the patch without logging in, either via the web interface or by pulling the 
relevant branch.  If you want to send comments via email based on this copy of 
the patch, then that’s up to you, though personally I’d much prefer the GitHub 
code review interface to anything email based.

David



smime.p7s
Description: S/MIME cryptographic signature


Re: how to reduce the size of /usr/share/i18n data?

2016-11-03 Thread David Chisnall
On 3 Nov 2016, at 19:34, Rodney W. Grimes  
wrote:
> 
> Depressing fact:  The i18n directory alone is larger than
> a full 386BSD 0.1+sources+patchkit install.

Is the depressing thing here that even something as recent as 386BSD 0.1 
assumed that ASCII was enough for the whole world?

David



smime.p7s
Description: S/MIME cryptographic signature


Re: Call for Testing: Switching back to our BSD licensed dtc(1)

2016-09-08 Thread David Chisnall
On 24 Jul 2016, at 12:42, David Chisnall <thera...@freebsd.org> wrote:
> 
> I’ve now fixed both of these in the version here:
> 
> https://github.com/davidchisnall/dtc

Andy filed a number of issues on GitHub, which are now all fixed.  In 
particular, /include/ should now work everywhere and /delete-node/ now works.

I’ve added a CMake build system and tests to the version on GitHub, which 
should make it easier for people to build out of tree (for example, in a port).

Please file issues as you find them.  I hope to look at the overlay stuff soon. 
 I believe that the support that it requires from dtc is relatively small.

I hope to find time to look at overlay support soon,

David



smime.p7s
Description: S/MIME cryptographic signature


Re: Wayland work status

2016-08-12 Thread David Chisnall
On 12 Aug 2016, at 00:18, Lundberg, Johannes  
wrote:
> 
> Currently by default evdev create /dev/input/eventX devices with 600
> permission. These need to be accessible for non-root users. What is the
> best solution? Should we create a "input" group similar to "video" group is
> being used for rights to access /dev/drm devices?

There is a similar problem for the drm devices (by default, users can’t use 3D 
acceleration).  A devfs.conf policy can change the permissions.  I’d suggest 
that we create a default group called something like console or local, put new 
users there by default, and make drm and evdev devices accessible by this group.

David



smime.p7s
Description: S/MIME cryptographic signature


Re: SafeStack in base

2016-07-28 Thread David Chisnall
On 27 Jul 2016, at 23:55, Shawn Webb  wrote:
> 
> I'm interested in getting SafeStack working in FreeBSD base. Below is a
> link to a simplistic (maybe too simplistic?) patch to enable SafeStack.
> The patch applies against HardenedBSD's hardened/current/master branch.
> Given how simple the patch is, it'd be extremely easy to port over to
> FreeBSD (just line numbers would change).

We’ve worked with the authors of the SafeStack work.  There are some changes to 
libc and a few other support libraries needed for it to work, which are in the 
GitHub repository.  They’ve also done some work to address issues of things 
like Firefox and v8 that need to be able to walk the stack, allocate their own 
stacks for userspace threads, and so on.

It was not enabled for FreeBSD 11 because SafeStack imposes a lot of long-term 
ABI constraints that it’s not clear we want to support indefinitely given the 
‘Missing the point(er)’ Oakland paper last year.  It does increase the work 
factor for attackers, so has some security benefit, but if bypassing it is 
something that’s going to be added to exploit toolkits then it’s little 
practical benefit.

One middle-ground that we’ve considered is only supporting it for statically 
linked binaries.  This absolves us of the need to support the ABI indefinitely, 
and still provides a lot of the benefit.

David



smime.p7s
Description: S/MIME cryptographic signature


Re: Call for Testing: Switching back to our BSD licensed dtc(1)

2016-07-24 Thread David Chisnall
Thanks,

On 19 Jul 2016, at 20:49, Emmanuel Vadot  wrote:
> 
> 
> Hello,
> 
> I've just tried bsd dtc on all arm dts that we have.
> It doesn't seems to handle multiple include directories.
> Here is how to reproduce :
> 
> $ export SRCROOT=/path/to/fbsd/src
> $ export MACHINE=arm
> $ cd $SRCROOT/sys/boot/fdt/dts/arm
> $ $SRCROOT/sys/tools/fdt/make_dtb.sh $SRCROOT/sys
> beaglebone-black.dts . converting beaglebone-black.dts
> -> ./beaglebone-black.dtb Unable to open file
> '/home/elbarto/Work/freebsd/freebsd.git//sys/boot/fdt/dts/arm/am33xx-clocks.dtsi'.
> No such file or directory Unable to open file
> '/home/elbarto/Work/freebsd/freebsd.git//sys/boot/fdt/dts/arm/tps65217.dtsi'.
> No such file or directory
> 
> Both dtsi files are include with /include/ (i.e. not handled by cpp).
> make_dtb.sh specify : -i $S/boot/fdt/dts/${MACHINE} -i
> $S/gnu/dts/${MACHINE}, it looks like the second one isn't added to the
> list.

It actually is added to the list and found.  The bug here is in error reporting 
- it was reporting an error for each file that it tried to open but couldn’t, 
even when it subsequently found the correct file.

> 
> Trying tegra124-jetson-tk1-fbsd.dts give this :
> converting tegra124-jetson-tk1-fbsd.dts
> -> /tmp/bsd_dtb//tegra124-jetson-tk1-fbsd.dtb Error on line 1214:
> Expected node name interrupt-affinity = <&{/cpus/cpu@0}>,
> ^
> Error on line 1214: Expected ; at end of property
>  interrupt-affinity = <&{/cpus/cpu@0}>,
> ^
> Failed to parse tree.  Unhappy face!

There was a FIXME relating to this in the code.  

I’ve now fixed both of these in the version here:

https://github.com/davidchisnall/dtc

I’ll push the changes to FreeBSD svn soon, but in the meantime anyone wanting 
to test can just clone that repo over the dtc directory in their src checkout.

David



smime.p7s
Description: S/MIME cryptographic signature


Re: Call for Testing: Switching back to our BSD licensed dtc(1)

2016-07-23 Thread David Chisnall
On 23 Jul 2016, at 05:16, Warner Losh <i...@bsdimp.com> wrote:
> 
> On Fri, Jul 22, 2016 at 1:03 AM, David Chisnall <thera...@freebsd.org> wrote:
>> On 22 Jul 2016, at 03:40, Warner Losh <i...@bsdimp.com> wrote:
>>> 
>>> On Wed, Jul 20, 2016 at 9:51 AM, David Chisnall <thera...@freebsd.org> 
>>> wrote:
>>>> On 20 Jul 2016, at 16:46, Warner Losh <i...@bsdimp.com> wrote:
>>>>> 
>>>>> I've been trying to get the final spec for it. Right now it's a
>>>>> disorganized series
>>>>> of patches, some of which have been merge some that haven't. I'll send 
>>>>> you a
>>>>> copy when I can find something better than "here's the code."
>>>> 
>>>> Thanks.  From the information I can find, it looks as if most of the 
>>>> machinery required to implement it is already in dtc, so it should 
>>>> (hopefully) just be a matter of adding a new keyword to detect plugins, a 
>>>> scan to find the cross-references (or possibly reusing the existing one) 
>>>> and then a little bit of extra logic.
>>> 
>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/overlay-notes.txt
>>> has the specs.
>> 
>> Hmm, that’s even less complete than the docs that I’d found.
> 
> Can you share that then?

I only found tutorial-style things:

https://www.raspberrypi.org/documentation/configuration/device-tree.md

https://learn.adafruit.com/introduction-to-the-beaglebone-black-device-tree/device-tree-overlays

I was hoping for something a bit more like a spec.


>> From that, it seems as if the only thing that dtc needs to do is support the 
>> /plugin/ syntax and emit a section describing unresolved references?
> 
> I believe so, yes.

That sounds like it should be easy.

> 
>> Or is dtc also expected to be able to do the merging?
> 
> I think that's TBD. We'll need, at the very least, an update libfdt
> from upstream that knows how to do the merging, as well as changes to
> /boot/loader to be able to pick and choose which plugins to add to the
> base. If we can do all that with the BSDL DTC and it passes all the
> other GPL test cases, then we may have a winner and we can get started
> integrating plugin support to /boot/loader. I know my RPi would be
> happier if it had a 'standard' DTB with a plugin for whatever 1-wire
> stuff I'm playing with today.

Patrick Wildt was running the GPL’d dtc test suite and I fixed all of the 
things that he reported broken.  We’re now using a less-efficient algorithm for 
calculating the cross-references so that we resolve things in the same order, 
which makes doing a diff on the dts produced by the two tools easier.

David



smime.p7s
Description: S/MIME cryptographic signature


Re: Call for Testing: Switching back to our BSD licensed dtc(1)

2016-07-22 Thread David Chisnall
On 22 Jul 2016, at 03:40, Warner Losh <i...@bsdimp.com> wrote:
> 
> On Wed, Jul 20, 2016 at 9:51 AM, David Chisnall <thera...@freebsd.org> wrote:
>> On 20 Jul 2016, at 16:46, Warner Losh <i...@bsdimp.com> wrote:
>>> 
>>> I've been trying to get the final spec for it. Right now it's a
>>> disorganized series
>>> of patches, some of which have been merge some that haven't. I'll send you a
>>> copy when I can find something better than "here's the code."
>> 
>> Thanks.  From the information I can find, it looks as if most of the 
>> machinery required to implement it is already in dtc, so it should 
>> (hopefully) just be a matter of adding a new keyword to detect plugins, a 
>> scan to find the cross-references (or possibly reusing the existing one) and 
>> then a little bit of extra logic.
> 
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/devicetree/overlay-notes.txt
> has the specs.

Hmm, that’s even less complete than the docs that I’d found.  From that, it 
seems as if the only thing that dtc needs to do is support the /plugin/ syntax 
and emit a section describing unresolved references?  Or is dtc also expected 
to be able to do the merging?

David




smime.p7s
Description: S/MIME cryptographic signature


Re: Call for Testing: Switching back to our BSD licensed dtc(1)

2016-07-20 Thread David Chisnall
On 20 Jul 2016, at 16:46, Warner Losh  wrote:
> 
> I've been trying to get the final spec for it. Right now it's a
> disorganized series
> of patches, some of which have been merge some that haven't. I'll send you a
> copy when I can find something better than "here's the code."

Thanks.  From the information I can find, it looks as if most of the machinery 
required to implement it is already in dtc, so it should (hopefully) just be a 
matter of adding a new keyword to detect plugins, a scan to find the 
cross-references (or possibly reusing the existing one) and then a little bit 
of extra logic.

David



smime.p7s
Description: S/MIME cryptographic signature


Re: Call for Testing: Switching back to our BSD licensed dtc(1)

2016-07-20 Thread David Chisnall
On 20 Jul 2016, at 05:48, Warner Losh  wrote:
> 
> My concern with this is overlay support. Overlay support for "shields" or
> "badges" is going into upstream GPL dtc shortly. The patches have been
> kicking around for a while and are almost ready to go in. We need this feature
> and I'd planned on bringing it into the tree as soon as it goes into the git
> repo. It's sorely needed for things like RPi and BBB that make it super
> easy to add boards with parts that need a bit more DTB into the FDT to
> describe them. There's been patches to bring this functionality into the
> loader and into the in-tree gpl dtc floating around for a while now.

Is the syntax / semantics for overlays documented somewhere?

David



smime.p7s
Description: S/MIME cryptographic signature


Re: FreeBSD-11.0-BETA1-amd64-disc1.iso is too big for my 700MB CD-r

2016-07-13 Thread David Chisnall
On 13 Jul 2016, at 10:17, O. Hartmann  wrote:
> 
> A CD is still a used media, but it starts getting squeezy on it as certain
> software starts to grow - as CLANG/LLVM does. Maybe it is time to have also 
> CDs
> as "miniboot" and DVDs for a more complete installation media?

I completely agree.  If you’re installing somewhere that’s not firewalled off 
completely from the Internet, then it’s typically faster to boot the bootonly 
ISO and then download the rest on the target machine (from a local mirror if 
necessary).  Even 100Mb/s ethernet is faster than most CD drives.  If you need 
offline installs, then the DVD with a bunch of other packages on it is probably 
what you want, not a CD that just contains the base system.

Even if you can’t boot from USB (as I can’t on one of my FreeBSD machines), it 
should be possible to stick the base distributions and a package repo on a USB 
stick, use the boot-only ISO to boot and then install from USB.

David



smime.p7s
Description: S/MIME cryptographic signature


Re: CURRENT: bhyve and Kernel SamePage Mergin

2016-06-09 Thread David Chisnall
If this paper is the one that I think it is, then I was one of the reviewers.  
Their attack is neat, but it depends quite a lot on being able to 
deterministically trigger deduplication.  Their proof-of-concept exploit was on 
Windows (and JavaScript attack was really fun) and I’m not convinced that it 
would work reliably on Linux or VMWare ESX, which both defer deduplication for 
as long as possible to avoid NUMA-related overheads.

We don’t currently have a FreeBSD implementation, but if someone wanted to 
provide one then a defence against this attack would be fairly simple: count 
the number of CoW faults that a process is receiving and if it reaches a 
certain threshold then remove all of its memory from the set of eligible pages. 
 The attack relies on being able to repeatedly trigger CoW faults and time 
whether they occur, with the same set of pages.  At least some existing 
implementations will make this impossible as these pages will repeatedly be 
deduplicated and then duplicated and this is already a pathological case that 
most memory deduplication implementations need to handle (as well as being a 
security hole, it’s also a big performance killer).

Kib has been working on ASLR for FreeBSD (I think it’s in 11?), but at this 
point it’s more of a checkbox item than a serious mitigation technique.  It 
adds a little bit of work for attackers, but there are so many attacks that can 
bypass ASLR even with strong entropy that it just increases the work factor 
slightly.  If you’re running code written in C, then you’re better off relying 
on Capscium compartmentalisation to limit what the attacker can do once they’ve 
compromised it.

David

> On 8 Jun 2016, at 16:01, O. Hartmann  wrote:
> 
> A couple of days I got as a responsible personell for a couple of systems a 
> warning about
> the vulnerabilities of the mechanism called "Kernel SamePage Mergin". On this 
> year's IEEE
> symposion there has been submitted a paper by Bosman et al., 2016, describing 
> an attack
> on KSM. This technique, also referred to as memory/page deduplication, seems 
> to be
> vulnerable by design under certain circumstances. I guess the experts of the 
> readers here
> do already know, but I consider myself a non-expert and therefore, I'd like 
> to ask about
> the status of that kind of development in FreeBSD. I read about a project of 
> last year's
> Google Summer of Code 2015 targetting KSM on FreeBSD.
> 
> In Linux, this deduplication techniques is implemented since kernel 2.6.38 
> and Windows
> Kernel uses this techniques since Windows 8.1 and sibblings (also Windows 
> Server). We
> were strongly advised to disable those "features" in Windows clients, servers 
> and Linux
> servers, if used.
> 
> Other papers describe successful attacks on memory contents and ASLR by 
> misusing KSM. On
> Windows, mmap() entropy is 19bit, on Linux usually 28bit. And FreeBSD (if
> planned/used/already implemented?)? 
> 
> If you are interested I could provide links or PDFs of the papers I already 
> gathered
> about that subject (it is not much, simply google for "KSM FReeBSD" or KSM 
> deduplication
> ASLR).
> 
> Thanks in advance,
> 
> oh



smime.p7s
Description: S/MIME cryptographic signature


Re: why 100 packages are evil

2016-04-25 Thread David Chisnall
On 25 Apr 2016, at 06:48, Gerrit Kühn  wrote:
> 
>> Yes.  It will be replaced by 'pkg upgrade' -- as far as I know, that's
>> the plan for 11.0-RELEASE.
> 
> Hm... I never had any troubles with freebsd-update, it always "just
> worked" for me. OTOH, I remember having several issues with pkg, requiring
> to fix databases manually and so on.

There are two kinds of issues with freebsd-update:

The first is fairly common: it’s slow and creates a lot of files.  If you read 
the forums, you’ll find a lot of issues about this.  Updates from one 
patchlevel to the next are pretty straightforward, but on both the VMs that I 
use for FreeBSD and a slow AMD machine with ZFS it takes around an hour 
(sometimes more) for freebsd-update to jump one major release.  After that, it 
takes pkg a minute or two to update the 2-3GB of packages that need upgrading.  
Minor releases can often take tens of minutes on these system.

The many files issue can cause inode exhaustion.  One one machine, I just 
checked and have 20K files in /var/db/freebsd-update/files.  If you’re using 
UFS for /var, it’s fairly easy for freebsd-update to run out of inodes.  Trying 
to recover a FreeBSD system that can no longer create files in /var is not the 
most fun uses of my time.

The second issue is that it sometimes just fails to work.  I have twice had 
freebsd-update manage to become confused about versions and install a kernel 
that couldn’t read the filesystem.  I’ve had similar confusion where (on a box 
that I administer mostly via the network and where physical access is a pain) 
had it install a version of ifconfig from an older userland than the kernel.  
These are all on machines where freebsd-update has been responsible for every 
upgrade after the initial install from CD.  Most depressingly, it spends ages 
doing checksums of every file in the system, determines that they don’t match 
the expected ones, and then installs the wrong one anyway.

I have been using the testing versions of pkg on most FreeBSD machines since it 
became available.  Since pkg 1.0 was released, I have had far fewer issues with 
pkg than with freebsd-update and almost all of those were to do with poor 
information in the ports tree and the rest were either UI or performance 
issues.  We have a lot tighter control over the packaging metadata for the base 
system.

David



smime.p7s
Description: S/MIME cryptographic signature


Re: [CFT] packaging the base system with pkg(8)

2016-04-22 Thread David Chisnall
On 21 Apr 2016, at 21:48, Dan Partelly  wrote:
> 
> Yes, you are right it misses the media change handler  in devd.conf. 
> maybe it should bementioned somewhere in a man page if it is not
> already there. Thanks for the pointer.
> 
> Anyway, if I would have written the system, what I would have done
> is to consolidate both user mode daemons into one and make this
> daemon a client of devd, fstyp a library, and handle all removable 
> media inside transparent to the user, without requiring any modifications
> to devd.conf, and without relaying on shell scripts. 
> 
> Is there any reason you did not took this approach ? This is not
> criticism, I am genuinely interested.

One of the current shortcomings of devd is that it does not provide a good 
mechanism for other running processes to request notification of events 
dynamically.  Ideally, when the automounter daemon starts, it should connect to 
devd via an IPC channel and request notification of the specific events that it 
wants.  This is a known problem (which extends to more than just the 
automounter) and there are some tentative plans to fix it, but they’re not yet 
concrete and won’t be in 11.0, though hopefully will appear at some point in 
the 11.x series.

David



smime.p7s
Description: S/MIME cryptographic signature


Re: [CFT] packaging the base system with pkg(8)

2016-04-20 Thread David Chisnall
On 20 Apr 2016, at 15:53, Paul Mather  wrote:
> 
> Arguably, a packaged base will make it easier to help people, because it 
> makes more explicit the dependencies of different parts of the system.  It's 
> been my experience that the interactions and impact of the various 
> /etc/src.conf settings are not entirely well known, at least to end-users.

In particular, with libxo output from pkg, we can get a machine-readable 
detailed dump of all of the base system packages installed and provide a single 
‘system dump’ command that includes this, relevant information from dmesg, and 
so on for bug triaging.

David



smime.p7s
Description: S/MIME cryptographic signature


Re: [CFT] packaging the base system with pkg(8)

2016-04-20 Thread David Chisnall
On 20 Apr 2016, at 06:06, Julian Elischer  wrote:
> 
> my problem with 400 packages is that is is hard to decide what you are 
> actually running.. or is it FreeBSD 11? is it FreeBSD 10.95342453?
> you have no way to tell exactly what you have without comparing all the 
> packages to a known list.
> uname doesn't mean much, nor does "__FreeBSD_version" if everything comes 
> with its own versions.

I think that it’s very important, for the purpose of a constructive discussion, 
to separate the two concerns:

1) The number of packages that the base system has.
2) The user interface by which the packages are presented.

I believe (and, please, correct me if I’m wrong), that all of the complaints in 
this thread have been about the UI, not about the underlying mechanism.  That’s 
not to say that they’re unimportant (quite the reverse), but that they can be 
solved concurrently with the task of preparing the base system for distribution 
in packaged form.

Having fine-grained packages makes a lot of things possible that are difficult 
otherwise, but we do need to fix the UI.


> the 'leaf' concept in pkg helps with this a bit, but we've always considered 
> FreeBSD bas as a sort of monalithic entity that moves forward together.
> 
> you are running 10.1p8 pr 10.2p1  that tells you all you need to know.
> If you now need to take into account 400 different dimensions you have a much 
> harder way to describe what you have..
> 
> I mentioned this before  but I think hte answer is to make a change on the 
> way that "meta packages" are displayed by default in pkg.

Part of the problem is that we don’t actually have metapackages.  A metapackage 
is a package that *contains* other packages.  What we actually have is empty 
packages that *depend on* other packages.  The package tool has no way of 
distinguishing a package that you install for the sole purpose of installing 
its dependencies from one that you install because you want it (though having 
no files inside it might serve as an heuristic that would work).

> If I install the meta package, I really don't want to see all the sub 
> packages tat are unchanged unless I add '-v'.  On the other hand if I upgrade 
> a sub package I want to see that in the context of the metapackage. Similarly 
> if I uninstall of the subpackages.

Doing this properly also requires the notion of optional default and 
non-default subpackages.  I should be prevented from uninstalling (at least, 
without a lot of -f) non-optional subpackages.  For example, on a small system 
where I’m not using zfs, I might uninstall the libzfs subpackage from 
freebsd-libs, but if I try to uninstall the libc package then the system should 
shout at me.

> 
> so something like this would remove most of my objections:
> 
> # pkg info
> = system packages
> FreeBSD-networking-11.0.2_1FreeBSD networking subsystem and 
> commands
> - ipfw-11.0.2-1   ipfw tools (uninstalled)
> - fbsd-tcpdump-11.0.2-1   Built in tcpdump tools (uninstalled)
> * openssl-11.0.2-2Openssl support (upgraded CVE-123456
> FreeBSD-base-base-11.0.2-1 The absolute minimum booting base 
> system
> [...]
>  external packages ==
> apache22-2.2.31Version 2.2.x of Apache web server with 
> prefork MPM.
> apr-1.5.2.1.5.4Apache Portability Library
> autoconf-2.69  Automatically configure source code on many 
> Un*x platforms
> autoconf-wrapper-20131203  Wrapper script for GNU autoconf
> [...]
> 
> 
> Maybe I uninstalled ipfw because I use pf and I install the ports tcpdump so 
> I can remove the built in one.
> I have installed a new openssl due to a bugfix..
> 
> This gives me a real instant feel for what I'm running..
> if I add -v  then I see all 400 packages, but I really don't want to see them 
> 99.99% of the time
> 
> I believe the "leaf" method gives close to this but if we could get the above 
> I'd have absolutely no objections.

Thank you for this suggestion.  I think that this is the sort of UI that makes 
a lot of sense (though having subpackage support would also be useful for 
ports).  It’s also the kind of thing that I think we could persuade the 
Foundation to fund if there is not enough volunteer time to implement it.

David



smime.p7s
Description: S/MIME cryptographic signature


Re: [CFT] packaging the base system with pkg(8)

2016-04-19 Thread David Chisnall
On 19 Apr 2016, at 08:44, Julian Elischer  wrote:
> 
>> All this can be done by meta-packages which depend on larger package groups.
> Currently Metapackage is a way to make 10 packages look like 11 packages.  
> The framework needs to understand to hide the 10 internal packages if they 
> are part of a metapackage.

I agree, and patches to do this are very welcome.  Currently, pkg is short of 
contributors.

I see basically three use cases for a packaged base:

1) People wanting a FreeBSD install to use as a server or workstation.  These 
people will install the FreeBSD 11 metapackage and not care that it is a few 
hundred MBs.  It would be nice if the pkg tool could present this as a single 
package in list views, but that’s a UI issue with pkg, not an issue with the 
number of packages in the base system.

2) People wanting to install embedded systems.  Anyone who has tried to run 
FreeBSD on a system with a small amount of flash storage will have encountered 
the pain of having to use some kind of ad-hoc update.  Being able to manage 
updates to these systems with the same packaging tool as you manage big systems 
is a big improvement.

3) People wanting to install service jails (sorry, containerised applications). 
 These want the smallest possible attack surface and so want the smallest 
amount of the base system that they can have.  Here, small packages are an 
advantage.  It will take a little while for ports to learn enough about the 
granularity of the base system for this to really be useful, but it would be 
great to be able to install nginx, for example, in a jail and have only the 
handful of libraries that it needs.

The big advantage of going with small packages initially, however, is that it 
will allow us to get some data on what the correct groupings are.  If we have 
large packages, then it’s very hard to tell which subsets of the packages 
people want.  That’s exactly the situation that we’re in now: we know some 
people don’t want docs or games, but that’s about all that we know.  It’s easy 
to move to a model where we have *fewer* packages in the future, but it’s 
harder to split them.  That also applies to dependencies.  If I know that a 
port depends on the shell, then it’s easy to update it from depending on a sh 
package to depending on a core system utilities package automatically, but it’s 
very hard to do an automatic update in the other direction.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Packaging the FreeBSD base system with pkg(8)

2016-04-05 Thread David Chisnall
On 5 Apr 2016, at 10:07, Gergely Czuczy  wrote:
> 
> Also, quite often entries from the base system are changed manually, think of 
> root's/toor's password. Are such cases going to be dealt with properly 
> between upgrades, including self-built-and-packaged base systems? Currently 
> it can be a PITA with mergemaster to handle things like master.passwd 
> properly between upgrades, automation so far wasn't famous on doing it 
> properly.

Mergemaster uses a 2-way merge.  It has the version that you have installed and 
the version that’s being proposed for installation.  Etcupdate and pkg perform 
a 3-way merge.  It has the pristine version, the version that you have made 
changes to, and the new version.  If you have changed an entry and so has the 
package, then you will get a conflict that you have to resolve manually.  If 
you have added lines and so has the upstream version, then that should cleanly 
apply.  Similarly, if you and upstream have both modified different lines, then 
there should be no problem.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread David Chisnall
On 24 Mar 2016, at 13:42, Damjan Jovanovic  wrote:
> 
> ELF itself is a disaster. Symbol lookup in ELF is process scoped, not
> library scoped like Windows's PE and Mac's Mach-O, so same named
> symbols from different libraries in the same process (loaded through
> any number of levels of indirection) can and do clash, resulting in
> memory corruption. This is why hacks like symbol versioning,
> RTLD_DEEPBIND on GNU's libc and -Bdirect on Solaris were invented.

This problem is addressed by some of the work that Sony has done recently that 
they are about to upstream to Clang/LLVM.

> We suffer from this problem badly on FreeBSD, as Clang's C++ standard
> library and GCC's standard library don't have fully compatible ABIs,
> so when both are loaded into the same process and the incompatible C++
> features are used -> memory corruption -> crash. Eg. compile Apache
> OpenOffice with GCC on a system built with Clang, and you'll see even
> the unit tests crash.

That shouldn’t happen, as libstd++ and libc++ have different symbols (libc++ 
puts its symbols in the __v1 namespace).  The problem can come from mixing 
libsupc++ and libcxxrt, but that’s only an issue if you have not built 
libstdc++ against libcxxrt.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread David Chisnall
On 24 Mar 2016, at 12:05, mokhi  wrote:
> 
> Hi.
> 
> I'm agreed with point you told about improvements we can do for fat
> format (or more).
> And I'm ready to do them (with your helps, sure :D).
> 
> But we need short steps and more of them (a local proverb :D) IMO.
> If we completely do this image activator, then we can have 2 sub plans
> for OSX emulation and/or fat data segment redesign.

FatELF binaries do not depend on this work.  Fat Mach-O binaries do, but the 
pain of working with Mach-O is not worth it (talk to some of the Apple 
toolchain team some time about how much effort Mach-O is - I’m glad it’s their 
problem and not mine).

I don’t believe that the work to support FatELF would be particularly large.  
The format is pretty simple (basically a small header that tells you where 
within the binaries to find the real ELF for your architecture).  Teaching all 
of the associated bits of the toolchain (especially debuggers) about it is a 
lot of tedious work, but not particularly hard if someone is motivated to do 
it.  Teaching clang and lld to produce fat binaries as part of normal ELF 
compilation would be a bit more work.

> I saw netbsd's way of mach-kernel/darwin emulation.
> They have been stopped in porting/simulating quartz (the reason
> described lack of developers' interest IIRC), and that relates to OSX
> emulating.

That wasn’t the only reason.  The XNU kernel interfaces for graphics and sound 
are large and mostly undocumented (at least, publicly) and change between OS X 
revisions.  Even if you implement *all* of this, then you’d still need most of 
an OS X userland to be able to run OS X applications.  This would involve 
violating the OS X EULA unless you ran it on a Mac and the only thing that 
you’d then be able to do that you couldn’t with OS X is run FreeBSD binaries in 
the background or in XQuartz (which you can already do pretty well with xhyve 
on OS X).  If you are willing to violate the OS X EULA then you should probably 
just run OS X in a VM.

> If we wanna complete/continue that way, first we need this image
> activator, what's your opinion about it?
> 
> BTW, in brief I believe we can have strategies to do (sub plans) and
> it worth (at least for me, because I'll learn good things). What's
> your opinion?

As a learning exercise, I definitely encourage you to continue.  Writing a new 
image activator will teach you a lot.  If you want to do some of the rtld work 
to make a partial Mach-O rtld then you’ll learn even more.  I just don’t think 
that the end result will be something that’s particularly useful to anyone.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FreeBSD MachO File format, your comments on it.

2016-03-24 Thread David Chisnall
Hi,

I’d slightly question the assertion that Mach-O is a well-designed format.  For 
example, it has a hard limit of 16 section types, doesn’t support COMDATs and 
so on.  OS X uses a load of magic section names to work around these 
limitations.

Note that a Mach-O image activator is relatively easy, but a Mach-O rtld is far 
more complex.  It might be possible to port dyld from OS X, but as I recall it 
depends quite heavily on the Mach kernel interfaces.

On fat binaries, note that the support in the file format is pretty trivial.  
Far more support is needed in the image activator and rtld to determine the 
correct parts and map only them.  If you’re interested in doing this work, then 
I’d recommend looking at this proposed specification for fat ELF binaries:

https://icculus.org/fatelf/

That said, I’m not totally convinced that fat binaries are actually a good 
solution (unless you’re willing to go a step further than Apple did and merge 
data sections) - NeXT managed very well shipping fat bundles without using fat 
binaries and even had a special mode in ditto to strip out the foreign 
architectures when copying a bundle from a network share to a local filesystem.

Persuading clang to emit FreeBSD Mach-O binaries is probably harder than you 
think.  It’s quite easy to persuade it that Mach is a valid file format for 
FreeBSD, but there are a *lot* of places where people conflate ‘is Mach’ with 
‘is Darwin’ in the Clang and LLVM sources.  Finding all of these and making 
sure that they’re really checking the correct one is difficult.

Emulating OS X binaries may be interesting.  NetBSD had a Mach / XNU compat 
layer for a while.  The problem here is that the graphics stack interfaces on 
OS X are completely different from any other *NIX system (as are the kernel 
interfaces for sound), so the most that they could do was run command-line and 
X11 Mac apps - not especially useful.  Actually emulating OS X apps will need 
far more than that - OS X ships with about 500MB of frameworks, many of which 
are used by most applications.  The GNUstep project is undermanned and hasn’t 
been able to keep up with the changes to the core Foundation and AppKit 
frameworks, let alone the rest.

David

> On 24 Mar 2016, at 09:13, mokhi  wrote:
> 
> Hi guys.
> I'm Mahdi Mokhtari (aka Mokhi between FreeBSD friends).
> 
> I am working on adding Mach-O binary format to supported formats for FreeBSD.
> Not for emulations on first step, but as a native supported format
> just like a.out [or Elf]
> (though it can go in both ways too).
> 
> There are good reasons to have Mach-O format support IMO.
> It's well/clear designed file format.
> Can supports multiple Arch by default (It's Fat Format).
> Because of its Fat Format support, it can even help porting/packaging easier 
> for
> projects such as Freebsd-arm or others IMO :D.
> At end (even not among its interesting parts, maybe :D) point, it
> leads and helps to have
> OSX emulation support on FreeBSD.
> 
> BTW, i've coded[1] Mach-O support for FreeBSD with helps of
> FreeBSD-ppl on IRC about various aspects of this works (from
> fundamental points of VM-MAP, to SysEntVec for Mach-O format) and
> with help of Elf and a.out format codes and mach-o references.
> It's in Alpha state (or before it) IMO, as I'm not sure about some of
> its parts, but I've tested a mach-o formatted binary with it and it at
> least loads and maps it segments correctly :D. (it was actually a
> simple "return 0" C Code, compiled in a OSX, if you know how can I
> force my FreeBSD clang to produce mach-o files instead of ELF I'd be
> happy to know it, and I appreciate :D)
> 
> I'd like to have your helps and comments on it, in hope to make it better
> and make it ready for review.
> 
> Thanks and thousands of regards, Mokhi.
> 
> ==
> [1] https://github.com/m0khi/FreeBSD_MachO
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: mount_smbfs(8): support for SMBv3.02?

2016-03-08 Thread David Chisnall
On 8 Mar 2016, at 17:59, Kurt Jaeger  wrote:
> 
>> Indeed.  Both Solaris and OS X have SMB2 implementations.  If
>> anyone is interested in working on this, then the Apple implementation
>> may provide some inspiration:
>> 
>> http://www.opensource.apple.com/source/smb/
> 
> Is there any way to download this as tgz or something ?
> 
> It looks painful to get it from that site.

Tarballs are here:

http://www.opensource.apple.com/tarballs/smb/

Latest one is here:

http://www.opensource.apple.com/tarballs/smb/smb-759.40.1.tar.gz

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: [CFT] packaging the base system with pkg(8)

2016-03-08 Thread David Chisnall
On 8 Mar 2016, at 15:14, Slawa Olhovchenkov  wrote:
> 
> Yes, I undertund this. But what profit of this? Addtional size is
> small, many small packages is bad. We already have expirense with
> spliting Xorg to many small packages -- no profit of this.

The X.org case is similar, but not quite the same.  The X.org split was to ease 
development, but came at a cost of packaging because you almost always want all 
of X (or, at least, most of it - there are a few things such as Xephyr that 
many users may want to skip).

In FreeBSD, we *do* have a compelling case for installing a small subset of the 
base system: service jails (or ‘containerised applications’ as the kids are 
calling them).  We want to be able to install, for example, owncloud and nginx 
or ejabberd in a jail with only the bare minimum required for them to start and 
run.  We want updates to these jails to be fast and we want disk usage (and 
install time) to be low.  In such a jail, I want a shell, the parts of sbin 
needed to do network setup, the libraries that these ports depend on, *and 
nothing else*.  We’re still a way away from doing that.

Comparing the installed sets can be simplified with some improvements to the 
pkg UI, for example allowing a set of packages to be aggregated into a single 
entry.  This is not quite the same as the metapackage concept.  If you install 
everything, then a FreeBSD-base-all metapackage might show up as a single thing 
unless you ask for a verbose output.  We can also present these in a 
hierarchical manner, so that you can drill down and see more detail if you want 
to.

In terms of comparing packages, if you’re doing that visually then you are 
likely to have problems anyway, unless your eyes and brain work far better than 
most humans.  We can make that much easier by providing libxo output in pkg and 
allowing you to have a simple jq script that tells you what the differences are.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: mount_smbfs(8): support for SMBv3.02?

2016-03-08 Thread David Chisnall
On 8 Mar 2016, at 13:19, Miroslav Lachman <000.f...@quip.cz> wrote:
> 
> It would be really nice if somebody can bring better support for FreeBSD's 
> SMB/CIFS mount. Maybe through FreeBSD Foundation projects.

Indeed.  Both Solaris and OS X have SMB2 implementations.  If anyone is 
interested in working on this, then the Apple implementation may provide some 
inspiration:

http://www.opensource.apple.com/source/smb/

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Packaging the FreeBSD base system with pkg(8)

2016-01-28 Thread David Chisnall
On 28 Jan 2016, at 17:45, NGie Cooper  wrote:
> 
> Also, consider that you're going to be allowing upgrades from older RELEASE 
> versions of the OS which might be using a fixed copy of pkgng -- how are you 
> going to support that?

I believe that the plan is to promote the pkg tool somewhat closer to the base 
system.  Upgrades will do the same sort of thing that they do currently for 
ports:

1. First check if the version of pkg is the latest
2. If not, upgrade it
3. Do the real upgrade

The package for package is simply a tarball.  It may be advantageous to 
separate the pkg and pkg-static binaries into different packages, so that pkg 
can always install pkg-static and pkg-static can always update pkg.

There is no guarantee that the pkg tool from X.Y can install any packages from 
X+n.Y.m other than the pkg-static binary, which can then upgrade the rest of 
the system.

The provision of pkg-static prevents us from being in the situation that I 
encountered trying to upgrade a Debian system (and ending up with a mess 
requiring a full reinstall) where apt needed a newer glibc and the glibc 
package needed a newer apt to install it.  We will always provide a pkg-static 
for every supported branch that can be installed by any earlier version of pkg 
(because it’s just extracting a single-file archive - and in the absolute worst 
case you can do this by hand) and can install newer packages.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Too low PTHREAD_STACK_MIN value?

2016-01-23 Thread David Chisnall
On 23 Jan 2016, at 08:58, Maxim Sobolev  wrote:
> 
> For what it's worth, I agree with David. This looks like definite misuse of 
> the constant. If app X requires min size of stack of Y, it's fullish of it if 
> to expect our PTHREAD_STACK_MIN somehow accommodate that. It should be really 
> using MAX(PTHREAD_STACK_MIN, Y) to set its stack instead. Should be easy to 
> patch and it needs to be reported to the upstream vendor(s) instead. Don't 
> forget that bumping this limit, no matter how small, will get multiplied by 
> the number of threads running, which could be in many thousands

After talking to Ed, I’m not sure I was correct in my initial assessment.  The 
code in pthread’s thread exit routine is calling the unwinder, and that’s 
what’s exhausting the stack space.  This means that a thread function that just 
does return 0 will run out of stack space.

That said, existing values of PTHREAD_STACK_MIN are part of the ABI and if 
we’re going to bump it then we need to make sure that we do something clever 
with existing binaries to ensure that, when they ask for 2KB of stack, we give 
them more (which can be problematic if they’re allocating their own stack).  
I’d much prefer a solution where we don’t expose the HP unwind interfaces from 
libeh (it’s fine to do so from a separate libunwind) and we don’t allocate that 
much space in the unwinder.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Too low PTHREAD_STACK_MIN value?

2016-01-22 Thread David Chisnall
On 21 Jan 2016, at 16:02, Ed Maste  wrote:
> 
> I found that lang/polyml uses PTHREAD_STACK_MIN for a trivial signal
> handler thread it creates[1]. They found it was too small and
> implemented a 4K minimum bound to fix polyml on FreeBSD[2]. Even if
> this isn't really the intended use of PTHREAD_STACK_MIN it suggests
> the 2K x86 minimum may indeed be too low.
> 
> I ran into this while trying LLVM's libunwind, which requires more
> stack space. 2K is certainly too low with LLVM libunwind. Is it
> reasonable to just increase it to say 8K?

I don’t really like this solution.  PTHREAD_STACK_MIN is the size for a stack 
that does not do anything.  You should never use it without adding the amount 
that you are going to need (which might be nothing if you are running code from 
a language that does not use a conventional C-style stack, but still wants to 
use OS threads).  Making it larger because a specific kind of thing that some 
consumers want to do with it needs more space is definitely against the spirit 
of the value and potentially harmful as it means that people using it correctly 
will be using a lot more memory per thread.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Is updating contrib/gcc desirable?

2015-12-31 Thread David Chisnall
On 31 Dec 2015, at 11:59, Yuri  wrote:
> 
> Would it be the right way of solving the problem if I submitted an update of 
> contrib/gcc and contrib/gcclibs from the gcc-5.3.0 tree?

No.

>  Any pitfalls with this?

The newer versions of GCC are GPLv3 and so are unacceptable for the FreeBSD 
base system.  Most of libgcc in base now comes from compiler-rt.  The correct 
solution would be to identify the missing functionality in compiler-rt so that 
it can be fixed (with patches ideally, but even a list of the missing functions 
and what they do would be helpful).  Things like libquadmath, which are only 
needed for the port, belong in ports.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: Dwarf problem with gcc and gdb

2015-12-08 Thread David Chisnall
The gdb in the base system doesn’t support DWARF4.  Use gdb791 or lldb-devel 
from ports (I believe gdb791 is probably a better bet on ARM, currently).

David

> On 8 Dec 2015, at 09:02, Ray Newman  wrote:
> 
> Hi,
> 
> Compiled using gcc (FreeBSD Ports Collection) 4.8.5 on arm (Raspberry Pi - 
> several versions); BSDmakefile attached (make test used).
> gdb gives:
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "armv6-marcel-freebsd"...Dwarf Error: wrong 
> version in compilation unit header (is 4, should be 2) [in module 
> /home/ray/mumps/mumps]
> 
> I need to fix this to find the *real* problem.
> 
> Thanks,  Ray
> 
> 
> # Makefile for MUMPS BSD
> # Copyright (c) Raymond Douglas Newman, 1999 - 2014
> # with help from Sam Habiel
> 
> CC  =   gcc
> LIBS  = -lm -lcrypt
> EXTRA = -O -Wall -Iinclude
> 
> .ifmake test
> EXTRA = -O0 -g  -gdwarf-2 -gstrict-dwarf -ggdb -Wall -Iinclude
> .endif
> 
> SUBDIRS=compile database init runtime seqio symbol util xcall
> 
> RM=rm -f
> 
> PROG  = mumps
> 
> OBJS=   compile/dollar.o \
>compile/eval.o \
>compile/localvar.o \
>compile/parse.o \
>compile/routine.o \
>database/db_buffer.o \
>database/db_daemon.o \
>database/db_get.o \
>database/db_ic.o \
>database/db_kill.o \
>database/db_locate.o \
>database/db_main.o \
>database/db_rekey.o \
>database/db_set.o \
>database/db_uci.o \
>database/db_util.o \
>database/db_view.o \
>init/init_create.o \
>init/init_run.o \
>init/init_start.o \
>init/mumps.o \
>runtime/runtime_attn.o \
>runtime/runtime_buildmvar.o \
>runtime/runtime_debug.o \
>runtime/runtime_func.o \
>runtime/runtime_math.o \
>runtime/runtime_pattern.o \
>runtime/runtime_run.o \
>runtime/runtime_ssvn.o \
>runtime/runtime_util.o \
>runtime/runtime_vars.o \
>seqio/SQ_Util.o \
>seqio/SQ_Signal.o \
>seqio/SQ_Device.o \
>seqio/SQ_File.o \
>seqio/SQ_Pipe.o \
>seqio/SQ_Seqio.o \
>seqio/SQ_Socket.o \
>seqio/SQ_Tcpip.o \
>symbol/symbol_new.o \
>symbol/symbol_util.o \
>util/util_key.o \
>util/util_lock.o \
>util/util_memory.o \
>util/util_routine.o \
>util/util_share.o \
>util/util_strerror.o \
>xcall/xcall.o
> 
> .c.o:
>${CC} ${EXTRA} -c $< -o $@
> 
> all: ${OBJS}
>${CC} ${EXTRA} -o ${PROG} ${OBJS} ${LIBS}
> 
> test: ${OBJS}
>${CC} ${EXTRA} -o ${PROG} ${OBJS} ${LIBS}
> 
> clean:
>rm -f ${OBJS} ${PROG} ${PROG}.core
> 

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: libXO-ification - tangent on JSON choices

2015-11-20 Thread David Chisnall
On 16 Nov 2015, at 21:04, Garance A Drosehn  wrote:
> 
> First let me say that I wish I had more time to contribute to the project,
> but I seem to be caught in variety of long drawn-out hassles in real-life.
> Otherwise I would already know the answer to this question:
> 
> Is there some specification for what JSON is created by the various FBSD
> utilities?  I did not see any discussion of that in the earlier threads
> on this.  I don't mean "what is syntatically correct JSON?", I mean some
> kind of guidelines of what property-names would be used across commands,
> and what values should be for those properties.

There is not, currently.  Until 11.0 is branched, it should be considered to be 
in flux, but after that then we are going to provide backwards compatibility 
(i.e. you can add fields to the JSON, but you can’t remove them).  Before this 
point, it would be good to have some consensus of on what properties should be 
common and what each should be providing, and regression tests for the JSON 
(the XML is isomorphic to the JSON, so probably doesn’t need testing 
separately).

If you want to provide a first draft of some recommendations, they would be 
very welcome,

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: libXO-ification - Why - and is it a symptom of deeper issues?

2015-11-16 Thread David Chisnall
On 16 Nov 2015, at 17:09, Elizabeth Myers  wrote:
> 
> It seems to boil down to the golden rule: he who has the gold, makes the
> rules. Juniper wanted it, they're a non-trivial donor to the FreeBSD
> foundation and employ many devs, so they got their way.
> 
> That's all there is to it.

I think that’s a mischaracterisation.  

(Core hat on:) Juniper’s status as a donor to the FreeBSD Foundation has 
absolutely no baring on their ability to get code committed. The libxo code was 
accepted because it solves a problem that a number of FreeBSD users (and 
downstream consumers of FreeBSD) have.

Libucl is primarily developed by a PhD student.  He is not backed by a large 
corporation or an organisation that donates to the FreeBSD Foundation.  His 
code is accepted for precisely the same reason as libxo: it solves a problem 
that many people have identified is real.

Development is, however, driven by people willing to actually do the work, and 
being willing to listen to feedback from other developers.  If someone started 
committing a load of code that is only of use to them and makes life harder for 
everyone else, then Core would be quick to request that it be reverted.  This 
rarely happens, because we try hard to avoid giving commit bits to people who 
don’t play well with others.

Phil has put a lot of effort into libxo and, most importantly, listened to 
community feedback.  For example, his recent changes to libxo from feedback at 
BSDCam (where he led a session discussing it and related topics) means that 
libxo can now be used to trivially add localisation to the a load of base 
system utilities.  This is something that was not in the Juniper system that 
inspired libxo, because it is not something that they need (Juniper’s interface 
provides a choice between US English and US English).

This is part of the reason why Phil was recently awarded his commit bit: he 
isn’t just writing code that Juniper wants, he’s writing code that benefits 
both Juniper and the wider community and is willing to adapt it to provide 
wider benefits.  This is *precisely* how open source is supposed to work: 
Juniper benefits by (eventually) being able to reduce their diffs to upstream, 
everyone else benefits from having the new features, and development is led by 
consensus of what is useful.

(Core hat off:) I slightly disagree with Alan’s comment that librarification of 
base system utilities addresses the same problems.  There are three related 
problems:

- Being able to expose the same functionality as the base system utilities to C 
code.
- Being able to expose this functionality via bindings to high-level languages.
- Being able to drive complex scripting from the command line and shell scripts.

Libxo directly addresses the last of these points and inefficiently addresses 
the first and second.  Librarification would address the first and (possibly) 
the second.  They are overlapping requirements.  For some the second, the 
combination would likely be the best solution for a lot of requirements (i.e. 
have library calls that produce the JSON that 
Lua/Python/Ruby/JavaScript/Intercal can turn into native objects).

I would very much like to see all of the base system functionality exposed in 
terms of libraries, but this is a huge challenge.  Good API design is *hard*.  
Tools like libucl and libxo allow people to build high-level wrappers and 
experiment with API design easily outside of the base system, in such a way 
that does not give us difficult C API compatibility requirements that we have 
to respect for the next few decades, and will allow us to be more informed when 
it comes to designing these APIs.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: libXO-ification - Why - and is it a symptom of deeper issues?

2015-11-15 Thread David Chisnall
On 15 Nov 2015, at 19:12, Simon J. Gerraty  wrote:
> 
> Garrett Cooper  wrote:
>> We lack a [dtd/json] spec for tools, so programming for xo'ification
>> doesn't seems like the best idea in the world to me from a end-user
>> sysadmin/developer perspective.
> 
> A dtd etc is good for sure, and we (Juniper) do have them, as well as
> ui-police to help ensure things go smoothly - even that doesn't catch
> everything.

At the GSoC Mentor Summit, someone from GitHub pointed me at a tool[1] for 
generating schemas from JSON (which can then be refined manually) and can be 
used for validating that the schema is still respected in newer versions.  I 
hope to integrate something like this into our regression test suite before 11.

For the people complaining that there are no command-line tools for filtering 
JSON in shell scripts, I’d suggest that they look at JQ[2].  I’m not sure if 
it’s worth bringing it into base, but being able to install it easily 
(textproc/jq, no dependencies) from ports suddenly makes a load of stuff 
trivial with shell scripts that were either incredibly hard or impossible prior 
to libxo.

David

[1] https://github.com/perenecabuto/json_schema_generator
[2] https://stedolan.github.io/jq/

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: FYI: SVN to GIT converter currently broken, github is falling behind

2015-11-08 Thread David Chisnall
On 8 Nov 2015, at 02:32, Ulrich Spörlein  wrote:
> 
> 
> Git commit hashes *might* change in the future. I really don't
> see how this is a big deal anyway.  It happened once and I'm trying to
> have it never happen again. But why are people afraid of this
> happening? Every "official" git commit is tagged with a SVN revision
> and the contents of those revisions are obviously correct (just not
> the ancestry and the commit objects, possibly). So it would be easy to
> write a script that replays VendorA's git history and swaps out the
> new official commits for the old official commits. There would be no
> merge conflicts.

Git commit hashes must not change, or we completely destroy the utility of the 
git mirror for all downstream users.  You can no longer do a merge from 
upstream git if the hashes in your local clone do not match the hashes 
downstream.  Your answer of expecting every downstream user of FreeBSD’s git 
repo (GitHub tracks over 600 of them, there are likely more that are using 
private clones) to write a script to fix the mess for themselves is completely 
unacceptable.

If there has been a window in which incorrect hashes were generated, this can 
be fixed by moving that to a branch, doing the correct thing in master, and 
then using git-imerge in rebase-with-history mode.  After the point of the fix, 
there will be a single set of commits, but before that there will be two 
options as parents, one for each version of the export.

Please remember that a guarantee of not changing the hashes of the history was 
one of the conditions that Core had for promoting this to an official FreeBSD 
service.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [PATCH] microoptimize by trying to avoid locking a locked mutex

2015-11-06 Thread David Chisnall
On 5 Nov 2015, at 16:55, Mateusz Guzik  wrote:
> 
>> Will that cause problems, especially for code inside a loop
>> where the compiler may decide to shuffle things around?
>> 
> 
> Lock value is volatile, so the compiler should not screw us up.

Note: volatile means do not reorder loads/stores *to this value*, the compiler 
is completely free to reorder loads/stores to other values around this one.  
_Atomic exists in C11 to do the right thing when you want to enforce stricter 
barriers.  In this case, we’re currently relying on the compiler not being able 
to see through the assembly.

At some point in the future, I imagine that we’ll move to using C11 atomic 
operations, which will allow the compiler more freedom.  The new reads that 
you’re currently doing should be replaced by explicit atomic reads with a 
relaxed memory order (to enforce atomicity with respect to that variable).  If 
we were to replace the mtx_lock field with an _Atomic variant, then the current 
code would make it a sequentially consistent read, which would add a lot of 
extra overhead.  Note also that it is undefined behaviour in C11 to do atomic 
and non-atomic accesses to the same underlying memory location.  Please 
consider the future maintainability of the code when making this change.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [CFT] Unicode collation string and reworked locale definitions

2015-11-02 Thread David Chisnall
On 1 Nov 2015, at 21:30, Baptiste Daroussin  wrote:
> 
> All issues reported has been fixed, except if more issues are reported, this
> will be merged into head next saturday: November 7th

That’s really excellent news!  Thanks for doing this.  Are there any good 
potential sources for the regex stuff?  I think std::regex in libc++ supports 
multibyte character sets, but is very full of templates and not very easy to 
translate into C.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: dtc(1): reproducible segmentation fault

2015-10-25 Thread David Chisnall
On 25 Oct 2015, at 00:07, George Abdelmalik  wrote:
> 
> You've beaten me to it with the fix before I could lodge the bug report :)
> 
> In your repo I've seen that the mmap(2) call now takes the MAP_PRIVATE flag. I
> applied that change locally to my source tree and that has fixed the problem.
> I've since re-read the mmap(2) man page to find out how that change could
> be influential...
> 
> [EINVAL]   None of MAP_ANON, MAP_PRIVATE, MAP_SHARED, or
>MAP_STACK was specified.  At least one of these flags
>must be included.
> 
> Although obvious to me now, I missed it on my previous reads.
> 
> Thanks for your assistance. I look forward to your coming set of changes. In
> my view DTC is an important tool and I would be willing contribute effort to
> making it feature parity with the GPL version if that is lacking.

It’s now committed (r289935).  Sorry for the delay - I meant to commit the 
changes in January and it slipped down my to-do list.

Please do test with any dts files that you have.  If you find bugs, then please 
report them and assign them to me.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: dtc(1): reproducible segmentation fault

2015-10-24 Thread David Chisnall
On 23 Oct 2015, at 17:40, Ian Lepore  wrote:
> 
> Don't cc me.  I looked at the in-tree dtc code once and decided it's
> too flawed to try to maintain, and it supports only a subset of the
> full dts syntax.  That's why we switched back to using the gnu dtc for
> buildkernel.  But I just discovered that for some reason gnu is not the
> copy of dtc that gets installed, it's just the one that gets used
> during a buildkernel.

Please assign the bug to me.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"


Re: dtc(1): reproducible segmentation fault

2015-10-24 Thread David Chisnall
On 24 Oct 2015, at 11:07, David Chisnall <thera...@freebsd.org> wrote:
> 
> On 23 Oct 2015, at 17:40, Ian Lepore <i...@freebsd.org> wrote:
>> 
>> Don't cc me.  I looked at the in-tree dtc code once and decided it's
>> too flawed to try to maintain, and it supports only a subset of the
>> full dts syntax.  That's why we switched back to using the gnu dtc for
>> buildkernel.  But I just discovered that for some reason gnu is not the
>> copy of dtc that gets installed, it's just the one that gets used
>> during a buildkernel.
> 
> Please assign the bug to me.

Actually, it looks as if this is one of the (many) bugs in dtc that I fixed in 
a bunch of changes that I made (and didn’t get around to committing) last 
Christmas (https://github.com/davidchisnall/dtc).  Patrick Wildt tested the 
version that I was working on with a load of things from the GPL dtc test suite 
and they all passed.  I’m now running a make universe with the new version, and 
I’ll commit if there are no problems.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Intel Haswell support - Any updates?

2015-09-17 Thread David Chisnall
On 17 Sep 2015, at 10:55, Lundberg, Johannes  
wrote:
> 
> Looking at those pages it seems like development has basically been
> standing still for the last couple of years.

I’m not sure why you’d have that impression.  The Haswell entry on that page 
links here:

https://wiki.freebsd.org/Graphics/Update%20i915%20GPU%20driver%20to%20Linux%203.8

This has a status update from July and links to this GitHub repo:

https://github.com/freebsd/freebsd-base-graphics/tree/drm-i915-update-38

The last graphics-related commit in that repo was 6 hours ago:

https://github.com/freebsd/freebsd-base-graphics/commits/drm-i915-update-38

It was preceded by a load of other work.  If you look at the changes since the 
last non-graphic-related commit in that branch (about a month ago) you’ll see a 
huge quantity of changes.

Of course, I’m sure that the graphics team would welcome assistance if people 
feel that their progress is too slow.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Intel Haswell support - Any updates?

2015-09-17 Thread David Chisnall
On 17 Sep 2015, at 11:31, Lundberg, Johannes  
wrote:
> 
> Anyway, I wish the foundation would support the graphics team by sponsoring 
> this development…

The Foundation did fund a lot of this work, and likely will again.  The problem 
is not willingness of the Foundation to fund it, nor availability of funds.  As 
with WiFi, it’s availability of people who have both the competence and 
interest to do the work and the availability to work as consultants for the 
Foundation.

Long term, the real solution is to convince GPU vendors to put as much effort 
into funding FreeBSD driver development as they do into funding Linux and 
Windows driver development.  The Foundation has been reaching out in this 
direction, but it’s far more compelling if people can document cases where lack 
of FreeBSD support has cost a vendor sales.  If you’ve bought a system with an 
nVidia or AMD GPU (or, ideally, if your company has bought a few thousand) 
because of lack of FreeBSD support from Intel, then let Intel know.  

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Intel Haswell support - Any updates?

2015-09-17 Thread David Chisnall
On 17 Sep 2015, at 14:41, Lundberg, Johannes  
wrote:
> 
> However, the problem now is not the driver right? But the whole graphics
> stack which has to be rewritten to work with new generation graphics like
> KMS, Wayland, etc?

There are lots of different components here that you’re conflating:

- KMS is the thing that allows the kernel to be responsible for setting the 
graphics mode, restoring it on context switches and so on.  This has worked on 
FreeBSD for a while.

- GEM / TTM are memory managers, they allow the kernel to manage memory (video 
memory and main memory allocated to the GPU) for GPU-using drivers.  These work 
on FreeBSD, but the supported version lags Linux slightly.  This is the focus 
of current work.

- The i915 driver.  This is the graphics driver for Intel GPUs.  It uses KMS 
and the memory manager functionality. Importing a newer version will be 
possible once the underlying parts are done.

- Wayland is a userland application that sits on top of DRI drivers, just as 
X.org does.  Most of the things Wayland needs that don’t work on FreeBSD are 
not graphics related.

The problem is that the developers *also* conflate them.  Upstream i915 
developers will add KPIs to GEM / TTM / KMS to make their lives easier and 
these features need bringing across to FreeBSD.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CPU Perf Power Mgt

2015-08-21 Thread David Chisnall
On 20 Aug 2015, at 17:44, Justin Hibbits jr...@alumni.cwru.edu wrote:
 
 There was a working group at BSDCan this year on power management, and
 what we need to / can do to bring it up to par with the modern world.
 Unfortunately, I haven't had any time lately to work on it, but you
 can read the notes at
 https://wiki.freebsd.org/201506DevSummit/ClockDomains
 
 In short, the goal is to add infrastructure to the kernel to support
 overall power management of the system, scaling beyond cpufreq/powerd.
 Looking for volunteers who could do some of this, due to my lack of
 time to work on it.

We also had a power management meeting at BSDCam this week.  Robin presented 
some slides (though the download seems to be broken for me at the moment):

https://wiki.freebsd.org/201508DevSummit?action=AttachFiledo=gettarget=power-management-for-freebsd-bsdcam-2015.pdf

This includes the design that ARM is trying to push into Linux, which looks 
like it would be adaptable to FreeBSD (and, on the plus side, our complete lack 
of existing power management in the kernel already means we don’t already have 
the legacy stuff that they are fighting in Linux).  The goal is to have the 
scheduler drive all of the (CPU) power management.  Any volunteers to work on 
this are welcome - ARM people are happy to provide advice.

David

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: duration of buildworld

2015-07-28 Thread David Chisnall
On 27 Jul 2015, at 20:49, Slawa Olhovchenkov s...@zxy.spb.ru wrote:
 
 No idea.
 May be someone know about current status swap in file on UFS.
 This is work for me some time ago.

It sounds as if the swap was working (as the build took a long time, it didn’t 
run out of memory).  My guess would be that, before the reboot, something had 
wired (or, if not, then touched frequently enough to keep it in core) a large 
chunk of memory, which forced the swapping.  Swap files are expected to be 
slower than swap partitions (though if you’re swapping to a disk, the seek 
times are likely to dominate in both cases).

I don’t suppose there’s  capture of the output of top from before the reboot?

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: eventfd lookalike in FreeBSD ?

2015-07-28 Thread David Chisnall
On 28 Jul 2015, at 18:23, Adrian Chadd adrian.ch...@gmail.com wrote:
 
 (What would be nice is having kqueue know about conditionals, so we
 can sleep on a cond as well as a kqueue fd+queue, but I can't have
 everything I want..)

I recently came across a need to do something like this.  Being able to add 
condvar / mutex pairs to a kqueue and wait on a set of condition variables, 
reacquiring the mutexes for any of the signalled ones, as well as waiting for 
kernel events would be very useful.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: eventfd lookalike in FreeBSD ?

2015-07-28 Thread David Chisnall
On 28 Jul 2015, at 18:33, Adrian Chadd adrian.ch...@gmail.com wrote:
 
 Windows has had this for years. It makes async network programming
 with thread worker queues significantly less abusive.

Can you do the same with Solaris completion ports?  It might be a good source 
of inspiration for a good API if so.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: gettimeofday((void *)-1, NULL) implicates core dump on recent FreeBSD 11-CURRENT

2015-07-09 Thread David Chisnall
On 9 Jul 2015, at 10:17, NGie Cooper yaneurab...@gmail.com wrote:
 
 I agree that this is less applicable for FreeBSD than NetBSD. Please
 keep in mind that contrib/netbsd-tests came from NetBSD, not FreeBSD.
 Peter Holm and I tried our best to vet out the issues with the test
 suite before integrating it in, but there might be issues due to
 implementation discrepancies between FreeBSD and NetBSD.

The high-level question is whether these are conformance tests (in which case 
they should be checking for specified behaviour) or regression tests (in which 
case they should be checking for known implementation details, but should 
explicitly document what those are).

When the gettimeofday implementation changed, it would have been nice to get 
two results:

- No conformance tests failed, change is still a valid implementation of POSIX 
gettimeofday()

- One regression test failed.  This test is assuming that gettimeofday() is a 
system call and so should be marked as an expected failure.

Unfortunately, the test suite currently conflates the two.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [RFC] Replace gnu groff in base by heirloom doctools

2015-05-14 Thread David Chisnall
On 14 May 2015, at 09:59, Baptiste Daroussin b...@freebsd.org wrote:
 
 On Thu, May 14, 2015 at 09:55:19AM +0100, David Chisnall wrote:
 On 14 May 2015, at 01:02, Baptiste Daroussin b...@freebsd.org wrote:
 
 - it is partially CDDL partially BSD license.
 
 We currently have a WITHOUT_CDDL knob that some people use.  If we don’t 
 build the CDDL parts, what will break?
 
 Exactly the same thing that breaks right now WITHOUT_GNU and/or WITHOUT_CXX 
 aka
 you won't have the main part of the toolchain (aka troff/nroff)

But man pages will still work via mandoc?  WITHOUT_GNU is known not to work 
(though we’re trying to address that with 11).  WITHOUT_CDDL is generally 
expected to give a working system currently.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [RFC] Replace gnu groff in base by heirloom doctools

2015-05-14 Thread David Chisnall
On 14 May 2015, at 01:02, Baptiste Daroussin b...@freebsd.org wrote:
 
 - it is partially CDDL partially BSD license.

We currently have a WITHOUT_CDDL knob that some people use.  If we don’t build 
the CDDL parts, what will break?

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: [RFC] Replace gnu groff in base by heirloom doctools

2015-05-14 Thread David Chisnall
On 14 May 2015, at 10:24, Baptiste Daroussin b...@freebsd.org wrote:
 
 On Thu, May 14, 2015 at 10:13:18AM +0100, David Chisnall wrote:
 On 14 May 2015, at 09:59, Baptiste Daroussin b...@freebsd.org wrote:
 
 On Thu, May 14, 2015 at 09:55:19AM +0100, David Chisnall wrote:
 On 14 May 2015, at 01:02, Baptiste Daroussin b...@freebsd.org wrote:
 
 - it is partially CDDL partially BSD license.
 
 We currently have a WITHOUT_CDDL knob that some people use.  If we don’t 
 build the CDDL parts, what will break?
 
 Exactly the same thing that breaks right now WITHOUT_GNU and/or WITHOUT_CXX 
 aka
 you won't have the main part of the toolchain (aka troff/nroff)
 
 But man pages will still work via mandoc?  WITHOUT_GNU is known not to work 
 (though we’re trying to address that with 11).  WITHOUT_CDDL is generally 
 expected to give a working system currently.
 
 Yes since I switched the default manpage renderer to mandoc(1) :) (meaning 
 that
 WITHOUT_GNU is also not blocking manpage rendering)

Sounds good to me.  Probably warrants a release notes entry specifically 
documenting that change, but otherwise sounds like a big improvement!

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Increase BUFSIZ to 8192

2015-05-13 Thread David Chisnall
On 13 May 2015, at 09:03, John-Mark Gurney j...@funkthat.com wrote:
 
 Poul-Henning Kamp wrote this message on Tue, May 12, 2015 at 06:31 +:
 
 In message 20150512032307.gp37...@funkthat.com, John-Mark Gurney writes:
 
 Also, you'd probably see even better performance by increasing the
 size to 64k, [...]
 
 easy:
  8K on 32bit
  64k on 64bit
 
 Sounds good to me...  Just for people who care... I did a quick set of
 benchmarks on sha256.. This is using my preliminary patch to use sse4
 optimized sha256...  But this should be the same for others...
 
 The numbers in ministat output are the time in seconds it takes my
 3.4GHz AMD A10-5700 APU running HEAD to process a 512MB file, so lower
 numbers are better..  I've processed them into easier to read format:
 BUFSIZ:   145MB/sec
 8k:   193MB/sec
 16k:  198MB/sec
 64k:  202MB/sec
 128k: 202MB/sec
 -t:   211MB/sec

It looks like most of the benefit is gained at 16KB.  Did you try running the 
benchmark with something else running at the same time to see if there is any 
advantage in trashing the caches a bit less (simple case, what happens if you 
run two instances of the same benchmark at once)?

I suspect that you’re about right anyway - I recently did some tests while 
playing with JavaScript FFI generation with a multithreaded process JavaScript 
environment calling out to OpenSSL to do SHA calculations and having each of 8 
threads reading in 128KB chunks gave the fastest performance (Core i7, 4 cores 
+ hyperthreading), with only a negligible gain over 64KB.  In all cases, the 
JavaScript implementation was significantly faster than the openssl tool, which 
used 8KB buffers.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: CFR: a new __unreachable() builtin

2015-05-13 Thread David Chisnall
On 13 May 2015, at 17:05, Pedro Giffuni p...@freebsd.org wrote:
 
 Hello;
 
 I am looking at the cdefs in other BSDs hoping to avoid adopting the
 same definitions with incompatible names and I noticed NetBSD is using
 a new __builtin_unreachable (void) function from gcc 4.6:
 
 https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html
 
 Apparently it was interesting enough that clang implemented it too so
 I created a code review differential for it.
 
 https://reviews.freebsd.org/D2536
 
 I don't want to add new C definitions unless they are going to be used
 so feel free to comment on the convenience or not of having it.

LLVM uses this quite heavily, in a macro that expands to something equivalent 
to assert(0  unreachable reached!”) in debug mode and 
__builtin_unreachable() in release mode.  When you’re debugging, you get errors 
if you reach unreachable code and in deployment the compiler gets a useful hint 
for optimisation.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: What to do about RCS/OpenRCS

2015-05-11 Thread David Chisnall
On 11 May 2015, at 17:27, Jos Backus j...@catnook.com wrote:
 
 I didn't miss anything. My point is that debating to update one piece of
 obsolete software with another is silly, and that FreeBSD should try to
 move forward in this area. But that's hard, as your response indicates.

Steve is correct, and you are missing the point.  Fossil, Git, Mercurial, and 
so on are all available as packages.  No one is suggesting using RCS in 
preference to any of them.  

Deleting RCS from the base system would be nice, but unfortunately we can’t 
because of scripts that depend on some components of RCS.  Replacing these with 
the OpenRCS equivalents (if they work) would allow us to remove a GPL’d piece 
of code from the base system.  As long as this doesn’t come with a 
functionality regression, this would be a nice thing to do.

Replacing RCS in the base system with Fossil solves no problems that actually 
exist.  It does not allow the scripts that rely on RCS to continue to work and 
it does not make Fossil easier to use (would you really want to stick with the 
one in the base system for the entire lifetime of a major release, rather than 
use the packaged one?).  It would only make sense if we were to move FreeBSD 
development to Fossil and currently there are a few showstoppers in Fossil that 
prevent this.

 This is the last I'll say about this, because it appears the community
 isn't ready. Have fun with your ancient version control while Linux
 continues to grow in market share. :-(

And now you’re moving beyond missing the point and just trolling.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

Re: Merging GitHub Pull Requests into Subversion using git-svn

2015-04-25 Thread David Chisnall
On 23 Apr 2015, at 00:12, Craig Rodrigues rodr...@freebsd.org wrote:
 
 While not as smooth as clicking a merge button in GitHub,
 this is a valid way to accept patches submitted via GitHub pull requests,
 and integrate them in our FreeBSD Subversion repo.

The merge button on GitHub does the wrong thing anyway (merges without 
fast-forward, so you end up with a tangled history), so (after the initial 
setup) the steps that I use for merging pull requests from GitHub projects are 
very similar (locally pull the branch with fast-fordward, test, push).  

David
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [CFT] Call for testing pkg 1.5.0

2015-04-01 Thread David Chisnall
On 1 Apr 2015, at 05:03, Rui Paulo rpa...@me.com wrote:
 
 That is expected.  WITH_PKG=devel is a make(1) option that only affects ports 
 (non-binary pkgs).

Are you sure?  I have it in make.conf on one of my systems where I never build 
ports manually (and don't even have a ports tree installed) and there I get 
this:

$ pkg -v
1.4.99.13

In a jail on the same machine without the make.conf entry, I get the stable 
version.  This is how I've been testing pkg-devel for a while.  Is there a 
different recommended way?

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: SSE in libthr

2015-03-28 Thread David Chisnall
On 28 Mar 2015, at 13:54, Julian Elischer jul...@freebsd.org wrote:
 
 the point is that clang will do this anywhere it can, because it isn't taking 
 into account the
 side effects, just the speed of the commands themselves.

This is also something that is not going to decrease.  Clang now enables the 
SLP vectoriser by default and this code is constantly being improved.  Current 
generation vector units are explicitly designed as targets for compiler 
autovectorisation, not for hand-tuned DSP code (which, increasingly, runs on 
the GPU anyway).  This means that we're increasingly going to see SSE/AVX/NEON 
usage in CPU-bound code, even without an explicit programmer decision to do so. 
 Optimising for the case when the vector unit is not used is about as sensible 
as optimising for the single-core case: it will affect some people, but 
generally not those who care about performance, and a decreasing number of 
people over time.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Jenkins build is still unstable: FreeBSD_HEAD-tests2 #867

2015-03-23 Thread David Chisnall
On 22 Mar 2015, at 22:01, Craig Rodrigues rodr...@freebsd.org wrote:
 
 Volatile is not the solution, it is completely orthogonal.  The correct
 way would be to use unsigned integers, for which wrapping is defined,
 then convert those back and forth when presenting the results to the
 user.
 
 
 OK, converting expr.y to use unsigned integers would require a bit of work.

Note that clang has, for a few releases, had builtins that allow 
overflow-checked operations and will generate very efficient code.  In 
op_times, I believe the following should work:

long long mul;
#if __has_builtin(__builtin_smulll_overflow)
if (__builtin_smulll_overflow(a-u.i, b-u.i, mul))
errx(ERR_EXIT, overflow); 
#else
mul = a-u.i * b-u.i;
#endif
r = make_integer(mul);

I don't know if recent versions of gcc implement these builtins yet.  I think 
they were added to clang around 3.4, possibly slightly earlier.  

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Massive libxo-zation that breaks everything

2015-03-03 Thread David Chisnall
On 3 Mar 2015, at 01:32, Andrey Chernov a...@freebsd.org wrote:
 
 So, why you ever need to modify wc? Just load wc inside your
 json/xml/etc writer, replacing its printf at the ld-elf.so level.

You can't get structured output from printf() because printf() takes 
unstructured input.  It's a string with some variables pasted in, but no 
awareness of context.

The libxo changes to the tools are simply marking the output as having a 
defined structure.  The library then translates this abstract structure into 
something that can be parsed easily by external tools.

If your argument is that the UNIX philosophy is simple tools doing one thing 
well, then please remember the context of this philosophy: It dates back to the 
original UNIX systems *that did not support shared libraries* and was an 
argument used to justify not implementing them.  This is why globbing is in the 
shell instead of a shared library and why some variant of mv *.a *.b works on 
every command-line interface except for a UNIX shell.

Even with that in mind, small changes to individual tools are a *lot* simpler 
than one massive monolithic tool that understands the output formats of every 
other tool in the base system and can transform them.  Why do you think a few 
library calls in each application is more complex *than an entire parser per 
tool*?

I am reminded of the time about ten years ago when I was writing a system info 
widget for a DE.  Getting the total amount of RAM, number of cores, and CPU 
speed on FreeBSD, OpenBSD, NetBSD, and Solaris proved to be a single sysctl for 
each attribute.  On Linux, the information was all there, in /proc/cpuinfo, so 
I had to write a parser.  Unfortunately, the format changed across kernel 
revisions and across architectures.  In the end, I had to link in a 300KB 
library to parse information that the kernel already had in a machine-readable 
format and present it to me in a machine-readable format.

Hopefully there's a lesson here that we can learn from: human-readable formats 
do not make good intermediate representations when communicating between tools.

If your argument is about maintainability of these changes, then please point 
to concrete instances where the changes are complex and difficult to maintain.

If your argument is about binary size, then it would be relatively easy for us 
to add a version of libxo for static linking into the versions in /rescue that 
only supported plain-text output, but again, please quantify your objections: 
What is the size increase that you're seeing and what kind of devices is this 
causing a problem for?


The general approach of making tools produce machine-readable output has been 
under discussion for *years* and the integration of libxo has been underway for 
several months, prior to which it was used inside Juniper.  If you had a better 
approach, then the correct time to raise it was back when the requirement to 
produce machine-readable output was raised.  The subject of this thread is just 
plain trolling (is everything broken by libxo?  No.  Is anything broken by 
libxo?  Occasionally, but new features often come with temporary breakage 
that's found and fixed in testing: that's why we have -CURRENT).

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Massive libxo-zation that breaks everything

2015-03-02 Thread David Chisnall
On 1 Mar 2015, at 21:29, Rui Paulo rpa...@me.com wrote:
 
 On Mar 1, 2015, at 11:11, David Chisnall thera...@freebsd.org wrote:
 How would it be in a port?  It involves modifying core utilities (some of 
 which, like ifconfig, rely on kernel APIs that change between releases) to 
 emit structured output. Maintaining two copies of each utility, one in the 
 base system with plain-text output only and another in ports with XML/JSON 
 output would be very painful.
 
 It would work fine if we had *libraries* for ifconfig/netstat/route/etc.  
 Obviously that's not the case and no one has stepped up to implement them.  
 I've also seen FreeBSD committers expressing their distaste for libraries for 
 trivial command line utilities, which implies they are unaware of another 
 world beyond the CLI.  :-)

I am completely in favour of libraries for the underlying functionality of 
these commands and would love to see all of the system management commands 
become thin wrappers around a library, though it's a lot of engineering work.  
In particular, these libraries will need to have stable APIs that we can 
support across multiple major releases, and getting those right is difficult.  
We really don't want to be stuck in 10 years maintaining a hastily designed API 
for a library.

I see one use of the libxo output as helping to design those APIs.  People are 
going to wrap various tools in libraries for their favourite scripting 
languages and this will give us a corpus for experimenting.

It's also worth noting that often invoking a tool and consuming its output is 
the easiest way to get a stable API and ABI where performance is not a primary 
concern (i.e. most management interfaces).

As to a world beyond the CLI, I saw a nice demo a few years ago of a terminal 
emulator that used WebKit and came with a hacked-up set of parsers for common 
tools.  I'd love to have something simpler (no need for a full WebKit - simple 
outline and table views would be enough and could be done with curses for ssh) 
for FreeBSD where I could type ls in the CLI and get a table view that I could 
then sort and filter by selecting column headings.  Those of us that have used 
Lisp and Smalltalk environments know that a CLI doesn't have to be a teletype 
emulator.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Massive libxo-zation that breaks everything

2015-03-02 Thread David Chisnall
On 2 Mar 2015, at 09:24, Harrison Grundy harrison.gru...@astrodoggroup.com 
wrote:
 
 It would seem like the libxo stuff runs the risk of becoming this same
 API.

Why?  The 'API' in the case of an libxo-ised program is a stream on stdout that 
is then consumed by a JSON or XML parser. XML and JSON are intrinsically 
extensible formats.  This is *the entire point* of libxo: that we can extend 
the output from these tools without breaking things that wish to consume them 
and which currently rely on fragile parsers.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Massive libxo-zation that breaks everything

2015-03-02 Thread David Chisnall
On 2 Mar 2015, at 09:16, Julian Elischer jul...@freebsd.org wrote:
 
 if we develop a suitable post processor with pluggable grammars, we save a 
 lot of work.
 given enough examples you could almost have automatically generated grammars.

This decoupled approach is problematic.  A large part of the point of libxo is 
to allow changing the human-readable output without breaking tools that consume 
the output.  Now I need to keep the tool that consumes it and the tool that 
produces it in sync, so that's an extra set of moving parts.  When you throw 
jails with multiple versions of world into the mix, it becomes a recipe for 
disaster.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Massive libxo-zation that breaks everything

2015-03-01 Thread David Chisnall
On 1 Mar 2015, at 18:49, Harrison Grundy harrison.gru...@astrodoggroup.com 
wrote:
 
 That does seem useful, but I'm not sure I see the reasoning behind
 putting into base, over a port or package, since processing XML in base
 is a pain, and it can't serve up JSON or HTML without additional
 utilities anyway.

How would it be in a port?  It involves modifying core utilities (some of 
which, like ifconfig, rely on kernel APIs that change between releases) to emit 
structured output.  Maintaining two copies of each utility, one in the base 
system with plain-text output only and another in ports with XML/JSON output 
would be very painful.

The goal of having machine-readable output in the base system is that people 
building systems atop FreeBSD will be able to expect a stable, 
machine-parsable, extensible, output from these tools.  If you're building a 
web admin GUI or automated administration tool for FreeBSD 11, or improving 
integration in your favourite DE, then you should be able to rely on the output 
from base system tools, without having to depend on anything external.

My only concern with libxo at present is that many of the modified tools are 
not emitting self-describing output (e.g. not specifying units for things), but 
that's something that we have a year to shake out before 11.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: HEADS-UP: Enabling WITH_DEBUG_FILES by default

2015-02-12 Thread David Chisnall
On 12 Feb 2015, at 04:49, Glen Barber g...@freebsd.org wrote:
 
 I would like to see it enabled by default for 11.0-RELEASE.

For everyone concerned about size, remember that the goal for 11.0-RELEASE is 
to use pkg for the base system, which makes it much easier to:

- Install some debug info later, on demand.
- Split debug info into separate packages so only the relevant bits can be 
installed when debugging a particular problem.

I probably don't want debug info for everything to be installed on my systems 
by default and upgraded all of the time, but I really do want to be able to 
just run a single pkg ins command to get the symbols for whatever it is that 
I'm experiencing problems with.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: PSA: If you run -current, beware!

2015-02-05 Thread David Chisnall
On 5 Feb 2015, at 07:48, Luigi Rizzo ri...@iet.unipi.it wrote:
 
 Rather than depending on a compiler option, wouldn't it be better/more
 robust to change ticks to unsigned, which has specified wrapping behavior?

Especially if we want to extend support for external toolchains.  gcc and clang 
support -fwrapv (though occasionally versions of both will not fully support 
it), but other compilers may well not have an equivalent.

Translating the code into C is a far more robust solution than the band-aid of 
telling the compiler to accept a language that is a bit like C and hoping that 
this will keep working across compiler implementations and versions.

Adding -fwrapv also defeats a number of compiler optimisations, so we are going 
to generate worse code for places where people used signed types correctly to 
work around places where they were used incorrectly.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: asr(4) error with new clang/llvm

2015-01-02 Thread David Chisnall
On 2 Jan 2015, at 05:00, Ed Maste ema...@freebsd.org wrote:
 
 It's a variable length array in a struct / union. Other than being
 confusing and now triggering a warning after the clang update it
 should be fine.
 
 Most likely we need to build asr with -Werror disabled for that
 warning, perhaps -Wno-error-array-bounds. I'll take a look tomorrow
 morning if nobody else gets to it first.

The correct solution is to declare the array to have 0 elements (although this 
will break C++ code).  A zero-length array at the end of a structure is 
specifically defined by the C standard (since C99) to be a variable-length 
array.  A length-one array was used in C89 prior to this for this purpose.  
Using a 1-element array in C is undefined behaviour.

Note that this change will also require fixing code that allocates it to 
allocate space for n elements not n-1.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [PATCH]: further shrinking of boot2

2014-11-21 Thread David Chisnall
Fir the clang 3.4 import, I had some patches (that never got applied), which 
used clang -O0 and then opt with some custom optimisation order to get a 
reasonable size saving.  It might be worth trying that, so that future changes 
to the default optimisation order don't make things worse again.

David

On 21 Nov 2014, at 12:56, Roman Divacky rdiva...@vlakno.cz wrote:

 Hi all!
 
 In an effort to help import clang3.5 I looked at squeezing a few more bytes
 from boot2.
 
 
http://rys.vlakno.cz/~rdivacky/boot2.diet.patch
 
 
 Please test and review the patch. It survived my qemu boot attempt so it's
 not completely broken. But I would like to have some more testing and review
 comments before I move forward with this.
 
 Fwiw, it shrinks boot2 by 16 bytes when compiled with clang34 and by 28 bytes
 when compiled with clang35.
 
 Thanks! Roman
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Finding a rogue src/sys commit with bisection?

2014-11-15 Thread David Chisnall
On 15 Nov 2014, at 18:56, Alfred Perlstein bri...@mu.org wrote:

 git clone --config remote.origin.fetch='+refs/notes/*:refs/notes/*' 
 https://github.com/freebsd/freebsd.git

You might want to add --depth 2000 (where 2000 is a guess at how many commits 
there have been since when it broke and now), to avoid downloading the entire 
history of the FreeBSD repo (not necessary if you have loads of bandwidth and 
disk space).  Just grabbing the last few thousand revisions is faster than an 
svn checkout.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: Changing timezone without reboot/restarting each service?

2014-11-11 Thread David Chisnall
On 11 Nov 2014, at 03:35, Allan Jude allanj...@freebsd.org wrote:

 jkh@ mentioned this specifically when he gave his talk at EuroBSDCon and
 MeetBSD, about how Apple solved this in LaunchD, because apparently
 originally libc DID check /etc/localtime constantly.

Darwin also has the notify(3) interface, which allows one-bit messages to be 
transmitted either from the kernel to userspace or from userspace to userspace 
very cheaply.  This is ideal for this sort of 'something you've cached is 
change, go and do something expensive to fix it' use (time zone has changed, 
network connectivity has changed, power state has changed).  You can assign 
notifications to either file descriptors (that can be monitored with kqueue), 
to signals, or to flags in memory for things where polling is cheaper (as in 
this case - the libc functions could just check whether a specific flag is set 
in memory when accessing the time zone info and load a newer one if changed).

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: HEADS UP: Standalone kernel debug files moving out of /boot/kernel/

2014-10-29 Thread David Chisnall
On 29 Oct 2014, at 03:11, Ed Maste ema...@freebsd.org wrote:

 /usr/lib/debug is the standard location for standalone debug data
 established by GDB, and seems like a decent enough location. I'll make
 sure to update the man page.

Do gdb and lldb also look in /usr/local/lib/debug?  If not, it would be great 
if we could at least teach lldb to do this so that we can start thinking about 
splitting debug info into separate packages for ports (and providing it as an 
optional install for everything).

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: HOWTO articles for migrating from Linux to FreeBSD, especially for pkg?

2014-10-21 Thread David Chisnall
On 21 Oct 2014, at 00:15, Mason Loring Bliss ma...@blisses.org wrote:

 The second thing that would be useful would be a series of cheat sheets for
 various things. This can either be equivalent commands or equivalent systems.
 Let new folks know that LUKS is GELI and that md-raid1 is gmirror and so
 forth. Show common package handling commands for various Linux flavours and
 map them to pkgng and ports. For instance, what's the equivalent of yum
 provides? Or what do I do in place of apt-cache search or zypper up or
 similar.

I agree that this would be useful, but it requires someone familiar with both 
systems to write.  Perhaps you could help by coming up with a list of things 
that you did frequently with Debian and a description of what they did, then 
someone more familiar with the FreeBSD side can help fill in any gaps where you 
haven't yet worked out what the FreeBSD equivalent is (or, if there isn't a 
FreeBSD equivalent, then we have a useful feature request).

 Other things in the grab bag... It's generally said that ports and pkgs
 shouldn't mix, but there are at least a couple instances where it's
 unavoidable:
 
 I bet roughly no one who installs Subversion wants the FreeBSD bug report
 headers baked in by default, but there they are unless you rebuild from ports
 with a non-default configuration.

It's worth noting that the FreeBSD headers don't affect operation.  Subversion 
only adds the headers to the commit message if they're modified.  I think that 
the fix for this is to add a line at the top saying

# Things below this line are only included if modified

I find that I do occasionally use those in other projects.

 If you want to watch DVDs on your FreeBSD workstation, it's necessary to
 install libdvdcss, but you can't get it from pkgng because it's not there.
 Again, you must build from ports.

Really?  I've installed vlc from packages and it seems able to play DVDs.  I 
don't remember having to do anything special.  Perhaps I had an old version of 
libdecss installed.  I thought that CSS had been ruled not to be an 'effective 
copyright protection mechanism' in the US, so wasn't covered by the DMCA 
anymore.

 I have nothing against ports, but people are warned off of mixing packages
 and ports when clearly it's necessary sometimes.
 
 Oh, here's one. I *was* horrified by ports at first, until someone told me
 about make config-recursive. It really makes me wonder why this isn't the
 default. I remember giving up on FreeBSD when 9.x was new because I had to
 build X from ports after the FreeBSD breach, and it seemed like the process
 was going to take a couple days of stuttering stops and starts as random
 packages I didn't want in some cases popped up between compiles. I learned
 some mechanism for saying just take the defaults but what I know now is
 that what I really wanted was make config-recursive. Why, out of curiosity,
 is it not the default? That would seem better than documenting it harder.

The recommended way of building packages now is to use Poudriere.  The 
Poudriere section in the handbook is still very new and contributions are 
*very* welcome.  I think that having an example of 'how to build libdecss from 
ports' there might be a good idea.

There is a plan that each package set should come with a package containing the 
matching ports tree, so that you can build package from ports that are 
compatible with the binary ones.  That should make a lot of this easier.

I think the two cases for Poudriere that need to be in the handbook are:

- How to build a few custom packages but mostly use upstream for an individual

- How to build a local package repository for your organisation with a load of 
custom config options for packages built from ports and some custom local-only 
ports

 Ah, and one more for the grab bag. I strongly suspect that many folks coming
 from Linux are going to bristle at the notion of using Sendmail. I used to
 run it so I wasn't terribly bothered by it, but maybe pre-populating rc.conf
 with obvious bits that people can see and turn off would be nice. OpenBSD has
 a nice model of populating rc.conf and sysctl.conf fully, and it ends up
 being a pleasant tool. Those awash in wonder, coming from Linux, can say,
 Look, it's all right here!

We put those things in /etc/defaults/rc.conf, which makes merges easier on 
upgrade: the user doesn't touch /etc/defaults/rc.conf and the update tool 
doesn't touch /etc/rc.conf.  Again, if the handbook doesn't tell you to look in 
/etc/defaults/rc.conf then that's an oversight that we should fix.

It might be a good idea to move this thread to the -docs mailing list, as it 
seems to have identified a number of shortcomings in our documentation and it 
would be a good idea to try to find some docs people willing to help get them 
fixed.

David


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, 

Re: Voxer using FreeBSD, BSDNow.tv interview

2014-10-20 Thread David Chisnall
On 19 Oct 2014, at 23:09, Craig Rodrigues rodr...@freebsd.org wrote:

 (2)  Most devops engineers in web/mobile companies are familiar with
Linux.  Any differences between Linux and FreeBSD in
 command-line
utilities are not show-stoppers, but they are annoyances.
Anything FreeBSD could do to help people used to Linux would be
 a big
help.  Allan Jude even brought up my request to symlink
 /bin/bash (
 https://lists.freebsd.org/pipermail/freebsd-ports/2014-September/095483.html
 ) :)

I presume that most of the relevant differences are for users / developers and 
not sysadmins?  It's worth noting that GNU coreutils, tar, bash, and a load of 
other things are in the ports repository.  I wonder if it's worth having a 
gnu-userland metaport, perhaps with something like the Solaris approach of 
sticking them all in a different tree so that you can just add that to the 
start of your PATH and have all of the GNU tools work by default.  

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: usb printer vs cups

2014-09-17 Thread David Chisnall
There are a couple of similar issues currently.  The other one that comes to 
mind is that every X11 application that needs to use OpenGL (or similar) must 
open /dev/dri/{something}, but the default permissions only permit root.

The correct solution is probably to ship a devfs.conf that puts these devices 
in the a sensible group.  For USB printers, we should probably have a printers 
group and make cupsd run with that group (or set the GUI of cups and printers 
to the same number if that's too difficult).  

David

On 17 Sep 2014, at 07:00, Andriy Gapon a...@freebsd.org wrote:

 
 Soliciting help.
 
  Forwarded Message 
 
 From my experience I think that cupsd executes backend tools with all uids and
 gids set to cups and no supplementary groups.  In the case of USB printers the
 backends need to access /dev/usbctl and /dev/usb/foobar that corresponds to a
 printer.  That means that the access to those devices must be somehow granted 
 to
 cups:cups.
 How do people solve this?  What kind of permissions / configuration do you 
 use?
 
 P.S.
 Maybe I over-generalized the issue to all USB printers.  My personal 
 experience
 is with an HP printer handled by hplip / hplip-plugin.
 -- 
 Andriy Gapon
 
 
 ___
 freebsd-current@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-current
 To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: [HEADSUP] pkg(8) is now the only package management tool

2014-09-02 Thread David Chisnall
On 2 Sep 2014, at 12:47, Michelle Sullivan miche...@sorbs.net wrote:

 I'm not happy that the EOL was
 not actually an EOL and it was actually a deadline.

I'm not sure what you think the difference is.  The EOL says 'the FreeBSD 
project no longer supports this configuration'.  If you are not relying on us 
for support (i.e. using an old or forked ports tree, or building your own 
packages), then things will continue to work.  If you are expecting (unpaid, 
volunteer) support from the project in the form of packages and a useable ports 
tree, then you need to use a supported configuration.

If being able to use the ports tree without installing pkg(8) is sufficiently 
valuable to you, then I can put you in touch with some companies that will 
backport things to a copy of the ports tree for your use (although the price 
tag will scale with the number of ports that you want to support).

If, however, your complaint is that it's hard to get new software certified for 
your system *then this change has absolutely no effect on you!*  If you're not 
worried about upgrading ports at all, then you can just stick with the ports 
tree version from the time of your release.  If you're able to upgrade ports, 
then upgrading the pkg port should not be an issue.

David

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


  1   2   3   >