Re: Adding a boot flag for No ASLR

[Kamil Rytarowski]

On 24.07.2018 14:53, Joerg Sonnenberger wrote:
> On Tue, Jul 24, 2018 at 06:44:52AM +0200, Martin Husemann wrote:
>> On Mon, Jul 23, 2018 at 11:02:04PM +0200, Kamil Rytarowski wrote:
>>> We need to maintain a function to translate certain ranges to
>>> shadow/meta/origin/etc. We cannot map arbitrarily wide ranges to them.
>>
>> Can we extend the pax note (or create a new one) and make the sanitizers
>> link that in? Then make the kernel reserve some (random) VA spaces
>> (details of what is needed in the note) and provide some way for the
>> sanitizers to find that special VAs (like from the aux vector)?
> 
> Yes, all sanitized binaries should contain a note if they require
> certain fixed mappings to be reserved. I don't think there is *any* need
> to disable ASLR beyond that.
> 
> Joerg
> 

Ok, Please prepare a patch for this and I can test it. I don't think it
will be sufficient (as noted in my previous mails).

On my end I will check whether reexec() in init(8) works and if not, I
will refer to paxctl(8).



signature.asc
Description: OpenPGP digital signature

Re: Adding a boot flag for No ASLR

[Joerg Sonnenberger]

On Tue, Jul 24, 2018 at 02:08:53PM +0200, Kamil Rytarowski wrote:
> I propose to move the code disabling PaX ASLR from bootloader and kernel
> as proposed in the patch by Siddharth and introduce it directly into the
> sanitizer, We can alter the CheckASLR() routine specific to NetBSD, with
> the following pseudo-code:

You can't disable ASLR at this point. It is too late. paxctl as hack is
good enough until the proper note processing and fixed VM space layout
is implemented.

Joerg

Re: Adding a boot flag for No ASLR

[Joerg Sonnenberger]

On Tue, Jul 24, 2018 at 06:44:52AM +0200, Martin Husemann wrote:
> On Mon, Jul 23, 2018 at 11:02:04PM +0200, Kamil Rytarowski wrote:
> > We need to maintain a function to translate certain ranges to
> > shadow/meta/origin/etc. We cannot map arbitrarily wide ranges to them.
> 
> Can we extend the pax note (or create a new one) and make the sanitizers
> link that in? Then make the kernel reserve some (random) VA spaces
> (details of what is needed in the note) and provide some way for the
> sanitizers to find that special VAs (like from the aux vector)?

Yes, all sanitized binaries should contain a note if they require
certain fixed mappings to be reserved. I don't think there is *any* need
to disable ASLR beyond that.

Joerg

Re: Adding a boot flag for No ASLR

[Kamil Rytarowski]

On 24.07.2018 06:44, Martin Husemann wrote:
> On Mon, Jul 23, 2018 at 11:02:04PM +0200, Kamil Rytarowski wrote:
>> We need to maintain a function to translate certain ranges to
>> shadow/meta/origin/etc. We cannot map arbitrarily wide ranges to them.
> 
> Can we extend the pax note (or create a new one) and make the sanitizers
> link that in? Then make the kernel reserve some (random) VA spaces
> (details of what is needed in the note) and provide some way for the
> sanitizers to find that special VAs (like from the aux vector)?
> 
> Martin
> 

PaX ELF Note is already inlined from lib/csu into every binary.

ASan, TSan and MSan are not designed for hardening, but for bug
detecting in the process of development.

I don't think that there is really need for changing the PaX ASLR code
to be compatible with them, it's sufficient that we can disable this
option. With !ASLR all currently known and potentially new problems are
gone.


We can handle it differently.

I propose to move the code disabling PaX ASLR from bootloader and kernel
as proposed in the patch by Siddharth and introduce it directly into the
sanitizer, We can alter the CheckASLR() routine specific to NetBSD, with
the following pseudo-code:

if (getpid() == 1) {
 disable_pax_aslr();
 reexec().
}

The reexec() functionality is already used by sanitizers and it is
supported on NetBSD.

This will be a special case for init(8) with a minimal extra code.

Pros:
 - no changes to the bootloader and booting process
 - no changes in the kernel
 - no new or changed ELF notes
 - no nee to rework PaX ASLR
 - no source code changed to init(8)

Cons:
 - it might seem ugly



signature.asc
Description: OpenPGP digital signature

Re: Adding a boot flag for No ASLR

[Martin Husemann]

On Mon, Jul 23, 2018 at 11:02:04PM +0200, Kamil Rytarowski wrote:
> We need to maintain a function to translate certain ranges to
> shadow/meta/origin/etc. We cannot map arbitrarily wide ranges to them.

Can we extend the pax note (or create a new one) and make the sanitizers
link that in? Then make the kernel reserve some (random) VA spaces
(details of what is needed in the note) and provide some way for the
sanitizers to find that special VAs (like from the aux vector)?

Martin

Re: Adding a boot flag for No ASLR

[Kamil Rytarowski]

On 23.07.2018 21:32, Joerg Sonnenberger wrote:
> On Mon, Jul 23, 2018 at 07:13:49PM +0200, Kamil Rytarowski wrote:
>> We need to have stack, heap and code of a program in predictable (and
>> quite narrow) ranges and thus ASLR disabled or less aggressive.
> 
> What for? Nothing in the sanitizer design should require that. The only
> requirement should be that the shadow area can be mapped at a fixed
> location.
> 

We need to maintain a function to translate certain ranges to
shadow/meta/origin/etc. We cannot map arbitrarily wide ranges to them.
Depending on the sanitizer there might be a need for multiple
shadow/origin/meta buffers - this is the case of TSan and MSan. ASan
uses a single shadow buffer and it sometimes works. ASan is also a
special case as it was designed to function in narrow address space
(below 32-bit). With TSan I've discovered that there is no room with the
current randomization to maintain a set of functions to map into meta
buffers.

So far there were no problems observed with mapping a buffer at a fixed
address (but I don't say that there are none). The issues pop up with
discovered mappings out of expected ranges and assert() in the best
case, or random corruption and crash in the worst one. It used to be
simple to miss that there is ASLR enabled for a process and debug a new
crash sanitizers - to prevent this, I've included upstream a function
CheckASLR() to detect it and report verbosely.

On the other hand, Linux ASLR and Fuchsia ASLR are compatible with the
sanitizers and data regions are in predictable and narrow ranges in the
process virtual address space.

> Joerg
> 




signature.asc
Description: OpenPGP digital signature

Re: Adding a boot flag for No ASLR

[Joerg Sonnenberger]

On Mon, Jul 23, 2018 at 07:13:49PM +0200, Kamil Rytarowski wrote:
> We need to have stack, heap and code of a program in predictable (and
> quite narrow) ranges and thus ASLR disabled or less aggressive.

What for? Nothing in the sanitizer design should require that. The only
requirement should be that the shadow area can be mapped at a fixed
location.

Joerg

Re: Adding a boot flag for No ASLR

[Kamil Rytarowski]

On 23.07.2018 15:09, Joerg Sonnenberger wrote:
> On Mon, Jul 23, 2018 at 06:24:09PM +0530, Siddharth Muralee wrote:
>>>
>>>
>>> (1) An implementation detail of userland shouldn't be leaked into the
>>> kernel boot (!) process.
>>>
>>
>> Okay. I think this makes sense(I am still pretty new to NetBSD) - Can you
>> suggest some other location/config that can be used.
> 
> paxctl.
> 

OK, this is another approach and it should work.

>>> (2) There is no fundamental issue that makes the sanitizers incompatible
>>> with ASLR. The only issue for asan and friends is the reservation of the
>>> shadow buffer and that can and should be handled explicitly.
>>>
>>
>> We have implemented the ATF tests for ASan - The tests work only 50% or
>> less when ASLR is on. To get perfect results I think ASLR needs to be off.
>> I guess Kamil can provide more info on this.
> 
> I'm very aware of the current situation. Ultimately, stack randomisation
> has the same issue. The way we setup the VM space of a new process is
> suboptimal for a world that wants to randomize things. I.e. at the
> moment, the VM commands (epp->ep_vmcmds) are executed in order and that
> makes placing fixed location objects difficult. What should happen is:
> (1) Each VM object should grow an object group field. VM objects in the
> same group are assigned a random location together. A special group
> field value of 0 means no randomisation.
> (2) Locations should be assigned first to fixed position fields and
> otherwise in descending order of size.
> (3) The stack of the main thread should be reserved and integrated into
> the VM object reservation just like the rest. The current stack
> randomisation should be removed.
> 
> It should be noted that (2) needs to deal with impossible allocations,
> so it should do one pass to size up each free range in the address space
> that can fit the requested object, pick a random value and then as
> second iteration find the correct range. to split.
> 

This part is not clear for us. If this means that we shall rework ASLR
in order to be compatible with sanitizers, than it's not viable on our
own as of now (mostly as unclear goals here and we have got enough
tasks)... but we are up to test a patch.



We need to have stack, heap and code of a program in predictable (and
quite narrow) ranges and thus ASLR disabled or less aggressive.

In the TSan example, we can reuse the Linux and FreeBSD address space
with the following assumptions:

C/C++ on linux/x86_64 and freebsd/x86_64
  1000 - 0080  : main binary and/or MAP_32BIT mappings
(512GB)
0040   - 0100  : -
0100   - 2000  : shadow
2000   - 3000  : -
3000   - 4000  : metainfo (memory blocks and sync objects)
4000   - 5500  : -
5500   - 5680  : pie binaries without ASLR or on 4.1+
kernels
5680   - 6000  : -
6000   - 6200  : traces
6200   - 7d00  : -
7b00   - 7c00  : heap
7c00   - 7e80  : -
7e80   - 8000  : modules and main thread stack

I've documented the NetBSD specifics as:

C/C++ on netbsd/amd64 can reuse the same mapping:
 * The address space starts from 0x1000 (option with 0x0) and ends with
   0x7f7ff000.
 * LoAppMem-kHeapMemEnd can be reused as it is.
 * No VDSO support.
 * No MidAppMem region.
 * No additional HeapMem region.

 * HiAppMem contains the stack, loader, shared libraries and heap.
 * Stack on NetBSD/amd64 has prereserved 128MB.
 * Heap grows downwards (top-down).
 * ASLR must be disabled per-process or globally.

 -- lib/tsan/rtl/tsan_platform.h

I've found it impossible to handle TSan with enabled ASLR, taking the
example of a running firefox, so a real-life application that we would
want to sanitize.

This approach of disabled ASLR ended up as needed for ASan and MSan too.

> Joerg
> 

On 23.07.2018 15:57, Mouse wrote:
>>> I have recently been working on adding a new boot flag for disabling
>>> ASLR during boot.  [...useful for some userland stuff...]
>
> What's wrong with just configuring a kernel without any ASLR at all for
> such work?
>

Nothing wrong and as stated in the beginning, we are doing it.

Building and running a custom kernel config or patching the kernel
sources seems just to be more than enough to run init(8) and other
programs with a sanitizer. It's certainly not convenient. With the
proposed bootloader flag we can reuse the same kernel without modifications.



signature.asc
Description: OpenPGP digital signature

Re: Adding a boot flag for No ASLR

[Joerg Sonnenberger]

On Mon, Jul 23, 2018 at 06:24:09PM +0530, Siddharth Muralee wrote:
> >
> >
> > (1) An implementation detail of userland shouldn't be leaked into the
> > kernel boot (!) process.
> >
> 
> Okay. I think this makes sense(I am still pretty new to NetBSD) - Can you
> suggest some other location/config that can be used.

paxctl.

> > (2) There is no fundamental issue that makes the sanitizers incompatible
> > with ASLR. The only issue for asan and friends is the reservation of the
> > shadow buffer and that can and should be handled explicitly.
> >
> 
> We have implemented the ATF tests for ASan - The tests work only 50% or
> less when ASLR is on. To get perfect results I think ASLR needs to be off.
> I guess Kamil can provide more info on this.

I'm very aware of the current situation. Ultimately, stack randomisation
has the same issue. The way we setup the VM space of a new process is
suboptimal for a world that wants to randomize things. I.e. at the
moment, the VM commands (epp->ep_vmcmds) are executed in order and that
makes placing fixed location objects difficult. What should happen is:
(1) Each VM object should grow an object group field. VM objects in the
same group are assigned a random location together. A special group
field value of 0 means no randomisation.
(2) Locations should be assigned first to fixed position fields and
otherwise in descending order of size.
(3) The stack of the main thread should be reserved and integrated into
the VM object reservation just like the rest. The current stack
randomisation should be removed.

It should be noted that (2) needs to deal with impossible allocations,
so it should do one pass to size up each free range in the address space
that can fit the requested object, pick a random value and then as
second iteration find the correct range. to split.

Joerg

Re: Adding a boot flag for No ASLR

[Mouse]

>> I have recently been working on adding a new boot flag for disabling
>> ASLR during boot.  [...useful for some userland stuff...]

What's wrong with just configuring a kernel without any ASLR at all for
such work?

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: Adding a boot flag for No ASLR

[Joerg Sonnenberger]

On Mon, Jul 23, 2018 at 06:11:49PM +0530, Siddharth Muralee wrote:
> Hello,
>I have recently been working on adding a new boot flag for disabling
> ASLR during boot. I feel that this is useful since MKSANITZER userland
> (specifically Address Sanitizer, Thread Sanitizer, and Memory Sanitizer)
> requires ASLR to be disabled. Till now it was hardcoded in the kernel to be
> disabled and this looks neater.

I'm strongly against this.

(1) An implementation detail of userland shouldn't be leaked into the
kernel boot (!) process.
(2) There is no fundamental issue that makes the sanitizers incompatible
with ASLR. The only issue for asan and friends is the reservation of the
shadow buffer and that can and should be handled explicitly.

Joerg

Re: Adding a boot flag for No ASLR

[Kamil Rytarowski]

On 23.07.2018 14:54, Siddharth Muralee wrote:
> 
> (1) An implementation detail of userland shouldn't be leaked into the
> kernel boot (!) process.
> 
> 
> Okay. I think this makes sense(I am still pretty new to NetBSD) - Can
> you suggest some other location/config that can be used.
> 

no-ASLR can be treated independently of the userland detail. I disable
it myself anyway using sysctl.conf(5) and I would switch to boot option.

For MKSANITIZER with the current PaX ASLR this is a mandatory option.

I think that leaking userland detail is overstatement.

Not sanitizing init(8) is not an option as it pulls in external
libraries that are sanitized -lutil and -lcrpyt.

> 
> (2) There is no fundamental issue that makes the sanitizers incompatible
> with ASLR. The only issue for asan and friends is the reservation of the
> shadow buffer and that can and should be handled explicitly.
> 
>  
> We have implemented the ATF tests for ASan - The tests work only 50% or
> less when ASLR is on. To get perfect results I think ASLR needs to be off.
> I guess Kamil can provide more info on this.
> 

Shadow and metadata memory is just a part of the implementation detail.
We are required to keep the remaining data of the program in address
space in predictable ranges. With PaX ASLR the ranges are fluid and
impossible to get functional.

OpenBSD due to ASR is impossible to get functional with these
sanitizers, it's the same story with HardenedBSD and their ASLR.

As an alternative there would be an option to highly reduce the entropy
of ASLR, but its purpose would be gone.

ASan is more tolerant to PaX ASLR and it sometimes works by in basic
scenarios. TSan and MSan are fully incompatible with PaX ASLR.

In the current version sanitizers (ASan, TSan, MSan) bail out on
detected ASLR with a failure rather than dying on some internal
corruption and crash.

> -- 
> Regards, 
>   Siddharth M
>   Third Year B.Tech (CSE) Student,
>   Amrita School of Engineering, Kollam
> /  Blog  /
> /---/
> /“Most people get ahead during the time that others waste.//"/




signature.asc
Description: OpenPGP digital signature

Re: Adding a boot flag for No ASLR

[Robert Elz]

Date:Mon, 23 Jul 2018 14:48:06 +0200
From:Martin Husemann 
Message-ID:  <20180723124806.gf18...@mail.duskware.de>

  | Why don't you just use /etc/sysctl.conf?

I think that would be a little late, by the time anything in userland
gets a chance to do anything at all (including just starting /sbin/init)
ASLR has already happened.

I think that having a boot time option to make it all go away is a
very good idea.   I have not yet looked at the patch though.

kre

Re: Adding a boot flag for No ASLR

[Siddharth Muralee]

>
>
> (1) An implementation detail of userland shouldn't be leaked into the
> kernel boot (!) process.
>

Okay. I think this makes sense(I am still pretty new to NetBSD) - Can you
suggest some other location/config that can be used.


> (2) There is no fundamental issue that makes the sanitizers incompatible
> with ASLR. The only issue for asan and friends is the reservation of the
> shadow buffer and that can and should be handled explicitly.
>

We have implemented the ATF tests for ASan - The tests work only 50% or
less when ASLR is on. To get perfect results I think ASLR needs to be off.
I guess Kamil can provide more info on this.

-- 
Regards,
  Siddharth M
  Third Year B.Tech (CSE) Student,
  Amrita School of Engineering, Kollam
*  Blog  *
*---*
*“Most people get ahead during the time that others waste.**"*

Re: Adding a boot flag for No ASLR

[Kamil Rytarowski]

On 23.07.2018 14:48, Martin Husemann wrote:
> On Mon, Jul 23, 2018 at 06:11:49PM +0530, Siddharth Muralee wrote:
>> Hello,
>>I have recently been working on adding a new boot flag for disabling
>> ASLR during boot. I feel that this is useful since MKSANITZER userland
>> (specifically Address Sanitizer, Thread Sanitizer, and Memory Sanitizer)
>> requires ASLR to be disabled. Till now it was hardcoded in the kernel to be
>> disabled and this looks neater.
> 
> Why don't you just use /etc/sysctl.conf?
> 

It must be disabled before executing init(8), the boot dies before
reaching rc.d/sysctl.

> Martin
> 




signature.asc
Description: OpenPGP digital signature

Re: Adding a boot flag for No ASLR

[Martin Husemann]

On Mon, Jul 23, 2018 at 06:11:49PM +0530, Siddharth Muralee wrote:
> Hello,
>I have recently been working on adding a new boot flag for disabling
> ASLR during boot. I feel that this is useful since MKSANITZER userland
> (specifically Address Sanitizer, Thread Sanitizer, and Memory Sanitizer)
> requires ASLR to be disabled. Till now it was hardcoded in the kernel to be
> disabled and this looks neater.

Why don't you just use /etc/sysctl.conf?

Martin

Adding a boot flag for No ASLR

[Siddharth Muralee]

Hello,
   I have recently been working on adding a new boot flag for disabling
ASLR during boot. I feel that this is useful since MKSANITZER userland
(specifically Address Sanitizer, Thread Sanitizer, and Memory Sanitizer)
requires ASLR to be disabled. Till now it was hardcoded in the kernel to be
disabled and this looks neater.

The current patch[1] is only for amd64 and it works fine for me.

I would like to get feedback on the same. I will also update the
documentation of boothowto(9), boot(8) and security(7) if the patch is
accepted.

[1]  http://netbsd.org/~kamil/mksanitizer-reports/boot_flag.diff
-- 
Regards,
  Siddharth M
  Third Year B.Tech (CSE) Student,
  Amrita School of Engineering, Kollam
*  Blog  *
*---*
*“Most people get ahead during the time that others waste.**"*