Re: qemu-x86_64, buster /sbin/ldconfig and setup_arg_pages (a mind dump)

[Richard Henderson]

On 1/17/20 7:33 AM, Alex Bennée wrote:
> So what do we take away from this?
> 
>  * we need testcases to exercise the memory layout of dynamic binaries
>  * "special" dynamic binaries can break our careful memory layout
>  * I feel as though I've trodden on a nest of vipers
> 
> Does any of this track with you? What is different about ldconfig that
> breaks our memory placement?

We've been through this bug before, but never came to a resolution.  See

https://bugs.launchpad.net/qemu/+bug/1749393
https://lists.nongnu.org/archive/html/qemu-devel/2018-03/msg04700.html


r~

qemu-x86_64, buster /sbin/ldconfig and setup_arg_pages (a mind dump)

[Alex Bennée]

Hi Richard,

While I was attempting to test the new vsyscall patches for x86 I
discovered I couldn't debootstrap an x86_64 buster image on my ARM box.
After digging further into it I discovered it was because executing
/sbin/ldconfig crashes and aborts the bootstrap.

This is helpfully reproducible on my main development system which is
also running buster:

  ./x86_64-linux-user/qemu-x86_64 /sbin/ldconfig
  setup_arg_pages: 004e
  target_set_brk: new_brk=004dfdf8
  do_brk() -> 004e (!new_brk)
  do_brk(004e11c0) -> do_brk: allocating 8192 => 7fb2dace5000
  004e (mapped_addr != -1 or brk_page)
  qemu: uncaught target signal 11 (Segmentation fault) - core dumped
  fish: Job 2, “./x86_64-linux-user/qemu-x86_64…” terminated by signal SIGSEGV 
(Address boundary error)

The failure of the second do_brk during the early setup of the binaries
TLS data area. However for some reason this isn't always the case. For
example with testthread which also uses TLS:

  ./x86_64-linux-user/qemu-x86_64 ./tests/tcg/x86_64-linux-user/testthread
  setup_arg_pages: 0040
  target_set_brk: new_brk=004c8558
  do_brk() -> 004c9000 (!new_brk)
  do_brk(004ca1c0) -> do_brk: allocating 8192 => 004c9000
  004ca1c0 (mapped_addr == brk_page)
  do_brk(004eb1c0) -> do_brk: allocating 135168 => 004cb000
  004eb1c0 (mapped_addr == brk_page)
  do_brk(004ec000) -> 004ec000 (new_brk <= brk_page)
  thread1: 0 hello1
  thread2: 0 hello2
  thread1: 1 hello1

Ultimately the failure is down to setup_arg_pages allocating too low in
the address space in the ldconfig case which leaves the second brk
unable to example it's region of memory. Turning on -d page and you can
see the region forming:

  page layout changed following target_mmap
  startend  size prot
  0040-00409000 9000 r--
  00409000-004ae000 000a5000 r-x
  004ae000-004d8000 0002a000 r--
  004d8000-004df000 7000 rw-
  004df000-004e 1000 ---
  page layout changed following target_mmap
  startend  size prot
  0040-00409000 9000 r--
  00409000-004ae000 000a5000 r-x
  004ae000-004d8000 0002a000 r--
  004d8000-0040008e1000 00809000 rw-
  setup_arg_pages: 004e
  guest_base  0x0
  page layout changed following binary load
  startend  size prot
  0040-00409000 9000 r--
  00409000-004ae000 000a5000 r-x
  004ae000-004d8000 0002a000 r--
  004d8000-004e 8000 rw-
  004e-004e1000 1000 ---
  004e1000-0040008e1000 0080 rw-
  start_brk   0x
  end_code0x004ad971
  start_code  0x00409000
  start_data  0x004d8778
  end_data0x004de510
  start_stack 0x0040008e02d0
  brk 0x004dfdf8
  entry   0x0040a370
  argv_start  0x0040008e02d8
  env_start   0x0040008e02e8
  auxv_start  0x0040008e0428
  target_set_brk: new_brk=004dfdf8
  page layout changed following target_mmap
  startend  size prot
  0040-00409000 9000 r--
  00409000-004ae000 000a5000 r-x
  004ae000-004d8000 0002a000 r--
  004d8000-004e 8000 rw-
  004e-004e1000 1000 ---
  004e1000-0040008e2000 00801000 rw-

So it looks like setup_arg_pages just creates a segment right in the
middle of a previously allocated block of storage. This is odd because
the loader basically just leaves it to mmap to pick a region:

error = target_mmap(0, size + guard, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

AFAICT this just depends on where we have allocated last, in the
testthread case we already have a high mapping to splat:

  page layout changed following target_mmap
  startend  size prot
  0040-00401000 1000 r--
  00401000-00495000 00094000 r-x
  00495000-004bc000 00027000 r--
  004bd000-004c9000 c000 rw-
  0040-004000801000 00801000 rw-
  setup_arg_pages: 0040
  guest_base  0x0
  page layout changed following binary load
  startend  size prot
  0040-00401000 1000 r--