Re: Debugging userspace with Nuttx protected build

[Gregory Nutt]

When I was originally developing the kernel mode, I wrote this: 
https://cwiki.apache.org/confluence/display/NUTTX/Memory+Configurations


That was mostly my notes of what I had worked on and my roadmap that was 
guiding me through the kernel mode development.  I burned out on that 
before finishing everything and since then several other people have 
worked with this build mode.  I don't know the current state.


I would be great to have a similar, updated document.  Although this old 
Wiki page is out of date, it still might answer some questions about 
what's there and why. There is even a section on "Partially Linked vs. 
Fully Linked Objects" in the section of things not finished.



On 4/24/2023 7:27 PM, Andrew Dennison wrote:

Are there any examples of a gdb setup to debug a userspace process? One key
issue is to setup the section offsets to match the final application
location once it has been loaded.

I was actually quite surprised that Nuttx applications are still partially
linked with a protected build and that all sections overlap (start at 0) in
the elf file. I had expected the application would be fully linked to
virtual addresses as per linux and another RTOS I used previously. Is this
something that can be changed relatively easily, or are there some
significant challenges?

Kind regards,

Andrew

Re: Debugging userspace with Nuttx protected build

[Tiago Medicci Serrano]

Hi!

I had no problems debugging NuttX with protected mode on ESP32-S3
(esp32s3-devkit:knsh).

After launching OpenOCD (https://github.com/espressif/openocd-esp32) with
`-c 'set ESP_RTOS none' -c "set ESP_FLASH_SIZE 0"`, just load the
userspace's symbols with `add-symbol-file` (with the address of the `.text`
region, that you can check with `objdump`, for instance)

xtensa-esp32s3-elf-gdb -ex "target extended-remote :"  -ex "symbol-file
nuttx"  -ex "add-symbol-file nuttx_user.elf 0x4209"  -ex "set remote
hardware-watchpoint-limit 2" -ex "mon reset halt" -ex "flushregs" -ex "set
print pretty" --tui

It works as expected on ESP32-S3.

Best regards,

Em ter., 25 de abr. de 2023 às 09:28, Gregory Nutt 
escreveu:

>
>  >>> I was actually quite surprised that Nuttx applications are still
> partially
>  >>> linked with a protected build and that all sections overlap (start
> at 0)in
>  >>> the elf file. I had expected the application would be fully linked to
>  >>> virtual addresses as per linux and another RTOS I used previously.
> Is this
>  >>> something that can be changed relatively easily, or are there some
>  >>> significant challenges?
>  >>
>  >> That would effect the build logic and the binary loader. The effort
>  >> would probably be significant.
>  >>
>  >> Are you doing a kernel build?  In that case, all applications are
>  >> partially linked ELF modules as you describe.  Linux positions .bss,
>  >> .data, immediately after .text in the virtual address space.  Heap and
>  >> thread stacks and other things are after that. So Linux applications
> can
>  >> be fully linked.
>  >
>  > I used a similar capability with a different small RTOS a while ago.
>  >
>  >> NuttX allocates everything from the heap when the module is loaded.  So
>  >> the application must be fully relocatable and address fix-ups are
>  >> needed.  So the ELF module is a partially linked relocatable ELF file.
>  >>
>  >
>  > For the kernel build (or MMU support in general) I'd assumed Nuttx would
>  > statically link and use the MMU to map the fixed elf section addresses
> to
>  > the pages allocated from the heap. This also helps startup performance
> as
>  > there are no relocations to process. But we're still learning the
> details
>  > of how Nuttx goes together.
>
> It was late last night when I responded.  In the clear light of morning
> I can see that my "justification" is lame and probably wrong.
>
> The REAL reason that the ELF modules work as they do is because they
> were designed to work in the FLAT build with all physical addressing.
> It is used with the KERNEL build because it works there too.
>
> That can't be changed only for the KERNEL build, but there could be a
> fully linked ELF binary just for the KERNEL build.  That should be too
> difficult.  Things that occur to me are:
>
>   * There are no tools for building such as executable.  It would
> probably have to be implemented within a Makefile as is the current
> ELF module build.
>   * There are no shared libraries so everything .. libc, syscalls, etc.
> ... has to be statically linked.  I think that is true of the
> existing ELF modules in the KERNEL build anyway.
>   * Some minor tweaks to crt0 might be necessary (?).
>
>

-- 
Tiago Medicci Serrano

Embedded Software Engineer
MSc Electronics/Microelectronics
m: +55 (19) 981403886 <+55+(19)+981403886>
e: tiago.medi...@gmail.com
a: Campinas, Brazil
Follow me:

Re: Debugging userspace with Nuttx protected build

[Gregory Nutt]

>>> I was actually quite surprised that Nuttx applications are still 
partially
>>> linked with a protected build and that all sections overlap (start 
at 0)in

>>> the elf file. I had expected the application would be fully linked to
>>> virtual addresses as per linux and another RTOS I used previously. 
Is this

>>> something that can be changed relatively easily, or are there some
>>> significant challenges?
>>
>> That would effect the build logic and the binary loader. The effort
>> would probably be significant.
>>
>> Are you doing a kernel build?  In that case, all applications are
>> partially linked ELF modules as you describe.  Linux positions .bss,
>> .data, immediately after .text in the virtual address space.  Heap and
>> thread stacks and other things are after that. So Linux applications can
>> be fully linked.
>
> I used a similar capability with a different small RTOS a while ago.
>
>> NuttX allocates everything from the heap when the module is loaded.  So
>> the application must be fully relocatable and address fix-ups are
>> needed.  So the ELF module is a partially linked relocatable ELF file.
>>
>
> For the kernel build (or MMU support in general) I'd assumed Nuttx would
> statically link and use the MMU to map the fixed elf section addresses to
> the pages allocated from the heap. This also helps startup performance as
> there are no relocations to process. But we're still learning the details
> of how Nuttx goes together.

It was late last night when I responded.  In the clear light of morning 
I can see that my "justification" is lame and probably wrong.


The REAL reason that the ELF modules work as they do is because they 
were designed to work in the FLAT build with all physical addressing.  
It is used with the KERNEL build because it works there too.


That can't be changed only for the KERNEL build, but there could be a 
fully linked ELF binary just for the KERNEL build.  That should be too 
difficult.  Things that occur to me are:


 * There are no tools for building such as executable.  It would
   probably have to be implemented within a Makefile as is the current
   ELF module build.
 * There are no shared libraries so everything .. libc, syscalls, etc.
   ... has to be statically linked.  I think that is true of the
   existing ELF modules in the KERNEL build anyway.
 * Some minor tweaks to crt0 might be necessary (?).

Re: Debugging userspace with Nuttx protected build

[Andrew Dennison]

On Tue, Apr 25, 2023, 3:10 PM Gregory Nutt  wrote:

>
> On 4/24/2023 7:27 PM, Andrew Dennison wrote:
> > Are there any examples of a gdb setup to debug a userspace process? One
> key
> > issue is to setup the section offsets to match the final application
> > location once it has been loaded.
>
> You don't describe your build, so I only give you a general answer.
>


This is for a kernel build targeting risvc32.



> This does not specifically address what you need to do, but might be
> helpful:
>
> https://cwiki.apache.org/confluence/display/NUTTX/Debugging+ELF+Loadable+Modules
>
> I think the basics would be the same for all builds.
>

Similar to debugging Linux kernel modules.


> - You need to be able to stop the ELF module (FLAT or KERNEL build) or
> in FLASH program (PROTECTED build).
>
> - Then you can just add the symbol table of the ELF module )or user
> space) to the debugger.
>
> > I was actually quite surprised that Nuttx applications are still
> partially
> > linked with a protected build and that all sections overlap (start at 0)
> in
> > the elf file. I had expected the application would be fully linked to
> > virtual addresses as per linux and another RTOS I used previously. Is
> this
> > something that can be changed relatively easily, or are there some
> > significant challenges?
>
> That would effect the build logic and the binary loader.  The effort
> would probably be significant.
>
> Are you doing a kernel build?  In that case, all applications are
> partially linked ELF modules as you describe.  Linux positions .bss,
> .data, immediately after .text in the virtual address space.  Heap and
> thread stacks and other things are after that. So Linux applications can
> be fully linked.
>

I used a similar capability with a different small RTOS a while ago.


> NuttX allocates everything from the heap when the module is loaded.  So
> the application must be fully relocatable and address fix-ups are
> needed.  So the ELF module is a partially linked relocatable ELF file.
>

For the kernel build (or MMU support in general) I'd assumed Nuttx would
statically link and use the MMU to map the fixed elf section addresses to
the pages allocated from the heap. This also helps startup performance as
there are no relocations to process. But we're still learning the details
of how Nuttx goes together.

Re: Debugging userspace with Nuttx protected build

[Brennan Ashton]

On Tue, Apr 25, 2023, 6:10 AM Gregory Nutt  wrote:

>
> On 4/24/2023 7:27 PM, Andrew Dennison wrote:
> > Are there any examples of a gdb setup to debug a userspace process? One
> key
> > issue is to setup the section offsets to match the final application
> > location once it has been loaded.
>

This won't help you today as it is not yet implemented, but longer term I
think it would be really nice to get ptrace support implemented so we could
allow for a debug server stub to run in userspace.

https://github.com/apache/nuttx/issues/2028

--Brennan

Re: Debugging userspace with Nuttx protected build

[Gregory Nutt]

On 4/24/2023 7:27 PM, Andrew Dennison wrote:

Are there any examples of a gdb setup to debug a userspace process? One key
issue is to setup the section offsets to match the final application
location once it has been loaded.


You don't describe your build, so I only give you a general answer.

This does not specifically address what you need to do, but might be 
helpful: 
https://cwiki.apache.org/confluence/display/NUTTX/Debugging+ELF+Loadable+Modules


I think the basics would be the same for all builds.

- You need to be able to stop the ELF module (FLAT or KERNEL build) or 
in FLASH program (PROTECTED build).


- Then you can just add the symbol table of the ELF module )or user 
space) to the debugger.



I was actually quite surprised that Nuttx applications are still partially
linked with a protected build and that all sections overlap (start at 0) in
the elf file. I had expected the application would be fully linked to
virtual addresses as per linux and another RTOS I used previously. Is this
something that can be changed relatively easily, or are there some
significant challenges?


That would effect the build logic and the binary loader.  The effort 
would probably be significant.


Are you doing a kernel build?  In that case, all applications are 
partially linked ELF modules as you describe.  Linux positions .bss, 
.data, immediately after .text in the virtual address space.  Heap and 
thread stacks and other things are after that. So Linux applications can 
be fully linked.


NuttX allocates everything from the heap when the module is loaded.  So 
the application must be fully relocatable and address fix-ups are 
needed.  So the ELF module is a partially linked relocatable ELF file.

Debugging userspace with Nuttx protected build

[Andrew Dennison]

Are there any examples of a gdb setup to debug a userspace process? One key
issue is to setup the section offsets to match the final application
location once it has been loaded.

I was actually quite surprised that Nuttx applications are still partially
linked with a protected build and that all sections overlap (start at 0) in
the elf file. I had expected the application would be fully linked to
virtual addresses as per linux and another RTOS I used previously. Is this
something that can be changed relatively easily, or are there some
significant challenges?

Kind regards,

Andrew