Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Amaan Cheval]

Minor update: I'll work on Chris' suggestion of using FreeBSD's
loader.efi and having that load our static hello.exe - it ought to be
a quicker test that way.
On Sun, Jun 10, 2018 at 9:34 PM Amaan Cheval  wrote:
>
> On Sun, Jun 10, 2018 at 12:38 AM Joel Sherrill  wrote:
> >
> >
> >
> > On Fri, Jun 8, 2018 at 7:45 PM, Chris Johns  wrote:
> >>
> >> On 9/6/18 10:00 am, Joel Sherrill wrote:
> >> > On Thu, Jun 7, 2018, 9:01 PM Chris Johns  >> > > wrote:
> >> > > and what
> >> > > discussions we need to have to decide between the "bundled 
> >> > kernel.so approach"
> >> > > (the one implemented here) vs. the "FreeBSD loader.efi+hello.exe"
> >> > approach. Let
> >> > > me know!
> >> > >
> >> >
> >> > I do not think I can help too much here. I understand the 
> >> > loader.efi+exe
> >> > solution and it should work because all RTEMS applications we have 
> >> > are
> >> > statically linked (I am assuming it is here). I have not looked at 
> >> > the details
> >> > being used with the -fPIC and .so solution so I cannot comment. I do 
> >> > have some
> >> > concerns the relocatable exe might expose some dark corners and 
> >> > issues in the
> >> > host tools we have, for example how does GDB find the base address 
> >> > of the image
> >> > so you can debug it? and is this just working or is it really 
> >> > suppose to work
> >> > this way?
> >> >
> >> >
> >> > All I can say is that with Deos/RTEMS, we use PIC on arm, PowerPC, and 
> >> > x86.
> >>
> >> I would hope a solution like Deos did provide a seamless way to do this 
> >> and I
> >> would also hope they support you.
> >
> >
> > I am not using their normal recommended tool setup for users. This is normal
> > RTEMS tools building our test executables. At one point, it was our gdb with
> > their qemu build. They use something like this strictly internally.
> >
> > These executables are statically linked EXCEPT for references to, in
> > the minimum case, two .so's from their environment. I set an argument to
> > ld to ensure all symbols are resolved at link time. Their boot process 
> > associates
> > the .so files with the partitions. It is dynamic loading but it is 
> > statically configured
> > and not touched after boot.
> >
> > I haven't had any special help from them in this area except figuring out 
> > the
> > arguments and linker scripts. When I have access to a build log, I am happy
> > to post the build of hello world for comparison.
> >
> > I have no idea how this compares to UEFI booting except to say that PIC
> > hasn't introduced any tool issues in our GNU tools. libdl may have issues
> > but we aren't using it yet. I can check if the tests pass or are disabled. I
> > don't remember. But that may be illuminating.
> >
> >>
> >>
> >> > We
> >> > have spent a lot of time debugging with gdb attached to qemu.
> >> How does GDB get the relocatable load address to map to the symbol table?
> >>
> >> The libdl code supports the same protocol/design as NetBSD and other 
> >> systems in
> >> informing GDB about the address of loaded modules. There is a series of 
> >> symbols
> >> and tables maintained that GDB knows to examine to find the load addresses 
> >> of
> >> object files.
> >>
> >> > I haven't seen any tools issues yet.
> >>
> >> Yet?  Once the path is settled it will be difficult to change so all I am 
> >> asking
> >> is the detail be checked and understood. RTEMS does not support shared 
> >> libraries
> >> the same way Linux or other Unix systems do. I do not understand enough of 
> >> the
> >> low level and the standards all this is based on to help decide.
> >>
> >> An example of an issue where a relocatable kernel with an unknown load 
> >> address
> >> creates a problem is libdl. The testsuite uses the 2-pass approach 
> >> (rtems-syms
> >> --embed) which should be OK however the other approach where the symbol 
> >> table is
> >> not embedded and built on the host would fail. It is a small issue but it 
> >> shows
> >> how things can subtly break.
> >
> >
> > I'm not relocating any RTEMS code with Deos. Our code is linked to a static 
> > address
> > ranges and invokes Deos methods in the shared libbary.
> >
> > I know this doesn't prove anything concretely about the UEFI exe  but it is 
> > the closest
> > example we have. PIC is likely OK. The .so magic could be problematic as it 
> > looks
> > like I effectively build a static exe.
>
> The Bsymbolic flag isn't a requirement for this system to work - it
> just helps eliminate the use of the GOT/PLT unnecessarily. Personally,
> I don't think I have the clarity to be able to say whether this is or
> isn't safe - I think the only way to be able to tell will be to
> continue with my work on it and to prove that it either does or
> doesn't, at least in the general case.
>
> What we aim with the hello.so method is the same as you said -
> effectively building a static fully resolved exe, with the difference
> 

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Amaan Cheval]

On Sun, Jun 10, 2018 at 12:38 AM Joel Sherrill  wrote:
>
>
>
> On Fri, Jun 8, 2018 at 7:45 PM, Chris Johns  wrote:
>>
>> On 9/6/18 10:00 am, Joel Sherrill wrote:
>> > On Thu, Jun 7, 2018, 9:01 PM Chris Johns > > > wrote:
>> > > and what
>> > > discussions we need to have to decide between the "bundled kernel.so 
>> > approach"
>> > > (the one implemented here) vs. the "FreeBSD loader.efi+hello.exe"
>> > approach. Let
>> > > me know!
>> > >
>> >
>> > I do not think I can help too much here. I understand the 
>> > loader.efi+exe
>> > solution and it should work because all RTEMS applications we have are
>> > statically linked (I am assuming it is here). I have not looked at the 
>> > details
>> > being used with the -fPIC and .so solution so I cannot comment. I do 
>> > have some
>> > concerns the relocatable exe might expose some dark corners and issues 
>> > in the
>> > host tools we have, for example how does GDB find the base address of 
>> > the image
>> > so you can debug it? and is this just working or is it really suppose 
>> > to work
>> > this way?
>> >
>> >
>> > All I can say is that with Deos/RTEMS, we use PIC on arm, PowerPC, and x86.
>>
>> I would hope a solution like Deos did provide a seamless way to do this and I
>> would also hope they support you.
>
>
> I am not using their normal recommended tool setup for users. This is normal
> RTEMS tools building our test executables. At one point, it was our gdb with
> their qemu build. They use something like this strictly internally.
>
> These executables are statically linked EXCEPT for references to, in
> the minimum case, two .so's from their environment. I set an argument to
> ld to ensure all symbols are resolved at link time. Their boot process 
> associates
> the .so files with the partitions. It is dynamic loading but it is statically 
> configured
> and not touched after boot.
>
> I haven't had any special help from them in this area except figuring out the
> arguments and linker scripts. When I have access to a build log, I am happy
> to post the build of hello world for comparison.
>
> I have no idea how this compares to UEFI booting except to say that PIC
> hasn't introduced any tool issues in our GNU tools. libdl may have issues
> but we aren't using it yet. I can check if the tests pass or are disabled. I
> don't remember. But that may be illuminating.
>
>>
>>
>> > We
>> > have spent a lot of time debugging with gdb attached to qemu.
>> How does GDB get the relocatable load address to map to the symbol table?
>>
>> The libdl code supports the same protocol/design as NetBSD and other systems 
>> in
>> informing GDB about the address of loaded modules. There is a series of 
>> symbols
>> and tables maintained that GDB knows to examine to find the load addresses of
>> object files.
>>
>> > I haven't seen any tools issues yet.
>>
>> Yet?  Once the path is settled it will be difficult to change so all I am 
>> asking
>> is the detail be checked and understood. RTEMS does not support shared 
>> libraries
>> the same way Linux or other Unix systems do. I do not understand enough of 
>> the
>> low level and the standards all this is based on to help decide.
>>
>> An example of an issue where a relocatable kernel with an unknown load 
>> address
>> creates a problem is libdl. The testsuite uses the 2-pass approach 
>> (rtems-syms
>> --embed) which should be OK however the other approach where the symbol 
>> table is
>> not embedded and built on the host would fail. It is a small issue but it 
>> shows
>> how things can subtly break.
>
>
> I'm not relocating any RTEMS code with Deos. Our code is linked to a static 
> address
> ranges and invokes Deos methods in the shared libbary.
>
> I know this doesn't prove anything concretely about the UEFI exe  but it is 
> the closest
> example we have. PIC is likely OK. The .so magic could be problematic as it 
> looks
> like I effectively build a static exe.

The Bsymbolic flag isn't a requirement for this system to work - it
just helps eliminate the use of the GOT/PLT unnecessarily. Personally,
I don't think I have the clarity to be able to say whether this is or
isn't safe - I think the only way to be able to tell will be to
continue with my work on it and to prove that it either does or
doesn't, at least in the general case.

What we aim with the hello.so method is the same as you said -
effectively building a static fully resolved exe, with the difference
that this _is_ relocatable depending on the UEFI firmware's access and
availability to memory.

I can't speak to how libdl is affected - but the bit from the ld
manual about "platforms which support shared libraries" doesn't imply
libdl / RTEMS needing to support shared libraries to me. What we build
in this method is hello.so (which includes all of RTEMS' kernel + the
user application (hello world here)) - the platform that needs to
support shared libraries then is 

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Amaan Cheval]

On Sat, Jun 9, 2018 at 5:26 AM Chris Johns  wrote:
>
> On 9/6/18 2:39 am, Amaan Cheval wrote:
> > On Fri, Jun 8, 2018 at 7:31 AM, Chris Johns  wrote:
> >> On 08/06/2018 01:50, Amaan Cheval wrote:
> >>>
> >>> Joel, Chris, I'd appreciate guidance on what I ought to work on next
> >>
> >> I would like to see the focus on the kernel context switcher, FPU support, 
> >> and
> >> then interrupts so we have the basic drivers we need like a tick interrupt 
> >> running.
> >>
> >> This assumes the loader issues we have not resolved do not effect this 
> >> work.
> >>
> >
> > They do affect it to a certain extent - I'd be working semi-blind
> > since without tying the loose ends required to boot, I wouldn't be
> > able to test the implementations of the areas you mentioned. I'm not
> > at that point yet, but I suspect I will be within a week or so, so the
> > sooner we determine the approach required for booting, the better.
>
> OK.
>
> >>> and what
> >>> discussions we need to have to decide between the "bundled kernel.so 
> >>> approach"
> >>> (the one implemented here) vs. the "FreeBSD loader.efi+hello.exe" 
> >>> approach. Let
> >>> me know!
> >>>
> >>
> >> I do not think I can help too much here. I understand the loader.efi+exe
> >> solution and it should work because all RTEMS applications we have are
> >> statically linked (I am assuming it is here). I have not looked at the 
> >> details
> >> being used with the -fPIC and .so solution so I cannot comment. I do have 
> >> some
> >> concerns the relocatable exe might expose some dark corners and issues in 
> >> the
> >> host tools we have, for example how does GDB find the base address of the 
> >> image
> >> so you can debug it? and is this just working or is it really suppose to 
> >> work
> >> this way?
> >
> > Well, these images won't simply _run_ through GDB, no - but here's
> > some stuff that may be helpful to see:
> > https://gist.github.com/AmaanC/4e1aaa2cbdda974b93c5a3e1eac5318a
>
> Interesting and thanks. Is this with QEMU?

No, it was only the dynamic/shared library "hello.so" (that I attached
in an earlier email, in case you want to play with it yourself).

>
> > One concern of yours was the unnecessary addition of the GOT/PLT.
> > Thankfully, through options like -Bsymbolic, we can circumvent the
> > GOT/PLT for all symbols which have already been resolved (as you'll
> > see happens for "InitializeLib", "Print", "boot_card", etc. (boot_card
> > because this is from my WIP version trying to get boot_card to work
> > with this method too)).
>
> The -Bsymbolic option is an example of my concern and stepping into a dark
> corner. It could also be my lack of understanding. Yes it works but is that
> always going to be the case? For example the GNU ld manual states:
>
>  "This option is only meaningful on ELF platforms which support shared
>   libraries and position independent executables."
>
> and technically we do not support shared libraries so is this usage a normal 
> use
> case? I do not know. Also Oracle says the option is somewhat historic:
>
>  https://docs.oracle.com/cd/E19957-01/806-0641/chapter4-16/index.html

That's a good point. I see this article which seems to me like it _is_
a fair concern, especially in the case of something as complex as
RTEMS.
https://software.intel.com/en-us/articles/performance-tools-for-software-developers-bsymbolic-can-cause-dangerous-side-effects

>
> > I'm definitely concerned, but having looked at it more, I can't find
> > anything specific that would genuinely cause problems besides
> > unresolved symbols - we could have a build-time check for them,
> > though, failing the build when that's the case (-Wl,-z,defs does it).
>
> If loader.efi+exe avoids this then it makes sense to me to do so. The less we
> bend or stretch the more stable the support will be over time.
>
> > So for next steps, I guess you'll be looking into how the use of -fPIC
> > may affect us, and we can work on addressing those concerns, right?
>
> I do not know because I do not know risks.
>
> > I
> > personally preferred the static build approach too, since that way we
> > can "plug" loaders in:
> > - loader.efi for UEFI firmware
> > - multiboot header + 32 to 64 bit mode code for Multiboot
> >
>
> Agreed.
>
> > That's a slight oversimplification since (1) needs to be packaged on
> > the filesystem with the static hello.exe
>
> Yes this is a good point, then again this target is a bit different to other
> targets. Users of PCs and similar hardware are use to boot loaders and 
> managing
> them plus handling the media needed. I do not think we will ever create a
> bootable within an RTEMS build.
>
> Could I install FreeBSD, load an RTEMS kernel onto the root directory and then
> boot it with the standard FreeBSD loader.efi chain?

If we go with the static approach (loader.efi+hello.exe), my guess is
yes, that ought to work in some sense - long mode code would run, but
if we expect the loader to configure hardware in a specific way or

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Joel Sherrill]

On Fri, Jun 8, 2018 at 7:45 PM, Chris Johns  wrote:

> On 9/6/18 10:00 am, Joel Sherrill wrote:
> > On Thu, Jun 7, 2018, 9:01 PM Chris Johns  > > wrote:
> > > and what
> > > discussions we need to have to decide between the "bundled
> kernel.so approach"
> > > (the one implemented here) vs. the "FreeBSD loader.efi+hello.exe"
> > approach. Let
> > > me know!
> > >
> >
> > I do not think I can help too much here. I understand the
> loader.efi+exe
> > solution and it should work because all RTEMS applications we have
> are
> > statically linked (I am assuming it is here). I have not looked at
> the details
> > being used with the -fPIC and .so solution so I cannot comment. I do
> have some
> > concerns the relocatable exe might expose some dark corners and
> issues in the
> > host tools we have, for example how does GDB find the base address
> of the image
> > so you can debug it? and is this just working or is it really
> suppose to work
> > this way?
> >
> >
> > All I can say is that with Deos/RTEMS, we use PIC on arm, PowerPC, and
> x86.
>
> I would hope a solution like Deos did provide a seamless way to do this
> and I
> would also hope they support you.
>

I am not using their normal recommended tool setup for users. This is normal
RTEMS tools building our test executables. At one point, it was our gdb with
their qemu build. They use something like this strictly internally.

These executables are statically linked EXCEPT for references to, in
the minimum case, two .so's from their environment. I set an argument to
ld to ensure all symbols are resolved at link time. Their boot process
associates
the .so files with the partitions. It is dynamic loading but it is
statically configured
and not touched after boot.

I haven't had any special help from them in this area except figuring out
the
arguments and linker scripts. When I have access to a build log, I am happy
to post the build of hello world for comparison.

I have no idea how this compares to UEFI booting except to say that PIC
hasn't introduced any tool issues in our GNU tools. libdl may have issues
but we aren't using it yet. I can check if the tests pass or are disabled. I
don't remember. But that may be illuminating.


>
> > We
> > have spent a lot of time debugging with gdb attached to qemu.
> How does GDB get the relocatable load address to map to the symbol table?
>
> The libdl code supports the same protocol/design as NetBSD and other
> systems in
> informing GDB about the address of loaded modules. There is a series of
> symbols
> and tables maintained that GDB knows to examine to find the load addresses
> of
> object files.
>
> > I haven't seen any tools issues yet.
>
> Yet?  Once the path is settled it will be difficult to change so all I am
> asking
> is the detail be checked and understood. RTEMS does not support shared
> libraries
> the same way Linux or other Unix systems do. I do not understand enough of
> the
> low level and the standards all this is based on to help decide.
>
> An example of an issue where a relocatable kernel with an unknown load
> address
> creates a problem is libdl. The testsuite uses the 2-pass approach
> (rtems-syms
> --embed) which should be OK however the other approach where the symbol
> table is
> not embedded and built on the host would fail. It is a small issue but it
> shows
> how things can subtly break.


I'm not relocating any RTEMS code with Deos. Our code is linked to a static
address
ranges and invokes Deos methods in the shared libbary.

I know this doesn't prove anything concretely about the UEFI exe  but it is
the closest
example we have. PIC is likely OK. The .so magic could be problematic as it
looks
like I effectively build a static exe.

>
> Chris
>
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Chris Johns]

On 9/6/18 10:00 am, Joel Sherrill wrote:
> On Thu, Jun 7, 2018, 9:01 PM Chris Johns  > wrote:
> > and what
> > discussions we need to have to decide between the "bundled kernel.so 
> approach"
> > (the one implemented here) vs. the "FreeBSD loader.efi+hello.exe"
> approach. Let
> > me know!
> >
> 
> I do not think I can help too much here. I understand the loader.efi+exe
> solution and it should work because all RTEMS applications we have are
> statically linked (I am assuming it is here). I have not looked at the 
> details
> being used with the -fPIC and .so solution so I cannot comment. I do have 
> some
> concerns the relocatable exe might expose some dark corners and issues in 
> the
> host tools we have, for example how does GDB find the base address of the 
> image
> so you can debug it? and is this just working or is it really suppose to 
> work
> this way?
> 
> 
> All I can say is that with Deos/RTEMS, we use PIC on arm, PowerPC, and x86.

I would hope a solution like Deos did provide a seamless way to do this and I
would also hope they support you.

> We
> have spent a lot of time debugging with gdb attached to qemu.
How does GDB get the relocatable load address to map to the symbol table?

The libdl code supports the same protocol/design as NetBSD and other systems in
informing GDB about the address of loaded modules. There is a series of symbols
and tables maintained that GDB knows to examine to find the load addresses of
object files.

> I haven't seen any tools issues yet. 

Yet?  Once the path is settled it will be difficult to change so all I am asking
is the detail be checked and understood. RTEMS does not support shared libraries
the same way Linux or other Unix systems do. I do not understand enough of the
low level and the standards all this is based on to help decide.

An example of an issue where a relocatable kernel with an unknown load address
creates a problem is libdl. The testsuite uses the 2-pass approach (rtems-syms
--embed) which should be OK however the other approach where the symbol table is
not embedded and built on the host would fail. It is a small issue but it shows
how things can subtly break.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Joel Sherrill]

On Thu, Jun 7, 2018, 9:01 PM Chris Johns  wrote:

> On 08/06/2018 01:50, Amaan Cheval wrote:
> >
> > Joel, Chris, I'd appreciate guidance on what I ought to work on next
>
> I would like to see the focus on the kernel context switcher, FPU support,
> and
> then interrupts so we have the basic drivers we need like a tick interrupt
> running.
>

+1

>
> This assumes the loader issues we have not resolved do not effect this
> work.
>
> > and what
> > discussions we need to have to decide between the "bundled kernel.so
> approach"
> > (the one implemented here) vs. the "FreeBSD loader.efi+hello.exe"
> approach. Let
> > me know!
> >
>
> I do not think I can help too much here. I understand the loader.efi+exe
> solution and it should work because all RTEMS applications we have are
> statically linked (I am assuming it is here). I have not looked at the
> details
> being used with the -fPIC and .so solution so I cannot comment. I do have
> some
> concerns the relocatable exe might expose some dark corners and issues in
> the
> host tools we have, for example how does GDB find the base address of the
> image
> so you can debug it? and is this just working or is it really suppose to
> work
> this way?
>

All I can say is that with Deos/RTEMS, we use PIC on arm, PowerPC, and x86.
We have spent a lot of time debugging with gdb attached to qemu. I haven't
seen any tools issues yet.



> Chris
>
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Chris Johns]

On 9/6/18 2:39 am, Amaan Cheval wrote:
> On Fri, Jun 8, 2018 at 7:31 AM, Chris Johns  wrote:
>> On 08/06/2018 01:50, Amaan Cheval wrote:
>>>
>>> Joel, Chris, I'd appreciate guidance on what I ought to work on next
>>
>> I would like to see the focus on the kernel context switcher, FPU support, 
>> and
>> then interrupts so we have the basic drivers we need like a tick interrupt 
>> running.
>>
>> This assumes the loader issues we have not resolved do not effect this work.
>>
> 
> They do affect it to a certain extent - I'd be working semi-blind
> since without tying the loose ends required to boot, I wouldn't be
> able to test the implementations of the areas you mentioned. I'm not
> at that point yet, but I suspect I will be within a week or so, so the
> sooner we determine the approach required for booting, the better.

OK.

>>> and what
>>> discussions we need to have to decide between the "bundled kernel.so 
>>> approach"
>>> (the one implemented here) vs. the "FreeBSD loader.efi+hello.exe" approach. 
>>> Let
>>> me know!
>>>
>>
>> I do not think I can help too much here. I understand the loader.efi+exe
>> solution and it should work because all RTEMS applications we have are
>> statically linked (I am assuming it is here). I have not looked at the 
>> details
>> being used with the -fPIC and .so solution so I cannot comment. I do have 
>> some
>> concerns the relocatable exe might expose some dark corners and issues in the
>> host tools we have, for example how does GDB find the base address of the 
>> image
>> so you can debug it? and is this just working or is it really suppose to work
>> this way?
> 
> Well, these images won't simply _run_ through GDB, no - but here's
> some stuff that may be helpful to see:
> https://gist.github.com/AmaanC/4e1aaa2cbdda974b93c5a3e1eac5318a

Interesting and thanks. Is this with QEMU?

> One concern of yours was the unnecessary addition of the GOT/PLT.
> Thankfully, through options like -Bsymbolic, we can circumvent the
> GOT/PLT for all symbols which have already been resolved (as you'll
> see happens for "InitializeLib", "Print", "boot_card", etc. (boot_card
> because this is from my WIP version trying to get boot_card to work
> with this method too)).

The -Bsymbolic option is an example of my concern and stepping into a dark
corner. It could also be my lack of understanding. Yes it works but is that
always going to be the case? For example the GNU ld manual states:

 "This option is only meaningful on ELF platforms which support shared
  libraries and position independent executables."

and technically we do not support shared libraries so is this usage a normal use
case? I do not know. Also Oracle says the option is somewhat historic:

 https://docs.oracle.com/cd/E19957-01/806-0641/chapter4-16/index.html

> I'm definitely concerned, but having looked at it more, I can't find
> anything specific that would genuinely cause problems besides
> unresolved symbols - we could have a build-time check for them,
> though, failing the build when that's the case (-Wl,-z,defs does it).

If loader.efi+exe avoids this then it makes sense to me to do so. The less we
bend or stretch the more stable the support will be over time.

> So for next steps, I guess you'll be looking into how the use of -fPIC
> may affect us, and we can work on addressing those concerns, right? 

I do not know because I do not know risks.

> I
> personally preferred the static build approach too, since that way we
> can "plug" loaders in:
> - loader.efi for UEFI firmware
> - multiboot header + 32 to 64 bit mode code for Multiboot
>

Agreed.

> That's a slight oversimplification since (1) needs to be packaged on
> the filesystem with the static hello.exe

Yes this is a good point, then again this target is a bit different to other
targets. Users of PCs and similar hardware are use to boot loaders and managing
them plus handling the media needed. I do not think we will ever create a
bootable within an RTEMS build.

Could I install FreeBSD, load an RTEMS kernel onto the root directory and then
boot it with the standard FreeBSD loader.efi chain?

> and (2) needs to be linked in
> with it, but I think the build system retains more of its separations
> from the boot method this way (though I can't be sure given my
> relatively naive view of RTEMS + autotools).

I can help here if you need it.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Amaan Cheval]

On Fri, Jun 8, 2018 at 7:31 AM, Chris Johns  wrote:
> On 08/06/2018 01:50, Amaan Cheval wrote:
>>
>> Joel, Chris, I'd appreciate guidance on what I ought to work on next
>
> I would like to see the focus on the kernel context switcher, FPU support, and
> then interrupts so we have the basic drivers we need like a tick interrupt 
> running.
>
> This assumes the loader issues we have not resolved do not effect this work.
>

They do affect it to a certain extent - I'd be working semi-blind
since without tying the loose ends required to boot, I wouldn't be
able to test the implementations of the areas you mentioned. I'm not
at that point yet, but I suspect I will be within a week or so, so the
sooner we determine the approach required for booting, the better.

>> and what
>> discussions we need to have to decide between the "bundled kernel.so 
>> approach"
>> (the one implemented here) vs. the "FreeBSD loader.efi+hello.exe" approach. 
>> Let
>> me know!
>>
>
> I do not think I can help too much here. I understand the loader.efi+exe
> solution and it should work because all RTEMS applications we have are
> statically linked (I am assuming it is here). I have not looked at the details
> being used with the -fPIC and .so solution so I cannot comment. I do have some
> concerns the relocatable exe might expose some dark corners and issues in the
> host tools we have, for example how does GDB find the base address of the 
> image
> so you can debug it? and is this just working or is it really suppose to work
> this way?

Well, these images won't simply _run_ through GDB, no - but here's
some stuff that may be helpful to see:
https://gist.github.com/AmaanC/4e1aaa2cbdda974b93c5a3e1eac5318a

One concern of yours was the unnecessary addition of the GOT/PLT.
Thankfully, through options like -Bsymbolic, we can circumvent the
GOT/PLT for all symbols which have already been resolved (as you'll
see happens for "InitializeLib", "Print", "boot_card", etc. (boot_card
because this is from my WIP version trying to get boot_card to work
with this method too)).

I'm definitely concerned, but having looked at it more, I can't find
anything specific that would genuinely cause problems besides
unresolved symbols - we could have a build-time check for them,
though, failing the build when that's the case (-Wl,-z,defs does it).

So for next steps, I guess you'll be looking into how the use of -fPIC
may affect us, and we can work on addressing those concerns, right? I
personally preferred the static build approach too, since that way we
can "plug" loaders in:
- loader.efi for UEFI firmware
- multiboot header + 32 to 64 bit mode code for Multiboot

That's a slight oversimplification since (1) needs to be packaged on
the filesystem with the static hello.exe and (2) needs to be linked in
with it, but I think the build system retains more of its separations
from the boot method this way (though I can't be sure given my
relatively naive view of RTEMS + autotools).

>
> Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Chris Johns]

On 08/06/2018 01:50, Amaan Cheval wrote:
> 
> Joel, Chris, I'd appreciate guidance on what I ought to work on next

I would like to see the focus on the kernel context switcher, FPU support, and
then interrupts so we have the basic drivers we need like a tick interrupt 
running.

This assumes the loader issues we have not resolved do not effect this work.

> and what
> discussions we need to have to decide between the "bundled kernel.so approach"
> (the one implemented here) vs. the "FreeBSD loader.efi+hello.exe" approach. 
> Let
> me know!
> 

I do not think I can help too much here. I understand the loader.efi+exe
solution and it should work because all RTEMS applications we have are
statically linked (I am assuming it is here). I have not looked at the details
being used with the -fPIC and .so solution so I cannot comment. I do have some
concerns the relocatable exe might expose some dark corners and issues in the
host tools we have, for example how does GDB find the base address of the image
so you can debug it? and is this just working or is it really suppose to work
this way?

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Amaan Cheval]

Chris, I've blogged a summary of the 2 approaches we can take in
integrating gnu-efi in case you missed the discussions earlier, btw.
The FreeBSD approach isn't as fleshed out, but I'll let you all know
what I find there.

https://blog.whatthedude.com/post/gnu-efi-kernel-so/

On Wed, Jun 6, 2018 at 2:30 PM, Amaan Cheval  wrote:
> I don't know yet, but I'll look into it. I'll pause the "hello.efi" approach
> work until we settle on a direction, yes? For now, primarily improving the
> stub, looking into the FreeBSD ld-elf.so, etc. Sound good?
>
> On Wed, Jun 6, 2018, 1:01 PM Chris Johns  wrote:
>>
>> On 6/6/18 5:22 pm, Amaan Cheval wrote:
>> > On Wed, Jun 6, 2018 at 12:45 PM, Chris Johns  wrote:
>> >> On 6/6/18 5:06 pm, Amaan Cheval wrote:
>> >>> On Wed, Jun 6, 2018 at 6:55 AM, Chris Johns  wrote:
>>  On 4/6/18 7:49 pm, Amaan Cheval wrote:
>> > Please let me know if that approach doesn't make sense - given that
>> > there is no dynamic loader in the RTEMS kernel as far as I know,
>> 
>>  There is a dynamic loader in RTEMS called libdl. It is not based
>>  around the ELF
>>  standard loader used for shared libraries and will not be. GOT and
>>  PLT do not
>>  add value to RTEMS because we do not have an active virtual address
>>  space with
>>  paging.
>> 
>>  The libdl code is currently supported with the i386 static builds. I
>>  would hope
>>  this would continue to be supported without major refactoring of that
>>  code.
>>  There are tests in the testsuite under libtests.
>> >>>
>> >>> Hm, so libdl can load static ELFs and handle those relocations itself?
>> >>> In that case we could go the "FreeBSD" way more easily as I outlined
>> >>> earlier; a loader UEFI application image (loader.efi) + the
>> >>> application image to be found on the filesystem and loaded through
>> >>> libdl, which we somehow call through loader.efi.
>> >>>
>> >>> loader.efi -> libdl -> hello.exe (static executable ELF now)
>> >>>
>> >>
>> >> libdl is not designed to do this and do not think it could be made too.
>> >> It needs
>> >> a full RTEMS kernel located to run.
>> >
>> > Ah, I see. In that case porting FreeBSD's ELF loader is the only
>> > viable option in this direction, I believe, since the ELF to be loaded
>> > would be the RTEMS kernel+the user app.
>> >
>>
>> Do we need to port it or can we use a released version from an installed
>> FreeBSD?
>>
>> >>
>> 
>> > what
>> > we really want _is_ a static file, but for it to be a relocatable
>> > PE,
>> > we need to convince GCC to spit out a relocatable but fully resolved
>> > shared library.
>> 
>>  It is not clear to me yet making the kernel relocatable is free of
>>  other
>>  possible issues. I will need to find time to review all the low level
>>  parts here
>>  and my time is currently limited.
>> 
>>  Does this UEFI work effect the score context switcher, interrupts,
>>  etc? If is
>>  does we will need to resolve what happens and if not should we
>>  consider leaving
>>  it if we can?
>> >>>
>> >>> It affects it in the sense that it's all compiled with fPIC, yes. It
>> >>> needs to be if we're bundling it all together (creating hello.efi,
>> >>> which includes the UEFI boot code, all of RTEMS, and the user app).
>> >>>
>>  For example can iPXE chain load a multiboot image?
>> >>>
>> >>> Yes, we could just use Multiboot too. One thing to note, though -
>> >>> Multiboot will drop us in 32-bit protected mode, whereas 64-bit UEFI
>> >>> firmware will load is into 64-bit protected mode.
>> >>
>> >> Ah OK this is a good point and boards like a Minnow have a 32bit or
>> >> 64bit BIOS
>> >> or what ever it is called on those boards so this may not work.
>> >>
>> >>> Supporting Multiboot
>> >>> too will involve adding some code to move to 64-bit mode before
>> >>> actually calling into the generalized 64-bit mode code.
>> >>
>> >> Can it be used as a step towards another solution?
>> >
>> > Not sure what you mean - like if it would be useful anywhere outside
>> > of the Multiboot work? If so, no, likely not, unless we also want
>> > legacy BIOS support eventually, in which case it could be part of the
>> > real mode->protected mode->long mode transition chain, but that's
>> > unlikely :P
>> >
>>
>> I was just wondering if a temporary short cut can be taken so we do not
>> spend
>> all our time on booting an image.
>>
>> Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Amaan Cheval]

I don't know yet, but I'll look into it. I'll pause the "hello.efi"
approach work until we settle on a direction, yes? For now, primarily
improving the stub, looking into the FreeBSD ld-elf.so, etc. Sound good?

On Wed, Jun 6, 2018, 1:01 PM Chris Johns  wrote:

> On 6/6/18 5:22 pm, Amaan Cheval wrote:
> > On Wed, Jun 6, 2018 at 12:45 PM, Chris Johns  wrote:
> >> On 6/6/18 5:06 pm, Amaan Cheval wrote:
> >>> On Wed, Jun 6, 2018 at 6:55 AM, Chris Johns  wrote:
>  On 4/6/18 7:49 pm, Amaan Cheval wrote:
> > Please let me know if that approach doesn't make sense - given that
> > there is no dynamic loader in the RTEMS kernel as far as I know,
> 
>  There is a dynamic loader in RTEMS called libdl. It is not based
> around the ELF
>  standard loader used for shared libraries and will not be. GOT and
> PLT do not
>  add value to RTEMS because we do not have an active virtual address
> space with
>  paging.
> 
>  The libdl code is currently supported with the i386 static builds. I
> would hope
>  this would continue to be supported without major refactoring of that
> code.
>  There are tests in the testsuite under libtests.
> >>>
> >>> Hm, so libdl can load static ELFs and handle those relocations itself?
> >>> In that case we could go the "FreeBSD" way more easily as I outlined
> >>> earlier; a loader UEFI application image (loader.efi) + the
> >>> application image to be found on the filesystem and loaded through
> >>> libdl, which we somehow call through loader.efi.
> >>>
> >>> loader.efi -> libdl -> hello.exe (static executable ELF now)
> >>>
> >>
> >> libdl is not designed to do this and do not think it could be made too.
> It needs
> >> a full RTEMS kernel located to run.
> >
> > Ah, I see. In that case porting FreeBSD's ELF loader is the only
> > viable option in this direction, I believe, since the ELF to be loaded
> > would be the RTEMS kernel+the user app.
> >
>
> Do we need to port it or can we use a released version from an installed
> FreeBSD?
>
> >>
> 
> > what
> > we really want _is_ a static file, but for it to be a relocatable PE,
> > we need to convince GCC to spit out a relocatable but fully resolved
> > shared library.
> 
>  It is not clear to me yet making the kernel relocatable is free of
> other
>  possible issues. I will need to find time to review all the low level
> parts here
>  and my time is currently limited.
> 
>  Does this UEFI work effect the score context switcher, interrupts,
> etc? If is
>  does we will need to resolve what happens and if not should we
> consider leaving
>  it if we can?
> >>>
> >>> It affects it in the sense that it's all compiled with fPIC, yes. It
> >>> needs to be if we're bundling it all together (creating hello.efi,
> >>> which includes the UEFI boot code, all of RTEMS, and the user app).
> >>>
>  For example can iPXE chain load a multiboot image?
> >>>
> >>> Yes, we could just use Multiboot too. One thing to note, though -
> >>> Multiboot will drop us in 32-bit protected mode, whereas 64-bit UEFI
> >>> firmware will load is into 64-bit protected mode.
> >>
> >> Ah OK this is a good point and boards like a Minnow have a 32bit or
> 64bit BIOS
> >> or what ever it is called on those boards so this may not work.
> >>
> >>> Supporting Multiboot
> >>> too will involve adding some code to move to 64-bit mode before
> >>> actually calling into the generalized 64-bit mode code.
> >>
> >> Can it be used as a step towards another solution?
> >
> > Not sure what you mean - like if it would be useful anywhere outside
> > of the Multiboot work? If so, no, likely not, unless we also want
> > legacy BIOS support eventually, in which case it could be part of the
> > real mode->protected mode->long mode transition chain, but that's
> > unlikely :P
> >
>
> I was just wondering if a temporary short cut can be taken so we do not
> spend
> all our time on booting an image.
>
> Chris
>
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Chris Johns]

On 6/6/18 5:22 pm, Amaan Cheval wrote:
> On Wed, Jun 6, 2018 at 12:45 PM, Chris Johns  wrote:
>> On 6/6/18 5:06 pm, Amaan Cheval wrote:
>>> On Wed, Jun 6, 2018 at 6:55 AM, Chris Johns  wrote:
 On 4/6/18 7:49 pm, Amaan Cheval wrote:
> Please let me know if that approach doesn't make sense - given that
> there is no dynamic loader in the RTEMS kernel as far as I know,

 There is a dynamic loader in RTEMS called libdl. It is not based around 
 the ELF
 standard loader used for shared libraries and will not be. GOT and PLT do 
 not
 add value to RTEMS because we do not have an active virtual address space 
 with
 paging.

 The libdl code is currently supported with the i386 static builds. I would 
 hope
 this would continue to be supported without major refactoring of that code.
 There are tests in the testsuite under libtests.
>>>
>>> Hm, so libdl can load static ELFs and handle those relocations itself?
>>> In that case we could go the "FreeBSD" way more easily as I outlined
>>> earlier; a loader UEFI application image (loader.efi) + the
>>> application image to be found on the filesystem and loaded through
>>> libdl, which we somehow call through loader.efi.
>>>
>>> loader.efi -> libdl -> hello.exe (static executable ELF now)
>>>
>>
>> libdl is not designed to do this and do not think it could be made too. It 
>> needs
>> a full RTEMS kernel located to run.
> 
> Ah, I see. In that case porting FreeBSD's ELF loader is the only
> viable option in this direction, I believe, since the ELF to be loaded
> would be the RTEMS kernel+the user app.
> 

Do we need to port it or can we use a released version from an installed 
FreeBSD?

>>

> what
> we really want _is_ a static file, but for it to be a relocatable PE,
> we need to convince GCC to spit out a relocatable but fully resolved
> shared library.

 It is not clear to me yet making the kernel relocatable is free of other
 possible issues. I will need to find time to review all the low level 
 parts here
 and my time is currently limited.

 Does this UEFI work effect the score context switcher, interrupts, etc? If 
 is
 does we will need to resolve what happens and if not should we consider 
 leaving
 it if we can?
>>>
>>> It affects it in the sense that it's all compiled with fPIC, yes. It
>>> needs to be if we're bundling it all together (creating hello.efi,
>>> which includes the UEFI boot code, all of RTEMS, and the user app).
>>>
 For example can iPXE chain load a multiboot image?
>>>
>>> Yes, we could just use Multiboot too. One thing to note, though -
>>> Multiboot will drop us in 32-bit protected mode, whereas 64-bit UEFI
>>> firmware will load is into 64-bit protected mode.
>>
>> Ah OK this is a good point and boards like a Minnow have a 32bit or 64bit 
>> BIOS
>> or what ever it is called on those boards so this may not work.
>>
>>> Supporting Multiboot
>>> too will involve adding some code to move to 64-bit mode before
>>> actually calling into the generalized 64-bit mode code.
>>
>> Can it be used as a step towards another solution?
> 
> Not sure what you mean - like if it would be useful anywhere outside
> of the Multiboot work? If so, no, likely not, unless we also want
> legacy BIOS support eventually, in which case it could be part of the
> real mode->protected mode->long mode transition chain, but that's
> unlikely :P
> 

I was just wondering if a temporary short cut can be taken so we do not spend
all our time on booting an image.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Amaan Cheval]

On Wed, Jun 6, 2018 at 12:45 PM, Chris Johns  wrote:
> On 6/6/18 5:06 pm, Amaan Cheval wrote:
>> On Wed, Jun 6, 2018 at 6:55 AM, Chris Johns  wrote:
>>> On 4/6/18 7:49 pm, Amaan Cheval wrote:
 Please let me know if that approach doesn't make sense - given that
 there is no dynamic loader in the RTEMS kernel as far as I know,
>>>
>>> There is a dynamic loader in RTEMS called libdl. It is not based around the 
>>> ELF
>>> standard loader used for shared libraries and will not be. GOT and PLT do 
>>> not
>>> add value to RTEMS because we do not have an active virtual address space 
>>> with
>>> paging.
>>>
>>> The libdl code is currently supported with the i386 static builds. I would 
>>> hope
>>> this would continue to be supported without major refactoring of that code.
>>> There are tests in the testsuite under libtests.
>>
>> Hm, so libdl can load static ELFs and handle those relocations itself?
>> In that case we could go the "FreeBSD" way more easily as I outlined
>> earlier; a loader UEFI application image (loader.efi) + the
>> application image to be found on the filesystem and loaded through
>> libdl, which we somehow call through loader.efi.
>>
>> loader.efi -> libdl -> hello.exe (static executable ELF now)
>>
>
> libdl is not designed to do this and do not think it could be made too. It 
> needs
> a full RTEMS kernel located to run.

Ah, I see. In that case porting FreeBSD's ELF loader is the only
viable option in this direction, I believe, since the ELF to be loaded
would be the RTEMS kernel+the user app.

>
>>>
 what
 we really want _is_ a static file, but for it to be a relocatable PE,
 we need to convince GCC to spit out a relocatable but fully resolved
 shared library.
>>>
>>> It is not clear to me yet making the kernel relocatable is free of other
>>> possible issues. I will need to find time to review all the low level parts 
>>> here
>>> and my time is currently limited.
>>>
>>> Does this UEFI work effect the score context switcher, interrupts, etc? If 
>>> is
>>> does we will need to resolve what happens and if not should we consider 
>>> leaving
>>> it if we can?
>>
>> It affects it in the sense that it's all compiled with fPIC, yes. It
>> needs to be if we're bundling it all together (creating hello.efi,
>> which includes the UEFI boot code, all of RTEMS, and the user app).
>>
>>> For example can iPXE chain load a multiboot image?
>>
>> Yes, we could just use Multiboot too. One thing to note, though -
>> Multiboot will drop us in 32-bit protected mode, whereas 64-bit UEFI
>> firmware will load is into 64-bit protected mode.
>
> Ah OK this is a good point and boards like a Minnow have a 32bit or 64bit BIOS
> or what ever it is called on those boards so this may not work.
>
>> Supporting Multiboot
>> too will involve adding some code to move to 64-bit mode before
>> actually calling into the generalized 64-bit mode code.
>
> Can it be used as a step towards another solution?

Not sure what you mean - like if it would be useful anywhere outside
of the Multiboot work? If so, no, likely not, unless we also want
legacy BIOS support eventually, in which case it could be part of the
real mode->protected mode->long mode transition chain, but that's
unlikely :P

>
>>
>>>
>>> Sorry to have disappeared for a period at a critical time in this 
>>> discussion, it
>>> was beyond my control.
>>
>> No worries! Hope everything's okay!
>>
>
> Thanks and yes.
>
>> I'll look into libdl further in the meantime and continue to polish up
>> the stub to reflect more of the x64 features, while we figure out our
>> boot direction.
>
> Great. I hope to be back full time next week and I can have a look.
>
> Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Chris Johns]

On 6/6/18 5:06 pm, Amaan Cheval wrote:
> On Wed, Jun 6, 2018 at 6:55 AM, Chris Johns  wrote:
>> On 4/6/18 7:49 pm, Amaan Cheval wrote:
>>> Please let me know if that approach doesn't make sense - given that
>>> there is no dynamic loader in the RTEMS kernel as far as I know,
>>
>> There is a dynamic loader in RTEMS called libdl. It is not based around the 
>> ELF
>> standard loader used for shared libraries and will not be. GOT and PLT do not
>> add value to RTEMS because we do not have an active virtual address space 
>> with
>> paging.
>>
>> The libdl code is currently supported with the i386 static builds. I would 
>> hope
>> this would continue to be supported without major refactoring of that code.
>> There are tests in the testsuite under libtests.
> 
> Hm, so libdl can load static ELFs and handle those relocations itself?
> In that case we could go the "FreeBSD" way more easily as I outlined
> earlier; a loader UEFI application image (loader.efi) + the
> application image to be found on the filesystem and loaded through
> libdl, which we somehow call through loader.efi.
> 
> loader.efi -> libdl -> hello.exe (static executable ELF now)
> 

libdl is not designed to do this and do not think it could be made too. It needs
a full RTEMS kernel located to run.

>>
>>> what
>>> we really want _is_ a static file, but for it to be a relocatable PE,
>>> we need to convince GCC to spit out a relocatable but fully resolved
>>> shared library.
>>
>> It is not clear to me yet making the kernel relocatable is free of other
>> possible issues. I will need to find time to review all the low level parts 
>> here
>> and my time is currently limited.
>>
>> Does this UEFI work effect the score context switcher, interrupts, etc? If is
>> does we will need to resolve what happens and if not should we consider 
>> leaving
>> it if we can?
> 
> It affects it in the sense that it's all compiled with fPIC, yes. It
> needs to be if we're bundling it all together (creating hello.efi,
> which includes the UEFI boot code, all of RTEMS, and the user app).
> 
>> For example can iPXE chain load a multiboot image?
> 
> Yes, we could just use Multiboot too. One thing to note, though -
> Multiboot will drop us in 32-bit protected mode, whereas 64-bit UEFI
> firmware will load is into 64-bit protected mode.

Ah OK this is a good point and boards like a Minnow have a 32bit or 64bit BIOS
or what ever it is called on those boards so this may not work.

> Supporting Multiboot
> too will involve adding some code to move to 64-bit mode before
> actually calling into the generalized 64-bit mode code.

Can it be used as a step towards another solution?

> 
>>
>> Sorry to have disappeared for a period at a critical time in this 
>> discussion, it
>> was beyond my control.
> 
> No worries! Hope everything's okay!
> 

Thanks and yes.

> I'll look into libdl further in the meantime and continue to polish up
> the stub to reflect more of the x64 features, while we figure out our
> boot direction.

Great. I hope to be back full time next week and I can have a look.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Amaan Cheval]

On Wed, Jun 6, 2018 at 6:55 AM, Chris Johns  wrote:
> On 4/6/18 7:49 pm, Amaan Cheval wrote:
>> Please let me know if that approach doesn't make sense - given that
>> there is no dynamic loader in the RTEMS kernel as far as I know,
>
> There is a dynamic loader in RTEMS called libdl. It is not based around the 
> ELF
> standard loader used for shared libraries and will not be. GOT and PLT do not
> add value to RTEMS because we do not have an active virtual address space with
> paging.
>
> The libdl code is currently supported with the i386 static builds. I would 
> hope
> this would continue to be supported without major refactoring of that code.
> There are tests in the testsuite under libtests.

Hm, so libdl can load static ELFs and handle those relocations itself?
In that case we could go the "FreeBSD" way more easily as I outlined
earlier; a loader UEFI application image (loader.efi) + the
application image to be found on the filesystem and loaded through
libdl, which we somehow call through loader.efi.

loader.efi -> libdl -> hello.exe (static executable ELF now)

>
>> what
>> we really want _is_ a static file, but for it to be a relocatable PE,
>> we need to convince GCC to spit out a relocatable but fully resolved
>> shared library.
>
> It is not clear to me yet making the kernel relocatable is free of other
> possible issues. I will need to find time to review all the low level parts 
> here
> and my time is currently limited.
>
> Does this UEFI work effect the score context switcher, interrupts, etc? If is
> does we will need to resolve what happens and if not should we consider 
> leaving
> it if we can?

It affects it in the sense that it's all compiled with fPIC, yes. It
needs to be if we're bundling it all together (creating hello.efi,
which includes the UEFI boot code, all of RTEMS, and the user app).

> For example can iPXE chain load a multiboot image?

Yes, we could just use Multiboot too. One thing to note, though -
Multiboot will drop us in 32-bit protected mode, whereas 64-bit UEFI
firmware will load is into 64-bit protected mode. Supporting Multiboot
too will involve adding some code to move to 64-bit mode before
actually calling into the generalized 64-bit mode code.

>
> Sorry to have disappeared for a period at a critical time in this discussion, 
> it
> was beyond my control.

No worries! Hope everything's okay!

I'll look into libdl further in the meantime and continue to polish up
the stub to reflect more of the x64 features, while we figure out our
boot direction.

>
> Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Chris Johns]

On 4/6/18 7:49 pm, Amaan Cheval wrote:
> Please let me know if that approach doesn't make sense - given that
> there is no dynamic loader in the RTEMS kernel as far as I know,

There is a dynamic loader in RTEMS called libdl. It is not based around the ELF
standard loader used for shared libraries and will not be. GOT and PLT do not
add value to RTEMS because we do not have an active virtual address space with
paging.

The libdl code is currently supported with the i386 static builds. I would hope
this would continue to be supported without major refactoring of that code.
There are tests in the testsuite under libtests.

> what
> we really want _is_ a static file, but for it to be a relocatable PE,
> we need to convince GCC to spit out a relocatable but fully resolved
> shared library.

It is not clear to me yet making the kernel relocatable is free of other
possible issues. I will need to find time to review all the low level parts here
and my time is currently limited.

Does this UEFI work effect the score context switcher, interrupts, etc? If is
does we will need to resolve what happens and if not should we consider leaving
it if we can? For example can iPXE chain load a multiboot image?

Sorry to have disappeared for a period at a critical time in this discussion, it
was beyond my control.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Joel Sherrill]

On Tue, Jun 5, 2018 at 4:27 PM, Amaan Cheval  wrote:

> Hi! Thanks for the guidance, both of you! I think we're quite close
> now to integrating gnu-efi in and finally having the stub port boot as
> a UEFI application image. All the test exe's generated on my local
> tree are now dynamic libraries, so both Newlib and crtbeginS/crtendS
> issues have been resolved (see point 2 below for the relevant WIP
> patches).
>
> Here's what I've done:
>
> - Updated amd64.cfg as Joel instructed earlier - here I put "-shared"
> in LDFLAGS instead of CPU_CFLAGS because of configure trying to use
> CPU_CFLAGS in general, running into unresolved symbols coming from
> GCC's stub crt0.c, and falling down. Putting it in LDFLAGS lets me
> work around this, but I'd appreciate a thumbs-up on this being okay:
>
> https://github.com/AmaanC/rtems-gsoc18/commit/
> 547ef85a7f176046b2cb06a34b1e312c4986e97f#diff-
> c64f46c71f828120e08bc4c8667f0525R18
>
>
This looks like the other BSPs.


> - Updated GCC as follows, building on top of Joel's WIP patch to
> minimize bsp_specs:
>
> https://github.com/AmaanC/gcc/pull/1


This doesn't have much code to review either. I think it is OK.


>
>
> I made it a PR on my fork so viewing it commit-wise or as a whole diff
> is easier for you to review. I'll look into how to make and use
> multilib variants next.
>
> Let me know what you think (about the LDFLAGS workaround, and the
> approach for the GCC patch so far).
>

The amd64.cfg file looks like I would expect. Not really a hack.

>
> On Mon, Jun 4, 2018 at 6:42 PM, Joel Sherrill  wrote:
> >
> >
> > On Mon, Jun 4, 2018 at 5:44 AM, Sebastian Huber
> >  wrote:
> >>
> >>
> >>
> >> - Am 4. Jun 2018 um 12:20 schrieb Amaan Cheval
> amaan.che...@gmail.com:
> >>
> >> > That's a good idea. That way based on the multilib variant, Newlib
> would
> >> > be
> >> > compiled using fPIC, yes?
> >>
> >> Yes.
> >
> >
> > This would be desirable for the i386 as well. Having the RTEMS libraries
> as
> > dynamic libraries would be more natural under Deos.
> >
> > Just a statement. Not a requirement on the GSoC project.
> >>
> >>
> >> >
> >> > Then I could simply figure out how to solve the crtbegin and crtend
> >> > dilemma
> >> > (which I believe should be easier), and use those to have a
> >> > dynamic/shared
> >> > RTEMS kernel + user application.
> >>
> >> These files will be compiled with -fPIC as well.
> >
> >
> >
> >>
> >>
> >> >
> >> > If that sounds right, I'll look into that first. Not familiar with the
> >> > GCC
> >> > source yet, but it should be doable.
> >>
> >> Sorry, I have no idea how the x86_64 configuration of GCC works for
> RTEMS.
> >>
> >> >
> >> >
> >> > On Mon, Jun 4, 2018, 3:43 PM Sebastian Huber <
> >> > sebastian.hu...@embedded-brains.de> wrote:
> >> >
> >> >> Hello Amaan,
> >> >>
> >> >> can't you add a new multilib variant which includes -fPIC to the GCC
> >> >> configuration for RTEMS?
> >> ___
> >> devel mailing list
> >> devel@rtems.org
> >> http://lists.rtems.org/mailman/listinfo/devel
> >
> >
>
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Amaan Cheval]

Hi! Thanks for the guidance, both of you! I think we're quite close
now to integrating gnu-efi in and finally having the stub port boot as
a UEFI application image. All the test exe's generated on my local
tree are now dynamic libraries, so both Newlib and crtbeginS/crtendS
issues have been resolved (see point 2 below for the relevant WIP
patches).

Here's what I've done:

- Updated amd64.cfg as Joel instructed earlier - here I put "-shared"
in LDFLAGS instead of CPU_CFLAGS because of configure trying to use
CPU_CFLAGS in general, running into unresolved symbols coming from
GCC's stub crt0.c, and falling down. Putting it in LDFLAGS lets me
work around this, but I'd appreciate a thumbs-up on this being okay:

https://github.com/AmaanC/rtems-gsoc18/commit/547ef85a7f176046b2cb06a34b1e312c4986e97f#diff-c64f46c71f828120e08bc4c8667f0525R18

- Updated GCC as follows, building on top of Joel's WIP patch to
minimize bsp_specs:

https://github.com/AmaanC/gcc/pull/1

I made it a PR on my fork so viewing it commit-wise or as a whole diff
is easier for you to review. I'll look into how to make and use
multilib variants next.

Let me know what you think (about the LDFLAGS workaround, and the
approach for the GCC patch so far).

On Mon, Jun 4, 2018 at 6:42 PM, Joel Sherrill  wrote:
>
>
> On Mon, Jun 4, 2018 at 5:44 AM, Sebastian Huber
>  wrote:
>>
>>
>>
>> - Am 4. Jun 2018 um 12:20 schrieb Amaan Cheval amaan.che...@gmail.com:
>>
>> > That's a good idea. That way based on the multilib variant, Newlib would
>> > be
>> > compiled using fPIC, yes?
>>
>> Yes.
>
>
> This would be desirable for the i386 as well. Having the RTEMS libraries as
> dynamic libraries would be more natural under Deos.
>
> Just a statement. Not a requirement on the GSoC project.
>>
>>
>> >
>> > Then I could simply figure out how to solve the crtbegin and crtend
>> > dilemma
>> > (which I believe should be easier), and use those to have a
>> > dynamic/shared
>> > RTEMS kernel + user application.
>>
>> These files will be compiled with -fPIC as well.
>
>
>
>>
>>
>> >
>> > If that sounds right, I'll look into that first. Not familiar with the
>> > GCC
>> > source yet, but it should be doable.
>>
>> Sorry, I have no idea how the x86_64 configuration of GCC works for RTEMS.
>>
>> >
>> >
>> > On Mon, Jun 4, 2018, 3:43 PM Sebastian Huber <
>> > sebastian.hu...@embedded-brains.de> wrote:
>> >
>> >> Hello Amaan,
>> >>
>> >> can't you add a new multilib variant which includes -fPIC to the GCC
>> >> configuration for RTEMS?
>> ___
>> devel mailing list
>> devel@rtems.org
>> http://lists.rtems.org/mailman/listinfo/devel
>
>
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Joel Sherrill]

On Mon, Jun 4, 2018 at 5:44 AM, Sebastian Huber <
sebastian.hu...@embedded-brains.de> wrote:

>
>
> - Am 4. Jun 2018 um 12:20 schrieb Amaan Cheval amaan.che...@gmail.com:
>
> > That's a good idea. That way based on the multilib variant, Newlib would
> be
> > compiled using fPIC, yes?
>
> Yes.
>

This would be desirable for the i386 as well. Having the RTEMS libraries as
dynamic libraries would be more natural under Deos.

Just a statement. Not a requirement on the GSoC project.

>
> >
> > Then I could simply figure out how to solve the crtbegin and crtend
> dilemma
> > (which I believe should be easier), and use those to have a
> dynamic/shared
> > RTEMS kernel + user application.
>
> These files will be compiled with -fPIC as well.
>



>
> >
> > If that sounds right, I'll look into that first. Not familiar with the
> GCC
> > source yet, but it should be doable.
>
> Sorry, I have no idea how the x86_64 configuration of GCC works for RTEMS.
>
> >
> >
> > On Mon, Jun 4, 2018, 3:43 PM Sebastian Huber <
> > sebastian.hu...@embedded-brains.de> wrote:
> >
> >> Hello Amaan,
> >>
> >> can't you add a new multilib variant which includes -fPIC to the GCC
> >> configuration for RTEMS?
> ___
> devel mailing list
> devel@rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
>
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Sebastian Huber]

- Am 4. Jun 2018 um 12:20 schrieb Amaan Cheval amaan.che...@gmail.com:

> That's a good idea. That way based on the multilib variant, Newlib would be
> compiled using fPIC, yes?

Yes.

> 
> Then I could simply figure out how to solve the crtbegin and crtend dilemma
> (which I believe should be easier), and use those to have a dynamic/shared
> RTEMS kernel + user application.

These files will be compiled with -fPIC as well.

> 
> If that sounds right, I'll look into that first. Not familiar with the GCC
> source yet, but it should be doable.

Sorry, I have no idea how the x86_64 configuration of GCC works for RTEMS.

> 
> 
> On Mon, Jun 4, 2018, 3:43 PM Sebastian Huber <
> sebastian.hu...@embedded-brains.de> wrote:
> 
>> Hello Amaan,
>>
>> can't you add a new multilib variant which includes -fPIC to the GCC
>> configuration for RTEMS?
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Amaan Cheval]

That's a good idea. That way based on the multilib variant, Newlib would be
compiled using fPIC, yes?

Then I could simply figure out how to solve the crtbegin and crtend dilemma
(which I believe should be easier), and use those to have a dynamic/shared
RTEMS kernel + user application.

If that sounds right, I'll look into that first. Not familiar with the GCC
source yet, but it should be doable.


On Mon, Jun 4, 2018, 3:43 PM Sebastian Huber <
sebastian.hu...@embedded-brains.de> wrote:

> Hello Amaan,
>
> can't you add a new multilib variant which includes -fPIC to the GCC
> configuration for RTEMS?
>
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Sebastian Huber]

Hello Amaan,

can't you add a new multilib variant which includes -fPIC to the GCC 
configuration for RTEMS?
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

[GSoC - x86_64 BSP] Using fPIC to compile RTEMS as a shared library

[Amaan Cheval]

Hi!

I figured I'd quickly confirm the direction I'm taking towards
compiling RTEMS as a dynamic/shared library.

Problems I've run into in setting up amd64.cfg to compile all of RTEMS
as a shared library:

- In the x86_64 tools, gcc's "-shared" flag has a different effect
than the "-Wl,-shared" flag used to pass the flag to the linker - the
former silently seems to compile a static library. The latter leads us
to our next issue

- Newlib seems to be compiled as a static libc.a. This leads to errors
such as the following:

https://gist.github.com/AmaanC/475bc0298697d22b944577ac80ec2736#file-rtems-make-fpic-log-L178

I believe this _should_ be solvable by compiling newlib as a shared
library, and _then_ linking the shared libc with RTEMS together. See
[1][2][3] for more.

Please let me know if that approach doesn't make sense - given that
there is no dynamic loader in the RTEMS kernel as far as I know, what
we really want _is_ a static file, but for it to be a relocatable PE,
we need to convince GCC to spit out a relocatable but fully resolved
shared library.

- Similarly, the gcc-compiled crtbegin and crtend also include static
relocations which are invalid when compiling with fPIC.

I think we can use crtbeginS.o and crtendS.o[4] in place of those -
that way, we might still be able to have GCC handle their inclusion,
without needing our bsp_specs. I'll look into this after figuring
newlib out.

---

I'd just like some confirmation on this being the correct path to
follow. I'm not quite sure, because if newlib is a shared library, I
think we'll need to divide the current build stage up to add stages
like:
- Compile librtemscpu.a and librtemsbsp.a with -fPIC
- Compile test_xyz.c while linking it with libc, librtems*, lib*efi,
etc. to create a fully resolved test_xyz.so
- Convert .so to PE using objcopy

Does that make sense to you all?

---

[1] 
https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=blob;f=newlib/README;h=e793d57ce75e56d1eb044e2c0325631e9eeef1af;hb=HEAD#l498
[2] https://sourceware.org/ml/newlib/2016/msg01106.html
[3] 
https://forum.osdev.org/viewtopic.php?p=276046=c7798911615bef866354e92a64125b1c#p276046
[4] https://dev.gentoo.org/~vapier/crt.txtz
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel