I've just spent the last two days in the same code porting the runtime to a similarly almost-linux-ELF system, so I may have some information that's useful to you. There are several variants on how Go does this depending on the system, so please bear with me if I slip up.
If we fix GOOS=linux, GOARCH=amd64, and -buildmode=exe, then: Go follows the ELF ABI for TLS. It uses a single TLS slot, named runtime.tlsg. It's at -8. With internal linking, this fixed slot is easy to ensure. All go code only generates the one. The only external C that's linked in is trusted not to take a slot away. (Notice there's an exception where cmd/link refuses to internally link the C variant of the net package on dragonfly, because dragonfly uses __thread for errno, meaning the libc might want that slot.) With external linking, runtime.tlsg is the only TLS LE variable. All the others have to be at least LD or GD, because they are coming from shared libraries. When CGO_ENABLED=0, there's a problem when new threads are created. Typically the libc creating the thread sets %fs to point to the TLS slots. But it's unset, because we created the thread without libc, via a direct clone syscall. So we need to emulate what the libc does, which we do in the runtime.settls function by calling arch_prctl. In particular we subtract 8 from the value we write into %fs so we get exactly what the libc would have done, we are not stealing the slot directly. In fact, after this is done, it's safe for that thread to run C code. Note that this function is *not* called when CGO_ENABLED=1. Hopefully the fact we are not clobbering %fs for this combination of build options makes your life easier. All of these facts change as you vary GOOS, GOARCH, buildmode, and CGO_ENABLED. (For example, -buildmode=pie implies PIC code which makes the TLS variable IE instead of LE, darwin/android use far off fixed slot offsets, ARM has a more accessible TLS register, etc.) The trickiest thing around is CanUse1InsnTLS in the compiler, which tripped me up for a while. Spend some time reading the comment at the call site of that function. Good luck! On Thu, Sep 1, 2016 at 7:05 AM, Nadav Har'El <[email protected]> wrote: > Hi, we are trying to run code compiled with the golang compiler to on the > OSv kernel. > OSv is a unikernel (kernel for a virtual-machine intending to run a single > program at a time). It is mostly compatible with the Linux glibc ABI, > meaning that you can "mostly" take unmodified Linux executable code and run > it on OSv. > > We're in fact very close to running golang-compiled code on OSv. But we're > having trouble with the code's use of TLS (thread-local storage), and I > wanted to ask for advice from the experts on golang's implementation. > > If I understand correctly, golang uses only one per-thread variable, "g" (or > maybe also a second one, "m"?). > > The ELF ABI has a standard (https://www.akkadia.org/drepper/tls.pdf) on how > several components of an executable linked at run time can "share" the > single fs_base offset (on x86) used to implement TLS. So while Go only needs > a single "g" thread-local variable, some other shared library loaded in the > same process might need an additional thread-local variable, and the ELF > standard nicely allows to pull that off. > > BUT, from the code we've inspected, it seems that golang-compiled code > doesn't use this technique. Instead, it seems it uses arch_prctl's > ARCH_SET_FS to completely take over the fs_base of Go threads. This would > cause big problems for us when these Go threads want to call other non-Go > functions (in OSv, the system calls are ordinary functions too). > > My question is - is there a "build mode" or something where Go can allocate > its "g" variable in the usual ELF way - and cooperate with other C code > running in the same process - instead of taking over the fs_base? > > I have read conflicting reports on what the "c-shared" mode does. On one > hand it seems it does *something* similar to what I want - it seems it has a > single TLS variable (in the "initial exec" variant of the ELF TLS standard). > On the other hand, I also read it still uses prctl()... > > So *should* c-shared do what I am looking for? Does it in fact do this > currently? > > Thanks. I'm looking forward to seeing Go code running on OSv soon :-) > > -- > Nadav Har'El > [email protected] > > -- > You received this message because you are subscribed to the Google Groups > "golang-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
