Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Ingo Molnar]

* Ingo Molnar  wrote:

> To quantify it: I just performed a test build of a Linux distro kernel config 
> (Fedora x86-64), and counted the number of callsites that use 'asm goto' 
> functionality with the v4.15 kernel (including drivers).
> 
> The results:
> 
> Linux distro | !CONFIG_TRACING
>  -
>  total # of functions :  191,567 | 184,443
>  total # of instructions  :   14,251,355 |  13,526,112
>  -
>  total # of spin_lock*() calls:   25,246 |  25,177
>  total # of mutex_lock*() calls   :   13,062 |  12,861
>  total # of kmalloc*() calls  :5,148 |   5,118
>  -
>  total # of 'asm goto' usage sites:   34,851 |  31,059
>  total # of 'asm goto' using functions:   18,209 |  16,089
>  -
>  percent of kernel functions using 'asm goto' : 9.5% |8.7%
>  -

Here's the size stats of kernel/sched/built-in.o for the same distro config:

 optimized | no asm goto
   -
   total # of functions :  765 |764
   total # of instructions  :   46,830 | 47,051

I.e. asm goto support reduces scheduler size by ~0.5%, which is a major 
generated 
code size reduction.

This doesn't count the live branch patching performance advantages: many of 
those 
asm goto usage sites are in hot paths, so the performance impact of it is much 
larger than that: easily a couple of percentage points in scheduler intensive 
benchmarks, as Peter mentioned.

For example here's a thread context switch benchmark comparison on a modern x86 
system running a v4.15 kernel:

  $ perf stat --repeat 20 --sync --null perf bench sched messaging -t -g 25

 no asm goto: 0.136778505 seconds time elapsed  ( stddev: +- 
0.55% )
  asm goto optimized: 0.133773904 seconds time elapsed  ( stddev: +- 
0.51% )

The asm goto enabled kernel is ~2.25% faster in this benchmark, and the 
performance penalty of not having asm goto support will only increase in the 
future.

i.e. it very much makes sense to implement asm goto support not just for 
compatibility reasons, but for performance reasons as well.

Thanks,

Ingo

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Ingo Molnar]

* Ingo Molnar  wrote:

> To quantify it: I just performed a test build of a Linux distro kernel config 
> (Fedora x86-64), and counted the number of callsites that use 'asm goto' 
> functionality with the v4.15 kernel (including drivers).
> 
> The results:
> 
> Linux distro | !CONFIG_TRACING
>  -
>  total # of functions :  191,567 | 184,443
>  total # of instructions  :   14,251,355 |  13,526,112
>  -
>  total # of spin_lock*() calls:   25,246 |  25,177
>  total # of mutex_lock*() calls   :   13,062 |  12,861
>  total # of kmalloc*() calls  :5,148 |   5,118
>  -
>  total # of 'asm goto' usage sites:   34,851 |  31,059
>  total # of 'asm goto' using functions:   18,209 |  16,089
>  -
>  percent of kernel functions using 'asm goto' : 9.5% |8.7%
>  -

Here's the size stats of kernel/sched/built-in.o for the same distro config:

 optimized | no asm goto
   -
   total # of functions :  765 |764
   total # of instructions  :   46,830 | 47,051

I.e. asm goto support reduces scheduler size by ~0.5%, which is a major 
generated 
code size reduction.

This doesn't count the live branch patching performance advantages: many of 
those 
asm goto usage sites are in hot paths, so the performance impact of it is much 
larger than that: easily a couple of percentage points in scheduler intensive 
benchmarks, as Peter mentioned.

For example here's a thread context switch benchmark comparison on a modern x86 
system running a v4.15 kernel:

  $ perf stat --repeat 20 --sync --null perf bench sched messaging -t -g 25

 no asm goto: 0.136778505 seconds time elapsed  ( stddev: +- 
0.55% )
  asm goto optimized: 0.133773904 seconds time elapsed  ( stddev: +- 
0.51% )

The asm goto enabled kernel is ~2.25% faster in this benchmark, and the 
performance penalty of not having asm goto support will only increase in the 
future.

i.e. it very much makes sense to implement asm goto support not just for 
compatibility reasons, but for performance reasons as well.

Thanks,

Ingo

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Peter Zijlstra]

On Wed, Feb 14, 2018 at 09:31:22AM -0800, Kees Cook wrote:
> On Wed, Feb 14, 2018 at 1:08 AM, Peter Zijlstra  wrote:
> > If you ever get to the point where it makes sense to build a kernel with
> > LLVM [...]
> 
> Just to speak to this point in particular: it makes sense to build
> kernels with LLVM right now. It's already happened; millions of Pixel
> 2 phones are already running with Clang-built kernels. And even if
> you're only interested in x86, that works too.

Only if you really don't care about performance. And as should be
evident from the rest of this thread, you soon won't be able to build
it anymore.

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Peter Zijlstra]

On Wed, Feb 14, 2018 at 09:31:22AM -0800, Kees Cook wrote:
> On Wed, Feb 14, 2018 at 1:08 AM, Peter Zijlstra  wrote:
> > If you ever get to the point where it makes sense to build a kernel with
> > LLVM [...]
> 
> Just to speak to this point in particular: it makes sense to build
> kernels with LLVM right now. It's already happened; millions of Pixel
> 2 phones are already running with Clang-built kernels. And even if
> you're only interested in x86, that works too.

Only if you really don't care about performance. And as should be
evident from the rest of this thread, you soon won't be able to build
it anymore.

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Linus Torvalds]

On Wed, Feb 14, 2018 at 7:31 AM, James Y Knight  wrote:
>
> IMO, inline asm is, generally, a valuable feature to provide in the
> compiler as an escape hatch, and asm goto is a relatively sane
> extension of it.

Side note: one thing that limits "asm goto" in gcc is the fact that
you can't have outputs.

If clang people are looking at "asm goto", the _syntax_ is actually
very straightforward, and there's really absolutely no point in trying
to make up some other interface.

But extending on what gcc does, and allowing outputs (possibly valid
in the fall-through case only, not in the cases where it jumps away to
a label) would be a big improvement on what gcc does.

At that point, we'd start to pester the gcc people to do the clang
thing, rather than the other way around. Because that's only fair ;)

  Linus

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Linus Torvalds]

On Wed, Feb 14, 2018 at 7:31 AM, James Y Knight  wrote:
>
> IMO, inline asm is, generally, a valuable feature to provide in the
> compiler as an escape hatch, and asm goto is a relatively sane
> extension of it.

Side note: one thing that limits "asm goto" in gcc is the fact that
you can't have outputs.

If clang people are looking at "asm goto", the _syntax_ is actually
very straightforward, and there's really absolutely no point in trying
to make up some other interface.

But extending on what gcc does, and allowing outputs (possibly valid
in the fall-through case only, not in the cases where it jumps away to
a label) would be a big improvement on what gcc does.

At that point, we'd start to pester the gcc people to do the clang
thing, rather than the other way around. Because that's only fair ;)

  Linus

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Linus Torvalds]

On Wed, Feb 14, 2018 at 2:34 AM, Peter Zijlstra  wrote:
>
> So it's far more than just tracepoints. We use it all over the kernel to
> do runtime branch patching.

Side note: I have a patch to the user access code to do "asm goto" for
the exception handling too.

In fact, the infrastructure is there already, but because we supported
older gcc versions that didn't have asm goto, I never applied the
final patch to actually make use of it.

See "unsafe_put_user()", which currently does

if (unlikely(__pu_err)) goto err_label;

rather than just have the exception table itself directly reference
"err_label", and not needing any code at all in the regular path.

I suspect there are other things like that, where the lack of "asm
goto" means that it's not worth the pain of having two code paths.
Once we update to a newer gcc requirement, those suddenly will also
very naturally use asm goto.

The other asm use that the clang people might want to look at is the
"condition codes as return value" one. That one we currently handle
with CC_SET/CC_OUT, and it's somewhat more amenable to those kinds of
games, so missing it isn't as nasty. But it *is* a useful "modern"
extension of gcc asms.

Maybe clang already supports cflags output, I haven't checked.

  Linus

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Linus Torvalds]

On Wed, Feb 14, 2018 at 2:34 AM, Peter Zijlstra  wrote:
>
> So it's far more than just tracepoints. We use it all over the kernel to
> do runtime branch patching.

Side note: I have a patch to the user access code to do "asm goto" for
the exception handling too.

In fact, the infrastructure is there already, but because we supported
older gcc versions that didn't have asm goto, I never applied the
final patch to actually make use of it.

See "unsafe_put_user()", which currently does

if (unlikely(__pu_err)) goto err_label;

rather than just have the exception table itself directly reference
"err_label", and not needing any code at all in the regular path.

I suspect there are other things like that, where the lack of "asm
goto" means that it's not worth the pain of having two code paths.
Once we update to a newer gcc requirement, those suddenly will also
very naturally use asm goto.

The other asm use that the clang people might want to look at is the
"condition codes as return value" one. That one we currently handle
with CC_SET/CC_OUT, and it's somewhat more amenable to those kinds of
games, so missing it isn't as nasty. But it *is* a useful "modern"
extension of gcc asms.

Maybe clang already supports cflags output, I haven't checked.

  Linus

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Kees Cook]

On Wed, Feb 14, 2018 at 1:08 AM, Peter Zijlstra  wrote:
> If you ever get to the point where it makes sense to build a kernel with
> LLVM [...]

Just to speak to this point in particular: it makes sense to build
kernels with LLVM right now. It's already happened; millions of Pixel
2 phones are already running with Clang-built kernels. And even if
you're only interested in x86, that works too.

-Kees

-- 
Kees Cook
Pixel Security

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Kees Cook]

On Wed, Feb 14, 2018 at 1:08 AM, Peter Zijlstra  wrote:
> If you ever get to the point where it makes sense to build a kernel with
> LLVM [...]

Just to speak to this point in particular: it makes sense to build
kernels with LLVM right now. It's already happened; millions of Pixel
2 phones are already running with Clang-built kernels. And even if
you're only interested in x86, that works too.

-Kees

-- 
Kees Cook
Pixel Security

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Andy Lutomirski]

On Wed, Feb 14, 2018 at 3:31 PM, James Y Knight  wrote:
> I'd be definitely in favor having clang support asm goto. I wouldn't
> want to exclude having other conversations about how to more directly
> provide compiler features that the linux kernel could use, ALSO, but I
> do not think that conversation should block implementing asm-goto.
>
> IMO, inline asm is, generally, a valuable feature to provide in the
> compiler as an escape hatch, and asm goto is a relatively sane
> extension of it. Supporting outgoing edges from an inline asm block is
> a reasonable thing for users to desire, and as far as anyone's said so
> far, seems like it ought to be fairly easily implementable, without
> causing bad side-effects in the compiler.
>
> Of course, we generally do want to minimize the need for users to use
> inline asm, by providing appropriate compiler support for the features
> people would otherwise be forced to implement using asm. But I don't
> see that as really any more important for asm-goto than any other
> inline asm. There will always be a desire for escape hatches, to do
> weird and unique things which aren't supported directly in the
> compiler. (Also, the kernel is a pretty special case in terms of its
> requirements, it seems exceedingly unlikely that we could ever provide
> enough compiler support to let it eliminate inline asm.).
>

If clang implements asm goto, could you also pretty please make it
better than GCC and support output constraints in the same asm
statements?  This would give us a very clean, short, and fast
implementation for __get_user() and __put_user(), and it would also
help put pressure on GCC to implement the same feature.

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Andy Lutomirski]

On Wed, Feb 14, 2018 at 3:31 PM, James Y Knight  wrote:
> I'd be definitely in favor having clang support asm goto. I wouldn't
> want to exclude having other conversations about how to more directly
> provide compiler features that the linux kernel could use, ALSO, but I
> do not think that conversation should block implementing asm-goto.
>
> IMO, inline asm is, generally, a valuable feature to provide in the
> compiler as an escape hatch, and asm goto is a relatively sane
> extension of it. Supporting outgoing edges from an inline asm block is
> a reasonable thing for users to desire, and as far as anyone's said so
> far, seems like it ought to be fairly easily implementable, without
> causing bad side-effects in the compiler.
>
> Of course, we generally do want to minimize the need for users to use
> inline asm, by providing appropriate compiler support for the features
> people would otherwise be forced to implement using asm. But I don't
> see that as really any more important for asm-goto than any other
> inline asm. There will always be a desire for escape hatches, to do
> weird and unique things which aren't supported directly in the
> compiler. (Also, the kernel is a pretty special case in terms of its
> requirements, it seems exceedingly unlikely that we could ever provide
> enough compiler support to let it eliminate inline asm.).
>

If clang implements asm goto, could you also pretty please make it
better than GCC and support output constraints in the same asm
statements?  This would give us a very clean, short, and fast
implementation for __get_user() and __put_user(), and it would also
help put pressure on GCC to implement the same feature.

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[James Y Knight]

I'd be definitely in favor having clang support asm goto. I wouldn't
want to exclude having other conversations about how to more directly
provide compiler features that the linux kernel could use, ALSO, but I
do not think that conversation should block implementing asm-goto.

IMO, inline asm is, generally, a valuable feature to provide in the
compiler as an escape hatch, and asm goto is a relatively sane
extension of it. Supporting outgoing edges from an inline asm block is
a reasonable thing for users to desire, and as far as anyone's said so
far, seems like it ought to be fairly easily implementable, without
causing bad side-effects in the compiler.

Of course, we generally do want to minimize the need for users to use
inline asm, by providing appropriate compiler support for the features
people would otherwise be forced to implement using asm. But I don't
see that as really any more important for asm-goto than any other
inline asm. There will always be a desire for escape hatches, to do
weird and unique things which aren't supported directly in the
compiler. (Also, the kernel is a pretty special case in terms of its
requirements, it seems exceedingly unlikely that we could ever provide
enough compiler support to let it eliminate inline asm.).

On Wed, Feb 14, 2018 at 3:33 AM, Yatsina, Marina
<marina.yats...@intel.com> wrote:
>
> Hi Kees,
>
> When I raised the question of whether we want to add support for "asm goto" 
> in llvm I got some feedback from the community that "asm goto" might not be 
> the best solution for the problem it was invented for (optimizing support for 
> tracepoints), so I stopped perusing this issue.
> I'm CC-ing the developers that participated in the original thread and a few 
> developers that might be interested in adding support of "asm goto".
> I'm also adding the llvm-dev mailing list, in case there are additional 
> parties interested in voicing their opinion.
>
> I hope this will give this issue a push forward and we will find a solution 
> that will not prevent llvm from compiling linux kernel.
>
> Thanks,
> Marina
>
> -Original Message-
> From: Kees Cook [mailto:keesc...@google.com]
> Sent: Wednesday, February 14, 2018 02:29
> To: David Woodhouse <dw...@infradead.org>; Chandler Carruth 
> <chandl...@google.com>; Yatsina, Marina <marina.yats...@intel.com>
> Cc: Guenter Roeck <li...@roeck-us.net>; X86 ML <x...@kernel.org>; LKML 
> <linux-kernel@vger.kernel.org>; Alan Cox <gno...@lxorguk.ukuu.org.uk>; Rik 
> van Riel <r...@redhat.com>; Andi Kleen <a...@linux.intel.com>; Josh Poimboeuf 
> <jpoim...@redhat.com>; Tom Lendacky <thomas.lenda...@amd.com>; Peter Zijlstra 
> <pet...@infradead.org>; Linus Torvalds <torva...@linux-foundation.org>; Jiri 
> Kosina <ji...@kernel.org>; Andy Lutomirski <l...@amacapital.net>; Hansen, 
> Dave <dave.han...@intel.com>; Tim Chen <tim.c.c...@linux.intel.com>; Greg 
> Kroah-Hartman <gre...@linux-foundation.org>; Paul Turner <p...@google.com>; 
> Stephen Hines <srhi...@google.com>; Nick Desaulniers <ndesaulni...@google.com>
> Subject: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang 
> support)
>
> On Tue, Feb 13, 2018 at 4:10 PM, David Woodhouse <dw...@infradead.org> wrote:
> > We also need to resolve the asm-goto thing.
>
> Yes, this is becoming much more urgent, assuming we'll be raising the minimum 
> GCC version soon and drop support for lacking asm-goto...
>
> Do you happen to know who the right people are to include to move the 
> discussion forward? I know various kernel folks that are passionate about it, 
> but I'm still getting to know who to talk with from llvm.
>
> I see an earlier thread here:
> http://lists.llvm.org/pipermail/llvm-dev/2017-April/111748.html
>
> It seems to end there? I'm still coming up to speed on it, so I'm likely 
> missing other context.
>
> -Kees
>
> --
> Kees Cook
> Pixel Security
> -
> Intel Israel (74) Limited
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[James Y Knight]

I'd be definitely in favor having clang support asm goto. I wouldn't
want to exclude having other conversations about how to more directly
provide compiler features that the linux kernel could use, ALSO, but I
do not think that conversation should block implementing asm-goto.

IMO, inline asm is, generally, a valuable feature to provide in the
compiler as an escape hatch, and asm goto is a relatively sane
extension of it. Supporting outgoing edges from an inline asm block is
a reasonable thing for users to desire, and as far as anyone's said so
far, seems like it ought to be fairly easily implementable, without
causing bad side-effects in the compiler.

Of course, we generally do want to minimize the need for users to use
inline asm, by providing appropriate compiler support for the features
people would otherwise be forced to implement using asm. But I don't
see that as really any more important for asm-goto than any other
inline asm. There will always be a desire for escape hatches, to do
weird and unique things which aren't supported directly in the
compiler. (Also, the kernel is a pretty special case in terms of its
requirements, it seems exceedingly unlikely that we could ever provide
enough compiler support to let it eliminate inline asm.).

On Wed, Feb 14, 2018 at 3:33 AM, Yatsina, Marina
 wrote:
>
> Hi Kees,
>
> When I raised the question of whether we want to add support for "asm goto" 
> in llvm I got some feedback from the community that "asm goto" might not be 
> the best solution for the problem it was invented for (optimizing support for 
> tracepoints), so I stopped perusing this issue.
> I'm CC-ing the developers that participated in the original thread and a few 
> developers that might be interested in adding support of "asm goto".
> I'm also adding the llvm-dev mailing list, in case there are additional 
> parties interested in voicing their opinion.
>
> I hope this will give this issue a push forward and we will find a solution 
> that will not prevent llvm from compiling linux kernel.
>
> Thanks,
> Marina
>
> -Original Message-
> From: Kees Cook [mailto:keesc...@google.com]
> Sent: Wednesday, February 14, 2018 02:29
> To: David Woodhouse ; Chandler Carruth 
> ; Yatsina, Marina 
> Cc: Guenter Roeck ; X86 ML ; LKML 
> ; Alan Cox ; Rik 
> van Riel ; Andi Kleen ; Josh Poimboeuf 
> ; Tom Lendacky ; Peter Zijlstra 
> ; Linus Torvalds ; Jiri 
> Kosina ; Andy Lutomirski ; Hansen, 
> Dave ; Tim Chen ; Greg 
> Kroah-Hartman ; Paul Turner ; 
> Stephen Hines ; Nick Desaulniers 
> Subject: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang 
> support)
>
> On Tue, Feb 13, 2018 at 4:10 PM, David Woodhouse  wrote:
> > We also need to resolve the asm-goto thing.
>
> Yes, this is becoming much more urgent, assuming we'll be raising the minimum 
> GCC version soon and drop support for lacking asm-goto...
>
> Do you happen to know who the right people are to include to move the 
> discussion forward? I know various kernel folks that are passionate about it, 
> but I'm still getting to know who to talk with from llvm.
>
> I see an earlier thread here:
> http://lists.llvm.org/pipermail/llvm-dev/2017-April/111748.html
>
> It seems to end there? I'm still coming up to speed on it, so I'm likely 
> missing other context.
>
> -Kees
>
> --
> Kees Cook
> Pixel Security
> -
> Intel Israel (74) Limited
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Ingo Molnar]

* Peter Zijlstra  wrote:

> On Wed, Feb 14, 2018 at 09:52:59AM +, Yatsina, Marina wrote:
> > Hi Peter,
> > 
> > When I started the original thread last year I was in favor of adding
> > "asm goto" and didn't understand why it wasn't done by that time.  The
> > feedback I got is that this feature (optimizing tracepoints) is very
> > useful and that we do want it in llvm, but perhaps there's a cleaner
> > way of implementing than "asm goto". An alternative suggestion arose
> > as well. 
> 
> So it's far more than just tracepoints. We use it all over the kernel to
> do runtime branch patching.
> 
> One example is avoiding the scheduler preemption callbacks if we know
> there are no users. This shaves a few % off a context switch
> micro-bench.
> 
> But it is really _all_ over the place.

To quantify it: I just performed a test build of a Linux distro kernel config 
(Fedora x86-64), and counted the number of callsites that use 'asm goto' 
functionality with the v4.15 kernel (including drivers).

The results:

Linux distro | !CONFIG_TRACING
 -
 total # of functions :  191,567 | 184,443
 total # of instructions  :   14,251,355 |  13,526,112
 -
 total # of spin_lock*() calls:   25,246 |  25,177
 total # of mutex_lock*() calls   :   13,062 |  12,861
 total # of kmalloc*() calls  :5,148 |   5,118
 -
 total # of 'asm goto' usage sites:   34,851 |  31,059
 total # of 'asm goto' using functions:   18,209 |  16,089
 -
 percent of kernel functions using 'asm goto' : 9.5% |8.7%
 -


( Note: I added the 'tracing disabled' column only to demonstrate how widely
'asm goto' is used beyond tracing - but since *all* major Linux 
distributions have tracing enabled it's the first column that matters 
in 
practice. )

So 'asm goto' is a major, major compiler feature the Linux kernel relies on:
in fact in the Linux kernel there's more 'asm goto' usage sites than there are
spin-lock critical sections!

Thanks,

Ingo

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Ingo Molnar]

* Peter Zijlstra  wrote:

> On Wed, Feb 14, 2018 at 09:52:59AM +, Yatsina, Marina wrote:
> > Hi Peter,
> > 
> > When I started the original thread last year I was in favor of adding
> > "asm goto" and didn't understand why it wasn't done by that time.  The
> > feedback I got is that this feature (optimizing tracepoints) is very
> > useful and that we do want it in llvm, but perhaps there's a cleaner
> > way of implementing than "asm goto". An alternative suggestion arose
> > as well. 
> 
> So it's far more than just tracepoints. We use it all over the kernel to
> do runtime branch patching.
> 
> One example is avoiding the scheduler preemption callbacks if we know
> there are no users. This shaves a few % off a context switch
> micro-bench.
> 
> But it is really _all_ over the place.

To quantify it: I just performed a test build of a Linux distro kernel config 
(Fedora x86-64), and counted the number of callsites that use 'asm goto' 
functionality with the v4.15 kernel (including drivers).

The results:

Linux distro | !CONFIG_TRACING
 -
 total # of functions :  191,567 | 184,443
 total # of instructions  :   14,251,355 |  13,526,112
 -
 total # of spin_lock*() calls:   25,246 |  25,177
 total # of mutex_lock*() calls   :   13,062 |  12,861
 total # of kmalloc*() calls  :5,148 |   5,118
 -
 total # of 'asm goto' usage sites:   34,851 |  31,059
 total # of 'asm goto' using functions:   18,209 |  16,089
 -
 percent of kernel functions using 'asm goto' : 9.5% |8.7%
 -


( Note: I added the 'tracing disabled' column only to demonstrate how widely
'asm goto' is used beyond tracing - but since *all* major Linux 
distributions have tracing enabled it's the first column that matters 
in 
practice. )

So 'asm goto' is a major, major compiler feature the Linux kernel relies on:
in fact in the Linux kernel there's more 'asm goto' usage sites than there are
spin-lock critical sections!

Thanks,

Ingo

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Thomas Gleixner]

On Wed, 14 Feb 2018, Peter Zijlstra wrote:
> On Wed, Feb 14, 2018 at 08:41:43AM +, Chandler Carruth wrote:
> > Marina, Kees:
> > 
> > I think the combination of LKML and the LLVM developer's list is probably
> > too large a set of folks to have a (somewhat) lengthy exploration of
> > options and how to go about this and find a good path forward.
> 
> As one of the original authors of the jump-label implementation I'm
> saying there are no options. We're not going to do a second parallel
> implementation. If you want to build the kernel, you get to implement
> asm-goto, teh end.

Amen to that.

As x86 maintainer I'm really interested and willing to support llvm/clang
for x86 and the kernel in general, but asm-goto is not longer negotiable.

asm-goto is in the kernel since Oct 2010 and the relevant llvm/clang bug
#9295 is open since Feb 2011. All clang/llvm folks came up with in those 7
years is occasional handwaving about better solutions.

We really try to support old tools and tools which lack features for quite
a long time, but that's not a free pass for eternity.

That said, I'm not willing to wait another 7 years and I'm going to pull
the plug on compilers w/o asm-goto support soon whether llvm/clang gets
their act together or not.

Hint: GCC is not the only major compiler which supports asm-goto.

Thanks,

tglx

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Thomas Gleixner]

On Wed, 14 Feb 2018, Peter Zijlstra wrote:
> On Wed, Feb 14, 2018 at 08:41:43AM +, Chandler Carruth wrote:
> > Marina, Kees:
> > 
> > I think the combination of LKML and the LLVM developer's list is probably
> > too large a set of folks to have a (somewhat) lengthy exploration of
> > options and how to go about this and find a good path forward.
> 
> As one of the original authors of the jump-label implementation I'm
> saying there are no options. We're not going to do a second parallel
> implementation. If you want to build the kernel, you get to implement
> asm-goto, teh end.

Amen to that.

As x86 maintainer I'm really interested and willing to support llvm/clang
for x86 and the kernel in general, but asm-goto is not longer negotiable.

asm-goto is in the kernel since Oct 2010 and the relevant llvm/clang bug
#9295 is open since Feb 2011. All clang/llvm folks came up with in those 7
years is occasional handwaving about better solutions.

We really try to support old tools and tools which lack features for quite
a long time, but that's not a free pass for eternity.

That said, I'm not willing to wait another 7 years and I'm going to pull
the plug on compilers w/o asm-goto support soon whether llvm/clang gets
their act together or not.

Hint: GCC is not the only major compiler which supports asm-goto.

Thanks,

tglx

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Peter Zijlstra]

On Wed, Feb 14, 2018 at 09:52:59AM +, Yatsina, Marina wrote:
> Hi Peter,
> 
> When I started the original thread last year I was in favor of adding
> "asm goto" and didn't understand why it wasn't done by that time.  The
> feedback I got is that this feature (optimizing tracepoints) is very
> useful and that we do want it in llvm, but perhaps there's a cleaner
> way of implementing than "asm goto". An alternative suggestion arose
> as well. 

So it's far more than just tracepoints. We use it all over the kernel to
do runtime branch patching.

One example is avoiding the scheduler preemption callbacks if we know
there are no users. This shaves a few % off a context switch
micro-bench.

But it is really _all_ over the place.

> I'm sure you can provide a lot of background for the decisions of why
> "asm goto" was chosen and which other alternatives were considered, as
> you were the one to implement this.

I have very little memories from back then, but it was mostly us asking
for label addresses in asm and them giving us asm-goto.

Using asm we can build our own primitives, and I realize the llvm
community doesn't like asm much, but then again, we treat C like a
glorified assembler and don't like our compilers too smart :-)

> Anyway, I think we should consider the alternatives and not take "asm
> goto" as a given.

Far too late for that, 7+ years ago when we did this was the time to
talk about alternatives, now we have this code base.

So we have the two jump_label things:

static __always_inline bool arch_static_branch(struct static_key *key, bool 
branch)
{
asm_volatile_goto("1:"
".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t"
".pushsection __jump_table,  \"aw\" \n\t"
_ASM_ALIGN "\n\t"
_ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t"
".popsection \n\t"
: :  "i" (key), "i" (branch) : : l_yes);

return false;
l_yes:
return true;
}

static __always_inline bool arch_static_branch_jump(struct static_key *key, 
bool branch)
{
asm_volatile_goto("1:"
".byte 0xe9\n\t .long %l[l_yes] - 2f\n\t"
"2:\n\t"
".pushsection __jump_table,  \"aw\" \n\t"
_ASM_ALIGN "\n\t"
_ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t"
".popsection \n\t"
: :  "i" (key), "i" (branch) : : l_yes);

return false;
l_yes:
return true;
}

Where we emit either a 5 byte jump or a 5 byte nop and write a special
section with meta-data for the branch point.

You could possibly capture all that with a built-in, but would have to
exactly match our meta-data section and then we'd still be up some creek
without no paddle when we need to change it.

But we also have:

static __always_inline __pure bool _static_cpu_has(u16 bit)
{
asm_volatile_goto("1: jmp 6f\n"
 "2:\n"
 ".skip -(((5f-4f) - (2b-1b)) > 0) * "
 "((5f-4f) - (2b-1b)),0x90\n"
 "3:\n"
 ".section .altinstructions,\"a\"\n"
 " .long 1b - .\n"  /* src offset */
 " .long 4f - .\n"  /* repl offset */
 " .word %P[always]\n"  /* always replace */
 " .byte 3b - 1b\n" /* src len */
 " .byte 5f - 4f\n" /* repl len */
 " .byte 3b - 2b\n" /* pad len */
 ".previous\n"
 ".section .altinstr_replacement,\"ax\"\n"
 "4: jmp %l[t_no]\n"
 "5:\n"
 ".previous\n"
 ".section .altinstructions,\"a\"\n"
 " .long 1b - .\n"  /* src offset */
 " .long 0\n"   /* no replacement */
 " .word %P[feature]\n" /* feature bit */
 " .byte 3b - 1b\n" /* src len */
 " .byte 0\n"   /* repl len */
 " .byte 0\n"   /* pad len */
 ".previous\n"
 ".section .altinstr_aux,\"ax\"\n"
 "6:\n"
 " testb %[bitnum],%[cap_byte]\n"
 " jnz %l[t_yes]\n"
 " jmp %l[t_no]\n"
 ".previous\n"
 : : [feature]  "i" (bit),
 [always]   "i" (X86_FEATURE_ALWAYS),
 [bitnum]   "i" (1 << (bit & 7)),
 [cap_byte] "m" (((const char 
*)boot_cpu_data.x86_capability)[bit >> 3])
 : : t_yes, t_no);
t_yes:
return true;
t_no:
return false;
}

Which does something similar, but with a completely different meta-data
section and a different pre-patch fallback path.

But we also do things like:

#define __GEN_RMWcc(fullop, var, cc, clobbers, ...) \
do {

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Peter Zijlstra]

On Wed, Feb 14, 2018 at 09:52:59AM +, Yatsina, Marina wrote:
> Hi Peter,
> 
> When I started the original thread last year I was in favor of adding
> "asm goto" and didn't understand why it wasn't done by that time.  The
> feedback I got is that this feature (optimizing tracepoints) is very
> useful and that we do want it in llvm, but perhaps there's a cleaner
> way of implementing than "asm goto". An alternative suggestion arose
> as well. 

So it's far more than just tracepoints. We use it all over the kernel to
do runtime branch patching.

One example is avoiding the scheduler preemption callbacks if we know
there are no users. This shaves a few % off a context switch
micro-bench.

But it is really _all_ over the place.

> I'm sure you can provide a lot of background for the decisions of why
> "asm goto" was chosen and which other alternatives were considered, as
> you were the one to implement this.

I have very little memories from back then, but it was mostly us asking
for label addresses in asm and them giving us asm-goto.

Using asm we can build our own primitives, and I realize the llvm
community doesn't like asm much, but then again, we treat C like a
glorified assembler and don't like our compilers too smart :-)

> Anyway, I think we should consider the alternatives and not take "asm
> goto" as a given.

Far too late for that, 7+ years ago when we did this was the time to
talk about alternatives, now we have this code base.

So we have the two jump_label things:

static __always_inline bool arch_static_branch(struct static_key *key, bool 
branch)
{
asm_volatile_goto("1:"
".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t"
".pushsection __jump_table,  \"aw\" \n\t"
_ASM_ALIGN "\n\t"
_ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t"
".popsection \n\t"
: :  "i" (key), "i" (branch) : : l_yes);

return false;
l_yes:
return true;
}

static __always_inline bool arch_static_branch_jump(struct static_key *key, 
bool branch)
{
asm_volatile_goto("1:"
".byte 0xe9\n\t .long %l[l_yes] - 2f\n\t"
"2:\n\t"
".pushsection __jump_table,  \"aw\" \n\t"
_ASM_ALIGN "\n\t"
_ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t"
".popsection \n\t"
: :  "i" (key), "i" (branch) : : l_yes);

return false;
l_yes:
return true;
}

Where we emit either a 5 byte jump or a 5 byte nop and write a special
section with meta-data for the branch point.

You could possibly capture all that with a built-in, but would have to
exactly match our meta-data section and then we'd still be up some creek
without no paddle when we need to change it.

But we also have:

static __always_inline __pure bool _static_cpu_has(u16 bit)
{
asm_volatile_goto("1: jmp 6f\n"
 "2:\n"
 ".skip -(((5f-4f) - (2b-1b)) > 0) * "
 "((5f-4f) - (2b-1b)),0x90\n"
 "3:\n"
 ".section .altinstructions,\"a\"\n"
 " .long 1b - .\n"  /* src offset */
 " .long 4f - .\n"  /* repl offset */
 " .word %P[always]\n"  /* always replace */
 " .byte 3b - 1b\n" /* src len */
 " .byte 5f - 4f\n" /* repl len */
 " .byte 3b - 2b\n" /* pad len */
 ".previous\n"
 ".section .altinstr_replacement,\"ax\"\n"
 "4: jmp %l[t_no]\n"
 "5:\n"
 ".previous\n"
 ".section .altinstructions,\"a\"\n"
 " .long 1b - .\n"  /* src offset */
 " .long 0\n"   /* no replacement */
 " .word %P[feature]\n" /* feature bit */
 " .byte 3b - 1b\n" /* src len */
 " .byte 0\n"   /* repl len */
 " .byte 0\n"   /* pad len */
 ".previous\n"
 ".section .altinstr_aux,\"ax\"\n"
 "6:\n"
 " testb %[bitnum],%[cap_byte]\n"
 " jnz %l[t_yes]\n"
 " jmp %l[t_no]\n"
 ".previous\n"
 : : [feature]  "i" (bit),
 [always]   "i" (X86_FEATURE_ALWAYS),
 [bitnum]   "i" (1 << (bit & 7)),
 [cap_byte] "m" (((const char 
*)boot_cpu_data.x86_capability)[bit >> 3])
 : : t_yes, t_no);
t_yes:
return true;
t_no:
return false;
}

Which does something similar, but with a completely different meta-data
section and a different pre-patch fallback path.

But we also do things like:

#define __GEN_RMWcc(fullop, var, cc, clobbers, ...) \
do {

RE: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Yatsina, Marina]

I believe we do have the infrastructure in llvm to implement a working patch.
The "asm goto" was introduced as a solution to a problem (optimizing 
tracepoints/ nop patching).
When we started the discussion in llvm about adding this feature, questions 
arose if we can design a better solution for this problem than "asm goto".

You can find a more about the discussion in the llvm community here:
http://lists.llvm.org/pipermail/llvm-dev/2017-April/111748.html

* This is the middle of the thread, and there are more messages later on.

Thanks,
Marina

-Original Message-
From: Greg Kroah-Hartman [mailto:gre...@linux-foundation.org] 
Sent: Wednesday, February 14, 2018 11:53
To: Chandler Carruth <chandl...@google.com>
Cc: Yatsina, Marina <marina.yats...@intel.com>; Kees Cook 
<keesc...@google.com>; David Woodhouse <dw...@infradead.org>; Kreitzer, David L 
<david.l.kreit...@intel.com>; Grischenko, Andrei L 
<andrei.l.grische...@intel.com>; r...@google.com; LLVM Developers 
<llvm-...@lists.llvm.org>; eh...@mozilla.com; Tayree, Coby 
<coby.tay...@intel.com>; Matthias Braun <ma...@braunis.de>; Dean Michael Berris 
<dean.ber...@gmail.com>; James Y Knight <jykni...@google.com>; Guenter Roeck 
<li...@roeck-us.net>; X86 ML <x...@kernel.org>; LKML 
<linux-kernel@vger.kernel.org>; Alan Cox <gno...@lxorguk.ukuu.org.uk>; Rik van 
Riel <r...@redhat.com>; Andi Kleen <a...@linux.intel.com>; Josh Poimboeuf 
<jpoim...@redhat.com>; Tom Lendacky <thomas.lenda...@amd.com>; Peter Zijlstra 
<pet...@infradead.org>; Linus Torvalds <torva...@linux-foundation.org>; Jiri 
Kosina <ji...@kernel.org>; Andy Lutomirski <l...@amacapital.net>; Hansen, Dave 
<dave.han...@intel.com>; Tim Chen <tim.c.c...@linux.intel.com>; Paul Turner 
<p...@google.com>; Stephen Hines <srhi...@google.com>; Nick Desaulniers 
<ndesaulni...@google.com>
Subject: Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add 
clang support)

On Wed, Feb 14, 2018 at 08:41:43AM +, Chandler Carruth wrote:
> Marina, Kees:
> 
> I think the combination of LKML and the LLVM developer's list is 
> probably too large a set of folks to have a (somewhat) lengthy 
> exploration of options and how to go about this and find a good path forward.

Why is there any need to discuss "options" here?  What is preventing the 
implementation of "asm goto" in llvm?  Is it the lack of a working patch, or 
something else?

thanks,

greg k-h
-
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

RE: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Yatsina, Marina]

I believe we do have the infrastructure in llvm to implement a working patch.
The "asm goto" was introduced as a solution to a problem (optimizing 
tracepoints/ nop patching).
When we started the discussion in llvm about adding this feature, questions 
arose if we can design a better solution for this problem than "asm goto".

You can find a more about the discussion in the llvm community here:
http://lists.llvm.org/pipermail/llvm-dev/2017-April/111748.html

* This is the middle of the thread, and there are more messages later on.

Thanks,
Marina

-Original Message-
From: Greg Kroah-Hartman [mailto:gre...@linux-foundation.org] 
Sent: Wednesday, February 14, 2018 11:53
To: Chandler Carruth 
Cc: Yatsina, Marina ; Kees Cook 
; David Woodhouse ; Kreitzer, David L 
; Grischenko, Andrei L 
; r...@google.com; LLVM Developers 
; eh...@mozilla.com; Tayree, Coby 
; Matthias Braun ; Dean Michael Berris 
; James Y Knight ; Guenter Roeck 
; X86 ML ; LKML 
; Alan Cox ; Rik van 
Riel ; Andi Kleen ; Josh Poimboeuf 
; Tom Lendacky ; Peter Zijlstra 
; Linus Torvalds ; Jiri 
Kosina ; Andy Lutomirski ; Hansen, Dave 
; Tim Chen ; Paul Turner 
; Stephen Hines ; Nick Desaulniers 

Subject: Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add 
clang support)

On Wed, Feb 14, 2018 at 08:41:43AM +, Chandler Carruth wrote:
> Marina, Kees:
> 
> I think the combination of LKML and the LLVM developer's list is 
> probably too large a set of folks to have a (somewhat) lengthy 
> exploration of options and how to go about this and find a good path forward.

Why is there any need to discuss "options" here?  What is preventing the 
implementation of "asm goto" in llvm?  Is it the lack of a working patch, or 
something else?

thanks,

greg k-h
-
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

RE: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Yatsina, Marina]

Hi Peter,

When I started the original thread last year I was in favor of adding "asm 
goto" and didn't understand why it wasn't done by that time.
The feedback I got is that this feature (optimizing tracepoints) is very useful 
and that we do want it in llvm, but perhaps there's a cleaner way of 
implementing than "asm goto". An alternative suggestion arose as well. 

I'm sure you can provide a lot of background for the decisions of why "asm 
goto" was chosen and which other alternatives were considered, as you were the 
one to implement this. This may shed a lot of light on the alternative 
suggestions that arose in my thread in the llvm community.
I think that including GCC folks in the "small team" discussion Chandler 
suggested might be a very good idea, so if you have some contacts, please send 
them to Chandler.

Anyway, I think we should consider the alternatives and not take "asm goto" as 
a given, even though this is the implementation of this feature for the past 7 
years and it works well.
I'm not saying linux kernel needs to rush and adopt an alternative solution 
ASAP, but we cannot ignore that it exists and may (or may not) be the direction 
of the future (for both gcc and llvm).

Thanks,
Marina

-Original Message-
From: Peter Zijlstra [mailto:pet...@infradead.org] 
Sent: Wednesday, February 14, 2018 11:09
To: Yatsina, Marina <marina.yats...@intel.com>
Cc: Kees Cook <keesc...@google.com>; David Woodhouse <dw...@infradead.org>; 
Chandler Carruth <chandl...@google.com>; Kreitzer, David L 
<david.l.kreit...@intel.com>; Grischenko, Andrei L 
<andrei.l.grische...@intel.com>; r...@google.com; LLVM Developers 
<llvm-...@lists.llvm.org>; eh...@mozilla.com; Tayree, Coby 
<coby.tay...@intel.com>; Matthias Braun <ma...@braunis.de>; Dean Michael Berris 
<dean.ber...@gmail.com>; James Y Knight <jykni...@google.com>; Guenter Roeck 
<li...@roeck-us.net>; X86 ML <x...@kernel.org>; LKML 
<linux-kernel@vger.kernel.org>; Alan Cox <gno...@lxorguk.ukuu.org.uk>; Rik van 
Riel <r...@redhat.com>; Andi Kleen <a...@linux.intel.com>; Josh Poimboeuf 
<jpoim...@redhat.com>; Tom Lendacky <thomas.lenda...@amd.com>; Linus Torvalds 
<torva...@linux-foundation.org>; Jiri Kosina <ji...@kernel.org>; Andy 
Lutomirski <l...@amacapital.net>; Hansen, Dave <dave.han...@intel.com>; Tim 
Chen <tim.c.c...@linux.intel.com>; Greg Kroah-Hartman 
<gre...@linux-foundation.org>; Paul Turner <p...@google.com>; Stephen Hines 
<srhi...@google.com>; Nick Desaulniers <ndesaulni...@google.com>
Subject: Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add 
clang support)

On Wed, Feb 14, 2018 at 08:33:39AM +, Yatsina, Marina wrote:
> I hope this will give this issue a push forward and we will find a 
> solution that will not prevent llvm from compiling linux kernel.

That solution is asm-goto. We (linux kernel) are not going to support something 
like it. It's going to be asm-goto or bust. The feature is over 7 years old by 
now, deal with it.

If you ever get to the point where it makes sense to build a kernel with LLVM, 
then you can bike-shed with the GCC team on new features, until then tough luck.
-
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

RE: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Yatsina, Marina]

Hi Peter,

When I started the original thread last year I was in favor of adding "asm 
goto" and didn't understand why it wasn't done by that time.
The feedback I got is that this feature (optimizing tracepoints) is very useful 
and that we do want it in llvm, but perhaps there's a cleaner way of 
implementing than "asm goto". An alternative suggestion arose as well. 

I'm sure you can provide a lot of background for the decisions of why "asm 
goto" was chosen and which other alternatives were considered, as you were the 
one to implement this. This may shed a lot of light on the alternative 
suggestions that arose in my thread in the llvm community.
I think that including GCC folks in the "small team" discussion Chandler 
suggested might be a very good idea, so if you have some contacts, please send 
them to Chandler.

Anyway, I think we should consider the alternatives and not take "asm goto" as 
a given, even though this is the implementation of this feature for the past 7 
years and it works well.
I'm not saying linux kernel needs to rush and adopt an alternative solution 
ASAP, but we cannot ignore that it exists and may (or may not) be the direction 
of the future (for both gcc and llvm).

Thanks,
Marina

-Original Message-
From: Peter Zijlstra [mailto:pet...@infradead.org] 
Sent: Wednesday, February 14, 2018 11:09
To: Yatsina, Marina 
Cc: Kees Cook ; David Woodhouse ; 
Chandler Carruth ; Kreitzer, David L 
; Grischenko, Andrei L 
; r...@google.com; LLVM Developers 
; eh...@mozilla.com; Tayree, Coby 
; Matthias Braun ; Dean Michael Berris 
; James Y Knight ; Guenter Roeck 
; X86 ML ; LKML 
; Alan Cox ; Rik van 
Riel ; Andi Kleen ; Josh Poimboeuf 
; Tom Lendacky ; Linus Torvalds 
; Jiri Kosina ; Andy 
Lutomirski ; Hansen, Dave ; Tim 
Chen ; Greg Kroah-Hartman 
; Paul Turner ; Stephen Hines 
; Nick Desaulniers 
Subject: Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add 
clang support)

On Wed, Feb 14, 2018 at 08:33:39AM +, Yatsina, Marina wrote:
> I hope this will give this issue a push forward and we will find a 
> solution that will not prevent llvm from compiling linux kernel.

That solution is asm-goto. We (linux kernel) are not going to support something 
like it. It's going to be asm-goto or bust. The feature is over 7 years old by 
now, deal with it.

If you ever get to the point where it makes sense to build a kernel with LLVM, 
then you can bike-shed with the GCC team on new features, until then tough luck.
-
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Greg Kroah-Hartman]

On Wed, Feb 14, 2018 at 08:41:43AM +, Chandler Carruth wrote:
> Marina, Kees:
> 
> I think the combination of LKML and the LLVM developer's list is probably
> too large a set of folks to have a (somewhat) lengthy exploration of
> options and how to go about this and find a good path forward.

Why is there any need to discuss "options" here?  What is preventing the
implementation of "asm goto" in llvm?  Is it the lack of a working
patch, or something else?

thanks,

greg k-h

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Greg Kroah-Hartman]

On Wed, Feb 14, 2018 at 08:41:43AM +, Chandler Carruth wrote:
> Marina, Kees:
> 
> I think the combination of LKML and the LLVM developer's list is probably
> too large a set of folks to have a (somewhat) lengthy exploration of
> options and how to go about this and find a good path forward.

Why is there any need to discuss "options" here?  What is preventing the
implementation of "asm goto" in llvm?  Is it the lack of a working
patch, or something else?

thanks,

greg k-h

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Peter Zijlstra]

On Wed, Feb 14, 2018 at 08:41:43AM +, Chandler Carruth wrote:
> Marina, Kees:
> 
> I think the combination of LKML and the LLVM developer's list is probably
> too large a set of folks to have a (somewhat) lengthy exploration of
> options and how to go about this and find a good path forward.

As one of the original authors of the jump-label implementation I'm
saying there are no options. We're not going to do a second parallel
implementation. If you want to build the kernel, you get to implement
asm-goto, teh end.

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Peter Zijlstra]

On Wed, Feb 14, 2018 at 08:41:43AM +, Chandler Carruth wrote:
> Marina, Kees:
> 
> I think the combination of LKML and the LLVM developer's list is probably
> too large a set of folks to have a (somewhat) lengthy exploration of
> options and how to go about this and find a good path forward.

As one of the original authors of the jump-label implementation I'm
saying there are no options. We're not going to do a second parallel
implementation. If you want to build the kernel, you get to implement
asm-goto, teh end.

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Peter Zijlstra]

On Wed, Feb 14, 2018 at 08:33:39AM +, Yatsina, Marina wrote:
> I hope this will give this issue a push forward and we will find a
> solution that will not prevent llvm from compiling linux kernel.

That solution is asm-goto. We (linux kernel) are not going to support
something like it. It's going to be asm-goto or bust. The feature is
over 7 years old by now, deal with it.

If you ever get to the point where it makes sense to build a kernel with
LLVM, then you can bike-shed with the GCC team on new features, until
then tough luck.

Re: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Peter Zijlstra]

On Wed, Feb 14, 2018 at 08:33:39AM +, Yatsina, Marina wrote:
> I hope this will give this issue a push forward and we will find a
> solution that will not prevent llvm from compiling linux kernel.

That solution is asm-goto. We (linux kernel) are not going to support
something like it. It's going to be asm-goto or bust. The feature is
over 7 years old by now, deal with it.

If you ever get to the point where it makes sense to build a kernel with
LLVM, then you can bike-shed with the GCC team on new features, until
then tough luck.

RE: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Yatsina, Marina]

Hi Kees,

When I raised the question of whether we want to add support for "asm goto" in 
llvm I got some feedback from the community that "asm goto" might not be the 
best solution for the problem it was invented for (optimizing support for 
tracepoints), so I stopped perusing this issue.
I'm CC-ing the developers that participated in the original thread and a few 
developers that might be interested in adding support of "asm goto".
I'm also adding the llvm-dev mailing list, in case there are additional parties 
interested in voicing their opinion.

I hope this will give this issue a push forward and we will find a solution 
that will not prevent llvm from compiling linux kernel.

Thanks,
Marina

-Original Message-
From: Kees Cook [mailto:keesc...@google.com] 
Sent: Wednesday, February 14, 2018 02:29
To: David Woodhouse <dw...@infradead.org>; Chandler Carruth 
<chandl...@google.com>; Yatsina, Marina <marina.yats...@intel.com>
Cc: Guenter Roeck <li...@roeck-us.net>; X86 ML <x...@kernel.org>; LKML 
<linux-kernel@vger.kernel.org>; Alan Cox <gno...@lxorguk.ukuu.org.uk>; Rik van 
Riel <r...@redhat.com>; Andi Kleen <a...@linux.intel.com>; Josh Poimboeuf 
<jpoim...@redhat.com>; Tom Lendacky <thomas.lenda...@amd.com>; Peter Zijlstra 
<pet...@infradead.org>; Linus Torvalds <torva...@linux-foundation.org>; Jiri 
Kosina <ji...@kernel.org>; Andy Lutomirski <l...@amacapital.net>; Hansen, Dave 
<dave.han...@intel.com>; Tim Chen <tim.c.c...@linux.intel.com>; Greg 
Kroah-Hartman <gre...@linux-foundation.org>; Paul Turner <p...@google.com>; 
Stephen Hines <srhi...@google.com>; Nick Desaulniers <ndesaulni...@google.com>
Subject: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang 
support)

On Tue, Feb 13, 2018 at 4:10 PM, David Woodhouse <dw...@infradead.org> wrote:
> We also need to resolve the asm-goto thing.

Yes, this is becoming much more urgent, assuming we'll be raising the minimum 
GCC version soon and drop support for lacking asm-goto...

Do you happen to know who the right people are to include to move the 
discussion forward? I know various kernel folks that are passionate about it, 
but I'm still getting to know who to talk with from llvm.

I see an earlier thread here:
http://lists.llvm.org/pipermail/llvm-dev/2017-April/111748.html

It seems to end there? I'm still coming up to speed on it, so I'm likely 
missing other context.

-Kees

--
Kees Cook
Pixel Security
-
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

RE: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang support)

[Yatsina, Marina]

Hi Kees,

When I raised the question of whether we want to add support for "asm goto" in 
llvm I got some feedback from the community that "asm goto" might not be the 
best solution for the problem it was invented for (optimizing support for 
tracepoints), so I stopped perusing this issue.
I'm CC-ing the developers that participated in the original thread and a few 
developers that might be interested in adding support of "asm goto".
I'm also adding the llvm-dev mailing list, in case there are additional parties 
interested in voicing their opinion.

I hope this will give this issue a push forward and we will find a solution 
that will not prevent llvm from compiling linux kernel.

Thanks,
Marina

-Original Message-
From: Kees Cook [mailto:keesc...@google.com] 
Sent: Wednesday, February 14, 2018 02:29
To: David Woodhouse ; Chandler Carruth 
; Yatsina, Marina 
Cc: Guenter Roeck ; X86 ML ; LKML 
; Alan Cox ; Rik van 
Riel ; Andi Kleen ; Josh Poimboeuf 
; Tom Lendacky ; Peter Zijlstra 
; Linus Torvalds ; Jiri 
Kosina ; Andy Lutomirski ; Hansen, Dave 
; Tim Chen ; Greg 
Kroah-Hartman ; Paul Turner ; 
Stephen Hines ; Nick Desaulniers 
Subject: clang asm-goto support (Was Re: [PATCH v2] x86/retpoline: Add clang 
support)

On Tue, Feb 13, 2018 at 4:10 PM, David Woodhouse  wrote:
> We also need to resolve the asm-goto thing.

Yes, this is becoming much more urgent, assuming we'll be raising the minimum 
GCC version soon and drop support for lacking asm-goto...

Do you happen to know who the right people are to include to move the 
discussion forward? I know various kernel folks that are passionate about it, 
but I'm still getting to know who to talk with from llvm.

I see an earlier thread here:
http://lists.llvm.org/pipermail/llvm-dev/2017-April/111748.html

It seems to end there? I'm still coming up to speed on it, so I'm likely 
missing other context.

-Kees

--
Kees Cook
Pixel Security
-
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.