Re: [clang] 24ab9b5 - Generalize the pass registration mechanism used by Polly to any third-party tool

2020-01-09 Thread Michael Kruse via cfe-commits
Am Mi., 8. Jan. 2020 um 17:20 Uhr schrieb Eric Christopher :
>
>
>
> On Wed, Jan 8, 2020 at 3:07 PM Michael Kruse  
> wrote:
>>
>> Am Di., 7. Jan. 2020 um 18:57 Uhr schrieb Eric Christopher via
>> cfe-commits :
>> >> Is there anything I should have done? Probably reaching llvm-dev before
>> >> commiting. Reaching the right reviewers has always been a challenge to 
>> >> me, I had
>> >> hoped that the mail to llvm-dev would trigger some subscription :-)
>> >
>> >
>> > Hrm. Probably finding some different reviewers, but I can't fault your 
>> > attempts here. Usually you can look at the last few people to make 
>> > substantial work in an area and loop them in via git log. :)
>>
>> In my experience adding more reviewers usually rarely to more people
>> looking into the patch. Especially the people that make substantial
>> work in an area and/or are code owners are added as reviewers to very
>> many commits and cannot look into all of them.
>
>
> As one of those added to a lot of patches I have a lot of sympathy here. The 
> answer isn't to avoid adding people with expertise in an area and no one on 
> the approval list had expertise or had recently hacked on the code.

For some parts you won't find people that recently worked on the code
other than refactoring, such as the legacy pass manager. Together with
Tobias Grosser, I worked out the mechanism that statically linked
Polly into tools which was generalized in this patch. I am not sure
what else would be required to approve this patch.


>
>>
>> Note that in the patch in question, there are comments from people not
>> in the reviewer list. For example, Mehdi with whom I also talked about
>> this patch at the last LLVM DevMtg.
>>
>
> Yes, but none of them ack'd it.

Mehdi did in the comments (and personally at the DevMtg). He just did
not approve the patch because I still had concerns. By coding policy,
only one approval is required.



>> >> >  ; CHECK-EP-VECTORIZER-START-NEXT: Running pass: NoOpFunctionPass
>> >> > +; CHECK-EXT: Running pass: {{.*}}::Bye on foo
>> >> >
>> >> > Why is this running on every test of the pass manager? It should be an 
>> >> > example
>> >> > run in the examples directory and not on by default? Same for every 
>> >> > other PM
>> >> > test. This seems like a bug?
>> >>
>> >> It's not. When the examples are active and if the appropriate cmake flag 
>> >> is set
>> >> (which is not the case by default), the pass is linked in statically, and 
>> >> is run
>> >> in the default pipeline. The CHECK-EXT prefix is disabled otherwise. 
>> >> That's one
>> >> of the configuration I did test :-)
>> >>
>> >
>> > I really don't think this is ideal. The examples directory shouldn't 
>> > affect tests being run or not or in what way. Can we back this part out 
>> > and talk about it a bit more? I don't think we should need to do this to 
>> > test the functionality.
>>
>> In a previous discussion [1] we discussed adding tutorial code to
>> repository which naturally should also be tested to avoid bit-rot. How
>> do you suggest to testing of examples should work? Note that LLVMHello
>> is also tested, but not in the example directory.
>
>
> llvm/tests/examples?

The point is, we don't have an established pattern of how to test
examples yet, and I would also like that we develop one. I also think
this was not in the scope of this patch.


Michael
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits


Re: [clang] 24ab9b5 - Generalize the pass registration mechanism used by Polly to any third-party tool

2020-01-08 Thread Eric Christopher via cfe-commits
On Wed, Jan 8, 2020 at 1:40 AM Serge Guelton  wrote:

> On Tue, Jan 07, 2020 at 04:57:29PM -0800, Eric Christopher wrote:
> > >  ; CHECK-EP-VECTORIZER-START-NEXT: Running pass: NoOpFunctionPass
> > > +; CHECK-EXT: Running pass: {{.*}}::Bye on foo
> > >
> > > Why is this running on every test of the pass manager? It should
> be an
> > example
> > > run in the examples directory and not on by default? Same for
> every other
> > PM
> > > test. This seems like a bug?
> >
> > It's not. When the examples are active and if the appropriate cmake
> flag is
> > set
> > (which is not the case by default), the pass is linked in
> statically, and
> > is run
> > in the default pipeline. The CHECK-EXT prefix is disabled otherwise.
> That's
> > one
> > of the configuration I did test :-)
> >
> >
> >
> > I really don't think this is ideal. The examples directory shouldn't
> affect
> > tests being run or not or in what way. Can we back this part out and
> talk about
> > it a bit more? I don't think we should need to do this to test the
> > functionality.
>
> I can just error if LLVM_BYE_LINK_INTO_TOOLS=ON, and remove that part. It
> means
> less testing, but somehow Polly is already some kind of test case. Would
> that be okay with you?
>

I'd really like to avoid linking bye into any of the tools by default and
instead just come up with a different test mechanism to show it's working.

-eric
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits


Re: [clang] 24ab9b5 - Generalize the pass registration mechanism used by Polly to any third-party tool

2020-01-08 Thread Eric Christopher via cfe-commits
On Wed, Jan 8, 2020 at 3:07 PM Michael Kruse 
wrote:

> Am Di., 7. Jan. 2020 um 18:57 Uhr schrieb Eric Christopher via
> cfe-commits :
> >> Is there anything I should have done? Probably reaching llvm-dev before
> >> commiting. Reaching the right reviewers has always been a challenge to
> me, I had
> >> hoped that the mail to llvm-dev would trigger some subscription :-)
> >
> >
> > Hrm. Probably finding some different reviewers, but I can't fault your
> attempts here. Usually you can look at the last few people to make
> substantial work in an area and loop them in via git log. :)
>
> In my experience adding more reviewers usually rarely to more people
> looking into the patch. Especially the people that make substantial
> work in an area and/or are code owners are added as reviewers to very
> many commits and cannot look into all of them.
>

As one of those added to a lot of patches I have a lot of sympathy here.
The answer isn't to avoid adding people with expertise in an area and no
one on the approval list had expertise or had recently hacked on the code.


> Note that in the patch in question, there are comments from people not
> in the reviewer list. For example, Mehdi with whom I also talked about
> this patch at the last LLVM DevMtg.
>
>
Yes, but none of them ack'd it.


> We also have a policy that allows post-commit reviews and reverts.
> Therefore I think we should not put that bar for allowing to commit
> too high.
>
>
I could, of course, have just asked this to be reverted I guess.


>
> >> >  ; CHECK-EP-VECTORIZER-START-NEXT: Running pass: NoOpFunctionPass
> >> > +; CHECK-EXT: Running pass: {{.*}}::Bye on foo
> >> >
> >> > Why is this running on every test of the pass manager? It should be
> an example
> >> > run in the examples directory and not on by default? Same for every
> other PM
> >> > test. This seems like a bug?
> >>
> >> It's not. When the examples are active and if the appropriate cmake
> flag is set
> >> (which is not the case by default), the pass is linked in statically,
> and is run
> >> in the default pipeline. The CHECK-EXT prefix is disabled otherwise.
> That's one
> >> of the configuration I did test :-)
> >>
> >
> > I really don't think this is ideal. The examples directory shouldn't
> affect tests being run or not or in what way. Can we back this part out and
> talk about it a bit more? I don't think we should need to do this to test
> the functionality.
>
> In a previous discussion [1] we discussed adding tutorial code to
> repository which naturally should also be tested to avoid bit-rot. How
> do you suggest to testing of examples should work? Note that LLVMHello
> is also tested, but not in the example directory.
>

llvm/tests/examples?

Thanks.

-eric
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits


Re: [clang] 24ab9b5 - Generalize the pass registration mechanism used by Polly to any third-party tool

2020-01-08 Thread Michael Kruse via cfe-commits
Am Di., 7. Jan. 2020 um 18:57 Uhr schrieb Eric Christopher via
cfe-commits :
>> Is there anything I should have done? Probably reaching llvm-dev before
>> commiting. Reaching the right reviewers has always been a challenge to me, I 
>> had
>> hoped that the mail to llvm-dev would trigger some subscription :-)
>
>
> Hrm. Probably finding some different reviewers, but I can't fault your 
> attempts here. Usually you can look at the last few people to make 
> substantial work in an area and loop them in via git log. :)

In my experience adding more reviewers usually rarely to more people
looking into the patch. Especially the people that make substantial
work in an area and/or are code owners are added as reviewers to very
many commits and cannot look into all of them.
Note that in the patch in question, there are comments from people not
in the reviewer list. For example, Mehdi with whom I also talked about
this patch at the last LLVM DevMtg.

We also have a policy that allows post-commit reviews and reverts.
Therefore I think we should not put that bar for allowing to commit
too high.


>> >  ; CHECK-EP-VECTORIZER-START-NEXT: Running pass: NoOpFunctionPass
>> > +; CHECK-EXT: Running pass: {{.*}}::Bye on foo
>> >
>> > Why is this running on every test of the pass manager? It should be an 
>> > example
>> > run in the examples directory and not on by default? Same for every other 
>> > PM
>> > test. This seems like a bug?
>>
>> It's not. When the examples are active and if the appropriate cmake flag is 
>> set
>> (which is not the case by default), the pass is linked in statically, and is 
>> run
>> in the default pipeline. The CHECK-EXT prefix is disabled otherwise. That's 
>> one
>> of the configuration I did test :-)
>>
>
> I really don't think this is ideal. The examples directory shouldn't affect 
> tests being run or not or in what way. Can we back this part out and talk 
> about it a bit more? I don't think we should need to do this to test the 
> functionality.

In a previous discussion [1] we discussed adding tutorial code to
repository which naturally should also be tested to avoid bit-rot. How
do you suggest to testing of examples should work? Note that LLVMHello
is also tested, but not in the example directory.

For this patch I suggested in the review to add another flag to the
Bye example and only if set the the Bye pass is added to the pass
manager (but in a different context: to avoid shell expansion hack
that would not run on Windows anyway). This would stop the pass
pipeline pass to fail because of a new pass in there and can be tested
separately. At the end I had no preference of how it should be tested.

[1] http://lists.llvm.org/pipermail/llvm-dev/2019-October/136159.html
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits


Re: [clang] 24ab9b5 - Generalize the pass registration mechanism used by Polly to any third-party tool

2020-01-08 Thread Serge Guelton via cfe-commits
On Tue, Jan 07, 2020 at 04:57:29PM -0800, Eric Christopher wrote:
> >  ; CHECK-EP-VECTORIZER-START-NEXT: Running pass: NoOpFunctionPass
> > +; CHECK-EXT: Running pass: {{.*}}::Bye on foo
> >
> > Why is this running on every test of the pass manager? It should be an
> example
> > run in the examples directory and not on by default? Same for every 
> other
> PM
> > test. This seems like a bug?
> 
> It's not. When the examples are active and if the appropriate cmake flag 
> is
> set
> (which is not the case by default), the pass is linked in statically, and
> is run
> in the default pipeline. The CHECK-EXT prefix is disabled otherwise. 
> That's
> one
> of the configuration I did test :-)
> 
> 
> 
> I really don't think this is ideal. The examples directory shouldn't affect
> tests being run or not or in what way. Can we back this part out and talk 
> about
> it a bit more? I don't think we should need to do this to test the
> functionality.

I can just error if LLVM_BYE_LINK_INTO_TOOLS=ON, and remove that part. It means
less testing, but somehow Polly is already some kind of test case. Would
that be okay with you?

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits


Re: [clang] 24ab9b5 - Generalize the pass registration mechanism used by Polly to any third-party tool

2020-01-07 Thread Eric Christopher via cfe-commits
Hi Serge!

Thanks for a lot of the answers here :)

>
> The proposed infrastructure makes this process smoother and non intrusive
> to the
> llvm-project codebase: all development can be done in a separate git repo,
> integration is controlled through cmake flags, and integration to
> clang/opt/bugpoint is built-in.
>
>
Sounds reasonably nice.


> > That being said, perhaps it is worth it? But I think we need to call out
> why we
> > want it. I would also have expected something to llvm-dev for a change
> of this
> > magnitude. I didn't see anyone from the pass manager hierarchy on the
> reviews
> > and the final reviewer wasn't someone who contributes to these areas
> typically.
>
> I've (obviously) mentioned this development on llvm-dev, see
>
> http://lists.llvm.org/pipermail/llvm-dev/2019-September/135326.html
>
> Is there anything I should have done? Probably reaching llvm-dev before
> commiting. Reaching the right reviewers has always been a challenge to me,
> I had
> hoped that the mail to llvm-dev would trigger some subscription :-)
>

Hrm. Probably finding some different reviewers, but I can't fault your
attempts here. Usually you can look at the last few people to make
substantial work in an area and loop them in via git log. :)

> In addition, what's with the OSX failure? It's currently turned off and
> was
> > breaking the bots, but does it mean that you don't expect this machinery
> to
> > work on OSX? That seems like a severely limiting factor for the project.
>
> I've setup github actions to test many configurations before merging [0],
> but missed one of
> them. I'm currently working on fixing that part.
>

Nice! Glad for the work here.


> >  ; CHECK-EP-VECTORIZER-START-NEXT: Running pass: NoOpFunctionPass
> > +; CHECK-EXT: Running pass: {{.*}}::Bye on foo
> >
> > Why is this running on every test of the pass manager? It should be an
> example
> > run in the examples directory and not on by default? Same for every
> other PM
> > test. This seems like a bug?
>
> It's not. When the examples are active and if the appropriate cmake flag
> is set
> (which is not the case by default), the pass is linked in statically, and
> is run
> in the default pipeline. The CHECK-EXT prefix is disabled otherwise.
> That's one
> of the configuration I did test :-)
>
>
I really don't think this is ideal. The examples directory shouldn't affect
tests being run or not or in what way. Can we back this part out and talk
about it a bit more? I don't think we should need to do this to test the
functionality.


> > Thanks!
>
> Thanks for giving me the opportunity to clarify some obscure points, hope
> it
> helps !
>
>
It very much did. Thanks for explaining.

-eric
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits


Re: [clang] 24ab9b5 - Generalize the pass registration mechanism used by Polly to any third-party tool

2020-01-07 Thread Serge Guelton via cfe-commits
On Mon, Jan 06, 2020 at 03:45:15PM -0800, Eric Christopher wrote:
> Hi Serge,

Hi Eric,

> I have a few questions here about this:
> 
> In general this appears to be a lot more complex than the existing plugin
> solutions and requires quite a lot of custom cmake rules that are difficult to
> maintain. Why do we want this in general for the project? I understand the
> desire to make polly less of a special case, but I don't think that the 
> overall
> complexity is worth the added ability to register polly plugins. Perhaps
> disabling the polly plugin registration scheme is a better option?

Indeed, the cmake machinery is now more generic (making Polly an application of
it, instead of an exception), at the expense of being more complex.

Why would we want that? When trying to write a custom pass, the usual process,
is to write a plugin that works correctly with opt, and that step is relatively
easy. To achieve a decent clang integration, it gets more difficult, and if one
still take the external plugin approach, one ends up with very long and non
standard command line like -Xclang -load -Xclang MyPlugin.so etc. So one need
another step to integrate the plugin into the static build, which means forking
llvm-project, rebasing every now and then etc.

The proposed infrastructure makes this process smoother and non intrusive to the
llvm-project codebase: all development can be done in a separate git repo,
integration is controlled through cmake flags, and integration to
clang/opt/bugpoint is built-in.

> That being said, perhaps it is worth it? But I think we need to call out why 
> we
> want it. I would also have expected something to llvm-dev for a change of this
> magnitude. I didn't see anyone from the pass manager hierarchy on the reviews
> and the final reviewer wasn't someone who contributes to these areas 
> typically.

I've (obviously) mentioned this development on llvm-dev, see

http://lists.llvm.org/pipermail/llvm-dev/2019-September/135326.html

Is there anything I should have done? Probably reaching llvm-dev before
commiting. Reaching the right reviewers has always been a challenge to me, I had
hoped that the mail to llvm-dev would trigger some subscription :-)

> In addition, what's with the OSX failure? It's currently turned off and was
> breaking the bots, but does it mean that you don't expect this machinery to
> work on OSX? That seems like a severely limiting factor for the project.

I've setup github actions to test many configurations before merging [0], but 
missed one of
them. I'm currently working on fixing that part.

> +One first needs to create an independent project and add it to either 
> ``tools/
> ``
> +or, using the MonoRepo layout, at the root of the repo alongside other
> projects.
> 
> This appears to be from an earlier incarnation of the patch? There's only one
> layout now.

Correct! I'll fix that.

> 
>  ; CHECK-EP-VECTORIZER-START-NEXT: Running pass: NoOpFunctionPass
> +; CHECK-EXT: Running pass: {{.*}}::Bye on foo
> 
> Why is this running on every test of the pass manager? It should be an example
> run in the examples directory and not on by default? Same for every other PM
> test. This seems like a bug?

It's not. When the examples are active and if the appropriate cmake flag is set
(which is not the case by default), the pass is linked in statically, and is run
in the default pipeline. The CHECK-EXT prefix is disabled otherwise. That's one
of the configuration I did test :-)

> Thanks!

Thanks for giving me the opportunity to clarify some obscure points, hope it
helps !


[0] 
https://github.com/serge-sans-paille/llvm-project/commit/37c9b5080acd3d7543347de109e44c402626d504/checks?check_suite_id=346849887

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits


Re: [clang] 24ab9b5 - Generalize the pass registration mechanism used by Polly to any third-party tool

2020-01-06 Thread Eric Christopher via cfe-commits
Hi Serge,

I have a few questions here about this:

In general this appears to be a lot more complex than the existing plugin
solutions and requires quite a lot of custom cmake rules that are difficult
to maintain. Why do we want this in general for the project? I understand
the desire to make polly less of a special case, but I don't think that the
overall complexity is worth the added ability to register polly plugins.
Perhaps disabling the polly plugin registration scheme is a better option?

That being said, perhaps it is worth it? But I think we need to call out
why we want it. I would also have expected something to llvm-dev for a
change of this magnitude. I didn't see anyone from the pass manager
hierarchy on the reviews and the final reviewer wasn't someone who
contributes to these areas typically.

In addition, what's with the OSX failure? It's currently turned off and was
breaking the bots, but does it mean that you don't expect this machinery to
work on OSX? That seems like a severely limiting factor for the project.

+One first needs to create an independent project and add it to either
``tools/``
+or, using the MonoRepo layout, at the root of the repo alongside other
projects.

This appears to be from an earlier incarnation of the patch? There's only
one layout now.


 ; CHECK-EP-VECTORIZER-START-NEXT: Running pass: NoOpFunctionPass
+; CHECK-EXT: Running pass: {{.*}}::Bye on foo

Why is this running on every test of the pass manager? It should be an
example run in the examples directory and not on by default? Same for every
other PM test. This seems like a bug?

Thanks!

-eric


On Thu, Jan 2, 2020 at 7:52 AM via cfe-commits 
wrote:

>
> Author: serge_sans_paille
> Date: 2020-01-02T16:45:31+01:00
> New Revision: 24ab9b537e61b3fe5e6a1019492ff6530d82a3ee
>
> URL:
> https://github.com/llvm/llvm-project/commit/24ab9b537e61b3fe5e6a1019492ff6530d82a3ee
> DIFF:
> https://github.com/llvm/llvm-project/commit/24ab9b537e61b3fe5e6a1019492ff6530d82a3ee.diff
>
> LOG: Generalize the pass registration mechanism used by Polly to any
> third-party tool
>
> There's quite a lot of references to Polly in the LLVM CMake codebase.
> However
> the registration pattern used by Polly could be useful to other external
> projects: thanks to that mechanism it would be possible to develop LLVM
> extension without touching the LLVM code base.
>
> This patch has two effects:
>
> 1. Remove all code specific to Polly in the llvm/clang codebase,
> replaicing it
>with a generic mechanism
>
> 2. Provide a generic mechanism to register compiler extensions.
>
> A compiler extension is similar to a pass plugin, with the notable
> difference
> that the compiler extension can be configured to be built dynamically (like
> plugins) or statically (like regular passes).
>
> As a result, people willing to add extra passes to clang/opt can do it
> using a
> separate code repo, but still have their pass be linked in clang/opt as
> built-in
> passes.
>
> Differential Revision: https://reviews.llvm.org/D61446
>
> Added:
> llvm/examples/Bye/Bye.cpp
> llvm/examples/Bye/CMakeLists.txt
> llvm/test/Feature/load_extension.ll
> polly/lib/Plugin/Polly.cpp
>
> Modified:
> clang/lib/CodeGen/BackendUtil.cpp
> clang/lib/CodeGen/CMakeLists.txt
> clang/tools/driver/CMakeLists.txt
> clang/tools/driver/cc1_main.cpp
> llvm/CMakeLists.txt
> llvm/cmake/modules/AddLLVM.cmake
> llvm/docs/WritingAnLLVMPass.rst
> llvm/examples/CMakeLists.txt
> llvm/include/llvm/Config/llvm-config.h.cmake
> llvm/test/Other/new-pm-defaults.ll
> llvm/test/Other/new-pm-thinlto-defaults.ll
> llvm/test/Other/opt-O0-pipeline.ll
> llvm/test/Other/opt-O2-pipeline.ll
> llvm/test/Other/opt-O3-pipeline.ll
> llvm/test/Other/opt-Os-pipeline.ll
> llvm/test/lit.cfg.py
> llvm/test/lit.site.cfg.py.in
> llvm/tools/CMakeLists.txt
> llvm/tools/bugpoint/CMakeLists.txt
> llvm/tools/bugpoint/bugpoint.cpp
> llvm/tools/opt/CMakeLists.txt
> llvm/tools/opt/NewPMDriver.cpp
> llvm/tools/opt/opt.cpp
> llvm/utils/gn/secondary/llvm/include/llvm/Config/BUILD.gn
> polly/include/polly/RegisterPasses.h
> polly/lib/CMakeLists.txt
> polly/lib/Support/RegisterPasses.cpp
> polly/test/Unit/lit.site.cfg.in
> polly/test/lit.site.cfg.in
> polly/test/update_check.py
>
> Removed:
> polly/lib/Polly.cpp
>
>
>
> 
> diff  --git a/clang/lib/CodeGen/BackendUtil.cpp
> b/clang/lib/CodeGen/BackendUtil.cpp
> index 645ef0165a53..ed881f2ddf68 100644
> --- a/clang/lib/CodeGen/BackendUtil.cpp
> +++ b/clang/lib/CodeGen/BackendUtil.cpp
> @@ -75,6 +75,10 @@
>  using namespace clang;
>  using namespace llvm;
>
> +#define HANDLE_EXTENSION(Ext)
>   \
> +  llvm::PassPluginLibraryInfo get##Ext##PluginInfo();
> +#include "llvm/Support/Extension.def"
> +
>  namespace {
>
>  // Default filename used for profile generation.
> @@ 

[clang] 24ab9b5 - Generalize the pass registration mechanism used by Polly to any third-party tool

2020-01-02 Thread via cfe-commits

Author: serge_sans_paille
Date: 2020-01-02T16:45:31+01:00
New Revision: 24ab9b537e61b3fe5e6a1019492ff6530d82a3ee

URL: 
https://github.com/llvm/llvm-project/commit/24ab9b537e61b3fe5e6a1019492ff6530d82a3ee
DIFF: 
https://github.com/llvm/llvm-project/commit/24ab9b537e61b3fe5e6a1019492ff6530d82a3ee.diff

LOG: Generalize the pass registration mechanism used by Polly to any 
third-party tool

There's quite a lot of references to Polly in the LLVM CMake codebase. However
the registration pattern used by Polly could be useful to other external
projects: thanks to that mechanism it would be possible to develop LLVM
extension without touching the LLVM code base.

This patch has two effects:

1. Remove all code specific to Polly in the llvm/clang codebase, replaicing it
   with a generic mechanism

2. Provide a generic mechanism to register compiler extensions.

A compiler extension is similar to a pass plugin, with the notable difference
that the compiler extension can be configured to be built dynamically (like
plugins) or statically (like regular passes).

As a result, people willing to add extra passes to clang/opt can do it using a
separate code repo, but still have their pass be linked in clang/opt as built-in
passes.

Differential Revision: https://reviews.llvm.org/D61446

Added: 
llvm/examples/Bye/Bye.cpp
llvm/examples/Bye/CMakeLists.txt
llvm/test/Feature/load_extension.ll
polly/lib/Plugin/Polly.cpp

Modified: 
clang/lib/CodeGen/BackendUtil.cpp
clang/lib/CodeGen/CMakeLists.txt
clang/tools/driver/CMakeLists.txt
clang/tools/driver/cc1_main.cpp
llvm/CMakeLists.txt
llvm/cmake/modules/AddLLVM.cmake
llvm/docs/WritingAnLLVMPass.rst
llvm/examples/CMakeLists.txt
llvm/include/llvm/Config/llvm-config.h.cmake
llvm/test/Other/new-pm-defaults.ll
llvm/test/Other/new-pm-thinlto-defaults.ll
llvm/test/Other/opt-O0-pipeline.ll
llvm/test/Other/opt-O2-pipeline.ll
llvm/test/Other/opt-O3-pipeline.ll
llvm/test/Other/opt-Os-pipeline.ll
llvm/test/lit.cfg.py
llvm/test/lit.site.cfg.py.in
llvm/tools/CMakeLists.txt
llvm/tools/bugpoint/CMakeLists.txt
llvm/tools/bugpoint/bugpoint.cpp
llvm/tools/opt/CMakeLists.txt
llvm/tools/opt/NewPMDriver.cpp
llvm/tools/opt/opt.cpp
llvm/utils/gn/secondary/llvm/include/llvm/Config/BUILD.gn
polly/include/polly/RegisterPasses.h
polly/lib/CMakeLists.txt
polly/lib/Support/RegisterPasses.cpp
polly/test/Unit/lit.site.cfg.in
polly/test/lit.site.cfg.in
polly/test/update_check.py

Removed: 
polly/lib/Polly.cpp



diff  --git a/clang/lib/CodeGen/BackendUtil.cpp 
b/clang/lib/CodeGen/BackendUtil.cpp
index 645ef0165a53..ed881f2ddf68 100644
--- a/clang/lib/CodeGen/BackendUtil.cpp
+++ b/clang/lib/CodeGen/BackendUtil.cpp
@@ -75,6 +75,10 @@
 using namespace clang;
 using namespace llvm;
 
+#define HANDLE_EXTENSION(Ext)  
\
+  llvm::PassPluginLibraryInfo get##Ext##PluginInfo();
+#include "llvm/Support/Extension.def"
+
 namespace {
 
 // Default filename used for profile generation.
@@ -1076,6 +1080,9 @@ void EmitAssemblyHelper::EmitAssemblyWithNewPassManager(
   << PluginFN << toString(PassPlugin.takeError());
 }
   }
+#define HANDLE_EXTENSION(Ext)  
\
+  get##Ext##PluginInfo().RegisterPassBuilderCallbacks(PB);
+#include "llvm/Support/Extension.def"
 
   LoopAnalysisManager LAM(CodeGenOpts.DebugPassManager);
   FunctionAnalysisManager FAM(CodeGenOpts.DebugPassManager);

diff  --git a/clang/lib/CodeGen/CMakeLists.txt 
b/clang/lib/CodeGen/CMakeLists.txt
index a3980637c592..d8b3c234a1ef 100644
--- a/clang/lib/CodeGen/CMakeLists.txt
+++ b/clang/lib/CodeGen/CMakeLists.txt
@@ -96,6 +96,8 @@ add_clang_library(clangCodeGen
   TargetInfo.cpp
   VarBypassDetector.cpp
 
+  ENABLE_PLUGINS
+
   DEPENDS
   ${codegen_deps}
 

diff  --git a/clang/tools/driver/CMakeLists.txt 
b/clang/tools/driver/CMakeLists.txt
index 52a95e6bc6b1..2b783cff0955 100644
--- a/clang/tools/driver/CMakeLists.txt
+++ b/clang/tools/driver/CMakeLists.txt
@@ -122,7 +122,3 @@ if(CLANG_ORDER_FILE AND
 set_target_properties(clang PROPERTIES LINK_DEPENDS ${CLANG_ORDER_FILE})
   endif()
 endif()
-
-if(WITH_POLLY AND LINK_POLLY_INTO_TOOLS)
-  target_link_libraries(clang PRIVATE Polly)
-endif(WITH_POLLY AND LINK_POLLY_INTO_TOOLS)

diff  --git a/clang/tools/driver/cc1_main.cpp b/clang/tools/driver/cc1_main.cpp
index 2b82a4378111..b551e9f4cf82 100644
--- a/clang/tools/driver/cc1_main.cpp
+++ b/clang/tools/driver/cc1_main.cpp
@@ -72,12 +72,6 @@ static void LLVMErrorHandler(void *UserData, const 
std::string ,
   exit(GenCrashDiag ? 70 : 1);
 }
 
-#ifdef LINK_POLLY_INTO_TOOLS
-namespace polly {
-void initializePollyPasses(llvm::PassRegistry );
-}
-#endif
-
 #ifdef CLANG_HAVE_RLIMITS
 #if defined(__linux__) && defined(__PIE__)
 static size_t