Re: [lldb-dev] [Openmp-dev] [cfe-dev] RFC: End-to-end testing

[David Blaikie via lldb-dev]

On Tue, Oct 8, 2019 at 12:46 PM David Greene  wrote:

> David Blaikie via Openmp-dev  writes:
>
> > I have a bit of concern about this sort of thing - worrying it'll lead to
> > people being less cautious about writing the more isolated tests.
>
> That's a fair concern.  Reviewers will still need to insist on small
> component-level tests to go along with patches.  We don't have to
> sacrifice one to get the other.
>
> > Dunno if they need a new place or should just be more stuff in
> test-suite,
> > though.
>
> There are at least two problems I see with using test-suite for this:
>
> - It is a separate repository and thus is not as convenient as tests
>   that live with the code.  One cannot commit an end-to-end test
>   atomically with the change meant to be tested.
>
> - It is full of large codes which is not the kind of testing I'm talking
>   about.
>

Oh, right - I'd forgotten that the test-suite wasn't part of the monorepo
(due to size, I can understand why) - fair enough. Makes sense to me to
have lit-style lightweight, targeted, but intentionally end-to-end tests.


>
> Let me describe how I recently added some testing in our downstream
> fork.
>
> - I implemented a new feature along with a C source test.
>
> - I used clang to generate asm from that test and captured the small
>   piece of it I wanted to check in an end-to-end test.
>
> - I used clang to generate IR just before the feature kicked in and
>   created an opt-style test for it.  Generating this IR is not always
>   straightfoward and it would be great to have better tools to do this,
>   but that's another discussion.
>
> - I took the IR out of opt (after running my feature) and created an
>   llc-style test out of it to check the generated asm.  The checks are
>   the same as in the original C end-to-end test.
>
> So the tests are checking at each stage that the expected input is
> generating the expected output and the end-to-end test checks that we go
> from source to asm correctly.
>
> These are all really small tests, easily runnable as part of the normal
> "make check" process.
>
>  -David
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] [Openmp-dev] [cfe-dev] RFC: End-to-end testing

[David Greene via lldb-dev]

David Blaikie via Openmp-dev  writes:

> I have a bit of concern about this sort of thing - worrying it'll lead to
> people being less cautious about writing the more isolated tests.

That's a fair concern.  Reviewers will still need to insist on small
component-level tests to go along with patches.  We don't have to
sacrifice one to get the other.

> Dunno if they need a new place or should just be more stuff in test-suite,
> though.

There are at least two problems I see with using test-suite for this:

- It is a separate repository and thus is not as convenient as tests
  that live with the code.  One cannot commit an end-to-end test
  atomically with the change meant to be tested.

- It is full of large codes which is not the kind of testing I'm talking
  about.

Let me describe how I recently added some testing in our downstream
fork.

- I implemented a new feature along with a C source test.

- I used clang to generate asm from that test and captured the small
  piece of it I wanted to check in an end-to-end test.

- I used clang to generate IR just before the feature kicked in and
  created an opt-style test for it.  Generating this IR is not always
  straightfoward and it would be great to have better tools to do this,
  but that's another discussion.

- I took the IR out of opt (after running my feature) and created an
  llc-style test out of it to check the generated asm.  The checks are
  the same as in the original C end-to-end test.

So the tests are checking at each stage that the expected input is
generating the expected output and the end-to-end test checks that we go
from source to asm correctly.

These are all really small tests, easily runnable as part of the normal
"make check" process.

 -David
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] [cfe-dev] RFC: End-to-end testing

[David Blaikie via lldb-dev]

I have a bit of concern about this sort of thing - worrying it'll lead to
people being less cautious about writing the more isolated tests. That
said, clearly there's value in end-to-end testing for all the reasons
you've mentioned (& we do see these problems in practice - recently DWARF
indexing broke when support for more nuanced language codes were added to
Clang).

Dunno if they need a new place or should just be more stuff in test-suite,
though.

On Tue, Oct 8, 2019 at 9:50 AM David Greene via cfe-dev <
cfe-...@lists.llvm.org> wrote:

> [ I am initially copying only a few lists since they seem like
>   the most impacted projects and I didn't want to spam all the mailing
>   lists.  Please let me know if other lists should be included. ]
>
> I submitted D68230 for review but this is not about that patch per se.
> The patch allows update_cc_test_checks.py to process tests that should
> check target asm rather than LLVM IR.  We use this facility downstream
> for our end-to-end tests.  It strikes me that it might be useful for
> upstream to do similar end-to-end testing.
>
> Now that the monorepo is about to become the canonical source of truth,
> we have an opportunity for convenient end-to-end testing that we didn't
> easily have before with svn (yes, it could be done but in an ugly way).
> AFAIK the only upstream end-to-end testing we have is in test-suite and
> many of those codes are very large and/or unfocused tests.
>
> With the monorepo we have a place to put lit-style tests that exercise
> multiple subprojects, for example tests that ensure the entire clang
> compilation pipeline executes correctly.  We could, for example, create
> a top-level "test" directory and put end-to-end tests there.  Some of
> the things that could be tested include:
>
> - Pipeline execution (debug-pass=Executions)
> - Optimization warnings/messages
> - Specific asm code sequences out of clang (e.g. ensure certain loops
>   are vectorized)
> - Pragma effects (e.g. ensure loop optimizations are honored)
> - Complete end-to-end PGO (generate a profile and re-compile)
> - GPU/accelerator offloading
> - Debuggability of clang-generated code
>
> Each of these things is tested to some degree within their own
> subprojects, but AFAIK there are currently no dedicated tests ensuring
> such things work through the entire clang pipeline flow and with other
> tools that make use of the results (debuggers, etc.).  It is relatively
> easy to break the pipeline while the individual subproject tests
> continue to pass.
>
> I realize that some folks prefer to work on only a portion of the
> monorepo (for example, they just hack on LLVM).  I am not sure how to
> address those developers WRT end-to-end testing.  On the one hand,
> requiring them to run end-to-end testing means they will have to at
> least check out and build the monorepo.  On the other hand, it seems
> less than ideal to have people developing core infrastructure and not
> running tests.
>
> I don't yet have a formal proposal but wanted to put this out to spur
> discussion and gather feedback and ideas.  Thank you for your interest
> and participation!
>
> -David
> ___
> cfe-dev mailing list
> cfe-...@lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] RFC: End-to-end testing

[via lldb-dev]

> -Original Message-
> From: cfe-dev  On Behalf Of David Greene
> via cfe-dev
> Sent: Tuesday, October 08, 2019 12:50 PM
> To: llvm-...@lists.llvm.org; cfe-...@lists.llvm.org; openmp-
> d...@lists.llvm.org; lldb-dev@lists.llvm.org
> Subject: [cfe-dev] RFC: End-to-end testing
> 
> [ I am initially copying only a few lists since they seem like
>   the most impacted projects and I didn't want to spam all the mailing
>   lists.  Please let me know if other lists should be included. ]
> 
> I submitted D68230 for review but this is not about that patch per se.
> The patch allows update_cc_test_checks.py to process tests that should
> check target asm rather than LLVM IR.  We use this facility downstream
> for our end-to-end tests.  It strikes me that it might be useful for
> upstream to do similar end-to-end testing.
> 
> Now that the monorepo is about to become the canonical source of truth,
> we have an opportunity for convenient end-to-end testing that we didn't
> easily have before with svn (yes, it could be done but in an ugly way).
> AFAIK the only upstream end-to-end testing we have is in test-suite and
> many of those codes are very large and/or unfocused tests.
> 
> With the monorepo we have a place to put lit-style tests that exercise
> multiple subprojects, for example tests that ensure the entire clang
> compilation pipeline executes correctly.  We could, for example, create
> a top-level "test" directory and put end-to-end tests there.  Some of
> the things that could be tested include:
> 
> - Pipeline execution (debug-pass=Executions)
> - Optimization warnings/messages
> - Specific asm code sequences out of clang (e.g. ensure certain loops
>   are vectorized)
> - Pragma effects (e.g. ensure loop optimizations are honored)
> - Complete end-to-end PGO (generate a profile and re-compile)
> - GPU/accelerator offloading
> - Debuggability of clang-generated code
> 
> Each of these things is tested to some degree within their own
> subprojects, but AFAIK there are currently no dedicated tests ensuring
> such things work through the entire clang pipeline flow and with other
> tools that make use of the results (debuggers, etc.).  It is relatively
> easy to break the pipeline while the individual subproject tests
> continue to pass.

I agree with all your points.  End-to-end testing is a major hole in the
project infrastructure; it has been largely left up to the individual
vendors/packagers/distributors.  The Clang tests verify that Clang will
produce some sort of not-unreasonable IR for given situations; the LLVM
tests verify that some (other) set of input IR will produce something
that looks not-unreasonable on the target side.  Very little connects
the two.

There is more than nothing:
- test-suite has some quantity of code that is compiled end-to-end for
  some targets.
- lldb can be set up to use the just-built Clang to compile its tests,
  but those are focused on debug info and are nothing like comprehensive.
- libcxx likely also can use the just-built Clang, although I've never
  tried it so I don't know for sure. It obviously exercises just the
  runtime side of things.
- compiler-rt likewise. The sanitizer tests in particular I'd expect to
  be using the just-built Clang.
- debuginfo-tests also uses the just-built Clang but is a pathetically
  small set, and again focused on debug info.

I'm not saying the LLVM Project should invest in a commercial suite
(although I'd expect vendors to do so; Sony does).  But a place to *put*
end-to-end tests seems entirely reasonable and useful.  Although I would
resist calling it simply "tests" (we have too many directories with that
name already).

> 
> I realize that some folks prefer to work on only a portion of the
> monorepo (for example, they just hack on LLVM).  I am not sure how to
> address those developers WRT end-to-end testing.  On the one hand,
> requiring them to run end-to-end testing means they will have to at
> least check out and build the monorepo.  On the other hand, it seems
> less than ideal to have people developing core infrastructure and not
> running tests.

People should obviously be running the tests for the project(s) they're
modifying.  People aren't expected to run everything.  That's why...

Bots.  "Don't argue with the bots."  I don't checkout and build and test
everything, and I've broken LLDB, compiler-rt, and probably others from
time to time.  Probably everybody has broken other projects unexpectedly.
That's what bots are for, to run the combinations and the projects that 
I don't have the infrastructure or resources to do myself.  It's not up
to me to run everything possible before committing; it IS up to me to
respond promptly to bot failures for my changes.  I don't see a new
end-to-end test project being any different in that respect.

> 
> I don't yet have a formal proposal but wanted to put this out to spur
> discussion and gather feedback and ideas.  Thank you for your interest
> and participation!

Thanks 

[lldb-dev] RFC: End-to-end testing

[David Greene via lldb-dev]

[ I am initially copying only a few lists since they seem like
  the most impacted projects and I didn't want to spam all the mailing
  lists.  Please let me know if other lists should be included. ]

I submitted D68230 for review but this is not about that patch per se.
The patch allows update_cc_test_checks.py to process tests that should
check target asm rather than LLVM IR.  We use this facility downstream
for our end-to-end tests.  It strikes me that it might be useful for
upstream to do similar end-to-end testing.

Now that the monorepo is about to become the canonical source of truth,
we have an opportunity for convenient end-to-end testing that we didn't
easily have before with svn (yes, it could be done but in an ugly way).
AFAIK the only upstream end-to-end testing we have is in test-suite and
many of those codes are very large and/or unfocused tests.

With the monorepo we have a place to put lit-style tests that exercise
multiple subprojects, for example tests that ensure the entire clang
compilation pipeline executes correctly.  We could, for example, create
a top-level "test" directory and put end-to-end tests there.  Some of
the things that could be tested include:

- Pipeline execution (debug-pass=Executions)
- Optimization warnings/messages
- Specific asm code sequences out of clang (e.g. ensure certain loops
  are vectorized)
- Pragma effects (e.g. ensure loop optimizations are honored)
- Complete end-to-end PGO (generate a profile and re-compile)
- GPU/accelerator offloading
- Debuggability of clang-generated code

Each of these things is tested to some degree within their own
subprojects, but AFAIK there are currently no dedicated tests ensuring
such things work through the entire clang pipeline flow and with other
tools that make use of the results (debuggers, etc.).  It is relatively
easy to break the pipeline while the individual subproject tests
continue to pass.

I realize that some folks prefer to work on only a portion of the
monorepo (for example, they just hack on LLVM).  I am not sure how to
address those developers WRT end-to-end testing.  On the one hand,
requiring them to run end-to-end testing means they will have to at
least check out and build the monorepo.  On the other hand, it seems
less than ideal to have people developing core infrastructure and not
running tests.

I don't yet have a formal proposal but wanted to put this out to spur
discussion and gather feedback and ideas.  Thank you for your interest
and participation!

-David
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] [CFP] LLVM devroom at FOSDEM 2020

[Kristof Beyls via lldb-dev]

CALL FOR PAPERS / PARTICIPATION

At FOSDEM 2020, LLVM will again participate with a dedicated devroom, on
Saturday February 1st, in Brussels.

As possibly the largest European Open Source Conference, FOSDEM attracts
more than 600 lectures and over 8000 hackers - many core contributors of
the world’s leading open source projects.

Complementing the LLVM developer meetings, the devroom at FOSDEM provides a
great opportunity for LLVM developers and the wider open source community
to get together, connect and discuss.

We invite academic, industrial and hobbyist speakers to present their work
on developing or using LLVM, Clang, LLDB, Compiler-rt, MLIR, flang, or any
of the other technologies in the LLVM project.

We are looking for:

   -

   Keynote speakers.
   -

   Technical presentations (default length of 40 minutes including
   questions) related to the development of LLVM technologies or use of those
   technologies in other projects.
   -

   Tutorials.
   -

   Lightning talks (default length of 5 minutes).
   -

   Demos.

The deadline for receiving proposals is Sunday November 24th, 2019.

Speakers will be notified of acceptance or rejection by December 13th.
Please find some advice on what constitutes a good proposal at the end of
this CFP.

To submit a proposal, please create an account on the FOSDEM interface (
https://penta.fosdem.org/user/new_account). If you already have an account
from previous years, please reuse that.

Submit your proposal following https://penta.fosdem.org/submission/FOSDEM20,
“Create Event”.

Please make sure you select "LLVM devroom" as the "Track”.

Presentations will be recorded and streamed. Sending your proposal implies
giving permission to be recorded.

Registration

FOSDEM does not require any registration and is free of charge.

Organization

The mailing list llvm-devroom at lists.fosdem.org can be used to discuss
issues of general interest related to the conference organization. Please,
do not reply to this email, as it is cross posted to many lists.

Financial support

There may be a possibility of limited funding to help presenters who could
not otherwise attend the conference.

If you need funding to be able to present at the meeting, or can help
provide sponsorship, please tell us on llvm-devroom at lists.fosdem.org.

Guidance on writing a proposal for the LLVM Dev Room

This is a guide to help you submit a good proposal and increase your
chances of your proposal being accepted.

If you have never presented at an LLVM meeting, then do not fear this
process. We are actively looking for new speakers who are excited about
LLVM and helping grow the community through these educational talks! You do
not need to be a long time developer to submit a proposal.

General Guidelines:

   -

   It should be clear from your abstract what your topic is, who your
   target audience is, and what are the takeaways for attendees. The program
   committee does not have time to read 10 page papers for each submission.
   -

   Talks about the use of an LLVM technology should include details about
   how LLVM is used and not only be about the resulting application.
   -

   Tutorials on “how to use X” in LLVM (or other subproject) are greatly
   desired and beneficial to many developers. Entry level topics are
   encouraged as well.
   -

   Typically a few paragraphs are sufficient.


Best regards,

LLVM @ FOSDEM organisers
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] [RFC] Adding a clang-style LLVM.h (or, "Are you tired of typing 'llvm::' everywhere ?")

[Jan Kratochvil via lldb-dev]

On Tue, 08 Oct 2019 15:42:25 +0200, Pavel Labath wrote:
> On 08/10/2019 10:14, Jan Kratochvil wrote:
> > If I should say something I would keep llvm::.
> >
> > My reason: The LLVM types are in many cases emulating classes adopted
> > in future C++ standards and I find more clear llvm:: vs. std:: than
> > "" vs. std::. Moreover when std:: is commonly omitted in other projects.
> 
> Which classes do you have in mind exactly? I know a lot of llvm *functions*
> mimic similar std:: versions, but I can't think of any *classes* right now.
> thinking here mainly of ADT classes like String/ArrayRef, Optional/Error,

llvm::StringRef vs. std::string, llvm::Optional vs. std::optional.

I do not want to argue whether it is ambiguous or not but you asked for an
+1/-1 opinion.


Jan
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] [RFC] Adding a clang-style LLVM.h (or, "Are you tired of typing 'llvm::' everywhere ?")

[Pavel Labath via lldb-dev]

On 08/10/2019 02:45, Larry D'Anna via lldb-dev wrote:

Pavel Labath said


some llvm classes, are so well-known and widely used, that qualifying
them with "llvm::" serves no useful purpose and only adds visual noise.
I'm thinking here mainly of ADT classes like String/ArrayRef,
Optional/Error, etc. I propose we stop explicitly qualifying these classes.

We can implement this proposal the same way as clang solved the same
problem, which is by creating a special LLVM.h

header in the Utility library. This header would adopt these classes
into the lldb_private namespace via a series of forward and "using"
declarations.

I think clang's LLVM.h is contains a well-balanced collection of adopted
classes, and it should cover the most widely-used classes in lldb too,
so I propose we use that as a starting point.


I think this is a great idea, particularly for llvm::Expected.   The signatures 
of functions
using Expected arer kind of noisy already, and adding llvm:: doesn’t help.

Anyone object to this idea?


I am in still in favour of that. :) I consider the following points to 
be the benefits of this proposal:

- consistency with llvm/clang/lld
- the extra llvm:: qualifications make people want to do away with the 
"cruft" via "auto", which *decreases* consistency with llvm 


- better formatting of code in the 80 columns we have available

On 08/10/2019 10:14, Jan Kratochvil wrote:
> If I should say something I would keep llvm::.
>
> My reason: The LLVM types are in many cases emulating classes adopted
> in future C++ standards and I find more clear llvm:: vs. std:: than
> "" vs. std::. Moreover when std:: is commonly omitted in other projects.


Which classes do you have in mind exactly? I know a lot of llvm 
*functions* mimic similar std:: versions, but I can't think of any 
*classes* right now. I mean StringRef is similar to std::string_view and 
so, but they still differ in the spelling of the base name...


Also, I am not proposing importing any llvm functions this way (and in 
fact, I am against a blanket "using namespace llvm", even in c++ files, 
save for some files with heavy llvm ties). It is true that often these 
llvm functions can be accessed unqualified thanks to ADL, but this 
proposal has nothing to do with that.


pl
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] [RFC] Adding a clang-style LLVM.h (or, "Are you tired of typing 'llvm::' everywhere ?")

[Jan Kratochvil via lldb-dev]

On Wed, 17 Apr 2019 12:39:51 +0200, Pavel Labath via lldb-dev wrote:
> some llvm classes, are so well-known and widely used, that qualifying them
> with "llvm::" serves no useful purpose and only adds visual noise. I'm
> thinking here mainly of ADT classes like String/ArrayRef, Optional/Error,
> etc. I propose we stop explicitly qualifying these classes.
...
> What do you think?

If I should say something I would keep llvm::.

My reason: The LLVM types are in many cases emulating classes adopted
in future C++ standards and I find more clear llvm:: vs. std:: than
"" vs. std::. Moreover when std:: is commonly omitted in other projects.


Jan
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev