[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

alexfh wrote:

I've added my findings with regard to this issue in 
https://github.com/llvm/llvm-project/pull/80442#discussion_r1476243065

Hope that helps understanding the problem.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

alexfh wrote:

I've sent the fix proposed above by @eaeltsin with amendment by @bgra8 as 
https://github.com/llvm/llvm-project/pull/80442. If a different fix is desired, 
please fix quickly or revert until the fix is available.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

bgra8 wrote:

Great find @eaeltsin !
Actually it is clear `BuffStart` and `BuffEnd` are derived from `BuffData`. We 
should just use `BuffData` in that call and all should be fine!

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

eaeltsin wrote:

Ok, the problem is in clang/lib/Frontend/TextDiagnostic.cpp:1352:
```
  std::unique_ptr[]> SourceStyles =
  highlightLines(BufStart, Lines.first, Lines.second, PP, LangOpts,
 DiagOpts->ShowColors, FID, SM);
```

Looks like passing `BufStart` creates a StringRef that doesn't cover all the 
data.

Replacing this to 
```
  std::unique_ptr[]> SourceStyles =
  highlightLines(StringRef(BufStart, BufEnd - BufStart), Lines.first, 
Lines.second, PP, LangOpts,
 DiagOpts->ShowColors, FID, SM);
```
makes the crash go away.

I don't know this code so cannot comment more on what actually happens. The 
only thing to add is that the crash occurred because offset returned in line 
1154 was way out of bounds of buffer created at line 1148.

Please fix.


https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

AaronBallman wrote:

> @AaronBallman
> 
> > Do you see this crash with community clang and no downstream changes?
> 
> Yes I can reproduce it with a vanila `clang` built from a git checkout at 
> this version, but the problem is extremely sensitive to the input headers, 
> their content and defines (`-Dbla`) in the compilation command.

Thank you, that's good to know you hit it with vanilla Clang! I think that 
between this information and the backtrace in #80127 we have enough of a reason 
to revert without a reproducer at hand. However, I think that if we don't have 
a reproducer by mid-next week (say 22:00 UTC on 2/7, but the date/time are 
negotiable) we should re-land the changes as-is to try to get more instances of 
the crash happening in the wild to help us solve the issue. WDYT?

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

bgra8 wrote:

@AaronBallman 

> Do you see this crash with community clang and no downstream changes?

Yes I can reproduce it with a vanila `clang` built  from a git checkout at this 
version, but the problem is extremely sensitive to  the input headers, their 
content and defines (`-Dbla`) in the compilation command.

> But if we get any evidence that other downstreams or users are hitting this 
> or the issue is happening for Google's test case with a vanilla Clang, then I 
> agree that a revert is appropriate until we have a solution.

This was already reported by another user in 
https://github.com/llvm/llvm-project/issues/80127. 

@tbaederr 

> That is, unless 
> https://github.com/llvm/llvm-project/commit/6d1d2c67e7597dc417c097d5027558b0159ed2e2
>  actually fixed the problem already, which would be good to check.

I already mentioned this does not fix the crash.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

AaronBallman wrote:

> This I think qualifies as a serious problem as it introduces a crash which 
> may occur in _any_ existent codebase. Anyone using the compiler at trunk and 
> encountering the crash has no way to know how to deal with it.

I think it's worth remembering that downstreams carry additional changes that 
the community has no visibility into, so the mere presence of a crash does not 
actually identify this change as the culprit without more investigation. The 
community cannot perform that investigation ourselves, we need Google to do 
that from their downstream or we need a reproducer that happens with vanilla 
Clang. Do you see this crash with community clang and no downstream changes?

In the short term (next 24hrs), I think the existing flag to disable color 
diagnostics should get you a path forward; but if you disagree, your downstream 
can certainly revert this patch locally until you're able to provide us with a 
way to reproduce the issue or have at least verified that the issue is not 
caused by changes in your downstream. But if we get any evidence that other 
downstreams or users are hitting this or the issue is happening for Google's 
test case with a vanilla Clang, then I agree that a revert is appropriate until 
we have a solution.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

tbaederr wrote:

> This I think qualifies as a serious problem as it introduces a crash which 
> may occur in _any_ existent codebase. Anyone using the compiler at trunk and 
> encountering the crash has no way to know how to deal with it.

Since we don't have a reproducer, the danger is that we revert the commit but 
never get such a reproducer, leaving us in a state where we can never re-commit 
safely.


If you have some sort of internal deployment, why not revert the commit 
locally? Or just apply something like
```diff
--- a/clang/lib/Frontend/TextDiagnostic.cpp
+++ b/clang/lib/Frontend/TextDiagnostic.cpp
@@ -1350,7 +1350,7 @@ void TextDiagnostic::emitSnippetAndCaret(
   // emit, starting from the first line.
   std::unique_ptr[]> SourceStyles =
   highlightLines(BufStart, Lines.first, Lines.second, PP, LangOpts,
- DiagOpts->ShowColors, FID, SM);
+ false, FID, SM);

   SmallVector LineRanges =
   prepareAndFilterRanges(Ranges, SM, Lines, FID, LangOpts);
```

That is, unless 
https://github.com/llvm/llvm-project/commit/6d1d2c67e7597dc417c097d5027558b0159ed2e2
 actually fixed the problem already, which would be good to check.





https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

bgra8 wrote:

> I hope you realize this is self-imposed limitation that Clang community 
> doesn't necessary have to respect.

Well, the actual LLVM policy is to revert to green: 
https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy

```
When should you revert your own change?

* Any time you learn of a serious problem with a change, you should revert it. 
We strongly encourage “revert to green” as opposed to “fixing forward”. We 
encourage reverting first, investigating offline, and then reapplying the fixed 
patch - possibly after another round of review if warranted.
```

This I think qualifies as a serious problem as it introduces a crash which may 
occur in *any* existent codebase. Anyone using the compiler at trunk and 
encountering the crash has no way to know how to deal with it.



https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Vlad Serebrennikov via cfe-commits]

Endilll wrote:

> We can't push internally a compiler that crashes unless we disable color 
> diagnostics.

I hope you realize this is self-imposed limitation that Clang community doesn't 
necessary have to respect.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

bgra8 wrote:

> I thought we are still talking about internal work being blocked on this, 
> which is much more severe issue.

We can't push internally a compiler that crashes unless we disable color 
diagnostics.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Vlad Serebrennikov via cfe-commits]

Endilll wrote:

> > As mentioned above, `-fno-color-diagnostics` is the flag you can deploy 
> > internally this very moment.
> 
> Disabling color diagnostics would make the job of other software engineers 
> harder, right? This hammer is a bit too big for the problem at hand, right?

I thought we are still talking about internal work being blocked on this, which 
is much more severe issue.


https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

bgra8 wrote:

> As mentioned above, `-fno-color-diagnostics` is the flag you can deploy 
> internally this very moment.

Disabling color diagnostics would make the job of other software engineers 
harder, right? This hammer is a bit too big for the problem at hand, right?

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Vlad Serebrennikov via cfe-commits]

Endilll wrote:

@bgra8 Thank you! This is helpful.

> The problem is extremely hard to reduce as it happens during preprocessing 
> and even removing unrelated headers causes the crash to disappear.

Yeah, crashes happening at this stage can easily be insane to reduce manually.

> This crash is blocking our internal work at Google. I second @eaeltsin's 
> proposal to put the highlightLines() function behind a flag at least until we 
> can get to the bottom of this.

As mentioned above, `-fno-color-diagnostics` is the flag you can deploy 
internally this very moment.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

bgra8 wrote:

https://github.com/llvm/llvm-project/pull/80023 does not fix the problem we're 
seeing here.

The problem is extremely hard to reduce as it happens during preprocessing and 
even removing unrelated headers causes the crash to disappear.

I've built `clang` with assertions enabled and debug information and got a bit 
of more information in the stack trace:

```
..
#6  0x666ce7a4 in __assert_fail ()
#7  0x5e86b604 in highlightLines (FileData=..., StartLineNumber=40, 
EndLineNumber=40, PP=0x68f3f80ec98, LangOpts=..., ShowColors=true, FID=..., 
SM=...) at /work/llvm-project/clang/lib/Frontend/TextDiagnostic.cpp:1156
#8  0x5e86a7d3 in clang::TextDiagnostic::emitSnippetAndCaret 
(this=0x68f3fc7cec0, Loc=..., Level=clang::DiagnosticsEngine::Note, Ranges=..., 
Hints=...) at /work/llvm-project/clang/lib/Frontend/TextDiagnostic.cpp:1352
#9  0x5e86e908 in clang::TextDiagnostic::emitCodeContext 
(this=0x68f3fc7cec0, Loc=..., Level=clang::DiagnosticsEngine::Note, Ranges=..., 
Hints=...) at 
/work/llvm-project/clang/include/clang/Frontend/TextDiagnostic.h:97
#10 0x5e844c4e in clang::DiagnosticRenderer::emitCaret 
(this=0x68f3fc7cec0, Loc=..., Level=clang::DiagnosticsEngine::Note, Ranges=..., 
Hints=...) at /work/llvm-project/clang/lib/Frontend/DiagnosticRenderer.cpp:429
#11 0x5e844482 in clang::DiagnosticRenderer::emitDiagnostic 
(this=0x68f3fc7cec0, Loc=..., Level=clang::DiagnosticsEngine::Note, 
Message=..., Ranges=..., FixItHints=..., D=...) at 
/work/llvm-project/clang/lib/Frontend/DiagnosticRenderer.cpp:127
#12 0x5e84619f in clang::DiagnosticRenderer::emitSingleMacroExpansion 
(this=0x68f3fc7cec0, Loc=..., Level=clang::DiagnosticsEngine::Warning, 
Ranges=...) at /work/llvm-project/clang/lib/Frontend/DiagnosticRenderer.cpp:454
#13 0x5e845044 in clang::DiagnosticRenderer::emitMacroExpansions 
(this=0x68f3fc7cec0, Loc=..., Level=clang::DiagnosticsEngine::Warning, 
Ranges=..., Hints=...) at 
/work/llvm-project/clang/lib/Frontend/DiagnosticRenderer.cpp:569
#14 0x5e84454c in clang::DiagnosticRenderer::emitDiagnostic 
(this=0x68f3fc7cec0, Loc=..., Level=clang::DiagnosticsEngine::Warning, 
Message=..., Ranges=..., FixItHints=..., D=...) at 
/work/llvm-project/clang/lib/Frontend/DiagnosticRenderer.cpp:132
#15 0x5e86866f in clang::TextDiagnosticPrinter::HandleDiagnostic 
(this=0x68f3fc1c900, Level=clang::DiagnosticsEngine::Warning, Info=...) at 
/work/llvm-project/clang/lib/Frontend/TextDiagnosticPrinter.cpp:151
#16 0x61435102 in clang::DiagnosticIDs::EmitDiag (this=0x68f3fc14bb0, 
Diag=..., DiagLevel=clang::DiagnosticIDs::Warning) at 
/work/llvm-project/clang/lib/Basic/DiagnosticIDs.cpp:823
#17 0x61434f44 in clang::DiagnosticIDs::ProcessDiag 
(this=0x68f3fc14bb0, Diag=...) at 
/work/llvm-project/clang/lib/Basic/DiagnosticIDs.cpp:815
#18 0x61429b49 in clang::DiagnosticsEngine::ProcessDiag 
(this=0x68f3f85c400) at 
/work/llvm-project/clang/include/clang/Basic/Diagnostic.h:1042
#19 0x61429ac9 in clang::DiagnosticsEngine::EmitCurrentDiagnostic 
(this=0x68f3f85c400, Force=false) at 
/work/llvm-project/clang/lib/Basic/Diagnostic.cpp:545
...
```

The failed assertion is in this block of code:

```
  const char *FirstLineStart =
  FileData.data() +
  SM.getDecomposedLoc(SM.translateLineCol(FID, StartLineNumber, 1)).second;
  if (const char *CheckPoint = PP->getCheckPoint(FID, FirstLineStart)) {
// THIS IS THE FAILED ASSERTION:
assert(CheckPoint >= Buff->getBufferStart() &&
   CheckPoint <= Buff->getBufferEnd());
assert(CheckPoint <= FirstLineStart);
size_t Offset = CheckPoint - Buff->getBufferStart();
L.seek(Offset, /*IsAtStartOfLine=*/false);
  }
```

The crash happens when the preprocessor outputs a warning related to this code:

```
#ifndef __GTS_H__
#define __GTS_H__

#include 
#include 
// #include "glib.h"

#define G_STRINGIFY(macro_or_string)G_STRINGIFY_ARG (macro_or_string)
#define G_STRINGIFY_ARG(contents)   #contents
#define _GLIB_GNUC_DO_PRAGMA(x) _Pragma(G_STRINGIFY (x))
#define GLIB_DEPRECATED_MACRO_FOR(f) \
  _GLIB_GNUC_DO_PRAGMA(GCC warning G_STRINGIFY (Deprecated pre-processor 
symbol: replace with #f))
#define GLIB_DEPRECATED_MACRO_IN_2_48_FOR(f) GLIB_DEPRECATED_MACRO_FOR (f)
#define G_INLINE_FUNC static inline GLIB_DEPRECATED_MACRO_IN_2_48_FOR(static 
inline)


/* Class declarations for base types */
G_INLINE_FUNC
gpointer gts_object_is_from_class  (gpointer object,
gpointer klass);

#endif /* __GTS_H__ */
```

In the previous code snippet the macro definitions are extracted from the 
`lib.h` header (provided by package `libglib2.0-dev`). Just compiling that does 
not reproduce the crash but it might give you an idea of the code that might 
setup the crash condition. The compiler output just before the crash is:

```
In file included from matrix.c:1:
gts.h:9:1: warning: Deprecated 

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

eaeltsin wrote:

> If you have a solid piece of internal testing blocked by this, can you 
> provide us with more data? Ideally a working reproducer. This would be more 
> productive than just reverting.

Sure, we are working on this and will share if we are successful. But please 
take into account it is often very non-trivial and time consuming to reduce a 
huge piece of internal code to a shareable state (might literally take days of 
work). Thus, having a way to proceed with further testing (== command-line flag 
to disable) in parallel with investigation is IMHO a must.




https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

tbaederr wrote:

I will also merge https://github.com/llvm/llvm-project/pull/80023 once the CI 
is green. That will fix one potential source of problems.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Vlad Serebrennikov via cfe-commits]

Endilll wrote:

> For us, these crashes block a solid piece of internal testing, thus new 
> unnoticed errors accumulate and make things exponentially harder for the 
> future.

If you have a solid piece of internal testing blocked by this, can you provide 
us with more data? Ideally a working reproducer. This would be more productive 
than just reverting.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

tbaederr wrote:

You can diasable it via `-fno-color-diagnostics`.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

eaeltsin wrote:

I don't understand the push back on introducing a command line flag to disable 
- it is needed anyway and doesn't prevent further work on the feature. 
Moreover, I think that all risky features should first appear under the flag, 
especially when not fully baked.

For us, these crashes block a solid piece of internal testing, thus new 
unnoticed errors accumulate and make things exponentially harder for the 
future. By insisting to block until this is resolved you basically block 
reports for other errors which might be very valuable for the authors of the 
new commits.


https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Vlad Serebrennikov via cfe-commits]

Endilll wrote:

@eaeltsin  I'm against reverting this or introducing a switch until we get a 
reproducer (or author figures out what's wrong). I believe it's a very 
worthwhile feature, and we want some actionable feedback to improve it. Author 
has been very responsive so far, so we are good on this side, too.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

eaeltsin wrote:

Though the crashes are hard to reproduce, they are here. 

If you don't want to revert to green while investigating, can you please 
introduce the command line switch to disable this soon? 

Thanks!

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

tbaederr wrote:

Looks a lot like the stack trace mentioned in 
https://github.com/llvm/llvm-project/issues/80127

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

bgra8 wrote:

@tbaederr early heads-up: this patch introduces a `clang` crash.

I don't have yet a reproducer, but here's the stack:

```
 #0 0x558360594cb8 llvm::sys::RunSignalHandlers() (bin/clang+0x8794cb8)
 #1 0x558360f6 CrashRecoverySignalHandler(int) (bin/clang+0x87555f6)
 #2 0x7f119d4f11c0 __restore_rt (/usr/grte/v5/lib64/libpthread.so.0+0x151c0)
 #3 0x55835db72d29 clang::Lexer::SkipWhitespace(clang::Token&, char const*, 
bool&) (bin/clang+0x5d72d29)
 #4 0x55835db77176 clang::Lexer::LexTokenInternal(clang::Token&, bool) 
(bin/clang+0x5d77176)
 #5 0x55835c3a6563 
clang::TextDiagnostic::emitSnippetAndCaret(clang::FullSourceLoc, 
clang::DiagnosticsEngine::Level, 
llvm::SmallVectorImpl&, 
llvm::ArrayRef) (bin/clang+0x45a6563)
 #6 0x55835c391ac8 
clang::DiagnosticRenderer::emitDiagnostic(clang::FullSourceLoc, 
clang::DiagnosticsEngine::Level, llvm::StringRef, 
llvm::ArrayRef, llvm::ArrayRef, 
llvm::PointerUnion) 
(bin/clang+0x4591ac8)
 #7 0x55835c393442 
clang::DiagnosticRenderer::emitSingleMacroExpansion(clang::FullSourceLoc, 
clang::DiagnosticsEngine::Level, llvm::ArrayRef) 
(bin/clang+0x4593442)
 #8 0x55835c392279 
clang::DiagnosticRenderer::emitMacroExpansions(clang::FullSourceLoc, 
clang::DiagnosticsEngine::Level, llvm::ArrayRef, 
llvm::ArrayRef) (bin/clang+0x4592279)
 #9 0x55835c391b13 
clang::DiagnosticRenderer::emitDiagnostic(clang::FullSourceLoc, 
clang::DiagnosticsEngine::Level, llvm::StringRef, 
llvm::ArrayRef, llvm::ArrayRef, 
llvm::PointerUnion) 
(bin/clang+0x4591b13)
#10 0x55835c3a2993 
clang::TextDiagnosticPrinter::HandleDiagnostic(clang::DiagnosticsEngine::Level, 
clang::Diagnostic const&) (bin/clang+0x45a2993)
#11 0x55835dc1dc50 
clang::DiagnosticIDs::ProcessDiag(clang::DiagnosticsEngine&) const 
(bin/clang+0x5e1dc50)
#12 0x55835dc17e98 clang::DiagnosticsEngine::EmitCurrentDiagnostic(bool) 
(bin/clang+0x5e17e98)
#13 0x55835dbd523a (anonymous 
namespace)::PragmaMessageHandler::HandlePragma(clang::Preprocessor&, 
clang::PragmaIntroducer, clang::Token&) (bin/clang+0x5dd523a)
#14 0x55835dbcf953 
clang::Preprocessor::HandlePragmaDirective(clang::PragmaIntroducer) 
(bin/clang+0x5dcf953)
#15 0x55835dbd00e5 clang::Preprocessor::Handle_Pragma(clang::Token&) 
(bin/clang+0x5dd00e5)
#16 0x55835dbc19cd 
clang::Preprocessor::HandleMacroExpandedIdentifier(clang::Token&, 
clang::MacroDefinition const&) (bin/clang+0x5dc19cd)
#17 0x55835dbe38e4 clang::Preprocessor::HandleIdentifier(clang::Token&) 
(bin/clang+0x5de38e4)
#18 0x55835dbe3d8d clang::Preprocessor::Lex(clang::Token&) 
(bin/clang+0x5de3d8d)
#19 0x55835c5aa29d clang::Parser::ConsumeAnyToken(bool) 
(bin/clang+0x47aa29d)
#20 0x55835c6297f4 
clang::Parser::ParseDeclarationSpecifiers(clang::DeclSpec&, 
clang::Parser::ParsedTemplateInfo const&, clang::AccessSpecifier, 
clang::Parser::DeclSpecContext, clang::Parser::LateParsedAttrList*, 
clang::ImplicitTypenameContext) (bin/clang+0x48297f4)
#21 0x55835c5b090f 
clang::Parser::ParseDeclOrFunctionDefInternal(clang::ParsedAttributes&, 
clang::ParsedAttributes&, clang::ParsingDeclSpec&, clang::AccessSpecifier) 
(bin/clang+0x47b090f)
#22 0x55835c5b064e 
clang::Parser::ParseDeclarationOrFunctionDefinition(clang::ParsedAttributes&, 
clang::ParsedAttributes&, clang::ParsingDeclSpec*, clang::AccessSpecifier) 
(bin/clang+0x47b064e)
#23 0x55835c5af7ba 
clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, 
clang::ParsedAttributes&, clang::ParsingDeclSpec*) (bin/clang+0x47af7ba)
#24 0x55835c5ad877 
clang::Parser::ParseTopLevelDecl(clang::OpaquePtr&, 
clang::Sema::ModuleImportState&) (bin/clang+0x47ad877)
#25 0x55835c5a885e clang::ParseAST(clang::Sema&, bool, bool) 
(bin/clang+0x47a885e)
#26 0x55835c2ec5fa clang::FrontendAction::Execute() (bin/clang+0x44ec5fa)
#27 0x55835c25d974 
clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) 
(bin/clang+0x445d974)
#28 0x55835b3b1585 
clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (bin/clang+0x35b1585)
#29 0x55835b3a5334 cc1_main(llvm::ArrayRef, char const*, 
void*) (bin/clang+0x35a5334)
#30 0x55835b3a2e64 ExecuteCC1Tool(llvm::SmallVectorImpl&, 
llvm::ToolContext const&) (bin/clang+0x35a2e64)
#31 0x55835c3f3c9e void llvm::function_ref::callback_fn>,
 std::__u::basic_string, 
std::__u::allocator>*, bool*) const::$_0>(long) (bin/clang+0x45f3c9e)
#32 0x5583605553cf 
llvm::CrashRecoveryContext::RunSafely(llvm::function_ref) 
(bin/clang+0x87553cf)
#33 0x55835c3f343b 
clang::driver::CC1Command::Execute(llvm::ArrayRef>,
 std::__u::basic_string, 
std::__u::allocator>*, bool*) const (bin/clang+0x45f343b)
#34 0x55835c3b2343 
clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, 
clang::driver::Command const*&, bool) const (bin/clang+0x45b2343)
#35 0x55835c3b288f 
clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, 
llvm::SmallVectorImpl>&, 
bool) const 

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

tbaederr wrote:

Here's gnome-terminal's default colors, first normal and then bright:
![Screenshot from 2024-01-30 
14-55-37](https://github.com/llvm/llvm-project/assets/49720664/9f225494-0cc9-4d53-a0e1-98b45b5b4c4b)

Here the same with xterm:
![Screenshot from 2024-01-30 
14-58-24](https://github.com/llvm/llvm-project/assets/49720664/b7e03a49-eacd-4fc6-bb30-b703df9dfceb)


This is Konsole:
![Screenshot from 2024-01-30 
15-01-06](https://github.com/llvm/llvm-project/assets/49720664/988e9a64-5899-4f23-baaf-c7961766ad4f)
 (I checked in an image editor and the blue used for `static_cast` is indeed 
every so slightly different)

And here is Konsole with its "Black on light yellow" color scheme:
![Screenshot from 2024-01-30 
15-05-16](https://github.com/llvm/llvm-project/assets/49720664/8ee801e0-5ce6-4047-8ca3-b4158d2a3068)





https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

tbaederr wrote:

Nice, thanks!

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Vlad Serebrennikov via cfe-commits]

Endilll wrote:

I also got in touch with Compiler Explorer devs. They should be able to pick a 
better color to render ANSI blue.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

tbaederr wrote:

Yeah, I read that too when first working on this. Let me try some stuff.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Vlad Serebrennikov via cfe-commits]

Endilll wrote:

Ah, I see where the limitation comes from.
According to 
[Wikipedia](https://en.wikipedia.org/wiki/ANSI_escape_code#3-bit_and_4-bit), 
there is bright blue color which many terminals display differently from 
regular blue. While this means we're expanding 3-bit color palette to 4-bit 
color palette, I'm not sure how much of a compatibility issue this is in 
practice.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

tbaederr wrote:

> But it's a bit off the point. We should pick colors that have good contrast 
> against both light and dark color themes by default, and not ask everyone to 
> change their color palette. Also make sure to zoom out when you look at this 
> yourself: issue is more apparent this way.

Of course, but a color theme should make sure that blue text is readable.

We have a _very_ limited set of colors available in `llvm::raw_ostream`. I 
could imagine yellow working better, but I have no idea if yellow on white will 
be significantly better to read on compiler-explorer.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Vlad Serebrennikov via cfe-commits]

Endilll wrote:

It looks a tinybit better in my local terminal, but still significantly harder 
to read than the rest of the text:
![image](https://github.com/llvm/llvm-project/assets/12883766/5b0136a5-012b-47f5-b888-787d1c060d54)

But it's a bit off the point. We should pick colors that have good contrast 
against both light and dark color themes by default, and not ask everyone to 
change their color palette. Also make sure to zoom out when you look at this 
yourself: issue is more apparent this way.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

tbaederr wrote:

Does it look that bad in your local terminal as well?

It's okay-ish with the default colors here:
![Screenshot from 2024-01-30 
09-48-38](https://github.com/llvm/llvm-project/assets/49720664/ca6fb7d1-d729-4fea-b0fd-0385b0e5fba2)

We might have to file a bug against compiler-explorer.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Vlad Serebrennikov via cfe-commits]

Endilll wrote:

The deep shade of blue doesn't make sense for both light and dark backgrounds:
![image](https://github.com/llvm/llvm-project/assets/12883766/e9ee6658-1ab1-422f-914b-5f95cac3846d)
![image](https://github.com/llvm/llvm-project/assets/12883766/986ae6f4-2fa9-4d2a-b661-47de3bf92cde)

It's too dark on a light background to be easily confused with black, and dark 
enough on a dark background to easily blend in.

Note that screenshots are intentionally big, so that diagnostics text is small 
enough to represent how it looks in the context. Isolated screenshots of 
diagnostic text (like ones posted in this thread earlier) make it too big, 
alleviating the issue.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

https://github.com/tbaederr closed 
https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

cor3ntin wrote:

I have confirmed that the libcxx issues are unrelated. Feel free to merge :)

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

cor3ntin wrote:

> Don't think I can get the Windows+libcxx pre-commit CI working :|
> 
> `libcxx/gdb/gdb_pretty_printer_test.sh.cpp` has `UNSUPPORTED` lines for clang 
> 16-18 but 19 is missing now. Not sure how this works for others.

We have this issue on other PRs, I think we can ignore. @ldionne 

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

tbaederr wrote:

Don't think I can get the Windows+libcxx pre-commit CI working :|

`libcxx/gdb/gdb_pretty_printer_test.sh.cpp` has `UNSUPPORTED` lines for clang 
16-18 but 19 is missing now. Not sure how this works for others.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 5014d98b1893225ebdcc1e9ce0941b348f504740 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 clang/docs/ReleaseNotes.rst   |   2 +
 clang/include/clang/Frontend/TextDiagnostic.h |  18 +-
 clang/include/clang/Lex/Preprocessor.h|  10 +
 clang/lib/Frontend/TextDiagnostic.cpp | 211 +-
 clang/lib/Frontend/TextDiagnosticPrinter.cpp  |   2 +-
 clang/lib/Lex/Preprocessor.cpp|  24 ++
 clang/test/Frontend/diagnostic-pipe.c |   9 +
 7 files changed, 262 insertions(+), 14 deletions(-)
 create mode 100644 clang/test/Frontend/diagnostic-pipe.c

diff --git a/clang/docs/ReleaseNotes.rst b/clang/docs/ReleaseNotes.rst
index 5330cd9caad801..66207fad313d1d 100644
--- a/clang/docs/ReleaseNotes.rst
+++ b/clang/docs/ReleaseNotes.rst
@@ -97,6 +97,8 @@ Attribute Changes in Clang
 
 Improvements to Clang's diagnostics
 ---
+- Clang now applies syntax highlighting to the code snippets it
+  prints.
 
 Improvements to Clang's time-trace
 --
diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca..a2fe8ae995423b 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -16,6 +16,7 @@
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
 #include "clang/Frontend/DiagnosticRenderer.h"
+#include "llvm/Support/raw_ostream.h"
 
 namespace clang {
 
@@ -33,14 +34,22 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  const Preprocessor *PP;
 
 public:
-  TextDiagnostic(raw_ostream ,
- const LangOptions ,
- DiagnosticOptions *DiagOpts);
+  TextDiagnostic(raw_ostream , const LangOptions ,
+ DiagnosticOptions *DiagOpts, const Preprocessor *PP = 
nullptr);
 
   ~TextDiagnostic() override;
 
+  struct StyleRange {
+unsigned Start;
+unsigned End;
+enum llvm::raw_ostream::Colors Color;
+StyleRange(unsigned S, unsigned E, enum llvm::raw_ostream::Colors C)
+: Start(S), End(E), Color(C){};
+  };
+
   /// Print the diagonstic level to a raw_ostream.
   ///
   /// This is a static helper that handles colorizing the level and formatting
@@ -104,7 +113,8 @@ class TextDiagnostic : public DiagnosticRenderer {
ArrayRef Hints);
 
   void emitSnippet(StringRef SourceLine, unsigned MaxLineNoDisplayWidth,
-   unsigned LineNo);
+   unsigned LineNo, unsigned DisplayLineNo,
+   ArrayRef Styles);
 
   void emitParseableFixits(ArrayRef Hints, const SourceManager );
 };
diff --git a/clang/include/clang/Lex/Preprocessor.h 
b/clang/include/clang/Lex/Preprocessor.h
index 2d9c53cdf5bde8..9d0d53129a12dd 100644
--- a/clang/include/clang/Lex/Preprocessor.h
+++ b/clang/include/clang/Lex/Preprocessor.h
@@ -284,6 +284,13 @@ class Preprocessor {
   /// The kind of translation unit we are processing.
   const TranslationUnitKind TUKind;
 
+  /// Returns a pointer into the given file's buffer that's guaranteed
+  /// to be between tokens. The returned pointer is always before \p Start.
+  /// The maximum distance betweenthe returned pointer and \p Start is
+  /// limited by a constant value, but also an implementation detail.
+  /// If no such check point exists, \c nullptr is returned.
+  const char *getCheckPoint(FileID FID, const char *Start) const;
+
 private:
   /// The code-completion handler.
   CodeCompletionHandler *CodeComplete = nullptr;
@@ -311,6 +318,9 @@ class Preprocessor {
   /// The import path for named module that we're currently processing.
   SmallVector, 2> 
NamedModuleImportPath;
 
+  llvm::DenseMap> CheckPoints;
+  unsigned CheckPointCounter = 0;
+
   /// Whether the import is an `@import` or a standard c++ modules import.
   bool IsAtImport = false;
 
diff --git a/clang/lib/Frontend/TextDiagnostic.cpp 
b/clang/lib/Frontend/TextDiagnostic.cpp
index 779dead5d058d1..291d71f6db61f1 100644
--- a/clang/lib/Frontend/TextDiagnostic.cpp
+++ b/clang/lib/Frontend/TextDiagnostic.cpp
@@ -12,6 +12,7 @@
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/ConvertUTF.h"
@@ -41,6 +42,14 @@ static const enum raw_ostream::Colors fatalColor = 
raw_ostream::RED;
 static const enum raw_ostream::Colors savedColor =
   raw_ostream::SAVEDCOLOR;
 
+// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
+// is already taken for 'note'. 

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 29c174e595dd839cf96ca54778712ab3bc543103 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 clang/docs/ReleaseNotes.rst   |   2 +
 clang/include/clang/Frontend/TextDiagnostic.h |  18 +-
 clang/include/clang/Lex/Preprocessor.h|  10 +
 clang/lib/Frontend/TextDiagnostic.cpp | 211 +-
 clang/lib/Frontend/TextDiagnosticPrinter.cpp  |   2 +-
 clang/lib/Lex/Preprocessor.cpp|  24 ++
 clang/test/Frontend/diagnostic-pipe.c |   9 +
 7 files changed, 262 insertions(+), 14 deletions(-)
 create mode 100644 clang/test/Frontend/diagnostic-pipe.c

diff --git a/clang/docs/ReleaseNotes.rst b/clang/docs/ReleaseNotes.rst
index db3d74e124e7d1d..6f5fe2049e242fc 100644
--- a/clang/docs/ReleaseNotes.rst
+++ b/clang/docs/ReleaseNotes.rst
@@ -97,6 +97,8 @@ Attribute Changes in Clang
 
 Improvements to Clang's diagnostics
 ---
+- Clang now applies syntax highlighting to the code snippets it
+  prints.
 
 Improvements to Clang's time-trace
 --
diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca8..a2fe8ae995423b9 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -16,6 +16,7 @@
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
 #include "clang/Frontend/DiagnosticRenderer.h"
+#include "llvm/Support/raw_ostream.h"
 
 namespace clang {
 
@@ -33,14 +34,22 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  const Preprocessor *PP;
 
 public:
-  TextDiagnostic(raw_ostream ,
- const LangOptions ,
- DiagnosticOptions *DiagOpts);
+  TextDiagnostic(raw_ostream , const LangOptions ,
+ DiagnosticOptions *DiagOpts, const Preprocessor *PP = 
nullptr);
 
   ~TextDiagnostic() override;
 
+  struct StyleRange {
+unsigned Start;
+unsigned End;
+enum llvm::raw_ostream::Colors Color;
+StyleRange(unsigned S, unsigned E, enum llvm::raw_ostream::Colors C)
+: Start(S), End(E), Color(C){};
+  };
+
   /// Print the diagonstic level to a raw_ostream.
   ///
   /// This is a static helper that handles colorizing the level and formatting
@@ -104,7 +113,8 @@ class TextDiagnostic : public DiagnosticRenderer {
ArrayRef Hints);
 
   void emitSnippet(StringRef SourceLine, unsigned MaxLineNoDisplayWidth,
-   unsigned LineNo);
+   unsigned LineNo, unsigned DisplayLineNo,
+   ArrayRef Styles);
 
   void emitParseableFixits(ArrayRef Hints, const SourceManager );
 };
diff --git a/clang/include/clang/Lex/Preprocessor.h 
b/clang/include/clang/Lex/Preprocessor.h
index 2d9c53cdf5bde8e..9d0d53129a12dd9 100644
--- a/clang/include/clang/Lex/Preprocessor.h
+++ b/clang/include/clang/Lex/Preprocessor.h
@@ -284,6 +284,13 @@ class Preprocessor {
   /// The kind of translation unit we are processing.
   const TranslationUnitKind TUKind;
 
+  /// Returns a pointer into the given file's buffer that's guaranteed
+  /// to be between tokens. The returned pointer is always before \p Start.
+  /// The maximum distance betweenthe returned pointer and \p Start is
+  /// limited by a constant value, but also an implementation detail.
+  /// If no such check point exists, \c nullptr is returned.
+  const char *getCheckPoint(FileID FID, const char *Start) const;
+
 private:
   /// The code-completion handler.
   CodeCompletionHandler *CodeComplete = nullptr;
@@ -311,6 +318,9 @@ class Preprocessor {
   /// The import path for named module that we're currently processing.
   SmallVector, 2> 
NamedModuleImportPath;
 
+  llvm::DenseMap> CheckPoints;
+  unsigned CheckPointCounter = 0;
+
   /// Whether the import is an `@import` or a standard c++ modules import.
   bool IsAtImport = false;
 
diff --git a/clang/lib/Frontend/TextDiagnostic.cpp 
b/clang/lib/Frontend/TextDiagnostic.cpp
index 779dead5d058d1a..291d71f6db61f17 100644
--- a/clang/lib/Frontend/TextDiagnostic.cpp
+++ b/clang/lib/Frontend/TextDiagnostic.cpp
@@ -12,6 +12,7 @@
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/ConvertUTF.h"
@@ -41,6 +42,14 @@ static const enum raw_ostream::Colors fatalColor = 
raw_ostream::RED;
 static const enum raw_ostream::Colors savedColor =
   raw_ostream::SAVEDCOLOR;
 
+// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
+// is already taken for 

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 2a0b02a285203228944d5dd206f968e776757993 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 clang/docs/ReleaseNotes.rst   |   2 +
 clang/include/clang/Frontend/TextDiagnostic.h |  18 +-
 clang/include/clang/Lex/Preprocessor.h|  10 +
 clang/lib/Frontend/TextDiagnostic.cpp | 211 +-
 clang/lib/Frontend/TextDiagnosticPrinter.cpp  |   2 +-
 clang/lib/Lex/Preprocessor.cpp|  24 ++
 clang/test/Frontend/diagnostic-pipe.c |   9 +
 7 files changed, 262 insertions(+), 14 deletions(-)
 create mode 100644 clang/test/Frontend/diagnostic-pipe.c

diff --git a/clang/docs/ReleaseNotes.rst b/clang/docs/ReleaseNotes.rst
index db3d74e124e7d1d..6f5fe2049e242fc 100644
--- a/clang/docs/ReleaseNotes.rst
+++ b/clang/docs/ReleaseNotes.rst
@@ -97,6 +97,8 @@ Attribute Changes in Clang
 
 Improvements to Clang's diagnostics
 ---
+- Clang now applies syntax highlighting to the code snippets it
+  prints.
 
 Improvements to Clang's time-trace
 --
diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca8..a2fe8ae995423b9 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -16,6 +16,7 @@
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
 #include "clang/Frontend/DiagnosticRenderer.h"
+#include "llvm/Support/raw_ostream.h"
 
 namespace clang {
 
@@ -33,14 +34,22 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  const Preprocessor *PP;
 
 public:
-  TextDiagnostic(raw_ostream ,
- const LangOptions ,
- DiagnosticOptions *DiagOpts);
+  TextDiagnostic(raw_ostream , const LangOptions ,
+ DiagnosticOptions *DiagOpts, const Preprocessor *PP = 
nullptr);
 
   ~TextDiagnostic() override;
 
+  struct StyleRange {
+unsigned Start;
+unsigned End;
+enum llvm::raw_ostream::Colors Color;
+StyleRange(unsigned S, unsigned E, enum llvm::raw_ostream::Colors C)
+: Start(S), End(E), Color(C){};
+  };
+
   /// Print the diagonstic level to a raw_ostream.
   ///
   /// This is a static helper that handles colorizing the level and formatting
@@ -104,7 +113,8 @@ class TextDiagnostic : public DiagnosticRenderer {
ArrayRef Hints);
 
   void emitSnippet(StringRef SourceLine, unsigned MaxLineNoDisplayWidth,
-   unsigned LineNo);
+   unsigned LineNo, unsigned DisplayLineNo,
+   ArrayRef Styles);
 
   void emitParseableFixits(ArrayRef Hints, const SourceManager );
 };
diff --git a/clang/include/clang/Lex/Preprocessor.h 
b/clang/include/clang/Lex/Preprocessor.h
index 2d9c53cdf5bde8e..9d0d53129a12dd9 100644
--- a/clang/include/clang/Lex/Preprocessor.h
+++ b/clang/include/clang/Lex/Preprocessor.h
@@ -284,6 +284,13 @@ class Preprocessor {
   /// The kind of translation unit we are processing.
   const TranslationUnitKind TUKind;
 
+  /// Returns a pointer into the given file's buffer that's guaranteed
+  /// to be between tokens. The returned pointer is always before \p Start.
+  /// The maximum distance betweenthe returned pointer and \p Start is
+  /// limited by a constant value, but also an implementation detail.
+  /// If no such check point exists, \c nullptr is returned.
+  const char *getCheckPoint(FileID FID, const char *Start) const;
+
 private:
   /// The code-completion handler.
   CodeCompletionHandler *CodeComplete = nullptr;
@@ -311,6 +318,9 @@ class Preprocessor {
   /// The import path for named module that we're currently processing.
   SmallVector, 2> 
NamedModuleImportPath;
 
+  llvm::DenseMap> CheckPoints;
+  unsigned CheckPointCounter = 0;
+
   /// Whether the import is an `@import` or a standard c++ modules import.
   bool IsAtImport = false;
 
diff --git a/clang/lib/Frontend/TextDiagnostic.cpp 
b/clang/lib/Frontend/TextDiagnostic.cpp
index 779dead5d058d1a..291d71f6db61f17 100644
--- a/clang/lib/Frontend/TextDiagnostic.cpp
+++ b/clang/lib/Frontend/TextDiagnostic.cpp
@@ -12,6 +12,7 @@
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/ConvertUTF.h"
@@ -41,6 +42,14 @@ static const enum raw_ostream::Colors fatalColor = 
raw_ostream::RED;
 static const enum raw_ostream::Colors savedColor =
   raw_ostream::SAVEDCOLOR;
 
+// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
+// is already taken for 

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

https://github.com/cor3ntin approved this pull request.

I am happy with that!
Lets merge it soon so that we can play with it :)

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 69aa9b23a7d5afb58b89a703e7755924561dd54f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 clang/docs/ReleaseNotes.rst   |   2 +
 clang/include/clang/Frontend/TextDiagnostic.h |  18 +-
 clang/include/clang/Lex/Preprocessor.h|  10 +
 clang/lib/Frontend/TextDiagnostic.cpp | 211 +-
 clang/lib/Frontend/TextDiagnosticPrinter.cpp  |   2 +-
 clang/lib/Lex/Preprocessor.cpp|  24 ++
 clang/test/Frontend/diagnostic-pipe.c |  16 ++
 7 files changed, 269 insertions(+), 14 deletions(-)
 create mode 100644 clang/test/Frontend/diagnostic-pipe.c

diff --git a/clang/docs/ReleaseNotes.rst b/clang/docs/ReleaseNotes.rst
index db3d74e124e7d1d..6f5fe2049e242fc 100644
--- a/clang/docs/ReleaseNotes.rst
+++ b/clang/docs/ReleaseNotes.rst
@@ -97,6 +97,8 @@ Attribute Changes in Clang
 
 Improvements to Clang's diagnostics
 ---
+- Clang now applies syntax highlighting to the code snippets it
+  prints.
 
 Improvements to Clang's time-trace
 --
diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca8..a2fe8ae995423b9 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -16,6 +16,7 @@
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
 #include "clang/Frontend/DiagnosticRenderer.h"
+#include "llvm/Support/raw_ostream.h"
 
 namespace clang {
 
@@ -33,14 +34,22 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  const Preprocessor *PP;
 
 public:
-  TextDiagnostic(raw_ostream ,
- const LangOptions ,
- DiagnosticOptions *DiagOpts);
+  TextDiagnostic(raw_ostream , const LangOptions ,
+ DiagnosticOptions *DiagOpts, const Preprocessor *PP = 
nullptr);
 
   ~TextDiagnostic() override;
 
+  struct StyleRange {
+unsigned Start;
+unsigned End;
+enum llvm::raw_ostream::Colors Color;
+StyleRange(unsigned S, unsigned E, enum llvm::raw_ostream::Colors C)
+: Start(S), End(E), Color(C){};
+  };
+
   /// Print the diagonstic level to a raw_ostream.
   ///
   /// This is a static helper that handles colorizing the level and formatting
@@ -104,7 +113,8 @@ class TextDiagnostic : public DiagnosticRenderer {
ArrayRef Hints);
 
   void emitSnippet(StringRef SourceLine, unsigned MaxLineNoDisplayWidth,
-   unsigned LineNo);
+   unsigned LineNo, unsigned DisplayLineNo,
+   ArrayRef Styles);
 
   void emitParseableFixits(ArrayRef Hints, const SourceManager );
 };
diff --git a/clang/include/clang/Lex/Preprocessor.h 
b/clang/include/clang/Lex/Preprocessor.h
index 2d9c53cdf5bde8e..9d0d53129a12dd9 100644
--- a/clang/include/clang/Lex/Preprocessor.h
+++ b/clang/include/clang/Lex/Preprocessor.h
@@ -284,6 +284,13 @@ class Preprocessor {
   /// The kind of translation unit we are processing.
   const TranslationUnitKind TUKind;
 
+  /// Returns a pointer into the given file's buffer that's guaranteed
+  /// to be between tokens. The returned pointer is always before \p Start.
+  /// The maximum distance betweenthe returned pointer and \p Start is
+  /// limited by a constant value, but also an implementation detail.
+  /// If no such check point exists, \c nullptr is returned.
+  const char *getCheckPoint(FileID FID, const char *Start) const;
+
 private:
   /// The code-completion handler.
   CodeCompletionHandler *CodeComplete = nullptr;
@@ -311,6 +318,9 @@ class Preprocessor {
   /// The import path for named module that we're currently processing.
   SmallVector, 2> 
NamedModuleImportPath;
 
+  llvm::DenseMap> CheckPoints;
+  unsigned CheckPointCounter = 0;
+
   /// Whether the import is an `@import` or a standard c++ modules import.
   bool IsAtImport = false;
 
diff --git a/clang/lib/Frontend/TextDiagnostic.cpp 
b/clang/lib/Frontend/TextDiagnostic.cpp
index 779dead5d058d1a..291d71f6db61f17 100644
--- a/clang/lib/Frontend/TextDiagnostic.cpp
+++ b/clang/lib/Frontend/TextDiagnostic.cpp
@@ -12,6 +12,7 @@
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/ConvertUTF.h"
@@ -41,6 +42,14 @@ static const enum raw_ostream::Colors fatalColor = 
raw_ostream::RED;
 static const enum raw_ostream::Colors savedColor =
   raw_ostream::SAVEDCOLOR;
 
+// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
+// is already taken for 

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= 
Message-ID:
In-Reply-To: 


https://github.com/AaronBallman approved this pull request.

LGTM but please do not land until after the Clang 18 branch next Tuesday.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 41bb52dc591ce1c154a4272abb738c80a7a57b90 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH 1/5] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 clang/include/clang/Frontend/TextDiagnostic.h |  18 +-
 clang/include/clang/Lex/Preprocessor.h|   5 +
 clang/lib/Frontend/TextDiagnostic.cpp | 197 +-
 clang/lib/Frontend/TextDiagnosticPrinter.cpp  |   2 +-
 clang/lib/Lex/Preprocessor.cpp|  24 +++
 5 files changed, 232 insertions(+), 14 deletions(-)

diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca..a2fe8ae995423b 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -16,6 +16,7 @@
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
 #include "clang/Frontend/DiagnosticRenderer.h"
+#include "llvm/Support/raw_ostream.h"
 
 namespace clang {
 
@@ -33,14 +34,22 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  const Preprocessor *PP;
 
 public:
-  TextDiagnostic(raw_ostream ,
- const LangOptions ,
- DiagnosticOptions *DiagOpts);
+  TextDiagnostic(raw_ostream , const LangOptions ,
+ DiagnosticOptions *DiagOpts, const Preprocessor *PP = 
nullptr);
 
   ~TextDiagnostic() override;
 
+  struct StyleRange {
+unsigned Start;
+unsigned End;
+enum llvm::raw_ostream::Colors Color;
+StyleRange(unsigned S, unsigned E, enum llvm::raw_ostream::Colors C)
+: Start(S), End(E), Color(C){};
+  };
+
   /// Print the diagonstic level to a raw_ostream.
   ///
   /// This is a static helper that handles colorizing the level and formatting
@@ -104,7 +113,8 @@ class TextDiagnostic : public DiagnosticRenderer {
ArrayRef Hints);
 
   void emitSnippet(StringRef SourceLine, unsigned MaxLineNoDisplayWidth,
-   unsigned LineNo);
+   unsigned LineNo, unsigned DisplayLineNo,
+   ArrayRef Styles);
 
   void emitParseableFixits(ArrayRef Hints, const SourceManager );
 };
diff --git a/clang/include/clang/Lex/Preprocessor.h 
b/clang/include/clang/Lex/Preprocessor.h
index 4ec21a8b6be2c8..d89e2be1bf5ff5 100644
--- a/clang/include/clang/Lex/Preprocessor.h
+++ b/clang/include/clang/Lex/Preprocessor.h
@@ -284,6 +284,8 @@ class Preprocessor {
   /// The kind of translation unit we are processing.
   const TranslationUnitKind TUKind;
 
+  const char *getCheckPoint(FileID FID, const char *Start) const;
+
 private:
   /// The code-completion handler.
   CodeCompletionHandler *CodeComplete = nullptr;
@@ -311,6 +313,9 @@ class Preprocessor {
   /// The import path for named module that we're currently processing.
   SmallVector, 2> 
NamedModuleImportPath;
 
+  llvm::DenseMap> CheckPoints;
+  unsigned CheckPointCounter = 0;
+
   /// Whether the import is an `@import` or a standard c++ modules import.
   bool IsAtImport = false;
 
diff --git a/clang/lib/Frontend/TextDiagnostic.cpp 
b/clang/lib/Frontend/TextDiagnostic.cpp
index 779dead5d058d1..691809ceded87b 100644
--- a/clang/lib/Frontend/TextDiagnostic.cpp
+++ b/clang/lib/Frontend/TextDiagnostic.cpp
@@ -12,6 +12,7 @@
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/ConvertUTF.h"
@@ -41,6 +42,14 @@ static const enum raw_ostream::Colors fatalColor = 
raw_ostream::RED;
 static const enum raw_ostream::Colors savedColor =
   raw_ostream::SAVEDCOLOR;
 
+// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
+// is already taken for 'note'. Green is already used to underline
+// source ranges. White and black are bad because of the usual
+// terminal backgrounds. Which leaves us only with TWO options.
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::YELLOW;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+
 /// Add highlights to differences in template strings.
 static void applyTemplateHighlighting(raw_ostream , StringRef Str,
   bool , bool Bold) {
@@ -644,10 +653,10 @@ static bool printWordWrapped(raw_ostream , StringRef 
Str, unsigned Columns,
   return Wrapped;
 }
 
-TextDiagnostic::TextDiagnostic(raw_ostream ,
-   const LangOptions ,
-   DiagnosticOptions 

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 41bb52dc591ce1c154a4272abb738c80a7a57b90 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH 1/4] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 clang/include/clang/Frontend/TextDiagnostic.h |  18 +-
 clang/include/clang/Lex/Preprocessor.h|   5 +
 clang/lib/Frontend/TextDiagnostic.cpp | 197 +-
 clang/lib/Frontend/TextDiagnosticPrinter.cpp  |   2 +-
 clang/lib/Lex/Preprocessor.cpp|  24 +++
 5 files changed, 232 insertions(+), 14 deletions(-)

diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca8..a2fe8ae995423b9 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -16,6 +16,7 @@
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
 #include "clang/Frontend/DiagnosticRenderer.h"
+#include "llvm/Support/raw_ostream.h"
 
 namespace clang {
 
@@ -33,14 +34,22 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  const Preprocessor *PP;
 
 public:
-  TextDiagnostic(raw_ostream ,
- const LangOptions ,
- DiagnosticOptions *DiagOpts);
+  TextDiagnostic(raw_ostream , const LangOptions ,
+ DiagnosticOptions *DiagOpts, const Preprocessor *PP = 
nullptr);
 
   ~TextDiagnostic() override;
 
+  struct StyleRange {
+unsigned Start;
+unsigned End;
+enum llvm::raw_ostream::Colors Color;
+StyleRange(unsigned S, unsigned E, enum llvm::raw_ostream::Colors C)
+: Start(S), End(E), Color(C){};
+  };
+
   /// Print the diagonstic level to a raw_ostream.
   ///
   /// This is a static helper that handles colorizing the level and formatting
@@ -104,7 +113,8 @@ class TextDiagnostic : public DiagnosticRenderer {
ArrayRef Hints);
 
   void emitSnippet(StringRef SourceLine, unsigned MaxLineNoDisplayWidth,
-   unsigned LineNo);
+   unsigned LineNo, unsigned DisplayLineNo,
+   ArrayRef Styles);
 
   void emitParseableFixits(ArrayRef Hints, const SourceManager );
 };
diff --git a/clang/include/clang/Lex/Preprocessor.h 
b/clang/include/clang/Lex/Preprocessor.h
index 4ec21a8b6be2c85..d89e2be1bf5ff55 100644
--- a/clang/include/clang/Lex/Preprocessor.h
+++ b/clang/include/clang/Lex/Preprocessor.h
@@ -284,6 +284,8 @@ class Preprocessor {
   /// The kind of translation unit we are processing.
   const TranslationUnitKind TUKind;
 
+  const char *getCheckPoint(FileID FID, const char *Start) const;
+
 private:
   /// The code-completion handler.
   CodeCompletionHandler *CodeComplete = nullptr;
@@ -311,6 +313,9 @@ class Preprocessor {
   /// The import path for named module that we're currently processing.
   SmallVector, 2> 
NamedModuleImportPath;
 
+  llvm::DenseMap> CheckPoints;
+  unsigned CheckPointCounter = 0;
+
   /// Whether the import is an `@import` or a standard c++ modules import.
   bool IsAtImport = false;
 
diff --git a/clang/lib/Frontend/TextDiagnostic.cpp 
b/clang/lib/Frontend/TextDiagnostic.cpp
index 779dead5d058d1a..691809ceded87bf 100644
--- a/clang/lib/Frontend/TextDiagnostic.cpp
+++ b/clang/lib/Frontend/TextDiagnostic.cpp
@@ -12,6 +12,7 @@
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/ConvertUTF.h"
@@ -41,6 +42,14 @@ static const enum raw_ostream::Colors fatalColor = 
raw_ostream::RED;
 static const enum raw_ostream::Colors savedColor =
   raw_ostream::SAVEDCOLOR;
 
+// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
+// is already taken for 'note'. Green is already used to underline
+// source ranges. White and black are bad because of the usual
+// terminal backgrounds. Which leaves us only with TWO options.
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::YELLOW;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+
 /// Add highlights to differences in template strings.
 static void applyTemplateHighlighting(raw_ostream , StringRef Str,
   bool , bool Bold) {
@@ -644,10 +653,10 @@ static bool printWordWrapped(raw_ostream , StringRef 
Str, unsigned Columns,
   return Wrapped;
 }
 
-TextDiagnostic::TextDiagnostic(raw_ostream ,
-   const LangOptions ,
-   DiagnosticOptions *DiagOpts)
-  : 

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 41bb52dc591ce1c154a4272abb738c80a7a57b90 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH 1/3] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 clang/include/clang/Frontend/TextDiagnostic.h |  18 +-
 clang/include/clang/Lex/Preprocessor.h|   5 +
 clang/lib/Frontend/TextDiagnostic.cpp | 197 +-
 clang/lib/Frontend/TextDiagnosticPrinter.cpp  |   2 +-
 clang/lib/Lex/Preprocessor.cpp|  24 +++
 5 files changed, 232 insertions(+), 14 deletions(-)

diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca..a2fe8ae995423b 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -16,6 +16,7 @@
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
 #include "clang/Frontend/DiagnosticRenderer.h"
+#include "llvm/Support/raw_ostream.h"
 
 namespace clang {
 
@@ -33,14 +34,22 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  const Preprocessor *PP;
 
 public:
-  TextDiagnostic(raw_ostream ,
- const LangOptions ,
- DiagnosticOptions *DiagOpts);
+  TextDiagnostic(raw_ostream , const LangOptions ,
+ DiagnosticOptions *DiagOpts, const Preprocessor *PP = 
nullptr);
 
   ~TextDiagnostic() override;
 
+  struct StyleRange {
+unsigned Start;
+unsigned End;
+enum llvm::raw_ostream::Colors Color;
+StyleRange(unsigned S, unsigned E, enum llvm::raw_ostream::Colors C)
+: Start(S), End(E), Color(C){};
+  };
+
   /// Print the diagonstic level to a raw_ostream.
   ///
   /// This is a static helper that handles colorizing the level and formatting
@@ -104,7 +113,8 @@ class TextDiagnostic : public DiagnosticRenderer {
ArrayRef Hints);
 
   void emitSnippet(StringRef SourceLine, unsigned MaxLineNoDisplayWidth,
-   unsigned LineNo);
+   unsigned LineNo, unsigned DisplayLineNo,
+   ArrayRef Styles);
 
   void emitParseableFixits(ArrayRef Hints, const SourceManager );
 };
diff --git a/clang/include/clang/Lex/Preprocessor.h 
b/clang/include/clang/Lex/Preprocessor.h
index 4ec21a8b6be2c8..d89e2be1bf5ff5 100644
--- a/clang/include/clang/Lex/Preprocessor.h
+++ b/clang/include/clang/Lex/Preprocessor.h
@@ -284,6 +284,8 @@ class Preprocessor {
   /// The kind of translation unit we are processing.
   const TranslationUnitKind TUKind;
 
+  const char *getCheckPoint(FileID FID, const char *Start) const;
+
 private:
   /// The code-completion handler.
   CodeCompletionHandler *CodeComplete = nullptr;
@@ -311,6 +313,9 @@ class Preprocessor {
   /// The import path for named module that we're currently processing.
   SmallVector, 2> 
NamedModuleImportPath;
 
+  llvm::DenseMap> CheckPoints;
+  unsigned CheckPointCounter = 0;
+
   /// Whether the import is an `@import` or a standard c++ modules import.
   bool IsAtImport = false;
 
diff --git a/clang/lib/Frontend/TextDiagnostic.cpp 
b/clang/lib/Frontend/TextDiagnostic.cpp
index 779dead5d058d1..691809ceded87b 100644
--- a/clang/lib/Frontend/TextDiagnostic.cpp
+++ b/clang/lib/Frontend/TextDiagnostic.cpp
@@ -12,6 +12,7 @@
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/ConvertUTF.h"
@@ -41,6 +42,14 @@ static const enum raw_ostream::Colors fatalColor = 
raw_ostream::RED;
 static const enum raw_ostream::Colors savedColor =
   raw_ostream::SAVEDCOLOR;
 
+// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
+// is already taken for 'note'. Green is already used to underline
+// source ranges. White and black are bad because of the usual
+// terminal backgrounds. Which leaves us only with TWO options.
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::YELLOW;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+
 /// Add highlights to differences in template strings.
 static void applyTemplateHighlighting(raw_ostream , StringRef Str,
   bool , bool Bold) {
@@ -644,10 +653,10 @@ static bool printWordWrapped(raw_ostream , StringRef 
Str, unsigned Columns,
   return Wrapped;
 }
 
-TextDiagnostic::TextDiagnostic(raw_ostream ,
-   const LangOptions ,
-   DiagnosticOptions *DiagOpts)
-  : DiagnosticRenderer(LangOpts, DiagOpts), 

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -1112,6 +1121,140 @@ prepareAndFilterRanges(const 
SmallVectorImpl ,
   return LineRanges;
 }
 
+/// Creates syntax highlighting information in form of StyleRanges.
+///
+/// The returned unique ptr has always exactly size
+/// (\p EndLineNumber - \p StartLineNumber + 1). Each SmallVector in there
+/// corresponds to syntax highlighting information in one line. In each line,
+/// the StyleRanges are non-overlapping and sorted from start to end of the
+/// line.
+std::unique_ptr[]>
+highlightLines(StringRef FileData, unsigned StartLineNumber,
+   unsigned EndLineNumber, const Preprocessor *PP,
+   const LangOptions , uint32_t MaxHighlightFileSize,
+   FileID FID, const SourceManager ) {
+  assert(StartLineNumber <= EndLineNumber);
+  auto SnippetRanges =
+  std::make_unique[]>(
+  EndLineNumber - StartLineNumber + 1);
+
+  if (!PP)
+return SnippetRanges;
+
+  // Might cause emission of another diagnostic.
+  if (PP->getIdentifierTable().getExternalIdentifierLookup())
+return SnippetRanges;
+
+  auto Buff = llvm::MemoryBuffer::getMemBuffer(FileData);
+  if (Buff->getBufferSize() > MaxHighlightFileSize)
+return SnippetRanges;
+
+  Lexer L{FID, *Buff, SM, LangOpts};
+  L.SetKeepWhitespaceMode(true);
+
+  // Classify the given token and append it to the given vector.
+  auto appendStyle =
+  [PP, ](SmallVector ,
+  const Token , unsigned Start, unsigned Length) -> void 
{
+if (T.is(tok::raw_identifier)) {
+  StringRef RawIdent = T.getRawIdentifier();
+  // Special case true/false/nullptr literals, since they will otherwise be
+  // treated as keywords.
+  if (RawIdent == "true" || RawIdent == "false" || RawIdent == "nullptr") {

AaronBallman wrote:

There's not a programmatic way to obtain those, but perhaps we could change the 
macros usable with TokenKinds.def to expose that. I was thinking we'd manually 
encode them here, but a programmatic way is more futureproof. I'd be 
comfortable going with your preference; if you do manually encode them, I'd 
recommend switching to a `StringSwitch` instead of a series of `||`.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 34ca28505542d55f62da80d8fd3c2561535185d3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH 1/2] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 clang/include/clang/Frontend/TextDiagnostic.h |  18 +-
 clang/include/clang/Lex/Preprocessor.h|   5 +
 clang/lib/Frontend/TextDiagnostic.cpp | 197 +-
 clang/lib/Frontend/TextDiagnosticPrinter.cpp  |   2 +-
 clang/lib/Lex/Preprocessor.cpp|  24 +++
 5 files changed, 232 insertions(+), 14 deletions(-)

diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca..a2fe8ae995423b 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -16,6 +16,7 @@
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
 #include "clang/Frontend/DiagnosticRenderer.h"
+#include "llvm/Support/raw_ostream.h"
 
 namespace clang {
 
@@ -33,14 +34,22 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  const Preprocessor *PP;
 
 public:
-  TextDiagnostic(raw_ostream ,
- const LangOptions ,
- DiagnosticOptions *DiagOpts);
+  TextDiagnostic(raw_ostream , const LangOptions ,
+ DiagnosticOptions *DiagOpts, const Preprocessor *PP = 
nullptr);
 
   ~TextDiagnostic() override;
 
+  struct StyleRange {
+unsigned Start;
+unsigned End;
+enum llvm::raw_ostream::Colors Color;
+StyleRange(unsigned S, unsigned E, enum llvm::raw_ostream::Colors C)
+: Start(S), End(E), Color(C){};
+  };
+
   /// Print the diagonstic level to a raw_ostream.
   ///
   /// This is a static helper that handles colorizing the level and formatting
@@ -104,7 +113,8 @@ class TextDiagnostic : public DiagnosticRenderer {
ArrayRef Hints);
 
   void emitSnippet(StringRef SourceLine, unsigned MaxLineNoDisplayWidth,
-   unsigned LineNo);
+   unsigned LineNo, unsigned DisplayLineNo,
+   ArrayRef Styles);
 
   void emitParseableFixits(ArrayRef Hints, const SourceManager );
 };
diff --git a/clang/include/clang/Lex/Preprocessor.h 
b/clang/include/clang/Lex/Preprocessor.h
index 4ec21a8b6be2c8..d89e2be1bf5ff5 100644
--- a/clang/include/clang/Lex/Preprocessor.h
+++ b/clang/include/clang/Lex/Preprocessor.h
@@ -284,6 +284,8 @@ class Preprocessor {
   /// The kind of translation unit we are processing.
   const TranslationUnitKind TUKind;
 
+  const char *getCheckPoint(FileID FID, const char *Start) const;
+
 private:
   /// The code-completion handler.
   CodeCompletionHandler *CodeComplete = nullptr;
@@ -311,6 +313,9 @@ class Preprocessor {
   /// The import path for named module that we're currently processing.
   SmallVector, 2> 
NamedModuleImportPath;
 
+  llvm::DenseMap> CheckPoints;
+  unsigned CheckPointCounter = 0;
+
   /// Whether the import is an `@import` or a standard c++ modules import.
   bool IsAtImport = false;
 
diff --git a/clang/lib/Frontend/TextDiagnostic.cpp 
b/clang/lib/Frontend/TextDiagnostic.cpp
index 779dead5d058d1..691809ceded87b 100644
--- a/clang/lib/Frontend/TextDiagnostic.cpp
+++ b/clang/lib/Frontend/TextDiagnostic.cpp
@@ -12,6 +12,7 @@
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/ConvertUTF.h"
@@ -41,6 +42,14 @@ static const enum raw_ostream::Colors fatalColor = 
raw_ostream::RED;
 static const enum raw_ostream::Colors savedColor =
   raw_ostream::SAVEDCOLOR;
 
+// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
+// is already taken for 'note'. Green is already used to underline
+// source ranges. White and black are bad because of the usual
+// terminal backgrounds. Which leaves us only with TWO options.
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::YELLOW;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+
 /// Add highlights to differences in template strings.
 static void applyTemplateHighlighting(raw_ostream , StringRef Str,
   bool , bool Bold) {
@@ -644,10 +653,10 @@ static bool printWordWrapped(raw_ostream , StringRef 
Str, unsigned Columns,
   return Wrapped;
 }
 
-TextDiagnostic::TextDiagnostic(raw_ostream ,
-   const LangOptions ,
-   DiagnosticOptions *DiagOpts)
-  : DiagnosticRenderer(LangOpts, DiagOpts), OS(OS) {}

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

@@ -1112,6 +1121,140 @@ prepareAndFilterRanges(const 
SmallVectorImpl ,
   return LineRanges;
 }
 
+/// Creates syntax highlighting information in form of StyleRanges.
+///
+/// The returned unique ptr has always exactly size
+/// (\p EndLineNumber - \p StartLineNumber + 1). Each SmallVector in there
+/// corresponds to syntax highlighting information in one line. In each line,
+/// the StyleRanges are non-overlapping and sorted from start to end of the
+/// line.
+std::unique_ptr[]>
+highlightLines(StringRef FileData, unsigned StartLineNumber,
+   unsigned EndLineNumber, const Preprocessor *PP,
+   const LangOptions , uint32_t MaxHighlightFileSize,
+   FileID FID, const SourceManager ) {
+  assert(StartLineNumber <= EndLineNumber);
+  auto SnippetRanges =
+  std::make_unique[]>(
+  EndLineNumber - StartLineNumber + 1);
+
+  if (!PP)
+return SnippetRanges;
+
+  // Might cause emission of another diagnostic.
+  if (PP->getIdentifierTable().getExternalIdentifierLookup())
+return SnippetRanges;
+
+  auto Buff = llvm::MemoryBuffer::getMemBuffer(FileData);
+  if (Buff->getBufferSize() > MaxHighlightFileSize)
+return SnippetRanges;
+
+  Lexer L{FID, *Buff, SM, LangOpts};
+  L.SetKeepWhitespaceMode(true);
+
+  // Classify the given token and append it to the given vector.
+  auto appendStyle =
+  [PP, ](SmallVector ,
+  const Token , unsigned Start, unsigned Length) -> void 
{
+if (T.is(tok::raw_identifier)) {
+  StringRef RawIdent = T.getRawIdentifier();
+  // Special case true/false/nullptr literals, since they will otherwise be
+  // treated as keywords.
+  if (RawIdent == "true" || RawIdent == "false" || RawIdent == "nullptr") {

tbaederr wrote:

Is there a programmatic way to obtain those? Or do you want me to manually 
encode them here?

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

tbaederr wrote:

> The only thing we need to make sure is that highlighting isn't done in CI/ 
> redirected output

It now respects `DiagOpts->ShowColors`, so that should work. But I guess you're 
talking about a test. :)



https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

cor3ntin wrote:

That looks reasonable.
I really think we should land this, with checkpoints early in the clang 19 
cycle to get actual data.
The only thing we need to make sure is that highlighting isn't done in CI/ 
redirected output

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 34ca28505542d55f62da80d8fd3c2561535185d3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 clang/include/clang/Frontend/TextDiagnostic.h |  18 +-
 clang/include/clang/Lex/Preprocessor.h|   5 +
 clang/lib/Frontend/TextDiagnostic.cpp | 197 +-
 clang/lib/Frontend/TextDiagnosticPrinter.cpp  |   2 +-
 clang/lib/Lex/Preprocessor.cpp|  24 +++
 5 files changed, 232 insertions(+), 14 deletions(-)

diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca..a2fe8ae995423b 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -16,6 +16,7 @@
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
 #include "clang/Frontend/DiagnosticRenderer.h"
+#include "llvm/Support/raw_ostream.h"
 
 namespace clang {
 
@@ -33,14 +34,22 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  const Preprocessor *PP;
 
 public:
-  TextDiagnostic(raw_ostream ,
- const LangOptions ,
- DiagnosticOptions *DiagOpts);
+  TextDiagnostic(raw_ostream , const LangOptions ,
+ DiagnosticOptions *DiagOpts, const Preprocessor *PP = 
nullptr);
 
   ~TextDiagnostic() override;
 
+  struct StyleRange {
+unsigned Start;
+unsigned End;
+enum llvm::raw_ostream::Colors Color;
+StyleRange(unsigned S, unsigned E, enum llvm::raw_ostream::Colors C)
+: Start(S), End(E), Color(C){};
+  };
+
   /// Print the diagonstic level to a raw_ostream.
   ///
   /// This is a static helper that handles colorizing the level and formatting
@@ -104,7 +113,8 @@ class TextDiagnostic : public DiagnosticRenderer {
ArrayRef Hints);
 
   void emitSnippet(StringRef SourceLine, unsigned MaxLineNoDisplayWidth,
-   unsigned LineNo);
+   unsigned LineNo, unsigned DisplayLineNo,
+   ArrayRef Styles);
 
   void emitParseableFixits(ArrayRef Hints, const SourceManager );
 };
diff --git a/clang/include/clang/Lex/Preprocessor.h 
b/clang/include/clang/Lex/Preprocessor.h
index 4ec21a8b6be2c8..d89e2be1bf5ff5 100644
--- a/clang/include/clang/Lex/Preprocessor.h
+++ b/clang/include/clang/Lex/Preprocessor.h
@@ -284,6 +284,8 @@ class Preprocessor {
   /// The kind of translation unit we are processing.
   const TranslationUnitKind TUKind;
 
+  const char *getCheckPoint(FileID FID, const char *Start) const;
+
 private:
   /// The code-completion handler.
   CodeCompletionHandler *CodeComplete = nullptr;
@@ -311,6 +313,9 @@ class Preprocessor {
   /// The import path for named module that we're currently processing.
   SmallVector, 2> 
NamedModuleImportPath;
 
+  llvm::DenseMap> CheckPoints;
+  unsigned CheckPointCounter = 0;
+
   /// Whether the import is an `@import` or a standard c++ modules import.
   bool IsAtImport = false;
 
diff --git a/clang/lib/Frontend/TextDiagnostic.cpp 
b/clang/lib/Frontend/TextDiagnostic.cpp
index 779dead5d058d1..691809ceded87b 100644
--- a/clang/lib/Frontend/TextDiagnostic.cpp
+++ b/clang/lib/Frontend/TextDiagnostic.cpp
@@ -12,6 +12,7 @@
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/ConvertUTF.h"
@@ -41,6 +42,14 @@ static const enum raw_ostream::Colors fatalColor = 
raw_ostream::RED;
 static const enum raw_ostream::Colors savedColor =
   raw_ostream::SAVEDCOLOR;
 
+// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
+// is already taken for 'note'. Green is already used to underline
+// source ranges. White and black are bad because of the usual
+// terminal backgrounds. Which leaves us only with TWO options.
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::YELLOW;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+
 /// Add highlights to differences in template strings.
 static void applyTemplateHighlighting(raw_ostream , StringRef Str,
   bool , bool Bold) {
@@ -644,10 +653,10 @@ static bool printWordWrapped(raw_ostream , StringRef 
Str, unsigned Columns,
   return Wrapped;
 }
 
-TextDiagnostic::TextDiagnostic(raw_ostream ,
-   const LangOptions ,
-   DiagnosticOptions *DiagOpts)
-  : DiagnosticRenderer(LangOpts, DiagOpts), OS(OS) {}
+TextDiagnostic::TextDiagnostic(raw_ostream , const LangOptions ,
+ 

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

> > > Here are the results for a quick implementation of check points: 
> > > http://llvm-compile-time-tracker.com/compare.php?from=12e425d0cf9bca072c7b2138e50acbc5f1cd818c=99f3a7853f9fa83bffe3b4d04e41e744169d426a=instructions:u
> > > With a little more fiddling: 
> > > http://llvm-compile-time-tracker.com/compare.php?from=12e425d0cf9bca072c7b2138e50acbc5f1cd818c=0ee6dd17747818b05a1d504e4916ce46ef061226=instructions:u
> > 
> > 
> > Imo this is reasonable, we should go in that direction. It's not free but 
> > it is predictable
> > Further improvements:
> > * cache the lookup of `CheckPoints[FID]` as it should not change between 
> > calls to `BeginSourceFile`
> > * Play with reserve and/or dequeue
> 
> Yeah, I like the direction it heads with checkpoints. The implementation is 
> pretty straight-forward, the performance is pretty consistent (and doesn't 
> seem to add significant overhead). Just for comparison though, how does the 
> stress test from the re-lexing approach perform with checkpoints? I expect 
> we'll still see an increase in compile times but hopefully not "took more 
> than an hour" levels of increase.

Before any of the changes in this PR:
```
real0m9.909s
user0m0.934s
sys 0m8.945s
```

with highlighting:
```
real1m3.245s
user0m53.403s
sys 0m9.682s
```

with the changes, but no highlighting, i.e. only the preprocessor changes:
```
real0m10.032s
user0m0.937s
sys 0m9.064s
```



https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

> > Here are the results for a quick implementation of check points: 
> > http://llvm-compile-time-tracker.com/compare.php?from=12e425d0cf9bca072c7b2138e50acbc5f1cd818c=99f3a7853f9fa83bffe3b4d04e41e744169d426a=instructions:u
> > With a little more fiddling: 
> > http://llvm-compile-time-tracker.com/compare.php?from=12e425d0cf9bca072c7b2138e50acbc5f1cd818c=0ee6dd17747818b05a1d504e4916ce46ef061226=instructions:u
> 
> Imo this is reasonable, we should go in that direction. It's not free but it 
> is predictable
> 
> Further improvements:
> 
> * cache the lookup of `CheckPoints[FID]` as it should not change between 
> calls to `BeginSourceFile`

That doesn't seem to make much of a difference:
http://llvm-compile-time-tracker.com/compare.php?from=a21c4f8efc38c739f4dca3282793f42274266d59=6dbc70d71750770dd8030a82f1fec1042e5a2bbc=instructions:u

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


AaronBallman wrote:

> > Here are the results for a quick implementation of check points: 
> > http://llvm-compile-time-tracker.com/compare.php?from=12e425d0cf9bca072c7b2138e50acbc5f1cd818c=99f3a7853f9fa83bffe3b4d04e41e744169d426a=instructions:u
> > With a little more fiddling: 
> > http://llvm-compile-time-tracker.com/compare.php?from=12e425d0cf9bca072c7b2138e50acbc5f1cd818c=0ee6dd17747818b05a1d504e4916ce46ef061226=instructions:u
> 
> Imo this is reasonable, we should go in that direction. It's not free but it 
> is predictable
> 
> Further improvements:
> * cache the lookup of `CheckPoints[FID]` as it should not change between 
> calls to `BeginSourceFile`
> * Play with reserve and/or dequeue

Yeah, I like the direction it heads with checkpoints. The implementation is 
pretty straight-forward, the performance is pretty consistent (and doesn't seem 
to add significant overhead). Just for comparison though, how does the stress 
test from the re-lexing approach perform with checkpoints? I expect we'll still 
see an increase in compile times but hopefully not "took more than an hour" 
levels of increase.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -954,6 +954,9 @@ void Preprocessor::Lex(Token ) {
 }
   }
 
+  if (CurLexer && ++CheckPointCounter == 1024)

AaronBallman wrote:

We might want to make this configurable? At the very least, we should have some 
comments explaining the 1024.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


cor3ntin wrote:

> Here are the results for a quick implementation of check points: 
> http://llvm-compile-time-tracker.com/compare.php?from=12e425d0cf9bca072c7b2138e50acbc5f1cd818c=99f3a7853f9fa83bffe3b4d04e41e744169d426a=instructions:u
> 
> With a little more fiddling: 
> http://llvm-compile-time-tracker.com/compare.php?from=12e425d0cf9bca072c7b2138e50acbc5f1cd818c=0ee6dd17747818b05a1d504e4916ce46ef061226=instructions:u

Imo this is reasonable, we should go in that direction. It's not free but it is 
predictable

Further improvements:
 * cache the lookup of `CheckPoints[FID]` as it should not change between calls 
to `BeginSourceFile`
 * Play with reserve and/or dequeue





https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

Here are the results for a quick implementation of check points: 
http://llvm-compile-time-tracker.com/compare.php?from=12e425d0cf9bca072c7b2138e50acbc5f1cd818c=99f3a7853f9fa83bffe3b4d04e41e744169d426a=instructions:u

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


AaronBallman wrote:

> So, I have a test file of about 4.5MB where every line is a `#warning`. Not 
> sure how I should create a "similar" file without diagnostics though.
> 
> ```
> # ls -lah test.cpp
> -rw-r--r--. 1 root root 4.5M Dec 12 02:52 test.cpp
> # wc -l test.cpp
> 357391 test.cpp
> ```
> 
> With a release build, the time is:
> 
> ```
> # time bin/clang++ -cc1 ./test.cpp > out.txt 2>&1
> real1m3.967s
> user0m52.142s
> sys 0m11.606s
> ```
> 
> this is _without_ any highlighting, since the file is too big.
> 
> Here's the same file with highlighting:
> 
> ```
> # time bin/clang++ -cc1 ./test.cpp -fmax-highlight-file-size=5242880 > 
> out.txt 2>&1
> real72m28.551s
> user72m11.610s
> sys 0m11.561s
> ```
> 
> (not that we definitely shouldn't take any longer here, since we don't emit 
> any highlighting anyway because we're piping into a file. Need to fix that).

It seems to me that we either pay a small cost (~.10% or less) for checkpoints 
or we make people who have a lot of diagnostics pay a huge cost (going from 1.5 
minutes to over an hour) for relexing, and that makes me think we should go 
with checkpoints instead of relexing the whole file. I realize this was a 
pathological test case and shouldn't be representative of real world code, but 
there are plenty of projects out there with *tons* of warnings being emitted 
and I don't think we can cause that much of a slowdown for their compile times. 
WDYT?

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

Ping

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

Ping

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

Ping

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

So, I have a test file of about 4.5MB where every line is a `#warning`. Not 
sure how I should create a "similar" file without diagnostics though.

```
# ls -lah test.cpp
-rw-r--r--. 1 root root 4.5M Dec 12 02:52 test.cpp
# wc -l test.cpp
357391 test.cpp
```

With a release build, the time is:
```
# time bin/clang++ -cc1 ./test.cpp > out.txt 2>&1
real1m3.967s
user0m52.142s
sys 0m11.606s
```
this is  _without_ any highlighting, since the file is too big.

Here's the same file with highlighting:
```
# time bin/clang++ -cc1 ./test.cpp -fmax-highlight-file-size=5242880 > out.txt 
2>&1
real72m28.551s
user72m11.610s
sys 0m11.561s
```

(not that we definitely shouldn't take any longer here, since we don't emit any 
highlighting anyway because we're piping into a file. Need to fix that).

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -53,6 +53,10 @@ LLVM_READNONE inline bool isASCII(int64_t c) { return 0 <= c 
&& c <= 127; }
 /// which is [a-zA-Z_].
 LLVM_READONLY inline bool isAsciiIdentifierStart(unsigned char c,
  bool AllowDollar = false) {
+  if (!AllowDollar) {
+return c == '_' || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z');
+  }
+

tbaederr wrote:

Ignore this, it's a leftover from a different patch

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


AaronBallman wrote:

> @tbaederr I will need time to review/discuss this patch with other folks 
> (@AaronBallman). I thought we had agreement on a direction before it got 
> reverted completely and it is not clear to me that doing whole file lexing 
> for each warning isn't going to negatively impact some folks.
> 
> But I haven't forgotten about it!

Re-lexing the file on each diagnostic works well enough for cases where there's 
only a few diagnostics in the file. But I do worry about the pathological case 
where there's a diagnostic on every line: that means compiling the file is 
O(N^2) in terms of lexing tokens whereas using checkpoints would still be O(N). 
Do you have an idea how bad the worst case performance looks compared to the 
best case? e.g., can you write a ~1MB test file that has no diagnostics and 
another similar ~1MB test file that has a warning diagnostic per line, and see 
what the performance difference is between runs of the compiler for those files?

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -1112,6 +1121,140 @@ prepareAndFilterRanges(const 
SmallVectorImpl ,
   return LineRanges;
 }
 
+/// Creates syntax highlighting information in form of StyleRanges.
+///
+/// The returned unique ptr has always exactly size
+/// (\p EndLineNumber - \p StartLineNumber + 1). Each SmallVector in there
+/// corresponds to syntax highlighting information in one line. In each line,
+/// the StyleRanges are non-overlapping and sorted from start to end of the
+/// line.
+std::unique_ptr[]>
+highlightLines(StringRef FileData, unsigned StartLineNumber,
+   unsigned EndLineNumber, const Preprocessor *PP,
+   const LangOptions , uint32_t MaxHighlightFileSize,
+   FileID FID, const SourceManager ) {
+  assert(StartLineNumber <= EndLineNumber);
+  auto SnippetRanges =
+  std::make_unique[]>(
+  EndLineNumber - StartLineNumber + 1);
+
+  if (!PP)
+return SnippetRanges;
+
+  // Might cause emission of another diagnostic.
+  if (PP->getIdentifierTable().getExternalIdentifierLookup())
+return SnippetRanges;
+
+  auto Buff = llvm::MemoryBuffer::getMemBuffer(FileData);
+  if (Buff->getBufferSize() > MaxHighlightFileSize)
+return SnippetRanges;
+
+  Lexer L{FID, *Buff, SM, LangOpts};
+  L.SetKeepWhitespaceMode(true);
+
+  // Classify the given token and append it to the given vector.
+  auto appendStyle =
+  [PP, ](SmallVector ,
+  const Token , unsigned Start, unsigned Length) -> void 
{
+if (T.is(tok::raw_identifier)) {
+  StringRef RawIdent = T.getRawIdentifier();
+  // Special case true/false/nullptr literals, since they will otherwise be
+  // treated as keywords.
+  if (RawIdent == "true" || RawIdent == "false" || RawIdent == "nullptr") {

AaronBallman wrote:

I think there are more cases like this: `__func__`, `__objc_yes`, `__objc_no`, 
`__null`, `__FUNCDNAME__`, `__FUNCSIG__`, `L__FUNCTION__`, and `L__FUNCSIG__`, 
at least from my reading of TokenKinds.def.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -53,6 +53,10 @@ LLVM_READNONE inline bool isASCII(int64_t c) { return 0 <= c 
&& c <= 127; }
 /// which is [a-zA-Z_].
 LLVM_READONLY inline bool isAsciiIdentifierStart(unsigned char c,
  bool AllowDollar = false) {
+  if (!AllowDollar) {
+return c == '_' || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z');
+  }
+

AaronBallman wrote:

Why do we need this change?

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


cor3ntin wrote:

@tbaederr I will need time to review/discuss this patch with other folks 
(@AaronBallman). 
I thought we had agreement on a direction before it got reverted completely and 
it is not clear to me that doing whole file lexing for each warning isn't going 
to negatively impact some folks.

But I haven't forgotten about it!

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

Ping

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

Ping

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -1288,11 +1291,33 @@ void TextDiagnostic::emitSnippet(StringRef SourceLine,
 
   // Print the source line one character at a time.
   bool PrintReversed = false;
+  bool HighlightingEnabled = DiagOpts->ShowColors;
   size_t I = 0;
   while (I < SourceLine.size()) {
 auto [Str, WasPrintable] =
 printableTextForNextCharacter(SourceLine, , DiagOpts->TabStop);
 
+// Just stop highlighting anything for this line if we found a 
non-printable
+// character.

tbaederr wrote:

>From some local testing, seems like this just works.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

FYI  I think this is ready for proper review now. We could still add some tests 
of course. It's using a maximum file size now and highlights the entire 
requested line range in one go.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 244bd962de82f3a7f65054086546170a88bdac6f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH 01/23] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 .../clang/Frontend/CodeSnippetHighlighter.h   |  46 +++
 clang/include/clang/Frontend/TextDiagnostic.h |   2 +
 clang/lib/Frontend/CMakeLists.txt |   1 +
 clang/lib/Frontend/CodeSnippetHighlighter.cpp | 120 ++
 clang/lib/Frontend/TextDiagnostic.cpp |  26 
 5 files changed, 195 insertions(+)
 create mode 100644 clang/include/clang/Frontend/CodeSnippetHighlighter.h
 create mode 100644 clang/lib/Frontend/CodeSnippetHighlighter.cpp

diff --git a/clang/include/clang/Frontend/CodeSnippetHighlighter.h 
b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
new file mode 100644
index 000..776954b59e2e1a8
--- /dev/null
+++ b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
@@ -0,0 +1,46 @@
+//===--- CodeSnippetHighlighter.h - Code snippet highlighting ---*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+#define LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+
+#include "clang/Basic/LangOptions.h"
+#include "llvm/ADT/SmallSet.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+namespace clang {
+
+struct StyleRange {
+  unsigned Start;
+  unsigned End;
+  const enum llvm::raw_ostream::Colors c;
+};
+
+class CodeSnippetHighlighter final {
+public:
+  CodeSnippetHighlighter() = default;
+
+  /// Produce StyleRanges for the given line.
+  /// The returned vector contains non-overlapping style ranges. They are 
sorted
+  /// from beginning of the line to the end.
+  std::vector highlightLine(llvm::StringRef SourceLine,
+const LangOptions );
+
+private:
+  bool Initialized = false;
+  /// Fills Keywords and Literals.
+  void ensureTokenData();
+
+  llvm::SmallSet Keywords;
+  llvm::SmallSet Literals;
+};
+
+} // namespace clang
+
+#endif
diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca8..59fd4d4f9408d48 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -15,6 +15,7 @@
 #ifndef LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
+#include "clang/Frontend/CodeSnippetHighlighter.h"
 #include "clang/Frontend/DiagnosticRenderer.h"
 
 namespace clang {
@@ -33,6 +34,7 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  CodeSnippetHighlighter SnippetHighlighter;
 
 public:
   TextDiagnostic(raw_ostream ,
diff --git a/clang/lib/Frontend/CMakeLists.txt 
b/clang/lib/Frontend/CMakeLists.txt
index 1e5f0a859dfd568..f3547f771593093 100644
--- a/clang/lib/Frontend/CMakeLists.txt
+++ b/clang/lib/Frontend/CMakeLists.txt
@@ -42,6 +42,7 @@ add_clang_library(clangFrontend
   TextDiagnosticPrinter.cpp
   VerifyDiagnosticConsumer.cpp
   InterfaceStubFunctionsConsumer.cpp
+  CodeSnippetHighlighter.cpp
 
   DEPENDS
   ClangDriverOptions
diff --git a/clang/lib/Frontend/CodeSnippetHighlighter.cpp 
b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
new file mode 100644
index 000..829a533ad2692e5
--- /dev/null
+++ b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
@@ -0,0 +1,120 @@
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+
+void CodeSnippetHighlighter::ensureTokenData() {
+  if (Initialized)
+return;
+
+  // List of keywords, literals and types we want to highlight.
+  // These are best-effort, as is everything we do wrt. highlighting.
+  Keywords.insert("_Static_assert");
+  Keywords.insert("auto");
+  Keywords.insert("concept");
+  

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 244bd962de82f3a7f65054086546170a88bdac6f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH 01/22] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 .../clang/Frontend/CodeSnippetHighlighter.h   |  46 +++
 clang/include/clang/Frontend/TextDiagnostic.h |   2 +
 clang/lib/Frontend/CMakeLists.txt |   1 +
 clang/lib/Frontend/CodeSnippetHighlighter.cpp | 120 ++
 clang/lib/Frontend/TextDiagnostic.cpp |  26 
 5 files changed, 195 insertions(+)
 create mode 100644 clang/include/clang/Frontend/CodeSnippetHighlighter.h
 create mode 100644 clang/lib/Frontend/CodeSnippetHighlighter.cpp

diff --git a/clang/include/clang/Frontend/CodeSnippetHighlighter.h 
b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
new file mode 100644
index 000..776954b59e2e1a8
--- /dev/null
+++ b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
@@ -0,0 +1,46 @@
+//===--- CodeSnippetHighlighter.h - Code snippet highlighting ---*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+#define LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+
+#include "clang/Basic/LangOptions.h"
+#include "llvm/ADT/SmallSet.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+namespace clang {
+
+struct StyleRange {
+  unsigned Start;
+  unsigned End;
+  const enum llvm::raw_ostream::Colors c;
+};
+
+class CodeSnippetHighlighter final {
+public:
+  CodeSnippetHighlighter() = default;
+
+  /// Produce StyleRanges for the given line.
+  /// The returned vector contains non-overlapping style ranges. They are 
sorted
+  /// from beginning of the line to the end.
+  std::vector highlightLine(llvm::StringRef SourceLine,
+const LangOptions );
+
+private:
+  bool Initialized = false;
+  /// Fills Keywords and Literals.
+  void ensureTokenData();
+
+  llvm::SmallSet Keywords;
+  llvm::SmallSet Literals;
+};
+
+} // namespace clang
+
+#endif
diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca8..59fd4d4f9408d48 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -15,6 +15,7 @@
 #ifndef LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
+#include "clang/Frontend/CodeSnippetHighlighter.h"
 #include "clang/Frontend/DiagnosticRenderer.h"
 
 namespace clang {
@@ -33,6 +34,7 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  CodeSnippetHighlighter SnippetHighlighter;
 
 public:
   TextDiagnostic(raw_ostream ,
diff --git a/clang/lib/Frontend/CMakeLists.txt 
b/clang/lib/Frontend/CMakeLists.txt
index 1e5f0a859dfd568..f3547f771593093 100644
--- a/clang/lib/Frontend/CMakeLists.txt
+++ b/clang/lib/Frontend/CMakeLists.txt
@@ -42,6 +42,7 @@ add_clang_library(clangFrontend
   TextDiagnosticPrinter.cpp
   VerifyDiagnosticConsumer.cpp
   InterfaceStubFunctionsConsumer.cpp
+  CodeSnippetHighlighter.cpp
 
   DEPENDS
   ClangDriverOptions
diff --git a/clang/lib/Frontend/CodeSnippetHighlighter.cpp 
b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
new file mode 100644
index 000..829a533ad2692e5
--- /dev/null
+++ b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
@@ -0,0 +1,120 @@
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+
+void CodeSnippetHighlighter::ensureTokenData() {
+  if (Initialized)
+return;
+
+  // List of keywords, literals and types we want to highlight.
+  // These are best-effort, as is everything we do wrt. highlighting.
+  Keywords.insert("_Static_assert");
+  Keywords.insert("auto");
+  Keywords.insert("concept");
+  Keywords.insert("const");
+  

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 244bd962de82f3a7f65054086546170a88bdac6f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH 01/22] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 .../clang/Frontend/CodeSnippetHighlighter.h   |  46 +++
 clang/include/clang/Frontend/TextDiagnostic.h |   2 +
 clang/lib/Frontend/CMakeLists.txt |   1 +
 clang/lib/Frontend/CodeSnippetHighlighter.cpp | 120 ++
 clang/lib/Frontend/TextDiagnostic.cpp |  26 
 5 files changed, 195 insertions(+)
 create mode 100644 clang/include/clang/Frontend/CodeSnippetHighlighter.h
 create mode 100644 clang/lib/Frontend/CodeSnippetHighlighter.cpp

diff --git a/clang/include/clang/Frontend/CodeSnippetHighlighter.h 
b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
new file mode 100644
index 000..776954b59e2e1a8
--- /dev/null
+++ b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
@@ -0,0 +1,46 @@
+//===--- CodeSnippetHighlighter.h - Code snippet highlighting ---*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+#define LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+
+#include "clang/Basic/LangOptions.h"
+#include "llvm/ADT/SmallSet.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+namespace clang {
+
+struct StyleRange {
+  unsigned Start;
+  unsigned End;
+  const enum llvm::raw_ostream::Colors c;
+};
+
+class CodeSnippetHighlighter final {
+public:
+  CodeSnippetHighlighter() = default;
+
+  /// Produce StyleRanges for the given line.
+  /// The returned vector contains non-overlapping style ranges. They are 
sorted
+  /// from beginning of the line to the end.
+  std::vector highlightLine(llvm::StringRef SourceLine,
+const LangOptions );
+
+private:
+  bool Initialized = false;
+  /// Fills Keywords and Literals.
+  void ensureTokenData();
+
+  llvm::SmallSet Keywords;
+  llvm::SmallSet Literals;
+};
+
+} // namespace clang
+
+#endif
diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca8..59fd4d4f9408d48 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -15,6 +15,7 @@
 #ifndef LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
+#include "clang/Frontend/CodeSnippetHighlighter.h"
 #include "clang/Frontend/DiagnosticRenderer.h"
 
 namespace clang {
@@ -33,6 +34,7 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  CodeSnippetHighlighter SnippetHighlighter;
 
 public:
   TextDiagnostic(raw_ostream ,
diff --git a/clang/lib/Frontend/CMakeLists.txt 
b/clang/lib/Frontend/CMakeLists.txt
index 1e5f0a859dfd568..f3547f771593093 100644
--- a/clang/lib/Frontend/CMakeLists.txt
+++ b/clang/lib/Frontend/CMakeLists.txt
@@ -42,6 +42,7 @@ add_clang_library(clangFrontend
   TextDiagnosticPrinter.cpp
   VerifyDiagnosticConsumer.cpp
   InterfaceStubFunctionsConsumer.cpp
+  CodeSnippetHighlighter.cpp
 
   DEPENDS
   ClangDriverOptions
diff --git a/clang/lib/Frontend/CodeSnippetHighlighter.cpp 
b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
new file mode 100644
index 000..829a533ad2692e5
--- /dev/null
+++ b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
@@ -0,0 +1,120 @@
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+
+void CodeSnippetHighlighter::ensureTokenData() {
+  if (Initialized)
+return;
+
+  // List of keywords, literals and types we want to highlight.
+  // These are best-effort, as is everything we do wrt. highlighting.
+  Keywords.insert("_Static_assert");
+  Keywords.insert("auto");
+  Keywords.insert("concept");
+  Keywords.insert("const");
+  

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -0,0 +1,187 @@
+//===-- CodeSnippetHighlighter.cpp - Code snippet highlighting --*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+using namespace clang;
+
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::CYAN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+
+llvm::SmallVector CodeSnippetHighlighter::highlightLine(
+unsigned LineNumber, const Preprocessor *PP, const LangOptions ,
+FileID FID, const SourceManager , const char *LineStart) {
+  std::chrono::steady_clock::time_point begin =
+  std::chrono::steady_clock::now();
+
+  if (!PP)
+return {};
+
+  // Might cause emission of another diagnostic.
+  if (PP->getIdentifierTable().getExternalIdentifierLookup())
+return {};
+
+  size_t NTokens = 0;
+  // Classify the given token and append it to the given vector.
+  auto appendStyle = [PP, ](llvm::SmallVector ,
+ const Token , unsigned Start,
+ unsigned Length) -> void {
+if (T.is(tok::raw_identifier)) {
+  StringRef RawIdent = T.getRawIdentifier();
+  // Special case true/false/nullptr literals, since they will otherwise be
+  // treated as keywords.
+  if (RawIdent == "true" || RawIdent == "false" || RawIdent == "nullptr") {
+Vec.push_back(StyleRange{Start, Start + Length, LiteralColor});
+  } else {
+const IdentifierInfo *II = PP->getIdentifierInfo(RawIdent);
+assert(II);
+if (II->isKeyword(LangOpts))
+  Vec.push_back(StyleRange{Start, Start + Length, KeywordColor});
+  }
+} else if (tok::isLiteral(T.getKind())) {
+  Vec.push_back(StyleRange{Start, Start + Length, LiteralColor});
+} else {
+  assert(T.is(tok::comment));
+  Vec.push_back(StyleRange{Start, Start + Length, CommentColor});
+}
+  };
+
+  // Figure out where to start lexing from.
+  auto Buff = SM.getBufferOrNone(FID);
+  assert(Buff);
+  Lexer L = Lexer(FID, *Buff, SM, LangOpts);
+  L.SetKeepWhitespaceMode(true);
+
+  // Seek to the last save point before the start of the line.
+  if (const char *Save = PP->getSaveFor(LineStart);
+  Buff->getBufferStart() <= Save && Save < Buff->getBufferEnd()) {
+size_t Offset = Save - Buff->getBufferStart();
+assert(Save >= Buff->getBufferStart());
+assert(Save <= Buff->getBufferEnd());
+
+L.seek(Offset, /*IsAtStartOfLine=*/true);
+  }
+
+  llvm::SmallVector LineRanges;
+  bool Stop = false;
+  while (!Stop) {
+++NTokens;
+Token T;
+Stop = L.LexFromRawLexer(T);
+if (T.is(tok::unknown))
+  continue;
+
+// We are only interested in identifiers, literals and comments.
+if (!T.is(tok::raw_identifier) && !T.is(tok::comment) &&
+!tok::isLiteral(T.getKind()))
+  continue;
+
+bool Invalid = false;
+unsigned EndLine = SM.getSpellingLineNumber(T.getEndLoc(), ) - 1;
+if (Invalid)
+  continue;
+
+if (EndLine < LineNumber)
+  continue;
+unsigned StartLine =
+SM.getSpellingLineNumber(T.getLocation(), ) - 1;
+if (Invalid)
+  continue;
+if (StartLine > LineNumber)
+  break;
+
+// Must have an intersection at this point
+assert(StartLine <= LineNumber && EndLine >= LineNumber);
+
+unsigned StartCol =
+SM.getSpellingColumnNumber(T.getLocation(), ) - 1;
+if (Invalid)
+  continue;
+
+// Simple tokens.
+if (StartLine == EndLine) {
+  appendStyle(LineRanges, T, StartCol, T.getLength());
+  continue;
+}
+unsigned NumLines = EndLine - StartLine;
+assert(NumLines >= 1);
+
+// For tokens that span multiple lines (think multiline comments), we
+// divide them into multiple StyleRanges.
+unsigned EndCol = SM.getSpellingColumnNumber(T.getEndLoc(), ) - 1;
+if (Invalid)
+  

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Tom Honermann via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -0,0 +1,187 @@
+//===-- CodeSnippetHighlighter.cpp - Code snippet highlighting --*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+using namespace clang;
+
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::CYAN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+
+llvm::SmallVector CodeSnippetHighlighter::highlightLine(
+unsigned LineNumber, const Preprocessor *PP, const LangOptions ,
+FileID FID, const SourceManager , const char *LineStart) {
+  std::chrono::steady_clock::time_point begin =
+  std::chrono::steady_clock::now();
+
+  if (!PP)
+return {};
+
+  // Might cause emission of another diagnostic.
+  if (PP->getIdentifierTable().getExternalIdentifierLookup())
+return {};
+
+  size_t NTokens = 0;
+  // Classify the given token and append it to the given vector.
+  auto appendStyle = [PP, ](llvm::SmallVector ,
+ const Token , unsigned Start,
+ unsigned Length) -> void {
+if (T.is(tok::raw_identifier)) {
+  StringRef RawIdent = T.getRawIdentifier();
+  // Special case true/false/nullptr literals, since they will otherwise be
+  // treated as keywords.
+  if (RawIdent == "true" || RawIdent == "false" || RawIdent == "nullptr") {
+Vec.push_back(StyleRange{Start, Start + Length, LiteralColor});
+  } else {
+const IdentifierInfo *II = PP->getIdentifierInfo(RawIdent);
+assert(II);
+if (II->isKeyword(LangOpts))
+  Vec.push_back(StyleRange{Start, Start + Length, KeywordColor});
+  }
+} else if (tok::isLiteral(T.getKind())) {
+  Vec.push_back(StyleRange{Start, Start + Length, LiteralColor});
+} else {
+  assert(T.is(tok::comment));
+  Vec.push_back(StyleRange{Start, Start + Length, CommentColor});
+}
+  };
+
+  // Figure out where to start lexing from.
+  auto Buff = SM.getBufferOrNone(FID);
+  assert(Buff);
+  Lexer L = Lexer(FID, *Buff, SM, LangOpts);
+  L.SetKeepWhitespaceMode(true);
+
+  // Seek to the last save point before the start of the line.
+  if (const char *Save = PP->getSaveFor(LineStart);
+  Buff->getBufferStart() <= Save && Save < Buff->getBufferEnd()) {
+size_t Offset = Save - Buff->getBufferStart();
+assert(Save >= Buff->getBufferStart());
+assert(Save <= Buff->getBufferEnd());
+
+L.seek(Offset, /*IsAtStartOfLine=*/true);
+  }
+
+  llvm::SmallVector LineRanges;
+  bool Stop = false;
+  while (!Stop) {
+++NTokens;
+Token T;
+Stop = L.LexFromRawLexer(T);
+if (T.is(tok::unknown))
+  continue;
+
+// We are only interested in identifiers, literals and comments.
+if (!T.is(tok::raw_identifier) && !T.is(tok::comment) &&
+!tok::isLiteral(T.getKind()))
+  continue;
+
+bool Invalid = false;
+unsigned EndLine = SM.getSpellingLineNumber(T.getEndLoc(), ) - 1;
+if (Invalid)
+  continue;
+
+if (EndLine < LineNumber)
+  continue;
+unsigned StartLine =
+SM.getSpellingLineNumber(T.getLocation(), ) - 1;
+if (Invalid)
+  continue;
+if (StartLine > LineNumber)
+  break;
+
+// Must have an intersection at this point
+assert(StartLine <= LineNumber && EndLine >= LineNumber);
+
+unsigned StartCol =
+SM.getSpellingColumnNumber(T.getLocation(), ) - 1;
+if (Invalid)
+  continue;
+
+// Simple tokens.
+if (StartLine == EndLine) {
+  appendStyle(LineRanges, T, StartCol, T.getLength());
+  continue;
+}
+unsigned NumLines = EndLine - StartLine;
+assert(NumLines >= 1);
+
+// For tokens that span multiple lines (think multiline comments), we
+// divide them into multiple StyleRanges.
+unsigned EndCol = SM.getSpellingColumnNumber(T.getEndLoc(), ) - 1;
+if (Invalid)
+  

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -0,0 +1,187 @@
+//===-- CodeSnippetHighlighter.cpp - Code snippet highlighting --*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+using namespace clang;
+
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::CYAN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+
+llvm::SmallVector CodeSnippetHighlighter::highlightLine(
+unsigned LineNumber, const Preprocessor *PP, const LangOptions ,
+FileID FID, const SourceManager , const char *LineStart) {
+  std::chrono::steady_clock::time_point begin =
+  std::chrono::steady_clock::now();
+
+  if (!PP)
+return {};
+
+  // Might cause emission of another diagnostic.
+  if (PP->getIdentifierTable().getExternalIdentifierLookup())
+return {};
+
+  size_t NTokens = 0;
+  // Classify the given token and append it to the given vector.
+  auto appendStyle = [PP, ](llvm::SmallVector ,
+ const Token , unsigned Start,
+ unsigned Length) -> void {
+if (T.is(tok::raw_identifier)) {
+  StringRef RawIdent = T.getRawIdentifier();
+  // Special case true/false/nullptr literals, since they will otherwise be
+  // treated as keywords.
+  if (RawIdent == "true" || RawIdent == "false" || RawIdent == "nullptr") {
+Vec.push_back(StyleRange{Start, Start + Length, LiteralColor});
+  } else {
+const IdentifierInfo *II = PP->getIdentifierInfo(RawIdent);
+assert(II);
+if (II->isKeyword(LangOpts))
+  Vec.push_back(StyleRange{Start, Start + Length, KeywordColor});
+  }
+} else if (tok::isLiteral(T.getKind())) {
+  Vec.push_back(StyleRange{Start, Start + Length, LiteralColor});
+} else {
+  assert(T.is(tok::comment));
+  Vec.push_back(StyleRange{Start, Start + Length, CommentColor});
+}
+  };
+
+  // Figure out where to start lexing from.
+  auto Buff = SM.getBufferOrNone(FID);
+  assert(Buff);
+  Lexer L = Lexer(FID, *Buff, SM, LangOpts);
+  L.SetKeepWhitespaceMode(true);
+
+  // Seek to the last save point before the start of the line.
+  if (const char *Save = PP->getSaveFor(LineStart);
+  Buff->getBufferStart() <= Save && Save < Buff->getBufferEnd()) {
+size_t Offset = Save - Buff->getBufferStart();
+assert(Save >= Buff->getBufferStart());
+assert(Save <= Buff->getBufferEnd());
+
+L.seek(Offset, /*IsAtStartOfLine=*/true);
+  }
+
+  llvm::SmallVector LineRanges;
+  bool Stop = false;
+  while (!Stop) {
+++NTokens;
+Token T;
+Stop = L.LexFromRawLexer(T);
+if (T.is(tok::unknown))
+  continue;
+
+// We are only interested in identifiers, literals and comments.
+if (!T.is(tok::raw_identifier) && !T.is(tok::comment) &&
+!tok::isLiteral(T.getKind()))
+  continue;
+
+bool Invalid = false;
+unsigned EndLine = SM.getSpellingLineNumber(T.getEndLoc(), ) - 1;
+if (Invalid)
+  continue;
+
+if (EndLine < LineNumber)
+  continue;
+unsigned StartLine =
+SM.getSpellingLineNumber(T.getLocation(), ) - 1;
+if (Invalid)
+  continue;
+if (StartLine > LineNumber)
+  break;
+
+// Must have an intersection at this point
+assert(StartLine <= LineNumber && EndLine >= LineNumber);
+
+unsigned StartCol =
+SM.getSpellingColumnNumber(T.getLocation(), ) - 1;
+if (Invalid)
+  continue;
+
+// Simple tokens.
+if (StartLine == EndLine) {
+  appendStyle(LineRanges, T, StartCol, T.getLength());
+  continue;
+}
+unsigned NumLines = EndLine - StartLine;
+assert(NumLines >= 1);
+
+// For tokens that span multiple lines (think multiline comments), we
+// divide them into multiple StyleRanges.
+unsigned EndCol = SM.getSpellingColumnNumber(T.getEndLoc(), ) - 1;
+if (Invalid)
+  

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -0,0 +1,185 @@
+//===-- CodeSnippetHighlighter.cpp - Code snippet highlighting --*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/CharInfo.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+using namespace clang;
+
+// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
+// is already taken for 'note'. Green is already used to underline
+// source ranges. White and black are bad because of the usual
+// terminal backgrounds. Which leaves us only with TWO options.
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::YELLOW;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+/// Maximum size of file we still highlight.
+static constexpr size_t MaxBufferSize = 1024 * 1024; // 1MB.
+
+llvm::SmallVector CodeSnippetHighlighter::highlightLine(
+unsigned LineNumber, const Preprocessor *PP, const LangOptions ,
+FileID FID, const SourceManager , const char *LineStart) {
+  std::chrono::steady_clock::time_point begin =
+  std::chrono::steady_clock::now();
+
+  if (!PP)
+return {};
+

AaronBallman wrote:

Thanks! I guess this stays then. :-)

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 001149c81ddeca2488597ebae3604efeaee1c490 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH 01/21] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 .../clang/Frontend/CodeSnippetHighlighter.h   |  46 +++
 clang/include/clang/Frontend/TextDiagnostic.h |   2 +
 clang/lib/Frontend/CMakeLists.txt |   1 +
 clang/lib/Frontend/CodeSnippetHighlighter.cpp | 120 ++
 clang/lib/Frontend/TextDiagnostic.cpp |  26 
 5 files changed, 195 insertions(+)
 create mode 100644 clang/include/clang/Frontend/CodeSnippetHighlighter.h
 create mode 100644 clang/lib/Frontend/CodeSnippetHighlighter.cpp

diff --git a/clang/include/clang/Frontend/CodeSnippetHighlighter.h 
b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
new file mode 100644
index 000..776954b59e2e1a8
--- /dev/null
+++ b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
@@ -0,0 +1,46 @@
+//===--- CodeSnippetHighlighter.h - Code snippet highlighting ---*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+#define LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+
+#include "clang/Basic/LangOptions.h"
+#include "llvm/ADT/SmallSet.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+namespace clang {
+
+struct StyleRange {
+  unsigned Start;
+  unsigned End;
+  const enum llvm::raw_ostream::Colors c;
+};
+
+class CodeSnippetHighlighter final {
+public:
+  CodeSnippetHighlighter() = default;
+
+  /// Produce StyleRanges for the given line.
+  /// The returned vector contains non-overlapping style ranges. They are 
sorted
+  /// from beginning of the line to the end.
+  std::vector highlightLine(llvm::StringRef SourceLine,
+const LangOptions );
+
+private:
+  bool Initialized = false;
+  /// Fills Keywords and Literals.
+  void ensureTokenData();
+
+  llvm::SmallSet Keywords;
+  llvm::SmallSet Literals;
+};
+
+} // namespace clang
+
+#endif
diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca8..59fd4d4f9408d48 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -15,6 +15,7 @@
 #ifndef LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
+#include "clang/Frontend/CodeSnippetHighlighter.h"
 #include "clang/Frontend/DiagnosticRenderer.h"
 
 namespace clang {
@@ -33,6 +34,7 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  CodeSnippetHighlighter SnippetHighlighter;
 
 public:
   TextDiagnostic(raw_ostream ,
diff --git a/clang/lib/Frontend/CMakeLists.txt 
b/clang/lib/Frontend/CMakeLists.txt
index 1e5f0a859dfd568..f3547f771593093 100644
--- a/clang/lib/Frontend/CMakeLists.txt
+++ b/clang/lib/Frontend/CMakeLists.txt
@@ -42,6 +42,7 @@ add_clang_library(clangFrontend
   TextDiagnosticPrinter.cpp
   VerifyDiagnosticConsumer.cpp
   InterfaceStubFunctionsConsumer.cpp
+  CodeSnippetHighlighter.cpp
 
   DEPENDS
   ClangDriverOptions
diff --git a/clang/lib/Frontend/CodeSnippetHighlighter.cpp 
b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
new file mode 100644
index 000..829a533ad2692e5
--- /dev/null
+++ b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
@@ -0,0 +1,120 @@
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+
+void CodeSnippetHighlighter::ensureTokenData() {
+  if (Initialized)
+return;
+
+  // List of keywords, literals and types we want to highlight.
+  // These are best-effort, as is everything we do wrt. highlighting.
+  Keywords.insert("_Static_assert");
+  Keywords.insert("auto");
+  Keywords.insert("concept");
+  Keywords.insert("const");
+  Keywords.insert("consteval");
+  

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -0,0 +1,185 @@
+//===-- CodeSnippetHighlighter.cpp - Code snippet highlighting --*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/CharInfo.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+using namespace clang;
+
+// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
+// is already taken for 'note'. Green is already used to underline
+// source ranges. White and black are bad because of the usual
+// terminal backgrounds. Which leaves us only with TWO options.
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::YELLOW;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+/// Maximum size of file we still highlight.
+static constexpr size_t MaxBufferSize = 1024 * 1024; // 1MB.
+
+llvm::SmallVector CodeSnippetHighlighter::highlightLine(
+unsigned LineNumber, const Preprocessor *PP, const LangOptions ,
+FileID FID, const SourceManager , const char *LineStart) {
+  std::chrono::steady_clock::time_point begin =
+  std::chrono::steady_clock::now();
+
+  if (!PP)
+return {};
+

tbaederr wrote:

`Preprocessor*` is what's passed to `TextDiagnosticPrinter::BeginSourceFile()`, 
and there are a couple of places that pass `nullptr`, here an excerpt from `ag`:
```
../clang/lib/Frontend/FrontendAction.cpp
580:bool FrontendAction::BeginSourceFile(CompilerInstance ,
699:CI.getDiagnosticClient().BeginSourceFile(CI.getLangOpts(), nullptr);
715:if (!BeginSourceFileAction(CI))
759:CI.getDiagnosticClient().BeginSourceFile(CI.getLangOpts(), nullptr);
763:if (!BeginSourceFileAction(CI))
815:  CI.getDiagnosticClient().BeginSourceFile(CI.getLangOpts(),
914:  if (!BeginSourceFileAction(CI))
1207:bool WrapperFrontendAction::BeginSourceFileAction(CompilerInstance ) {
1210:  auto Ret = WrappedAction->BeginSourceFileAction(CI);
1211:  // BeginSourceFileAction may change CurrentInput, e.g. during module 
builds.

``

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -0,0 +1,187 @@
+//===-- CodeSnippetHighlighter.cpp - Code snippet highlighting --*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+using namespace clang;
+
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::CYAN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+
+llvm::SmallVector CodeSnippetHighlighter::highlightLine(
+unsigned LineNumber, const Preprocessor *PP, const LangOptions ,
+FileID FID, const SourceManager , const char *LineStart) {
+  std::chrono::steady_clock::time_point begin =
+  std::chrono::steady_clock::now();
+
+  if (!PP)
+return {};
+
+  // Might cause emission of another diagnostic.
+  if (PP->getIdentifierTable().getExternalIdentifierLookup())
+return {};
+
+  size_t NTokens = 0;
+  // Classify the given token and append it to the given vector.
+  auto appendStyle = [PP, ](llvm::SmallVector ,
+ const Token , unsigned Start,
+ unsigned Length) -> void {
+if (T.is(tok::raw_identifier)) {
+  StringRef RawIdent = T.getRawIdentifier();
+  // Special case true/false/nullptr literals, since they will otherwise be
+  // treated as keywords.
+  if (RawIdent == "true" || RawIdent == "false" || RawIdent == "nullptr") {
+Vec.push_back(StyleRange{Start, Start + Length, LiteralColor});
+  } else {
+const IdentifierInfo *II = PP->getIdentifierInfo(RawIdent);
+assert(II);
+if (II->isKeyword(LangOpts))
+  Vec.push_back(StyleRange{Start, Start + Length, KeywordColor});
+  }
+} else if (tok::isLiteral(T.getKind())) {
+  Vec.push_back(StyleRange{Start, Start + Length, LiteralColor});
+} else {
+  assert(T.is(tok::comment));
+  Vec.push_back(StyleRange{Start, Start + Length, CommentColor});
+}
+  };
+
+  // Figure out where to start lexing from.
+  auto Buff = SM.getBufferOrNone(FID);
+  assert(Buff);
+  Lexer L = Lexer(FID, *Buff, SM, LangOpts);
+  L.SetKeepWhitespaceMode(true);
+
+  // Seek to the last save point before the start of the line.
+  if (const char *Save = PP->getSaveFor(LineStart);
+  Buff->getBufferStart() <= Save && Save < Buff->getBufferEnd()) {
+size_t Offset = Save - Buff->getBufferStart();
+assert(Save >= Buff->getBufferStart());
+assert(Save <= Buff->getBufferEnd());
+
+L.seek(Offset, /*IsAtStartOfLine=*/true);
+  }
+
+  llvm::SmallVector LineRanges;
+  bool Stop = false;
+  while (!Stop) {
+++NTokens;
+Token T;
+Stop = L.LexFromRawLexer(T);
+if (T.is(tok::unknown))
+  continue;
+
+// We are only interested in identifiers, literals and comments.
+if (!T.is(tok::raw_identifier) && !T.is(tok::comment) &&
+!tok::isLiteral(T.getKind()))
+  continue;
+
+bool Invalid = false;
+unsigned EndLine = SM.getSpellingLineNumber(T.getEndLoc(), ) - 1;
+if (Invalid)
+  continue;
+
+if (EndLine < LineNumber)
+  continue;
+unsigned StartLine =
+SM.getSpellingLineNumber(T.getLocation(), ) - 1;
+if (Invalid)
+  continue;
+if (StartLine > LineNumber)
+  break;
+
+// Must have an intersection at this point
+assert(StartLine <= LineNumber && EndLine >= LineNumber);
+
+unsigned StartCol =
+SM.getSpellingColumnNumber(T.getLocation(), ) - 1;
+if (Invalid)
+  continue;
+
+// Simple tokens.
+if (StartLine == EndLine) {
+  appendStyle(LineRanges, T, StartCol, T.getLength());
+  continue;
+}
+unsigned NumLines = EndLine - StartLine;
+assert(NumLines >= 1);
+
+// For tokens that span multiple lines (think multiline comments), we
+// divide them into multiple StyleRanges.
+unsigned EndCol = SM.getSpellingColumnNumber(T.getEndLoc(), ) - 1;
+if (Invalid)
+  

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -0,0 +1,185 @@
+//===-- CodeSnippetHighlighter.cpp - Code snippet highlighting --*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/CharInfo.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+using namespace clang;
+
+// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
+// is already taken for 'note'. Green is already used to underline
+// source ranges. White and black are bad because of the usual
+// terminal backgrounds. Which leaves us only with TWO options.
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::YELLOW;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+/// Maximum size of file we still highlight.
+static constexpr size_t MaxBufferSize = 1024 * 1024; // 1MB.
+
+llvm::SmallVector CodeSnippetHighlighter::highlightLine(
+unsigned LineNumber, const Preprocessor *PP, const LangOptions ,
+FileID FID, const SourceManager , const char *LineStart) {
+  std::chrono::steady_clock::time_point begin =
+  std::chrono::steady_clock::now();
+
+  if (!PP)
+return {};
+

AaronBallman wrote:

Under what circumstances is `PP` null? Can we turn this into an assert? Or 
better yet, can we turn it into a const reference?

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -0,0 +1,187 @@
+//===-- CodeSnippetHighlighter.cpp - Code snippet highlighting --*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+using namespace clang;
+
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::CYAN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+
+llvm::SmallVector CodeSnippetHighlighter::highlightLine(
+unsigned LineNumber, const Preprocessor *PP, const LangOptions ,
+FileID FID, const SourceManager , const char *LineStart) {
+  std::chrono::steady_clock::time_point begin =
+  std::chrono::steady_clock::now();
+
+  if (!PP)
+return {};
+
+  // Might cause emission of another diagnostic.
+  if (PP->getIdentifierTable().getExternalIdentifierLookup())
+return {};
+
+  size_t NTokens = 0;
+  // Classify the given token and append it to the given vector.
+  auto appendStyle = [PP, ](llvm::SmallVector ,
+ const Token , unsigned Start,
+ unsigned Length) -> void {
+if (T.is(tok::raw_identifier)) {
+  StringRef RawIdent = T.getRawIdentifier();
+  // Special case true/false/nullptr literals, since they will otherwise be
+  // treated as keywords.
+  if (RawIdent == "true" || RawIdent == "false" || RawIdent == "nullptr") {
+Vec.push_back(StyleRange{Start, Start + Length, LiteralColor});
+  } else {
+const IdentifierInfo *II = PP->getIdentifierInfo(RawIdent);
+assert(II);
+if (II->isKeyword(LangOpts))
+  Vec.push_back(StyleRange{Start, Start + Length, KeywordColor});
+  }
+} else if (tok::isLiteral(T.getKind())) {
+  Vec.push_back(StyleRange{Start, Start + Length, LiteralColor});
+} else {
+  assert(T.is(tok::comment));
+  Vec.push_back(StyleRange{Start, Start + Length, CommentColor});
+}
+  };
+
+  // Figure out where to start lexing from.
+  auto Buff = SM.getBufferOrNone(FID);
+  assert(Buff);
+  Lexer L = Lexer(FID, *Buff, SM, LangOpts);
+  L.SetKeepWhitespaceMode(true);
+
+  // Seek to the last save point before the start of the line.
+  if (const char *Save = PP->getSaveFor(LineStart);
+  Buff->getBufferStart() <= Save && Save < Buff->getBufferEnd()) {
+size_t Offset = Save - Buff->getBufferStart();
+assert(Save >= Buff->getBufferStart());
+assert(Save <= Buff->getBufferEnd());
+
+L.seek(Offset, /*IsAtStartOfLine=*/true);
+  }
+
+  llvm::SmallVector LineRanges;
+  bool Stop = false;
+  while (!Stop) {
+++NTokens;
+Token T;
+Stop = L.LexFromRawLexer(T);
+if (T.is(tok::unknown))
+  continue;
+
+// We are only interested in identifiers, literals and comments.
+if (!T.is(tok::raw_identifier) && !T.is(tok::comment) &&
+!tok::isLiteral(T.getKind()))
+  continue;
+
+bool Invalid = false;
+unsigned EndLine = SM.getSpellingLineNumber(T.getEndLoc(), ) - 1;
+if (Invalid)
+  continue;
+
+if (EndLine < LineNumber)
+  continue;
+unsigned StartLine =
+SM.getSpellingLineNumber(T.getLocation(), ) - 1;
+if (Invalid)
+  continue;
+if (StartLine > LineNumber)
+  break;
+
+// Must have an intersection at this point
+assert(StartLine <= LineNumber && EndLine >= LineNumber);
+
+unsigned StartCol =
+SM.getSpellingColumnNumber(T.getLocation(), ) - 1;
+if (Invalid)
+  continue;
+
+// Simple tokens.
+if (StartLine == EndLine) {
+  appendStyle(LineRanges, T, StartCol, T.getLength());
+  continue;
+}
+unsigned NumLines = EndLine - StartLine;
+assert(NumLines >= 1);
+
+// For tokens that span multiple lines (think multiline comments), we
+// divide them into multiple StyleRanges.
+unsigned EndCol = SM.getSpellingColumnNumber(T.getEndLoc(), ) - 1;
+if (Invalid)
+  

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -0,0 +1,185 @@
+//===-- CodeSnippetHighlighter.cpp - Code snippet highlighting --*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/CharInfo.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+using namespace clang;
+
+// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
+// is already taken for 'note'. Green is already used to underline
+// source ranges. White and black are bad because of the usual
+// terminal backgrounds. Which leaves us only with TWO options.
+static constexpr raw_ostream::Colors CommentColor = raw_ostream::YELLOW;
+static constexpr raw_ostream::Colors LiteralColor = raw_ostream::GREEN;
+static constexpr raw_ostream::Colors KeywordColor = raw_ostream::BLUE;
+/// Maximum size of file we still highlight.
+static constexpr size_t MaxBufferSize = 1024 * 1024; // 1MB.
+
+llvm::SmallVector CodeSnippetHighlighter::highlightLine(
+unsigned LineNumber, const Preprocessor *PP, const LangOptions ,
+FileID FID, const SourceManager , const char *LineStart) {
+  std::chrono::steady_clock::time_point begin =
+  std::chrono::steady_clock::now();
+
+  if (!PP)
+return {};
+
+  // Might cause emission of another diagnostic.
+  if (PP->getIdentifierTable().getExternalIdentifierLookup())
+return {};
+
+  auto Buff = SM.getBufferOrNone(FID);
+  if (!Buff || Buff->getBufferSize() > MaxBufferSize)
+return {};
+
+  Lexer L = Lexer(FID, *Buff, SM, LangOpts);

AaronBallman wrote:

```suggestion
  Lexer L{FID, *Buff, SM, LangOpts};
```

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= ,
Timm =?utf-8?q?B=C3=A4der?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

Ping

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 8e0d084529d159ee80dd3971f9a5bb112e24bb86 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH 01/21] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 .../clang/Frontend/CodeSnippetHighlighter.h   |  46 +++
 clang/include/clang/Frontend/TextDiagnostic.h |   2 +
 clang/lib/Frontend/CMakeLists.txt |   1 +
 clang/lib/Frontend/CodeSnippetHighlighter.cpp | 120 ++
 clang/lib/Frontend/TextDiagnostic.cpp |  26 
 5 files changed, 195 insertions(+)
 create mode 100644 clang/include/clang/Frontend/CodeSnippetHighlighter.h
 create mode 100644 clang/lib/Frontend/CodeSnippetHighlighter.cpp

diff --git a/clang/include/clang/Frontend/CodeSnippetHighlighter.h 
b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
new file mode 100644
index 000..776954b59e2e1a8
--- /dev/null
+++ b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
@@ -0,0 +1,46 @@
+//===--- CodeSnippetHighlighter.h - Code snippet highlighting ---*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+#define LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+
+#include "clang/Basic/LangOptions.h"
+#include "llvm/ADT/SmallSet.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+namespace clang {
+
+struct StyleRange {
+  unsigned Start;
+  unsigned End;
+  const enum llvm::raw_ostream::Colors c;
+};
+
+class CodeSnippetHighlighter final {
+public:
+  CodeSnippetHighlighter() = default;
+
+  /// Produce StyleRanges for the given line.
+  /// The returned vector contains non-overlapping style ranges. They are 
sorted
+  /// from beginning of the line to the end.
+  std::vector highlightLine(llvm::StringRef SourceLine,
+const LangOptions );
+
+private:
+  bool Initialized = false;
+  /// Fills Keywords and Literals.
+  void ensureTokenData();
+
+  llvm::SmallSet Keywords;
+  llvm::SmallSet Literals;
+};
+
+} // namespace clang
+
+#endif
diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca8..59fd4d4f9408d48 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -15,6 +15,7 @@
 #ifndef LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
+#include "clang/Frontend/CodeSnippetHighlighter.h"
 #include "clang/Frontend/DiagnosticRenderer.h"
 
 namespace clang {
@@ -33,6 +34,7 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  CodeSnippetHighlighter SnippetHighlighter;
 
 public:
   TextDiagnostic(raw_ostream ,
diff --git a/clang/lib/Frontend/CMakeLists.txt 
b/clang/lib/Frontend/CMakeLists.txt
index 1e5f0a859dfd568..f3547f771593093 100644
--- a/clang/lib/Frontend/CMakeLists.txt
+++ b/clang/lib/Frontend/CMakeLists.txt
@@ -42,6 +42,7 @@ add_clang_library(clangFrontend
   TextDiagnosticPrinter.cpp
   VerifyDiagnosticConsumer.cpp
   InterfaceStubFunctionsConsumer.cpp
+  CodeSnippetHighlighter.cpp
 
   DEPENDS
   ClangDriverOptions
diff --git a/clang/lib/Frontend/CodeSnippetHighlighter.cpp 
b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
new file mode 100644
index 000..829a533ad2692e5
--- /dev/null
+++ b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
@@ -0,0 +1,120 @@
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+
+void CodeSnippetHighlighter::ensureTokenData() {
+  if (Initialized)
+return;
+
+  // List of keywords, literals and types we want to highlight.
+  // These are best-effort, as is everything we do wrt. highlighting.
+  Keywords.insert("_Static_assert");
+  Keywords.insert("auto");
+  Keywords.insert("concept");
+  Keywords.insert("const");
+  Keywords.insert("consteval");
+  

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

> > I've already switched to the max file size variant now, is that alright or 
> > do you want me to pursue the checkpoint version again?
> 
> Oh wow, there was a failure of communication somewhere. Having checkpoints, 
> AND disabling it entirely for large files is what i thought we were going to 
> do.

No, my point was that we can do it for "small" (but actually large) source 
files in a few milliseconds per diagnostic, so not a problem, but for larger 
files (generated or from a unity build) files, it might become too slow. So if 
we just limit ourselves to, let's say, 1MB files, then it's feasible. The 
implementation of this is easier, and, most importantly, it has no negative 
effect on lexing.

The check point approach makes regular lexing slower - even if we reserve the 
1000 checkpoints as you mentioned, we still add more overhead for ever token we 
lex (how much overhead depends of course).

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


cor3ntin wrote:

> I've already switched to the max file size variant now, is that alright or do 
> you want me to pursue the checkpoint version again?

Oh wow, there was a failure of communication somewhere.
Having checkpoints, AND disabling it entirely for large files is what i thought 
we were going to do.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

> > Kind of - [this 
> > run](http://llvm-compile-time-tracker.com/compare.php?from=6ade5183232dc1398205d7c9dbe21243b2560837=5ad516d39b6706af6fd55eded264aba4ba432ad2=instructions:u)
> >  only shows noticeable slowdown in one test. But we also only highlight the 
> > main file right now - that means we don't get syntax highlighting in 
> > included files and implementing that would probably make it a bit slower as 
> > well since we'd have to map the save points to their FileID.
> 
> I think we should do it for includes to. Did you play with `reserve`? I think 
> it would be reasonable to preallocate maybe ~1k check points - which should 
> just be 8kb per file, which would be reasonable

I've already switched to the max file size variant now, is that alright or do 
you want me to pursue the checkpoint version again?

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


AaronBallman wrote:

> > Regardless of the approach we take in the end - what do you think about 
> > tests for this? Should we have a flag similar to 
> > `-fdiagnostics-print-source-range-info` for the code style ranges?
> 
> I have no opinion. @AaronBallman ?

I could have sworn we already had one with `-fcolor-diagnostics 
-fansi-escape-codes` and piping the output to a file instead of the terminal? 
e.g.,
```
F:\source\llvm-project>llvm\out\build\x64-Debug\bin\clang.exe -Weverything 
-fsyntax-only -fansi-escape-codes -fcolor-diagnostics 
"C:\Users\aballman\OneDrive - Intel Corporation\Desktop\test.cpp" > foo.txt 2>&1

F:\source\llvm-project>cat foo.txt
←[1mC:\Users\aballman\OneDrive - Intel Corporation\Desktop\test.cpp:1:6: 
←[0m←[0;1;35mwarning: ←[0m←[1mno previous prototype for function 'func' 
[-Wmissing-prototypes]←[0m
1 | void func(int i, unsigned u) {←[0m
  | ←[0;1;32m ^
←[0m←[1mC:\Users\aballman\OneDrive - Intel Corporation\Desktop\test.cpp:1:1: 
←[0m←[0;1;36mnote: ←[0mdeclare 'static' if the function is not intended to be 
used outside of this translation unit←[0m
1 | void func(int i, unsigned u) {←[0m
  | ←[0;1;32m^
←[0m  | ←[0;32mstatic
←[0m←[1mC:\Users\aballman\OneDrive - Intel Corporation\Desktop\test.cpp:2:9: 
←[0m←[0;1;35mwarning: ←[0m←[1mcomparison of integers of different signs: 'int' 
and 'unsigned int' [-Wsign-compare]←[0m
2 |   if (i < u) return;←[0m
  | ←[0;1;32m  ~ ^ ~
←[0m2 warnings generated.
```
Would it work to reuse that?

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -0,0 +1,185 @@
+//===-- CodeSnippetHighlighter.cpp - Code snippet highlighting --*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/CharInfo.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+using namespace clang;
+
+// Magenta is taken for 'warning'. Red is already 'error' and 'cya'
+// is already taken for 'note'. Green is already used to underline
+// source ranges. White and black are bad because of the usual
+// terminal backgrounds. Which leaves us only with TWO options.

tbaederr wrote:

We have three colors, but one of them must be one that's already taken. Right 
now that's green for literals :(

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -0,0 +1,185 @@
+//===-- CodeSnippetHighlighter.cpp - Code snippet highlighting --*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/CharInfo.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+using namespace clang;
+
+// Magenta is taken for 'warning'. Red is already 'error' and 'cya'

AaronBallman wrote:

```suggestion
// Magenta is taken for 'warning'. Red is already 'error' and 'cyan'
```

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 



@@ -0,0 +1,185 @@
+//===-- CodeSnippetHighlighter.cpp - Code snippet highlighting --*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/CharInfo.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+using namespace clang;
+
+// Magenta is taken for 'warning'. Red is already 'error' and 'cya'
+// is already taken for 'note'. Green is already used to underline
+// source ranges. White and black are bad because of the usual
+// terminal backgrounds. Which leaves us only with TWO options.

AaronBallman wrote:

Heh, if we only have two options, why do we have three below? :-)

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


cor3ntin wrote:

> Kind of - [this 
> run](http://llvm-compile-time-tracker.com/compare.php?from=6ade5183232dc1398205d7c9dbe21243b2560837=5ad516d39b6706af6fd55eded264aba4ba432ad2=instructions:u)
>  only shows noticeable slowdown in one test. But we also only highlight the 
> main file right now - that means we don't get syntax highlighting in included 
> files and implementing that would probably make it a bit slower as well since 
> we'd have to map the save points to their FileID.

I think we should do it for includes to.
Did you play with `reserve`? I think it would be reasonable to preallocate 
maybe ~1k check points - which should just be 8kb per file, which would be 
reasonable

> Regardless of the approach we take in the end - what do you think about tests 
> for this? Should we have a flag similar to 
> `-fdiagnostics-print-source-range-info` for the code style ranges?

I have no opinion. @AaronBallman ?


https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 8e0d084529d159ee80dd3971f9a5bb112e24bb86 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH 01/20] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 .../clang/Frontend/CodeSnippetHighlighter.h   |  46 +++
 clang/include/clang/Frontend/TextDiagnostic.h |   2 +
 clang/lib/Frontend/CMakeLists.txt |   1 +
 clang/lib/Frontend/CodeSnippetHighlighter.cpp | 120 ++
 clang/lib/Frontend/TextDiagnostic.cpp |  26 
 5 files changed, 195 insertions(+)
 create mode 100644 clang/include/clang/Frontend/CodeSnippetHighlighter.h
 create mode 100644 clang/lib/Frontend/CodeSnippetHighlighter.cpp

diff --git a/clang/include/clang/Frontend/CodeSnippetHighlighter.h 
b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
new file mode 100644
index 000..776954b59e2e1a8
--- /dev/null
+++ b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
@@ -0,0 +1,46 @@
+//===--- CodeSnippetHighlighter.h - Code snippet highlighting ---*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+#define LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+
+#include "clang/Basic/LangOptions.h"
+#include "llvm/ADT/SmallSet.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+namespace clang {
+
+struct StyleRange {
+  unsigned Start;
+  unsigned End;
+  const enum llvm::raw_ostream::Colors c;
+};
+
+class CodeSnippetHighlighter final {
+public:
+  CodeSnippetHighlighter() = default;
+
+  /// Produce StyleRanges for the given line.
+  /// The returned vector contains non-overlapping style ranges. They are 
sorted
+  /// from beginning of the line to the end.
+  std::vector highlightLine(llvm::StringRef SourceLine,
+const LangOptions );
+
+private:
+  bool Initialized = false;
+  /// Fills Keywords and Literals.
+  void ensureTokenData();
+
+  llvm::SmallSet Keywords;
+  llvm::SmallSet Literals;
+};
+
+} // namespace clang
+
+#endif
diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca8..59fd4d4f9408d48 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -15,6 +15,7 @@
 #ifndef LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
+#include "clang/Frontend/CodeSnippetHighlighter.h"
 #include "clang/Frontend/DiagnosticRenderer.h"
 
 namespace clang {
@@ -33,6 +34,7 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  CodeSnippetHighlighter SnippetHighlighter;
 
 public:
   TextDiagnostic(raw_ostream ,
diff --git a/clang/lib/Frontend/CMakeLists.txt 
b/clang/lib/Frontend/CMakeLists.txt
index 1e5f0a859dfd568..f3547f771593093 100644
--- a/clang/lib/Frontend/CMakeLists.txt
+++ b/clang/lib/Frontend/CMakeLists.txt
@@ -42,6 +42,7 @@ add_clang_library(clangFrontend
   TextDiagnosticPrinter.cpp
   VerifyDiagnosticConsumer.cpp
   InterfaceStubFunctionsConsumer.cpp
+  CodeSnippetHighlighter.cpp
 
   DEPENDS
   ClangDriverOptions
diff --git a/clang/lib/Frontend/CodeSnippetHighlighter.cpp 
b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
new file mode 100644
index 000..829a533ad2692e5
--- /dev/null
+++ b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
@@ -0,0 +1,120 @@
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+
+void CodeSnippetHighlighter::ensureTokenData() {
+  if (Initialized)
+return;
+
+  // List of keywords, literals and types we want to highlight.
+  // These are best-effort, as is everything we do wrt. highlighting.
+  Keywords.insert("_Static_assert");
+  Keywords.insert("auto");
+  Keywords.insert("concept");
+  Keywords.insert("const");
+  Keywords.insert("consteval");
+  Keywords.insert("constexpr");
+  

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

Regardless of the approach we take in the end - what do you think about tests 
for this? Should we have a flag similar to 
`-fdiagnostics-print-source-range-info` for the code style ranges?

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


https://github.com/tbaederr updated 
https://github.com/llvm/llvm-project/pull/66514

>From 8e0d084529d159ee80dd3971f9a5bb112e24bb86 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Timm=20B=C3=A4der?= 
Date: Fri, 15 Sep 2023 15:51:39 +0200
Subject: [PATCH 01/20] [clang][Diagnostics] Highlight code snippets

Add some primitive syntax highlighting to our code snippet output.
---
 .../clang/Frontend/CodeSnippetHighlighter.h   |  46 +++
 clang/include/clang/Frontend/TextDiagnostic.h |   2 +
 clang/lib/Frontend/CMakeLists.txt |   1 +
 clang/lib/Frontend/CodeSnippetHighlighter.cpp | 120 ++
 clang/lib/Frontend/TextDiagnostic.cpp |  26 
 5 files changed, 195 insertions(+)
 create mode 100644 clang/include/clang/Frontend/CodeSnippetHighlighter.h
 create mode 100644 clang/lib/Frontend/CodeSnippetHighlighter.cpp

diff --git a/clang/include/clang/Frontend/CodeSnippetHighlighter.h 
b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
new file mode 100644
index 000..776954b59e2e1a8
--- /dev/null
+++ b/clang/include/clang/Frontend/CodeSnippetHighlighter.h
@@ -0,0 +1,46 @@
+//===--- CodeSnippetHighlighter.h - Code snippet highlighting ---*- C++ 
-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+#define LLVM_CLANG_FRONTEND_CODESNIPPETHIGHLIGHTER_H
+
+#include "clang/Basic/LangOptions.h"
+#include "llvm/ADT/SmallSet.h"
+#include "llvm/Support/raw_ostream.h"
+#include 
+
+namespace clang {
+
+struct StyleRange {
+  unsigned Start;
+  unsigned End;
+  const enum llvm::raw_ostream::Colors c;
+};
+
+class CodeSnippetHighlighter final {
+public:
+  CodeSnippetHighlighter() = default;
+
+  /// Produce StyleRanges for the given line.
+  /// The returned vector contains non-overlapping style ranges. They are 
sorted
+  /// from beginning of the line to the end.
+  std::vector highlightLine(llvm::StringRef SourceLine,
+const LangOptions );
+
+private:
+  bool Initialized = false;
+  /// Fills Keywords and Literals.
+  void ensureTokenData();
+
+  llvm::SmallSet Keywords;
+  llvm::SmallSet Literals;
+};
+
+} // namespace clang
+
+#endif
diff --git a/clang/include/clang/Frontend/TextDiagnostic.h 
b/clang/include/clang/Frontend/TextDiagnostic.h
index 7eb0ab0cdc9bca8..59fd4d4f9408d48 100644
--- a/clang/include/clang/Frontend/TextDiagnostic.h
+++ b/clang/include/clang/Frontend/TextDiagnostic.h
@@ -15,6 +15,7 @@
 #ifndef LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 #define LLVM_CLANG_FRONTEND_TEXTDIAGNOSTIC_H
 
+#include "clang/Frontend/CodeSnippetHighlighter.h"
 #include "clang/Frontend/DiagnosticRenderer.h"
 
 namespace clang {
@@ -33,6 +34,7 @@ namespace clang {
 /// printing coming out of libclang.
 class TextDiagnostic : public DiagnosticRenderer {
   raw_ostream 
+  CodeSnippetHighlighter SnippetHighlighter;
 
 public:
   TextDiagnostic(raw_ostream ,
diff --git a/clang/lib/Frontend/CMakeLists.txt 
b/clang/lib/Frontend/CMakeLists.txt
index 1e5f0a859dfd568..f3547f771593093 100644
--- a/clang/lib/Frontend/CMakeLists.txt
+++ b/clang/lib/Frontend/CMakeLists.txt
@@ -42,6 +42,7 @@ add_clang_library(clangFrontend
   TextDiagnosticPrinter.cpp
   VerifyDiagnosticConsumer.cpp
   InterfaceStubFunctionsConsumer.cpp
+  CodeSnippetHighlighter.cpp
 
   DEPENDS
   ClangDriverOptions
diff --git a/clang/lib/Frontend/CodeSnippetHighlighter.cpp 
b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
new file mode 100644
index 000..829a533ad2692e5
--- /dev/null
+++ b/clang/lib/Frontend/CodeSnippetHighlighter.cpp
@@ -0,0 +1,120 @@
+
+#include "clang/Frontend/CodeSnippetHighlighter.h"
+#include "clang/Basic/DiagnosticOptions.h"
+#include "clang/Basic/SourceManager.h"
+#include "clang/Lex/Lexer.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace clang;
+
+void CodeSnippetHighlighter::ensureTokenData() {
+  if (Initialized)
+return;
+
+  // List of keywords, literals and types we want to highlight.
+  // These are best-effort, as is everything we do wrt. highlighting.
+  Keywords.insert("_Static_assert");
+  Keywords.insert("auto");
+  Keywords.insert("concept");
+  Keywords.insert("const");
+  Keywords.insert("consteval");
+  Keywords.insert("constexpr");
+  

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Timm Baeder via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


tbaederr wrote:

> > > I really like this direction, thanks!
> > > I left a few comments, i will do another pass later
> > 
> > 
> > So you'd accept the lexing slowdown instead of using a maximum file size 
> > for highlighting?
> 
> Do I understand correctly that with the latest change there is no noticeable 
> perf impact? Otherwise, yes I'm happy limiting to 2MB - it seems the biggest 
> files in clang are about 500K. So it should only disable highlighting on 
> generated files, which seems reasonable

Kind of  - [this 
run](http://llvm-compile-time-tracker.com/compare.php?from=6ade5183232dc1398205d7c9dbe21243b2560837=5ad516d39b6706af6fd55eded264aba4ba432ad2=instructions:u)
 only shows noticeable slowdown in one test. But we also only highlight the 
main file right now - that means we don't get syntax highlighting in included 
files and implementing that would probably make it a bit slower as well since 
we'd have to map the save points to their FileID.

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[via cfe-commits]

Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= ,
Timm =?utf-8?q?Bäder?= 
Message-ID:
In-Reply-To: 


cor3ntin wrote:

> > I really like this direction, thanks!
> > I left a few comments, i will do another pass later
> 
> So you'd accept the lexing slowdown instead of using a maximum file size for 
> highlighting?

Do I understand correctly that with the latest change there is no noticeable 
perf impact?
Otherwise, yes I'm happy limiting to 2MB - it seems the biggest files in clang 
are about 500K. So it should only disable highlighting on generated files, 
which seems reasonable

https://github.com/llvm/llvm-project/pull/66514
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits