[PATCH] D76663: [clangd] Support new semanticTokens request from LSP 3.16.

[Sam McCall via Phabricator via cfe-commits]

This revision was automatically updated to reflect the committed changes.
Closed by commit rG71177ac16801: [clangd] Support new semanticTokens request 
from LSP 3.16. (authored by sammccall).

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D76663/new/

https://reviews.llvm.org/D76663

Files:
  clang-tools-extra/clangd/ClangdLSPServer.cpp
  clang-tools-extra/clangd/ClangdLSPServer.h
  clang-tools-extra/clangd/ClangdServer.cpp
  clang-tools-extra/clangd/ClangdServer.h
  clang-tools-extra/clangd/Protocol.cpp
  clang-tools-extra/clangd/Protocol.h
  clang-tools-extra/clangd/SemanticHighlighting.cpp
  clang-tools-extra/clangd/SemanticHighlighting.h
  clang-tools-extra/clangd/test/initialize-params.test
  clang-tools-extra/clangd/test/semantic-tokens.test
  clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp

Index: clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
===
--- clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
+++ clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
@@ -720,6 +720,41 @@
   ASSERT_EQ(Counter.Count, 1);
 }
 
+TEST(SemanticHighlighting, toSemanticTokens) {
+  auto CreatePosition = [](int Line, int Character) -> Position {
+Position Pos;
+Pos.line = Line;
+Pos.character = Character;
+return Pos;
+  };
+
+  std::vector Tokens = {
+  {HighlightingKind::Variable,
+   Range{CreatePosition(1, 1), CreatePosition(1, 5)}},
+  {HighlightingKind::Function,
+   Range{CreatePosition(3, 4), CreatePosition(3, 7)}},
+  {HighlightingKind::Variable,
+   Range{CreatePosition(3, 8), CreatePosition(3, 12)}},
+  };
+
+  std::vector Results = toSemanticTokens(Tokens);
+  EXPECT_EQ(Tokens.size(), Results.size());
+  EXPECT_EQ(Results[0].tokenType, unsigned(HighlightingKind::Variable));
+  EXPECT_EQ(Results[0].deltaLine, 1u);
+  EXPECT_EQ(Results[0].deltaStart, 1u);
+  EXPECT_EQ(Results[0].length, 4u);
+
+  EXPECT_EQ(Results[1].tokenType, unsigned(HighlightingKind::Function));
+  EXPECT_EQ(Results[1].deltaLine, 2u);
+  EXPECT_EQ(Results[1].deltaStart, 4u);
+  EXPECT_EQ(Results[1].length, 3u);
+
+  EXPECT_EQ(Results[2].tokenType, unsigned(HighlightingKind::Variable));
+  EXPECT_EQ(Results[2].deltaLine, 0u);
+  EXPECT_EQ(Results[2].deltaStart, 4u);
+  EXPECT_EQ(Results[2].length, 4u);
+}
+
 TEST(SemanticHighlighting, toTheiaSemanticHighlightingInformation) {
   auto CreatePosition = [](int Line, int Character) -> Position {
 Position Pos;
Index: clang-tools-extra/clangd/test/semantic-tokens.test
===
--- /dev/null
+++ clang-tools-extra/clangd/test/semantic-tokens.test
@@ -0,0 +1,22 @@
+# RUN: clangd -lit-test < %s | FileCheck -strict-whitespace %s
+{"jsonrpc":"2.0","id":0,"method":"initialize","params":{}}
+---
+{"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"test:///foo.cpp","languageId":"cpp","text":"int x = 2;"}}}
+---
+{"jsonrpc":"2.0","id":1,"method":"textDocument/semanticTokens","params":{"textDocument":{"uri":"test:///foo.cpp"}}}
+# CHECK:   "id": 1,
+# CHECK-NEXT:  "jsonrpc": "2.0",
+# CHECK-NEXT:  "result": {
+# CHECK-NEXT:"data": [
+#  First line, char 5, variable, no modifiers.
+# CHECK-NEXT:  0,
+# CHECK-NEXT:  4,
+# CHECK-NEXT:  1,
+# CHECK-NEXT:  0,
+# CHECK-NEXT:  0
+# CHECK-NEXT:]
+# CHECK-NEXT:  }
+---
+{"jsonrpc":"2.0","id":2,"method":"shutdown"}
+---
+{"jsonrpc":"2.0","method":"exit"}
Index: clang-tools-extra/clangd/test/initialize-params.test
===
--- clang-tools-extra/clangd/test/initialize-params.test
+++ clang-tools-extra/clangd/test/initialize-params.test
@@ -38,6 +38,16 @@
 # CHECK-NEXT:  "referencesProvider": true,
 # CHECK-NEXT:  "renameProvider": true,
 # CHECK-NEXT:  "selectionRangeProvider": true,
+# CHECK-NEXT:  "semanticTokensProvider": {
+# CHECK-NEXT:"documentProvider": true,
+# CHECK-NEXT:"legend": {
+# CHECK-NEXT:  "tokenModifiers": [],
+# CHECK-NEXT:  "tokenTypes": [
+# CHECK-NEXT:"variable",
+# CHECK:   ]
+# CHECK-NEXT:},
+# CHECK-NEXT:"rangeProvider": false
+# CHECK-NEXT:  },
 # CHECK-NEXT:  "signatureHelpProvider": {
 # CHECK-NEXT:"triggerCharacters": [
 # CHECK-NEXT:  "(",
Index: clang-tools-extra/clangd/SemanticHighlighting.h
===
--- clang-tools-extra/clangd/SemanticHighlighting.h
+++ clang-tools-extra/clangd/SemanticHighlighting.h
@@ -6,8 +6,21 @@
 //
 //===--===//
 //
-// An implementation of semantic highlighting based on this proposal:
-// https://github.com/microsoft/vscode-languageserver-node/pull/367 in clangd.
+// 

[PATCH] D76663: [clangd] Support new semanticTokens request from LSP 3.16.

[Haojian Wu via Phabricator via cfe-commits]

hokein accepted this revision.
hokein added a comment.
This revision is now accepted and ready to land.

Thanks!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D76663/new/

https://reviews.llvm.org/D76663



___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D76663: [clangd] Support new semanticTokens request from LSP 3.16.

[Sam McCall via Phabricator via cfe-commits]

sammccall added inline comments.



Comment at: clang-tools-extra/clangd/SemanticHighlighting.cpp:452
+  const HighlightingToken *Last = nullptr;
+  for (const HighlightingToken  : Tokens) {
+Result.emplace_back();

hokein wrote:
> note that we don't calculate the column offset for `InactiveCode` token 
> (`{(line, 0), (line, 0)}`), I think we probably calculate the range with the 
> new implementation, maybe add a FIXME.
Right, yes. Just skipping over them for now, with a fixme.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D76663/new/

https://reviews.llvm.org/D76663



___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D76663: [clangd] Support new semanticTokens request from LSP 3.16.

[Sam McCall via Phabricator via cfe-commits]

sammccall updated this revision to Diff 252906.
sammccall marked 6 inline comments as done.
sammccall added a comment.

Address review comments.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D76663/new/

https://reviews.llvm.org/D76663

Files:
  clang-tools-extra/clangd/ClangdLSPServer.cpp
  clang-tools-extra/clangd/ClangdLSPServer.h
  clang-tools-extra/clangd/ClangdServer.cpp
  clang-tools-extra/clangd/ClangdServer.h
  clang-tools-extra/clangd/Protocol.cpp
  clang-tools-extra/clangd/Protocol.h
  clang-tools-extra/clangd/SemanticHighlighting.cpp
  clang-tools-extra/clangd/SemanticHighlighting.h
  clang-tools-extra/clangd/test/initialize-params.test
  clang-tools-extra/clangd/test/semantic-tokens.test
  clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp

Index: clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
===
--- clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
+++ clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
@@ -720,6 +720,41 @@
   ASSERT_EQ(Counter.Count, 1);
 }
 
+TEST(SemanticHighlighting, toSemanticTokens) {
+  auto CreatePosition = [](int Line, int Character) -> Position {
+Position Pos;
+Pos.line = Line;
+Pos.character = Character;
+return Pos;
+  };
+
+  std::vector Tokens = {
+  {HighlightingKind::Variable,
+   Range{CreatePosition(1, 1), CreatePosition(1, 5)}},
+  {HighlightingKind::Function,
+   Range{CreatePosition(3, 4), CreatePosition(3, 7)}},
+  {HighlightingKind::Variable,
+   Range{CreatePosition(3, 8), CreatePosition(3, 12)}},
+  };
+
+  std::vector Results = toSemanticTokens(Tokens);
+  EXPECT_EQ(Tokens.size(), Results.size());
+  EXPECT_EQ(Results[0].tokenType, unsigned(HighlightingKind::Variable));
+  EXPECT_EQ(Results[0].deltaLine, 1u);
+  EXPECT_EQ(Results[0].deltaStart, 1u);
+  EXPECT_EQ(Results[0].length, 4u);
+
+  EXPECT_EQ(Results[1].tokenType, unsigned(HighlightingKind::Function));
+  EXPECT_EQ(Results[1].deltaLine, 2u);
+  EXPECT_EQ(Results[1].deltaStart, 4u);
+  EXPECT_EQ(Results[1].length, 3u);
+
+  EXPECT_EQ(Results[2].tokenType, unsigned(HighlightingKind::Variable));
+  EXPECT_EQ(Results[2].deltaLine, 0u);
+  EXPECT_EQ(Results[2].deltaStart, 4u);
+  EXPECT_EQ(Results[2].length, 4u);
+}
+
 TEST(SemanticHighlighting, toTheiaSemanticHighlightingInformation) {
   auto CreatePosition = [](int Line, int Character) -> Position {
 Position Pos;
Index: clang-tools-extra/clangd/test/semantic-tokens.test
===
--- /dev/null
+++ clang-tools-extra/clangd/test/semantic-tokens.test
@@ -0,0 +1,22 @@
+# RUN: clangd -lit-test < %s | FileCheck -strict-whitespace %s
+{"jsonrpc":"2.0","id":0,"method":"initialize","params":{}}
+---
+{"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"test:///foo.cpp","languageId":"cpp","text":"int x = 2;"}}}
+---
+{"jsonrpc":"2.0","id":1,"method":"textDocument/semanticTokens","params":{"textDocument":{"uri":"test:///foo.cpp"}}}
+# CHECK:   "id": 1,
+# CHECK-NEXT:  "jsonrpc": "2.0",
+# CHECK-NEXT:  "result": {
+# CHECK-NEXT:"data": [
+#  First line, char 5, variable, no modifiers.
+# CHECK-NEXT:  0,
+# CHECK-NEXT:  4,
+# CHECK-NEXT:  1,
+# CHECK-NEXT:  0,
+# CHECK-NEXT:  0
+# CHECK-NEXT:]
+# CHECK-NEXT:  }
+---
+{"jsonrpc":"2.0","id":2,"method":"shutdown"}
+---
+{"jsonrpc":"2.0","method":"exit"}
Index: clang-tools-extra/clangd/test/initialize-params.test
===
--- clang-tools-extra/clangd/test/initialize-params.test
+++ clang-tools-extra/clangd/test/initialize-params.test
@@ -38,6 +38,16 @@
 # CHECK-NEXT:  "referencesProvider": true,
 # CHECK-NEXT:  "renameProvider": true,
 # CHECK-NEXT:  "selectionRangeProvider": true,
+# CHECK-NEXT:  "semanticTokensProvider": {
+# CHECK-NEXT:"documentProvider": true,
+# CHECK-NEXT:"legend": {
+# CHECK-NEXT:  "tokenModifiers": [],
+# CHECK-NEXT:  "tokenTypes": [
+# CHECK-NEXT:"variable",
+# CHECK:   ]
+# CHECK-NEXT:},
+# CHECK-NEXT:"rangeProvider": false
+# CHECK-NEXT:  },
 # CHECK-NEXT:  "signatureHelpProvider": {
 # CHECK-NEXT:"triggerCharacters": [
 # CHECK-NEXT:  "(",
Index: clang-tools-extra/clangd/SemanticHighlighting.h
===
--- clang-tools-extra/clangd/SemanticHighlighting.h
+++ clang-tools-extra/clangd/SemanticHighlighting.h
@@ -6,8 +6,21 @@
 //
 //===--===//
 //
-// An implementation of semantic highlighting based on this proposal:
-// https://github.com/microsoft/vscode-languageserver-node/pull/367 in clangd.
+// This file supports semantic highlighting: 

[PATCH] D76663: [clangd] Support new semanticTokens request from LSP 3.16.

[Haojian Wu via Phabricator via cfe-commits]

hokein added a comment.

In D76663#1942273 , @sammccall wrote:

> For some context here, I was doing some digging as Heyward Fann is looking at 
> hooking up our existing syntax highlighting to coc.nvim, and he and Jack Guo 
> (of jackguo380/vim-lsp-cxx-highlight) were asking about the protocol.
>
> The new LSP protocol looks really solid, including better incremental 
> highlight support than the Theia proposal, though this patch doesn't 
> implement incremental yet. (In particular, no sending thousands of 
> re-highlights when inserting a new line). It's also request-response rather 
> than notification-based, which is easier to implement on the server side. 
> Also the VSCode client-side of our highlighting feels like significant 
> technical debt we could be rid of.
>
> So I think we should try to support the new LSP and drop the older Theia one 
> ASAP (clangd 12?), even if semantic highlighting isn't a really high priority 
> for us.


Thanks for the summary, sounds a good plan.

The patch is mostly good, just a few nits. I will try to play it with VSCode.




Comment at: clang-tools-extra/clangd/ClangdLSPServer.cpp:597
+ llvm::json::Object{
+ {"documentProvider", true},
+ {"legend",

nit: shall we explicitly set the `rangeProvider` to false as we don't support 
the semantic token range request now.



Comment at: clang-tools-extra/clangd/Protocol.h:1377
+};
+llvm::json::Value toJSON(const SemanticTokens );
+

nit: FStatus => Tokens.



Comment at: clang-tools-extra/clangd/SemanticHighlighting.cpp:449
+std::vector
+toSemanticTokens(llvm::ArrayRef Tokens) {
+  std::vector Result;

nit: assert the Tokens is sorted?



Comment at: clang-tools-extra/clangd/SemanticHighlighting.cpp:452
+  const HighlightingToken *Last = nullptr;
+  for (const HighlightingToken  : Tokens) {
+Result.emplace_back();

note that we don't calculate the column offset for `InactiveCode` token 
(`{(line, 0), (line, 0)}`), I think we probably calculate the range with the 
new implementation, maybe add a FIXME.



Comment at: clang-tools-extra/clangd/SemanticHighlighting.cpp:489
+return "member";
+  case HighlightingKind::StaticMethod:
+return "function";

It seems ok now, but I think for static method, we should set the modifier to 
`static`, maybe add a FIXME to support token modifiers


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D76663/new/

https://reviews.llvm.org/D76663



___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D76663: [clangd] Support new semanticTokens request from LSP 3.16.

[Sam McCall via Phabricator via cfe-commits]

sammccall added a comment.

For some context here, I was doing some digging as Heyward Fann is looking at 
hooking up our existing syntax highlighting to coc.nvim, and he and Jack Guo 
(of jackguo380/vim-lsp-cxx-highlight) were asking about the protocol.

The new LSP protocol looks really solid, including better incremental highlight 
support than the Theia proposal, though this patch doesn't implement 
incremental yet. (In particular, no sending thousands of re-highlights when 
inserting a new line). It's also request-response rather than 
notification-based, which is easier to implement on the server side. Also the 
VSCode client-side of our highlighting feels like significant technical debt we 
could be rid of.

So I think we should try to support the new LSP and drop the older Theia one 
ASAP (clangd 12?), even if semantic highlighting isn't a really high priority 
for us.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D76663/new/

https://reviews.llvm.org/D76663



___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D76663: [clangd] Support new semanticTokens request from LSP 3.16.

[Sam McCall via Phabricator via cfe-commits]

sammccall updated this revision to Diff 252676.
sammccall added a comment.

Update SemanticHighlighting comment


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D76663/new/

https://reviews.llvm.org/D76663

Files:
  clang-tools-extra/clangd/ClangdLSPServer.cpp
  clang-tools-extra/clangd/ClangdLSPServer.h
  clang-tools-extra/clangd/ClangdServer.cpp
  clang-tools-extra/clangd/ClangdServer.h
  clang-tools-extra/clangd/Protocol.cpp
  clang-tools-extra/clangd/Protocol.h
  clang-tools-extra/clangd/SemanticHighlighting.cpp
  clang-tools-extra/clangd/SemanticHighlighting.h
  clang-tools-extra/clangd/test/initialize-params.test
  clang-tools-extra/clangd/test/semantic-tokens.test
  clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp

Index: clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
===
--- clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
+++ clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
@@ -720,6 +720,41 @@
   ASSERT_EQ(Counter.Count, 1);
 }
 
+TEST(SemanticHighlighting, toSemanticTokens) {
+  auto CreatePosition = [](int Line, int Character) -> Position {
+Position Pos;
+Pos.line = Line;
+Pos.character = Character;
+return Pos;
+  };
+
+  std::vector Tokens = {
+  {HighlightingKind::Variable,
+   Range{CreatePosition(1, 1), CreatePosition(1, 5)}},
+  {HighlightingKind::Function,
+   Range{CreatePosition(3, 4), CreatePosition(3, 7)}},
+  {HighlightingKind::Variable,
+   Range{CreatePosition(3, 8), CreatePosition(3, 12)}},
+  };
+
+  std::vector Results = toSemanticTokens(Tokens);
+  EXPECT_EQ(Tokens.size(), Results.size());
+  EXPECT_EQ(Results[0].tokenType, unsigned(HighlightingKind::Variable));
+  EXPECT_EQ(Results[0].deltaLine, 1u);
+  EXPECT_EQ(Results[0].deltaStart, 1u);
+  EXPECT_EQ(Results[0].length, 4u);
+
+  EXPECT_EQ(Results[1].tokenType, unsigned(HighlightingKind::Function));
+  EXPECT_EQ(Results[1].deltaLine, 2u);
+  EXPECT_EQ(Results[1].deltaStart, 4u);
+  EXPECT_EQ(Results[1].length, 3u);
+
+  EXPECT_EQ(Results[2].tokenType, unsigned(HighlightingKind::Variable));
+  EXPECT_EQ(Results[2].deltaLine, 0u);
+  EXPECT_EQ(Results[2].deltaStart, 4u);
+  EXPECT_EQ(Results[2].length, 4u);
+}
+
 TEST(SemanticHighlighting, toTheiaSemanticHighlightingInformation) {
   auto CreatePosition = [](int Line, int Character) -> Position {
 Position Pos;
Index: clang-tools-extra/clangd/test/semantic-tokens.test
===
--- /dev/null
+++ clang-tools-extra/clangd/test/semantic-tokens.test
@@ -0,0 +1,22 @@
+# RUN: clangd -lit-test < %s | FileCheck -strict-whitespace %s
+{"jsonrpc":"2.0","id":0,"method":"initialize","params":{}}
+---
+{"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"test:///foo.cpp","languageId":"cpp","text":"int x = 2;"}}}
+---
+{"jsonrpc":"2.0","id":1,"method":"textDocument/semanticTokens","params":{"textDocument":{"uri":"test:///foo.cpp"}}}
+# CHECK:   "id": 1,
+# CHECK-NEXT:  "jsonrpc": "2.0",
+# CHECK-NEXT:  "result": {
+# CHECK-NEXT:"data": [
+#  First line, char 5, variable, no modifiers.
+# CHECK-NEXT:  0,
+# CHECK-NEXT:  4,
+# CHECK-NEXT:  1,
+# CHECK-NEXT:  0,
+# CHECK-NEXT:  0
+# CHECK-NEXT:]
+# CHECK-NEXT:  }
+---
+{"jsonrpc":"2.0","id":2,"method":"shutdown"}
+---
+{"jsonrpc":"2.0","method":"exit"}
Index: clang-tools-extra/clangd/test/initialize-params.test
===
--- clang-tools-extra/clangd/test/initialize-params.test
+++ clang-tools-extra/clangd/test/initialize-params.test
@@ -38,6 +38,15 @@
 # CHECK-NEXT:  "referencesProvider": true,
 # CHECK-NEXT:  "renameProvider": true,
 # CHECK-NEXT:  "selectionRangeProvider": true,
+# CHECK-NEXT:  "semanticTokensProvider": {
+# CHECK-NEXT:"documentProvider": true,
+# CHECK-NEXT:"legend": {
+# CHECK-NEXT:  "tokenModifiers": [],
+# CHECK-NEXT:  "tokenTypes": [
+# CHECK-NEXT:"variable",
+# CHECK:   ]
+# CHECK-NEXT:}
+# CHECK-NEXT:  },
 # CHECK-NEXT:  "signatureHelpProvider": {
 # CHECK-NEXT:"triggerCharacters": [
 # CHECK-NEXT:  "(",
Index: clang-tools-extra/clangd/SemanticHighlighting.h
===
--- clang-tools-extra/clangd/SemanticHighlighting.h
+++ clang-tools-extra/clangd/SemanticHighlighting.h
@@ -6,8 +6,21 @@
 //
 //===--===//
 //
-// An implementation of semantic highlighting based on this proposal:
-// https://github.com/microsoft/vscode-languageserver-node/pull/367 in clangd.
+// This file supports semantic highlighting: categorizing tokens in the file so
+// that the editor can color/style them 

[PATCH] D76663: [clangd] Support new semanticTokens request from LSP 3.16.

[Sam McCall via Phabricator via cfe-commits]

sammccall created this revision.
sammccall added a reviewer: hokein.
Herald added subscribers: cfe-commits, usaxena95, kadircet, arphaman, jkorous, 
MaskRay, ilya-biryukov.
Herald added a project: clang.

This is a simpler request/response protocol.

Reference: 
https://github.com/microsoft/vscode-languageserver-node/blob/master/protocol/src/protocol.semanticTokens.proposed.ts

No attempt to support incremental formatting (yet).


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D76663

Files:
  clang-tools-extra/clangd/ClangdLSPServer.cpp
  clang-tools-extra/clangd/ClangdLSPServer.h
  clang-tools-extra/clangd/ClangdServer.cpp
  clang-tools-extra/clangd/ClangdServer.h
  clang-tools-extra/clangd/Protocol.cpp
  clang-tools-extra/clangd/Protocol.h
  clang-tools-extra/clangd/SemanticHighlighting.cpp
  clang-tools-extra/clangd/SemanticHighlighting.h
  clang-tools-extra/clangd/test/initialize-params.test
  clang-tools-extra/clangd/test/semantic-tokens.test
  clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp

Index: clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
===
--- clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
+++ clang-tools-extra/clangd/unittests/SemanticHighlightingTests.cpp
@@ -720,6 +720,41 @@
   ASSERT_EQ(Counter.Count, 1);
 }
 
+TEST(SemanticHighlighting, toSemanticTokens) {
+  auto CreatePosition = [](int Line, int Character) -> Position {
+Position Pos;
+Pos.line = Line;
+Pos.character = Character;
+return Pos;
+  };
+
+  std::vector Tokens = {
+  {HighlightingKind::Variable,
+   Range{CreatePosition(1, 1), CreatePosition(1, 5)}},
+  {HighlightingKind::Function,
+   Range{CreatePosition(3, 4), CreatePosition(3, 7)}},
+  {HighlightingKind::Variable,
+   Range{CreatePosition(3, 8), CreatePosition(3, 12)}},
+  };
+
+  std::vector Results = toSemanticTokens(Tokens);
+  EXPECT_EQ(Tokens.size(), Results.size());
+  EXPECT_EQ(Results[0].tokenType, unsigned(HighlightingKind::Variable));
+  EXPECT_EQ(Results[0].deltaLine, 1u);
+  EXPECT_EQ(Results[0].deltaStart, 1u);
+  EXPECT_EQ(Results[0].length, 4u);
+
+  EXPECT_EQ(Results[1].tokenType, unsigned(HighlightingKind::Function));
+  EXPECT_EQ(Results[1].deltaLine, 2u);
+  EXPECT_EQ(Results[1].deltaStart, 4u);
+  EXPECT_EQ(Results[1].length, 3u);
+
+  EXPECT_EQ(Results[2].tokenType, unsigned(HighlightingKind::Variable));
+  EXPECT_EQ(Results[2].deltaLine, 0u);
+  EXPECT_EQ(Results[2].deltaStart, 4u);
+  EXPECT_EQ(Results[2].length, 4u);
+}
+
 TEST(SemanticHighlighting, toTheiaSemanticHighlightingInformation) {
   auto CreatePosition = [](int Line, int Character) -> Position {
 Position Pos;
Index: clang-tools-extra/clangd/test/semantic-tokens.test
===
--- /dev/null
+++ clang-tools-extra/clangd/test/semantic-tokens.test
@@ -0,0 +1,22 @@
+# RUN: clangd -lit-test < %s | FileCheck -strict-whitespace %s
+{"jsonrpc":"2.0","id":0,"method":"initialize","params":{}}
+---
+{"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"uri":"test:///foo.cpp","languageId":"cpp","text":"int x = 2;"}}}
+---
+{"jsonrpc":"2.0","id":1,"method":"textDocument/semanticTokens","params":{"textDocument":{"uri":"test:///foo.cpp"}}}
+# CHECK:   "id": 1,
+# CHECK-NEXT:  "jsonrpc": "2.0",
+# CHECK-NEXT:  "result": {
+# CHECK-NEXT:"data": [
+#  First line, char 5, variable, no modifiers.
+# CHECK-NEXT:  0,
+# CHECK-NEXT:  4,
+# CHECK-NEXT:  1,
+# CHECK-NEXT:  0,
+# CHECK-NEXT:  0
+# CHECK-NEXT:]
+# CHECK-NEXT:  }
+---
+{"jsonrpc":"2.0","id":2,"method":"shutdown"}
+---
+{"jsonrpc":"2.0","method":"exit"}
Index: clang-tools-extra/clangd/test/initialize-params.test
===
--- clang-tools-extra/clangd/test/initialize-params.test
+++ clang-tools-extra/clangd/test/initialize-params.test
@@ -38,6 +38,15 @@
 # CHECK-NEXT:  "referencesProvider": true,
 # CHECK-NEXT:  "renameProvider": true,
 # CHECK-NEXT:  "selectionRangeProvider": true,
+# CHECK-NEXT:  "semanticTokensProvider": {
+# CHECK-NEXT:"documentProvider": true,
+# CHECK-NEXT:"legend": {
+# CHECK-NEXT:  "tokenModifiers": [],
+# CHECK-NEXT:  "tokenTypes": [
+# CHECK-NEXT:"variable",
+# CHECK:   ]
+# CHECK-NEXT:}
+# CHECK-NEXT:  },
 # CHECK-NEXT:  "signatureHelpProvider": {
 # CHECK-NEXT:"triggerCharacters": [
 # CHECK-NEXT:  "(",
Index: clang-tools-extra/clangd/SemanticHighlighting.h
===
--- clang-tools-extra/clangd/SemanticHighlighting.h
+++ clang-tools-extra/clangd/SemanticHighlighting.h
@@ -75,6 +75,9 @@
 // main AST.
 std::vector getSemanticHighlightings(ParsedAST );
 
+std::vector toSemanticTokens(llvm::ArrayRef);