[PATCH] D135356: [Format] Fix crash when hitting eof while lexing JS template string

[Gulfem Savrun Yeniceri via Phabricator via cfe-commits]

gulfem added a comment.

In D135356#3840428 , @sammccall wrote:

> Fixed by 4b53c00173163774d32125fbcae283a46a9a4b19 
>  I think.

It fixed the test error that we are seeing, thanks!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D135356/new/

https://reviews.llvm.org/D135356

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D135356: [Format] Fix crash when hitting eof while lexing JS template string

[Sam McCall via Phabricator via cfe-commits]

sammccall added a comment.

Fixed by 4b53c00173163774d32125fbcae283a46a9a4b19 
 I think.

(@kadircet suggested a possible second crash, I couldn't get it to crash so 
included the testcase with this patch. Turns out it does crash...)


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D135356/new/

https://reviews.llvm.org/D135356

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D135356: [Format] Fix crash when hitting eof while lexing JS template string

[Gulfem Savrun Yeniceri via Phabricator via cfe-commits]

gulfem added a comment.

We are seeing the same error reported above:

  FormatTests: clang/lib/Lex/Lexer.cpp:1151: SourceLocation 
clang::Lexer::getSourceLocation(const char *, unsigned int) const: Assertion 
`Loc >= BufferStart && Loc <= BufferEnd && "Location out of range for this 
buffer!"' failed.

https://luci-milo.appspot.com/ui/p/fuchsia/builders/toolchain.ci/clang-linux-x64/b8801040768684259873/overview


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D135356/new/

https://reviews.llvm.org/D135356

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D135356: [Format] Fix crash when hitting eof while lexing JS template string

[Nico Weber via Phabricator via cfe-commits]

thakis added a comment.

This breaks tests: http://45.33.8.238/linux/88341/step_7.txt

Please take a look and revert for now if it takes a while to fix.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D135356/new/

https://reviews.llvm.org/D135356

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D135356: [Format] Fix crash when hitting eof while lexing JS template string

[Sam McCall via Phabricator via cfe-commits]

This revision was landed with ongoing or failed builds.
This revision was automatically updated to reflect the committed changes.
Closed by commit rG882a05afa17f: [Format] Fix crash when hitting eof while 
lexing JS template string (authored by sammccall).

Changed prior to commit:
  https://reviews.llvm.org/D135356?vs=465699=465742#toc

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D135356/new/

https://reviews.llvm.org/D135356

Files:
  clang/lib/Format/FormatTokenLexer.cpp
  clang/unittests/Format/FormatTestJS.cpp


Index: clang/unittests/Format/FormatTestJS.cpp
===
--- clang/unittests/Format/FormatTestJS.cpp
+++ clang/unittests/Format/FormatTestJS.cpp
@@ -2145,6 +2145,8 @@
 
   // Crashed at some point.
   verifyFormat("}");
+  verifyFormat("`");
+  verifyFormat("`\\");
 }
 
 TEST_F(FormatTestJS, TaggedTemplateStrings) {
Index: clang/lib/Format/FormatTokenLexer.cpp
===
--- clang/lib/Format/FormatTokenLexer.cpp
+++ clang/lib/Format/FormatTokenLexer.cpp
@@ -760,6 +760,7 @@
   for (; Offset != Lex->getBuffer().end(); ++Offset) {
 if (Offset[0] == '`') {
   StateStack.pop();
+  ++Offset;
   break;
 }
 if (Offset[0] == '\\') {
@@ -768,12 +769,12 @@
Offset[1] == '{') {
   // '${' introduces an expression interpolation in the template string.
   StateStack.push(LexerState::NORMAL);
-  ++Offset;
+  Offset += 2;
   break;
 }
   }
 
-  StringRef LiteralText(TmplBegin, Offset - TmplBegin + 1);
+  StringRef LiteralText(TmplBegin, Offset - TmplBegin);
   BacktickToken->setType(TT_TemplateString);
   BacktickToken->Tok.setKind(tok::string_literal);
   BacktickToken->TokenText = LiteralText;
@@ -794,9 +795,7 @@
   StartColumn, Style.TabWidth, Encoding);
   }
 
-  SourceLocation loc = Offset < Lex->getBuffer().end()
-   ? Lex->getSourceLocation(Offset + 1)
-   : SourceMgr.getLocForEndOfFile(ID);
+  SourceLocation loc = Lex->getSourceLocation(Offset);
   resetLexer(SourceMgr.getFileOffset(loc));
 }
 


Index: clang/unittests/Format/FormatTestJS.cpp
===
--- clang/unittests/Format/FormatTestJS.cpp
+++ clang/unittests/Format/FormatTestJS.cpp
@@ -2145,6 +2145,8 @@
 
   // Crashed at some point.
   verifyFormat("}");
+  verifyFormat("`");
+  verifyFormat("`\\");
 }
 
 TEST_F(FormatTestJS, TaggedTemplateStrings) {
Index: clang/lib/Format/FormatTokenLexer.cpp
===
--- clang/lib/Format/FormatTokenLexer.cpp
+++ clang/lib/Format/FormatTokenLexer.cpp
@@ -760,6 +760,7 @@
   for (; Offset != Lex->getBuffer().end(); ++Offset) {
 if (Offset[0] == '`') {
   StateStack.pop();
+  ++Offset;
   break;
 }
 if (Offset[0] == '\\') {
@@ -768,12 +769,12 @@
Offset[1] == '{') {
   // '${' introduces an expression interpolation in the template string.
   StateStack.push(LexerState::NORMAL);
-  ++Offset;
+  Offset += 2;
   break;
 }
   }
 
-  StringRef LiteralText(TmplBegin, Offset - TmplBegin + 1);
+  StringRef LiteralText(TmplBegin, Offset - TmplBegin);
   BacktickToken->setType(TT_TemplateString);
   BacktickToken->Tok.setKind(tok::string_literal);
   BacktickToken->TokenText = LiteralText;
@@ -794,9 +795,7 @@
   StartColumn, Style.TabWidth, Encoding);
   }
 
-  SourceLocation loc = Offset < Lex->getBuffer().end()
-   ? Lex->getSourceLocation(Offset + 1)
-   : SourceMgr.getLocForEndOfFile(ID);
+  SourceLocation loc = Lex->getSourceLocation(Offset);
   resetLexer(SourceMgr.getFileOffset(loc));
 }
 
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D135356: [Format] Fix crash when hitting eof while lexing JS template string

[Sam McCall via Phabricator via cfe-commits]

sammccall marked an inline comment as done.
sammccall added inline comments.



Comment at: clang/lib/Format/FormatTokenLexer.cpp:767
 if (Offset[0] == '\\') {
   ++Offset; // Skip the escaped character.
 } else if (Offset + 1 < Lex->getBuffer().end() && Offset[0] == '$' &&

kadircet wrote:
> it's a separate concern, but feels like this could also trigger some crashes 
> (e.g. an input like "`\")
Good catch. I think you're right, but:
- I'm not sure
- that example doesn't crash under asan, and I didn't find a crashing one
- that pattern occurs a bunch more times in this file
- I really want to have an isolated minimal fix + test as this bug is causing 
an outage
- I don't really have spare time now to dive deeper into this


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D135356/new/

https://reviews.llvm.org/D135356

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D135356: [Format] Fix crash when hitting eof while lexing JS template string

[Kadir Cetinkaya via Phabricator via cfe-commits]

kadircet accepted this revision.
kadircet added a comment.
This revision is now accepted and ready to land.

thanks!




Comment at: clang/lib/Format/FormatTokenLexer.cpp:767
 if (Offset[0] == '\\') {
   ++Offset; // Skip the escaped character.
 } else if (Offset + 1 < Lex->getBuffer().end() && Offset[0] == '$' &&

it's a separate concern, but feels like this could also trigger some crashes 
(e.g. an input like "`\")



Comment at: clang/lib/Format/FormatTokenLexer.cpp:772
   StateStack.push(LexerState::NORMAL);
   ++Offset;
+  ++Offset;

nit: `Offset += 2;` instead?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D135356/new/

https://reviews.llvm.org/D135356

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D135356: [Format] Fix crash when hitting eof while lexing JS template string

[Sam McCall via Phabricator via cfe-commits]

sammccall created this revision.
sammccall added a reviewer: kadircet.
Herald added a project: All.
sammccall requested review of this revision.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

Different loop termination conditions resulted in confusion of whether
*Offset was intended to be inside or outside the token.
This ultimately led to constructing an out-of-range SourceLocation.

Fix by making Offset consistently point *after* the token.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D135356

Files:
  clang/lib/Format/FormatTokenLexer.cpp
  clang/unittests/Format/FormatTestJS.cpp


Index: clang/unittests/Format/FormatTestJS.cpp
===
--- clang/unittests/Format/FormatTestJS.cpp
+++ clang/unittests/Format/FormatTestJS.cpp
@@ -2145,6 +2145,7 @@
 
   // Crashed at some point.
   verifyFormat("}");
+  verifyFormat("`");
 }
 
 TEST_F(FormatTestJS, TaggedTemplateStrings) {
Index: clang/lib/Format/FormatTokenLexer.cpp
===
--- clang/lib/Format/FormatTokenLexer.cpp
+++ clang/lib/Format/FormatTokenLexer.cpp
@@ -760,6 +760,7 @@
   for (; Offset != Lex->getBuffer().end(); ++Offset) {
 if (Offset[0] == '`') {
   StateStack.pop();
+  ++Offset;
   break;
 }
 if (Offset[0] == '\\') {
@@ -769,11 +770,12 @@
   // '${' introduces an expression interpolation in the template string.
   StateStack.push(LexerState::NORMAL);
   ++Offset;
+  ++Offset;
   break;
 }
   }
 
-  StringRef LiteralText(TmplBegin, Offset - TmplBegin + 1);
+  StringRef LiteralText(TmplBegin, Offset - TmplBegin);
   BacktickToken->setType(TT_TemplateString);
   BacktickToken->Tok.setKind(tok::string_literal);
   BacktickToken->TokenText = LiteralText;
@@ -794,9 +796,7 @@
   StartColumn, Style.TabWidth, Encoding);
   }
 
-  SourceLocation loc = Offset < Lex->getBuffer().end()
-   ? Lex->getSourceLocation(Offset + 1)
-   : SourceMgr.getLocForEndOfFile(ID);
+  SourceLocation loc = Lex->getSourceLocation(Offset);
   resetLexer(SourceMgr.getFileOffset(loc));
 }
 


Index: clang/unittests/Format/FormatTestJS.cpp
===
--- clang/unittests/Format/FormatTestJS.cpp
+++ clang/unittests/Format/FormatTestJS.cpp
@@ -2145,6 +2145,7 @@
 
   // Crashed at some point.
   verifyFormat("}");
+  verifyFormat("`");
 }
 
 TEST_F(FormatTestJS, TaggedTemplateStrings) {
Index: clang/lib/Format/FormatTokenLexer.cpp
===
--- clang/lib/Format/FormatTokenLexer.cpp
+++ clang/lib/Format/FormatTokenLexer.cpp
@@ -760,6 +760,7 @@
   for (; Offset != Lex->getBuffer().end(); ++Offset) {
 if (Offset[0] == '`') {
   StateStack.pop();
+  ++Offset;
   break;
 }
 if (Offset[0] == '\\') {
@@ -769,11 +770,12 @@
   // '${' introduces an expression interpolation in the template string.
   StateStack.push(LexerState::NORMAL);
   ++Offset;
+  ++Offset;
   break;
 }
   }
 
-  StringRef LiteralText(TmplBegin, Offset - TmplBegin + 1);
+  StringRef LiteralText(TmplBegin, Offset - TmplBegin);
   BacktickToken->setType(TT_TemplateString);
   BacktickToken->Tok.setKind(tok::string_literal);
   BacktickToken->TokenText = LiteralText;
@@ -794,9 +796,7 @@
   StartColumn, Style.TabWidth, Encoding);
   }
 
-  SourceLocation loc = Offset < Lex->getBuffer().end()
-   ? Lex->getSourceLocation(Offset + 1)
-   : SourceMgr.getLocForEndOfFile(ID);
+  SourceLocation loc = Lex->getSourceLocation(Offset);
   resetLexer(SourceMgr.getFileOffset(loc));
 }
 
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits