[PATCH] D125311: [pseudo] Share the underly payload when stripping comments for a token stream

[Haojian Wu via Phabricator via cfe-commits]

This revision was automatically updated to reflect the committed changes.
Closed by commit rG76910d4a56c8: [pseudo] Share the underly payload when 
stripping comments for a token stream (authored by hokein).

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D125311/new/

https://reviews.llvm.org/D125311

Files:
  clang-tools-extra/pseudo/include/clang-pseudo/Token.h
  clang-tools-extra/pseudo/lib/Lex.cpp
  clang-tools-extra/pseudo/lib/Token.cpp


Index: clang-tools-extra/pseudo/lib/Token.cpp
===
--- clang-tools-extra/pseudo/lib/Token.cpp
+++ clang-tools-extra/pseudo/lib/Token.cpp
@@ -116,7 +116,7 @@
 }
 
 TokenStream stripComments(const TokenStream ) {
-  TokenStream Out;
+  TokenStream Out(Input.getPayload());
   for (const Token  : Input.tokens()) {
 if (T.Kind == tok::comment)
   continue;
Index: clang-tools-extra/pseudo/lib/Lex.cpp
===
--- clang-tools-extra/pseudo/lib/Lex.cpp
+++ clang-tools-extra/pseudo/lib/Lex.cpp
@@ -77,7 +77,7 @@
   auto CleanedStorage = std::make_shared();
   clang::IdentifierTable Identifiers(LangOpts);
   TokenStream Result(CleanedStorage);
-
+  Result.addPayload(Code.getPayload());
   for (auto Tok : Code.tokens()) {
 if (Tok.flag(LexFlags::NeedsCleaning)) {
   // Remove escaped newlines and trigraphs.
Index: clang-tools-extra/pseudo/include/clang-pseudo/Token.h
===
--- clang-tools-extra/pseudo/include/clang-pseudo/Token.h
+++ clang-tools-extra/pseudo/include/clang-pseudo/Token.h
@@ -170,6 +170,18 @@
 return Storage[1];
   }
 
+  /// Returns the shared payload.
+  std::shared_ptr getPayload() const { return Payload; }
+  /// Adds the given payload to the stream.
+  void addPayload(std::shared_ptr P) {
+if (!Payload)
+  Payload = std::move(P);
+else
+  Payload = std::make_shared<
+  std::pair, std::shared_ptr>>(
+  std::move(P), std::move(Payload));
+  }
+
   /// Print the tokens in this stream to the output stream.
   ///
   /// The presence of newlines/spaces is preserved, but not the quantity.


Index: clang-tools-extra/pseudo/lib/Token.cpp
===
--- clang-tools-extra/pseudo/lib/Token.cpp
+++ clang-tools-extra/pseudo/lib/Token.cpp
@@ -116,7 +116,7 @@
 }
 
 TokenStream stripComments(const TokenStream ) {
-  TokenStream Out;
+  TokenStream Out(Input.getPayload());
   for (const Token  : Input.tokens()) {
 if (T.Kind == tok::comment)
   continue;
Index: clang-tools-extra/pseudo/lib/Lex.cpp
===
--- clang-tools-extra/pseudo/lib/Lex.cpp
+++ clang-tools-extra/pseudo/lib/Lex.cpp
@@ -77,7 +77,7 @@
   auto CleanedStorage = std::make_shared();
   clang::IdentifierTable Identifiers(LangOpts);
   TokenStream Result(CleanedStorage);
-
+  Result.addPayload(Code.getPayload());
   for (auto Tok : Code.tokens()) {
 if (Tok.flag(LexFlags::NeedsCleaning)) {
   // Remove escaped newlines and trigraphs.
Index: clang-tools-extra/pseudo/include/clang-pseudo/Token.h
===
--- clang-tools-extra/pseudo/include/clang-pseudo/Token.h
+++ clang-tools-extra/pseudo/include/clang-pseudo/Token.h
@@ -170,6 +170,18 @@
 return Storage[1];
   }
 
+  /// Returns the shared payload.
+  std::shared_ptr getPayload() const { return Payload; }
+  /// Adds the given payload to the stream.
+  void addPayload(std::shared_ptr P) {
+if (!Payload)
+  Payload = std::move(P);
+else
+  Payload = std::make_shared<
+  std::pair, std::shared_ptr>>(
+  std::move(P), std::move(Payload));
+  }
+
   /// Print the tokens in this stream to the output stream.
   ///
   /// The presence of newlines/spaces is preserved, but not the quantity.
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D125311: [pseudo] Share the underly payload when stripping comments for a token stream

[Haojian Wu via Phabricator via cfe-commits]

hokein added a comment.

In D125311#3524190 , @sammccall wrote:

> In D125311#3523388 , @hokein wrote:
>
>> In D125311#3503452 , @sammccall 
>> wrote:
>>
>>> This looks good, but if that's the idiom we should add it to cook() also.
>>
>> cook is more tricky, it takes a TokenStream, and returns a new one with an 
>> allocator payload:
>>
>> - the input TokenStream doesn't have a payload, this is OK
>> - the input TokenStream has a payload, but what if the type of payload is 
>> not an allocator?
>
> Payload is opaque, it can be a pair of {new payload, inner payload}. e.g.
>
>   void TokenStream::addPayload(shared_ptr P) {
> Payload = make_shared, 
> shared_ptr>>(std::move(P), std::move(Payload));
>   }

Ohh, right, this is clever!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D125311/new/

https://reviews.llvm.org/D125311

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D125311: [pseudo] Share the underly payload when stripping comments for a token stream

[Haojian Wu via Phabricator via cfe-commits]

hokein updated this revision to Diff 430646.
hokein added a comment.

add addPayload method, and share payload in cook as well


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D125311/new/

https://reviews.llvm.org/D125311

Files:
  clang-tools-extra/pseudo/include/clang-pseudo/Token.h
  clang-tools-extra/pseudo/lib/Lex.cpp
  clang-tools-extra/pseudo/lib/Token.cpp


Index: clang-tools-extra/pseudo/lib/Token.cpp
===
--- clang-tools-extra/pseudo/lib/Token.cpp
+++ clang-tools-extra/pseudo/lib/Token.cpp
@@ -116,7 +116,7 @@
 }
 
 TokenStream stripComments(const TokenStream ) {
-  TokenStream Out;
+  TokenStream Out(Input.getPayload());
   for (const Token  : Input.tokens()) {
 if (T.Kind == tok::comment)
   continue;
Index: clang-tools-extra/pseudo/lib/Lex.cpp
===
--- clang-tools-extra/pseudo/lib/Lex.cpp
+++ clang-tools-extra/pseudo/lib/Lex.cpp
@@ -77,7 +77,7 @@
   auto CleanedStorage = std::make_shared();
   clang::IdentifierTable Identifiers(LangOpts);
   TokenStream Result(CleanedStorage);
-
+  Result.addPayload(Code.getPayload());
   for (auto Tok : Code.tokens()) {
 if (Tok.flag(LexFlags::NeedsCleaning)) {
   // Remove escaped newlines and trigraphs.
Index: clang-tools-extra/pseudo/include/clang-pseudo/Token.h
===
--- clang-tools-extra/pseudo/include/clang-pseudo/Token.h
+++ clang-tools-extra/pseudo/include/clang-pseudo/Token.h
@@ -161,6 +161,18 @@
 return Storage[1];
   }
 
+  /// Returns the shared payload.
+  std::shared_ptr getPayload() const { return Payload; }
+  /// Adds the given payload to the stream.
+  void addPayload(std::shared_ptr P) {
+if (!Payload)
+  Payload = std::move(P);
+else
+  Payload = std::make_shared<
+  std::pair, std::shared_ptr>>(
+  std::move(P), std::move(Payload));
+  }
+
   /// Print the tokens in this stream to the output stream.
   ///
   /// The presence of newlines/spaces is preserved, but not the quantity.


Index: clang-tools-extra/pseudo/lib/Token.cpp
===
--- clang-tools-extra/pseudo/lib/Token.cpp
+++ clang-tools-extra/pseudo/lib/Token.cpp
@@ -116,7 +116,7 @@
 }
 
 TokenStream stripComments(const TokenStream ) {
-  TokenStream Out;
+  TokenStream Out(Input.getPayload());
   for (const Token  : Input.tokens()) {
 if (T.Kind == tok::comment)
   continue;
Index: clang-tools-extra/pseudo/lib/Lex.cpp
===
--- clang-tools-extra/pseudo/lib/Lex.cpp
+++ clang-tools-extra/pseudo/lib/Lex.cpp
@@ -77,7 +77,7 @@
   auto CleanedStorage = std::make_shared();
   clang::IdentifierTable Identifiers(LangOpts);
   TokenStream Result(CleanedStorage);
-
+  Result.addPayload(Code.getPayload());
   for (auto Tok : Code.tokens()) {
 if (Tok.flag(LexFlags::NeedsCleaning)) {
   // Remove escaped newlines and trigraphs.
Index: clang-tools-extra/pseudo/include/clang-pseudo/Token.h
===
--- clang-tools-extra/pseudo/include/clang-pseudo/Token.h
+++ clang-tools-extra/pseudo/include/clang-pseudo/Token.h
@@ -161,6 +161,18 @@
 return Storage[1];
   }
 
+  /// Returns the shared payload.
+  std::shared_ptr getPayload() const { return Payload; }
+  /// Adds the given payload to the stream.
+  void addPayload(std::shared_ptr P) {
+if (!Payload)
+  Payload = std::move(P);
+else
+  Payload = std::make_shared<
+  std::pair, std::shared_ptr>>(
+  std::move(P), std::move(Payload));
+  }
+
   /// Print the tokens in this stream to the output stream.
   ///
   /// The presence of newlines/spaces is preserved, but not the quantity.
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D125311: [pseudo] Share the underly payload when stripping comments for a token stream

[Sam McCall via Phabricator via cfe-commits]

sammccall added a comment.

In D125311#3523388 , @hokein wrote:

> In D125311#3503452 , @sammccall 
> wrote:
>
>> This looks good, but if that's the idiom we should add it to cook() also.
>
> cook is more tricky, it takes a TokenStream, and returns a new one with an 
> allocator payload:
>
> - the input TokenStream doesn't have a payload, this is OK
> - the input TokenStream has a payload, but what if the type of payload is not 
> an allocator?

Payload is opaque, it can be a pair of {new payload, inner payload}. e.g.

  void TokenStream::addPayload(shared_ptr P) {
Payload = make_shared, 
shared_ptr>>(std::move(P), std::move(Payload));
  }


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D125311/new/

https://reviews.llvm.org/D125311

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D125311: [pseudo] Share the underly payload when stripping comments for a token stream

[Haojian Wu via Phabricator via cfe-commits]

hokein added a comment.

In D125311#3503452 , @sammccall wrote:

> This looks good, but if that's the idiom we should add it to cook() also.

cook is more tricky, it takes a TokenStream, and returns a new one with an 
allocator payload:

- the input TokenStream doesn't have a payload, this is OK
- the input TokenStream has a payload, but what if the type of payload is not 
an allocator?

Possible solution: 1) always assert the input TokenStream doesn't have a 
payload or the payload is allocator type 2) merge the lex and cook function, it 
looks like we always use the cook one.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D125311/new/

https://reviews.llvm.org/D125311

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D125311: [pseudo] Share the underly payload when stripping comments for a token stream

[Sam McCall via Phabricator via cfe-commits]

sammccall added a comment.

This looks good, but if that's the idiom we should add it to cook() also.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D125311/new/

https://reviews.llvm.org/D125311

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D125311: [pseudo] Share the underly payload when stripping comments for a token stream

[Haojian Wu via Phabricator via cfe-commits]

hokein added a comment.

Example: 
https://github.com/llvm/llvm-project/blob/main/clang-tools-extra/pseudo/tool/ClangPseudo.cpp#L88


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D125311/new/

https://reviews.llvm.org/D125311

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D125311: [pseudo] Share the underly payload when stripping comments for a token stream

[Haojian Wu via Phabricator via cfe-commits]

hokein created this revision.
hokein added a reviewer: sammccall.
Herald added a project: All.
hokein requested review of this revision.
Herald added a subscriber: alextsao1999.
Herald added a project: clang-tools-extra.

`stripComments(cook(...))` is a common pattern being written.
Without this patch, this has a use-after-free issue (cook returns a temporary
TokenStream object which has its own payload, but the payload is not
shared with the one returned by stripComments).


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D125311

Files:
  clang-tools-extra/pseudo/include/clang-pseudo/Token.h
  clang-tools-extra/pseudo/lib/Token.cpp


Index: clang-tools-extra/pseudo/lib/Token.cpp
===
--- clang-tools-extra/pseudo/lib/Token.cpp
+++ clang-tools-extra/pseudo/lib/Token.cpp
@@ -116,7 +116,7 @@
 }
 
 TokenStream stripComments(const TokenStream ) {
-  TokenStream Out;
+  TokenStream Out(Input.getPayload());
   for (const Token  : Input.tokens()) {
 if (T.Kind == tok::comment)
   continue;
Index: clang-tools-extra/pseudo/include/clang-pseudo/Token.h
===
--- clang-tools-extra/pseudo/include/clang-pseudo/Token.h
+++ clang-tools-extra/pseudo/include/clang-pseudo/Token.h
@@ -161,6 +161,9 @@
 return Storage[1];
   }
 
+  /// Returns the shared payload.
+  std::shared_ptr getPayload() const { return Payload; }
+
   /// Print the tokens in this stream to the output stream.
   ///
   /// The presence of newlines/spaces is preserved, but not the quantity.


Index: clang-tools-extra/pseudo/lib/Token.cpp
===
--- clang-tools-extra/pseudo/lib/Token.cpp
+++ clang-tools-extra/pseudo/lib/Token.cpp
@@ -116,7 +116,7 @@
 }
 
 TokenStream stripComments(const TokenStream ) {
-  TokenStream Out;
+  TokenStream Out(Input.getPayload());
   for (const Token  : Input.tokens()) {
 if (T.Kind == tok::comment)
   continue;
Index: clang-tools-extra/pseudo/include/clang-pseudo/Token.h
===
--- clang-tools-extra/pseudo/include/clang-pseudo/Token.h
+++ clang-tools-extra/pseudo/include/clang-pseudo/Token.h
@@ -161,6 +161,9 @@
 return Storage[1];
   }
 
+  /// Returns the shared payload.
+  std::shared_ptr getPayload() const { return Payload; }
+
   /// Print the tokens in this stream to the output stream.
   ///
   /// The presence of newlines/spaces is preserved, but not the quantity.
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits