Re: [clang] 33a745e - [clang][clang-scan-deps] Add support for extracting full module dependencies.

[Reid Kleckner via cfe-commits]

You could hack it to standardize the yaml output to forward slashes with
sed, or clang might wish to do it internally.

My current position is that we should pick the path style and slash
direction closest to the point at which the path will be emitted. In the
past we've tried to make arguments that slashes should be internally one
way or another depending on if it's Windows, MinGW, MSVC, or some other
criteria, but the codebase is inconsistent. Now I lean towards
canonicalizing when printing based on the format in use.
- For example, if emitting codeview, try to standardize to backslashes to
match MSVC.
- If emitting DWARF, maybe use forward slashes since the consumer will
probably be a gnu tool.
- When implementing -print-*-path in the driver, the user is probably a
Makefile, so use forward slashes.
- When emitting this YAML stuff, it's our own format, so do whatever is
convenient.

On Wed, Oct 30, 2019 at 4:03 PM Michael Spencer 
wrote:

> On Wed, Oct 30, 2019 at 3:54 PM Reid Kleckner via cfe-commits <
> cfe-commits@lists.llvm.org> wrote:
>
>> I XFAILED the test on Windows in 52194350cfe. The [/\\] regexes
>> might need to match two slashes, and the PREFIX variable ends up having
>> inconsistent slash direction.
>>
>
> Hmm, fixing the slashes should be easy, but I'm not sure there's a good
> way to get the PREFIX right. I feel like FileCheck needs an ignore slash
> direction mode.
>
> - Michael Spencer
>
>
>>
>> On Wed, Oct 30, 2019 at 3:27 PM Michael Spencer via cfe-commits <
>> cfe-commits@lists.llvm.org> wrote:
>>
>>>
>>> Author: Michael Spencer
>>> Date: 2019-10-30T15:27:27-07:00
>>> New Revision: 33a745e6fe7e81d3793f7831d2832aa0785ef327
>>>
>>> URL:
>>> https://github.com/llvm/llvm-project/commit/33a745e6fe7e81d3793f7831d2832aa0785ef327
>>> DIFF:
>>> https://github.com/llvm/llvm-project/commit/33a745e6fe7e81d3793f7831d2832aa0785ef327.diff
>>>
>>> LOG: [clang][clang-scan-deps] Add support for extracting full module
>>> dependencies.
>>>
>>> This is a recommit of d8a4ef0e685c with the nondeterminism fixed.
>>>
>>> This adds experimental support for extracting a Clang module dependency
>>> graph
>>> from a compilation database. The output format is experimental and will
>>> change.
>>> It is currently a concatenation of JSON outputs for each compilation.
>>> Future
>>> patches will change this to deduplicate modules between compilations.
>>>
>>> Differential Revision: https://reviews.llvm.org/D69420
>>>
>>> Added:
>>> clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
>>> clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp
>>> clang/test/ClangScanDeps/modules-full.cpp
>>>
>>> Modified:
>>>
>>> clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>>>
>>> clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
>>>
>>> clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
>>> clang/lib/Tooling/DependencyScanning/CMakeLists.txt
>>> clang/lib/Tooling/DependencyScanning/DependencyScanningService.cpp
>>> clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
>>> clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
>>> clang/tools/clang-scan-deps/ClangScanDeps.cpp
>>>
>>> Removed:
>>>
>>>
>>>
>>>
>>> 
>>> diff  --git
>>> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>>> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>>> index fd8ed80b143c..76edf150dbee 100644
>>> ---
>>> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>>> +++
>>> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>>> @@ -30,15 +30,30 @@ enum class ScanningMode {
>>>MinimizedSourcePreprocessing
>>>  };
>>>
>>> +/// The format that is output by the dependency scanner.
>>> +enum class ScanningOutputFormat {
>>> +  /// This is the Makefile compatible dep format. This will include all
>>> of the
>>> +  /// deps necessary for an implicit modules build, but won't include
>>> any
>>> +  /// intermodule dependency information.
>>> +  Make,
>>> +
>>> +  /// This outputs the full module dependency graph suitable for use for
>>> +  /// explicitly building modules.
>>> +  Full,
>>> +};
>>> +
>>>  /// The dependency scanning service contains the shared state that is
>>> used by
>>>  /// the invidual dependency scanning workers.
>>>  class DependencyScanningService {
>>>  public:
>>> -  DependencyScanningService(ScanningMode Mode, bool ReuseFileManager =
>>> true,
>>> +  DependencyScanningService(ScanningMode Mode, ScanningOutputFormat
>>> Format,
>>> +bool ReuseFileManager = true,
>>>  bool SkipExcludedPPRanges = true);
>>>
>>>ScanningMode getMode() const { return Mode; }
>>>
>>> +  ScanningOutputFormat getFormat() const { return Format; }
>>> +
>>>bool 

Re: [clang] 33a745e - [clang][clang-scan-deps] Add support for extracting full module dependencies.

[Michael Spencer via cfe-commits]

On Wed, Oct 30, 2019 at 3:54 PM Reid Kleckner via cfe-commits <
cfe-commits@lists.llvm.org> wrote:

> I XFAILED the test on Windows in 52194350cfe. The [/\\] regexes might need
> to match two slashes, and the PREFIX variable ends up having inconsistent
> slash direction.
>

Hmm, fixing the slashes should be easy, but I'm not sure there's a good way
to get the PREFIX right. I feel like FileCheck needs an ignore slash
direction mode.

- Michael Spencer


>
> On Wed, Oct 30, 2019 at 3:27 PM Michael Spencer via cfe-commits <
> cfe-commits@lists.llvm.org> wrote:
>
>>
>> Author: Michael Spencer
>> Date: 2019-10-30T15:27:27-07:00
>> New Revision: 33a745e6fe7e81d3793f7831d2832aa0785ef327
>>
>> URL:
>> https://github.com/llvm/llvm-project/commit/33a745e6fe7e81d3793f7831d2832aa0785ef327
>> DIFF:
>> https://github.com/llvm/llvm-project/commit/33a745e6fe7e81d3793f7831d2832aa0785ef327.diff
>>
>> LOG: [clang][clang-scan-deps] Add support for extracting full module
>> dependencies.
>>
>> This is a recommit of d8a4ef0e685c with the nondeterminism fixed.
>>
>> This adds experimental support for extracting a Clang module dependency
>> graph
>> from a compilation database. The output format is experimental and will
>> change.
>> It is currently a concatenation of JSON outputs for each compilation.
>> Future
>> patches will change this to deduplicate modules between compilations.
>>
>> Differential Revision: https://reviews.llvm.org/D69420
>>
>> Added:
>> clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
>> clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp
>> clang/test/ClangScanDeps/modules-full.cpp
>>
>> Modified:
>>
>> clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>>
>> clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
>>
>> clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
>> clang/lib/Tooling/DependencyScanning/CMakeLists.txt
>> clang/lib/Tooling/DependencyScanning/DependencyScanningService.cpp
>> clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
>> clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
>> clang/tools/clang-scan-deps/ClangScanDeps.cpp
>>
>> Removed:
>>
>>
>>
>>
>> 
>> diff  --git
>> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>> index fd8ed80b143c..76edf150dbee 100644
>> ---
>> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>> +++
>> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>> @@ -30,15 +30,30 @@ enum class ScanningMode {
>>MinimizedSourcePreprocessing
>>  };
>>
>> +/// The format that is output by the dependency scanner.
>> +enum class ScanningOutputFormat {
>> +  /// This is the Makefile compatible dep format. This will include all
>> of the
>> +  /// deps necessary for an implicit modules build, but won't include any
>> +  /// intermodule dependency information.
>> +  Make,
>> +
>> +  /// This outputs the full module dependency graph suitable for use for
>> +  /// explicitly building modules.
>> +  Full,
>> +};
>> +
>>  /// The dependency scanning service contains the shared state that is
>> used by
>>  /// the invidual dependency scanning workers.
>>  class DependencyScanningService {
>>  public:
>> -  DependencyScanningService(ScanningMode Mode, bool ReuseFileManager =
>> true,
>> +  DependencyScanningService(ScanningMode Mode, ScanningOutputFormat
>> Format,
>> +bool ReuseFileManager = true,
>>  bool SkipExcludedPPRanges = true);
>>
>>ScanningMode getMode() const { return Mode; }
>>
>> +  ScanningOutputFormat getFormat() const { return Format; }
>> +
>>bool canReuseFileManager() const { return ReuseFileManager; }
>>
>>bool canSkipExcludedPPRanges() const { return SkipExcludedPPRanges; }
>> @@ -49,6 +64,7 @@ class DependencyScanningService {
>>
>>  private:
>>const ScanningMode Mode;
>> +  const ScanningOutputFormat Format;
>>const bool ReuseFileManager;
>>/// Set to true to use the preprocessor optimization that skips
>> excluded PP
>>/// ranges by bumping the buffer pointer in the lexer instead of
>> lexing the
>>
>> diff  --git
>> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
>> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
>> index c950cbe167cd..78b49e4fa0c5 100644
>> ---
>> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
>> +++
>> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
>> @@ -40,6 +40,7 @@ class DependencyScanningTool {
>>  StringRef CWD);
>>
>>  private:
>> +  const ScanningOutputFormat Format;
>>DependencyScanningWorker Worker;

Re: [clang] 33a745e - [clang][clang-scan-deps] Add support for extracting full module dependencies.

[Reid Kleckner via cfe-commits]

I XFAILED the test on Windows in 52194350cfe. The [/\\] regexes might need
to match two slashes, and the PREFIX variable ends up having inconsistent
slash direction.

On Wed, Oct 30, 2019 at 3:27 PM Michael Spencer via cfe-commits <
cfe-commits@lists.llvm.org> wrote:

>
> Author: Michael Spencer
> Date: 2019-10-30T15:27:27-07:00
> New Revision: 33a745e6fe7e81d3793f7831d2832aa0785ef327
>
> URL:
> https://github.com/llvm/llvm-project/commit/33a745e6fe7e81d3793f7831d2832aa0785ef327
> DIFF:
> https://github.com/llvm/llvm-project/commit/33a745e6fe7e81d3793f7831d2832aa0785ef327.diff
>
> LOG: [clang][clang-scan-deps] Add support for extracting full module
> dependencies.
>
> This is a recommit of d8a4ef0e685c with the nondeterminism fixed.
>
> This adds experimental support for extracting a Clang module dependency
> graph
> from a compilation database. The output format is experimental and will
> change.
> It is currently a concatenation of JSON outputs for each compilation.
> Future
> patches will change this to deduplicate modules between compilations.
>
> Differential Revision: https://reviews.llvm.org/D69420
>
> Added:
> clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
> clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp
> clang/test/ClangScanDeps/modules-full.cpp
>
> Modified:
>
> clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
> clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
>
> clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
> clang/lib/Tooling/DependencyScanning/CMakeLists.txt
> clang/lib/Tooling/DependencyScanning/DependencyScanningService.cpp
> clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
> clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
> clang/tools/clang-scan-deps/ClangScanDeps.cpp
>
> Removed:
>
>
>
>
> 
> diff  --git
> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
> index fd8ed80b143c..76edf150dbee 100644
> ---
> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
> +++
> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
> @@ -30,15 +30,30 @@ enum class ScanningMode {
>MinimizedSourcePreprocessing
>  };
>
> +/// The format that is output by the dependency scanner.
> +enum class ScanningOutputFormat {
> +  /// This is the Makefile compatible dep format. This will include all
> of the
> +  /// deps necessary for an implicit modules build, but won't include any
> +  /// intermodule dependency information.
> +  Make,
> +
> +  /// This outputs the full module dependency graph suitable for use for
> +  /// explicitly building modules.
> +  Full,
> +};
> +
>  /// The dependency scanning service contains the shared state that is
> used by
>  /// the invidual dependency scanning workers.
>  class DependencyScanningService {
>  public:
> -  DependencyScanningService(ScanningMode Mode, bool ReuseFileManager =
> true,
> +  DependencyScanningService(ScanningMode Mode, ScanningOutputFormat
> Format,
> +bool ReuseFileManager = true,
>  bool SkipExcludedPPRanges = true);
>
>ScanningMode getMode() const { return Mode; }
>
> +  ScanningOutputFormat getFormat() const { return Format; }
> +
>bool canReuseFileManager() const { return ReuseFileManager; }
>
>bool canSkipExcludedPPRanges() const { return SkipExcludedPPRanges; }
> @@ -49,6 +64,7 @@ class DependencyScanningService {
>
>  private:
>const ScanningMode Mode;
> +  const ScanningOutputFormat Format;
>const bool ReuseFileManager;
>/// Set to true to use the preprocessor optimization that skips
> excluded PP
>/// ranges by bumping the buffer pointer in the lexer instead of lexing
> the
>
> diff  --git
> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
> index c950cbe167cd..78b49e4fa0c5 100644
> ---
> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
> +++
> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
> @@ -40,6 +40,7 @@ class DependencyScanningTool {
>  StringRef CWD);
>
>  private:
> +  const ScanningOutputFormat Format;
>DependencyScanningWorker Worker;
>const tooling::CompilationDatabase 
>  };
>
> diff  --git
> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
> index 45c9fb4f029d..689119330c41 100644
> ---
> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
> +++
> 

[clang] 33a745e - [clang][clang-scan-deps] Add support for extracting full module dependencies.

[Michael Spencer via cfe-commits]

Author: Michael Spencer
Date: 2019-10-30T15:27:27-07:00
New Revision: 33a745e6fe7e81d3793f7831d2832aa0785ef327

URL: 
https://github.com/llvm/llvm-project/commit/33a745e6fe7e81d3793f7831d2832aa0785ef327
DIFF: 
https://github.com/llvm/llvm-project/commit/33a745e6fe7e81d3793f7831d2832aa0785ef327.diff

LOG: [clang][clang-scan-deps] Add support for extracting full module 
dependencies.

This is a recommit of d8a4ef0e685c with the nondeterminism fixed.

This adds experimental support for extracting a Clang module dependency graph
from a compilation database. The output format is experimental and will change.
It is currently a concatenation of JSON outputs for each compilation. Future
patches will change this to deduplicate modules between compilations.

Differential Revision: https://reviews.llvm.org/D69420

Added: 
clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp
clang/test/ClangScanDeps/modules-full.cpp

Modified: 
clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
clang/lib/Tooling/DependencyScanning/CMakeLists.txt
clang/lib/Tooling/DependencyScanning/DependencyScanningService.cpp
clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
clang/tools/clang-scan-deps/ClangScanDeps.cpp

Removed: 




diff  --git 
a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h 
b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
index fd8ed80b143c..76edf150dbee 100644
--- a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
+++ b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
@@ -30,15 +30,30 @@ enum class ScanningMode {
   MinimizedSourcePreprocessing
 };
 
+/// The format that is output by the dependency scanner.
+enum class ScanningOutputFormat {
+  /// This is the Makefile compatible dep format. This will include all of the
+  /// deps necessary for an implicit modules build, but won't include any
+  /// intermodule dependency information.
+  Make,
+
+  /// This outputs the full module dependency graph suitable for use for
+  /// explicitly building modules.
+  Full,
+};
+
 /// The dependency scanning service contains the shared state that is used by
 /// the invidual dependency scanning workers.
 class DependencyScanningService {
 public:
-  DependencyScanningService(ScanningMode Mode, bool ReuseFileManager = true,
+  DependencyScanningService(ScanningMode Mode, ScanningOutputFormat Format,
+bool ReuseFileManager = true,
 bool SkipExcludedPPRanges = true);
 
   ScanningMode getMode() const { return Mode; }
 
+  ScanningOutputFormat getFormat() const { return Format; }
+
   bool canReuseFileManager() const { return ReuseFileManager; }
 
   bool canSkipExcludedPPRanges() const { return SkipExcludedPPRanges; }
@@ -49,6 +64,7 @@ class DependencyScanningService {
 
 private:
   const ScanningMode Mode;
+  const ScanningOutputFormat Format;
   const bool ReuseFileManager;
   /// Set to true to use the preprocessor optimization that skips excluded PP
   /// ranges by bumping the buffer pointer in the lexer instead of lexing the

diff  --git 
a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h 
b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
index c950cbe167cd..78b49e4fa0c5 100644
--- a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
+++ b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
@@ -40,6 +40,7 @@ class DependencyScanningTool {
 StringRef CWD);
 
 private:
+  const ScanningOutputFormat Format;
   DependencyScanningWorker Worker;
   const tooling::CompilationDatabase 
 };

diff  --git 
a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h 
b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
index 45c9fb4f029d..689119330c41 100644
--- a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
+++ b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
@@ -15,6 +15,8 @@
 #include "clang/Frontend/PCHContainerOperations.h"
 #include "clang/Lex/PreprocessorExcludedConditionalDirectiveSkipMapping.h"
 #include "clang/Tooling/CompilationDatabase.h"
+#include "clang/Tooling/DependencyScanning/DependencyScanningService.h"
+#include "clang/Tooling/DependencyScanning/ModuleDepCollector.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/FileSystem.h"
 #include 
@@ -26,7 +28,6 @@ class