[llvm] [clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Jan Svoboda via cfe-commits]

@@ -0,0 +1,48 @@
+//===-- Utils.h 
---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// Functions required by both the frontend and the module build daemon
+//
+//===--===//
+
+#ifndef LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_UTILS_H
+#define LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_UTILS_H
+
+#include "llvm/Support/Error.h"
+
+#include 
+
+#ifdef _WIN32
+#include 
+#else
+#include 
+#endif
+
+namespace clang::tooling::cc1modbuildd {
+
+constexpr const std::string_view SOCKET_FILE_NAME = "mbd.sock";
+constexpr const std::string_view STDOUT_FILE_NAME = "mbd.out";
+constexpr const std::string_view STDERR_FILE_NAME = "mbd.err";
+constexpr const std::string_view MODULE_BUILD_DAEMON_FLAG = "-cc1modbuildd";
+
+// A llvm::raw_socket_stream uses sockaddr_un
+constexpr const size_t SOCKET_ADDR_MAX_LENGTH = sizeof(sockaddr_un::sun_path);
+
+constexpr const size_t BASEPATH_MAX_LENGTH =
+SOCKET_ADDR_MAX_LENGTH - std::string_view(SOCKET_FILE_NAME).length();

jansvoboda11 wrote:

Is constructing new `std::string_view` from `SOCKET_FILE_NAME` necessary?

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[llvm] [clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Jan Svoboda via cfe-commits]

@@ -383,6 +383,9 @@ class FrontendOptions {
   LLVM_PREFERRED_TYPE(bool)
   unsigned ModulesShareFileManager : 1;
 
+  /// Connect to module build daemon
+  unsigned ModuleBuildDaemon : 1;

jansvoboda11 wrote:

Mark this with `LLVM_PREFERRED_TYPE(bool)`, like the other bool-like bitfield 
in this class.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[llvm] [clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Jan Svoboda via cfe-commits]

@@ -0,0 +1,48 @@
+//===-- Utils.h 
---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// Functions required by both the frontend and the module build daemon
+//
+//===--===//
+
+#ifndef LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_UTILS_H
+#define LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_UTILS_H
+
+#include "llvm/Support/Error.h"
+
+#include 
+
+#ifdef _WIN32
+#include 
+#else
+#include 
+#endif
+
+namespace clang::tooling::cc1modbuildd {
+
+constexpr const std::string_view SOCKET_FILE_NAME = "mbd.sock";

jansvoboda11 wrote:

You should be able to drop `const` here, since that's implied by `constexpr`.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[llvm] [clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Jan Svoboda via cfe-commits]

@@ -0,0 +1,60 @@
+//=== SocketMsgSupport.cpp 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/raw_socket_stream.h"
+
+#include 
+#include 
+
+using namespace llvm;
+
+namespace clang::tooling::cc1modbuildd {
+
+Expected>
+connectToSocket(StringRef SocketPath) {
+
+  Expected> MaybeClient =
+  raw_socket_stream::createConnectedUnix(SocketPath);
+  if (!MaybeClient)
+return std::move(MaybeClient.takeError());
+
+  return std::move(*MaybeClient);
+}
+
+Expected readBufferFromSocket(raw_socket_stream ) {
+
+  constexpr const unsigned short MAX_BUFFER = 4096;
+  char Buffer[MAX_BUFFER];
+  std::string ReturnBuffer;
+
+  ssize_t n = 0;
+  while ((n = Socket.read(Buffer, MAX_BUFFER)) > 0) {
+ReturnBuffer.append(Buffer, n);
+// Read until \n... encountered which is the last line of a YAML document
+if (ReturnBuffer.find("\n...") != std::string::npos)

jansvoboda11 wrote:

IIUC `"\n..."` is always going to be at the end of `ReturnBuffer`, right? Could 
we replace the call to `std::string::find()` with 
`StringRef(ReturnBuffer).ends_with("\n...")`? Considering `"\n..."` constant, I 
think this would reduce the complexity of the `while` loop from `O(n*n)` to 
`O(n)`.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[llvm] [clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Jan Svoboda via cfe-commits]

jansvoboda11 wrote:

> This seems like something that shouldn't be in the compiler binary itself. 
> There should be a separate binary for this type of thing.

Discussed here: 
https://discourse.llvm.org/t/rfc-modules-build-daemon-build-system-agnostic-support-for-explicitly-built-modules/71524

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[llvm] [clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Nico Weber via cfe-commits]

nico wrote:

This seems like something that shouldn't be in the compiler binary itself. 
There should be a separate binary for this type of thing.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

@@ -0,0 +1,135 @@
+//===- SocketMsgSupport.h 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_SOCKETMSGSUPPORT_H
+#define LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_SOCKETMSGSUPPORT_H
+
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+
+using namespace clang;
+using namespace llvm;
+
+namespace cc1modbuildd {

cpsughrue wrote:

You're right, thanks for pointing it out!

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

@@ -0,0 +1,41 @@
+//===-- Client.h 
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_CLIENT_H
+#define LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_CLIENT_H
+
+#include "clang/Frontend/CompilerInstance.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/YAMLParser.h"
+#include "llvm/Support/YAMLTraits.h"
+
+using namespace clang;
+using namespace llvm;

cpsughrue wrote:

Thanks!

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue edited 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

@@ -3738,6 +3737,19 @@ static bool RenderModulesOptions(Compilation , const 
Driver ,
Std->containsValue("c++latest") || Std->containsValue("gnu++latest"));
   bool HaveModules = HaveStdCXXModules;
 
+  // -fmodule-build-daemon enables module build daemon functionality
+  if (Args.hasArg(options::OPT_fmodule_build_daemon))
+Args.AddLastArg(CmdArgs, options::OPT_fmodule_build_daemon);
+
+  // by default module build daemon socket address and output files are saved
+  // under /tmp/ but that can be overridden by providing the

cpsughrue wrote:

The `/tmp` is provided by `llvm::sys::path::system_temp_directory` which is 
portable

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[via cfe-commits]

@@ -0,0 +1,135 @@
+//===- SocketMsgSupport.h 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_SOCKETMSGSUPPORT_H
+#define LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_SOCKETMSGSUPPORT_H
+
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+
+using namespace clang;
+using namespace llvm;
+
+namespace cc1modbuildd {

tschuett wrote:

I believe that you still have to wrap everything into:
```
clang::tooling::
```

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[via cfe-commits]

@@ -0,0 +1,41 @@
+//===-- Client.h 
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_CLIENT_H
+#define LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_CLIENT_H
+
+#include "clang/Frontend/CompilerInstance.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/YAMLParser.h"
+#include "llvm/Support/YAMLTraits.h"
+
+using namespace clang;
+using namespace llvm;

tschuett wrote:

Please don't do this in headers. Every user will suffer from it.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -15,21 +12,12 @@ int main() {return 0;}
 // RUN: %clang -fmodule-build-daemon=mbd-handshake -Rmodule-build-daemon 
%t/main.c &> %t/output-existing
 // RUN: cat %t/output-existing | FileCheck %s --check-prefix=CHECK-EXIST
 
-// CHECK: remark: Trying to spawn module build daemon [-Rmodule-build-daemon]
+// COM: Check that clang invocation can spawn and handshake with module build 
daemon

Bigcheese wrote:

No need for `COM:`.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

Bigcheese wrote:

> I had a couple of questions. I guess my main concern here is that the TODO 
> for portability might turn out to be quite heavy lifting. Perhaps we should 
> be looking to abstract the socket-like interface at a lower level.

We have someone working on a simple socket library for LLVM support that can 
handle the portability issues, plan is to land that first.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

Bigcheese wrote:

> > Implementing a timeout as part of this patch as a safety measure seems 
> > worth while too but I'm not sure I understand your final solution. You all 
> > spawned a clang job that had to communicate with the daemon to make sure 
> > one did not exist?
> 
> It is something like this: 
> https://github.com/apple/llvm-project/blob/next/clang/test/CAS/fdepscan-daemon.c
> 
> The "daemon" binary can be launched as a normal process and it takes `--` 
> option, which has the arguments of the clang process it needs to run that 
> talk to itself. The lifetime of the "daemon" is now bounded to that normal 
> process and will not be left behind after testing is done.

I agree with doing this for all the non daemon launch tests. For those you can 
just run clang in a way that lit doesn't care if it fails, kill the daemon, 
then check the output of clang.

It doesn't really need to do a fully clean shutdown as no other process should 
be connecting, it can just call `shutdownDaemon`.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue updated 
https://github.com/llvm/llvm-project/pull/67562

>From 5def87462e3b48cfebafdc2526ac929f5cb9cea3 Mon Sep 17 00:00:00 2001
From: cpsughrue 
Date: Sun, 9 Jul 2023 23:19:58 -0400
Subject: [PATCH 1/6] [clang][MBD] set up module build daemon infrastructure

The module build daemon (mbd) will serve as a cc1 tool that helps convert
implicit module command lines to explicit module command lines. A clang
invocation will check to see if a mbd exists, if not the invocation will spawn
one. After the mbd is up and running and a handshake has successfully been
carried out between the mbd and clang invocation the clang invocation will
share it's command line with the mbd. The mbd will then scan the translation
unit and build all modular dependencies before returning a modified cc1 command
 line such that all arguments related to modules are converted from the
implicit option to their explicit option.

This commit sets up the basic mbd infrastructure. Including the ability to
spawn a mbd and carry out a handshake between the clang invocation and mbd.

RFC: 
https://discourse.llvm.org/t/rfc-modules-build-daemon-build-system-agnostic-support-for-explicitly-built-modules/71524
---
 clang/include/clang/Driver/Options.td |  12 +
 .../include/clang/Frontend/FrontendOptions.h  |   7 +
 .../clang/Tooling/ModuleBuildDaemon/Client.h  |  44 +++
 .../ModuleBuildDaemon/SocketMsgSupport.h  | 134 +
 .../Tooling/ModuleBuildDaemon/SocketSupport.h |  31 ++
 .../clang/Tooling/ModuleBuildDaemon/Utils.h   |  28 ++
 clang/lib/Driver/ToolChains/Clang.cpp |  14 +-
 clang/lib/Tooling/CMakeLists.txt  |   1 +
 .../Tooling/ModuleBuildDaemon/CMakeLists.txt  |   9 +
 .../lib/Tooling/ModuleBuildDaemon/Client.cpp  | 167 +++
 .../ModuleBuildDaemon/SocketSupport.cpp   | 128 +
 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp |  32 +++
 clang/test/Driver/unknown-arg.c   |   2 +-
 clang/test/ModuleBuildDaemon/handshake.c  |  18 ++
 clang/test/ModuleBuildDaemon/launch.c |  14 +
 clang/tools/driver/CMakeLists.txt |   3 +
 clang/tools/driver/cc1_main.cpp   |  28 +-
 clang/tools/driver/cc1modbuildd_main.cpp  | 267 ++
 clang/tools/driver/driver.cpp |  17 +-
 19 files changed, 947 insertions(+), 9 deletions(-)
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Client.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketSupport.h
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Utils.h
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/CMakeLists.txt
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Client.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/SocketSupport.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp
 create mode 100644 clang/test/ModuleBuildDaemon/handshake.c
 create mode 100644 clang/test/ModuleBuildDaemon/launch.c
 create mode 100644 clang/tools/driver/cc1modbuildd_main.cpp

diff --git a/clang/include/clang/Driver/Options.td 
b/clang/include/clang/Driver/Options.td
index efd90942948af27..d31d28255e81a8a 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -2836,6 +2836,18 @@ defm declspec : BoolOption<"f", "declspec",
   NegFlag,
   BothFlags<[], [ClangOption, CC1Option],
   " __declspec as a keyword">>, Group;
+
+def fmodule_build_daemon : Flag<["-"], "fmodule-build-daemon">, Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality">,
+  MarshallingInfoFlag>;
+def fmodule_build_daemon_EQ : Joined<["-"], "fmodule-build-daemon=">, 
Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality and defines location of 
output files">,
+  MarshallingInfoString>;
+
 def fmodules_cache_path : Joined<["-"], "fmodules-cache-path=">, 
Group,
   Flags<[NoXarchOption]>, Visibility<[ClangOption, CC1Option]>,
   MetaVarName<"">,
diff --git a/clang/include/clang/Frontend/FrontendOptions.h 
b/clang/include/clang/Frontend/FrontendOptions.h
index 117e35de6f76c4c..8ce97a57d413c0b 100644
--- a/clang/include/clang/Frontend/FrontendOptions.h
+++ b/clang/include/clang/Frontend/FrontendOptions.h
@@ -350,6 +350,9 @@ class FrontendOptions {
   /// Whether to share the FileManager when building modules.
   unsigned ModulesShareFileManager : 1;
 
+  /// Connect to module build daemon
+  unsigned ModuleBuildDaemon : 1;
+
   CodeCompleteOptions CodeCompleteOpts;
 
   /// Specifies the output format of the AST.
@@ -435,6 +438,10 @@ class FrontendOptions {
   /// The output file, if any.
   std::string OutputFile;
 
+  /// If given, the path to the module build daemon's output files and socket
+  /// address
+  std::string 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

@@ -0,0 +1,134 @@
+//===- SocketMsgSupport.h 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_SOCKETMSGSUPPORT_H
+#define LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_SOCKETMSGSUPPORT_H
+
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+
+using namespace clang;
+using namespace llvm;
+
+namespace cc1modbuildd {
+
+enum class ActionType { HANDSHAKE };
+enum class StatusType { REQUEST, SUCCESS, FAILURE };
+
+struct BaseMsg {
+  ActionType MsgAction;
+  StatusType MsgStatus;
+
+  BaseMsg() = default;
+  BaseMsg(ActionType Action, StatusType Status)
+  : MsgAction(Action), MsgStatus(Status) {}
+};
+
+struct HandshakeMsg : public BaseMsg {
+  HandshakeMsg() = default;
+  HandshakeMsg(ActionType Action, StatusType Status)
+  : BaseMsg(Action, Status) {}
+};
+
+template  std::string getBufferFromSocketMsg(T Msg) {
+  static_assert(std::is_base_of::value,
+"T must inherit from cc1modbuildd::BaseMsg");
+
+  std::string Buffer;
+  llvm::raw_string_ostream OS(Buffer);
+  llvm::yaml::Output YamlOut(OS);
+
+  YamlOut << Msg;
+  return Buffer;
+}
+
+template  Expected getSocketMsgFromBuffer(StringRef Buffer) {
+  static_assert(std::is_base_of::value,
+"T must inherit from cc1modbuildd::BaseMsg");
+
+  T ClientRequest;
+  llvm::yaml::Input YamlIn(Buffer);
+  YamlIn >> ClientRequest;
+
+  if (YamlIn.error()) {
+std::string Msg = "Syntax or semantic error during YAML parsing";

cpsughrue wrote:

`YamlIn.error()` dumps an error message describing the syntax or semantic error 
if there is one. I added a comment explaining that

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

@@ -0,0 +1,211 @@
+//===- Client.cpp 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendDiagnostic.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace clang;
+using namespace llvm;
+using namespace cc1modbuildd;
+
+std::string cc1modbuildd::getBasePath() {
+  llvm::BLAKE3 Hash;
+  Hash.update(getClangFullVersion());
+  auto HashResult = Hash.final();
+  uint64_t HashValue =
+  llvm::support::endian::read(
+  HashResult.data());
+  std::string Key = toString(llvm::APInt(64, HashValue), 36, /*Signed*/ false);
+
+  // Set paths
+  SmallString<128> BasePath;
+  llvm::sys::path::system_temp_directory(/*erasedOnReboot*/ true, BasePath);
+  llvm::sys::path::append(BasePath, "clang-" + Key);
+  return BasePath.c_str();
+}
+
+llvm::Error cc1modbuildd::attemptHandshake(int SocketFD,
+   DiagnosticsEngine ) {
+
+  HandshakeMsg Request{ActionType::HANDSHAKE, StatusType::REQUEST};
+  std::string Buffer = getBufferFromSocketMsg(Request);
+
+  // Send HandshakeMsg to module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Trying to send HandshakeMsg to module build daemon";
+  if (llvm::Error Err = writeToSocket(Buffer, SocketFD))
+return std::move(Err);
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Successfully sent HandshakeMsg to module build daemon";
+
+  // Receive response from module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Waiting to receive module build daemon response";
+  Expected MaybeServerResponse =
+  readSocketMsgFromSocket(SocketFD);
+  if (!MaybeServerResponse)
+return std::move(MaybeServerResponse.takeError());
+  HandshakeMsg ServerResponse = std::move(*MaybeServerResponse);
+
+  assert(ServerResponse.MsgAction == ActionType::HANDSHAKE &&
+ "Response ActionType should only ever be HANDSHAKE");
+
+  if (ServerResponse.MsgStatus == StatusType::SUCCESS) {
+Diag.Report(diag::remark_module_build_daemon)
+<< "Successfully received HandshakeMsg from module build daemon";
+return llvm::Error::success();
+  }
+
+  return llvm::make_error(
+  "Received failed handshake response from module build daemon",
+  inconvertibleErrorCode());
+}
+
+llvm::Error cc1modbuildd::spawnModuleBuildDaemon(StringRef BasePath,
+ const char *Argv0,
+ DiagnosticsEngine ) {
+  std::string BasePathStr = BasePath.str();
+  const char *Args[] = {Argv0, "-cc1modbuildd", BasePathStr.c_str(), nullptr};
+  pid_t pid;
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Trying to spawn module build daemon";
+  int EC = posix_spawn(, Args[0],
+   /*file_actions*/ nullptr,
+   /*spawnattr*/ nullptr, const_cast(Args),
+   /*envp*/ nullptr);
+  if (EC)
+return createStringError(std::error_code(EC, std::generic_category()),
+ "Failed to spawn module build daemon");
+
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Successfully spawned module build daemon";
+  return llvm::Error::success();
+}
+
+Expected cc1modbuildd::getModuleBuildDaemon(const char *Argv0,
+ StringRef BasePath,
+ DiagnosticsEngine ) {
+
+  SmallString<128> SocketPath = BasePath;
+  llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+
+  if (llvm::sys::fs::exists(SocketPath)) {
+Expected MaybeFD = connectToSocket(SocketPath);
+if (MaybeFD) {
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Module build daemon already exists";
+  return std::move(*MaybeFD);
+}
+consumeError(MaybeFD.takeError());
+  }
+
+  if (llvm::Error Err =
+  cc1modbuildd::spawnModuleBuildDaemon(BasePath, Argv0, Diag))
+return std::move(Err);
+
+  const unsigned int MICROSEC_IN_SEC = 100;
+  constexpr unsigned int MAX_WAIT_TIME = 30 * 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Iain Sandoe via cfe-commits]

https://github.com/iains commented:

I had a couple of questions.  I  guess my main concern here is that the TODO 
for portability might turn out to be quite heavy lifting.  Perhaps we should be 
looking to abstract the socket-like interface at a lower level.


https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Steven Wu via cfe-commits]

cachemeifyoucan wrote:

> Implementing a timeout as part of this patch as a safety measure seems worth 
> while too but I'm not sure I understand your final solution. You all spawned 
> a clang job that had to communicate with the daemon to make sure one did not 
> exist?

It is something like this: 
https://github.com/apple/llvm-project/blob/next/clang/test/CAS/fdepscan-daemon.c

The "daemon" binary can be launched as a normal process and it takes `--` 
option, which has the arguments of the clang process it needs to run that talk 
to itself. The lifetime of the "daemon" is now bounded to that normal process 
and will not be left behind after testing is done.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Iain Sandoe via cfe-commits]

https://github.com/iains edited https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Iain Sandoe via cfe-commits]

@@ -3738,6 +3737,19 @@ static bool RenderModulesOptions(Compilation , const 
Driver ,
Std->containsValue("c++latest") || Std->containsValue("gnu++latest"));
   bool HaveModules = HaveStdCXXModules;
 
+  // -fmodule-build-daemon enables module build daemon functionality
+  if (Args.hasArg(options::OPT_fmodule_build_daemon))
+Args.AddLastArg(CmdArgs, options::OPT_fmodule_build_daemon);
+
+  // by default module build daemon socket address and output files are saved
+  // under /tmp/ but that can be overridden by providing the

iains wrote:

is that a VFS /tmp - i.e. it will be available on all supported platforms?


https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Iain Sandoe via cfe-commits]

@@ -0,0 +1,134 @@
+//===- SocketMsgSupport.h 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_SOCKETMSGSUPPORT_H
+#define LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_SOCKETMSGSUPPORT_H
+
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+
+using namespace clang;
+using namespace llvm;
+
+namespace cc1modbuildd {
+
+enum class ActionType { HANDSHAKE };
+enum class StatusType { REQUEST, SUCCESS, FAILURE };
+
+struct BaseMsg {
+  ActionType MsgAction;
+  StatusType MsgStatus;
+
+  BaseMsg() = default;
+  BaseMsg(ActionType Action, StatusType Status)
+  : MsgAction(Action), MsgStatus(Status) {}
+};
+
+struct HandshakeMsg : public BaseMsg {
+  HandshakeMsg() = default;
+  HandshakeMsg(ActionType Action, StatusType Status)
+  : BaseMsg(Action, Status) {}
+};
+
+template  std::string getBufferFromSocketMsg(T Msg) {
+  static_assert(std::is_base_of::value,
+"T must inherit from cc1modbuildd::BaseMsg");
+
+  std::string Buffer;
+  llvm::raw_string_ostream OS(Buffer);
+  llvm::yaml::Output YamlOut(OS);
+
+  YamlOut << Msg;
+  return Buffer;
+}
+
+template  Expected getSocketMsgFromBuffer(StringRef Buffer) {
+  static_assert(std::is_base_of::value,
+"T must inherit from cc1modbuildd::BaseMsg");
+
+  T ClientRequest;
+  llvm::yaml::Input YamlIn(Buffer);
+  YamlIn >> ClientRequest;
+
+  if (YamlIn.error()) {
+std::string Msg = "Syntax or semantic error during YAML parsing";

iains wrote:

is there any additional information that can be provided to help the user 
figure out what the actual issue is?

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Iain Sandoe via cfe-commits]

@@ -0,0 +1,41 @@
+//===-- Client.h 
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_CLIENT_H
+#define LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_CLIENT_H
+
+#include "clang/Frontend/CompilerInstance.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/YAMLParser.h"
+#include "llvm/Support/YAMLTraits.h"
+
+using namespace clang;
+using namespace llvm;
+
+namespace cc1modbuildd {
+
+// Returns where to store log files and socket address. Of the format
+// /tmp/clang-/

iains wrote:

`BLAKE3HashOfClagnFullVersion` typo for Clang?

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Iain Sandoe via cfe-commits]

iains wrote:

A possible mechanism might be to have a watchdog process, with which instances 
of the server are registered with a time-to-live.  I guess this might be easier 
to construct as a scheme for test-suites than for end-user.


https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

cpsughrue wrote:

> I haven't read the PR in details but one thing we found difficult to do is 
> how to write a test for daemon spawning logic, because if anything went 
> wrong, you left an unbounded daemon process on CI node.
> 
> The strategy in this PR (try to kill it after the test finishes) definitely 
> doesn't work because 1. if anything failed, you will not hit kill. 2. you are 
> going to kill the daemon from a different test that runs in parallel.

1. Great point, I did not think of that.
2.  The tests are set up so that each one spawns, interacts with, and kills its 
own daemon, so parallel tests shouldn't kill one another's daemon.

> Other things we tried and know didn't work:
> 
> * explicitly start/stop daemon with command: you might not hit stop due to 
> failure
> * have a timeout and let the daemon terminate after that: you can have some 
> infinite loop that prevents daemon cleanup
> 
> What we ended up is to have a server process that spawns clang job that needs 
> to talk to it. We eliminated the run off daemons but we don't really have 
> coverage for how clang spawns a daemon when needed.

Implementing a timeout as part of this patch as a safety measure seems worth 
while too but I'm not sure I understand your final solution. You all spawned a 
clang job that had to communicate with the daemon to make sure one did not 
exist?

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Steven Wu via cfe-commits]

@@ -0,0 +1,302 @@
+//===--- cc1modbuildd_main.cpp - Clang CC1 Module Build Daemon 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Support/Threading.h"
+#include "llvm/Support/YAMLParser.h"
+#include "llvm/Support/YAMLTraits.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace llvm;
+using namespace clang;
+using namespace cc1modbuildd;
+
+// Create unbuffered STDOUT stream so that any logging done by module build
+// daemon can be viewed without having to terminate the process
+static raw_fd_ostream _outs() {
+  static raw_fd_ostream S(STDOUT_FILENO, false, true);
+  return S;
+}
+
+namespace {
+
+struct ClientConnection {
+  int ClientFD;
+  std::string Buffer;
+};
+
+class ModuleBuildDaemonServer {
+public:
+  SmallString<256> SocketPath;
+  SmallString<256> STDERR;
+  SmallString<256> STDOUT;
+
+  ModuleBuildDaemonServer(StringRef Path, ArrayRef Argv)
+  : SocketPath(Path), STDERR(Path), STDOUT(Path) {
+llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+llvm::sys::path::append(STDOUT, STDOUT_FILE_NAME);
+llvm::sys::path::append(STDERR, STDERR_FILE_NAME);
+  }
+
+  ~ModuleBuildDaemonServer() { shutdownDaemon(); }
+
+  int forkDaemon();
+  int createDaemonSocket();
+  int listenForClients();
+
+  static void handleClient(ClientConnection Connection);
+
+  void shutdownDaemon() {
+int SocketFD = ListenSocketFD.load();
+
+unlink(SocketPath.c_str());
+shutdown(SocketFD, SHUT_RD);
+close(SocketFD);
+exit(EXIT_SUCCESS);

cachemeifyoucan wrote:

Right, I like a soft shutdown mode, which daemon can still finish jobs that are 
in process but not accepting new jobs, at the same time, allows a new daemon to 
spawn and start working before it completely terminates itself.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Steven Wu via cfe-commits]

https://github.com/cachemeifyoucan commented:

I haven't read the PR in details but one thing we found difficult to do is how 
to write a test for daemon spawning logic, because if anything went wrong, you 
left an unbounded daemon process on CI node.

The strategy in this PR (try to kill it after the test finishes) definitely 
doesn't work because 1. if anything failed, you will not hit kill. 2. you are 
going to kill the daemon from a different test that runs in parallel.

Other things we tried and know didn't work:
* explicitly start/stop daemon with command: you might not hit stop due to 
failure
* have a timeout and let the daemon terminate after that: you can have some 
infinite loop that prevents daemon cleanup

What we ended up is to have a server process that spawns clang job that needs 
to talk to it. We eliminated the run off daemons but we don't really have 
coverage for how clang spawns a daemon when needed.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,302 @@
+//===--- cc1modbuildd_main.cpp - Clang CC1 Module Build Daemon 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Support/Threading.h"
+#include "llvm/Support/YAMLParser.h"
+#include "llvm/Support/YAMLTraits.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace llvm;
+using namespace clang;
+using namespace cc1modbuildd;
+
+// Create unbuffered STDOUT stream so that any logging done by module build
+// daemon can be viewed without having to terminate the process
+static raw_fd_ostream _outs() {
+  static raw_fd_ostream S(STDOUT_FILENO, false, true);
+  return S;
+}
+
+namespace {
+
+struct ClientConnection {
+  int ClientFD;
+  std::string Buffer;
+};
+
+class ModuleBuildDaemonServer {
+public:
+  SmallString<256> SocketPath;
+  SmallString<256> STDERR;
+  SmallString<256> STDOUT;
+
+  ModuleBuildDaemonServer(StringRef Path, ArrayRef Argv)
+  : SocketPath(Path), STDERR(Path), STDOUT(Path) {
+llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+llvm::sys::path::append(STDOUT, STDOUT_FILE_NAME);
+llvm::sys::path::append(STDERR, STDERR_FILE_NAME);
+  }
+
+  ~ModuleBuildDaemonServer() { shutdownDaemon(); }
+
+  int forkDaemon();
+  int createDaemonSocket();
+  int listenForClients();
+
+  static void handleClient(ClientConnection Connection);
+
+  void shutdownDaemon() {
+int SocketFD = ListenSocketFD.load();
+
+unlink(SocketPath.c_str());
+shutdown(SocketFD, SHUT_RD);
+close(SocketFD);
+exit(EXIT_SUCCESS);

Bigcheese wrote:

This should be `_Exit` as `exit` is not signal safe (it calls destructors and 
does some other work).

Longer term I think we'll want to make this just set a flag saying we want to 
shut down, and then stop accepting new clients and shut down when they are done.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,211 @@
+//===- Client.cpp 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendDiagnostic.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace clang;
+using namespace llvm;
+using namespace cc1modbuildd;
+
+std::string cc1modbuildd::getBasePath() {
+  llvm::BLAKE3 Hash;
+  Hash.update(getClangFullVersion());
+  auto HashResult = Hash.final();
+  uint64_t HashValue =
+  llvm::support::endian::read(
+  HashResult.data());
+  std::string Key = toString(llvm::APInt(64, HashValue), 36, /*Signed*/ false);
+
+  // Set paths
+  SmallString<128> BasePath;
+  llvm::sys::path::system_temp_directory(/*erasedOnReboot*/ true, BasePath);
+  llvm::sys::path::append(BasePath, "clang-" + Key);
+  return BasePath.c_str();
+}
+
+llvm::Error cc1modbuildd::attemptHandshake(int SocketFD,
+   DiagnosticsEngine ) {
+
+  HandshakeMsg Request{ActionType::HANDSHAKE, StatusType::REQUEST};
+  std::string Buffer = getBufferFromSocketMsg(Request);
+
+  // Send HandshakeMsg to module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Trying to send HandshakeMsg to module build daemon";
+  if (llvm::Error Err = writeToSocket(Buffer, SocketFD))
+return std::move(Err);
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Successfully sent HandshakeMsg to module build daemon";
+
+  // Receive response from module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Waiting to receive module build daemon response";
+  Expected MaybeServerResponse =
+  readSocketMsgFromSocket(SocketFD);
+  if (!MaybeServerResponse)
+return std::move(MaybeServerResponse.takeError());
+  HandshakeMsg ServerResponse = std::move(*MaybeServerResponse);
+
+  assert(ServerResponse.MsgAction == ActionType::HANDSHAKE &&
+ "Response ActionType should only ever be HANDSHAKE");
+
+  if (ServerResponse.MsgStatus == StatusType::SUCCESS) {
+Diag.Report(diag::remark_module_build_daemon)
+<< "Successfully received HandshakeMsg from module build daemon";
+return llvm::Error::success();
+  }
+
+  return llvm::make_error(
+  "Received failed handshake response from module build daemon",
+  inconvertibleErrorCode());
+}
+
+llvm::Error cc1modbuildd::spawnModuleBuildDaemon(StringRef BasePath,
+ const char *Argv0,
+ DiagnosticsEngine ) {
+  std::string BasePathStr = BasePath.str();
+  const char *Args[] = {Argv0, "-cc1modbuildd", BasePathStr.c_str(), nullptr};
+  pid_t pid;
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Trying to spawn module build daemon";
+  int EC = posix_spawn(, Args[0],
+   /*file_actions*/ nullptr,
+   /*spawnattr*/ nullptr, const_cast(Args),
+   /*envp*/ nullptr);
+  if (EC)
+return createStringError(std::error_code(EC, std::generic_category()),
+ "Failed to spawn module build daemon");
+
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Successfully spawned module build daemon";
+  return llvm::Error::success();
+}
+
+Expected cc1modbuildd::getModuleBuildDaemon(const char *Argv0,
+ StringRef BasePath,
+ DiagnosticsEngine ) {
+
+  SmallString<128> SocketPath = BasePath;
+  llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+
+  if (llvm::sys::fs::exists(SocketPath)) {
+Expected MaybeFD = connectToSocket(SocketPath);
+if (MaybeFD) {
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Module build daemon already exists";
+  return std::move(*MaybeFD);
+}
+consumeError(MaybeFD.takeError());
+  }
+
+  if (llvm::Error Err =
+  cc1modbuildd::spawnModuleBuildDaemon(BasePath, Argv0, Diag))
+return std::move(Err);
+
+  const unsigned int MICROSEC_IN_SEC = 100;
+  constexpr unsigned int MAX_WAIT_TIME = 30 * 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,211 @@
+//===- Client.cpp 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendDiagnostic.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace clang;
+using namespace llvm;
+using namespace cc1modbuildd;
+
+std::string cc1modbuildd::getBasePath() {
+  llvm::BLAKE3 Hash;
+  Hash.update(getClangFullVersion());
+  auto HashResult = Hash.final();
+  uint64_t HashValue =
+  llvm::support::endian::read(
+  HashResult.data());
+  std::string Key = toString(llvm::APInt(64, HashValue), 36, /*Signed*/ false);
+
+  // Set paths
+  SmallString<128> BasePath;
+  llvm::sys::path::system_temp_directory(/*erasedOnReboot*/ true, BasePath);
+  llvm::sys::path::append(BasePath, "clang-" + Key);
+  return BasePath.c_str();
+}
+
+llvm::Error cc1modbuildd::attemptHandshake(int SocketFD,
+   DiagnosticsEngine ) {
+
+  HandshakeMsg Request{ActionType::HANDSHAKE, StatusType::REQUEST};
+  std::string Buffer = getBufferFromSocketMsg(Request);
+
+  // Send HandshakeMsg to module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Trying to send HandshakeMsg to module build daemon";
+  if (llvm::Error Err = writeToSocket(Buffer, SocketFD))
+return std::move(Err);
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Successfully sent HandshakeMsg to module build daemon";
+
+  // Receive response from module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Waiting to receive module build daemon response";
+  Expected MaybeServerResponse =
+  readSocketMsgFromSocket(SocketFD);
+  if (!MaybeServerResponse)
+return std::move(MaybeServerResponse.takeError());
+  HandshakeMsg ServerResponse = std::move(*MaybeServerResponse);
+
+  assert(ServerResponse.MsgAction == ActionType::HANDSHAKE &&
+ "Response ActionType should only ever be HANDSHAKE");
+
+  if (ServerResponse.MsgStatus == StatusType::SUCCESS) {
+Diag.Report(diag::remark_module_build_daemon)
+<< "Successfully received HandshakeMsg from module build daemon";
+return llvm::Error::success();
+  }
+
+  return llvm::make_error(
+  "Received failed handshake response from module build daemon",
+  inconvertibleErrorCode());
+}
+
+llvm::Error cc1modbuildd::spawnModuleBuildDaemon(StringRef BasePath,
+ const char *Argv0,
+ DiagnosticsEngine ) {
+  std::string BasePathStr = BasePath.str();
+  const char *Args[] = {Argv0, "-cc1modbuildd", BasePathStr.c_str(), nullptr};
+  pid_t pid;
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Trying to spawn module build daemon";
+  int EC = posix_spawn(, Args[0],
+   /*file_actions*/ nullptr,
+   /*spawnattr*/ nullptr, const_cast(Args),
+   /*envp*/ nullptr);
+  if (EC)
+return createStringError(std::error_code(EC, std::generic_category()),
+ "Failed to spawn module build daemon");
+
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Successfully spawned module build daemon";
+  return llvm::Error::success();
+}
+
+Expected cc1modbuildd::getModuleBuildDaemon(const char *Argv0,
+ StringRef BasePath,
+ DiagnosticsEngine ) {
+
+  SmallString<128> SocketPath = BasePath;
+  llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+
+  if (llvm::sys::fs::exists(SocketPath)) {
+Expected MaybeFD = connectToSocket(SocketPath);
+if (MaybeFD) {
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Module build daemon already exists";
+  return std::move(*MaybeFD);
+}
+consumeError(MaybeFD.takeError());
+  }
+
+  if (llvm::Error Err =
+  cc1modbuildd::spawnModuleBuildDaemon(BasePath, Argv0, Diag))
+return std::move(Err);
+
+  const unsigned int MICROSEC_IN_SEC = 100;
+  constexpr unsigned int MAX_WAIT_TIME = 30 * 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,109 @@
+//===- SocketSupport.cpp 
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Config/llvm-config.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+Expected cc1modbuildd::createSocket() {
+  int FD;
+  if ((FD = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) {
+std::string Msg = "socket create error: " + std::string(strerror(errno));
+return createStringError(inconvertibleErrorCode(), Msg);
+  }
+  return FD;
+}
+
+Expected cc1modbuildd::connectToSocket(StringRef SocketPath) {
+
+  Expected MaybeFD = cc1modbuildd::createSocket();
+  if (!MaybeFD)
+return std::move(MaybeFD.takeError());
+
+  int FD = std::move(*MaybeFD);
+
+  struct sockaddr_un Addr;
+  memset(, 0, sizeof(Addr));
+  Addr.sun_family = AF_UNIX;
+  strncpy(Addr.sun_path, SocketPath.str().c_str(), sizeof(Addr.sun_path) - 1);
+
+  if (connect(FD, (struct sockaddr *), sizeof(Addr)) == -1) {
+close(FD);
+std::string msg = "socket connect error: " + std::string(strerror(errno));
+return createStringError(inconvertibleErrorCode(), msg);
+  }
+  return FD;
+}
+
+llvm::Error cc1modbuildd::readFromSocket(int FD, std::string ) {
+
+  char Buffer[MAX_BUFFER];
+  ssize_t n;
+
+  while ((n = read(FD, Buffer, MAX_BUFFER)) > 0) {
+
+BufferConsumer.assign(Buffer, n);
+// Read until ...\n encountered (last line of YAML document)
+if (BufferConsumer.find("...\n") != std::string::npos)

Bigcheese wrote:

This should be "\n...", we just care that the `...` is at the start of a line.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,128 @@
+//===- SocketSupport.cpp 
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Config/llvm-config.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+Expected cc1modbuildd::createSocket() {
+  int FD;
+  if ((FD = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) {
+std::string Msg = "socket create error: " + std::string(strerror(errno));
+return createStringError(inconvertibleErrorCode(), Msg);

Bigcheese wrote:

I think you can just pass the error code instead of `inconvertibleErrorCode()` 
along with the string.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,109 @@
+//===- SocketSupport.cpp 
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Config/llvm-config.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+Expected cc1modbuildd::createSocket() {
+  int FD;
+  if ((FD = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) {
+std::string Msg = "socket create error: " + std::string(strerror(errno));
+return createStringError(inconvertibleErrorCode(), Msg);
+  }
+  return FD;
+}
+
+Expected cc1modbuildd::connectToSocket(StringRef SocketPath) {
+
+  Expected MaybeFD = cc1modbuildd::createSocket();
+  if (!MaybeFD)
+return std::move(MaybeFD.takeError());
+
+  int FD = std::move(*MaybeFD);
+
+  struct sockaddr_un Addr;
+  memset(, 0, sizeof(Addr));
+  Addr.sun_family = AF_UNIX;
+  strncpy(Addr.sun_path, SocketPath.str().c_str(), sizeof(Addr.sun_path) - 1);
+
+  if (connect(FD, (struct sockaddr *), sizeof(Addr)) == -1) {
+close(FD);
+std::string msg = "socket connect error: " + std::string(strerror(errno));
+return createStringError(inconvertibleErrorCode(), msg);
+  }
+  return FD;
+}
+
+llvm::Error cc1modbuildd::readFromSocket(int FD, std::string ) {
+
+  char Buffer[MAX_BUFFER];
+  ssize_t n;
+
+  while ((n = read(FD, Buffer, MAX_BUFFER)) > 0) {
+
+BufferConsumer.assign(Buffer, n);

Bigcheese wrote:

This should be append.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,109 @@
+//===- SocketSupport.cpp 
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Config/llvm-config.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+Expected cc1modbuildd::createSocket() {
+  int FD;
+  if ((FD = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) {
+std::string Msg = "socket create error: " + std::string(strerror(errno));
+return createStringError(inconvertibleErrorCode(), Msg);
+  }
+  return FD;
+}
+
+Expected cc1modbuildd::connectToSocket(StringRef SocketPath) {
+
+  Expected MaybeFD = cc1modbuildd::createSocket();
+  if (!MaybeFD)
+return std::move(MaybeFD.takeError());
+
+  int FD = std::move(*MaybeFD);
+
+  struct sockaddr_un Addr;
+  memset(, 0, sizeof(Addr));
+  Addr.sun_family = AF_UNIX;
+  strncpy(Addr.sun_path, SocketPath.str().c_str(), sizeof(Addr.sun_path) - 1);
+
+  if (connect(FD, (struct sockaddr *), sizeof(Addr)) == -1) {
+close(FD);
+std::string msg = "socket connect error: " + std::string(strerror(errno));
+return createStringError(inconvertibleErrorCode(), msg);
+  }
+  return FD;
+}
+
+llvm::Error cc1modbuildd::readFromSocket(int FD, std::string ) {
+
+  char Buffer[MAX_BUFFER];
+  ssize_t n;
+
+  while ((n = read(FD, Buffer, MAX_BUFFER)) > 0) {
+
+BufferConsumer.assign(Buffer, n);
+// Read until ...\n encountered (last line of YAML document)
+if (BufferConsumer.find("...\n") != std::string::npos)
+  break;
+  }
+
+  if (n < 0) {
+std::string Msg = "socket read error: " + std::string(strerror(errno));
+return llvm::make_error(Msg, inconvertibleErrorCode());
+  }
+  if (n == 0)
+return llvm::make_error("EOF", inconvertibleErrorCode());
+  return llvm::Error::success();
+}
+
+llvm::Error cc1modbuildd::writeToSocket(StringRef Buffer, int WriteFD) {
+
+  ssize_t BytesToWrite = static_cast(Buffer.size());
+  const char *Bytes = Buffer.data();
+
+  while (BytesToWrite) {
+ssize_t BytesWritten = write(WriteFD, Bytes, BytesToWrite);
+if (BytesWritten == -1) {
+  std::string Msg = "socket write error: " + std::string(strerror(errno));
+  return llvm::make_error(Msg, inconvertibleErrorCode());
+}
+
+if (!BytesWritten || BytesWritten > BytesToWrite)

Bigcheese wrote:

`write` won't ever return BytesWritten > BytesToWrite.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,211 @@
+//===- Client.cpp 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendDiagnostic.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace clang;
+using namespace llvm;
+using namespace cc1modbuildd;
+
+std::string cc1modbuildd::getBasePath() {
+  llvm::BLAKE3 Hash;
+  Hash.update(getClangFullVersion());
+  auto HashResult = Hash.final();
+  uint64_t HashValue =
+  llvm::support::endian::read(
+  HashResult.data());
+  std::string Key = toString(llvm::APInt(64, HashValue), 36, /*Signed*/ false);
+
+  // Set paths
+  SmallString<128> BasePath;
+  llvm::sys::path::system_temp_directory(/*erasedOnReboot*/ true, BasePath);
+  llvm::sys::path::append(BasePath, "clang-" + Key);
+  return BasePath.c_str();
+}
+
+llvm::Error cc1modbuildd::attemptHandshake(int SocketFD,
+   DiagnosticsEngine ) {
+
+  HandshakeMsg Request{ActionType::HANDSHAKE, StatusType::REQUEST};
+  std::string Buffer = getBufferFromSocketMsg(Request);
+
+  // Send HandshakeMsg to module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Trying to send HandshakeMsg to module build daemon";
+  if (llvm::Error Err = writeToSocket(Buffer, SocketFD))
+return std::move(Err);
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Successfully sent HandshakeMsg to module build daemon";
+
+  // Receive response from module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Waiting to receive module build daemon response";
+  Expected MaybeServerResponse =
+  readSocketMsgFromSocket(SocketFD);
+  if (!MaybeServerResponse)
+return std::move(MaybeServerResponse.takeError());
+  HandshakeMsg ServerResponse = std::move(*MaybeServerResponse);
+
+  assert(ServerResponse.MsgAction == ActionType::HANDSHAKE &&
+ "Response ActionType should only ever be HANDSHAKE");
+
+  if (ServerResponse.MsgStatus == StatusType::SUCCESS) {
+Diag.Report(diag::remark_module_build_daemon)
+<< "Successfully received HandshakeMsg from module build daemon";
+return llvm::Error::success();
+  }
+
+  return llvm::make_error(
+  "Received failed handshake response from module build daemon",
+  inconvertibleErrorCode());
+}
+
+llvm::Error cc1modbuildd::spawnModuleBuildDaemon(StringRef BasePath,
+ const char *Argv0,
+ DiagnosticsEngine ) {
+  std::string BasePathStr = BasePath.str();
+  const char *Args[] = {Argv0, "-cc1modbuildd", BasePathStr.c_str(), nullptr};
+  pid_t pid;
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Trying to spawn module build daemon";
+  int EC = posix_spawn(, Args[0],
+   /*file_actions*/ nullptr,
+   /*spawnattr*/ nullptr, const_cast(Args),
+   /*envp*/ nullptr);
+  if (EC)
+return createStringError(std::error_code(EC, std::generic_category()),
+ "Failed to spawn module build daemon");
+
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Successfully spawned module build daemon";
+  return llvm::Error::success();
+}
+
+Expected cc1modbuildd::getModuleBuildDaemon(const char *Argv0,
+ StringRef BasePath,
+ DiagnosticsEngine ) {
+
+  SmallString<128> SocketPath = BasePath;
+  llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+
+  if (llvm::sys::fs::exists(SocketPath)) {
+Expected MaybeFD = connectToSocket(SocketPath);
+if (MaybeFD) {
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Module build daemon already exists";
+  return std::move(*MaybeFD);
+}
+consumeError(MaybeFD.takeError());
+  }
+
+  if (llvm::Error Err =
+  cc1modbuildd::spawnModuleBuildDaemon(BasePath, Argv0, Diag))
+return std::move(Err);
+
+  const unsigned int MICROSEC_IN_SEC = 100;
+  constexpr unsigned int MAX_WAIT_TIME = 30 * 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,211 @@
+//===- Client.cpp 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendDiagnostic.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace clang;
+using namespace llvm;
+using namespace cc1modbuildd;
+
+std::string cc1modbuildd::getBasePath() {
+  llvm::BLAKE3 Hash;
+  Hash.update(getClangFullVersion());
+  auto HashResult = Hash.final();
+  uint64_t HashValue =
+  llvm::support::endian::read(
+  HashResult.data());
+  std::string Key = toString(llvm::APInt(64, HashValue), 36, /*Signed*/ false);
+
+  // Set paths
+  SmallString<128> BasePath;
+  llvm::sys::path::system_temp_directory(/*erasedOnReboot*/ true, BasePath);
+  llvm::sys::path::append(BasePath, "clang-" + Key);
+  return BasePath.c_str();
+}
+
+llvm::Error cc1modbuildd::attemptHandshake(int SocketFD,
+   DiagnosticsEngine ) {
+
+  HandshakeMsg Request{ActionType::HANDSHAKE, StatusType::REQUEST};
+  std::string Buffer = getBufferFromSocketMsg(Request);
+
+  // Send HandshakeMsg to module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Trying to send HandshakeMsg to module build daemon";
+  if (llvm::Error Err = writeToSocket(Buffer, SocketFD))
+return std::move(Err);
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Successfully sent HandshakeMsg to module build daemon";
+
+  // Receive response from module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Waiting to receive module build daemon response";
+  Expected MaybeServerResponse =
+  readSocketMsgFromSocket(SocketFD);
+  if (!MaybeServerResponse)
+return std::move(MaybeServerResponse.takeError());
+  HandshakeMsg ServerResponse = std::move(*MaybeServerResponse);
+
+  assert(ServerResponse.MsgAction == ActionType::HANDSHAKE &&
+ "Response ActionType should only ever be HANDSHAKE");
+
+  if (ServerResponse.MsgStatus == StatusType::SUCCESS) {
+Diag.Report(diag::remark_module_build_daemon)
+<< "Successfully received HandshakeMsg from module build daemon";

Bigcheese wrote:

This should just be handled by the other successfully connected remark.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,211 @@
+//===- Client.cpp 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendDiagnostic.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace clang;
+using namespace llvm;
+using namespace cc1modbuildd;
+
+std::string cc1modbuildd::getBasePath() {
+  llvm::BLAKE3 Hash;
+  Hash.update(getClangFullVersion());
+  auto HashResult = Hash.final();
+  uint64_t HashValue =
+  llvm::support::endian::read(
+  HashResult.data());
+  std::string Key = toString(llvm::APInt(64, HashValue), 36, /*Signed*/ false);
+
+  // Set paths
+  SmallString<128> BasePath;
+  llvm::sys::path::system_temp_directory(/*erasedOnReboot*/ true, BasePath);
+  llvm::sys::path::append(BasePath, "clang-" + Key);
+  return BasePath.c_str();
+}
+
+llvm::Error cc1modbuildd::attemptHandshake(int SocketFD,
+   DiagnosticsEngine ) {
+
+  HandshakeMsg Request{ActionType::HANDSHAKE, StatusType::REQUEST};
+  std::string Buffer = getBufferFromSocketMsg(Request);
+
+  // Send HandshakeMsg to module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Trying to send HandshakeMsg to module build daemon";
+  if (llvm::Error Err = writeToSocket(Buffer, SocketFD))
+return std::move(Err);
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Successfully sent HandshakeMsg to module build daemon";
+
+  // Receive response from module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Waiting to receive module build daemon response";
+  Expected MaybeServerResponse =
+  readSocketMsgFromSocket(SocketFD);
+  if (!MaybeServerResponse)
+return std::move(MaybeServerResponse.takeError());
+  HandshakeMsg ServerResponse = std::move(*MaybeServerResponse);
+
+  assert(ServerResponse.MsgAction == ActionType::HANDSHAKE &&
+ "Response ActionType should only ever be HANDSHAKE");
+
+  if (ServerResponse.MsgStatus == StatusType::SUCCESS) {
+Diag.Report(diag::remark_module_build_daemon)
+<< "Successfully received HandshakeMsg from module build daemon";
+return llvm::Error::success();
+  }
+
+  return llvm::make_error(
+  "Received failed handshake response from module build daemon",
+  inconvertibleErrorCode());
+}
+
+llvm::Error cc1modbuildd::spawnModuleBuildDaemon(StringRef BasePath,
+ const char *Argv0,
+ DiagnosticsEngine ) {
+  std::string BasePathStr = BasePath.str();
+  const char *Args[] = {Argv0, "-cc1modbuildd", BasePathStr.c_str(), nullptr};
+  pid_t pid;
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Trying to spawn module build daemon";
+  int EC = posix_spawn(, Args[0],
+   /*file_actions*/ nullptr,
+   /*spawnattr*/ nullptr, const_cast(Args),
+   /*envp*/ nullptr);
+  if (EC)
+return createStringError(std::error_code(EC, std::generic_category()),
+ "Failed to spawn module build daemon");
+
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Successfully spawned module build daemon";
+  return llvm::Error::success();
+}
+
+Expected cc1modbuildd::getModuleBuildDaemon(const char *Argv0,
+ StringRef BasePath,
+ DiagnosticsEngine ) {
+
+  SmallString<128> SocketPath = BasePath;
+  llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+
+  if (llvm::sys::fs::exists(SocketPath)) {
+Expected MaybeFD = connectToSocket(SocketPath);
+if (MaybeFD) {
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Module build daemon already exists";

Bigcheese wrote:

This remark is redundant.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,211 @@
+//===- Client.cpp 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendDiagnostic.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace clang;
+using namespace llvm;
+using namespace cc1modbuildd;
+
+std::string cc1modbuildd::getBasePath() {
+  llvm::BLAKE3 Hash;
+  Hash.update(getClangFullVersion());
+  auto HashResult = Hash.final();
+  uint64_t HashValue =
+  llvm::support::endian::read(
+  HashResult.data());
+  std::string Key = toString(llvm::APInt(64, HashValue), 36, /*Signed*/ false);
+
+  // Set paths
+  SmallString<128> BasePath;
+  llvm::sys::path::system_temp_directory(/*erasedOnReboot*/ true, BasePath);
+  llvm::sys::path::append(BasePath, "clang-" + Key);
+  return BasePath.c_str();
+}
+
+llvm::Error cc1modbuildd::attemptHandshake(int SocketFD,
+   DiagnosticsEngine ) {
+
+  HandshakeMsg Request{ActionType::HANDSHAKE, StatusType::REQUEST};
+  std::string Buffer = getBufferFromSocketMsg(Request);
+
+  // Send HandshakeMsg to module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Trying to send HandshakeMsg to module build daemon";
+  if (llvm::Error Err = writeToSocket(Buffer, SocketFD))
+return std::move(Err);
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Successfully sent HandshakeMsg to module build daemon";
+
+  // Receive response from module build daemon
+  Diag.Report(diag::remark_module_build_daemon)
+  << "Waiting to receive module build daemon response";

Bigcheese wrote:

For the remarks here I think we should just have a single "Attempting to 
connect to ".

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -261,6 +261,11 @@ def err_test_module_file_extension_version : Error<
   "test module file extension '%0' has different version (%1.%2) than expected 
"
   "(%3.%4)">;
 
+def warn_module_build_daemon : Warning<"%0">,
+  InGroup;
+def remark_module_build_daemon : Remark<"%0">,
+  InGroup;

Bigcheese wrote:

These should end up being separate warnings/remarks for each case.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue updated 
https://github.com/llvm/llvm-project/pull/67562

>From 5def87462e3b48cfebafdc2526ac929f5cb9cea3 Mon Sep 17 00:00:00 2001
From: cpsughrue 
Date: Sun, 9 Jul 2023 23:19:58 -0400
Subject: [PATCH 1/5] [clang][MBD] set up module build daemon infrastructure

The module build daemon (mbd) will serve as a cc1 tool that helps convert
implicit module command lines to explicit module command lines. A clang
invocation will check to see if a mbd exists, if not the invocation will spawn
one. After the mbd is up and running and a handshake has successfully been
carried out between the mbd and clang invocation the clang invocation will
share it's command line with the mbd. The mbd will then scan the translation
unit and build all modular dependencies before returning a modified cc1 command
 line such that all arguments related to modules are converted from the
implicit option to their explicit option.

This commit sets up the basic mbd infrastructure. Including the ability to
spawn a mbd and carry out a handshake between the clang invocation and mbd.

RFC: 
https://discourse.llvm.org/t/rfc-modules-build-daemon-build-system-agnostic-support-for-explicitly-built-modules/71524
---
 clang/include/clang/Driver/Options.td |  12 +
 .../include/clang/Frontend/FrontendOptions.h  |   7 +
 .../clang/Tooling/ModuleBuildDaemon/Client.h  |  44 +++
 .../ModuleBuildDaemon/SocketMsgSupport.h  | 134 +
 .../Tooling/ModuleBuildDaemon/SocketSupport.h |  31 ++
 .../clang/Tooling/ModuleBuildDaemon/Utils.h   |  28 ++
 clang/lib/Driver/ToolChains/Clang.cpp |  14 +-
 clang/lib/Tooling/CMakeLists.txt  |   1 +
 .../Tooling/ModuleBuildDaemon/CMakeLists.txt  |   9 +
 .../lib/Tooling/ModuleBuildDaemon/Client.cpp  | 167 +++
 .../ModuleBuildDaemon/SocketSupport.cpp   | 128 +
 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp |  32 +++
 clang/test/Driver/unknown-arg.c   |   2 +-
 clang/test/ModuleBuildDaemon/handshake.c  |  18 ++
 clang/test/ModuleBuildDaemon/launch.c |  14 +
 clang/tools/driver/CMakeLists.txt |   3 +
 clang/tools/driver/cc1_main.cpp   |  28 +-
 clang/tools/driver/cc1modbuildd_main.cpp  | 267 ++
 clang/tools/driver/driver.cpp |  17 +-
 19 files changed, 947 insertions(+), 9 deletions(-)
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Client.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketSupport.h
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Utils.h
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/CMakeLists.txt
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Client.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/SocketSupport.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp
 create mode 100644 clang/test/ModuleBuildDaemon/handshake.c
 create mode 100644 clang/test/ModuleBuildDaemon/launch.c
 create mode 100644 clang/tools/driver/cc1modbuildd_main.cpp

diff --git a/clang/include/clang/Driver/Options.td 
b/clang/include/clang/Driver/Options.td
index efd90942948af27..d31d28255e81a8a 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -2836,6 +2836,18 @@ defm declspec : BoolOption<"f", "declspec",
   NegFlag,
   BothFlags<[], [ClangOption, CC1Option],
   " __declspec as a keyword">>, Group;
+
+def fmodule_build_daemon : Flag<["-"], "fmodule-build-daemon">, Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality">,
+  MarshallingInfoFlag>;
+def fmodule_build_daemon_EQ : Joined<["-"], "fmodule-build-daemon=">, 
Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality and defines location of 
output files">,
+  MarshallingInfoString>;
+
 def fmodules_cache_path : Joined<["-"], "fmodules-cache-path=">, 
Group,
   Flags<[NoXarchOption]>, Visibility<[ClangOption, CC1Option]>,
   MetaVarName<"">,
diff --git a/clang/include/clang/Frontend/FrontendOptions.h 
b/clang/include/clang/Frontend/FrontendOptions.h
index 117e35de6f76c4c..8ce97a57d413c0b 100644
--- a/clang/include/clang/Frontend/FrontendOptions.h
+++ b/clang/include/clang/Frontend/FrontendOptions.h
@@ -350,6 +350,9 @@ class FrontendOptions {
   /// Whether to share the FileManager when building modules.
   unsigned ModulesShareFileManager : 1;
 
+  /// Connect to module build daemon
+  unsigned ModuleBuildDaemon : 1;
+
   CodeCompleteOptions CodeCompleteOpts;
 
   /// Specifies the output format of the AST.
@@ -435,6 +438,10 @@ class FrontendOptions {
   /// The output file, if any.
   std::string OutputFile;
 
+  /// If given, the path to the module build daemon's output files and socket
+  /// address
+  std::string 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue updated 
https://github.com/llvm/llvm-project/pull/67562

>From 5def87462e3b48cfebafdc2526ac929f5cb9cea3 Mon Sep 17 00:00:00 2001
From: cpsughrue 
Date: Sun, 9 Jul 2023 23:19:58 -0400
Subject: [PATCH 1/4] [clang][MBD] set up module build daemon infrastructure

The module build daemon (mbd) will serve as a cc1 tool that helps convert
implicit module command lines to explicit module command lines. A clang
invocation will check to see if a mbd exists, if not the invocation will spawn
one. After the mbd is up and running and a handshake has successfully been
carried out between the mbd and clang invocation the clang invocation will
share it's command line with the mbd. The mbd will then scan the translation
unit and build all modular dependencies before returning a modified cc1 command
 line such that all arguments related to modules are converted from the
implicit option to their explicit option.

This commit sets up the basic mbd infrastructure. Including the ability to
spawn a mbd and carry out a handshake between the clang invocation and mbd.

RFC: 
https://discourse.llvm.org/t/rfc-modules-build-daemon-build-system-agnostic-support-for-explicitly-built-modules/71524
---
 clang/include/clang/Driver/Options.td |  12 +
 .../include/clang/Frontend/FrontendOptions.h  |   7 +
 .../clang/Tooling/ModuleBuildDaemon/Client.h  |  44 +++
 .../ModuleBuildDaemon/SocketMsgSupport.h  | 134 +
 .../Tooling/ModuleBuildDaemon/SocketSupport.h |  31 ++
 .../clang/Tooling/ModuleBuildDaemon/Utils.h   |  28 ++
 clang/lib/Driver/ToolChains/Clang.cpp |  14 +-
 clang/lib/Tooling/CMakeLists.txt  |   1 +
 .../Tooling/ModuleBuildDaemon/CMakeLists.txt  |   9 +
 .../lib/Tooling/ModuleBuildDaemon/Client.cpp  | 167 +++
 .../ModuleBuildDaemon/SocketSupport.cpp   | 128 +
 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp |  32 +++
 clang/test/Driver/unknown-arg.c   |   2 +-
 clang/test/ModuleBuildDaemon/handshake.c  |  18 ++
 clang/test/ModuleBuildDaemon/launch.c |  14 +
 clang/tools/driver/CMakeLists.txt |   3 +
 clang/tools/driver/cc1_main.cpp   |  28 +-
 clang/tools/driver/cc1modbuildd_main.cpp  | 267 ++
 clang/tools/driver/driver.cpp |  17 +-
 19 files changed, 947 insertions(+), 9 deletions(-)
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Client.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketSupport.h
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Utils.h
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/CMakeLists.txt
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Client.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/SocketSupport.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp
 create mode 100644 clang/test/ModuleBuildDaemon/handshake.c
 create mode 100644 clang/test/ModuleBuildDaemon/launch.c
 create mode 100644 clang/tools/driver/cc1modbuildd_main.cpp

diff --git a/clang/include/clang/Driver/Options.td 
b/clang/include/clang/Driver/Options.td
index efd90942948af27..d31d28255e81a8a 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -2836,6 +2836,18 @@ defm declspec : BoolOption<"f", "declspec",
   NegFlag,
   BothFlags<[], [ClangOption, CC1Option],
   " __declspec as a keyword">>, Group;
+
+def fmodule_build_daemon : Flag<["-"], "fmodule-build-daemon">, Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality">,
+  MarshallingInfoFlag>;
+def fmodule_build_daemon_EQ : Joined<["-"], "fmodule-build-daemon=">, 
Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality and defines location of 
output files">,
+  MarshallingInfoString>;
+
 def fmodules_cache_path : Joined<["-"], "fmodules-cache-path=">, 
Group,
   Flags<[NoXarchOption]>, Visibility<[ClangOption, CC1Option]>,
   MetaVarName<"">,
diff --git a/clang/include/clang/Frontend/FrontendOptions.h 
b/clang/include/clang/Frontend/FrontendOptions.h
index 117e35de6f76c4c..8ce97a57d413c0b 100644
--- a/clang/include/clang/Frontend/FrontendOptions.h
+++ b/clang/include/clang/Frontend/FrontendOptions.h
@@ -350,6 +350,9 @@ class FrontendOptions {
   /// Whether to share the FileManager when building modules.
   unsigned ModulesShareFileManager : 1;
 
+  /// Connect to module build daemon
+  unsigned ModuleBuildDaemon : 1;
+
   CodeCompleteOptions CodeCompleteOpts;
 
   /// Specifies the output format of the AST.
@@ -435,6 +438,10 @@ class FrontendOptions {
   /// The output file, if any.
   std::string OutputFile;
 
+  /// If given, the path to the module build daemon's output files and socket
+  /// address
+  std::string 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue edited 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue updated 
https://github.com/llvm/llvm-project/pull/67562

>From 5def87462e3b48cfebafdc2526ac929f5cb9cea3 Mon Sep 17 00:00:00 2001
From: cpsughrue 
Date: Sun, 9 Jul 2023 23:19:58 -0400
Subject: [PATCH 1/3] [clang][MBD] set up module build daemon infrastructure

The module build daemon (mbd) will serve as a cc1 tool that helps convert
implicit module command lines to explicit module command lines. A clang
invocation will check to see if a mbd exists, if not the invocation will spawn
one. After the mbd is up and running and a handshake has successfully been
carried out between the mbd and clang invocation the clang invocation will
share it's command line with the mbd. The mbd will then scan the translation
unit and build all modular dependencies before returning a modified cc1 command
 line such that all arguments related to modules are converted from the
implicit option to their explicit option.

This commit sets up the basic mbd infrastructure. Including the ability to
spawn a mbd and carry out a handshake between the clang invocation and mbd.

RFC: 
https://discourse.llvm.org/t/rfc-modules-build-daemon-build-system-agnostic-support-for-explicitly-built-modules/71524
---
 clang/include/clang/Driver/Options.td |  12 +
 .../include/clang/Frontend/FrontendOptions.h  |   7 +
 .../clang/Tooling/ModuleBuildDaemon/Client.h  |  44 +++
 .../ModuleBuildDaemon/SocketMsgSupport.h  | 134 +
 .../Tooling/ModuleBuildDaemon/SocketSupport.h |  31 ++
 .../clang/Tooling/ModuleBuildDaemon/Utils.h   |  28 ++
 clang/lib/Driver/ToolChains/Clang.cpp |  14 +-
 clang/lib/Tooling/CMakeLists.txt  |   1 +
 .../Tooling/ModuleBuildDaemon/CMakeLists.txt  |   9 +
 .../lib/Tooling/ModuleBuildDaemon/Client.cpp  | 167 +++
 .../ModuleBuildDaemon/SocketSupport.cpp   | 128 +
 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp |  32 +++
 clang/test/Driver/unknown-arg.c   |   2 +-
 clang/test/ModuleBuildDaemon/handshake.c  |  18 ++
 clang/test/ModuleBuildDaemon/launch.c |  14 +
 clang/tools/driver/CMakeLists.txt |   3 +
 clang/tools/driver/cc1_main.cpp   |  28 +-
 clang/tools/driver/cc1modbuildd_main.cpp  | 267 ++
 clang/tools/driver/driver.cpp |  17 +-
 19 files changed, 947 insertions(+), 9 deletions(-)
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Client.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketSupport.h
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Utils.h
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/CMakeLists.txt
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Client.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/SocketSupport.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp
 create mode 100644 clang/test/ModuleBuildDaemon/handshake.c
 create mode 100644 clang/test/ModuleBuildDaemon/launch.c
 create mode 100644 clang/tools/driver/cc1modbuildd_main.cpp

diff --git a/clang/include/clang/Driver/Options.td 
b/clang/include/clang/Driver/Options.td
index efd90942948af27..d31d28255e81a8a 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -2836,6 +2836,18 @@ defm declspec : BoolOption<"f", "declspec",
   NegFlag,
   BothFlags<[], [ClangOption, CC1Option],
   " __declspec as a keyword">>, Group;
+
+def fmodule_build_daemon : Flag<["-"], "fmodule-build-daemon">, Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality">,
+  MarshallingInfoFlag>;
+def fmodule_build_daemon_EQ : Joined<["-"], "fmodule-build-daemon=">, 
Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality and defines location of 
output files">,
+  MarshallingInfoString>;
+
 def fmodules_cache_path : Joined<["-"], "fmodules-cache-path=">, 
Group,
   Flags<[NoXarchOption]>, Visibility<[ClangOption, CC1Option]>,
   MetaVarName<"">,
diff --git a/clang/include/clang/Frontend/FrontendOptions.h 
b/clang/include/clang/Frontend/FrontendOptions.h
index 117e35de6f76c4c..8ce97a57d413c0b 100644
--- a/clang/include/clang/Frontend/FrontendOptions.h
+++ b/clang/include/clang/Frontend/FrontendOptions.h
@@ -350,6 +350,9 @@ class FrontendOptions {
   /// Whether to share the FileManager when building modules.
   unsigned ModulesShareFileManager : 1;
 
+  /// Connect to module build daemon
+  unsigned ModuleBuildDaemon : 1;
+
   CodeCompleteOptions CodeCompleteOpts;
 
   /// Specifies the output format of the AST.
@@ -435,6 +438,10 @@ class FrontendOptions {
   /// The output file, if any.
   std::string OutputFile;
 
+  /// If given, the path to the module build daemon's output files and socket
+  /// address
+  std::string 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

@@ -0,0 +1,18 @@
+// Check that clang invocation can spawn and handshake with module build daemon
+
+// REQUIRES: !system-windows
+
+//  RUN: if pgrep -f "cc1modbuildd mbd-handshake"; then pkill -f "cc1modbuildd 
mbd-handshake"; fi
+//  RUN: rm -rf mbd-handshake %t

cpsughrue wrote:

I could also set up the client to have a max length of 108 too which would 
guarantee they always match.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue updated 
https://github.com/llvm/llvm-project/pull/67562

>From 9c8e6c71084e1f0e7a96987b7aa2d251b02bea48 Mon Sep 17 00:00:00 2001
From: cpsughrue 
Date: Sun, 9 Jul 2023 23:19:58 -0400
Subject: [PATCH 1/3] [clang][MBD] set up module build daemon infrastructure

The module build daemon (mbd) will serve as a cc1 tool that helps convert
implicit module command lines to explicit module command lines. A clang
invocation will check to see if a mbd exists, if not the invocation will spawn
one. After the mbd is up and running and a handshake has successfully been
carried out between the mbd and clang invocation the clang invocation will
share it's command line with the mbd. The mbd will then scan the translation
unit and build all modular dependencies before returning a modified cc1 command
 line such that all arguments related to modules are converted from the
implicit option to their explicit option.

This commit sets up the basic mbd infrastructure. Including the ability to
spawn a mbd and carry out a handshake between the clang invocation and mbd.

RFC: 
https://discourse.llvm.org/t/rfc-modules-build-daemon-build-system-agnostic-support-for-explicitly-built-modules/71524
---
 clang/include/clang/Driver/Options.td |  12 +
 .../include/clang/Frontend/FrontendOptions.h  |   7 +
 .../clang/Tooling/ModuleBuildDaemon/Client.h  |  44 +++
 .../ModuleBuildDaemon/SocketMsgSupport.h  | 134 +
 .../Tooling/ModuleBuildDaemon/SocketSupport.h |  31 ++
 .../clang/Tooling/ModuleBuildDaemon/Utils.h   |  28 ++
 clang/lib/Driver/ToolChains/Clang.cpp |  14 +-
 clang/lib/Tooling/CMakeLists.txt  |   1 +
 .../Tooling/ModuleBuildDaemon/CMakeLists.txt  |   9 +
 .../lib/Tooling/ModuleBuildDaemon/Client.cpp  | 167 +++
 .../ModuleBuildDaemon/SocketSupport.cpp   | 128 +
 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp |  32 +++
 clang/test/Driver/unknown-arg.c   |   2 +-
 clang/test/ModuleBuildDaemon/handshake.c  |  18 ++
 clang/test/ModuleBuildDaemon/launch.c |  14 +
 clang/tools/driver/CMakeLists.txt |   3 +
 clang/tools/driver/cc1_main.cpp   |  28 +-
 clang/tools/driver/cc1modbuildd_main.cpp  | 267 ++
 clang/tools/driver/driver.cpp |  17 +-
 19 files changed, 947 insertions(+), 9 deletions(-)
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Client.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketSupport.h
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Utils.h
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/CMakeLists.txt
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Client.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/SocketSupport.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp
 create mode 100644 clang/test/ModuleBuildDaemon/handshake.c
 create mode 100644 clang/test/ModuleBuildDaemon/launch.c
 create mode 100644 clang/tools/driver/cc1modbuildd_main.cpp

diff --git a/clang/include/clang/Driver/Options.td 
b/clang/include/clang/Driver/Options.td
index f573f5a06ceb501..3887dad2ef77f58 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -2836,6 +2836,18 @@ defm declspec : BoolOption<"f", "declspec",
   NegFlag,
   BothFlags<[], [ClangOption, CC1Option],
   " __declspec as a keyword">>, Group;
+
+def fmodule_build_daemon : Flag<["-"], "fmodule-build-daemon">, Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality">,
+  MarshallingInfoFlag>;
+def fmodule_build_daemon_EQ : Joined<["-"], "fmodule-build-daemon=">, 
Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality and defines location of 
output files">,
+  MarshallingInfoString>;
+
 def fmodules_cache_path : Joined<["-"], "fmodules-cache-path=">, 
Group,
   Flags<[NoXarchOption]>, Visibility<[ClangOption, CC1Option]>,
   MetaVarName<"">,
diff --git a/clang/include/clang/Frontend/FrontendOptions.h 
b/clang/include/clang/Frontend/FrontendOptions.h
index 117e35de6f76c4c..8ce97a57d413c0b 100644
--- a/clang/include/clang/Frontend/FrontendOptions.h
+++ b/clang/include/clang/Frontend/FrontendOptions.h
@@ -350,6 +350,9 @@ class FrontendOptions {
   /// Whether to share the FileManager when building modules.
   unsigned ModulesShareFileManager : 1;
 
+  /// Connect to module build daemon
+  unsigned ModuleBuildDaemon : 1;
+
   CodeCompleteOptions CodeCompleteOpts;
 
   /// Specifies the output format of the AST.
@@ -435,6 +438,10 @@ class FrontendOptions {
   /// The output file, if any.
   std::string OutputFile;
 
+  /// If given, the path to the module build daemon's output files and socket
+  /// address
+  std::string 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[via cfe-commits]

github-actions[bot] wrote:




:warning: C/C++ code formatter, clang-format found issues in your code. 
:warning:



You can test this locally with the following command:


``bash
git-clang-format --diff b4a89a5edf08af9e03324274ebdce7e032ce 
18d413991f49162923172fbb8ee5016eef954636 -- 
clang/include/clang/Tooling/ModuleBuildDaemon/Client.h 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketSupport.h 
clang/include/clang/Tooling/ModuleBuildDaemon/Utils.h 
clang/lib/Tooling/ModuleBuildDaemon/Client.cpp 
clang/lib/Tooling/ModuleBuildDaemon/SocketSupport.cpp 
clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp 
clang/test/ModuleBuildDaemon/handshake.c clang/test/ModuleBuildDaemon/launch.c 
clang/tools/driver/cc1modbuildd_main.cpp 
clang/include/clang/Frontend/FrontendOptions.h 
clang/lib/Driver/ToolChains/Clang.cpp clang/test/Driver/unknown-arg.c 
clang/tools/driver/cc1_main.cpp clang/tools/driver/driver.cpp
``





View the diff from clang-format here.


``diff
diff --git a/clang/lib/Tooling/ModuleBuildDaemon/Client.cpp 
b/clang/lib/Tooling/ModuleBuildDaemon/Client.cpp
index 576a3c0ce9c0..766b78ee940d 100644
--- a/clang/lib/Tooling/ModuleBuildDaemon/Client.cpp
+++ b/clang/lib/Tooling/ModuleBuildDaemon/Client.cpp
@@ -126,7 +126,7 @@ Expected cc1modbuildd::getModuleBuildDaemon(const char 
*Argv0,
 return std::move(*MaybeFD);
   consumeError(MaybeFD.takeError());
 }
-
+
 CumulativeTime += WaitTime;
 // Exponential backoff
 WaitTime = WaitTime * 2;

``




https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue updated 
https://github.com/llvm/llvm-project/pull/67562

>From 9c8e6c71084e1f0e7a96987b7aa2d251b02bea48 Mon Sep 17 00:00:00 2001
From: cpsughrue 
Date: Sun, 9 Jul 2023 23:19:58 -0400
Subject: [PATCH 1/3] [clang][MBD] set up module build daemon infrastructure

The module build daemon (mbd) will serve as a cc1 tool that helps convert
implicit module command lines to explicit module command lines. A clang
invocation will check to see if a mbd exists, if not the invocation will spawn
one. After the mbd is up and running and a handshake has successfully been
carried out between the mbd and clang invocation the clang invocation will
share it's command line with the mbd. The mbd will then scan the translation
unit and build all modular dependencies before returning a modified cc1 command
 line such that all arguments related to modules are converted from the
implicit option to their explicit option.

This commit sets up the basic mbd infrastructure. Including the ability to
spawn a mbd and carry out a handshake between the clang invocation and mbd.

RFC: 
https://discourse.llvm.org/t/rfc-modules-build-daemon-build-system-agnostic-support-for-explicitly-built-modules/71524
---
 clang/include/clang/Driver/Options.td |  12 +
 .../include/clang/Frontend/FrontendOptions.h  |   7 +
 .../clang/Tooling/ModuleBuildDaemon/Client.h  |  44 +++
 .../ModuleBuildDaemon/SocketMsgSupport.h  | 134 +
 .../Tooling/ModuleBuildDaemon/SocketSupport.h |  31 ++
 .../clang/Tooling/ModuleBuildDaemon/Utils.h   |  28 ++
 clang/lib/Driver/ToolChains/Clang.cpp |  14 +-
 clang/lib/Tooling/CMakeLists.txt  |   1 +
 .../Tooling/ModuleBuildDaemon/CMakeLists.txt  |   9 +
 .../lib/Tooling/ModuleBuildDaemon/Client.cpp  | 167 +++
 .../ModuleBuildDaemon/SocketSupport.cpp   | 128 +
 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp |  32 +++
 clang/test/Driver/unknown-arg.c   |   2 +-
 clang/test/ModuleBuildDaemon/handshake.c  |  18 ++
 clang/test/ModuleBuildDaemon/launch.c |  14 +
 clang/tools/driver/CMakeLists.txt |   3 +
 clang/tools/driver/cc1_main.cpp   |  28 +-
 clang/tools/driver/cc1modbuildd_main.cpp  | 267 ++
 clang/tools/driver/driver.cpp |  17 +-
 19 files changed, 947 insertions(+), 9 deletions(-)
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Client.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketSupport.h
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Utils.h
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/CMakeLists.txt
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Client.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/SocketSupport.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp
 create mode 100644 clang/test/ModuleBuildDaemon/handshake.c
 create mode 100644 clang/test/ModuleBuildDaemon/launch.c
 create mode 100644 clang/tools/driver/cc1modbuildd_main.cpp

diff --git a/clang/include/clang/Driver/Options.td 
b/clang/include/clang/Driver/Options.td
index f573f5a06ceb501..3887dad2ef77f58 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -2836,6 +2836,18 @@ defm declspec : BoolOption<"f", "declspec",
   NegFlag,
   BothFlags<[], [ClangOption, CC1Option],
   " __declspec as a keyword">>, Group;
+
+def fmodule_build_daemon : Flag<["-"], "fmodule-build-daemon">, Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality">,
+  MarshallingInfoFlag>;
+def fmodule_build_daemon_EQ : Joined<["-"], "fmodule-build-daemon=">, 
Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality and defines location of 
output files">,
+  MarshallingInfoString>;
+
 def fmodules_cache_path : Joined<["-"], "fmodules-cache-path=">, 
Group,
   Flags<[NoXarchOption]>, Visibility<[ClangOption, CC1Option]>,
   MetaVarName<"">,
diff --git a/clang/include/clang/Frontend/FrontendOptions.h 
b/clang/include/clang/Frontend/FrontendOptions.h
index 117e35de6f76c4c..8ce97a57d413c0b 100644
--- a/clang/include/clang/Frontend/FrontendOptions.h
+++ b/clang/include/clang/Frontend/FrontendOptions.h
@@ -350,6 +350,9 @@ class FrontendOptions {
   /// Whether to share the FileManager when building modules.
   unsigned ModulesShareFileManager : 1;
 
+  /// Connect to module build daemon
+  unsigned ModuleBuildDaemon : 1;
+
   CodeCompleteOptions CodeCompleteOpts;
 
   /// Specifies the output format of the AST.
@@ -435,6 +438,10 @@ class FrontendOptions {
   /// The output file, if any.
   std::string OutputFile;
 
+  /// If given, the path to the module build daemon's output files and socket
+  /// address
+  std::string 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

@@ -0,0 +1,128 @@
+//===- SocketSupport.cpp 
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Config/llvm-config.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+Expected cc1modbuildd::createSocket() {
+  int FD;
+  if ((FD = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) {
+std::string Msg = "socket create error: " + std::string(strerror(errno));
+return createStringError(inconvertibleErrorCode(), Msg);

cpsughrue wrote:

I've been trying to write error message that contain the errno description and 
some developer-provided context. For example, `std::perror("Socket bind error: 
");` might print something like `"Socket bind error: No such file or 
directory"` which follows the format `"developer-provided context: errno 
description"`. I think that format does a good job of providing a helpful error 
message.

Is there a way to return developer-provided context alongside the llvm::Error 
formed from errno? When returning a `llvm::Error` with something like `return 
llvm::errorCodeToError(std::error_code(EIO, std::generic_category()));` I have 
not been able to figure out how to return anything other then the error code.


https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue unresolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue unresolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue unresolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue unresolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue unresolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue unresolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue unresolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue unresolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue unresolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue unresolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue resolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue resolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue resolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue resolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue resolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue resolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue resolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue resolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue resolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue resolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

@@ -0,0 +1,267 @@
+//===--- cc1modbuildd_main.cpp - Clang CC1 Module Build Daemon 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Support/Threading.h"
+#include "llvm/Support/YAMLParser.h"
+#include "llvm/Support/YAMLTraits.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace llvm;
+using namespace clang;
+using namespace cc1modbuildd;
+
+// Create unbuffered STDOUT stream so that any logging done by module build
+// daemon can be viewed without having to terminate the process
+static raw_fd_ostream _outs() {
+  static raw_fd_ostream S(STDOUT_FILENO, false, true);
+  return S;
+}
+
+namespace {
+
+class ModuleBuildDaemonServer {
+public:
+  SmallString<128> BasePath;
+  SmallString<128> SocketPath;
+  SmallString<128> PidPath;
+
+  ModuleBuildDaemonServer(SmallString<128> Path, ArrayRef Argv)

cpsughrue wrote:

Is there an issue with making `Path` a `StringRef`?

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

@@ -0,0 +1,267 @@
+//===--- cc1modbuildd_main.cpp - Clang CC1 Module Build Daemon 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Support/Threading.h"
+#include "llvm/Support/YAMLParser.h"
+#include "llvm/Support/YAMLTraits.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace llvm;
+using namespace clang;
+using namespace cc1modbuildd;
+
+// Create unbuffered STDOUT stream so that any logging done by module build
+// daemon can be viewed without having to terminate the process
+static raw_fd_ostream _outs() {
+  static raw_fd_ostream S(STDOUT_FILENO, false, true);
+  return S;
+}
+
+namespace {
+
+class ModuleBuildDaemonServer {
+public:
+  SmallString<128> BasePath;
+  SmallString<128> SocketPath;
+  SmallString<128> PidPath;

cpsughrue wrote:

I use those variables with `llvm::sys::path::append(SmallVectorImpl, Twine)` so 
I think they have to be `SmallString`

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue resolved 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue updated 
https://github.com/llvm/llvm-project/pull/67562

>From 9c8e6c71084e1f0e7a96987b7aa2d251b02bea48 Mon Sep 17 00:00:00 2001
From: cpsughrue 
Date: Sun, 9 Jul 2023 23:19:58 -0400
Subject: [PATCH 1/2] [clang][MBD] set up module build daemon infrastructure

The module build daemon (mbd) will serve as a cc1 tool that helps convert
implicit module command lines to explicit module command lines. A clang
invocation will check to see if a mbd exists, if not the invocation will spawn
one. After the mbd is up and running and a handshake has successfully been
carried out between the mbd and clang invocation the clang invocation will
share it's command line with the mbd. The mbd will then scan the translation
unit and build all modular dependencies before returning a modified cc1 command
 line such that all arguments related to modules are converted from the
implicit option to their explicit option.

This commit sets up the basic mbd infrastructure. Including the ability to
spawn a mbd and carry out a handshake between the clang invocation and mbd.

RFC: 
https://discourse.llvm.org/t/rfc-modules-build-daemon-build-system-agnostic-support-for-explicitly-built-modules/71524
---
 clang/include/clang/Driver/Options.td |  12 +
 .../include/clang/Frontend/FrontendOptions.h  |   7 +
 .../clang/Tooling/ModuleBuildDaemon/Client.h  |  44 +++
 .../ModuleBuildDaemon/SocketMsgSupport.h  | 134 +
 .../Tooling/ModuleBuildDaemon/SocketSupport.h |  31 ++
 .../clang/Tooling/ModuleBuildDaemon/Utils.h   |  28 ++
 clang/lib/Driver/ToolChains/Clang.cpp |  14 +-
 clang/lib/Tooling/CMakeLists.txt  |   1 +
 .../Tooling/ModuleBuildDaemon/CMakeLists.txt  |   9 +
 .../lib/Tooling/ModuleBuildDaemon/Client.cpp  | 167 +++
 .../ModuleBuildDaemon/SocketSupport.cpp   | 128 +
 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp |  32 +++
 clang/test/Driver/unknown-arg.c   |   2 +-
 clang/test/ModuleBuildDaemon/handshake.c  |  18 ++
 clang/test/ModuleBuildDaemon/launch.c |  14 +
 clang/tools/driver/CMakeLists.txt |   3 +
 clang/tools/driver/cc1_main.cpp   |  28 +-
 clang/tools/driver/cc1modbuildd_main.cpp  | 267 ++
 clang/tools/driver/driver.cpp |  17 +-
 19 files changed, 947 insertions(+), 9 deletions(-)
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Client.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketSupport.h
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Utils.h
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/CMakeLists.txt
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Client.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/SocketSupport.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp
 create mode 100644 clang/test/ModuleBuildDaemon/handshake.c
 create mode 100644 clang/test/ModuleBuildDaemon/launch.c
 create mode 100644 clang/tools/driver/cc1modbuildd_main.cpp

diff --git a/clang/include/clang/Driver/Options.td 
b/clang/include/clang/Driver/Options.td
index f573f5a06ceb501..3887dad2ef77f58 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -2836,6 +2836,18 @@ defm declspec : BoolOption<"f", "declspec",
   NegFlag,
   BothFlags<[], [ClangOption, CC1Option],
   " __declspec as a keyword">>, Group;
+
+def fmodule_build_daemon : Flag<["-"], "fmodule-build-daemon">, Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality">,
+  MarshallingInfoFlag>;
+def fmodule_build_daemon_EQ : Joined<["-"], "fmodule-build-daemon=">, 
Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality and defines location of 
output files">,
+  MarshallingInfoString>;
+
 def fmodules_cache_path : Joined<["-"], "fmodules-cache-path=">, 
Group,
   Flags<[NoXarchOption]>, Visibility<[ClangOption, CC1Option]>,
   MetaVarName<"">,
diff --git a/clang/include/clang/Frontend/FrontendOptions.h 
b/clang/include/clang/Frontend/FrontendOptions.h
index 117e35de6f76c4c..8ce97a57d413c0b 100644
--- a/clang/include/clang/Frontend/FrontendOptions.h
+++ b/clang/include/clang/Frontend/FrontendOptions.h
@@ -350,6 +350,9 @@ class FrontendOptions {
   /// Whether to share the FileManager when building modules.
   unsigned ModulesShareFileManager : 1;
 
+  /// Connect to module build daemon
+  unsigned ModuleBuildDaemon : 1;
+
   CodeCompleteOptions CodeCompleteOpts;
 
   /// Specifies the output format of the AST.
@@ -435,6 +438,10 @@ class FrontendOptions {
   /// The output file, if any.
   std::string OutputFile;
 
+  /// If given, the path to the module build daemon's output files and socket
+  /// address
+  std::string 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

@@ -0,0 +1,18 @@
+// Check that clang invocation can spawn and handshake with module build daemon
+
+// REQUIRES: !system-windows
+
+//  RUN: if pgrep -f "cc1modbuildd mbd-handshake"; then pkill -f "cc1modbuildd 
mbd-handshake"; fi
+//  RUN: rm -rf mbd-handshake %t

cpsughrue wrote:

Looks like the header file `un.h`, provided by 
glibc-headers-2.34-60.el9.x86_64, sets the max unix socket address length to 
108 characters.

```cpp
// /usr/include/sys/un.h
/* Structure describing the address of an AF_LOCAL (aka AF_UNIX) socket.  */
struct sockaddr_un
  {
__SOCKADDR_COMMON (sun_);
char sun_path[108]; /* Path name.  */
  };
```

`%t` results in a socket address of 
`/home/cpsughrue/repos/llvm-project-fork/build/tools/clang/test/ModuleBuildDaemon/Output/handshake.c.tmp/mbd.sock`
 which is 112 characters. So, in this case, the clang invocation looks to 
communicate over `mbd.sock` under 
`/home/cpsughrue/repos/llvm-project-fork/build/tools/clang/test/ModuleBuildDaemon/Output/handshake.c.tmp`
 but the daemon chops off `.sock` and creates the address `mbd`. 

I can shorten the file name to `hand.c` or use 
`-fmodule-build-daemon=handshake-mbd` which places the socket file at 
`llvm-project/build/tools/clang/test/ModuleBuildDaemon/handshake-mbd/mbd.sock`. 
Any preference?

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,267 @@
+//===--- cc1modbuildd_main.cpp - Clang CC1 Module Build Daemon 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Support/Threading.h"
+#include "llvm/Support/YAMLParser.h"
+#include "llvm/Support/YAMLTraits.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace llvm;
+using namespace clang;
+using namespace cc1modbuildd;
+
+// Create unbuffered STDOUT stream so that any logging done by module build
+// daemon can be viewed without having to terminate the process
+static raw_fd_ostream _outs() {
+  static raw_fd_ostream S(STDOUT_FILENO, false, true);
+  return S;
+}
+
+namespace {
+
+class ModuleBuildDaemonServer {
+public:
+  SmallString<128> BasePath;
+  SmallString<128> SocketPath;
+  SmallString<128> PidPath;
+
+  ModuleBuildDaemonServer(SmallString<128> Path, ArrayRef Argv)
+  : BasePath(Path), SocketPath(Path) {
+llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+  }
+
+  ~ModuleBuildDaemonServer() { shutdownDaemon(SIGTERM); }
+
+  int forkDaemon();
+  int launchDaemon();
+  int listenForClients();
+
+  static void handleClient(int Client);
+
+  void shutdownDaemon(int signal) {
+unlink(SocketPath.c_str());
+shutdown(ListenSocketFD, SHUT_RD);
+close(ListenSocketFD);
+exit(EXIT_SUCCESS);
+  }
+
+private:
+  // Initializes and returns DiagnosticsEngine
+  pid_t Pid = -1;
+  int ListenSocketFD = -1;
+};
+
+// Required to handle SIGTERM by calling Shutdown
+ModuleBuildDaemonServer *DaemonPtr = nullptr;
+void handleSignal(int Signal) {
+  if (DaemonPtr != nullptr) {
+DaemonPtr->shutdownDaemon(Signal);

Bigcheese wrote:

I did some more checking on this. `DaemonPtr` is fine as the signal handler is 
installed after this is set. As for `shutdownDaemon`, all the functions it 
currently calls are signal safe according to POSIX, but `ListenSocketFD` also 
needs to be atomic as it is written after the signal handler is installed.

Basically you need to treat the signal handler as running in another thread 
that starts and synchronizes with when the signal handler gets installed.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,267 @@
+//===--- cc1modbuildd_main.cpp - Clang CC1 Module Build Daemon 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Support/Threading.h"
+#include "llvm/Support/YAMLParser.h"
+#include "llvm/Support/YAMLTraits.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace llvm;
+using namespace clang;
+using namespace cc1modbuildd;
+
+// Create unbuffered STDOUT stream so that any logging done by module build
+// daemon can be viewed without having to terminate the process
+static raw_fd_ostream _outs() {
+  static raw_fd_ostream S(STDOUT_FILENO, false, true);
+  return S;
+}
+
+namespace {
+
+class ModuleBuildDaemonServer {
+public:
+  SmallString<128> BasePath;
+  SmallString<128> SocketPath;
+  SmallString<128> PidPath;
+
+  ModuleBuildDaemonServer(SmallString<128> Path, ArrayRef Argv)
+  : BasePath(Path), SocketPath(Path) {
+llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+  }
+
+  ~ModuleBuildDaemonServer() { shutdownDaemon(SIGTERM); }
+
+  int forkDaemon();
+  int launchDaemon();
+  int listenForClients();
+
+  static void handleClient(int Client);
+
+  void shutdownDaemon(int signal) {
+unlink(SocketPath.c_str());
+shutdown(ListenSocketFD, SHUT_RD);
+close(ListenSocketFD);
+exit(EXIT_SUCCESS);
+  }
+
+private:
+  // Initializes and returns DiagnosticsEngine
+  pid_t Pid = -1;
+  int ListenSocketFD = -1;
+};
+
+// Required to handle SIGTERM by calling Shutdown
+ModuleBuildDaemonServer *DaemonPtr = nullptr;
+void handleSignal(int Signal) {
+  if (DaemonPtr != nullptr) {
+DaemonPtr->shutdownDaemon(Signal);
+  }
+}
+} // namespace
+
+static bool verbose = false;
+static void verbose_print(const llvm::Twine ) {
+  if (verbose) {
+unbuff_outs() << message << '\n';
+  }
+}
+
+// Forks and detaches process, creating module build daemon
+int ModuleBuildDaemonServer::forkDaemon() {
+
+  pid_t pid = fork();
+
+  if (pid < 0) {
+exit(EXIT_FAILURE);
+  }
+  if (pid > 0) {
+exit(EXIT_SUCCESS);
+  }
+
+  Pid = getpid();
+
+  close(STDIN_FILENO);
+  close(STDOUT_FILENO);
+  close(STDERR_FILENO);
+
+  SmallString<128> STDOUT = BasePath;
+  llvm::sys::path::append(STDOUT, STDOUT_FILE_NAME);
+  freopen(STDOUT.c_str(), "a", stdout);
+
+  SmallString<128> STDERR = BasePath;
+  llvm::sys::path::append(STDERR, STDERR_FILE_NAME);
+  freopen(STDERR.c_str(), "a", stderr);
+
+  if (signal(SIGTERM, handleSignal) == SIG_ERR) {
+errs() << "failed to handle SIGTERM" << '\n';
+exit(EXIT_FAILURE);
+  }
+  if (signal(SIGHUP, SIG_IGN) == SIG_ERR) {
+errs() << "failed to ignore SIGHUP" << '\n';
+exit(EXIT_FAILURE);
+  }
+  if (setsid() == -1) {
+errs() << "setsid failed" << '\n';
+exit(EXIT_FAILURE);
+  }
+
+  return EXIT_SUCCESS;
+}
+
+// Creates unix socket for IPC with module build daemon
+int ModuleBuildDaemonServer::launchDaemon() {
+
+  // new socket
+  if ((ListenSocketFD = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) {
+std::perror("Socket create error: ");
+exit(EXIT_FAILURE);
+  }
+
+  struct sockaddr_un addr;
+  memset(, 0, sizeof(struct sockaddr_un));
+  addr.sun_family = AF_UNIX;
+  strncpy(addr.sun_path, SocketPath.c_str(), sizeof(addr.sun_path) - 1);
+
+  // bind to local address
+  if (bind(ListenSocketFD, (struct sockaddr *), sizeof(addr)) == -1) {
+
+// If the socket address is already in use, exit because another module
+// build daemon has successfully launched. When translation units are
+// compiled in parallel, until the socket file is created, all clang
+// invocations will spawn a module build daemon.
+if (errno == EADDRINUSE) {
+  close(ListenSocketFD);
+  exit(EXIT_SUCCESS);
+}
+std::perror("Socket bind error: ");
+exit(EXIT_FAILURE);
+  }
+  verbose_print("mbd created and binded to socket address at: " + SocketPath);
+
+  // set socket to accept incoming connection request
+  unsigned MaxBacklog = llvm::hardware_concurrency().compute_thread_count();
+  if (listen(ListenSocketFD, MaxBacklog) == -1) {
+std::perror("Socket listen error: ");
+exit(EXIT_FAILURE);
+  }
+
+  return 0;
+}
+
+// Function submitted to thread pool with each client connection. Not
+// responsible for closing client connections
+void 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,267 @@
+//===--- cc1modbuildd_main.cpp - Clang CC1 Module Build Daemon 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Support/Threading.h"
+#include "llvm/Support/YAMLParser.h"
+#include "llvm/Support/YAMLTraits.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace llvm;
+using namespace clang;
+using namespace cc1modbuildd;
+
+// Create unbuffered STDOUT stream so that any logging done by module build
+// daemon can be viewed without having to terminate the process
+static raw_fd_ostream _outs() {
+  static raw_fd_ostream S(STDOUT_FILENO, false, true);
+  return S;
+}
+
+namespace {
+
+class ModuleBuildDaemonServer {
+public:
+  SmallString<128> BasePath;
+  SmallString<128> SocketPath;
+  SmallString<128> PidPath;
+
+  ModuleBuildDaemonServer(SmallString<128> Path, ArrayRef Argv)
+  : BasePath(Path), SocketPath(Path) {
+llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+  }
+
+  ~ModuleBuildDaemonServer() { shutdownDaemon(SIGTERM); }
+
+  int forkDaemon();
+  int launchDaemon();
+  int listenForClients();
+
+  static void handleClient(int Client);
+
+  void shutdownDaemon(int signal) {
+unlink(SocketPath.c_str());
+shutdown(ListenSocketFD, SHUT_RD);
+close(ListenSocketFD);
+exit(EXIT_SUCCESS);
+  }
+
+private:
+  // Initializes and returns DiagnosticsEngine
+  pid_t Pid = -1;
+  int ListenSocketFD = -1;
+};
+
+// Required to handle SIGTERM by calling Shutdown
+ModuleBuildDaemonServer *DaemonPtr = nullptr;
+void handleSignal(int Signal) {
+  if (DaemonPtr != nullptr) {
+DaemonPtr->shutdownDaemon(Signal);

Bigcheese wrote:

This may race with other threads, not sure.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,267 @@
+//===--- cc1modbuildd_main.cpp - Clang CC1 Module Build Daemon 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Support/Threading.h"
+#include "llvm/Support/YAMLParser.h"
+#include "llvm/Support/YAMLTraits.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace llvm;
+using namespace clang;
+using namespace cc1modbuildd;
+
+// Create unbuffered STDOUT stream so that any logging done by module build
+// daemon can be viewed without having to terminate the process
+static raw_fd_ostream _outs() {
+  static raw_fd_ostream S(STDOUT_FILENO, false, true);
+  return S;
+}
+
+namespace {
+
+class ModuleBuildDaemonServer {
+public:
+  SmallString<128> BasePath;
+  SmallString<128> SocketPath;
+  SmallString<128> PidPath;
+
+  ModuleBuildDaemonServer(SmallString<128> Path, ArrayRef Argv)
+  : BasePath(Path), SocketPath(Path) {
+llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+  }
+
+  ~ModuleBuildDaemonServer() { shutdownDaemon(SIGTERM); }
+
+  int forkDaemon();
+  int launchDaemon();
+  int listenForClients();
+
+  static void handleClient(int Client);
+
+  void shutdownDaemon(int signal) {
+unlink(SocketPath.c_str());
+shutdown(ListenSocketFD, SHUT_RD);
+close(ListenSocketFD);
+exit(EXIT_SUCCESS);
+  }
+
+private:
+  // Initializes and returns DiagnosticsEngine
+  pid_t Pid = -1;
+  int ListenSocketFD = -1;
+};
+
+// Required to handle SIGTERM by calling Shutdown
+ModuleBuildDaemonServer *DaemonPtr = nullptr;
+void handleSignal(int Signal) {
+  if (DaemonPtr != nullptr) {
+DaemonPtr->shutdownDaemon(Signal);
+  }
+}
+} // namespace
+
+static bool verbose = false;
+static void verbose_print(const llvm::Twine ) {
+  if (verbose) {
+unbuff_outs() << message << '\n';
+  }
+}
+
+// Forks and detaches process, creating module build daemon
+int ModuleBuildDaemonServer::forkDaemon() {
+
+  pid_t pid = fork();
+
+  if (pid < 0) {
+exit(EXIT_FAILURE);
+  }
+  if (pid > 0) {
+exit(EXIT_SUCCESS);
+  }
+
+  Pid = getpid();
+
+  close(STDIN_FILENO);
+  close(STDOUT_FILENO);
+  close(STDERR_FILENO);
+
+  SmallString<128> STDOUT = BasePath;
+  llvm::sys::path::append(STDOUT, STDOUT_FILE_NAME);
+  freopen(STDOUT.c_str(), "a", stdout);
+
+  SmallString<128> STDERR = BasePath;
+  llvm::sys::path::append(STDERR, STDERR_FILE_NAME);
+  freopen(STDERR.c_str(), "a", stderr);
+
+  if (signal(SIGTERM, handleSignal) == SIG_ERR) {
+errs() << "failed to handle SIGTERM" << '\n';
+exit(EXIT_FAILURE);
+  }
+  if (signal(SIGHUP, SIG_IGN) == SIG_ERR) {
+errs() << "failed to ignore SIGHUP" << '\n';
+exit(EXIT_FAILURE);
+  }
+  if (setsid() == -1) {
+errs() << "setsid failed" << '\n';
+exit(EXIT_FAILURE);
+  }
+
+  return EXIT_SUCCESS;
+}
+
+// Creates unix socket for IPC with module build daemon
+int ModuleBuildDaemonServer::launchDaemon() {
+
+  // new socket
+  if ((ListenSocketFD = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) {
+std::perror("Socket create error: ");
+exit(EXIT_FAILURE);
+  }
+
+  struct sockaddr_un addr;
+  memset(, 0, sizeof(struct sockaddr_un));
+  addr.sun_family = AF_UNIX;
+  strncpy(addr.sun_path, SocketPath.c_str(), sizeof(addr.sun_path) - 1);
+
+  // bind to local address
+  if (bind(ListenSocketFD, (struct sockaddr *), sizeof(addr)) == -1) {
+
+// If the socket address is already in use, exit because another module
+// build daemon has successfully launched. When translation units are
+// compiled in parallel, until the socket file is created, all clang
+// invocations will spawn a module build daemon.
+if (errno == EADDRINUSE) {
+  close(ListenSocketFD);
+  exit(EXIT_SUCCESS);
+}
+std::perror("Socket bind error: ");
+exit(EXIT_FAILURE);
+  }
+  verbose_print("mbd created and binded to socket address at: " + SocketPath);
+
+  // set socket to accept incoming connection request
+  unsigned MaxBacklog = llvm::hardware_concurrency().compute_thread_count();
+  if (listen(ListenSocketFD, MaxBacklog) == -1) {
+std::perror("Socket listen error: ");
+exit(EXIT_FAILURE);
+  }
+
+  return 0;
+}
+
+// Function submitted to thread pool with each client connection. Not
+// responsible for closing client connections
+void 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,18 @@
+// Check that clang invocation can spawn and handshake with module build daemon
+
+// REQUIRES: !system-windows
+
+//  RUN: if pgrep -f "cc1modbuildd mbd-handshake"; then pkill -f "cc1modbuildd 
mbd-handshake"; fi
+//  RUN: rm -rf mbd-handshake %t
+//  RUN: split-file %s %t
+
+//--- main.c
+int main() {return 0;}
+
+// RUN: %clang -fmodule-build-daemon=mbd-handshake %t/main.c > output
+// RUN: cat output | FileCheck %s
+
+// CHECK: Completed successfull handshake with module build daemon
+
+// RUN: if pgrep -f "cc1modbuildd mbd-handshake"; then pkill -f "cc1modbuildd 
mbd-handshake"; fi
+// RUN: rm -rf mbd-handshake %t

Bigcheese wrote:

Don't delete the temp dir at the end.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,167 @@
+//===- Client.cpp 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace clang;
+using namespace llvm;
+using namespace cc1modbuildd;
+
+std::string cc1modbuildd::getBasePath() {
+  llvm::BLAKE3 Hash;
+  Hash.update(getClangFullVersion());
+  auto HashResult = Hash.final();
+  uint64_t HashValue =
+  llvm::support::endian::read(
+  HashResult.data());
+  std::string Key = toString(llvm::APInt(64, HashValue), 36, /*Signed*/ false);
+
+  // set paths
+  SmallString<128> BasePath;
+  llvm::sys::path::system_temp_directory(/*erasedOnReboot*/ true, BasePath);
+  llvm::sys::path::append(BasePath, "clang-" + Key);
+  return BasePath.c_str();
+}
+
+llvm::Error cc1modbuildd::attemptHandshake(int SocketFD) {
+
+  HandshakeMsg Request{ActionType::HANDSHAKE, StatusType::REQUEST};
+  std::string Buffer = getBufferFromSocketMsg(Request);
+
+  if (llvm::Error Err = writeToSocket(Buffer, SocketFD))
+return std::move(Err);
+
+  Expected MaybeServerResponse =
+  readSocketMsgFromSocket(SocketFD);
+  if (!MaybeServerResponse)
+return std::move(MaybeServerResponse.takeError());
+
+  HandshakeMsg ServerResponse = std::move(*MaybeServerResponse);
+
+  assert(ServerResponse.MsgAction == ActionType::HANDSHAKE &&
+ "At this point response ActionType should only ever be HANDSHAKE");
+
+  if (ServerResponse.MsgStatus == StatusType::SUCCESS)
+return llvm::Error::success();
+
+  return llvm::make_error(
+  "Received failed handshake response from module build daemon",
+  inconvertibleErrorCode());
+}
+
+llvm::Error cc1modbuildd::spawnModuleBuildDaemon(StringRef BasePath,
+ const char *Argv0) {
+  std::string BasePathStr = BasePath.str();
+  const char *Args[] = {Argv0, "-cc1modbuildd", BasePathStr.c_str(), nullptr};
+  pid_t pid;
+  int EC = posix_spawn(, Args[0],
+   /*file_actions*/ nullptr,
+   /*spawnattr*/ nullptr, const_cast(Args),
+   /*envp*/ nullptr);
+  if (EC)
+return createStringError(std::error_code(EC, std::generic_category()),
+ "failed to spawn module build daemon");
+
+  return llvm::Error::success();
+}
+
+Expected cc1modbuildd::getModuleBuildDaemon(const char *Argv0,
+ StringRef BasePath) {
+
+  SmallString<128> SocketPath = BasePath;
+  llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+
+  if (llvm::sys::fs::exists(SocketPath)) {
+Expected MaybeFD = connectToSocket(SocketPath);
+if (MaybeFD)
+  return std::move(*MaybeFD);
+consumeError(MaybeFD.takeError());
+  }
+
+  if (llvm::Error Err = cc1modbuildd::spawnModuleBuildDaemon(BasePath, Argv0))
+return std::move(Err);
+
+  const unsigned int MICROSEC_IN_SEC = 100;
+  constexpr unsigned int MAX_TIME = 30 * MICROSEC_IN_SEC;
+  const unsigned short INTERVAL = 100;
+
+  unsigned int CumulativeTime = 0;
+  unsigned int WaitTime = 0;
+
+  while (CumulativeTime <= MAX_TIME) {
+// Wait a bit then check to see if the module build daemon has initialized
+usleep(WaitTime);

Bigcheese wrote:

This ends up sleeping for 0 every time.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,18 @@
+// Check that clang invocation can spawn and handshake with module build daemon

Bigcheese wrote:

This test should also check that clang can connect to an existing MBD without 
spawning a new one.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,128 @@
+//===- SocketSupport.cpp 
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Config/llvm-config.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+Expected cc1modbuildd::createSocket() {
+  int FD;
+  if ((FD = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) {
+std::string Msg = "socket create error: " + std::string(strerror(errno));
+return createStringError(inconvertibleErrorCode(), Msg);
+  }
+  return FD;
+}
+
+Expected cc1modbuildd::connectToSocket(StringRef SocketPath) {
+
+  Expected MaybeFD = cc1modbuildd::createSocket();
+  if (!MaybeFD)
+return std::move(MaybeFD.takeError());
+
+  int FD = std::move(*MaybeFD);
+
+  struct sockaddr_un Addr;
+  memset(, 0, sizeof(Addr));
+  Addr.sun_family = AF_UNIX;
+  strncpy(Addr.sun_path, SocketPath.str().c_str(), sizeof(Addr.sun_path) - 1);
+
+  if (connect(FD, (struct sockaddr *), sizeof(Addr)) == -1) {
+close(FD);
+std::string msg = "socket connect error: " + std::string(strerror(errno));
+return createStringError(inconvertibleErrorCode(), msg);
+  }
+  return FD;
+}
+
+Expected cc1modbuildd::connectAndWriteToSocket(std::string Buffer,
+StringRef SocketPath) {
+
+  Expected MaybeConnectedFD = connectToSocket(SocketPath);
+  if (!MaybeConnectedFD)
+return std::move(MaybeConnectedFD.takeError());
+
+  int ConnectedFD = std::move(*MaybeConnectedFD);
+  llvm::Error Err = writeToSocket(Buffer, ConnectedFD);
+  if (Err)
+return std::move(Err);
+
+  return ConnectedFD;
+}
+
+Expected cc1modbuildd::readFromSocket(int FD) {
+
+  const size_t BUFFER_SIZE = 4096;
+  std::vector Buffer(BUFFER_SIZE);
+  size_t TotalBytesRead = 0;
+
+  ssize_t n;
+  while ((n = read(FD, Buffer.data() + TotalBytesRead,
+   Buffer.size() - TotalBytesRead)) > 0) {
+
+TotalBytesRead += n;
+// Read until ...\n encountered (last line of YAML document)
+if (std::string([TotalBytesRead - 4], 4) == "...\n")
+  break;
+if (Buffer.size() - TotalBytesRead < BUFFER_SIZE)
+  Buffer.resize(Buffer.size() + BUFFER_SIZE);

Bigcheese wrote:

This ends up dropping any other messages if they get read at the same time, and 
may miss `...` in the middle of a buffer. You need to look for the first 
occurrence of `\n...` anywhere in the buffer, and safe the rest of it for the 
next read.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,128 @@
+//===- SocketSupport.cpp 
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Config/llvm-config.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+Expected cc1modbuildd::createSocket() {
+  int FD;
+  if ((FD = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) {
+std::string Msg = "socket create error: " + std::string(strerror(errno));
+return createStringError(inconvertibleErrorCode(), Msg);

Bigcheese wrote:

You should be able to just form an `llvm::Error` from `errno` here.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,31 @@
+//===-- SocketSupport.h 
---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#ifndef LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_SOCKETSUPPORT_H
+#define LLVM_CLANG_TOOLING_MODULEBUILDDAEMON_SOCKETSUPPORT_H
+
+#include "clang/Frontend/CompilerInstance.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/YAMLParser.h"
+#include "llvm/Support/YAMLTraits.h"
+
+using namespace clang;
+using namespace llvm;
+
+namespace cc1modbuildd {
+
+Expected createSocket();
+Expected connectToSocket(StringRef SocketPath);
+Expected connectAndWriteToSocket(std::string Buffer, StringRef 
SocketPath);

Bigcheese wrote:

The write functions should take a `StringRef`.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

https://github.com/Bigcheese requested changes to this pull request.


https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,167 @@
+//===- Client.cpp 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/ModuleBuildDaemon/Client.h"
+#include "clang/Basic/Version.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/BLAKE3.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace clang;
+using namespace llvm;
+using namespace cc1modbuildd;
+
+std::string cc1modbuildd::getBasePath() {
+  llvm::BLAKE3 Hash;
+  Hash.update(getClangFullVersion());
+  auto HashResult = Hash.final();
+  uint64_t HashValue =
+  llvm::support::endian::read(
+  HashResult.data());
+  std::string Key = toString(llvm::APInt(64, HashValue), 36, /*Signed*/ false);
+
+  // set paths
+  SmallString<128> BasePath;
+  llvm::sys::path::system_temp_directory(/*erasedOnReboot*/ true, BasePath);
+  llvm::sys::path::append(BasePath, "clang-" + Key);
+  return BasePath.c_str();
+}
+
+llvm::Error cc1modbuildd::attemptHandshake(int SocketFD) {
+
+  HandshakeMsg Request{ActionType::HANDSHAKE, StatusType::REQUEST};
+  std::string Buffer = getBufferFromSocketMsg(Request);
+
+  if (llvm::Error Err = writeToSocket(Buffer, SocketFD))
+return std::move(Err);
+
+  Expected MaybeServerResponse =
+  readSocketMsgFromSocket(SocketFD);
+  if (!MaybeServerResponse)
+return std::move(MaybeServerResponse.takeError());
+
+  HandshakeMsg ServerResponse = std::move(*MaybeServerResponse);
+
+  assert(ServerResponse.MsgAction == ActionType::HANDSHAKE &&
+ "At this point response ActionType should only ever be HANDSHAKE");
+
+  if (ServerResponse.MsgStatus == StatusType::SUCCESS)
+return llvm::Error::success();
+
+  return llvm::make_error(
+  "Received failed handshake response from module build daemon",
+  inconvertibleErrorCode());
+}
+
+llvm::Error cc1modbuildd::spawnModuleBuildDaemon(StringRef BasePath,
+ const char *Argv0) {
+  std::string BasePathStr = BasePath.str();
+  const char *Args[] = {Argv0, "-cc1modbuildd", BasePathStr.c_str(), nullptr};
+  pid_t pid;
+  int EC = posix_spawn(, Args[0],
+   /*file_actions*/ nullptr,
+   /*spawnattr*/ nullptr, const_cast(Args),
+   /*envp*/ nullptr);
+  if (EC)
+return createStringError(std::error_code(EC, std::generic_category()),
+ "failed to spawn module build daemon");
+
+  return llvm::Error::success();
+}
+
+Expected cc1modbuildd::getModuleBuildDaemon(const char *Argv0,
+ StringRef BasePath) {
+
+  SmallString<128> SocketPath = BasePath;
+  llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+
+  if (llvm::sys::fs::exists(SocketPath)) {
+Expected MaybeFD = connectToSocket(SocketPath);
+if (MaybeFD)
+  return std::move(*MaybeFD);
+consumeError(MaybeFD.takeError());
+  }
+
+  if (llvm::Error Err = cc1modbuildd::spawnModuleBuildDaemon(BasePath, Argv0))
+return std::move(Err);
+
+  const unsigned int MICROSEC_IN_SEC = 100;
+  constexpr unsigned int MAX_TIME = 30 * MICROSEC_IN_SEC;
+  const unsigned short INTERVAL = 100;
+
+  unsigned int CumulativeTime = 0;
+  unsigned int WaitTime = 0;
+
+  while (CumulativeTime <= MAX_TIME) {
+// Wait a bit then check to see if the module build daemon has initialized
+usleep(WaitTime);
+
+if (llvm::sys::fs::exists(SocketPath)) {
+  Expected MaybeFD = connectToSocket(SocketPath);
+  if (MaybeFD)
+return std::move(*MaybeFD);
+  consumeError(MaybeFD.takeError());
+}
+
+CumulativeTime += INTERVAL;

Bigcheese wrote:

I think we want an exponential backoff here rather than constant.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,18 @@
+// Check that clang invocation can spawn and handshake with module build daemon
+
+// REQUIRES: !system-windows
+
+//  RUN: if pgrep -f "cc1modbuildd mbd-handshake"; then pkill -f "cc1modbuildd 
mbd-handshake"; fi
+//  RUN: rm -rf mbd-handshake %t

Bigcheese wrote:

`mbd-handshake` should be `%t/mbd-handshake` in these tests.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -2836,6 +2836,18 @@ defm declspec : BoolOption<"f", "declspec",
   NegFlag,
   BothFlags<[], [ClangOption, CC1Option],
   " __declspec as a keyword">>, Group;
+
+def fmodule_build_daemon : Flag<["-"], "fmodule-build-daemon">, Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality">,
+  MarshallingInfoFlag>;
+def fmodule_build_daemon_EQ : Joined<["-"], "fmodule-build-daemon=">, 
Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality and defines location of 
output files">,

Bigcheese wrote:

I think this is less "output files" and more just the lock and log files.

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

@@ -0,0 +1,44 @@
+//===- Protocol.h 
-===//

Bigcheese wrote:

Client.h

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Michael Spencer via cfe-commits]

https://github.com/Bigcheese edited 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue updated 
https://github.com/llvm/llvm-project/pull/67562

>From 9c8e6c71084e1f0e7a96987b7aa2d251b02bea48 Mon Sep 17 00:00:00 2001
From: cpsughrue 
Date: Sun, 9 Jul 2023 23:19:58 -0400
Subject: [PATCH] [clang][MBD] set up module build daemon infrastructure

The module build daemon (mbd) will serve as a cc1 tool that helps convert
implicit module command lines to explicit module command lines. A clang
invocation will check to see if a mbd exists, if not the invocation will spawn
one. After the mbd is up and running and a handshake has successfully been
carried out between the mbd and clang invocation the clang invocation will
share it's command line with the mbd. The mbd will then scan the translation
unit and build all modular dependencies before returning a modified cc1 command
 line such that all arguments related to modules are converted from the
implicit option to their explicit option.

This commit sets up the basic mbd infrastructure. Including the ability to
spawn a mbd and carry out a handshake between the clang invocation and mbd.

RFC: 
https://discourse.llvm.org/t/rfc-modules-build-daemon-build-system-agnostic-support-for-explicitly-built-modules/71524
---
 clang/include/clang/Driver/Options.td |  12 +
 .../include/clang/Frontend/FrontendOptions.h  |   7 +
 .../clang/Tooling/ModuleBuildDaemon/Client.h  |  44 +++
 .../ModuleBuildDaemon/SocketMsgSupport.h  | 134 +
 .../Tooling/ModuleBuildDaemon/SocketSupport.h |  31 ++
 .../clang/Tooling/ModuleBuildDaemon/Utils.h   |  28 ++
 clang/lib/Driver/ToolChains/Clang.cpp |  14 +-
 clang/lib/Tooling/CMakeLists.txt  |   1 +
 .../Tooling/ModuleBuildDaemon/CMakeLists.txt  |   9 +
 .../lib/Tooling/ModuleBuildDaemon/Client.cpp  | 167 +++
 .../ModuleBuildDaemon/SocketSupport.cpp   | 128 +
 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp |  32 +++
 clang/test/Driver/unknown-arg.c   |   2 +-
 clang/test/ModuleBuildDaemon/handshake.c  |  18 ++
 clang/test/ModuleBuildDaemon/launch.c |  14 +
 clang/tools/driver/CMakeLists.txt |   3 +
 clang/tools/driver/cc1_main.cpp   |  28 +-
 clang/tools/driver/cc1modbuildd_main.cpp  | 267 ++
 clang/tools/driver/driver.cpp |  17 +-
 19 files changed, 947 insertions(+), 9 deletions(-)
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Client.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketSupport.h
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Utils.h
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/CMakeLists.txt
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Client.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/SocketSupport.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp
 create mode 100644 clang/test/ModuleBuildDaemon/handshake.c
 create mode 100644 clang/test/ModuleBuildDaemon/launch.c
 create mode 100644 clang/tools/driver/cc1modbuildd_main.cpp

diff --git a/clang/include/clang/Driver/Options.td 
b/clang/include/clang/Driver/Options.td
index f573f5a06ceb501..3887dad2ef77f58 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -2836,6 +2836,18 @@ defm declspec : BoolOption<"f", "declspec",
   NegFlag,
   BothFlags<[], [ClangOption, CC1Option],
   " __declspec as a keyword">>, Group;
+
+def fmodule_build_daemon : Flag<["-"], "fmodule-build-daemon">, Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality">,
+  MarshallingInfoFlag>;
+def fmodule_build_daemon_EQ : Joined<["-"], "fmodule-build-daemon=">, 
Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality and defines location of 
output files">,
+  MarshallingInfoString>;
+
 def fmodules_cache_path : Joined<["-"], "fmodules-cache-path=">, 
Group,
   Flags<[NoXarchOption]>, Visibility<[ClangOption, CC1Option]>,
   MetaVarName<"">,
diff --git a/clang/include/clang/Frontend/FrontendOptions.h 
b/clang/include/clang/Frontend/FrontendOptions.h
index 117e35de6f76c4c..8ce97a57d413c0b 100644
--- a/clang/include/clang/Frontend/FrontendOptions.h
+++ b/clang/include/clang/Frontend/FrontendOptions.h
@@ -350,6 +350,9 @@ class FrontendOptions {
   /// Whether to share the FileManager when building modules.
   unsigned ModulesShareFileManager : 1;
 
+  /// Connect to module build daemon
+  unsigned ModuleBuildDaemon : 1;
+
   CodeCompleteOptions CodeCompleteOpts;
 
   /// Specifies the output format of the AST.
@@ -435,6 +438,10 @@ class FrontendOptions {
   /// The output file, if any.
   std::string OutputFile;
 
+  /// If given, the path to the module build daemon's output files and socket
+  /// address
+  std::string 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue updated 
https://github.com/llvm/llvm-project/pull/67562

>From ecea06aeedac75896e4d243420bc590e07891713 Mon Sep 17 00:00:00 2001
From: cpsughrue 
Date: Sun, 9 Jul 2023 23:19:58 -0400
Subject: [PATCH] [clang][MBD] set up module build daemon infrastructure

The module build daemon (mbd) will serve as a cc1 tool that helps convert
implicit module command lines to explicit module command lines. A clang
invocation will check to see if a mbd exists, if not the invocation will spawn
one. After the mbd is up and running and a handshake has successfully been
carried out between the mbd and clang invocation the clang invocation will
share it's command line with the mbd. The mbd will then scan the translation
unit and build all modular dependencies before returning a modified cc1 command
 line such that all arguments related to modules are converted from the
implicit option to their explicit option.

This commit sets up the basic mbd infrastructure. Including the ability to
spawn a mbd and carry out a handshake between the clang invocation and mbd.

RFC: 
https://discourse.llvm.org/t/rfc-modules-build-daemon-build-system-agnostic-support-for-explicitly-built-modules/71524
---
 clang/include/clang/Driver/Options.td |  12 +
 .../include/clang/Frontend/FrontendOptions.h  |   7 +
 .../clang/Tooling/ModuleBuildDaemon/Client.h  |  44 +++
 .../ModuleBuildDaemon/SocketMsgSupport.h  | 134 +
 .../Tooling/ModuleBuildDaemon/SocketSupport.h |  31 ++
 .../clang/Tooling/ModuleBuildDaemon/Utils.h   |  28 ++
 clang/lib/Driver/ToolChains/Clang.cpp |  14 +-
 clang/lib/Tooling/CMakeLists.txt  |   1 +
 .../Tooling/ModuleBuildDaemon/CMakeLists.txt  |   9 +
 .../lib/Tooling/ModuleBuildDaemon/Client.cpp  | 167 +++
 .../ModuleBuildDaemon/SocketSupport.cpp   | 128 +
 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp |  32 +++
 clang/test/Driver/unknown-arg.c   |   2 +-
 clang/test/ModuleBuildDaemon/handshake.c  |  18 ++
 clang/test/ModuleBuildDaemon/launch.c |  14 +
 clang/tools/driver/CMakeLists.txt |   3 +
 clang/tools/driver/cc1_main.cpp   |  28 +-
 clang/tools/driver/cc1modbuildd_main.cpp  | 267 ++
 clang/tools/driver/driver.cpp |  17 +-
 19 files changed, 947 insertions(+), 9 deletions(-)
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Client.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketSupport.h
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Utils.h
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/CMakeLists.txt
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Client.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/SocketSupport.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp
 create mode 100644 clang/test/ModuleBuildDaemon/handshake.c
 create mode 100644 clang/test/ModuleBuildDaemon/launch.c
 create mode 100644 clang/tools/driver/cc1modbuildd_main.cpp

diff --git a/clang/include/clang/Driver/Options.td 
b/clang/include/clang/Driver/Options.td
index f573f5a06ceb501..3887dad2ef77f58 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -2836,6 +2836,18 @@ defm declspec : BoolOption<"f", "declspec",
   NegFlag,
   BothFlags<[], [ClangOption, CC1Option],
   " __declspec as a keyword">>, Group;
+
+def fmodule_build_daemon : Flag<["-"], "fmodule-build-daemon">, Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality">,
+  MarshallingInfoFlag>;
+def fmodule_build_daemon_EQ : Joined<["-"], "fmodule-build-daemon=">, 
Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality and defines location of 
output files">,
+  MarshallingInfoString>;
+
 def fmodules_cache_path : Joined<["-"], "fmodules-cache-path=">, 
Group,
   Flags<[NoXarchOption]>, Visibility<[ClangOption, CC1Option]>,
   MetaVarName<"">,
diff --git a/clang/include/clang/Frontend/FrontendOptions.h 
b/clang/include/clang/Frontend/FrontendOptions.h
index 117e35de6f76c4c..8ce97a57d413c0b 100644
--- a/clang/include/clang/Frontend/FrontendOptions.h
+++ b/clang/include/clang/Frontend/FrontendOptions.h
@@ -350,6 +350,9 @@ class FrontendOptions {
   /// Whether to share the FileManager when building modules.
   unsigned ModulesShareFileManager : 1;
 
+  /// Connect to module build daemon
+  unsigned ModuleBuildDaemon : 1;
+
   CodeCompleteOptions CodeCompleteOpts;
 
   /// Specifies the output format of the AST.
@@ -435,6 +438,10 @@ class FrontendOptions {
   /// The output file, if any.
   std::string OutputFile;
 
+  /// If given, the path to the module build daemon's output files and socket
+  /// address
+  std::string 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue deleted 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue deleted 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue updated 
https://github.com/llvm/llvm-project/pull/67562

>From 4153aeb4e06c6bec6e27dd2c814708a3a8b8c842 Mon Sep 17 00:00:00 2001
From: cpsughrue 
Date: Sun, 9 Jul 2023 23:19:58 -0400
Subject: [PATCH] [clang][MBD] set up module build daemon infrastructure

The module build daemon (mbd) will serve as a cc1 tool that helps convert
implicit module command lines to explicit module command lines. A clang
invocation will check to see if a mbd exists, if not the invocation will spawn
one. After the mbd is up and running and a handshake has successfully been
carried out between the mbd and clang invocation the clang invocation will
share it's command line with the mbd. The mbd will then scan the translation
unit and build all modular dependencies before returning a modified cc1 command
 line such that all arguments related to modules are converted from the
implicit option to their explicit option.

This commit sets up the basic mbd infrastructure. Including the ability to
spawn a mbd and carry out a handshake between the clang invocation and mbd.

RFC: 
https://discourse.llvm.org/t/rfc-modules-build-daemon-build-system-agnostic-support-for-explicitly-built-modules/71524
---
 clang/include/clang/Driver/Options.td |  12 +
 .../include/clang/Frontend/FrontendOptions.h  |   7 +
 .../clang/Tooling/ModuleBuildDaemon/Client.h  |  44 +++
 .../ModuleBuildDaemon/SocketMsgSupport.h  | 134 +
 .../Tooling/ModuleBuildDaemon/SocketSupport.h |  31 ++
 .../clang/Tooling/ModuleBuildDaemon/Utils.h   |  28 ++
 clang/lib/Driver/ToolChains/Clang.cpp |  14 +-
 clang/lib/Tooling/CMakeLists.txt  |   1 +
 .../Tooling/ModuleBuildDaemon/CMakeLists.txt  |   9 +
 .../lib/Tooling/ModuleBuildDaemon/Client.cpp  | 167 +++
 .../ModuleBuildDaemon/SocketSupport.cpp   | 128 +
 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp |  32 +++
 clang/test/Driver/unknown-arg.c   |   2 +-
 clang/test/ModuleBuildDaemon/handshake.c  |  18 ++
 clang/test/ModuleBuildDaemon/launch.c |  14 +
 clang/tools/driver/CMakeLists.txt |   3 +
 clang/tools/driver/cc1_main.cpp   |  28 +-
 clang/tools/driver/cc1modbuildd_main.cpp  | 268 ++
 clang/tools/driver/driver.cpp |  17 +-
 19 files changed, 948 insertions(+), 9 deletions(-)
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Client.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h
 create mode 100644 
clang/include/clang/Tooling/ModuleBuildDaemon/SocketSupport.h
 create mode 100644 clang/include/clang/Tooling/ModuleBuildDaemon/Utils.h
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/CMakeLists.txt
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Client.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/SocketSupport.cpp
 create mode 100644 clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp
 create mode 100644 clang/test/ModuleBuildDaemon/handshake.c
 create mode 100644 clang/test/ModuleBuildDaemon/launch.c
 create mode 100644 clang/tools/driver/cc1modbuildd_main.cpp

diff --git a/clang/include/clang/Driver/Options.td 
b/clang/include/clang/Driver/Options.td
index f573f5a06ceb501..3887dad2ef77f58 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -2836,6 +2836,18 @@ defm declspec : BoolOption<"f", "declspec",
   NegFlag,
   BothFlags<[], [ClangOption, CC1Option],
   " __declspec as a keyword">>, Group;
+
+def fmodule_build_daemon : Flag<["-"], "fmodule-build-daemon">, Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality">,
+  MarshallingInfoFlag>;
+def fmodule_build_daemon_EQ : Joined<["-"], "fmodule-build-daemon=">, 
Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality and defines location of 
output files">,
+  MarshallingInfoString>;
+
 def fmodules_cache_path : Joined<["-"], "fmodules-cache-path=">, 
Group,
   Flags<[NoXarchOption]>, Visibility<[ClangOption, CC1Option]>,
   MetaVarName<"">,
diff --git a/clang/include/clang/Frontend/FrontendOptions.h 
b/clang/include/clang/Frontend/FrontendOptions.h
index 117e35de6f76c4c..8ce97a57d413c0b 100644
--- a/clang/include/clang/Frontend/FrontendOptions.h
+++ b/clang/include/clang/Frontend/FrontendOptions.h
@@ -350,6 +350,9 @@ class FrontendOptions {
   /// Whether to share the FileManager when building modules.
   unsigned ModulesShareFileManager : 1;
 
+  /// Connect to module build daemon
+  unsigned ModuleBuildDaemon : 1;
+
   CodeCompleteOptions CodeCompleteOpts;
 
   /// Specifies the output format of the AST.
@@ -435,6 +438,10 @@ class FrontendOptions {
   /// The output file, if any.
   std::string OutputFile;
 
+  /// If given, the path to the module build daemon's output files and socket
+  /// address
+  std::string 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue edited 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

https://github.com/cpsughrue edited 
https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

@@ -0,0 +1,273 @@
+//===--- cc1modbuildd_main.cpp - Clang CC1 Module Build Daemon 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/DependencyScanning/DependencyScanningService.h"
+#include "clang/Tooling/DependencyScanning/DependencyScanningTool.h"
+#include "clang/Tooling/DependencyScanning/ModuleDepCollector.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/SocketSupport.h"
+#include "clang/Tooling/ModuleBuildDaemon/Utils.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Support/Threading.h"
+#include "llvm/Support/YAMLParser.h"
+#include "llvm/Support/YAMLTraits.h"
+
+// TODO: Make portable
+#if LLVM_ON_UNIX
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+using namespace llvm;
+using namespace clang;
+using namespace tooling::dependencies;
+using namespace cc1modbuildd;
+
+// Create unbuffered STDOUT stream so that any logging done by module build
+// daemon can be viewed without having to terminate the process
+static raw_fd_ostream _outs() {
+  static raw_fd_ostream S(STDOUT_FILENO, false, true);
+  return S;
+}
+
+namespace {
+
+class ModuleBuildDaemonServer {
+public:
+  SmallString<128> BasePath;
+  SmallString<128> SocketPath;
+  SmallString<128> PidPath;
+
+  ModuleBuildDaemonServer(SmallString<128> Path, ArrayRef Argv)
+  : BasePath(Path), SocketPath(Path) {
+llvm::sys::path::append(SocketPath, SOCKET_FILE_NAME);
+  }
+
+  ~ModuleBuildDaemonServer() { shutdownDaemon(SIGTERM); }
+
+  int forkDaemon();
+  int launchDaemon();
+  int listenForClients();
+
+  static void handleClient(int Client);
+
+  void shutdownDaemon(int signal) {
+unlink(SocketPath.c_str());
+shutdown(ListenSocketFD, SHUT_RD);
+close(ListenSocketFD);
+exit(EXIT_SUCCESS);
+  }
+
+private:
+  // Initializes and returns DiagnosticsEngine
+  pid_t Pid = -1;
+  int ListenSocketFD = -1;
+};
+
+// Required to handle SIGTERM by calling Shutdown
+ModuleBuildDaemonServer *DaemonPtr = nullptr;
+void handleSignal(int Signal) {
+  if (DaemonPtr != nullptr) {
+DaemonPtr->shutdownDaemon(Signal);
+  }
+}
+} // namespace
+
+static bool verbose = false;
+static void verbose_print(const llvm::Twine ) {
+  if (verbose) {
+unbuff_outs() << message << '\n';
+  }
+}
+
+// Forks and detaches process, creating module build daemon
+int ModuleBuildDaemonServer::forkDaemon() {
+
+  pid_t pid = fork();
+
+  if (pid < 0) {
+exit(EXIT_FAILURE);
+  }
+  if (pid > 0) {
+exit(EXIT_SUCCESS);
+  }
+
+  Pid = getpid();
+
+  close(STDIN_FILENO);
+  close(STDOUT_FILENO);
+  close(STDERR_FILENO);
+
+  SmallString<128> STDOUT = BasePath;
+  llvm::sys::path::append(STDOUT, STDOUT_FILE_NAME);
+  freopen(STDOUT.c_str(), "a", stdout);
+
+  SmallString<128> STDERR = BasePath;
+  llvm::sys::path::append(STDERR, STDERR_FILE_NAME);
+  freopen(STDERR.c_str(), "a", stderr);
+
+  if (signal(SIGTERM, handleSignal) == SIG_ERR) {
+errs() << "failed to handle SIGTERM" << '\n';
+exit(EXIT_FAILURE);
+  }
+  if (signal(SIGHUP, SIG_IGN) == SIG_ERR) {
+errs() << "failed to ignore SIGHUP" << '\n';
+exit(EXIT_FAILURE);
+  }
+  if (setsid() == -1) {
+errs() << "setsid failed" << '\n';
+exit(EXIT_FAILURE);
+  }
+
+  return EXIT_SUCCESS;
+}
+
+// Creates unix socket for IPC with module build daemon
+int ModuleBuildDaemonServer::launchDaemon() {
+
+  // new socket
+  if ((ListenSocketFD = socket(AF_UNIX, SOCK_STREAM, 0)) == -1) {
+std::perror("Socket create error: ");
+exit(EXIT_FAILURE);
+  }
+
+  struct sockaddr_un addr;
+  memset(, 0, sizeof(struct sockaddr_un));
+  addr.sun_family = AF_UNIX;
+  strncpy(addr.sun_path, SocketPath.c_str(), sizeof(addr.sun_path) - 1);
+
+  // bind to local address
+  if (bind(ListenSocketFD, (struct sockaddr *), sizeof(addr)) == -1) {
+
+// If the socket address is already in use, exit because another module
+// build daemon has successfully launched. When translation units are
+// compiled in parallel, until the socket file is created, all clang
+// invocations will spawn a module build daemon.
+if (errno == EADDRINUSE) {
+  close(ListenSocketFD);
+  exit(EXIT_SUCCESS);
+}
+std::perror("Socket bind error: ");
+exit(EXIT_FAILURE);
+  }
+  verbose_print("mbd created and binded to socket address at: " + SocketPath);
+
+  // set socket to accept incoming connection request
+  unsigned MaxBacklog = llvm::hardware_concurrency().compute_thread_count();
+  if 

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[Connor Sughrue via cfe-commits]

@@ -0,0 +1,273 @@
+//===--- cc1modbuildd_main.cpp - Clang CC1 Module Build Daemon 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Tooling/DependencyScanning/DependencyScanningService.h"
+#include "clang/Tooling/DependencyScanning/DependencyScanningTool.h"
+#include "clang/Tooling/DependencyScanning/ModuleDepCollector.h"

cpsughrue wrote:

Dependency scanning headers are not needed for the initial module build daemon 
commit

https://github.com/llvm/llvm-project/pull/67562
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[clang] [clang][MBD] set up module build daemon infrastructure (PR #67562)

[via cfe-commits]

llvmbot wrote:




@llvm/pr-subscribers-clang-modules


Changes

The module build daemon (mbd) will be a cc1 tool that helps convert implicit 
module command lines to explicit module command lines. The basic workflow is as 
follows: A clang invocation will check whether a mbd exists. If not, the 
invocation will spawn one. After the mbd is up and running and a handshake has 
successfully been carried out between the mbd and clang invocation, the clang 
invocation will share its command line with the mbd. The mbd will then scan the 
translation unit and build all modular dependencies before returning a modified 
cc1 command line such that all arguments related to modules are converted from 
the implicit option to their explicit option.

This commit sets up the basic mbd infrastructure. Including the ability to 
spawn a mbd and carry out a handshake between the clang invocation and mbd.

RFC: 
https://discourse.llvm.org/t/rfc-modules-build-daemon-build-system-agnostic-support-for-explicitly-built-modules/71524

---

Patch is 40.58 KiB, truncated to 20.00 KiB below, full version: 
https://github.com/llvm/llvm-project/pull/67562.diff


19 Files Affected:

- (modified) clang/include/clang/Driver/Options.td (+12) 
- (modified) clang/include/clang/Frontend/FrontendOptions.h (+7) 
- (added) clang/include/clang/Tooling/ModuleBuildDaemon/Client.h (+44) 
- (added) clang/include/clang/Tooling/ModuleBuildDaemon/SocketMsgSupport.h 
(+134) 
- (added) clang/include/clang/Tooling/ModuleBuildDaemon/SocketSupport.h (+31) 
- (added) clang/include/clang/Tooling/ModuleBuildDaemon/Utils.h (+28) 
- (modified) clang/lib/Driver/ToolChains/Clang.cpp (+13-1) 
- (modified) clang/lib/Tooling/CMakeLists.txt (+1) 
- (added) clang/lib/Tooling/ModuleBuildDaemon/CMakeLists.txt (+9) 
- (added) clang/lib/Tooling/ModuleBuildDaemon/Client.cpp (+167) 
- (added) clang/lib/Tooling/ModuleBuildDaemon/SocketSupport.cpp (+128) 
- (added) clang/lib/Tooling/ModuleBuildDaemon/Utils.cpp (+32) 
- (modified) clang/test/Driver/unknown-arg.c (+1-1) 
- (added) clang/test/ModuleBuildDaemon/handshake.c (+18) 
- (added) clang/test/ModuleBuildDaemon/launch.c (+14) 
- (modified) clang/tools/driver/CMakeLists.txt (+3) 
- (modified) clang/tools/driver/cc1_main.cpp (+23-5) 
- (added) clang/tools/driver/cc1modbuildd_main.cpp (+273) 
- (modified) clang/tools/driver/driver.cpp (+15-2) 


``diff
diff --git a/clang/include/clang/Driver/Options.td 
b/clang/include/clang/Driver/Options.td
index f573f5a06ceb501..06575f4b73bd47a 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -2836,6 +2836,18 @@ defm declspec : BoolOption<"f", "declspec",
   NegFlag,
   BothFlags<[], [ClangOption, CC1Option],
   " __declspec as a keyword">>, Group;
+
+def fmodule_build_daemon : Flag<["-"], "fmodule-build-daemon">, Group,
+  Flags<[NoXarchOption]>, 
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality">,
+  MarshallingInfoFlag>;
+def fmodule_build_daemon_EQ : Joined<["-"], "fmodule-build-daemon=">, 
Group,
+  Flags<[NoXarchOption]>,
+  Visibility<[ClangOption, CC1Option]>,
+  HelpText<"Enables module build daemon functionality and defines location of 
output files">,
+  MarshallingInfoString>;
+
 def fmodules_cache_path : Joined<["-"], "fmodules-cache-path=">, 
Group,
   Flags<[NoXarchOption]>, Visibility<[ClangOption, CC1Option]>,
   MetaVarName<"">,
diff --git a/clang/include/clang/Frontend/FrontendOptions.h 
b/clang/include/clang/Frontend/FrontendOptions.h
index 117e35de6f76c4c..8ce97a57d413c0b 100644
--- a/clang/include/clang/Frontend/FrontendOptions.h
+++ b/clang/include/clang/Frontend/FrontendOptions.h
@@ -350,6 +350,9 @@ class FrontendOptions {
   /// Whether to share the FileManager when building modules.
   unsigned ModulesShareFileManager : 1;
 
+  /// Connect to module build daemon
+  unsigned ModuleBuildDaemon : 1;
+
   CodeCompleteOptions CodeCompleteOpts;
 
   /// Specifies the output format of the AST.
@@ -435,6 +438,10 @@ class FrontendOptions {
   /// The output file, if any.
   std::string OutputFile;
 
+  /// If given, the path to the module build daemon's output files and socket
+  /// address
+  std::string ModuleBuildDaemonPath;
+
   /// If given, the new suffix for fix-it rewritten files.
   std::string FixItSuffix;
 
diff --git a/clang/include/clang/Tooling/ModuleBuildDaemon/Client.h 
b/clang/include/clang/Tooling/ModuleBuildDaemon/Client.h
new file mode 100644
index 000..d7506fa3011cffb
--- /dev/null
+++ b/clang/include/clang/Tooling/ModuleBuildDaemon/Client.h
@@ -0,0 +1,44 @@
+//===- Protocol.h 
-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+