[PATCH] D155078: [ci] Make libc++ and Clang CI scripts independent

[Aaron Ballman via Phabricator via cfe-commits]

aaron.ballman added a comment.

In D155078#4494167 , @ldionne wrote:

> In D155078#4493995 , @aaron.ballman 
> wrote:
>
>> In D155078#4493468 , @ldionne 
>> wrote:
>>
>>> Ok, this is ready to go. More context around this change:
>>>
>>> I created a clang-specific CI pipeline 
>>> (https://buildkite.com/llvm-project/clang-ci) and Clang CI jobs will now 
>>> trigger inside that pipeline instead of inside the libc++ pipeline 
>>> (https://buildkite.com/llvm-project/libcxx-ci). The problem this solves is 
>>> that 90% of the jobs of the libc++ CI pipelines are otherwise Clang jobs, 
>>> which makes it nearly impossible for libc++ to figure out information about 
>>> the jobs we're actually running.
>>>
>>> - For now, this does not have any impact on the fact that Clang CI is still 
>>> utilizing libc++ resources (although this is next on my list of things to 
>>> address).
>>> - This doesn't have any impact on whether we'll eventually want to move to 
>>> Github Actions or some other CI system. This simply fixes the current 
>>> system before the LLVM 17 release. In particular, I would like to avoid 
>>> bikeshedding the location of the `generate-buildkite-pipeline` scripts 
>>> under `.ci` -- it doesn't matter, I just wanted a neutral 
>>> (non-libcxx-specific) location and that can be changed later once CI has 
>>> stabilized.
>>
>> Please excuse my ignorance, but does this change precommit CI test coverage 
>> when reviews are created/updated? e.g., if a review goes up with changes 
>> only to Clang, will the review still get a libc++ test run to ensure the 
>> Clang changes don't break libc++?
>
> It actually fixes it. Previously, if both libc++ and Clang were modified in a 
> review, only the libc++ CI was triggered. After this patch:
>
> - If Clang is modified, the Clang pre-commit CI is triggered (which includes 
> running the libc++ test suite w/ just-built clang in C++03, C++20 and Modules 
> IIRC)
> - If Libc++ is modified, the libc++ pre-commit CI is triggered (which 
> includes the 60ish jobs we have)
> - If both are modified, both are triggered.

Thank you, this sounds great to me then! LGTM. :-)


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D155078/new/

https://reviews.llvm.org/D155078

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D155078: [ci] Make libc++ and Clang CI scripts independent

[Mark de Wever via Phabricator via cfe-commits]

Mordante added a comment.

Thanks for working on this! I really like the direction. I mainly skimmed the 
code since the patch as already landed.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D155078/new/

https://reviews.llvm.org/D155078

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D155078: [ci] Make libc++ and Clang CI scripts independent

[Louis Dionne via Phabricator via cfe-commits]

This revision was landed with ongoing or failed builds.
This revision was automatically updated to reflect the committed changes.
Closed by commit rG298f207e0649: [ci] Make libc++ and Clang CI scripts 
independent (authored by ldionne).

Changed prior to commit:
  https://reviews.llvm.org/D155078?vs=539553=539645#toc

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D155078/new/

https://reviews.llvm.org/D155078

Files:
  .ci/generate-buildkite-pipeline-premerge
  .ci/generate-buildkite-pipeline-scheduled
  libcxx/utils/ci/buildkite-pipeline-premerge.sh
  libcxx/utils/ci/buildkite-pipeline-snapshot.sh
  libcxx/utils/ci/generate-buildkite-pipeline

Index: libcxx/utils/ci/generate-buildkite-pipeline
===
--- libcxx/utils/ci/generate-buildkite-pipeline
+++ libcxx/utils/ci/generate-buildkite-pipeline
@@ -9,18 +9,9 @@
 
 #
 # This script generates the appropriate libc++ CI pipeline based on which project(s) were changed.
+# TODO: Remove this once the libcxx CI pipeline uploads the buildkite-pipeline.yml file directly.
 #
 
 if git diff --name-only HEAD~1 | grep -q -E "^libcxx/|^libcxxabi/|^libunwind/|^runtimes/|^cmake/"; then
-  LIBCXX_CHANGED=true
-fi
-
-if git diff --name-only HEAD~1 | grep -q -E "^clang/"; then
-  CLANG_CHANGED=true
-fi
-
-if [[ "${CLANG_CHANGED}" == "true" && "${LIBCXX_CHANGED}" != "true" ]]; then
-  cat clang/utils/ci/buildkite-pipeline.yml
-else
   cat libcxx/utils/ci/buildkite-pipeline.yml
 fi
Index: libcxx/utils/ci/buildkite-pipeline-snapshot.sh
===
--- libcxx/utils/ci/buildkite-pipeline-snapshot.sh
+++ libcxx/utils/ci/buildkite-pipeline-snapshot.sh
@@ -7,18 +7,5 @@
 #
 #===--===##
 
-#
-# This file generates a Buildkite pipeline that triggers the libc++ CI jobs.
-# See https://buildkite.com/docs/agent/v3/cli-pipeline#pipeline-format.
-#
-# Invoked by CI on full builds.
-#
-
-cat 

[PATCH] D155078: [ci] Make libc++ and Clang CI scripts independent

[Louis Dionne via Phabricator via cfe-commits]

ldionne added a comment.

I will land this because I can tell this is correct after following the CI 
pipeline's execution, and 
https://github.com/google/llvm-premerge-checks/pull/452 has already landed so 
the CI could start becoming red anytime now. If there are concerns about Clang 
having its own CI pipeline (which I think is only a huge improvement), please 
raise them and we can address that concern post-commit.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D155078/new/

https://reviews.llvm.org/D155078

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D155078: [ci] Make libc++ and Clang CI scripts independent

[Louis Dionne via Phabricator via cfe-commits]

ldionne added a comment.

In D155078#4493995 , @aaron.ballman 
wrote:

> In D155078#4493468 , @ldionne wrote:
>
>> Ok, this is ready to go. More context around this change:
>>
>> I created a clang-specific CI pipeline 
>> (https://buildkite.com/llvm-project/clang-ci) and Clang CI jobs will now 
>> trigger inside that pipeline instead of inside the libc++ pipeline 
>> (https://buildkite.com/llvm-project/libcxx-ci). The problem this solves is 
>> that 90% of the jobs of the libc++ CI pipelines are otherwise Clang jobs, 
>> which makes it nearly impossible for libc++ to figure out information about 
>> the jobs we're actually running.
>>
>> - For now, this does not have any impact on the fact that Clang CI is still 
>> utilizing libc++ resources (although this is next on my list of things to 
>> address).
>> - This doesn't have any impact on whether we'll eventually want to move to 
>> Github Actions or some other CI system. This simply fixes the current system 
>> before the LLVM 17 release. In particular, I would like to avoid 
>> bikeshedding the location of the `generate-buildkite-pipeline` scripts under 
>> `.ci` -- it doesn't matter, I just wanted a neutral (non-libcxx-specific) 
>> location and that can be changed later once CI has stabilized.
>
> Please excuse my ignorance, but does this change precommit CI test coverage 
> when reviews are created/updated? e.g., if a review goes up with changes only 
> to Clang, will the review still get a libc++ test run to ensure the Clang 
> changes don't break libc++?

It actually fixes it. Previously, if both libc++ and Clang were modified in a 
review, only the libc++ CI was triggered. After this patch:

- If Clang is modified, the Clang pre-commit CI is triggered (which includes 
running the libc++ test suite w/ just-built clang in C++03, C++20 and Modules 
IIRC)
- If Libc++ is modified, the libc++ pre-commit CI is triggered (which includes 
the 60ish jobs we have)
- If both are modified, both are triggered.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D155078/new/

https://reviews.llvm.org/D155078

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D155078: [ci] Make libc++ and Clang CI scripts independent

[Aaron Ballman via Phabricator via cfe-commits]

aaron.ballman added a comment.

In D155078#4493468 , @ldionne wrote:

> Ok, this is ready to go. More context around this change:
>
> I created a clang-specific CI pipeline 
> (https://buildkite.com/llvm-project/clang-ci) and Clang CI jobs will now 
> trigger inside that pipeline instead of inside the libc++ pipeline 
> (https://buildkite.com/llvm-project/libcxx-ci). The problem this solves is 
> that 90% of the jobs of the libc++ CI pipelines are otherwise Clang jobs, 
> which makes it nearly impossible for libc++ to figure out information about 
> the jobs we're actually running.
>
> - For now, this does not have any impact on the fact that Clang CI is still 
> utilizing libc++ resources (although this is next on my list of things to 
> address).
> - This doesn't have any impact on whether we'll eventually want to move to 
> Github Actions or some other CI system. This simply fixes the current system 
> before the LLVM 17 release. In particular, I would like to avoid bikeshedding 
> the location of the `generate-buildkite-pipeline` scripts under `.ci` -- it 
> doesn't matter, I just wanted a neutral (non-libcxx-specific) location and 
> that can be changed later once CI has stabilized.

Please excuse my ignorance, but does this change precommit CI test coverage 
when reviews are created/updated? e.g., if a review goes up with changes only 
to Clang, will the review still get a libc++ test run to ensure the Clang 
changes don't break libc++?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D155078/new/

https://reviews.llvm.org/D155078

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D155078: [ci] Make libc++ and Clang CI scripts independent

[Louis Dionne via Phabricator via cfe-commits]

ldionne added a comment.

See https://github.com/google/llvm-premerge-checks/pull/452 for the other side 
of this change. https://github.com/google/llvm-premerge-checks/pull/452 must 
land after this patch.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D155078/new/

https://reviews.llvm.org/D155078

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D155078: [ci] Make libc++ and Clang CI scripts independent

[Nikolas Klauser via Phabricator via cfe-commits]

philnik accepted this revision.
philnik added a comment.
This revision is now accepted and ready to land.

I like this a lot! LGTM.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D155078/new/

https://reviews.llvm.org/D155078

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D155078: [ci] Make libc++ and Clang CI scripts independent

[Louis Dionne via Phabricator via cfe-commits]

ldionne updated this revision to Diff 539553.
ldionne added a comment.

Poke CI after fixing Clang CI pipeline clone URL.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D155078/new/

https://reviews.llvm.org/D155078

Files:
  .ci/generate-buildkite-pipeline-premerge
  .ci/generate-buildkite-pipeline-scheduled
  clang/foo
  libcxx/utils/ci/buildkite-pipeline-premerge.sh
  libcxx/utils/ci/buildkite-pipeline-snapshot.sh
  libcxx/utils/ci/generate-buildkite-pipeline

Index: libcxx/utils/ci/generate-buildkite-pipeline
===
--- libcxx/utils/ci/generate-buildkite-pipeline
+++ libcxx/utils/ci/generate-buildkite-pipeline
@@ -9,18 +9,9 @@
 
 #
 # This script generates the appropriate libc++ CI pipeline based on which project(s) were changed.
+# TODO: Remove this once the libcxx CI pipeline uploads the buildkite-pipeline.yml file directly.
 #
 
 if git diff --name-only HEAD~1 | grep -q -E "^libcxx/|^libcxxabi/|^libunwind/|^runtimes/|^cmake/"; then
-  LIBCXX_CHANGED=true
-fi
-
-if git diff --name-only HEAD~1 | grep -q -E "^clang/"; then
-  CLANG_CHANGED=true
-fi
-
-if [[ "${CLANG_CHANGED}" == "true" && "${LIBCXX_CHANGED}" != "true" ]]; then
-  cat clang/utils/ci/buildkite-pipeline.yml
-else
   cat libcxx/utils/ci/buildkite-pipeline.yml
 fi
Index: libcxx/utils/ci/buildkite-pipeline-snapshot.sh
===
--- libcxx/utils/ci/buildkite-pipeline-snapshot.sh
+++ libcxx/utils/ci/buildkite-pipeline-snapshot.sh
@@ -7,18 +7,5 @@
 #
 #===--===##
 
-#
-# This file generates a Buildkite pipeline that triggers the libc++ CI jobs.
-# See https://buildkite.com/docs/agent/v3/cli-pipeline#pipeline-format.
-#
-# Invoked by CI on full builds.
-#
-
-cat 

[PATCH] D155078: [ci] Make libc++ and Clang CI scripts independent

[Louis Dionne via Phabricator via cfe-commits]

ldionne added a comment.

Ok, this is ready to go. More context around this change:

I created a clang-specific CI pipeline 
(https://buildkite.com/llvm-project/clang-ci) and Clang CI jobs will now 
trigger inside that pipeline instead of inside the libc++ pipeline 
(https://buildkite.com/llvm-project/libcxx-ci). The problem this solves is that 
90% of the jobs of the libc++ CI pipelines are otherwise Clang jobs, which 
makes it nearly impossible for libc++ to figure out information about the jobs 
we're actually running.

- For now, this does not have any impact on the fact that Clang CI is still 
utilizing libc++ resources (although this is next on my list of things to 
address).
- This doesn't have any impact on whether we'll eventually want to move to 
Github Actions or some other CI system. This simply fixes the current system 
before the LLVM 17 release. In particular, I would like to avoid bikeshedding 
the location of the `generate-buildkite-pipeline` scripts under `.ci` -- it 
doesn't matter, I just wanted a neutral (non-libcxx-specific) location and that 
can be changed later once CI is out of constant-red.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D155078/new/

https://reviews.llvm.org/D155078

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D155078: [ci] Make libc++ and Clang CI scripts independent

[Louis Dionne via Phabricator via cfe-commits]

ldionne updated this revision to Diff 539534.
ldionne added a comment.

Poke CI after fixing pipeline settings.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D155078/new/

https://reviews.llvm.org/D155078

Files:
  .ci/generate-buildkite-pipeline-premerge
  .ci/generate-buildkite-pipeline-scheduled
  clang/foo
  libcxx/utils/ci/buildkite-pipeline-premerge.sh
  libcxx/utils/ci/buildkite-pipeline-snapshot.sh
  libcxx/utils/ci/generate-buildkite-pipeline

Index: libcxx/utils/ci/generate-buildkite-pipeline
===
--- libcxx/utils/ci/generate-buildkite-pipeline
+++ libcxx/utils/ci/generate-buildkite-pipeline
@@ -9,18 +9,9 @@
 
 #
 # This script generates the appropriate libc++ CI pipeline based on which project(s) were changed.
+# TODO: Remove this once the libcxx CI pipeline uploads the buildkite-pipeline.yml file directly.
 #
 
 if git diff --name-only HEAD~1 | grep -q -E "^libcxx/|^libcxxabi/|^libunwind/|^runtimes/|^cmake/"; then
-  LIBCXX_CHANGED=true
-fi
-
-if git diff --name-only HEAD~1 | grep -q -E "^clang/"; then
-  CLANG_CHANGED=true
-fi
-
-if [[ "${CLANG_CHANGED}" == "true" && "${LIBCXX_CHANGED}" != "true" ]]; then
-  cat clang/utils/ci/buildkite-pipeline.yml
-else
   cat libcxx/utils/ci/buildkite-pipeline.yml
 fi
Index: libcxx/utils/ci/buildkite-pipeline-snapshot.sh
===
--- libcxx/utils/ci/buildkite-pipeline-snapshot.sh
+++ libcxx/utils/ci/buildkite-pipeline-snapshot.sh
@@ -7,18 +7,5 @@
 #
 #===--===##
 
-#
-# This file generates a Buildkite pipeline that triggers the libc++ CI jobs.
-# See https://buildkite.com/docs/agent/v3/cli-pipeline#pipeline-format.
-#
-# Invoked by CI on full builds.
-#
-
-cat 

[PATCH] D155078: [ci] Make libc++ and Clang CI scripts independent

[Louis Dionne via Phabricator via cfe-commits]

ldionne updated this revision to Diff 539526.
ldionne added a comment.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

Touch a file inside Clang to test the triggering.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D155078/new/

https://reviews.llvm.org/D155078

Files:
  .ci/generate-buildkite-pipeline-premerge
  .ci/generate-buildkite-pipeline-scheduled
  clang/foo
  libcxx/utils/ci/buildkite-pipeline-premerge.sh
  libcxx/utils/ci/buildkite-pipeline-snapshot.sh
  libcxx/utils/ci/generate-buildkite-pipeline

Index: libcxx/utils/ci/generate-buildkite-pipeline
===
--- libcxx/utils/ci/generate-buildkite-pipeline
+++ libcxx/utils/ci/generate-buildkite-pipeline
@@ -9,18 +9,9 @@
 
 #
 # This script generates the appropriate libc++ CI pipeline based on which project(s) were changed.
+# TODO: Remove this once the libcxx CI pipeline uploads the buildkite-pipeline.yml file directly.
 #
 
 if git diff --name-only HEAD~1 | grep -q -E "^libcxx/|^libcxxabi/|^libunwind/|^runtimes/|^cmake/"; then
-  LIBCXX_CHANGED=true
-fi
-
-if git diff --name-only HEAD~1 | grep -q -E "^clang/"; then
-  CLANG_CHANGED=true
-fi
-
-if [[ "${CLANG_CHANGED}" == "true" && "${LIBCXX_CHANGED}" != "true" ]]; then
-  cat clang/utils/ci/buildkite-pipeline.yml
-else
   cat libcxx/utils/ci/buildkite-pipeline.yml
 fi
Index: libcxx/utils/ci/buildkite-pipeline-snapshot.sh
===
--- libcxx/utils/ci/buildkite-pipeline-snapshot.sh
+++ libcxx/utils/ci/buildkite-pipeline-snapshot.sh
@@ -7,18 +7,5 @@
 #
 #===--===##
 
-#
-# This file generates a Buildkite pipeline that triggers the libc++ CI jobs.
-# See https://buildkite.com/docs/agent/v3/cli-pipeline#pipeline-format.
-#
-# Invoked by CI on full builds.
-#
-
-cat