llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT--> @llvm/pr-subscribers-clang-ssaf Author: Balázs Benics (steakhal) <details> <summary>Changes</summary> This reverts the SSAF orchestrator/driver stack landed on 2026-10-13/14, as requested by multiple maintainers in #<!-- -->231999. The changes broke upstream and downstream builders (red for >1 day) and violate LLVM layering rules, with no straightforward forward fix in a reasonable timeframe. Reasons for reverting - GNU Make made a mandatory dependency for lit tests (previously optional per the GettingStarted requirements (https://llvm.org/docs/GettingStarted.html#software)) — broke all macOS builders, which don't ship GNU Make. - New layering dependencies from clang/ onto clang-tools-extra and libc++ — clang/test gained a dep on clang-apply-replacements; tests assume libc++ exists, which some targets (e.g. Solaris) don't provide. - Windows path length — a tool name pushed build paths past MAX_PATH (260). Commits reverted (newest first) - 6abd4709cff4 — [gn] port clang-ssaf-driver (#<!-- -->232118) - 1c563eafd44e — [gn] port clang-ssaf-merge-conflict-report (#<!-- -->232115) - 8a96bd506570 — [gn] port clang-ssaf-merge-transformation-report (#<!-- -->232114) - 73109fa8894a — [gn] port clang-ssaf-orchestrator-make (#<!-- -->232113) - a8797d63e97f — Clang ssaf driver 4 (#<!-- -->231999) - ba6d822f43f0 — Merge transformation reports per link unit (#<!-- -->231957) - 3e93f25d161f — Put clang-ssaf-orchestrator-make in LLVM_BINARY_DIR (#<!-- -->231930) - f32ac72b4f1e — Add clang-ssaf-merge-conflict-report (#<!-- -->231731) - 07195e29cdba — Add clang-ssaf-merge-transformation-report (#<!-- -->231664) - 9d4c617fc399 — Interpose ld.lld (#<!-- -->231720) - 9d9ad8a3dbc5 — Clang ssaf orchestrator fixes (#<!-- -->231678) - 4d2bc2c3a7a4 — Interpose ar (#<!-- -->231618) - 2250a9708e9e — Add a transformation pass (#<!-- -->231601) - dbde204202ad — Add clang-ssaf-orchestrator-make (#<!-- -->231583) Complaints referenced - #<!-- -->231583 — clang tests failing (Kewen12): #<!-- -->231583#issuecomment-6153318133 - #<!-- -->231583 — breaks Solaris, libc++ not always present (rorth): #<!-- -->231583#issuecomment-6153866977 - #<!-- -->231583 — GNU Make now required, broke macOS builders, RFC expected (petrhosek): #<!-- -->231583#issuecomment-6182524527 - #<!-- -->231583 — 3 builders still failing, fix or revert (kovdan01): #<!-- -->231583#issuecomment-6186432153 - #<!-- -->231664 — Windows MAX_PATH exceeded by tool name (yueshe): #<!-- -->231664#issuecomment-6175672277 - #<!-- -->231678 — fix incomplete, 14 ssaf tests still failing (Kewen12): #<!-- -->231678#issuecomment-6157991885 - #<!-- -->231720 — still failing after fix (dyung): #<!-- -->231720#issuecomment-6175099823 - #<!-- -->231720 — still failing downstream, revert the stack (Prabhuk): #<!-- -->231720#issuecomment-6179425292 - #<!-- -->231999 — clang/ should not depend on clang-tools-extra (nico): #<!-- -->231999#discussion_r4277237257 - #<!-- -->231999 — unreviewed stack, revert all (nikic): #<!-- -->231999#issuecomment-6187057955 - #<!-- -->231999 — second reverting, layering violations (petrhosek): #<!-- -->231999#issuecomment-6187297359 Build-bot failures referenced - Builder 10 (clang tests): https://lab.llvm.org/buildbot/#/builders/10/builds/37473 - Solaris/amd64 (staging 120): https://lab.llvm.org/staging/#/builders/120/builds/22263 - intel-sycl-gpu (225): https://lab.llvm.org/buildbot/#/builders/225/builds/21732 - llvm-clang-x86_64-sie-ubuntu-fast (144): https://lab.llvm.org/buildbot/#/builders/144/builds/68697 - llvm-clang-ubuntu-x-aarch64-pac-ret (229): https://lab.llvm.org/buildbot/#/builders/229 - llvm-clang-ubuntu-x-aarch64-pauthtest (227): https://lab.llvm.org/buildbot/#/builders/227 - Builder 10 after first fix (#<!-- -->231678): https://lab.llvm.org/buildbot/#/builders/10/builds/37490 - llvm-clang-x86_64-sie-ubuntu-fast after ld.lld fix (#<!-- -->231720): https://lab.llvm.org/buildbot/#/builders/144/builds/68731 Authored-By: claude --- Patch is 483.36 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/232274.diff 150 Files Affected: - (removed) clang/docs/ScalableStaticAnalysis/developer-docs/MakeOrchestratorShims.md (-155) - (removed) clang/docs/ScalableStaticAnalysis/user-docs/Driver.md (-108) - (removed) clang/docs/ScalableStaticAnalysis/user-docs/MakeProjects.md (-323) - (modified) clang/docs/ScalableStaticAnalysis/user-docs/SourceEditGeneration.md (-28) - (removed) clang/test/Analysis/Scalable/ssaf-driver/apply-clang-apply-replacements.test (-38) - (removed) clang/test/Analysis/Scalable/ssaf-driver/apply.test (-86) - (removed) clang/test/Analysis/Scalable/ssaf-driver/archive-conflict.test (-56) - (removed) clang/test/Analysis/Scalable/ssaf-driver/bad-args.test (-21) - (removed) clang/test/Analysis/Scalable/ssaf-driver/build-failure.test (-17) - (removed) clang/test/Analysis/Scalable/ssaf-driver/conflicts.test (-58) - (removed) clang/test/Analysis/Scalable/ssaf-driver/help.test (-28) - (removed) clang/test/Analysis/Scalable/ssaf-driver/lit.local.cfg (-53) - (removed) clang/test/Analysis/Scalable/ssaf-driver/report.test (-48) - (removed) clang/test/Analysis/Scalable/ssaf-driver/transformation.test (-72) - (removed) clang/test/Analysis/Scalable/ssaf-driver/unittests.test (-8) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/Inputs/conflicts-a.sarif (-118) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/Inputs/conflicts-b.sarif (-118) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/Inputs/conflicts-c.sarif (-67) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/Inputs/no-conflicts.sarif (-15) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/Inputs/other-tool.sarif (-15) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/Inputs/two-runs.sarif (-24) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/cli-errors.test (-69) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/empty.test (-14) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/help.test (-19) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/merge.test (-28) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/nested.test (-17) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/order-invariance.test (-8) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/src-edit-merge-reports.test (-26) - (removed) clang/test/Analysis/Scalable/ssaf-merge-conflict-report/version.test (-5) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/Inputs/bad-artifact-index.sarif (-71) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/Inputs/bad-rule-index.sarif (-71) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/Inputs/empty.sarif (-21) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/Inputs/other-transformation.sarif (-21) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/Inputs/tu-a.sarif (-106) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/Inputs/tu-b.sarif (-118) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/Inputs/tu-c.sarif (-71) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/Inputs/tu-clang-order.sarif (-124) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/Inputs/two-runs.sarif (-36) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/clang-artifact-order.test (-23) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/cli-errors.test (-87) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/empty.test (-16) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/help.test (-19) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/merge.test (-51) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/nested.test (-16) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/order-invariance.test (-8) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/plugin-reports.test (-47) - (removed) clang/test/Analysis/Scalable/ssaf-merge-transformation-report/version.test (-5) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/bad-args.test (-13) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/bare-linker.test (-46) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/build-failures.test (-35) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/extract-summaries.test (-67) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/failure-handling.test (-78) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/help.test (-31) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/incremental-tree.test (-58) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/interposition.test (-88) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/library-only.test (-35) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/link-driver-ld-lld.test (-41) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/link-driver.test (-29) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/link-unit-selection.test (-91) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/link-units.test (-54) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/lit.local.cfg (-65) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/path-interposition.test (-42) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/prebuilt-archive.test (-44) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/recursive-make.test (-61) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/same-basename-objects.test (-61) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/shim-outside-a-run.test (-17) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/transformation-reports.test (-88) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/two-pass-compile-and-link.test (-44) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/two-pass.test (-68) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/unittests.test (-8) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/unreachable-compiler.test (-27) - (removed) clang/test/Analysis/Scalable/ssaf-orchestrator-make/whole-program-analysis.test (-59) - (modified) clang/test/CMakeLists.txt (-12) - (modified) clang/tools/CMakeLists.txt (-6) - (removed) clang/tools/clang-ssaf-driver/CMakeLists.txt (-75) - (removed) clang/tools/clang-ssaf-driver/README.md (-35) - (removed) clang/tools/clang-ssaf-driver/bin/clang-ssaf-driver (-15) - (removed) clang/tools/clang-ssaf-driver/lib/clang_ssaf_driver/__init__.py (-10) - (removed) clang/tools/clang-ssaf-driver/lib/clang_ssaf_driver/apply.py (-81) - (removed) clang/tools/clang-ssaf-driver/lib/clang_ssaf_driver/driver.py (-105) - (removed) clang/tools/clang-ssaf-driver/lib/clang_ssaf_driver/logging_diagnostics.py (-31) - (removed) clang/tools/clang-ssaf-driver/lib/clang_ssaf_driver/make_orchestrator.py (-64) - (removed) clang/tools/clang-ssaf-driver/lib/clang_ssaf_driver/options.py (-159) - (removed) clang/tools/clang-ssaf-driver/lib/clang_ssaf_driver/patch.py (-129) - (removed) clang/tools/clang-ssaf-driver/lib/clang_ssaf_driver/report.py (-136) - (removed) clang/tools/clang-ssaf-driver/lib/clang_ssaf_driver/source_edits.py (-186) - (removed) clang/tools/clang-ssaf-driver/lib/clang_ssaf_driver/transformations.py (-18) - (removed) clang/tools/clang-ssaf-driver/lib/clang_ssaf_driver/version.py.in (-5) - (removed) clang/tools/clang-ssaf-driver/tests/__init__.py (-3) - (removed) clang/tools/clang-ssaf-driver/tests/test_apply.py (-118) - (removed) clang/tools/clang-ssaf-driver/tests/test_driver.py (-124) - (removed) clang/tools/clang-ssaf-driver/tests/test_make_orchestrator.py (-137) - (removed) clang/tools/clang-ssaf-driver/tests/test_options.py (-107) - (removed) clang/tools/clang-ssaf-driver/tests/test_patch.py (-237) - (removed) clang/tools/clang-ssaf-driver/tests/test_report.py (-204) - (removed) clang/tools/clang-ssaf-driver/tests/test_source_edits.py (-171) - (removed) clang/tools/clang-ssaf-merge-conflict-report/CMakeLists.txt (-12) - (removed) clang/tools/clang-ssaf-merge-conflict-report/MergeConflictReport.cpp (-249) - (removed) clang/tools/clang-ssaf-merge-transformation-report/CMakeLists.txt (-12) - (removed) clang/tools/clang-ssaf-merge-transformation-report/MergeTransformationReport.cpp (-386) - (removed) clang/tools/clang-ssaf-orchestrator-make/CMakeLists.txt (-95) - (removed) clang/tools/clang-ssaf-orchestrator-make/README.md (-51) - (removed) clang/tools/clang-ssaf-orchestrator-make/bin/clang-ssaf-orchestrator-make (-15) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/__init__.py (-10) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/build_graph.py (-73) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/config.py (-142) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/config_orchestrator.py (-189) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/config_shims.py (-33) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/failure_logs.py (-53) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/interposition.py (-31) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/logging_diagnostics.py (-49) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/options.py (-192) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/orchestrator.py (-264) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/shims/__init__.py (-3) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/shims/ar.py (-102) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/shims/clang.py (-158) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/shims/commands.py (-57) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/shims/driver_jobs.py (-58) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/shims/ld.py (-94) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/shims/run.py (-23) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/shims/shim.py (-279) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/ssaf_artifact_repository.py (-76) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/utils.py (-54) - (removed) clang/tools/clang-ssaf-orchestrator-make/lib/clang_ssaf_orchestrator_make/version.py.in (-5) - (removed) clang/tools/clang-ssaf-orchestrator-make/shims/shim (-45) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/__init__.py (-3) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/configs.py (-101) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_ar_shim.py (-149) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_build_graph.py (-222) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_clang_shim.py (-293) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_commands.py (-124) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_config.py (-142) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_config_orchestrator.py (-418) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_config_shims.py (-63) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_driver_jobs.py (-150) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_failure_logs.py (-93) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_interposition.py (-69) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_ld_shim.py (-239) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_logging_diagnostics.py (-123) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_options.py (-88) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_orchestrator.py (-383) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_run.py (-84) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_shim.py (-414) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_ssaf_artifact_repository.py (-218) - (removed) clang/tools/clang-ssaf-orchestrator-make/tests/test_utils.py (-94) - (modified) llvm/utils/gn/secondary/clang/test/BUILD.gn (-4) - (removed) llvm/utils/gn/secondary/clang/tools/clang-ssaf-driver/BUILD.gn (-41) - (removed) llvm/utils/gn/secondary/clang/tools/clang-ssaf-merge-conflict-report/BUILD.gn (-8) - (removed) llvm/utils/gn/secondary/clang/tools/clang-ssaf-merge-transformation-report/BUILD.gn (-8) - (removed) llvm/utils/gn/secondary/clang/tools/clang-ssaf-orchestrator-make/BUILD.gn (-64) ``````````diff diff --git a/clang/docs/ScalableStaticAnalysis/developer-docs/MakeOrchestratorShims.md b/clang/docs/ScalableStaticAnalysis/developer-docs/MakeOrchestratorShims.md deleted file mode 100644 index e251d98e52f722..00000000000000 --- a/clang/docs/ScalableStaticAnalysis/developer-docs/MakeOrchestratorShims.md +++ /dev/null @@ -1,155 +0,0 @@ -# Make Orchestrator Shims - -`clang-ssaf-orchestrator-make` analyses a project by riding along with its -build. It does that with a set of small programs — *shims* — that stand in -for the tools the build invokes. This describes how they are selected, what -each one does, and how they learn what run they are part of; the user-facing -behaviour is in {doc}`../user-docs/MakeProjects`. - -## Setting up the build's environment - -At the start of a run the orchestrator does two things to the environment the -build will see. - -It fills `<output-dir>/shims/` with links named after the build tools, all -pointing at one shim program, and prepends that directory to `PATH`. A build -can spell the compiler several ways, so several names lead to the same shim: - -```text -cc, clang, c++, clang++ -> clang -ld -> ld -ar -> ar -``` - -It also writes the configuration for the shims into a JSON document and puts -that document's path in `SSAF_CONFIG`. A shim that finds no `SSAF_CONFIG` -reports an error and exits without running anything. - -## Running the binary tool - -A shim has to run the tool the build asked for, by the name the build spelled, -but on the modified `PATH` that name finds the shim itself. -`SSAFWorkflowConfig.restored_path_env_var` therefore holds the build's `PATH` -as it was before the shim link directory went in front of it. -`Shim.bin_tool_path` looks the tool up on that `PATH`, and -`Shim.get_unmodified_env` gives the tool an environment with that `PATH` -restored and `SSAF_CONFIG` removed. - -## The shim program - -`libexec/ssaf-orchestrator-shims/` holds one program, `shim`, and every link -points at it. It puts the package on `sys.path`, looks up the name it was -invoked under in `SHIM_FOR_LINK_NAME`, the table the links were made from, and -imports the module that stands in for that tool. It is a file rather than a -package module because it is symlinked under several names, and `argv[0]` is -what tells it which tool it stands in for. - -The shim bodies live in the package, as `clang_ssaf_orchestrator_make.shims`, -split by what they do: - -```text -shim.py the Shim class, its diagnostics, the transformation merges -commands.py Files, is_object, is_archive, artifact_of, summary_of, - filter_build_artifacts, unique -driver_jobs.py Job and the -### listing reader -run.py run_shim -clang.py ld.py ar.py -``` - -Every shim has the same shape, which `run_shim` implements: run the tool the -build asked for, then do the SSAF work. A failure in the first half is the -build's own. A failure in the second is logged, and the build carries on. - -## The clang shim has two roles - -One program serves both, and a marker on the command line tells them apart. - -**Invoked as the clang driver.** The build spelled `cc`, `clang`, `c++` or -`clang++`, and `PATH` found the link. The shim runs the binary tool on the -build's command line untouched, asks it with `-###` how that command -decomposes into jobs, and forwards each job to the shim that stands in for its -tool. - -**Invoked as cc1 by the shim's driver role.** A forwarded job ends in -`--ssaf-shim-skip-binary-tool-invocation`, which only the driver role appends. -The binary tool has therefore already run, and the shim does only the SSAF -work: it runs the SSAF clang over the job with the `--ssaf-*` options added. -`Shim.__init__` removes the marker as the command enters the shim, so nothing -downstream sees it. - -## Matching a job to a shim - -For each job in the driver's `-###` listing, take the program the job names, -strip its directory and any trailing `-<digits>`, and look the result up among -the links: - -```text -/…/toolchains/…/usr/bin/clang -> clang -> <output-dir>/shims/clang -/…/build/bin/clang-<N> -> clang -> <output-dir>/shims/clang -/usr/bin/ld -> ld -> <output-dir>/shims/ld -/usr/bin/dsymutil -> dsymutil -> no link; nothing stands in -``` - -The suffix is stripped because a driver names its own binary `clang` when -installed and `clang-<N>`, after its major version, in a build tree. A job -with no matching link is noted and skipped: the build's compiler has already -run it, and there is no SSAF step for its tool. - -## Recording what the build assembled - -`ld` and `ar` each read their command line for the artifact it produces and -the artifacts it consumes, and record them in -`<output-dir>/build-graph/<path>.buildstep.json`. The top-most link unit is -found from these records, and they tell an artifact the build assembled from -one it never touched. - -Everything a shim writes is named for a build artifact's path plus a suffix, -such as `ssaf-artifacts/<path>.summary.json`. - -## Failure logs - -A failure log is a diagnostic about a command rather than something the -command produced, so it lives apart, under `failure-logs/`. The run counts -every log at its end, which is how a failure a shim logged and moved past -still fails the run. - -A log is named for the artifact it concerns: the output of the command that -failed, or the input whose summary is missing. A command that names no output, -such as `clang -c a.c b.c`, and an internal error in a shim are logged instead -under the directory the command ran in, named for a hash of the command: - -```text -failure-logs/<path>.<kind>.log named for an artifact -failure-logs/<dir>/<hash>.<kind>.log named for a command -``` - -The kind says what went wrong; the kinds are the values of `SSAFFailureKind`: - -```text -extract summary extraction for a compilation -transform source-edit generation for a compilation -link clang-ssaf-linker over a link's inputs -archive the same for an archive -merge clang-ssaf-src-edit-merge over a link's or an archive's inputs -transformation-report - clang-ssaf-merge-transformation-report over the same inputs -conflict-report - clang-ssaf-merge-conflict-report over the same inputs and the - conflicts of the merge itself -forward handing a job to the shim that stands in for its tool -missing an artifact with nothing analysed behind it -driver-jobs the driver refused to list a command's jobs -internal an unhandled exception inside a shim -``` - -## Carrying the link unit name into the second pass - -One piece of state flows from the first pass to the second: the top-most link -unit's own name, which is not an artifact id and is unknown until the first -pass's linker has written the link unit summary. - -The orchestrator reads it from that summary, as the `lu_namespace` component -whose kind is `LinkUnit`, and puts it in the transformation pass's -configuration; the clang shim passes it as `--ssaf-link-unit-id=` to every -source-edit generation. This is why the second pass's configuration file -cannot be written up front. diff --git a/clang/docs/ScalableStaticAnalysis/user-docs/Driver.md b/clang/docs/ScalableStaticAnalysis/user-docs/Driver.md deleted file mode 100644 index 86c892b3497c57..00000000000000 --- a/clang/docs/ScalableStaticAnalysis/user-docs/Driver.md +++ /dev/null @@ -1,108 +0,0 @@ -# Transforming a Project - -`clang-ssaf-driver` runs a source transformation over a whole project as it -builds, and presents what it found: the source edits as a patch, the -transformation report as diagnostics and as SARIF, and, if asked, the edits -applied to the sources. It drives projects built with make through -`clang-ssaf-orchestrator-make`, whose requirements it shares; see -{doc}`MakeProjects`. - -## Usage - -Run it from the project root, with its options before the build command: - -```console -$ clang-ssaf-driver --transformation=cpp-bounded-buffers -- make -j8 -``` - -The driver knows which summaries and which whole-program analysis each -transformation needs: - -```{eval-rst} -.. list-table:: - :header-rows: 1 - - * - Transformation - - What it does - * - ``cpp-bounded-buffers`` - - Rewrites the raw pointers and arrays that need bounds into - ``bounded_ptr`` and ``bounded_array``. -``` - -## Results - -A run writes two files into the output directory: - -- `<transformation>.patch`, the source edits as one unified diff whose paths - are relative to the project root, for `patch -p1` from there; -- `<transformation>.sarif`, the transformation report: the findings of every - translation unit, and every conflict between source edits. - -Both are named on stdout, after the orchestrator's own results: - -```text -patch: /work/proj/ssaf-output/cpp-bounded-buffers.patch -report: /work/proj/ssaf-output/cpp-bounded-buffers.sarif -``` - -The report is also printed on stderr, one line per finding the way a compiler -prints a diagnostic, and closes with a count for each rule: - -```text -src/shared.h:1:8: note: declarator of a multi-declarator group is not yet rewritten [cpp-bounded-buffers-skipped] -src/shared.h:3:1: error: 2 overlapping replacement(s) at file:///work/proj/src/shared.h byte 20 were dropped; resolve manually. [clang-reforge-replacement-conflict] -src/shared.h:5:1: note: candidate edit: "bounded_ptr<int> " -src/shared.h:3:1: note: candidate edit: "bounded_ptr<long> " -1 clang-reforge-replacement-conflict, 1 cpp-bounded-buffers-skipped reported -``` - -A finding is a place the transformation did not rewrite. A conflict is a place -where translation units asked for different edits; none of them is made, and -an edit that overlaps a conflict is left out of the patch too, with a warning. -Both are left to be done by hand, and neither changes the exit status. - -## Applying the edits - -`--apply` also rewrites the sources, using `clang-apply-replacements` from -clang-tools-extra. It is looked for on `PATH`, or named with -`--clang-apply-replacements=<path>`, before anything is built. The patch is -written either way, and the run fails if the sources do not end up the way the -patch shows them. - -## Options - -```{eval-rst} -.. list-table:: - :header-rows: 1 - - * - Option - - Purpose - * - ``--transformation=<name>`` - - The transformation to run. Required. - * - ``-o``, ``--output-dir=<dir>`` - - Where the results and the SSAF artifacts go. Defaults to - ``ssaf-output``, and is the orchestrator's output directory. - * - ``--apply`` - - Also apply the source edits. - * - ``--clang-apply-replacements=<path>`` - - Path to ``clang-apply-replacements``. Defaults to the name on - ``PATH``. - * - ``--ssaf-clang=<path>``, ``--ssaf-linker=<path>``, - ``--ssaf-analyzer=<path>``, ``--ssaf-src-edit-merge=<path>``, - ``--ssaf-merge-transformation-report=<path>``, - ``--ssaf-merge-conflict-report=<path>``, - ``--ssaf-extra-clang-arg=<arg>``, ``-k``, ``-v`` - - Passed to the orchestrator; see {doc}`MakeProjects`. - * - ``--version`` - - The clang version this was built from. -``` - -## What is not supported - -**Edits outside the project root.** A file the project does not contain is -left out of the patch and is not applied to; the driver names it in a warning. - -**Compilations that run in another directory**, as in recursive make or -`make -C`. A source edit names its file relative to where its compilation ran, -and is merged where the link runs, so such an edit can be lost or land on the -wrong file. Run the build from the project root. diff --git a/clang/docs/ScalableStaticAnalysis/user-docs/MakeProjects.md b/clang/docs/ScalableStaticAnalysis/user-docs/MakeProjects.md deleted file mode 100644 index ae163b8c138902..00000000000000 --- a/clang/docs/ScalableStaticAnalysis/user-docs/MakeProjects.md +++ /dev/null @@ -1,323 +0,0 @@ -# Make-Based Projects - -Running SSAF over a whole project means running summary extraction for every -translation unit the project compiles, and linking those summaries the way the -build links the objects they describe. `clang-ssaf-orchestrator-make` does that -for projects built with make. The design based on tool shims is documented in -{doc}`../developer-docs/MakeOrchestratorShims`. - -The build is executed normally, and each step it takes is paired with an SSAF -step: - -- A **compilation** gets a second compilation — the frontend job the build's - own compiler said it would run, with the `--ssaf-*` options appended and - nothing taken away, so extraction sees exactly what the build compiles. -- An **archive** and a **link** get a `clang-ssaf-linker` - invocation over the summaries of what went into them. These are not copies - of the build's command and carry no `--ssaf-*` options. - -The mapping onto the SSAF tools is direct: - -```{eval-rst} -.. list-table:: - :header-rows: 1 - - * - Build step - - SSAF step - * - ``$(CXX) -c -o parser.o parser.cpp`` - - ``clang -cc1 … -o parser.o -x c++ parser.cpp - --ssaf-extract-summaries=…`` - * - ``$(AR) rcs libparse.a parser.o lexer.o`` - - ``clang-ssaf-linker static-library create parser.json lexer.json`` - * - ``$(CXX) -o app main.o libparse.a`` - - ``clang-ssaf-linker main.json libparse.json`` -``` - -## Usage - -Run it from the project root; that is where the build runs. - -There are three workflows, each going one step further than the last: - -- **Summary extraction and linking**, with `--extract-summaries`. Every - translation unit gets a summary, and the summaries are linked the way the - build links the objects, into one link unit summary. -- **Whole-program analysis**, adding `--analysis`. `clang-ssaf-analyzer` runs - the named analyses over the link unit summary. -- **Whole-program transformation**, adding `--transformation`. A second build - generates source edits and transformation reports from the analysis result, - and they are merged across the whole program, together with the reports of - the edits that conflicted. - -Summary extraction and linking: - -```console -$ clang-ssaf-orchestrator-make \ - --extract-summaries=PointerFlow,UnsafeBufferUsage \ - -o ssaf-output \ - -- make -j8 -``` - -Whole-program analysis: - -```console -$ clang-ssaf-orchestrator-make \ - --extract-summaries=PointerFlow,UnsafeBufferUsage \ - --analysis=UnsafeBufferReachableAnalysisResult \ - -o ssaf-output \ - -- make -j8 -``` - -Whole-program transformation, which ends in merged source edits and reports: - -```console -$ clang-ssaf-orchestrator-make \ - --extract-summaries=PointerFlow,UnsafeBufferUsage \ - --analysis=UnsafeBufferReachableAnalysisResult \ - --transformation=cpp-bounded-buffers \ - -o ssaf-output \ - -- make -j8 -``` - -For a project configured by autoconf, run `configure` first and give the -orchestrator only the `make` invocation. - -## Options - -```{eval-rst} -.. list-table:: - :header-rows: 1 - - * - Option - - Purpose - * - ``--extract-summaries=<name,...>`` - - Summary extractors to run over every translation unit. Required. - * - ``--analysis=<name>`` - - Whole-program analysis to run on the link unit summary. Repeatable. - Without it the run stops after summary extraction. - * - ``--transformation=<name>`` - - Source transformation to run in the second pass. Requires - ``--analysis``. Without it the run stops after the analysis. - * - ``-o``, ``--output-dir=<dir>`` - - Where the SSAF artifacts go, made absolute. Defaults to - ``ssaf-output``. It must be absent, empty, or an earlier run's. - * - ``--ssaf-clang=<path>`` - - SSAF-capable clang used for the SSAF compilations. Defaults to - ``clang`` on ``PATH``. - * - ``--ssaf-linker=<path>`` - - Path to ``clang-ssaf-linker``. Defaults to the name on ``PATH``. - * - ``--ssaf-analyzer=<path>`` - - Path to ``clang-ssaf-analyzer``. Defaults to the name on ``PATH``, and - is only required with ``--analysis``. - * - ``--ssaf-src-edit-merge=<path>`` - - Path to ``clang-ssaf-src-edit-merge``. Defaults to the name on - ``PATH``, and is only required with ``--transformation``. - * - ``--ssaf-merge-transformation-report=<path>`` - - Path to ``clang-ssaf-merge-transformation-report``. Defaults to the - name on ``PATH``, and is only required with ``--transformation``. - * - ``--ssaf-merge-conflict-report=<path>`` - - Path to ``clang-ssaf-merge-conflict-report``. Defaults to the name on - ``PATH``, and is only required with ``--transformation``. - * - ``--ssaf-extra-clang-arg=<arg>`` - - Extra argument for every SSAF compilation. Repeatable. - * - ``-k``, ``--keep-going`` - - Exit successfully even though some SSAF steps failed. The failures are - still reported. - * - ``-v``, ``--verbose`` - - Print every command the orchestrator and the shims run, on stderr. - * - ``--version`` - - The clang version this was built from. -``` - -Each `--ssaf-*` tool must resolve to one executable file. A value naming more -than one word, a directory, something that is not executable, or a path inside -our own installed shim directory — which is what an earlier run's links on -`PATH` look like — is refused, and every unresolvable tool is reported at -once. - -## Artifacts - -A build artifact is named by its resolved **path**, so two spellings of one -file name the same artifact. That path is what a buildstep record names, and -the value of `--ssaf-compilation-unit-id=`. - -An SSAF artifact is filed under the same path with its leading separator -dropped, so an object at `/work/parse/build/parser.o` gets a summary at -`<output-dir>/ssaf-artifacts/work/parse/build/parser.o.summary.json`. - -No part of the path is elided, because the path is what tells two -translation units apart. A build that compiles one source into -`build/first/parser.o` and another into `build/second/parser.o` has two -translation units, and only the whole path tells them apart. - -```{eval-rst} -.. list-table:: - :header-rows: 1 - - * - Path - - Contents - * - ``<output-dir>/ssaf-artifacts/<path>.summary.json`` - - What SSAF holds for ``<path>``: a TU summary where the path names a - compilation's object, and whatever the SSAF linker wrote where the - build assembled it. - * - ``<output-dir>/build-graph/<path>.buildstep.json`` - - Which build artifacts were inputs for the ``<path>`` build artifact. - * - ``<output-dir>/wpa.json`` - - Whole-program analysis result. - * - ``<output-dir>/ssaf-artifacts/<path>.edits.yaml`` - - Source edits for the ``<path>`` build artifact. - * - ``<output-dir>/ssaf-artifacts/<path>.conflicts.sarif`` - - Edits dropped because they conflicted when the build assembled - ``<path>``. - * - ``<output-dir>/ssaf-artifacts/<path>.conflict-report.sarif`` - - Every conflict behind the ``<path>`` build artifact: its own, and its - inputs'. - * - ``<output-dir>/ssaf-artifacts/<path>.report.sarif`` - - Transformation report for the ``<path>`` build artifact: a - compilation's own, and its inputs' merged where the build assembled it. - * - ``<output-dir>/config-analysis.json`` - - Configuration for the analysis pass, the first build pass. - * - ``<output-dir>/config-transformation.json`` - - Configuration for the transformation pass, the second build pass. - * - ``<output-dir>/shims/`` - - Links to the shims named so they interpose the binary tools, recreated - on every run. - * - ``<output-dir>/.clang-ssaf-orchestrator-make`` - - SSAF ownership attribute for cleanup. - * - ``<output-dir>/failure-logs/<path>.<kind>.log`` - - What went wrong for ``<path>``: a command writing it whose SSAF step - failed, or an input that had no summary. - * - ``<output-dir>/failure-logs/<dir>/<hash>.<kind>.log`` - - What went wrong for a command that names no output, and any internal - error in a shim; ``<hash>`` names the command and ``<dir>`` is where - it ran. Every ``.log`` anywhere under the output directory counts - against the run; a log is always a failure. -``` - -The buildstep records are what the top-most link unit is found from. An -input... [truncated] `````````` </details> https://github.com/llvm/llvm-project/pull/232274 _______________________________________________ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
