https://github.com/jtb20 updated https://github.com/llvm/llvm-project/pull/242480
>From c82a3baddb6f84f283ff1de47ed0b63854162c2f Mon Sep 17 00:00:00 2001 From: Julian Brown <[email protected]> Date: Thu, 12 Nov 2026 17:02:53 -0600 Subject: [PATCH] [Clang][OpenMP] Serialize lockstep warp lanes across omp_set_lock regions Where the threads of a wavefront/warp run in lockstep (as for many typical GPUs), multiple lanes reaching an omp_set_lock call may deadlock (or trap, on AMDGPU) if they contend for a lock. This is actually permitted by the spec (OpenMP 6.0, 1.2 Execution Model: "A program that relies on concurrent execution of such diverging code paths for the purposes of synchronization may deadlock."). However, in some common cases we can do better. This patch allows set/unset pairs to work as long as their lock expressions are the same, they are in the same compound statement, and no intervening code has control-flow that enters or leaves the sequence or various other constructs. I.e.: omp_set_lock(L); ... omp_unset_lock(L); is emitted through the lane serialization that OpenMPIRBuilder provides for critical regions, so that each lane takes and releases the lock in turn. This is done only where every lane is certain to release every lock it takes before the next lane's turn. A new warning (with explanatory note) is added in Sema for patterns which can confidently be predicted to allow multiple lanes of a wavefront to reach an omp_set_lock when the serialization *cannot* be done. The name is -Wopenmp-lockstep-lock. This patch is a little invasive, and not actually required for OpenMP conformance (at least as of 6.0). As such the previous patches can be committed without it if desired. Assisted-By: Claude --- clang/docs/OpenMPSupport.md | 24 ++ clang/docs/ReleaseNotes.md | 8 + clang/include/clang/AST/OpenMPLockSequence.h | 87 +++++ clang/include/clang/Basic/DiagnosticGroups.td | 3 +- .../clang/Basic/DiagnosticSemaKinds.td | 18 + clang/include/clang/Sema/SemaOpenMP.h | 12 + clang/lib/AST/CMakeLists.txt | 1 + clang/lib/AST/OpenMPLockSequence.cpp | 177 +++++++++ clang/lib/CodeGen/CGOpenMPRuntime.h | 10 + clang/lib/CodeGen/CGOpenMPRuntimeGPU.cpp | 15 + clang/lib/CodeGen/CGOpenMPRuntimeGPU.h | 6 + clang/lib/CodeGen/CGStmt.cpp | 13 +- clang/lib/Sema/SemaOpenMP.cpp | 165 +++++++++ clang/lib/Sema/SemaStmt.cpp | 5 +- .../OpenMP/amdgcn_lock_lane_serialization.c | 337 ++++++++++++++++++ .../OpenMP/amdgcn_lock_lane_serialization.cpp | 158 ++++++++ .../test/OpenMP/target_lock_lane_warning.cpp | 233 ++++++++++++ offload/test/offloading/target_lock_lanes.c | 115 ++++++ 18 files changed, 1384 insertions(+), 3 deletions(-) create mode 100644 clang/include/clang/AST/OpenMPLockSequence.h create mode 100644 clang/lib/AST/OpenMPLockSequence.cpp create mode 100644 clang/test/OpenMP/amdgcn_lock_lane_serialization.c create mode 100644 clang/test/OpenMP/amdgcn_lock_lane_serialization.cpp create mode 100644 clang/test/OpenMP/target_lock_lane_warning.cpp create mode 100644 offload/test/offloading/target_lock_lanes.c diff --git a/clang/docs/OpenMPSupport.md b/clang/docs/OpenMPSupport.md index 0978d7269d0ca7..0582be3eea5277 100644 --- a/clang/docs/OpenMPSupport.md +++ b/clang/docs/OpenMPSupport.md @@ -84,6 +84,30 @@ between the threads in the parallel regions. Often, the optimizer is able to reduce the cost of `Generic` mode to the level of `Cuda` mode, but the flag, as well as other assumption flags, can be used for tuning. +### Locks and critical regions on GPUs + +On a GPU, the threads of a warp (an AMD GPU wavefront) may execute a single +instruction stream, so in OpenMP 6.0 terms a warp is a *progress unit*. +Divergent threads in such a progress unit that synchronize with each other, e.g. +by calling `omp_set_lock(lock)`, may deadlock. A `parallel` on the GPU behaves +as if it had a `safesync` clause whose width is the warp size. + +- A `critical` region is run by the threads of each warp one at a time. +- A sequence `omp_set_lock(lock); ...; omp_unset_lock(lock);` is also run one + thread at a time, if + - both calls are statements of the same block, with the same lock expression; + - the code between them can be entered only at the start and left only at + the end, contains no OpenMP directive and declares nothing at its top level; + - it uses the lock routines only in further such pairs, properly nested, at + the top level of the block. +- Any other `omp_set_lock` that several threads of a warp reach together may + deadlock if they contend for the same lock. On AMD GPUs the device runtime + traps instead when they reach it together, with an explanatory message. + +Clang warns (`-Wopenmp-lockstep-lock`) about each call to `omp_set_lock` that +cannot be handled this way and that is within a construct where multiple lanes +may be active. + ### Features not supported or with limited support for Cuda devices - Cancellation constructs are not supported. diff --git a/clang/docs/ReleaseNotes.md b/clang/docs/ReleaseNotes.md index 3c24a33e59f08d..af2cdf9df7007f 100644 --- a/clang/docs/ReleaseNotes.md +++ b/clang/docs/ReleaseNotes.md @@ -1740,6 +1740,14 @@ The `alpha.cplusplus.UseAfterLifetimeEnd` checker was renamed to `alpha.core.Use another lane of the same wavefront spins for it; `omp_set_lock` traps in that case rather than hang. +- On GPUs, whose threads of a warp may run in lockstep, where + `omp_set_lock(lock)` and the matching `omp_unset_lock(lock)` are in the same + block, with code between them that can only be entered at the start and left + at the end, the threads of a warp now run that sequence one at a time. This + makes the common pattern of every thread taking the same lock work. When an + `omp_set_lock` cannot be handled this way and several threads of a warp may + reach it together, a new warning `-Wopenmp-lockstep-lock` explains the reason. + ### SYCL Support - On a `spirv32-unknown-unknown` or `spirv64-unknown-unknown` offload target, diff --git a/clang/include/clang/AST/OpenMPLockSequence.h b/clang/include/clang/AST/OpenMPLockSequence.h new file mode 100644 index 00000000000000..4757103756819f --- /dev/null +++ b/clang/include/clang/AST/OpenMPLockSequence.h @@ -0,0 +1,87 @@ +//===- OpenMPLockSequence.h - OpenMP lock sequences in a block --*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// Where the threads of a wavefront run in lockstep, a thread that takes an +// OpenMP lock cannot release it while another thread of its wavefront spins for +// the same lock. A compound statement containing +// +// omp_set_lock(L); ... omp_unset_lock(L); +// +// can instead be run by the threads of a wavefront one at a time, provided that +// each thread releases every lock it takes before the next thread's turn. The +// analysis in this file is used both by code generation and semantic analysis; +// the latter warns about omp_set_lock cases that cannot be handled like this. +// +//===----------------------------------------------------------------------===// + +#ifndef LLVM_CLANG_AST_OPENMPLOCKSEQUENCE_H +#define LLVM_CLANG_AST_OPENMPLOCKSEQUENCE_H + +#include "clang/Basic/LLVM.h" + +namespace clang { + +class ASTContext; +class CallExpr; +class Decl; +class Stmt; + +/// Reasons why an omp_set_lock ... omp_unset_lock sequence might be rejected. +/// Keep this list aligned with the %select in +/// warn_omp_lock_not_lane_serialized. +enum class OMPLockSequenceRejection : unsigned { + /// No matching omp_unset_lock follows it in the same compound statement. + NoUnset, + /// Control can enter or leave the code before the matching omp_unset_lock + /// other than at its ends. + Escape, + /// A lock routine is used before the matching omp_unset_lock other than in + /// properly nested pairs at the top level of the compound statement. + OtherLockUse, + /// An OpenMP directive comes before the matching omp_unset_lock. + Directive, + /// A variable is declared before the matching omp_unset_lock. + Declaration, + /// The lock expression has side effects. + SideEffects, + /// The omp_set_lock statement is labelled. + Labelled, + /// The omp_set_lock call is not a statement of a compound statement. + NotInBlock, +}; + +/// The result of looking for a lock sequence at the start of some statements. +struct OMPLockSequence { + /// The omp_set_lock call made by the first statement, if it makes one. + const CallExpr *SetCall = nullptr; + /// The number of statements in the sequence, or 0 if there is none. + unsigned Length = 0; + /// Why there is no sequence, if SetCall is set and Length is 0. + OMPLockSequenceRejection Rejection = OMPLockSequenceRejection::NoUnset; +}; + +/// Returns true if \p D is one of the OpenMP lock routines that set, unset or +/// test a lock. +bool isOMPLockRoutine(const Decl *D); + +/// Returns true if \p CE calls omp_set_lock. +bool isOMPSetLockCall(const CallExpr *CE); + +/// Looks for a sequence `omp_set_lock(L); ... omp_unset_lock(L);` that the +/// threads of a warp can run one at a time at the start of \p Stmts, the +/// statements of a compound statement not yet considered. The sequence must use +/// the same side-effect-free lock expression in both calls; the statements in +/// between must be enterable only at the start and leavable only at the end, +/// declare nothing, contain no OpenMP directive, and use the lock routines only +/// in further such pairs, properly nested, among themselves. +OMPLockSequence matchOMPLockSequence(ArrayRef<Stmt *> Stmts, + const ASTContext &Ctx); + +} // namespace clang + +#endif // LLVM_CLANG_AST_OPENMPLOCKSEQUENCE_H diff --git a/clang/include/clang/Basic/DiagnosticGroups.td b/clang/include/clang/Basic/DiagnosticGroups.td index 4d50ecb82747e9..4445786bb45865 100644 --- a/clang/include/clang/Basic/DiagnosticGroups.td +++ b/clang/include/clang/Basic/DiagnosticGroups.td @@ -1803,11 +1803,12 @@ def OpenMPPre51Compat : DiagGroup<"pre-openmp-51-compat">; def OpenMP51Ext : DiagGroup<"openmp-51-extensions">; def OpenMPExtensions : DiagGroup<"openmp-extensions">; def OpenMPTargetException : DiagGroup<"openmp-target-exception">; +def OpenMPLockstepLock : DiagGroup<"openmp-lockstep-lock">; def OpenMPFuture : DiagGroup<"openmp-future">; def OpenMP : DiagGroup<"openmp", [ SourceUsesOpenMP, OpenMPClauses, OpenMPLoopForm, OpenMPTarget, OpenMPMapping, OpenMP51Ext, OpenMPExtensions, OpenMPTargetException, - OpenMPFuture + OpenMPFuture, OpenMPLockstepLock ]>; // OpenACC warnings. diff --git a/clang/include/clang/Basic/DiagnosticSemaKinds.td b/clang/include/clang/Basic/DiagnosticSemaKinds.td index dff45a081f2e8c..68a4c89da4e868 100644 --- a/clang/include/clang/Basic/DiagnosticSemaKinds.td +++ b/clang/include/clang/Basic/DiagnosticSemaKinds.td @@ -12395,6 +12395,24 @@ def warn_omp_declare_target_local_nohost : Warning<"'device_type(nohost)' is not yet supported with 'local' clause; " "treating as 'device_type(any)'">, InGroup<OpenMPTarget>; +def warn_omp_lock_not_lane_serialized : Warning< + "'omp_set_lock' may be reached by several threads of a warp at once, but " + "they cannot be made to take the lock one at a time because " + // The cases follow OMPLockSequenceRejection. + "%select{no matching 'omp_unset_lock' follows it in the same block|" + "control can enter or leave the locked code other than at its ends|" + "the locked code uses a lock routine other than in properly nested " + "set/unset pairs|" + "the locked code contains an OpenMP directive|" + "the locked code declares a variable|" + "the lock expression has side effects|" + "the call is labelled; put the locked code in a block of its own|" + "the call is not a statement of its own in a block}0; threads of a warp " + "that contend for the lock will trap or hang">, + InGroup<OpenMPLockstepLock>; +def note_omp_lock_lockstep_context : Note< + "threads of a warp may execute this '#pragma omp %0' region together; use " + "'single', 'masked' or 'critical' to run code on one thread at a time">; def warn_omp_not_in_target_context : Warning< "declaration is not declared in any declare target region">, InGroup<OpenMPTarget>; diff --git a/clang/include/clang/Sema/SemaOpenMP.h b/clang/include/clang/Sema/SemaOpenMP.h index a8a291d02722ce..6f4e5df82b1732 100644 --- a/clang/include/clang/Sema/SemaOpenMP.h +++ b/clang/include/clang/Sema/SemaOpenMP.h @@ -29,6 +29,7 @@ #include "clang/Sema/Ownership.h" #include "clang/Sema/SemaBase.h" #include "llvm/ADT/DenseMap.h" +#include "llvm/ADT/DenseSet.h" #include "llvm/Frontend/OpenMP/OMP.h.inc" #include "llvm/Frontend/OpenMP/OMPConstants.h" #include <array> @@ -76,6 +77,12 @@ class SemaOpenMP : public SemaBase { unsigned DiagID, const FunctionDecl *FD); + /// Where the threads of a warp run in lockstep on the device, warns about + /// each call to omp_set_lock in the compound statement \p CS that several + /// threads of a warp may reach together, but that does not begin a sequence + /// they can be made to run one at a time (see OpenMPLockSequence.h). + void diagnoseLockstepLocks(const CompoundStmt *CS); + /// Creates a SemaDiagnosticBuilder that emits the diagnostic if the current /// context is "used as host code". /// @@ -1542,6 +1549,11 @@ class SemaOpenMP : public SemaBase { private: void *VarDataSharingAttributesStack; + /// Locations of calls to omp_set_lock already diagnosed by + /// diagnoseLockstepLocks, so that each template instantiation does not + /// repeat the warning. + llvm::DenseSet<SourceLocation> DiagnosedLockstepLockCalls; + /// Number of nested '#pragma omp declare target' directives. SmallVector<DeclareTargetContextInfo, 4> DeclareTargetNesting; diff --git a/clang/lib/AST/CMakeLists.txt b/clang/lib/AST/CMakeLists.txt index 5d910d71d7d74b..0a9f2a78b22f42 100644 --- a/clang/lib/AST/CMakeLists.txt +++ b/clang/lib/AST/CMakeLists.txt @@ -110,6 +110,7 @@ add_clang_library(clangAST ODRHash.cpp OpenACCClause.cpp OpenMPClause.cpp + OpenMPLockSequence.cpp OSLog.cpp ParentMap.cpp PrintfFormatString.cpp diff --git a/clang/lib/AST/OpenMPLockSequence.cpp b/clang/lib/AST/OpenMPLockSequence.cpp new file mode 100644 index 00000000000000..0979075aa684cf --- /dev/null +++ b/clang/lib/AST/OpenMPLockSequence.cpp @@ -0,0 +1,177 @@ +//===- OpenMPLockSequence.cpp - OpenMP lock sequences in a block ----------===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// + +#include "clang/AST/OpenMPLockSequence.h" +#include "clang/AST/ASTContext.h" +#include "clang/AST/Expr.h" +#include "clang/AST/ExprCXX.h" +#include "clang/AST/StmtCXX.h" +#include "clang/AST/StmtObjC.h" +#include "clang/AST/StmtOpenMP.h" +#include "llvm/ADT/FoldingSet.h" +#include "llvm/ADT/STLExtras.h" +#include "llvm/ADT/StringSwitch.h" +#include <optional> + +using namespace clang; + +bool clang::isOMPLockRoutine(const Decl *D) { + const auto *FD = dyn_cast_or_null<FunctionDecl>(D); + if (!FD || !FD->isExternC() || !FD->getIdentifier()) + return false; + return llvm::StringSwitch<bool>(FD->getName()) + .Cases({"omp_set_lock", "omp_unset_lock", "omp_test_lock", + "omp_set_nest_lock", "omp_unset_nest_lock", "omp_test_nest_lock"}, + true) + .Default(false); +} + +bool clang::isOMPSetLockCall(const CallExpr *CE) { + const FunctionDecl *FD = CE->getDirectCallee(); + return isOMPLockRoutine(FD) && FD->getName() == "omp_set_lock"; +} + +/// If \p S is the statement `Name(Arg);` calling the OpenMP lock routine +/// \p Name, return the call. +static const CallExpr *getLockCall(const Stmt *S, StringRef Name) { + const auto *E = dyn_cast<Expr>(S); + if (!E) + return nullptr; + const auto *CE = dyn_cast<CallExpr>(E->IgnoreImplicit()->IgnoreParens()); + if (!CE || CE->getNumArgs() != 1) + return nullptr; + const FunctionDecl *FD = CE->getDirectCallee(); + if (!isOMPLockRoutine(FD) || FD->getName() != Name) + return nullptr; + return CE; +} + +/// If \p S is the statement `Name(Arg);` calling the OpenMP lock routine +/// \p Name, return Arg. +static const Expr *getLockCallArg(const Stmt *S, StringRef Name) { + const CallExpr *CE = getLockCall(S, Name); + return CE ? CE->getArg(0)->IgnoreParenImpCasts() : nullptr; +} + +/// Checks that control can enter \p S only at its start and leave it only at +/// its end, and that \p S contains no OpenMP directive and makes no use of an +/// OpenMP lock routine. Returns why not, if it does not. \p InLoop and +/// \p InSwitch say whether an enclosing loop or switch statement is itself part +/// of the sequence being checked. +static std::optional<OMPLockSequenceRejection> +checkSerializable(const Stmt *S, bool InLoop, bool InSwitch) { + using R = OMPLockSequenceRejection; + if (!S) + return std::nullopt; + if (isa<OMPExecutableDirective>(S)) + return R::Directive; + if (isa<ReturnStmt, GotoStmt, IndirectGotoStmt, LabelStmt, CoreturnStmt, + CoroutineSuspendExpr, CXXTryStmt, SEHTryStmt, SEHLeaveStmt, + ObjCAtTryStmt, ObjCAtThrowStmt>(S)) + return R::Escape; + if (const auto *Asm = dyn_cast<GCCAsmStmt>(S); Asm && Asm->isAsmGoto()) + return R::Escape; + // A lock taken or released conditionally, or other than in a pair at the top + // level of the sequence, could be held from one thread's turn to the next. + if (const auto *DRE = dyn_cast<DeclRefExpr>(S); + DRE && isOMPLockRoutine(DRE->getDecl())) + return R::OtherLockUse; + if (isa<BreakStmt>(S)) + return InLoop || InSwitch ? std::nullopt : std::optional(R::Escape); + if (isa<ContinueStmt>(S)) + return InLoop ? std::nullopt : std::optional(R::Escape); + if (isa<SwitchCase>(S) && !InSwitch) + return R::Escape; + // The bodies of lambdas and blocks belong to other functions. + if (isa<LambdaExpr, BlockExpr>(S)) + return std::nullopt; + bool IsLoop = isa<ForStmt, WhileStmt, DoStmt, CXXForRangeStmt>(S); + bool IsSwitch = isa<SwitchStmt>(S); + for (const Stmt *Child : S->children()) + if (auto Why = + checkSerializable(Child, InLoop || IsLoop, InSwitch || IsSwitch)) + return Why; + return std::nullopt; +} + +OMPLockSequence clang::matchOMPLockSequence(ArrayRef<Stmt *> Stmts, + const ASTContext &Ctx) { + using R = OMPLockSequenceRejection; + OMPLockSequence Result; + if (Stmts.empty()) + return Result; + + // Code generation cannot put a label inside the loop that runs the threads + // through the sequence one at a time, since control would enter the loop + // through it. + const Stmt *First = Stmts.front(); + if (isa<LabelStmt, SwitchCase>(First)) { + const Stmt *Sub = First; + while (const auto *L = dyn_cast<LabelStmt>(Sub)) + Sub = L->getSubStmt(); + while (const auto *C = dyn_cast<SwitchCase>(Sub)) + Sub = C->getSubStmt(); + if ((Result.SetCall = getLockCall(Sub, "omp_set_lock"))) + Result.Rejection = R::Labelled; + return Result; + } + + Result.SetCall = getLockCall(First, "omp_set_lock"); + if (!Result.SetCall) + return Result; + auto GetLockID = + [&](const Expr *Lock) -> std::optional<llvm::FoldingSetNodeID> { + if (Lock->HasSideEffects(Ctx)) + return std::nullopt; + llvm::FoldingSetNodeID ID; + Lock->Profile(ID, Ctx, /*Canonical=*/true); + return ID; + }; + auto Reject = [&](R Why) { + Result.Rejection = Why; + return Result; + }; + + // The locks set in the sequence and not yet unset, innermost last. + SmallVector<llvm::FoldingSetNodeID, 2> Held; + auto ID = GetLockID(Result.SetCall->getArg(0)->IgnoreParenImpCasts()); + if (!ID) + return Reject(R::SideEffects); + Held.push_back(*ID); + for (unsigned I = 1, E = Stmts.size(); I != E; ++I) { + const Stmt *S = Stmts[I]; + if (const Expr *Set = getLockCallArg(S, "omp_set_lock")) { + auto ID = GetLockID(Set); + if (!ID) + return Reject(R::SideEffects); + if (llvm::is_contained(Held, *ID)) + return Reject(R::OtherLockUse); + Held.push_back(*ID); + continue; + } + if (const Expr *Unset = getLockCallArg(S, "omp_unset_lock")) { + auto ID = GetLockID(Unset); + if (!ID) + return Reject(R::SideEffects); + if (*ID != Held.back()) + return Reject(R::OtherLockUse); + Held.pop_back(); + if (Held.empty()) { + Result.Length = I + 1; + return Result; + } + continue; + } + // A declaration would remain in scope after the sequence ends. + if (isa<DeclStmt>(S)) + return Reject(R::Declaration); + if (auto Why = checkSerializable(S, /*InLoop=*/false, /*InSwitch=*/false)) + return Reject(*Why); + } + return Reject(R::NoUnset); +} diff --git a/clang/lib/CodeGen/CGOpenMPRuntime.h b/clang/lib/CodeGen/CGOpenMPRuntime.h index 47c05d62a3b412..3189130ca343b2 100644 --- a/clang/lib/CodeGen/CGOpenMPRuntime.h +++ b/clang/lib/CodeGen/CGOpenMPRuntime.h @@ -804,6 +804,16 @@ class CGOpenMPRuntime { OpenMPSeverityClauseKind Severity = OMPC_SEVERITY_fatal, const Expr *Message = nullptr); + /// Called while emitting the statements of a compound statement in device + /// code, with \p Stmts the statements not yet emitted. If a sequence of them + /// starting with the first needs to be emitted as a whole, emits it and + /// returns the number of statements emitted; otherwise returns 0 and emits + /// nothing. + virtual unsigned emitStmtSequence(CodeGenFunction &CGF, + ArrayRef<Stmt *> Stmts) { + return 0; + } + /// Emits a critical region. /// \param CriticalName Name of the critical region. /// \param CriticalOpGen Generator for the statement associated with the given diff --git a/clang/lib/CodeGen/CGOpenMPRuntimeGPU.cpp b/clang/lib/CodeGen/CGOpenMPRuntimeGPU.cpp index 2d4e8577b88b82..6f17eff44924cc 100644 --- a/clang/lib/CodeGen/CGOpenMPRuntimeGPU.cpp +++ b/clang/lib/CodeGen/CGOpenMPRuntimeGPU.cpp @@ -18,6 +18,7 @@ #include "clang/AST/Attr.h" #include "clang/AST/DeclOpenMP.h" #include "clang/AST/OpenMPClause.h" +#include "clang/AST/OpenMPLockSequence.h" #include "clang/AST/StmtOpenMP.h" #include "clang/AST/StmtVisitor.h" #include "llvm/ADT/SmallPtrSet.h" @@ -1389,6 +1390,20 @@ void CGOpenMPRuntimeGPU::emitLaneSerializedRegion( CGF.Builder.restoreIP(AfterIP); } +unsigned CGOpenMPRuntimeGPU::emitStmtSequence(CodeGenFunction &CGF, + ArrayRef<Stmt *> Stmts) { + if (!CGM.getTarget().getGridValue().GV_Lockstep_Lanes) + return 0; + OMPLockSequence Seq = matchOMPLockSequence(Stmts, CGM.getContext()); + if (!Seq.Length) + return 0; + emitLaneSerializedRegion(CGF, [&] { + for (const Stmt *S : Stmts.take_front(Seq.Length)) + CGF.EmitStmt(S); + }); + return Seq.Length; +} + void CGOpenMPRuntimeGPU::emitCriticalRegion( CodeGenFunction &CGF, StringRef CriticalName, const RegionCodeGenTy &CriticalOpGen, SourceLocation Loc, diff --git a/clang/lib/CodeGen/CGOpenMPRuntimeGPU.h b/clang/lib/CodeGen/CGOpenMPRuntimeGPU.h index b30542366ae07c..799c8c86f888f6 100644 --- a/clang/lib/CodeGen/CGOpenMPRuntimeGPU.h +++ b/clang/lib/CodeGen/CGOpenMPRuntimeGPU.h @@ -290,6 +290,12 @@ class CGOpenMPRuntimeGPU : public CGOpenMPRuntime { SourceLocation Loc, const Expr *Hint = nullptr) override; + /// Where the threads of a warp run in lockstep, emits a sequence starting + /// omp_set_lock(L) and ending with the matching omp_unset_lock(L) so that + /// the threads of a warp run through it one at a time. + unsigned emitStmtSequence(CodeGenFunction &CGF, + ArrayRef<Stmt *> Stmts) override; + /// Emit a code for reduction clause. /// /// \param Privates List of private copies for original reduction arguments. diff --git a/clang/lib/CodeGen/CGStmt.cpp b/clang/lib/CodeGen/CGStmt.cpp index 455c4352661043..ae1b76df27cf9f 100644 --- a/clang/lib/CodeGen/CGStmt.cpp +++ b/clang/lib/CodeGen/CGStmt.cpp @@ -588,10 +588,21 @@ CodeGenFunction::EmitCompoundStmtWithoutScope(const CompoundStmt &S, bool GetLast, AggValueSlot AggSlot) { + // The OpenMP device runtime may need to emit a run of statements as a whole. + bool OpenMPDevice = + getLangOpts().OpenMP && getLangOpts().OpenMPIsTargetDevice; for (CompoundStmt::const_body_iterator I = S.body_begin(), E = S.body_end() - GetLast; - I != E; ++I) + I != E; ++I) { + if (OpenMPDevice) { + if (unsigned N = CGM.getOpenMPRuntime().emitStmtSequence( + *this, llvm::ArrayRef(I, E))) { + I += N - 1; + continue; + } + } EmitStmt(*I); + } Address RetAlloca = Address::invalid(); if (GetLast) { diff --git a/clang/lib/Sema/SemaOpenMP.cpp b/clang/lib/Sema/SemaOpenMP.cpp index 24782f5a8c417c..75a3af560efad5 100644 --- a/clang/lib/Sema/SemaOpenMP.cpp +++ b/clang/lib/Sema/SemaOpenMP.cpp @@ -24,6 +24,7 @@ #include "clang/AST/DeclOpenMP.h" #include "clang/AST/DynamicRecursiveASTVisitor.h" #include "clang/AST/OpenMPClause.h" +#include "clang/AST/OpenMPLockSequence.h" #include "clang/AST/RecursiveASTVisitor.h" #include "clang/AST/Stmt.h" #include "clang/AST/StmtCXX.h" @@ -47,6 +48,7 @@ #include "llvm/ADT/SetVector.h" #include "llvm/ADT/SmallSet.h" #include "llvm/ADT/StringExtras.h" +#include "llvm/Frontend/OpenMP/OMP.h" #include "llvm/Frontend/OpenMP/OMPAssume.h" #include "llvm/Frontend/OpenMP/OMPConstants.h" #include "llvm/Frontend/OpenMP/OMPVersion.h" @@ -652,6 +654,15 @@ class DSAStackTy { const llvm::function_ref<bool(OpenMPDirectiveKind)> DPred, unsigned Level) const; + /// Calls \p F with the kind and location of each directive enclosing the + /// current point, the current directive first, until it returns true. + void forEachDirective( + llvm::function_ref<bool(OpenMPDirectiveKind, SourceLocation)> F) const { + for (const_iterator I = begin(), E = end(); I != E; ++I) + if (F(I->Directive, I->ConstructLoc)) + return; + } + /// Finds a directive which matches specified \a DPred predicate. bool hasDirective( const llvm::function_ref<bool( @@ -2176,6 +2187,160 @@ SemaOpenMP::diagIfOpenMPDeviceCode(SourceLocation Loc, unsigned DiagID, return SemaDiagnosticBuilder(Kind, Loc, DiagID, FD, SemaRef); } +namespace { +/// How the threads of a warp may run the code that a construct encloses. +enum class LockstepContextKind { + /// The construct does not decide; look further out. + Undecided, + /// One thread at a time. + OneThread, + /// Several threads of a warp together. + Lockstep, + /// The sequential part of a target or teams region. + Sequential, +}; + +/// The innermost construct that decides how the threads of a warp may run some +/// code. +struct LockstepContext { + LockstepContextKind Kind = LockstepContextKind::Undecided; + OpenMPDirectiveKind Directive = OMPD_unknown; + SourceLocation Loc; + bool InTarget = false; + + /// Takes into account a construct of kind \p DKind at \p DLoc that encloses + /// the code in the context. Returns true if it decides. + bool enclose(OpenMPDirectiveKind DKind, SourceLocation DLoc); +}; +} // namespace + +bool LockstepContext::enclose(OpenMPDirectiveKind DKind, SourceLocation DLoc) { + // The leaf constructs of a combined construct are nested in order, so the + // last that decides is the innermost. + LockstepContextKind New = LockstepContextKind::Undecided; + for (OpenMPDirectiveKind Leaf : + llvm::reverse(llvm::omp::getLeafConstructsOrSelf(DKind))) { + switch (Leaf) { + case OMPD_single: + case OMPD_masked: + case OMPD_master: + case OMPD_critical: + case OMPD_ordered_blockassoc: + // A sections region runs each section, explicit or implicit, on one thread. + case OMPD_sections: + case OMPD_section: + New = LockstepContextKind::OneThread; + break; + default: + if (isOpenMPParallelDirective(Leaf) || + isOpenMPWorksharingDirective(Leaf) || + isOpenMPGenericLoopDirective(Leaf)) + New = LockstepContextKind::Lockstep; + else if (isOpenMPTargetExecutionDirective(Leaf) || + isOpenMPTeamsDirective(Leaf)) + New = LockstepContextKind::Sequential; + break; + } + if (New != LockstepContextKind::Undecided) + break; + } + InTarget |= isOpenMPTargetExecutionDirective(DKind); + if (New == LockstepContextKind::Undecided) + return false; + Kind = New; + Directive = DKind; + Loc = DLoc; + return true; +} + +void SemaOpenMP::diagnoseLockstepLocks(const CompoundStmt *CS) { + assert(getLangOpts().OpenMP && getLangOpts().OpenMPIsTargetDevice && + "Expected OpenMP device compilation."); + const TargetInfo &TI = getASTContext().getTargetInfo(); + if (!TI.getTriple().isGPU() || !TI.getGridValue().GV_Lockstep_Lanes) + return; + // The lock expressions in a template are only known once it is instantiated. + if (SemaRef.CurContext->isDependentContext()) + return; + // The callers of a lambda or block are not known. + if (isa_and_nonnull<sema::LambdaScopeInfo, sema::BlockScopeInfo>( + SemaRef.getCurFunction())) + return; + + LockstepContext Context; + DSAStack->forEachDirective([&](OpenMPDirectiveKind K, SourceLocation L) { + return Context.enclose(K, L); + }); + Context.InTarget |= isInOpenMPTargetExecutionDirective(); + const FunctionDecl *FD = SemaRef.getCurFunctionDecl(); + + auto Diagnose = [&](const CallExpr *CE, OMPLockSequenceRejection Why, + const LockstepContext &C) { + if (C.Kind != LockstepContextKind::Lockstep || + !DiagnosedLockstepLockCalls.insert(CE->getExprLoc()).second) + return; + // Code in a target region is emitted for the device; elsewhere, only if + // the function is. + auto Report = [&](SourceLocation Loc, unsigned DiagID) { + return C.InTarget + ? SemaDiagnosticBuilder(SemaDiagnosticBuilder::K_Immediate, + Loc, DiagID, FD, SemaRef) + : diagIfOpenMPDeviceCode(Loc, DiagID, FD); + }; + Report(CE->getExprLoc(), diag::warn_omp_lock_not_lane_serialized) + << static_cast<unsigned>(Why); + Report(C.Loc, diag::note_omp_lock_lockstep_context) + << getOpenMPDirectiveName(C.Directive, + getLangOpts().getOpenMPVersion()); + }; + + // Calls to omp_set_lock nested within a statement of the block, other than + // in nested blocks, which were checked when they were built. + std::function<void(const Stmt *, LockstepContext, const CallExpr *)> Walk = + [&](const Stmt *S, LockstepContext C, const CallExpr *Skip) { + if (!S || isa<CompoundStmt, LambdaExpr, BlockExpr>(S)) + return; + if (const auto *D = dyn_cast<OMPExecutableDirective>(S)) { + // A construct that does not decide leaves the context as it was. + LockstepContext Inner; + Inner.InTarget = C.InTarget; + if (!Inner.enclose(D->getDirectiveKind(), D->getBeginLoc())) { + Inner.Kind = C.Kind; + Inner.Directive = C.Directive; + Inner.Loc = C.Loc; + } + if (D->hasAssociatedStmt()) + Walk(D->getAssociatedStmt(), Inner, Skip); + return; + } + if (const auto *Captured = dyn_cast<CapturedStmt>(S)) { + Walk(Captured->getCapturedStmt(), C, Skip); + return; + } + if (const auto *CE = dyn_cast<CallExpr>(S); + CE && CE != Skip && isOMPSetLockCall(CE)) + Diagnose(CE, OMPLockSequenceRejection::NotInBlock, C); + for (const Stmt *Child : S->children()) + Walk(Child, C, Skip); + }; + + // Check the block as code generation will emit it: a sequence that the + // threads of a warp can run one at a time is emitted as a whole. + ArrayRef<Stmt *> Body = CS->body(); + for (unsigned I = 0, E = Body.size(); I != E;) { + OMPLockSequence Seq = + matchOMPLockSequence(Body.drop_front(I), getASTContext()); + if (Seq.Length) { + I += Seq.Length; + continue; + } + if (Seq.SetCall) + Diagnose(Seq.SetCall, Seq.Rejection, Context); + Walk(Body[I], Context, Seq.SetCall); + ++I; + } +} + SemaBase::SemaDiagnosticBuilder SemaOpenMP::diagIfOpenMPHostCode(SourceLocation Loc, unsigned DiagID, const FunctionDecl *FD) { diff --git a/clang/lib/Sema/SemaStmt.cpp b/clang/lib/Sema/SemaStmt.cpp index 4bf5b3f3ec6b74..ca0209d604205d 100644 --- a/clang/lib/Sema/SemaStmt.cpp +++ b/clang/lib/Sema/SemaStmt.cpp @@ -519,7 +519,10 @@ StmtResult Sema::ActOnCompoundStmt(SourceLocation L, SourceLocation R, : getCurCompoundScope().InitialFPFeatures; FPOptionsOverride FPDiff = getCurFPFeatures().getChangesFrom(FPO); - return CompoundStmt::Create(Context, Elts, FPDiff, L, R); + CompoundStmt *CS = CompoundStmt::Create(Context, Elts, FPDiff, L, R); + if (getLangOpts().OpenMP && getLangOpts().OpenMPIsTargetDevice) + OpenMP().diagnoseLockstepLocks(CS); + return CS; } ExprResult diff --git a/clang/test/OpenMP/amdgcn_lock_lane_serialization.c b/clang/test/OpenMP/amdgcn_lock_lane_serialization.c new file mode 100644 index 00000000000000..135db7a77ad8a9 --- /dev/null +++ b/clang/test/OpenMP/amdgcn_lock_lane_serialization.c @@ -0,0 +1,337 @@ +// REQUIRES: amdgpu-registered-target + +// On AMDGPU, whose lanes run in lockstep, an omp_set_lock ... omp_unset_lock +// sequence with straight-line control flow between the calls is run by the +// lanes of a wavefront one at a time, so that lanes contending for the lock do +// not deadlock. Anything that could leave a lock held from one lane's turn to +// the next, or leave the sequence other than at its end, is left alone. + +// RUN: %clang_cc1 -fopenmp -x c -triple x86_64-unknown-unknown -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm-bc %s -o %t-host.bc +// RUN: %clang_cc1 -fopenmp -x c -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s +// RUN: %clang_cc1 -fopenmp -x c -triple x86_64-unknown-unknown -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -o - | FileCheck %s --check-prefix=HOST +// expected-no-diagnostics + +typedef struct omp_lock_t { + void *_lk; +} omp_lock_t; +typedef struct omp_nest_lock_t { + void *_lk; +} omp_nest_lock_t; +extern void omp_set_lock(omp_lock_t *); +extern void omp_unset_lock(omp_lock_t *); +extern int omp_test_lock(omp_lock_t *); +extern void omp_set_nest_lock(omp_nest_lock_t *); +extern void omp_unset_nest_lock(omp_nest_lock_t *); + +// HOST-NOT: __kmpc_warp_active_thread_mask + +#pragma omp declare target +omp_lock_t L, M, Ls[4]; +omp_nest_lock_t N; +int X; +void use(int); +void with_lock_routine(void (*)(omp_lock_t *), omp_lock_t *); + +// CHECK-LABEL: define {{.*}}void @straight( +// CHECK: %omp.lanes.mask = call i64 @__kmpc_warp_active_thread_mask() +// CHECK: omp.lanes.header: +// CHECK: omp.lanes.body: +// CHECK-NEXT: call void @omp_set_lock(ptr noundef {{.*}}@L{{.*}}) +// CHECK: call void @omp_unset_lock(ptr noundef {{.*}}@L{{.*}}) +// CHECK: omp.lanes.latch: +// CHECK-NEXT: call void @__kmpc_syncwarp(i64 %omp.lanes.mask) +// CHECK: omp.lanes.exit: +// CHECK: call void @use( +void straight(void) { + omp_set_lock(&L); + X++; + omp_unset_lock(&L); + use(X); +} + +// Control flow that stays inside the region, and a lock chosen per thread. +// CHECK-LABEL: define {{.*}}void @nested_flow( +// CHECK: call i64 @__kmpc_warp_active_thread_mask() +// CHECK: omp.lanes.body: +// CHECK: call void @omp_set_lock( +// CHECK: call void @use( +// CHECK: call void @omp_unset_lock( +void nested_flow(int N) { + omp_set_lock(&Ls[N % 4]); + for (int I = 0; I < N; ++I) { + if (I == 3) + break; + use(I); + } + omp_unset_lock(&Ls[N % 4]); +} + +// The lanes are serialized from each omp_set_lock to its own omp_unset_lock. +// CHECK-LABEL: define {{.*}}void @two_regions( +// CHECK: call i64 @__kmpc_warp_active_thread_mask() +// CHECK: call void @omp_set_lock(ptr noundef {{.*}}@L{{.*}}) +// CHECK: call void @omp_unset_lock(ptr noundef {{.*}}@L{{.*}}) +// CHECK: call i64 @__kmpc_warp_active_thread_mask() +// CHECK: call void @omp_set_lock(ptr noundef {{.*}}@M{{.*}}) +// CHECK: call void @omp_unset_lock(ptr noundef {{.*}}@M{{.*}}) +// CHECK: ret void +void two_regions(void) { + omp_set_lock(&L); + X++; + omp_unset_lock(&L); + omp_set_lock(&M); + X++; + omp_unset_lock(&M); +} + +// A pair nested properly inside another is part of the outer sequence. +// CHECK-LABEL: define {{.*}}void @nested_pairs( +// CHECK: call i64 @__kmpc_warp_active_thread_mask() +// CHECK: call void @omp_set_lock(ptr noundef {{.*}}@L{{.*}}) +// CHECK: call void @omp_set_lock(ptr noundef {{.*}}@M{{.*}}) +// CHECK: call void @omp_unset_lock(ptr noundef {{.*}}@M{{.*}}) +// CHECK: call void @omp_unset_lock(ptr noundef {{.*}}@L{{.*}}) +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void nested_pairs(void) { + omp_set_lock(&L); + omp_set_lock(&M); + X++; + omp_unset_lock(&M); + omp_unset_lock(&L); +} + +// A sequence inside a nested block, and a switch inside the sequence. +// CHECK-LABEL: define {{.*}}void @in_block( +// CHECK: call i64 @__kmpc_warp_active_thread_mask() +// CHECK: ret void +void in_block(int C) { + if (C) { + omp_set_lock(&L); + switch (C) { + case 1: + X++; + break; + default: + break; + } + omp_unset_lock(&L); + } +} + +// The following are left alone. + +// CHECK-LABEL: define {{.*}}void @mismatched( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void mismatched(void) { + omp_set_lock(&L); + X++; + omp_unset_lock(&M); +} + +// CHECK-LABEL: define {{.*}}void @early_return( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void early_return(int C) { + omp_set_lock(&L); + if (C) + return; + omp_unset_lock(&L); +} + +// CHECK-LABEL: define {{.*}}void @escaping_break( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void escaping_break(int C) { + for (;;) { + omp_set_lock(&L); + if (C) + break; + omp_unset_lock(&L); + } +} + +// CHECK-LABEL: define {{.*}}void @declaration( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void declaration(void) { + omp_set_lock(&L); + int Y = X; + omp_unset_lock(&L); + use(Y); +} + +// CHECK-LABEL: define {{.*}}void @directive( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void directive(void) { + omp_set_lock(&L); +#pragma omp barrier + omp_unset_lock(&L); +} + +// CHECK-LABEL: define {{.*}}void @side_effect( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void side_effect(int N) { + omp_set_lock(&Ls[N++]); + omp_unset_lock(&Ls[N++]); +} + +// The unset does not follow the set on every path. + +// CHECK-LABEL: define {{.*}}void @set_only( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void set_only(void) { + omp_set_lock(&L); + X++; +} + +// CHECK-LABEL: define {{.*}}void @conditional_unset( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void conditional_unset(int C) { + omp_set_lock(&L); + X++; + if (C) + omp_unset_lock(&L); +} + +// CHECK-LABEL: define {{.*}}void @unset_in_both_branches( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void unset_in_both_branches(int C) { + omp_set_lock(&L); + if (C) { + X++; + omp_unset_lock(&L); + } else { + omp_unset_lock(&L); + } +} + +// CHECK-LABEL: define {{.*}}void @unset_before_set( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void unset_before_set(void) { + omp_unset_lock(&L); + X++; + omp_set_lock(&L); +} + +// CHECK-LABEL: define {{.*}}void @escaping_continue( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void escaping_continue(int C) { + for (int I = 0; I < C; ++I) { + omp_set_lock(&L); + if (I == 3) + continue; + omp_unset_lock(&L); + } +} + +// CHECK-LABEL: define {{.*}}void @goto_out( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void goto_out(int C) { + omp_set_lock(&L); + if (C) + goto out; + omp_unset_lock(&L); +out: + X++; +} + +// CHECK-LABEL: define {{.*}}void @label_inside( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void label_inside(int C) { + if (C) + goto in; + omp_set_lock(&L); +in: + X++; + omp_unset_lock(&L); +} + +// CHECK-LABEL: define {{.*}}void @case_label_inside( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void case_label_inside(int C) { + switch (C) { + case 0: + omp_set_lock(&L); + case 1: + X++; + omp_unset_lock(&L); + } +} + +// A lock could be held from one lane's turn to the next. + +// CHECK-LABEL: define {{.*}}void @overlapping_pairs( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void overlapping_pairs(void) { + omp_set_lock(&L); + omp_set_lock(&M); + X++; + omp_unset_lock(&L); + omp_unset_lock(&M); +} + +// The first omp_set_lock stays outside; the pair that follows it is serialized. +// CHECK-LABEL: define {{.*}}void @relocked( +// CHECK: call void @omp_set_lock(ptr noundef {{.*}}@L{{.*}}) +// CHECK-NEXT: %omp.lanes.mask = call i64 @__kmpc_warp_active_thread_mask() +// CHECK: ret void +void relocked(void) { + omp_set_lock(&L); + omp_set_lock(&L); + omp_unset_lock(&L); +} + +// CHECK-LABEL: define {{.*}}void @unset_and_reset_conditionally( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void unset_and_reset_conditionally(int C) { + omp_set_lock(&L); + if (C) { + omp_unset_lock(&L); + omp_set_lock(&L); + } + omp_unset_lock(&L); +} + +// CHECK-LABEL: define {{.*}}void @test_lock_inside( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void test_lock_inside(void) { + omp_set_lock(&L); + if (omp_test_lock(&M)) + X++; + omp_unset_lock(&L); +} + +// CHECK-LABEL: define {{.*}}void @nest_lock_inside( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void nest_lock_inside(void) { + omp_set_lock(&L); + omp_set_nest_lock(&N); + omp_unset_nest_lock(&N); + omp_unset_lock(&L); +} + +// CHECK-LABEL: define {{.*}}void @lock_routine_as_value( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void +void lock_routine_as_value(void) { + omp_set_lock(&L); + with_lock_routine(omp_set_lock, &M); + omp_unset_lock(&L); +} +#pragma omp end declare target diff --git a/clang/test/OpenMP/amdgcn_lock_lane_serialization.cpp b/clang/test/OpenMP/amdgcn_lock_lane_serialization.cpp new file mode 100644 index 00000000000000..204781d3efffcc --- /dev/null +++ b/clang/test/OpenMP/amdgcn_lock_lane_serialization.cpp @@ -0,0 +1,158 @@ +// REQUIRES: amdgpu-registered-target + +// omp_set_lock ... omp_unset_lock sequences in C++ on AMDGPU: in templates, +// where the decision is made for each instantiation, and alongside lambdas, +// classes and exceptions. + +// RUN: %clang_cc1 -fopenmp -x c++ -std=c++17 -triple x86_64-unknown-unknown -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm-bc %s -o %t-host.bc +// RUN: %clang_cc1 -fopenmp -x c++ -std=c++17 -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o - | FileCheck %s --implicit-check-not=invoke --implicit-check-not=landingpad +// RUN: %clang_cc1 -fopenmp -x c++ -std=c++17 -fexceptions -fcxx-exceptions -triple x86_64-unknown-unknown -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm-bc %s -o %t-host-eh.bc +// RUN: %clang_cc1 -fopenmp -x c++ -std=c++17 -fexceptions -fcxx-exceptions -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-eh.bc -o - | FileCheck %s --implicit-check-not=invoke --implicit-check-not=landingpad +// expected-no-diagnostics + +struct omp_lock_t { + void *_lk; +}; +extern "C" void omp_set_lock(omp_lock_t *); +extern "C" void omp_unset_lock(omp_lock_t *); + +// Not the OpenMP routines. +namespace mine { +void omp_set_lock(omp_lock_t *); +void omp_unset_lock(omp_lock_t *); +} // namespace mine + +void may_throw(); + +// Exceptions are not supported in device code, so with or without +// -fexceptions no call in a sequence can leave it by unwinding: there is no +// invoke or landingpad anywhere in the output. + +#pragma omp declare target +omp_lock_t L, Ls[4]; +int X; + +template <typename T> void bump(T &V) { + omp_set_lock(&L); + V += 1; + may_throw(); + omp_unset_lock(&L); +} + +template <int I> void bump_indexed() { + omp_set_lock(&Ls[I]); + X++; + omp_unset_lock(&Ls[I]); +} + +template <int I, int J> void bump_pair() { + omp_set_lock(&Ls[I]); + X++; + omp_unset_lock(&Ls[J]); +} + +template <bool B> void unset_if() { + omp_set_lock(&L); + X++; + if constexpr (B) + omp_unset_lock(&L); +} + +template <typename T> void return_if(T C) { + omp_set_lock(&L); + if (C) + return; + omp_unset_lock(&L); +} + +template <typename T> struct Guarded { + omp_lock_t Lock; + T Val; + void bump() { + omp_set_lock(&Lock); + ++Val; + omp_unset_lock(&Lock); + } +}; + +void with_lambda(int C) { + omp_set_lock(&L); + [&] { + if (C) + return; + X++; + }(); + omp_unset_lock(&L); +} + +void not_openmp() { + mine::omp_set_lock(&L); + X++; + mine::omp_unset_lock(&L); +} + +void instantiate(int &I, double &D, Guarded<int> &G) { + bump(I); + bump(D); + bump_indexed<2>(); + bump_pair<1, 1>(); + bump_pair<1, 2>(); + unset_if<true>(); + unset_if<false>(); + return_if(I); + G.bump(); +} +#pragma omp end declare target + +// A lambda's return leaves only the lambda. +// CHECK-LABEL: define {{.*}}void @_Z11with_lambdai( +// CHECK: call i64 @__kmpc_warp_active_thread_mask() +// CHECK: ret void + +// CHECK-LABEL: define {{.*}}void @_Z10not_openmpv( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void + +// Each instantiation is considered on its own. +// CHECK-LABEL: define {{.*}}void @_Z4bumpIiEvRT_( +// CHECK: call i64 @__kmpc_warp_active_thread_mask() +// CHECK: omp.lanes.body: +// CHECK-NEXT: call void @omp_set_lock( +// CHECK: call void @_Z9may_throwv() +// CHECK: call void @omp_unset_lock( +// CHECK: ret void + +// CHECK-LABEL: define {{.*}}void @_Z4bumpIdEvRT_( +// CHECK: call i64 @__kmpc_warp_active_thread_mask() +// CHECK: ret void + +// CHECK-LABEL: define {{.*}}void @_Z12bump_indexedILi2EEvv( +// CHECK: call i64 @__kmpc_warp_active_thread_mask() +// CHECK: ret void + +// The two lock expressions are the same once I and J are substituted... +// CHECK-LABEL: define {{.*}}void @_Z9bump_pairILi1ELi1EEvv( +// CHECK: call i64 @__kmpc_warp_active_thread_mask() +// CHECK: ret void + +// ...but not here. +// CHECK-LABEL: define {{.*}}void @_Z9bump_pairILi1ELi2EEvv( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void + +// The unset is conditional in the template, whichever branch survives. +// CHECK-LABEL: define {{.*}}void @_Z8unset_ifILb1EEvv( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void + +// CHECK-LABEL: define {{.*}}void @_Z8unset_ifILb0EEvv( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void + +// CHECK-LABEL: define {{.*}}void @_Z9return_ifIiEvT_( +// CHECK-NOT: __kmpc_warp_active_thread_mask +// CHECK: ret void + +// CHECK-LABEL: define {{.*}}void @_ZN7GuardedIiE4bumpEv( +// CHECK: call i64 @__kmpc_warp_active_thread_mask() +// CHECK: ret void diff --git a/clang/test/OpenMP/target_lock_lane_warning.cpp b/clang/test/OpenMP/target_lock_lane_warning.cpp new file mode 100644 index 00000000000000..68c1c1d51a2020 --- /dev/null +++ b/clang/test/OpenMP/target_lock_lane_warning.cpp @@ -0,0 +1,233 @@ +// REQUIRES: amdgpu-registered-target + +// On a GPU, whose threads of a warp may run in lockstep, warn about an +// omp_set_lock that several threads of a warp may reach together but that +// cannot be serialized across them, and say why. + +// RUN: %clang_cc1 -fopenmp -x c++ -triple x86_64-unknown-unknown -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm-bc %s -o %t-host.bc -verify=host +// RUN: %clang_cc1 -fopenmp -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o /dev/null -verify +// RUN: %clang_cc1 -fopenmp -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -fsyntax-only %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -verify +// RUN: %clang_cc1 -fopenmp -x c++ -triple amdgcn-amd-amdhsa -fopenmp-targets=amdgcn-amd-amdhsa -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host.bc -o /dev/null -Wno-openmp-lockstep-lock -verify=quiet +// RUN: %clang_cc1 -fopenmp -x c++ -triple x86_64-unknown-unknown -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm-bc %s -o %t-host-nv.bc +// RUN: %clang_cc1 -fopenmp -x c++ -triple nvptx64-nvidia-cuda -fopenmp-targets=nvptx64-nvidia-cuda -emit-llvm %s -fopenmp-is-target-device -fopenmp-host-ir-file-path %t-host-nv.bc -o /dev/null -verify + +// host-no-diagnostics +// quiet-no-diagnostics + +struct omp_lock_t { + void *_lk; +}; +extern "C" void omp_set_lock(omp_lock_t *); +extern "C" void omp_unset_lock(omp_lock_t *); + +#pragma omp declare target +omp_lock_t L, M, Ls[4]; +int X; +void use(int); +// The callers are not known, so there is nothing to say. +void acquire() { omp_set_lock(&L); } +#pragma omp end declare target + +void reasons(int C, int N) { + // expected-note@+1 7 {{threads of a warp may execute this '#pragma omp target parallel' region together; use 'single', 'masked' or 'critical' to run code on one thread at a time}} +#pragma omp target parallel + { + // Each case is in a block of its own, since the search for the matching + // omp_unset_lock stops at the end of the block. + { + omp_set_lock(&L); // expected-warning {{because no matching 'omp_unset_lock' follows it in the same block; threads of a warp that contend for the lock will trap or hang}} + X++; + } + { + omp_set_lock(&M); // expected-warning {{because no matching 'omp_unset_lock' follows it in the same block}} + X++; + } + for (int I = 0; I < N; ++I) { + omp_set_lock(&L); // expected-warning {{because control can enter or leave the locked code other than at its ends}} + if (I == 3) + continue; + omp_unset_lock(&L); + } + { + omp_set_lock(&L); // expected-warning {{because the locked code uses a lock routine other than in properly nested set/unset pairs}} + X++; + if (C) + omp_unset_lock(&L); + } + { + omp_set_lock(&L); // expected-warning {{because the locked code contains an OpenMP directive}} +#pragma omp barrier + omp_unset_lock(&L); + } + { + omp_set_lock(&L); // expected-warning {{because the locked code declares a variable}} + int Y = X; + omp_unset_lock(&L); + use(Y); + } + { + omp_set_lock(&Ls[N++]); // expected-warning {{because the lock expression has side effects}} + omp_unset_lock(&Ls[N]); + } + } +} + +void contexts(int C) { + // A worksharing loop inside a combined construct. + // expected-note@+1 {{this '#pragma omp target teams distribute parallel for' region}} +#pragma omp target teams distribute parallel for + for (int I = 0; I < 100; ++I) { + omp_set_lock(&L); // expected-warning {{because the locked code uses a lock routine}} + if (C) + omp_unset_lock(&L); + } + + // Nothing to say where one thread at a time runs the code, or where it is + // serialized. +#pragma omp target parallel + { + omp_set_lock(&L); + X++; + omp_unset_lock(&L); + +#pragma omp single + { + omp_set_lock(&L); + if (C) + omp_unset_lock(&L); + } +#pragma omp masked + omp_set_lock(&M); +#pragma omp critical + { + omp_set_lock(&M); + } + } + + // The sequential part of a target region, and one thread per team. +#pragma omp target + { + omp_set_lock(&L); + if (C) + omp_unset_lock(&L); + } +#pragma omp target teams distribute + for (int I = 0; I < 100; ++I) { + omp_set_lock(&L); + if (C) + omp_unset_lock(&L); + } +} + +void one_thread(int C) { + // The masked part of a combined construct runs the code on one thread. +#pragma omp target +#pragma omp parallel masked + { + omp_set_lock(&L); + if (C) + omp_unset_lock(&L); + } + + // So does each section, explicit or implicit, of a sections region. +#pragma omp target +#pragma omp parallel sections + { + { + omp_set_lock(&L); + if (C) + omp_unset_lock(&L); + } +#pragma omp section + { + omp_set_lock(&L); + if (C) + omp_unset_lock(&L); + } + } +} + +void statement_forms(int C) { + // expected-note@+1 2 {{this '#pragma omp target parallel' region}} +#pragma omp target parallel + { + switch (C) { + case 1: + omp_set_lock(&L); // expected-warning {{because the call is labelled; put the locked code in a block of its own}} + X++; + omp_unset_lock(&L); + break; + case 2: { + omp_set_lock(&L); + X++; + omp_unset_lock(&L); + break; + } + } + if (C) { + omp_set_lock(&M); + omp_unset_lock(&M); + } + if (C) + omp_set_lock(&M); // expected-warning {{because the call is not a statement of its own in a block}} + } + + // A construct whose body is not a block. + // expected-note@+1 {{this '#pragma omp target parallel' region}} +#pragma omp target parallel + omp_set_lock(&L); // expected-warning {{because the call is not a statement of its own in a block}} + + // The callers of a lambda are not known. +#pragma omp target parallel + { + auto F = [&] { + omp_set_lock(&L); + if (C) + omp_unset_lock(&L); + }; + F(); + } +} + +// A function reached from a target region is emitted for the device; one only +// called on the host is not. +void device_helper(int C) { + // expected-note@+1 {{this '#pragma omp parallel' region}} +#pragma omp parallel + { + omp_set_lock(&L); // expected-warning {{because the locked code uses a lock routine}} + if (C) + omp_unset_lock(&L); + } +} + +void host_helper(int C) { +#pragma omp parallel + { + omp_set_lock(&L); + if (C) + omp_unset_lock(&L); + } +} + +void callers(int C) { +#pragma omp target + device_helper(C); // expected-note {{called by 'callers'}} + host_helper(C); +} + +// One warning however many instantiations there are. +template <int N> void instantiated(int C) { + // expected-note@+1 {{this '#pragma omp target parallel' region}} +#pragma omp target parallel + { + omp_set_lock(&Ls[N]); // expected-warning {{because the locked code uses a lock routine}} + if (C) + omp_unset_lock(&Ls[N]); + } +} + +void instantiate(int C) { + instantiated<1>(C); // expected-note {{in instantiation of function template specialization 'instantiated<1>' requested here}} + instantiated<2>(C); +} diff --git a/offload/test/offloading/target_lock_lanes.c b/offload/test/offloading/target_lock_lanes.c new file mode 100644 index 00000000000000..9bf9e496e7606f --- /dev/null +++ b/offload/test/offloading/target_lock_lanes.c @@ -0,0 +1,115 @@ +// RUN: %libomptarget-compile-run-and-check-generic +// RUN: %libomptarget-compileopt-run-and-check-generic + +// REQUIRES: amdgpu + +// Every thread of a team takes the same lock, including threads that share a +// wavefront. The compiler serializes the lanes of a wavefront across each +// omp_set_lock ... omp_unset_lock sequence below, since a lane that holds the +// lock could not otherwise release it while another lane of the same wavefront +// spins for it. +// +// Locks may only be shared within a contention group, so each lock is created, +// used and destroyed by the threads of a single team. + +#include <omp.h> +#include <stdio.h> + +#define TEAMS 4 + +#pragma omp declare target +omp_lock_t Lock; +omp_lock_t Locks[4]; +omp_lock_t TeamLocks[TEAMS]; + +void bump(volatile int *P) { *P = *P + 1; } +#pragma omp end declare target + +int main() { + int Count = 0, Calls = 0, Shared = 0, Expected = 0, Even = 0; + int Hist[4] = {0, 0, 0, 0}; + +#pragma omp target parallel num_threads(256) \ + map(tofrom : Count, Calls, Shared, Expected, Even, Hist) + { +#pragma omp single + { + omp_init_lock(&Lock); + for (int I = 0; I < 4; ++I) + omp_init_lock(&Locks[I]); + } + + int Id = omp_get_thread_num(); + volatile int *C = &Count; + + // Straight-line code. + omp_set_lock(&Lock); + *C = *C + 1; + omp_unset_lock(&Lock); + + // A function call and nested control flow inside the region. + omp_set_lock(&Lock); + if (Id % 2 == 0) + bump(&Calls); + for (int I = 0; I < 2; ++I) + bump(&Shared); + omp_unset_lock(&Lock); + + // A lock picked per thread; lanes of a wavefront want different locks. + omp_set_lock(&Locks[Id % 4]); + bump(&Hist[Id % 4]); + omp_unset_lock(&Locks[Id % 4]); + +#pragma omp atomic + Expected += 1; + if (Id % 2 == 0) { +#pragma omp atomic + Even += 1; + } + +#pragma omp barrier +#pragma omp single + { + omp_destroy_lock(&Lock); + for (int I = 0; I < 4; ++I) + omp_destroy_lock(&Locks[I]); + } + } + + // Several teams at once, each with a lock of its own. + int TeamCount[TEAMS] = {0}, TeamThreads[TEAMS] = {0}; +#pragma omp target teams num_teams(TEAMS) map(tofrom : TeamCount, TeamThreads) +#pragma omp parallel num_threads(128) + { + int T = omp_get_team_num(); +#pragma omp single + omp_init_lock(&TeamLocks[T]); + + omp_set_lock(&TeamLocks[T]); + bump(&TeamCount[T]); + omp_unset_lock(&TeamLocks[T]); + +#pragma omp atomic + TeamThreads[T] += 1; + +#pragma omp barrier +#pragma omp single + omp_destroy_lock(&TeamLocks[T]); + } + int TeamsOK = 1; + for (int T = 0; T < TEAMS; ++T) + TeamsOK &= TeamCount[T] == TeamThreads[T]; + + // CHECK: Count OK + printf("Count %s\n", Count == Expected && Count > 0 ? "OK" : "BAD"); + // CHECK: Calls OK + printf("Calls %s\n", Calls == Even ? "OK" : "BAD"); + // CHECK: Shared OK + printf("Shared %s\n", Shared == 2 * Expected ? "OK" : "BAD"); + // CHECK: Hist OK + printf("Hist %s\n", + Hist[0] + Hist[1] + Hist[2] + Hist[3] == Expected ? "OK" : "BAD"); + // CHECK: Teams OK + printf("Teams %s\n", TeamsOK ? "OK" : "BAD"); + return 0; +} _______________________________________________ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
