llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-llvm-transforms
Author: Tomas Matheson (tommat01)
<details>
<summary>Changes</summary>
The deferecenceable at-point semanatics change has caused a regression on
AArch64 for an openmp parallel region that looks like this:
```cpp
#include <omp.h>
double sum_if(double *values, const bool *enabled, int count) {
double sum = 0.0;
#pragma omp parallel reduction(+ : sum)
{
int threads = omp_get_num_threads();
int thread = omp_get_thread_num();
for (int i = thread * count / threads; i < (thread + 1) * count /
threads; ++i)
if (enabled[i])
sum += values[i];
}
return sum;
}
```
This parallel region is offloaded by clang (in the AST) and looks something
like this:
```cpp
#include <omp.h>
double sum_if(double *values, const bool *enabled, int count) {
double sum = 0.0;
magic threading {
sum_if_omp_outlined(count, enabled, sum, values);
}
return sum;
}
void sum_if_omp_outlined(int& count, bool*& enabled, double& sum,
double*& values) {
int threads = omp_get_num_threads();
int thread = omp_get_thread_num();
for (int i = thread * count / threads; i < (thread + 1) * count / threads;
++i)
if (enabled[i])
sum += values[i];
}
```
The captured values (count, enabled, sum, values) are stored on the stack and
references to them are passed to the outlined function via the broker
`__kmpc_fork_call`. Something like:
```cpp
int count_slot = count;
bool*& enabled_slot = enabled;
double& sum_slot = sum;
double*& values_slot = values;
__kmpc_fork_call(..., &sum_if_omp_outlined, count_slot, enabled_slot,
sum_slot, values_slot);
```
In the outlined function, these pointers to capture slots are marked
`dereferenceable`. But unfortunately, with the new at-point semantics, the
OpenMP runtime calls prevent this property from being applied at the use site
(anything after the `omp_*` calls). These functions are not marked `nofree` and
therefore they could potentially free the captured pointers (although that
seems unlikely).
*This prevents the dereferencing of the slot-pointer `values_slot` to the slot
value inside the loop*, and therefore prevents that dereference from being
hoisted outside the loop. This in turn prevents the loop from being vectorized
etc. Here is the IR for the outlined function:
```llvm
define internal void @<!-- -->sum_if.omp_outlined(
ptr noalias noundef %.global_tid.,
ptr noalias noundef %.bound_tid.,
ptr noundef nonnull align 4 dereferenceable(4) %count_cap,
ptr noundef nonnull align 8 dereferenceable(8) %enabled_cap,
ptr noundef nonnull align 8 dereferenceable(8) %sum_cap,
# In the outlined function, this is a pointer to values (double** values).
# At this point it is dereferencable.
ptr noundef nonnull align 8 dereferenceable(8) %values_cap) {
entry:
# call the openmp functions. At this point, %values_cap is no longer
dereferencable.
%call = invoke i32 @<!-- -->omp_get_num_threads()
%call3 = invoke i32 @<!-- -->omp_get_thread_num()
for.body:
# load the if-condition from another captured variable enabled[i]
# This matters because it makes the load of values[i2] speculative.
# Without `dereferencable` on %values_cap, whether it is dereferenceable or
not
# might depend on the value in enabled[i].
%condition_byte = load i8, ptr %enabled_offset
%condition = trunc i8 %condition_byte to i1
br i1 %condition, label %if.then, label %if.end
if.then:
# If successful, then load values[i].
# %values_cap is reloaded every time.
# If it is dereferencable, it will be hoisted out of the loop.
%base_ptr = load ptr, ptr %values_cap, align 8 # <----------
%arrayidx7 = getelementptr inbounds double, ptr %base_ptr, i64 %idxprom6
# Load the actual value and update sum.
%value = load double, ptr %arrayidx7, align 8, !tbaa !23
%old_sum1 = load double, ptr %sum1, align 8, !tbaa !23
%add8 = fadd fast double %old_sum1, %value
store double %add8, ptr %sum1, align 8, !tbaa !23
br label %if.end
}
```
There are a number of ways that we could fix this. A couple that I considered:
1. Annotate the `omp_` function calls with appropriate attributes like
`nofree`. This can already be done for clang-generated runtime calls by adding
the flag `-mllvm -openmp-ir-builder-optimistic-attributes` but not for
user-written calls like these ones. We could extend it to cover user calls too,
but it is not clear how widely used that flag is, which openmp implementations
it is valid for, and it does not solve the problem for arbitrary `opaque()`
external function calls that the user might make. We might want to extend the
flag to do this anyway.
2. When clang is outlining the parallel region, we could re-materialise the
values from the capture slot pointers at the top of the function and then
replace all dereferences of these pointers with the value stored in the slot.
This avoids the problem of whether they are dereferenceable by removing the
pointer entirely. It requires some non-trivial analysis to determine if the
pointer is aliased or the value is mutated etc. It doesn't seem like the AST is
the right place for this, but it does work.
3. Allow the OpenMP Attributor to do the interprocedural analysis necessary to
see across the `__kmpc_fork_call` and outlined function to determine that these
capture slot pointers are not aliased and are therefore safely dereferenceable.
This is the approach I've taken here.
*The main change here is to seed `AANoAlias` for OpenMP for callbacks on the
host.* This allows it to derive `noalias`, `capture(...)` etc for the slot
pointers which is sufficient to allow the original dereference of the slot
pointer to be hoisted again.
There were a few other minor issues that required fixing to get this to work
for the specific benchmark code in question:
- The `posix_memalign` libcall was missing a `captures(none)` annotation for
the first argument.
- The Attributor was the callee rather than the call-site incorrectly.
These are contained in separate commits; I can split out to separate PRs if
necessary.
---
Patch is 54.70 KiB, truncated to 20.00 KiB below, full version:
https://github.com/llvm/llvm-project/pull/218453.diff
13 Files Affected:
- (modified) clang/test/OpenMP/bug54082.c (+9-11)
- (added) clang/test/OpenMP/callback_capture_lifetime.cpp (+41)
- (added) flang/test/Integration/OpenMP/callback-capture-lifetime.f90 (+48)
- (modified) llvm/lib/Transforms/IPO/AttributorAttributes.cpp (+1-1)
- (modified) llvm/lib/Transforms/IPO/OpenMPOpt.cpp (+19-2)
- (modified) llvm/lib/Transforms/Utils/BuildLibCalls.cpp (+3)
- (added) llvm/test/Transforms/Attributor/callback-noalias.ll (+33)
- (modified) llvm/test/Transforms/Attributor/callbacks.ll (+28-26)
- (modified) llvm/test/Transforms/InferFunctionAttrs/annotate.ll (+1-1)
- (added) llvm/test/Transforms/OpenMP/callback-capture-lifetime.ll (+262)
- (added) llvm/test/Transforms/OpenMP/callback-capture-posix-memalign.ll (+63)
- (modified) llvm/test/Transforms/OpenMP/parallel_deletion.ll (+10-10)
- (modified) llvm/test/Transforms/OpenMP/parallel_region_merging.ll (+6-6)
``````````diff
diff --git a/clang/test/OpenMP/bug54082.c b/clang/test/OpenMP/bug54082.c
index 6b8c93e9ffc96..b317a88d8aacb 100644
--- a/clang/test/OpenMP/bug54082.c
+++ b/clang/test/OpenMP/bug54082.c
@@ -69,43 +69,41 @@ void foo() {
// CHECK-NEXT: [[X_TRAITS:%.*]] = alloca [1 x
[[STRUCT_OMP_ALLOCTRAIT_T:%.*]]], align 16
// CHECK-NEXT: [[X_ALLOC:%.*]] = alloca i64, align 8
// CHECK-NEXT: call void @llvm.lifetime.start.p0(ptr nonnull [[X_TRAITS]])
#[[ATTR4:[0-9]+]]
-// CHECK-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr noundef nonnull align
16 dereferenceable(16) [[X_TRAITS]], ptr noundef nonnull align 16
dereferenceable(16) @__const.foo.x_traits, i64 16, i1 false)
+// CHECK-NEXT: call void @llvm.memcpy.p0.p0.i64(ptr noundef nonnull
writeonly align 16 dereferenceable(16) [[X_TRAITS]], ptr noundef nonnull
readonly align 16 dereferenceable(16) @__const.foo.x_traits, i64 16, i1 false)
// CHECK-NEXT: call void @llvm.lifetime.start.p0(ptr nonnull [[X_ALLOC]])
#[[ATTR4]]
// CHECK-NEXT: [[CALL:%.*]] = call i64 @omp_init_allocator(i64 noundef 0,
i32 noundef 1, ptr noundef nonnull [[X_TRAITS]]) #[[ATTR4]]
// CHECK-NEXT: store i64 [[CALL]], ptr [[X_ALLOC]], align 8, !tbaa
[[LONG_TBAA7:![0-9]+]]
-// CHECK-NEXT: call void (ptr, i32, ptr, ...) @__kmpc_fork_call(ptr nonnull
@[[GLOB2:[0-9]+]], i32 1, ptr nonnull @foo.omp_outlined, ptr nonnull
[[X_ALLOC]])
+// CHECK-NEXT: call void (ptr, i32, ptr, ...) @__kmpc_fork_call(ptr nonnull
@[[GLOB2:[0-9]+]], i32 1, ptr nonnull @foo.omp_outlined, ptr noalias nonnull
readonly captures(none) [[X_ALLOC]])
// CHECK-NEXT: call void @llvm.lifetime.end.p0(ptr nonnull [[X_ALLOC]])
#[[ATTR4]]
// CHECK-NEXT: call void @llvm.lifetime.end.p0(ptr nonnull [[X_TRAITS]])
#[[ATTR4]]
// CHECK-NEXT: ret void
//
//
// CHECK-LABEL: define internal void @foo.omp_outlined(
-// CHECK-SAME: ptr noalias nofree noundef readonly captures(none)
[[DOTGLOBAL_TID_:%.*]], ptr noalias nofree readnone captures(none)
[[DOTBOUND_TID_:%.*]], ptr nofree noundef nonnull readonly align 8
captures(none) dereferenceable(8) [[X_ALLOC:%.*]]) #[[ATTR3:[0-9]+]] {
+// CHECK-SAME: ptr noalias nofree noundef readonly captures(none)
[[DOTGLOBAL_TID_:%.*]], ptr noalias nofree readnone captures(none)
[[DOTBOUND_TID_:%.*]], ptr noalias noundef nonnull readonly align 8
captures(none) dereferenceable(8) [[X_ALLOC:%.*]]) #[[ATTR3:[0-9]+]] {
// CHECK-NEXT: [[ENTRY:.*:]]
// CHECK-NEXT: [[DOTOMP_LB:%.*]] = alloca i32, align 4
// CHECK-NEXT: [[DOTOMP_UB:%.*]] = alloca i32, align 4
// CHECK-NEXT: [[DOTOMP_STRIDE:%.*]] = alloca i32, align 4
// CHECK-NEXT: [[DOTOMP_IS_LAST:%.*]] = alloca i32, align 4
-// CHECK-NEXT: call void @llvm.lifetime.start.p0(ptr nonnull [[DOTOMP_LB]])
#[[ATTR4]]
+// CHECK-NEXT: call void @llvm.lifetime.start.p0(ptr nonnull [[DOTOMP_LB]])
#[[ATTR6:[0-9]+]]
// CHECK-NEXT: store i32 0, ptr [[DOTOMP_LB]], align 4, !tbaa
[[INT_TBAA9:![0-9]+]]
-// CHECK-NEXT: call void @llvm.lifetime.start.p0(ptr nonnull [[DOTOMP_UB]])
#[[ATTR4]]
+// CHECK-NEXT: call void @llvm.lifetime.start.p0(ptr nonnull [[DOTOMP_UB]])
#[[ATTR6]]
// CHECK-NEXT: store i32 1023, ptr [[DOTOMP_UB]], align 4, !tbaa
[[INT_TBAA9]]
-// CHECK-NEXT: call void @llvm.lifetime.start.p0(ptr nonnull
[[DOTOMP_STRIDE]]) #[[ATTR4]]
+// CHECK-NEXT: call void @llvm.lifetime.start.p0(ptr nonnull
[[DOTOMP_STRIDE]]) #[[ATTR6]]
// CHECK-NEXT: store i32 1, ptr [[DOTOMP_STRIDE]], align 4, !tbaa
[[INT_TBAA9]]
-// CHECK-NEXT: call void @llvm.lifetime.start.p0(ptr nonnull
[[DOTOMP_IS_LAST]]) #[[ATTR4]]
+// CHECK-NEXT: call void @llvm.lifetime.start.p0(ptr nonnull
[[DOTOMP_IS_LAST]]) #[[ATTR6]]
// CHECK-NEXT: store i32 0, ptr [[DOTOMP_IS_LAST]], align 4, !tbaa
[[INT_TBAA9]]
// CHECK-NEXT: [[TMP0:%.*]] = load i32, ptr [[DOTGLOBAL_TID_]], align 4,
!tbaa [[INT_TBAA9]]
// CHECK-NEXT: [[TMP1:%.*]] = load i64, ptr [[X_ALLOC]], align 8, !tbaa
[[LONG_TBAA7]]
// CHECK-NEXT: [[CONV:%.*]] = inttoptr i64 [[TMP1]] to ptr
-// CHECK-NEXT: [[DOTX__VOID_ADDR:%.*]] = tail call ptr @__kmpc_alloc(i32
[[TMP0]], i64 8, ptr [[CONV]])
+// CHECK-NEXT: [[DOTX__VOID_ADDR:%.*]] = tail call ptr @__kmpc_alloc(i32
[[TMP0]], i64 8, ptr [[CONV]]) #[[ATTR4]]
// CHECK-NEXT: call void @__kmpc_for_static_init_4(ptr nonnull
@[[GLOB1:[0-9]+]], i32 [[TMP0]], i32 34, ptr nonnull [[DOTOMP_IS_LAST]], ptr
nonnull [[DOTOMP_LB]], ptr nonnull [[DOTOMP_UB]], ptr nonnull
[[DOTOMP_STRIDE]], i32 1, i32 1)
// CHECK-NEXT: [[TMP2:%.*]] = load i32, ptr [[DOTOMP_UB]], align 4, !tbaa
[[INT_TBAA9]]
// CHECK-NEXT: [[COND:%.*]] = call i32 @llvm.smin.i32(i32 [[TMP2]], i32
1023)
// CHECK-NEXT: store i32 [[COND]], ptr [[DOTOMP_UB]], align 4, !tbaa
[[INT_TBAA9]]
// CHECK-NEXT: call void @__kmpc_for_static_fini(ptr nonnull @[[GLOB1]],
i32 [[TMP0]])
-// CHECK-NEXT: [[TMP3:%.*]] = load i64, ptr [[X_ALLOC]], align 8, !tbaa
[[LONG_TBAA7]]
-// CHECK-NEXT: [[CONV5:%.*]] = inttoptr i64 [[TMP3]] to ptr
-// CHECK-NEXT: call void @__kmpc_free(i32 [[TMP0]], ptr
[[DOTX__VOID_ADDR]], ptr [[CONV5]])
+// CHECK-NEXT: call void @__kmpc_free(i32 [[TMP0]], ptr
[[DOTX__VOID_ADDR]], ptr [[CONV]])
// CHECK-NEXT: call void @llvm.lifetime.end.p0(ptr nonnull
[[DOTOMP_IS_LAST]]) #[[ATTR4]]
// CHECK-NEXT: call void @llvm.lifetime.end.p0(ptr nonnull
[[DOTOMP_STRIDE]]) #[[ATTR4]]
// CHECK-NEXT: call void @llvm.lifetime.end.p0(ptr nonnull [[DOTOMP_UB]])
#[[ATTR4]]
diff --git a/clang/test/OpenMP/callback_capture_lifetime.cpp
b/clang/test/OpenMP/callback_capture_lifetime.cpp
new file mode 100644
index 0000000000000..c4e1195c550c7
--- /dev/null
+++ b/clang/test/OpenMP/callback_capture_lifetime.cpp
@@ -0,0 +1,41 @@
+// Verify that OpenMP callback analysis proves the captured pointer containers
+// noalias, allowing their loads to be hoisted out of the loop across an
+// unrelated opaque call.
+//
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -O2 -ffast-math -fopenmp \
+// RUN: -x c++ -emit-llvm -o - %s \
+// RUN: | FileCheck %s
+
+extern "C" int omp_get_num_threads();
+extern "C" int omp_get_thread_num();
+extern "C" void opaque();
+
+double sum_if(double *values, const bool *enabled, int count) {
+ double sum = 0.0;
+#pragma omp parallel reduction(+ : sum)
+ {
+ int threads = omp_get_num_threads();
+ int thread = omp_get_thread_num();
+
+ for (int i = thread * count / threads;
+ i < (thread + 1) * count / threads; ++i) {
+ opaque();
+ if (enabled[i])
+ sum += values[i];
+ }
+ }
+ return sum;
+}
+
+// CHECK-LABEL: define internal void @_Z6sum_ifPdPKbi.omp_outlined(
+// CHECK-SAME: ptr noalias noundef nonnull readonly align 8 captures(none)
dereferenceable(8) %enabled,
+// CHECK-SAME: {{.*}}ptr noalias noundef nonnull readonly align 8
captures(none) dereferenceable(8) %values)
+// CHECK: call i32 @omp_get_num_threads()
+// CHECK: call i32 @omp_get_thread_num()
+// CHECK: for.body.lr.ph:
+// CHECK: [[VALUES:%.*]] = load ptr, ptr %values
+// CHECK: for.body:
+// CHECK-NOT: load ptr, ptr %values
+// CHECK: call void @opaque()
+// CHECK: if.then:
+// CHECK: getelementptr {{.*}}, ptr [[VALUES]],
diff --git a/flang/test/Integration/OpenMP/callback-capture-lifetime.f90
b/flang/test/Integration/OpenMP/callback-capture-lifetime.f90
new file mode 100644
index 0000000000000..742cc8ca7f765
--- /dev/null
+++ b/flang/test/Integration/OpenMP/callback-capture-lifetime.f90
@@ -0,0 +1,48 @@
+!===----------------------------------------------------------------------===!
+! This directory can be used to add Integration tests involving multiple
+! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
+! contain executable tests. We should only add tests here sparingly and only
+! if there is no other way to test. Repeat this message in each test that is
+! added to this directory and sub-directories.
+!===----------------------------------------------------------------------===!
+
+! Verify that host OpenMP callback analysis makes Flang's aggregate capture
+! container noalias, allowing its fields to be loaded before an opaque call.
+!
+! RUN: %flang_fc1 -O2 -fopenmp -emit-llvm %s -o - | FileCheck %s
+
+subroutine capture_lifetime(input, enabled)
+ integer(8), intent(in) :: input
+ logical(1), intent(in) :: enabled(64)
+ integer(8) :: captured
+ integer :: i
+
+ interface
+ subroutine opaque()
+ end subroutine
+ subroutine use_value(value)
+ integer(8), value :: value
+ end subroutine
+ end interface
+
+ captured = input
+ !$omp parallel shared(captured, enabled) private(i)
+ call opaque()
+ do i = 1, 64
+ if (enabled(i)) call use_value(captured)
+ end do
+ !$omp end parallel
+end subroutine
+
+! CHECK-LABEL: define internal void @capture_lifetime_..omp_par(
+! CHECK-SAME: ptr noalias readonly captures(none) [[CAPTURES:%.*]])
+! CHECK: omp.par.entry:
+! CHECK-NEXT: [[ENABLED:%.*]] = load ptr, ptr [[CAPTURES]], align 8
+! CHECK-NEXT: [[CAPTURE_FIELD:%.*]] = getelementptr i8, ptr [[CAPTURES]], i64 8
+! CHECK-NEXT: [[CAPTURED:%.*]] = load ptr, ptr [[CAPTURE_FIELD]], align 8
+! CHECK-NEXT: tail call void @opaque_()
+! CHECK: omp.par.region3:
+! CHECK-NOT: load ptr, ptr [[CAPTURE_FIELD]]
+! CHECK: omp.par.region4:
+! CHECK-NOT: load ptr, ptr [[CAPTURE_FIELD]]
+! CHECK: load i64, ptr [[CAPTURED]], align 8
diff --git a/llvm/lib/Transforms/IPO/AttributorAttributes.cpp
b/llvm/lib/Transforms/IPO/AttributorAttributes.cpp
index 8d16f49525ca8..f30395679a51d 100644
--- a/llvm/lib/Transforms/IPO/AttributorAttributes.cpp
+++ b/llvm/lib/Transforms/IPO/AttributorAttributes.cpp
@@ -3907,7 +3907,7 @@ struct AANoAliasCallSiteArgument final : AANoAliasImpl {
const AAMemoryBehavior &MemBehaviorAA,
const CallBase &CB, unsigned OtherArgNo) {
// We do not need to worry about aliasing with the underlying IRP.
- if (this->getCalleeArgNo() == (int)OtherArgNo)
+ if (this->getCallSiteArgNo() == (int)OtherArgNo)
return false;
// If it is not a pointer or pointer vector we do not alias.
diff --git a/llvm/lib/Transforms/IPO/OpenMPOpt.cpp
b/llvm/lib/Transforms/IPO/OpenMPOpt.cpp
index 22a5efb402504..0fff56cf17ced 100644
--- a/llvm/lib/Transforms/IPO/OpenMPOpt.cpp
+++ b/llvm/lib/Transforms/IPO/OpenMPOpt.cpp
@@ -5627,10 +5627,27 @@ void OpenMPOpt::registerAAs(bool IsModulePass) {
}
}
+ // Seed the generic noalias deduction for callback-mapped pointer arguments
+ // in host regions. It will inspect every callback and direct call site
before
+ // manifesting the attribute.
+ if (!isOpenMPDevice(M)) {
+ for (Function *F : SCC) {
+ if (F->isDeclaration())
+ continue;
+ for (Use &U : F->uses()) {
+ AbstractCallSite ACS(&U);
+ if (!ACS || !ACS.isCallbackCall())
+ continue;
+ for (Argument &Arg : F->args())
+ if (Arg.getType()->isPointerTy() && ACS.getCallArgOperandNo(Arg) >=
0)
+ A.getOrCreateAAFor<AANoAlias>(IRPosition::argument(Arg));
+ }
+ }
+ return;
+ }
+
// Create an ExecutionDomain AA for every function and a HeapToStack AA for
// every function if there is a device kernel.
- if (!isOpenMPDevice(M))
- return;
for (auto *F : SCC) {
if (F->isDeclaration())
diff --git a/llvm/lib/Transforms/Utils/BuildLibCalls.cpp
b/llvm/lib/Transforms/Utils/BuildLibCalls.cpp
index 0afcc9df40070..c3621d56ec453 100644
--- a/llvm/lib/Transforms/Utils/BuildLibCalls.cpp
+++ b/llvm/lib/Transforms/Utils/BuildLibCalls.cpp
@@ -534,6 +534,9 @@ bool llvm::inferNonMandatoryLibFuncAttrs(Function &F,
Changed |= setDoesNotCapture(F, 0);
Changed |= setOnlyReadsMemory(F, 0);
break;
+ case LibFunc_posix_memalign:
+ Changed |= setDoesNotCapture(F, 0);
+ break;
case LibFunc_aligned_alloc:
Changed |= setAlignedAllocParam(F, 0);
Changed |= setAllocSize(F, 1, std::nullopt);
diff --git a/llvm/test/Transforms/Attributor/callback-noalias.ll
b/llvm/test/Transforms/Attributor/callback-noalias.ll
new file mode 100644
index 0000000000000..aecd2a062a3a4
--- /dev/null
+++ b/llvm/test/Transforms/Attributor/callback-noalias.ll
@@ -0,0 +1,33 @@
+; RUN: opt -passes=attributor -S < %s | FileCheck %s
+; RUN: opt -passes=attributor-cgscc -S < %s | FileCheck %s
+
+; A callback argument number is not necessarily the corresponding broker
+; operand number. Make sure the alias comparison skips the actual call-site
+; operand, rather than an unrelated mapped argument.
+
+; CHECK-LABEL: define internal void @callback(
+; CHECK-SAME: ptr nofree noundef nonnull writeonly align 8 captures(none)
dereferenceable(8) %a,
+; CHECK-SAME: ptr nofree noundef nonnull readonly align 8 captures(none)
dereferenceable(8) %b)
+
+define void @caller(i64 %value) {
+ %slot = alloca i64, align 8
+ store i64 %value, ptr %slot, align 8
+ call void @broker(ptr @callback, ptr %slot, ptr %slot)
+ ret void
+}
+
+define internal void @callback(ptr align 8 dereferenceable(8) %a,
+ ptr align 8 dereferenceable(8) %b) {
+ store i64 0, ptr %a, align 8
+ call void @sync()
+ %value = load i64, ptr %b, align 8
+ call void @use(i64 %value)
+ ret void
+}
+
+declare !callback !0 void @broker(ptr, ptr, ptr)
+declare void @sync()
+declare void @use(i64) memory(none)
+
+!0 = !{!1}
+!1 = !{i64 0, i64 1, i64 2, i1 false}
diff --git a/llvm/test/Transforms/Attributor/callbacks.ll
b/llvm/test/Transforms/Attributor/callbacks.ll
index 80a0b2befbbee..884496e1dab9a 100644
--- a/llvm/test/Transforms/Attributor/callbacks.ll
+++ b/llvm/test/Transforms/Attributor/callbacks.ll
@@ -53,7 +53,7 @@ define internal void @t0_callback_callee(ptr %is_not_null,
ptr %ptr, ptr %a, i64
; TUNIT-LABEL: define {{[^@]+}}@t0_callback_callee
; TUNIT-SAME: (ptr nofree noundef nonnull writeonly align 4 captures(none)
dereferenceable(4) [[IS_NOT_NULL:%.*]], ptr nofree noundef nonnull readonly
align 8 captures(none) dereferenceable(4) [[PTR:%.*]], ptr align 256 [[A:%.*]],
i64 [[B:%.*]], ptr noalias nofree noundef nonnull readonly align 64
captures(none) dereferenceable(8) [[C:%.*]]) {
; TUNIT-NEXT: entry:
-; TUNIT-NEXT: [[PTR_VAL:%.*]] = load i32, ptr [[PTR]], align 8
+; TUNIT-NEXT: [[PTR_VAL:%.*]] = load i32, ptr [[PTR]], align 8,
!invariant.load [[META0:![0-9]+]]
; TUNIT-NEXT: store i32 [[PTR_VAL]], ptr [[IS_NOT_NULL]], align 4
; TUNIT-NEXT: [[TMP0:%.*]] = load ptr, ptr [[C]], align 64
; TUNIT-NEXT: tail call void @t0_check(ptr align 256 [[A]], i64 noundef 99,
ptr align 32 [[TMP0]])
@@ -62,9 +62,9 @@ define internal void @t0_callback_callee(ptr %is_not_null,
ptr %ptr, ptr %a, i64
; CGSCC-LABEL: define {{[^@]+}}@t0_callback_callee
; CGSCC-SAME: (ptr nofree noundef nonnull writeonly align 4 captures(none)
dereferenceable(4) [[IS_NOT_NULL:%.*]], ptr nofree noundef nonnull readonly
align 8 captures(none) dereferenceable(4) [[PTR:%.*]], ptr align 256 [[A:%.*]],
i64 [[B:%.*]], ptr noalias nofree noundef nonnull readonly align 64
captures(none) dereferenceable(8) [[C:%.*]]) {
; CGSCC-NEXT: entry:
-; CGSCC-NEXT: [[PTR_VAL:%.*]] = load i32, ptr [[PTR]], align 8
+; CGSCC-NEXT: [[PTR_VAL:%.*]] = load i32, ptr [[PTR]], align 8,
!invariant.load [[META0:![0-9]+]]
; CGSCC-NEXT: store i32 [[PTR_VAL]], ptr [[IS_NOT_NULL]], align 4
-; CGSCC-NEXT: [[TMP0:%.*]] = load ptr, ptr [[C]], align 64
+; CGSCC-NEXT: [[TMP0:%.*]] = load ptr, ptr [[C]], align 64, !invariant.load
[[META0]]
; CGSCC-NEXT: tail call void @t0_check(ptr align 256 [[A]], i64 noundef 99,
ptr [[TMP0]])
; CGSCC-NEXT: ret void
;
@@ -95,7 +95,7 @@ define void @t1_caller(ptr noalias %a) {
; TUNIT-NEXT: [[PTR:%.*]] = alloca i32, align 128
; TUNIT-NEXT: store i32 42, ptr [[B]], align 32
; TUNIT-NEXT: store ptr [[B]], ptr [[C]], align 64
-; TUNIT-NEXT: call void (ptr, ptr, ptr, ...) @t1_callback_broker(ptr
noundef null, ptr noalias noundef nonnull align 128 captures(none)
dereferenceable(4) [[PTR]], ptr noundef nonnull captures(none)
@t1_callback_callee, ptr align 256 captures(none) [[A]], i64 undef, ptr noalias
nofree noundef nonnull readonly align 64 captures(none) dereferenceable(8)
[[C]])
+; TUNIT-NEXT: call void (ptr, ptr, ptr, ...) @t1_callback_broker(ptr
noundef null, ptr noalias noundef nonnull align 128 captures(none)
dereferenceable(4) [[PTR]], ptr noundef nonnull captures(none)
@t1_callback_callee, ptr noalias align 256 captures(none) [[A]], i64 undef, ptr
noalias nofree noundef nonnull readonly align 64 captures(none)
dereferenceable(8) [[C]])
; TUNIT-NEXT: ret void
;
; CGSCC-LABEL: define {{[^@]+}}@t1_caller
@@ -106,7 +106,7 @@ define void @t1_caller(ptr noalias %a) {
; CGSCC-NEXT: [[PTR:%.*]] = alloca i32, align 128
; CGSCC-NEXT: store i32 42, ptr [[B]], align 32
; CGSCC-NEXT: store ptr [[B]], ptr [[C]], align 64
-; CGSCC-NEXT: call void (ptr, ptr, ptr, ...) @t1_callback_broker(ptr
noundef null, ptr noalias noundef nonnull align 128 captures(none)
dereferenceable(4) [[PTR]], ptr noundef nonnull captures(none)
@t1_callback_callee, ptr align 256 captures(none) [[A]], i64 noundef 99, ptr
noalias nofree noundef nonnull readonly align 64 captures(none)
dereferenceable(8) [[C]])
+; CGSCC-NEXT: call void (ptr, ptr, ptr, ...) @t1_callback_broker(ptr
noundef null, ptr noalias noundef nonnull align 128 captures(none)
dereferenceable(4) [[PTR]], ptr noundef nonnull captures(none)
@t1_callback_callee, ptr noalias align 256 captures(none) [[A]], i64 noundef
99, ptr noalias nofree noundef nonnull readonly align 64 captures(none)
dereferenceable(8) [[C]])
; CGSCC-NEXT: ret void
;
entry:
@@ -125,9 +125,9 @@ define internal void @t1_callback_callee(ptr %is_not_null,
ptr %ptr, ptr %a, i64
;
; TUNIT: Function Attrs: nosync
; TUNIT-LABEL: define {{[^@]+}}@t1_callback_callee
-; TUNIT-SAME: (ptr nofree noundef nonnull writeonly align 4 captures(none)
dereferenceable(4) [[IS_NOT_NULL:%.*]], ptr nofree noundef nonnull readonly
align 8 captures(none) dereferenceable(4) [[PTR:%.*]], ptr align 256
captures(none) [[A:%.*]], i64 [[B:%.*]], ptr noalias nofree noundef nonnull
readonly align 64 captures(none) dereferenceable(8) [[C:%.*]])
#[[ATTR0:[0-9]+]] {
+; TUNIT-SAME: (ptr nofree noundef nonnull writeonly align 4 captures(none)
dereferenceable(4) [[IS_NOT_NULL:%.*]], ptr nofree noundef nonnull readonly
align 8 captures(none) dereferenceable(4) [[PTR:%.*]], ptr noalias align 256
captures(none) [[A:%.*]], i64 [[B:%.*]], ptr noalias nofree noundef nonnull
readonly align 64 captures(none) dereferenceable(8) [[C:%.*]])
#[[ATTR0:[0-9]+]] {
; TUNIT-NEXT: entry:
-; TUNIT-NEXT: [[PTR_VAL:%.*]] = load i32, ptr [[PTR]], align 8
+; TUNIT-NEXT: [[PTR_VAL:%.*]] = load i32, ptr [[PTR]], align 8,
!invariant.load [[META0]]
; TUNIT-NEXT: store i32 [[PTR_VAL]], ptr [[IS_NOT_NULL]], align 4
; TUNIT-NEXT: [[TMP0:%.*]] = load ptr, ptr [[C]], align 64
; TUNIT-NEXT: tail call void @t1_check(ptr align 256 captures(none) [[A]],
i64 noundef 99, ptr align 32 captures(none) [[TMP0]])
@@ -135,11 +135,11 @@ define internal void @t1_callback_callee(ptr
%is_not_null, ptr %ptr, ptr %a, i64
;
; CGSCC: Function Attrs: nosync
; CGSCC-LABEL: define {{[^@]+}}@t1_callback_callee
-; CGSCC-SAME: (ptr nofree noundef nonnull writeonly align 4 captures(none)
dereferenceable(4) [[IS_NOT_NULL:%.*]], ptr nofree noundef nonnull readonly
align 8 captures(none) dereferenceable(4) [[PTR:%.*]], ptr align 256
captures(none) [[A:%.*]], i64 [[B:%.*]], ptr noalias nofree noundef nonnull
readonly align 64 captures(none) dereferenceable(8) [[C:%.*]])
#[[ATTR0:[0-9]+]] {
+; CGSCC-SAME: (ptr nofree noundef nonnull writeonly align 4 captures(none)
dereferenceable(4) [[IS_NOT_NULL:%.*]], ptr nofree noundef nonnull readonly
align 8 captures(none) dereferenceable(4) [[PTR:%.*]], ptr noalias align 256
captures(none) [[A:%.*]], i64 [[B:%.*]], ptr noalias nofree noundef nonnull
readonly align 64 captures(none) dereferenceable(8) [[C:%.*]])
#[[ATTR0:[0-9]+]] {
; CGSCC-NEXT: entry:
-; CGSCC-NEXT: [[PTR_VAL:%.*]] = load i32, ptr [[PTR]], align 8
+; CGSCC-NEXT: [[PTR_VAL:%.*]] = load i32, ptr [[PTR]], align 8,
!invariant.load [[META0]]
; CGSCC-NEXT: store i32 [[PTR_VAL]], ptr [[IS_NOT_NULL]], align 4
-; CGSCC-NEXT: [[TMP0:%.*]] = load ptr, ptr [[C]], align 64
+; CGSCC-NEXT: [[TMP0:%.*]] = load ptr, ptr [[C]], align 64, !invariant.load
[[META0]]
; CGSCC-NEXT: tail call void @t1_check(ptr align 256 captures(none) [[A]],
i64 noundef 99, ptr ca...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/218453
_______________________________________________
cfe-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
