[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/eddyz87 closed https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/eddyz87 updated https://github.com/llvm/llvm-project/pull/84410 >From 1cb6905e726b1f6c6cb71ea1e4d9ab4405aedeaf Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 26 Jan 2024 04:18:32 +0200 Subject: [PATCH] [BPF] Add addr_space_cast BPF instruction addr_space_cast instruction === This commit aims to support BPF arena kernel side feature [0]: - arena is a memory region accessible from both BPF program and userspace; - base pointers for this memory region differ between kernel and user spaces; - `dst_reg = addr_space_cast(src_reg, dst_addr_space, src_addr_space)` translates src_reg, a pointer in src_addr_space to dst_reg, equivalent pointer in dst_addr_space, {src,dst}_addr_space are immediate constants; - number 0 is assigned to kernel address space; - number 1 is assigned to user address space. [0] https://lore.kernel.org/bpf/20240206220441.38311-1-alexei.starovoi...@gmail.com/ On the LLVM side, the goal is to make load and store operations on arena pointers "transparent" for BPF programs: - assume that pointers with non-zero address space are pointers to arena memory; - assume that arena is identified by address space number; - assume that address space zero corresponds to kernel address space; - assume that every BPF-side load or store from arena is done via pointer in user address space, thus convert base pointers using `addr_space_cast(src_reg, 0, 1)`; Only load, store, cmpxchg and atomicrmw IR instructions are handled by this transformation. For example, the following C code: #define __as __attribute__((address_space(1))) void copy(int __as *from, int __as *to) { *to = *from; } Compiled to the following IR: define void @copy(ptr addrspace(1) %from, ptr addrspace(1) %to) { entry: %0 = load i32, ptr addrspace(1) %from, align 4 store i32 %0, ptr addrspace(1) %to, align 4 ret void } Is transformed to: %to2 = addrspacecast ptr addrspace(1) %to to ptr ;; ! %from1 = addrspacecast ptr addrspace(1) %from to ptr ;; ! %0 = load i32, ptr %from1, align 4, !tbaa !3 store i32 %0, ptr %to2, align 4, !tbaa !3 ret void And compiled as: r2 = addr_space_cast(r2, 0, 1) r1 = addr_space_cast(r1, 0, 1) r1 = *(u32 *)(r1 + 0) *(u32 *)(r2 + 0) = r1 exit Internally: - piggy-back `BPFCheckAndAdjustIR` pass to insert address space casts for base pointer of memory access instructions, when base pointer has non-zero address space; - modify `BPFInstrInfo.td` and `BPFIselLowering.cpp` to allow translation of `addrspacecast` instruction: - define new machine instruction: `ADDR_SPACE_CAST`; - define pattern to select `ADDR_SPACE_CAST` for `addrspacecast` ISD nodes. Global variables in .arena section == Make it so that all globals within same address space reside in section with name ".arena.N", where N is number of the address space. E.g. for the following C program: ```c __as const char a[2] = {1,2}; __as char b[2] = {3,4}; __as char c[2]; ... ``` Generate the following layout: ``` $ clang -O2 --target=bpf t.c -c -o - \ | llvm-readelf --sections --symbols - ... Section Headers: [Nr] Name TypeAddress OffSize ES Flg Lk Inf Al ... [ 4] .arena.272PROGBITS e8 18 00 WA 0 0 4 ... Symbol table '.symtab' contains 8 entries: Num:Value Size TypeBind Vis Ndx Name ... 3: 8 OBJECT GLOBAL DEFAULT 4 a 4: 0008 8 OBJECT GLOBAL DEFAULT 4 b 5: 0010 8 OBJECT GLOBAL DEFAULT 4 c ... ^^^ Note section index ``` Convert 'addrcast M->N -> GEP -> addrcast N->M' to just GEP === For BPF GEP would adjust pointer using same offset in any address space, thus transformation of form: %inner = addrspacecast N->M %ptr %gep = getelementptr %inner, ... %outer = addrspacecast M->N %gep to just: %gep = getelementptr %ptr, ... is valid. Applying such transformation helps with C patterns that use e.g. (void *) casts to offsets w/o actual memory access: #define container_of(ptr, type, member) \ ({ \ void __arena *__mptr = (void *)(ptr); \ ((type *)(__mptr - offsetof(type, member)));\ }) (Note the address space cast on first body line) --- clang/lib/Basic/Targets/BPF.cpp | 3 + .../test/Preprocessor/bpf-predefined-macros.c | 8 ++ .../lib/Target/BPF/AsmParser/BPFAsmParser.cpp | 1 + llvm/lib/Target/BPF/BPF.h | 8 ++ .../Target/BPF/BPFASpaceCastSimplifyPass.cpp | 92 ++ llvm/lib/Target/BPF/BPFCheckAndAdj
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,52 @@ +; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 4 +; RUN: opt --bpf-check-and-opt-ir -S -mtriple=bpf-pc-linux < %s | FileCheck %s + +; Generated from the following C code: +; +; extern int __uptr *magic1(); +; extern int __uptr *magic2(); +; +; void test(long i) { +; int __uptr *a; +; +; if (i > 42) +; a = magic1(); +; else +; a = magic2(); +; a[5] = 7; +; } +; +; Using the following command: +; +; clang --target=bpf -O2 -S -emit-llvm -o t.ll t.c + +; Function Attrs: nounwind eddyz87 wrote: Done. https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/eddyz87 updated https://github.com/llvm/llvm-project/pull/84410 >From 503c1abc8dd63ec1500d1ed867a4bfefc1aed062 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 26 Jan 2024 04:18:32 +0200 Subject: [PATCH 1/4] [BPF] Add addr_space_cast BPF instruction This commit aims to support BPF arena kernel side feature [0]: - arena is a memory region accessible from both BPF program and userspace; - base pointers for this memory region differ between kernel and user spaces; - `dst_reg = addr_space_cast(src_reg, dst_addr_space, src_addr_space)` translates src_reg, a pointer in src_addr_space to dst_reg, equivalent pointer in dst_addr_space, {src,dst}_addr_space are immediate constants; - number 0 is assigned to kernel address space; - number 1 is assigned to user address space. On the LLVM side, the goal is to make load and store operations on arena pointers "transparent" for BPF programs: - assume that pointers with non-zero address space are pointers to arena memory; - assume that arena is identified by address space number; - assume that address space zero corresponds to kernel address space; - assume that every BPF-side load or store from arena is done via pointer in user address space, thus convert base pointers using `addr_space_cast(src_reg, 0, 1)`; Only load, store, cmpxchg and atomicrmw IR instructions are handled by this transformation. For example, the following C code: #define __as __attribute__((address_space(1))) void copy(int __as *from, int __as *to) { *to = *from; } Compiled to the following IR: define void @copy(ptr addrspace(1) %from, ptr addrspace(1) %to) { entry: %0 = load i32, ptr addrspace(1) %from, align 4 store i32 %0, ptr addrspace(1) %to, align 4 ret void } Is transformed to: %to2 = addrspacecast ptr addrspace(1) %to to ptr ;; ! %from1 = addrspacecast ptr addrspace(1) %from to ptr ;; ! %0 = load i32, ptr %from1, align 4, !tbaa !3 store i32 %0, ptr %to2, align 4, !tbaa !3 ret void And compiled as: r2 = addr_space_cast(r2, 0, 1) r1 = addr_space_cast(r1, 0, 1) r1 = *(u32 *)(r1 + 0) *(u32 *)(r2 + 0) = r1 exit Internally: - piggy-back `BPFCheckAndAdjustIR` pass to insert address space casts for base pointer of memory access instructions, when base pointer has non-zero address space; - modify `BPFInstrInfo.td` and `BPFIselLowering.cpp` to allow translation of `addrspacecast` instruction: - define new machine instruction: `ADDR_SPACE_CAST`; - define pattern to select `ADDR_SPACE_CAST` for `addrspacecast` ISD nodes. [0] https://lore.kernel.org/bpf/20240206220441.38311-1-alexei.starovoi...@gmail.com/ --- .../lib/Target/BPF/AsmParser/BPFAsmParser.cpp | 1 + llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp | 104 ++ llvm/lib/Target/BPF/BPFInstrInfo.td | 29 + .../test/CodeGen/BPF/addr-space-auto-casts.ll | 78 + llvm/test/CodeGen/BPF/addr-space-cast.ll | 22 llvm/test/CodeGen/BPF/addr-space-gep-chain.ll | 25 + llvm/test/CodeGen/BPF/addr-space-phi.ll | 53 + .../test/CodeGen/BPF/assembler-disassembler.s | 7 ++ 8 files changed, 319 insertions(+) create mode 100644 llvm/test/CodeGen/BPF/addr-space-auto-casts.ll create mode 100644 llvm/test/CodeGen/BPF/addr-space-cast.ll create mode 100644 llvm/test/CodeGen/BPF/addr-space-gep-chain.ll create mode 100644 llvm/test/CodeGen/BPF/addr-space-phi.ll diff --git a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp index 0d1eef60c3b550..3145bc3d19f5dc 100644 --- a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp +++ b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp @@ -271,6 +271,7 @@ struct BPFOperand : public MCParsedAsmOperand { .Case("xchg32_32", true) .Case("cmpxchg_64", true) .Case("cmpxchg32_32", true) +.Case("addr_space_cast", true) .Default(false); } }; diff --git a/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp b/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp index 81effc9b1db46c..d39a6b57aafed4 100644 --- a/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp +++ b/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp @@ -14,6 +14,8 @@ // optimizations are done and those builtins can be removed. // - remove llvm.bpf.getelementptr.and.load builtins. // - remove llvm.bpf.getelementptr.and.store builtins. +// - for loads and stores with base addresses from non-zero address space +// cast base address to zero address space (support for BPF arenas). // //===--===// @@ -55,6 +57,7 @@ class BPFCheckAndAdjustIR final : public ModulePass { bool removeCompareBuiltin(Module &M); bool sinkMinMax(Module &M); bool removeGEPBuiltins(Module &M); + bool insertASpaceCasts(Module &M); }; } // End anonymous namespace @@ -416,11 +419,112 @@ bool BPFCheckAndAdjustIR::removeGEPBu
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,52 @@ +; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 4 +; RUN: opt --bpf-check-and-opt-ir -S -mtriple=bpf-pc-linux < %s | FileCheck %s + +; Generated from the following C code: +; +; extern int __uptr *magic1(); +; extern int __uptr *magic2(); +; +; void test(long i) { +; int __uptr *a; +; +; if (i > 42) +; a = magic1(); +; else +; a = magic2(); +; a[5] = 7; +; } +; +; Using the following command: +; +; clang --target=bpf -O2 -S -emit-llvm -o t.ll t.c + +; Function Attrs: nounwind eddyz87 wrote: Oh, I see what you mean. I cleaned up the attributes by hand because those were not necessary for the test and have not adjusted the comment. Will change, thank you. https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/inclyc edited https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,52 @@ +; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 4 +; RUN: opt --bpf-check-and-opt-ir -S -mtriple=bpf-pc-linux < %s | FileCheck %s + +; Generated from the following C code: +; +; extern int __uptr *magic1(); +; extern int __uptr *magic2(); +; +; void test(long i) { +; int __uptr *a; +; +; if (i > 42) +; a = magic1(); +; else +; a = magic2(); +; a[5] = 7; +; } +; +; Using the following command: +; +; clang --target=bpf -O2 -S -emit-llvm -o t.ll t.c + +; Function Attrs: nounwind inclyc wrote: No, it do has 'nounwind' attr because it is attributed with `#0`, however, in our testing ir I see `#0` is omitted https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/eddyz87 approved this pull request. https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/eddyz87 edited https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/eddyz87 updated https://github.com/llvm/llvm-project/pull/84410 >From 0575ca506badd58d870ce03b1f2e26e7a3d6168b Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 26 Jan 2024 04:18:32 +0200 Subject: [PATCH 1/4] [BPF] Add addr_space_cast BPF instruction This commit aims to support BPF arena kernel side feature [0]: - arena is a memory region accessible from both BPF program and userspace; - base pointers for this memory region differ between kernel and user spaces; - `dst_reg = addr_space_cast(src_reg, dst_addr_space, src_addr_space)` translates src_reg, a pointer in src_addr_space to dst_reg, equivalent pointer in dst_addr_space, {src,dst}_addr_space are immediate constants; - number 0 is assigned to kernel address space; - number 1 is assigned to user address space. On the LLVM side, the goal is to make load and store operations on arena pointers "transparent" for BPF programs: - assume that pointers with non-zero address space are pointers to arena memory; - assume that arena is identified by address space number; - assume that address space zero corresponds to kernel address space; - assume that every BPF-side load or store from arena is done via pointer in user address space, thus convert base pointers using `addr_space_cast(src_reg, 0, 1)`; Only load, store, cmpxchg and atomicrmw IR instructions are handled by this transformation. For example, the following C code: #define __as __attribute__((address_space(1))) void copy(int __as *from, int __as *to) { *to = *from; } Compiled to the following IR: define void @copy(ptr addrspace(1) %from, ptr addrspace(1) %to) { entry: %0 = load i32, ptr addrspace(1) %from, align 4 store i32 %0, ptr addrspace(1) %to, align 4 ret void } Is transformed to: %to2 = addrspacecast ptr addrspace(1) %to to ptr ;; ! %from1 = addrspacecast ptr addrspace(1) %from to ptr ;; ! %0 = load i32, ptr %from1, align 4, !tbaa !3 store i32 %0, ptr %to2, align 4, !tbaa !3 ret void And compiled as: r2 = addr_space_cast(r2, 0, 1) r1 = addr_space_cast(r1, 0, 1) r1 = *(u32 *)(r1 + 0) *(u32 *)(r2 + 0) = r1 exit Internally: - piggy-back `BPFCheckAndAdjustIR` pass to insert address space casts for base pointer of memory access instructions, when base pointer has non-zero address space; - modify `BPFInstrInfo.td` and `BPFIselLowering.cpp` to allow translation of `addrspacecast` instruction: - define new machine instruction: `ADDR_SPACE_CAST`; - define pattern to select `ADDR_SPACE_CAST` for `addrspacecast` ISD nodes. [0] https://lore.kernel.org/bpf/20240206220441.38311-1-alexei.starovoi...@gmail.com/ --- .../lib/Target/BPF/AsmParser/BPFAsmParser.cpp | 1 + llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp | 104 ++ llvm/lib/Target/BPF/BPFInstrInfo.td | 29 + .../test/CodeGen/BPF/addr-space-auto-casts.ll | 78 + llvm/test/CodeGen/BPF/addr-space-cast.ll | 22 llvm/test/CodeGen/BPF/addr-space-gep-chain.ll | 25 + llvm/test/CodeGen/BPF/addr-space-phi.ll | 54 + .../test/CodeGen/BPF/assembler-disassembler.s | 7 ++ 8 files changed, 320 insertions(+) create mode 100644 llvm/test/CodeGen/BPF/addr-space-auto-casts.ll create mode 100644 llvm/test/CodeGen/BPF/addr-space-cast.ll create mode 100644 llvm/test/CodeGen/BPF/addr-space-gep-chain.ll create mode 100644 llvm/test/CodeGen/BPF/addr-space-phi.ll diff --git a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp index 0d1eef60c3b550..3145bc3d19f5dc 100644 --- a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp +++ b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp @@ -271,6 +271,7 @@ struct BPFOperand : public MCParsedAsmOperand { .Case("xchg32_32", true) .Case("cmpxchg_64", true) .Case("cmpxchg32_32", true) +.Case("addr_space_cast", true) .Default(false); } }; diff --git a/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp b/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp index 81effc9b1db46c..d39a6b57aafed4 100644 --- a/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp +++ b/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp @@ -14,6 +14,8 @@ // optimizations are done and those builtins can be removed. // - remove llvm.bpf.getelementptr.and.load builtins. // - remove llvm.bpf.getelementptr.and.store builtins. +// - for loads and stores with base addresses from non-zero address space +// cast base address to zero address space (support for BPF arenas). // //===--===// @@ -55,6 +57,7 @@ class BPFCheckAndAdjustIR final : public ModulePass { bool removeCompareBuiltin(Module &M); bool sinkMinMax(Module &M); bool removeGEPBuiltins(Module &M); + bool insertASpaceCasts(Module &M); }; } // End anonymous namespace @@ -416,11 +419,112 @@ bool BPFCheckAndAdjustIR::removeGEPBu
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,92 @@ +//=== BPFIRPeephole.cpp - IR Peephole Transformation --===// eddyz87 wrote: My bad. Thank you for taking a look at this MR, will push updates with fixes soon. https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,52 @@ +; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 4 +; RUN: opt --bpf-check-and-opt-ir -S -mtriple=bpf-pc-linux < %s | FileCheck %s + +; Generated from the following C code: +; +; extern int __uptr *magic1(); +; extern int __uptr *magic2(); +; +; void test(long i) { +; int __uptr *a; +; +; if (i > 42) +; a = magic1(); +; else +; a = magic2(); +; a[5] = 7; +; } +; +; Using the following command: +; +; clang --target=bpf -O2 -S -emit-llvm -o t.ll t.c + +; Function Attrs: nounwind eddyz87 wrote: Retested this, the attrs are like in the test: ```c $ cat test.c #define __uptr __attribute__((address_space(1))) extern int __uptr *magic1(); extern int __uptr *magic2(); void test(long i) { int __uptr *a; if (i > 42) a = magic1(); else a = magic2(); a[5] = 7; } $ clang --target=bpf -O2 -S -emit-llvm -o - test.c | head -n10 ; ModuleID = 'test.c' source_filename = "test.c" target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128" target triple = "bpf" ; Function Attrs: nounwind define dso_local void @test(i64 noundef %i) local_unnamed_addr #0 { entry: %cmp = icmp sgt i64 %i, 42 br i1 %cmp, label %if.then, label %if.else ``` https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -31,6 +31,9 @@ def SDT_BPFMEMCPY : SDTypeProfile<0, 4, [SDTCisVT<0, i64>, SDTCisVT<1, i64>, SDTCisVT<2, i64>, SDTCisVT<3, i64>]>; +def SDT_BPFAddrSpaceCast : SDTypeProfile<0, 3, [SDTCisPtrTy<0>, eddyz87 wrote: That works, thanks for the tip. I ended up with the following `td` definitions: ```llvm def ADDR_SPACE_CAST : ALU_RR { bits<64> dst_as; bits<64> src_as; let Inst{47-32} = 1; let Inst{31-16} = dst_as{15-0}; let Inst{15-0} = src_as{15-0}; } def SrcAddrSpace : SDNodeXFormgetTargetConstant( cast(N)->getSrcAddressSpace(), SDLoc(N), MVT::i64); }]>; def DstAddrSpace : SDNodeXFormgetTargetConstant( cast(N)->getDestAddressSpace(), SDLoc(N), MVT::i64); }]>; def : Pat<(addrspacecast:$this GPR:$src), (ADDR_SPACE_CAST $src, (DstAddrSpace $this), (SrcAddrSpace $this))>; ``` https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -638,6 +643,31 @@ SDValue BPFTargetLowering::LowerDYNAMIC_STACKALLOC(SDValue Op, return DAG.getMergeValues(Ops, SDLoc()); } +SDValue BPFTargetLowering::LowerADDRSPACECAST(SDValue Op, + SelectionDAG &DAG) const { + auto *ACast = dyn_cast(Op); + const SDValue &Ptr = ACast->getOperand(0); + unsigned SrcAS = ACast->getSrcAddressSpace(); + unsigned DstAS = ACast->getDestAddressSpace(); + SDLoc DL(Op); + + if (SrcAS > 0 && DstAS > 0) { +SmallString<64> Msg; +raw_svector_ostream OS(Msg); +OS << "Can't cast address space " << SrcAS << " to " << DstAS + << ": either source or destination address space has to be zero"; +fail(DL, DAG, Msg); eddyz87 wrote: Actually, I think this check is no longer necessary. It remains from a previous version, where `cast_kern` and `cast_user` instructions were used, modeled as an address space cast at IR level. There either source or destination address space had to be zero. Sorry for confusion, I should have removed this when switching MR to `addr_space_cast` instruction. https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -31,6 +31,9 @@ def SDT_BPFMEMCPY : SDTypeProfile<0, 4, [SDTCisVT<0, i64>, SDTCisVT<1, i64>, SDTCisVT<2, i64>, SDTCisVT<3, i64>]>; +def SDT_BPFAddrSpaceCast : SDTypeProfile<0, 3, [SDTCisPtrTy<0>, inclyc wrote: (question) Why not just use the common "addrspacecast" ISD node? https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,92 @@ +//=== BPFIRPeephole.cpp - IR Peephole Transformation --===// inclyc wrote: nit https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -638,6 +643,31 @@ SDValue BPFTargetLowering::LowerDYNAMIC_STACKALLOC(SDValue Op, return DAG.getMergeValues(Ops, SDLoc()); } +SDValue BPFTargetLowering::LowerADDRSPACECAST(SDValue Op, + SelectionDAG &DAG) const { + auto *ACast = dyn_cast(Op); + const SDValue &Ptr = ACast->getOperand(0); + unsigned SrcAS = ACast->getSrcAddressSpace(); + unsigned DstAS = ACast->getDestAddressSpace(); + SDLoc DL(Op); + + if (SrcAS > 0 && DstAS > 0) { +SmallString<64> Msg; +raw_svector_ostream OS(Msg); +OS << "Can't cast address space " << SrcAS << " to " << DstAS + << ": either source or destination address space has to be zero"; +fail(DL, DAG, Msg); inclyc wrote: Our `fail` method is used for `DiagnosticInfoUnsupported` diagnostic, should we create a new diagnostic kind? https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
@@ -0,0 +1,52 @@ +; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 4 +; RUN: opt --bpf-check-and-opt-ir -S -mtriple=bpf-pc-linux < %s | FileCheck %s + +; Generated from the following C code: +; +; extern int __uptr *magic1(); +; extern int __uptr *magic2(); +; +; void test(long i) { +; int __uptr *a; +; +; if (i > 42) +; a = magic1(); +; else +; a = magic2(); +; a[5] = 7; +; } +; +; Using the following command: +; +; clang --target=bpf -O2 -S -emit-llvm -o t.ll t.c + +; Function Attrs: nounwind inclyc wrote: this function attrs looks outdated https://github.com/llvm/llvm-project/pull/84410 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] BPF address space insn (PR #84410)
llvmbot wrote: @llvm/pr-subscribers-clang Author: None (4ast) Changes --- Patch is 34.73 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/84410.diff 23 Files Affected: - (modified) clang/lib/Basic/Targets/BPF.cpp (+3) - (modified) clang/test/Preprocessor/bpf-predefined-macros.c (+8) - (modified) llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp (+1) - (modified) llvm/lib/Target/BPF/BPF.h (+8) - (added) llvm/lib/Target/BPF/BPFASpaceCastSimplifyPass.cpp (+92) - (modified) llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp (+116) - (modified) llvm/lib/Target/BPF/BPFISelLowering.cpp (+32) - (modified) llvm/lib/Target/BPF/BPFISelLowering.h (+3-1) - (modified) llvm/lib/Target/BPF/BPFInstrInfo.td (+21) - (modified) llvm/lib/Target/BPF/BPFTargetMachine.cpp (+5) - (modified) llvm/lib/Target/BPF/CMakeLists.txt (+1) - (added) llvm/test/CodeGen/BPF/addr-space-auto-casts.ll (+78) - (added) llvm/test/CodeGen/BPF/addr-space-cast.ll (+22) - (added) llvm/test/CodeGen/BPF/addr-space-gep-chain.ll (+25) - (added) llvm/test/CodeGen/BPF/addr-space-globals.ll (+30) - (added) llvm/test/CodeGen/BPF/addr-space-globals2.ll (+25) - (added) llvm/test/CodeGen/BPF/addr-space-phi.ll (+52) - (added) llvm/test/CodeGen/BPF/addr-space-simplify-1.ll (+19) - (added) llvm/test/CodeGen/BPF/addr-space-simplify-2.ll (+21) - (added) llvm/test/CodeGen/BPF/addr-space-simplify-3.ll (+26) - (added) llvm/test/CodeGen/BPF/addr-space-simplify-4.ll (+21) - (added) llvm/test/CodeGen/BPF/addr-space-simplify-5.ll (+25) - (modified) llvm/test/CodeGen/BPF/assembler-disassembler.s (+7) ``diff diff --git a/clang/lib/Basic/Targets/BPF.cpp b/clang/lib/Basic/Targets/BPF.cpp index e3fbbb720d0694..26a54f631fcfc4 100644 --- a/clang/lib/Basic/Targets/BPF.cpp +++ b/clang/lib/Basic/Targets/BPF.cpp @@ -35,6 +35,9 @@ void BPFTargetInfo::getTargetDefines(const LangOptions &Opts, Builder.defineMacro("__BPF_CPU_VERSION__", "0"); return; } + + Builder.defineMacro("__BPF_FEATURE_ARENA_CAST"); + if (CPU.empty() || CPU == "generic" || CPU == "v1") { Builder.defineMacro("__BPF_CPU_VERSION__", "1"); return; diff --git a/clang/test/Preprocessor/bpf-predefined-macros.c b/clang/test/Preprocessor/bpf-predefined-macros.c index ff4d00ac3bcfcc..fea24d1ea0ff7b 100644 --- a/clang/test/Preprocessor/bpf-predefined-macros.c +++ b/clang/test/Preprocessor/bpf-predefined-macros.c @@ -61,6 +61,9 @@ int r; #ifdef __BPF_FEATURE_ST int s; #endif +#ifdef __BPF_FEATURE_ARENA_CAST +int t; +#endif // CHECK: int b; // CHECK: int c; @@ -90,6 +93,11 @@ int s; // CPU_V4: int r; // CPU_V4: int s; +// CPU_V1: int t; +// CPU_V2: int t; +// CPU_V3: int t; +// CPU_V4: int t; + // CPU_GENERIC: int g; // CPU_PROBE: int f; diff --git a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp index 90697c6645be2f..3fbd14facaf804 100644 --- a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp +++ b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp @@ -270,6 +270,7 @@ struct BPFOperand : public MCParsedAsmOperand { .Case("xchg32_32", true) .Case("cmpxchg_64", true) .Case("cmpxchg32_32", true) +.Case("addr_space_cast", true) .Default(false); } }; diff --git a/llvm/lib/Target/BPF/BPF.h b/llvm/lib/Target/BPF/BPF.h index 5c77d183e1ef3d..bbdbdbbde53228 100644 --- a/llvm/lib/Target/BPF/BPF.h +++ b/llvm/lib/Target/BPF/BPF.h @@ -66,6 +66,14 @@ class BPFIRPeepholePass : public PassInfoMixin { static bool isRequired() { return true; } }; +class BPFASpaceCastSimplifyPass +: public PassInfoMixin { +public: + PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM); + + static bool isRequired() { return true; } +}; + class BPFAdjustOptPass : public PassInfoMixin { public: PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM); diff --git a/llvm/lib/Target/BPF/BPFASpaceCastSimplifyPass.cpp b/llvm/lib/Target/BPF/BPFASpaceCastSimplifyPass.cpp new file mode 100644 index 00..d32cecc073fab9 --- /dev/null +++ b/llvm/lib/Target/BPF/BPFASpaceCastSimplifyPass.cpp @@ -0,0 +1,92 @@ +//=== BPFIRPeephole.cpp - IR Peephole Transformation --===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===--===// + +#include "BPF.h" +#include + +#define DEBUG_TYPE "bpf-aspace-simplify" + +using namespace llvm; + +namespace { + +struct CastGEPCast { + AddrSpaceCastInst *OuterCast; + + // Match chain of instructions: + // %inner = addrspacecast N->M + // %gep = getelementptr %inner, ... + // %outer = addrspacecast M->N %gep + // Where I is %outer. + static std::optional match(Value *I) { +auto *OuterCast = dyn_cast(I); +if (!OuterCast) +
[clang] [llvm] BPF address space insn (PR #84410)
https://github.com/4ast created https://github.com/llvm/llvm-project/pull/84410 None >From dd19f019066852dba15259cd201ce1b94fa5b6e6 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 26 Jan 2024 04:18:32 +0200 Subject: [PATCH 1/4] [BPF] Add addr_space_cast BPF instruction This commit aims to support BPF arena kernel side feature [0]: - arena is a memory region accessible from both BPF program and userspace; - base pointers for this memory region differ between kernel and user spaces; - `dst_reg = addr_space_cast(src_reg, dst_addr_space, src_addr_space)` translates src_reg, a pointer in src_addr_space to dst_reg, equivalent pointer in dst_addr_space, {src,dst}_addr_space are immediate constants; - number 0 is assigned to kernel address space; - number 1 is assigned to user address space. On the LLVM side, the goal is to make load and store operations on arena pointers "transparent" for BPF programs: - assume that pointers with non-zero address space are pointers to arena memory; - assume that arena is identified by address space number; - assume that address space zero corresponds to kernel address space; - assume that every BPF-side load or store from arena is done via pointer in user address space, thus convert base pointers using `addr_space_cast(src_reg, 0, 1)`; Only load, store, cmpxchg and atomicrmw IR instructions are handled by this transformation. For example, the following C code: #define __as __attribute__((address_space(1))) void copy(int __as *from, int __as *to) { *to = *from; } Compiled to the following IR: define void @copy(ptr addrspace(1) %from, ptr addrspace(1) %to) { entry: %0 = load i32, ptr addrspace(1) %from, align 4 store i32 %0, ptr addrspace(1) %to, align 4 ret void } Is transformed to: %to2 = addrspacecast ptr addrspace(1) %to to ptr ;; ! %from1 = addrspacecast ptr addrspace(1) %from to ptr ;; ! %0 = load i32, ptr %from1, align 4, !tbaa !3 store i32 %0, ptr %to2, align 4, !tbaa !3 ret void And compiled as: r2 = addr_space_cast(r2, 0, 1) r1 = addr_space_cast(r1, 0, 1) r1 = *(u32 *)(r1 + 0) *(u32 *)(r2 + 0) = r1 exit Internally: - piggy-back `BPFCheckAndAdjustIR` pass to insert address space casts for base pointer of memory access instructions, when base pointer has non-zero address space; - modify `BPFInstrInfo.td` and `BPFIselLowering.cpp` to allow translation of `addrspacecast` instruction: - define new SDNode type `BPFAddrSpaceCast`; - lower `addrspacecast` as such SDNode; - define new machine instruction: `ADDR_SPACE_CAST`; - define pattern to select `ADDR_SPACE_CAST` for `BPFAddrSpaceCast` nodes. [0] https://lore.kernel.org/bpf/20240206220441.38311-1-alexei.starovoi...@gmail.com/ --- .../lib/Target/BPF/AsmParser/BPFAsmParser.cpp | 1 + llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp | 104 ++ llvm/lib/Target/BPF/BPFISelLowering.cpp | 32 ++ llvm/lib/Target/BPF/BPFISelLowering.h | 4 +- llvm/lib/Target/BPF/BPFInstrInfo.td | 21 .../test/CodeGen/BPF/addr-space-auto-casts.ll | 78 + llvm/test/CodeGen/BPF/addr-space-cast.ll | 22 llvm/test/CodeGen/BPF/addr-space-gep-chain.ll | 25 + llvm/test/CodeGen/BPF/addr-space-phi.ll | 52 + .../test/CodeGen/BPF/assembler-disassembler.s | 7 ++ 10 files changed, 345 insertions(+), 1 deletion(-) create mode 100644 llvm/test/CodeGen/BPF/addr-space-auto-casts.ll create mode 100644 llvm/test/CodeGen/BPF/addr-space-cast.ll create mode 100644 llvm/test/CodeGen/BPF/addr-space-gep-chain.ll create mode 100644 llvm/test/CodeGen/BPF/addr-space-phi.ll diff --git a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp index 90697c6645be2f..3fbd14facaf804 100644 --- a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp +++ b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp @@ -270,6 +270,7 @@ struct BPFOperand : public MCParsedAsmOperand { .Case("xchg32_32", true) .Case("cmpxchg_64", true) .Case("cmpxchg32_32", true) +.Case("addr_space_cast", true) .Default(false); } }; diff --git a/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp b/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp index 81effc9b1db46c..d39a6b57aafed4 100644 --- a/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp +++ b/llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp @@ -14,6 +14,8 @@ // optimizations are done and those builtins can be removed. // - remove llvm.bpf.getelementptr.and.load builtins. // - remove llvm.bpf.getelementptr.and.store builtins. +// - for loads and stores with base addresses from non-zero address space +// cast base address to zero address space (support for BPF arenas). // //===--===// @@ -55,6 +57,7 @@ class BPFCheckAndAdjustIR final : public ModulePass { bool removeCompareBu