llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT--> @llvm/pr-subscribers-backend-amdgpu Author: Dhruva Chakrabarti (dhruvachak) <details> <summary>Changes</summary> GCNDownwardRPTracker::advanceToNext() added an instruction's defs to the live set while the uses that die at that instruction were still in it, and only dropped them in the following advanceBeforeNext(). This could lead to inflated register pressure reporting. Apply the transfer function in the order the generic tracker uses in RegPressureTracker::advance(): retire the lanes that die at the instruction first, then add its defs. Early-clobber defs are the exception. PHIs are skipped because nothing dies there. All of these are now done by advanceToNext(). advanceBeforeNext() now only drops dead def lanes, the uses having been retired already. Two callers depend on the previous ordering and are updated: - SIFormMemoryClauses::checkPressure() deliberately keeps dying uses live, because every def in a clause is early-clobber and stays live to the end of the clause. It now asks for that explicitly via RetireDeadUses=false. - The next-use distance printer reports distances for every register live at the instruction, including the ones it consumes, so it adds the live-in lanes of the uses back. This patch affects default codegen, not just the GCNTrackers path. This is because region pressure is measured with the downward tracker regardless of GCNTrackers. The trackers become measurably more accurate: in regpressure_printer.mir the upward and downward trackers now report identical pressure for all ten functions, where five previously disagreed, and in schedule-regpressure-ilp-metric-spills.mir the GCN trackers stop emitting twelve spills and agree with the generic tracker that none are needed. Among the LIT tests changed, 35 of 47 assembly tests have the same spill and instruction counts, differing only in numbering or order. Among the remaining, 5 have reduced instruction count. load-global-i16.ll is the most notable, where 32 of the 52 removed instructions are spill/reload pairs. rewrite-vgpr-mfma-to-agpr-spill-multi-store-codegen.ll no longer has regalloc spills that the rewrite phase eliminated. The number of AGPRs drops from 128 to 114 for this test. 6 tests grow by a handful. The only outlier is amdgcn.bitcast.1024bit.ll, at +115 combined spills and reloads. That is not a scheduling regression: for the worst function the post-scheduler MIR is identical in both builds, and the allocator diverges only because LiveIntervals is maintained incrementally. Recomputing live ranges would help but that is not part of this patch. --- Patch is 3.00 MiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/228294.diff 60 Files Affected: - (modified) llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp (+17-2) - (modified) llvm/lib/Target/AMDGPU/GCNRegPressure.cpp (+45-12) - (modified) llvm/lib/Target/AMDGPU/GCNRegPressure.h (+4-1) - (modified) llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp (+5-4) - (modified) llvm/test/CodeGen/AMDGPU/GlobalISel/fshl.ll (+158-158) - (modified) llvm/test/CodeGen/AMDGPU/GlobalISel/fshr.ll (+144-144) - (modified) llvm/test/CodeGen/AMDGPU/GlobalISel/uaddsat.ll (+2-2) - (modified) llvm/test/CodeGen/AMDGPU/addrspacecast.ll (+204-216) - (modified) llvm/test/CodeGen/AMDGPU/amdgcn.bitcast.1024bit.ll (+8762-8544) - (modified) llvm/test/CodeGen/AMDGPU/amdgcn.bitcast.256bit.ll (+252-252) - (modified) llvm/test/CodeGen/AMDGPU/amdgcn.bitcast.384bit.ll (+68-68) - (modified) llvm/test/CodeGen/AMDGPU/amdgcn.bitcast.448bit.ll (+76-76) - (modified) llvm/test/CodeGen/AMDGPU/amdgcn.bitcast.512bit.ll (+3014-3003) - (modified) llvm/test/CodeGen/AMDGPU/amdgcn.bitcast.576bit.ll (+177-177) - (modified) llvm/test/CodeGen/AMDGPU/amdgcn.bitcast.640bit.ll (+462-462) - (modified) llvm/test/CodeGen/AMDGPU/amdgcn.bitcast.704bit.ll (+362-389) - (modified) llvm/test/CodeGen/AMDGPU/amdgcn.bitcast.768bit.ll (+165-165) - (modified) llvm/test/CodeGen/AMDGPU/bf16.ll (+147-147) - (modified) llvm/test/CodeGen/AMDGPU/callee-special-input-vgprs-packed.ll (+6-4) - (modified) llvm/test/CodeGen/AMDGPU/callee-special-input-vgprs.ll (+6-4) - (modified) llvm/test/CodeGen/AMDGPU/debug-value-scheduler-crash.mir (+19-19) - (modified) llvm/test/CodeGen/AMDGPU/div_i128.ll (+75-75) - (modified) llvm/test/CodeGen/AMDGPU/div_v2i128.ll (+231-231) - (modified) llvm/test/CodeGen/AMDGPU/fceil64.ll (+105-105) - (modified) llvm/test/CodeGen/AMDGPU/fmaximum.ll (+105-52) - (modified) llvm/test/CodeGen/AMDGPU/fminimum.ll (+105-52) - (modified) llvm/test/CodeGen/AMDGPU/freeze.ll (+380-389) - (modified) llvm/test/CodeGen/AMDGPU/gfx-callable-argument-types.ll (+3-3) - (modified) llvm/test/CodeGen/AMDGPU/high-RP-reschedule.mir (+4-2) - (modified) llvm/test/CodeGen/AMDGPU/llvm.exp.f64.ll (+285-285) - (modified) llvm/test/CodeGen/AMDGPU/llvm.exp10.f64.ll (+449-449) - (modified) llvm/test/CodeGen/AMDGPU/llvm.exp2.f64.ll (+351-351) - (modified) llvm/test/CodeGen/AMDGPU/llvm.maximum.f16.ll (+75-75) - (modified) llvm/test/CodeGen/AMDGPU/llvm.maximum.f32.ll (+108-108) - (modified) llvm/test/CodeGen/AMDGPU/llvm.maximum.f64.ll (+78-78) - (modified) llvm/test/CodeGen/AMDGPU/llvm.minimum.f16.ll (+75-75) - (modified) llvm/test/CodeGen/AMDGPU/llvm.minimum.f32.ll (+108-108) - (modified) llvm/test/CodeGen/AMDGPU/llvm.minimum.f64.ll (+78-78) - (modified) llvm/test/CodeGen/AMDGPU/load-constant-i32.ll (+43-37) - (modified) llvm/test/CodeGen/AMDGPU/load-global-i16.ll (+594-646) - (modified) llvm/test/CodeGen/AMDGPU/load-global-i32.ll (+237-235) - (modified) llvm/test/CodeGen/AMDGPU/load-global-i8.ll (+232-232) - (modified) llvm/test/CodeGen/AMDGPU/machine-scheduler-sink-trivial-remats-debug.mir (+7-3) - (modified) llvm/test/CodeGen/AMDGPU/machine-scheduler-sink-trivial-remats.mir (+52-52) - (modified) llvm/test/CodeGen/AMDGPU/promote-constOffset-to-imm.ll (+83-81) - (modified) llvm/test/CodeGen/AMDGPU/regpressure_printer.mir (+206-409) - (modified) llvm/test/CodeGen/AMDGPU/rem_i128.ll (+83-83) - (modified) llvm/test/CodeGen/AMDGPU/rewrite-vgpr-mfma-to-agpr-spill-multi-store-codegen.ll (+400-424) - (modified) llvm/test/CodeGen/AMDGPU/sched-assert-dead-def-subreg-use-other-subreg.mir (+1-1) - (modified) llvm/test/CodeGen/AMDGPU/sched-dynamic-vgpr-blocks.mir (+3-3) - (modified) llvm/test/CodeGen/AMDGPU/sched_mfma_rewrite_copies.mir (+13-17) - (modified) llvm/test/CodeGen/AMDGPU/schedule-regpressure-ilp-metric-spills.mir (+3-6) - (modified) llvm/test/CodeGen/AMDGPU/spill-agpr.ll (+1-1) - (modified) llvm/test/CodeGen/AMDGPU/uaddsat.ll (+24-24) - (modified) llvm/test/CodeGen/AMDGPU/vector-reduce-fmax.ll (+42-42) - (modified) llvm/test/CodeGen/AMDGPU/vector-reduce-fmaximum.ll (+110-110) - (modified) llvm/test/CodeGen/AMDGPU/vector-reduce-fmin.ll (+42-42) - (modified) llvm/test/CodeGen/AMDGPU/vector-reduce-fminimum.ll (+110-110) - (modified) llvm/test/CodeGen/AMDGPU/vector-reduce-mul.ll (+138-138) - (modified) llvm/test/CodeGen/AMDGPU/vgpr-mark-last-scratch-load.ll (+4-4) ``````````diff diff --git a/llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp b/llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp index a667b96c0d089..d880bfe6912bb 100644 --- a/llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp +++ b/llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp @@ -2451,11 +2451,26 @@ void printNextUseDistancesAsJson(json::OStream &J, const MachineFunction &MF, RPTracker.reset(MI, MBB.end()); RPTracker.advance(); + // Report distance for every register live at MI including the ones + // that the tracker retired. Add their live-in lanes back. + GCNRPTracker::LiveRegSet LiveAtMI = RPTracker.getLiveRegs(); + if (!MI.isDebugInstr()) { + SlotIndex BaseIdx = LIS.getInstructionIndex(MI).getBaseIndex(); + for (const MachineOperand &MO : MI.all_uses()) { + Register Reg = MO.getReg(); + if (!Reg.isVirtual() || !MO.readsReg()) + continue; + LaneBitmask Mask = getLiveLaneMask(Reg, BaseIdx, LIS, MRI); + if (Mask.any()) + LiveAtMI[Reg] |= Mask; + } + } + UseDistancePair Furthest; UseDistancePair FurthestSubreg; RelevantUses.clear(); - NUA.getNextUseDistances(RPTracker.getLiveRegs(), MI, Furthest, - &FurthestSubreg, &RelevantUses); + NUA.getNextUseDistances(LiveAtMI, MI, Furthest, &FurthestSubreg, + &RelevantUses); J.objectBegin(); printInstrMember(J, MST, MI, NUAImpl); diff --git a/llvm/lib/Target/AMDGPU/GCNRegPressure.cpp b/llvm/lib/Target/AMDGPU/GCNRegPressure.cpp index a371c32af0786..b364a741cc012 100644 --- a/llvm/lib/Target/AMDGPU/GCNRegPressure.cpp +++ b/llvm/lib/Target/AMDGPU/GCNRegPressure.cpp @@ -720,18 +720,22 @@ bool GCNDownwardRPTracker::advanceBeforeNext(MachineInstr *MI, assert(SI.isValid()); - // Remove dead registers or mask bits. SmallSet<Register, 8> SeenRegs; - for (auto &MO : CurrMI->operands()) { - if (!MO.isReg() || !MO.getReg().isVirtual()) - continue; - if (MO.isUse() && !MO.readsReg()) - continue; - if (!UseInternalIterator && MO.isDef()) - continue; - if (!SeenRegs.insert(MO.getReg()).second) - continue; - retireVirtReg(MO.getReg(), SI); + if (UseInternalIterator) { + // Drop dead def lanes. + for (const MachineOperand &MO : CurrMI->all_defs()) { + Register Reg = MO.getReg(); + if (Reg.isVirtual() && SeenRegs.insert(Reg).second) + retireVirtReg(Reg, SI); + } + } else { + // Defs have not been added yet, so the only lanes that can be dropped are + // uses that are already dead. + for (const MachineOperand &MO : CurrMI->all_uses()) { + Register Reg = MO.getReg(); + if (MO.readsReg() && Reg.isVirtual() && SeenRegs.insert(Reg).second) + retireVirtReg(Reg, SI); + } } MaxPressure = max(MaxPressure, CurPressure); @@ -742,7 +746,8 @@ bool GCNDownwardRPTracker::advanceBeforeNext(MachineInstr *MI, } void GCNDownwardRPTracker::advanceToNext(MachineInstr *MI, - bool UseInternalIterator) { + bool UseInternalIterator, + bool RetireDeadUses) { if (UseInternalIterator) { LastTrackedMI = &*NextMI++; NextMI = skipDebugInstructionsForward(NextMI, MBBEnd); @@ -752,6 +757,34 @@ void GCNDownwardRPTracker::advanceToNext(MachineInstr *MI, const MachineInstr *CurrMI = LastTrackedMI; + if (RetireDeadUses) { + // Mirror ECDefPressure handling from GCNUpwardRPTracker::recede(). + GCNRegPressure ECDefPressure; + for (const MachineOperand &MO : CurrMI->all_defs()) { + Register Reg = MO.getReg(); + if (Reg.isVirtual() && MO.isEarlyClobber()) + ECDefPressure.inc(Reg, LaneBitmask::getNone(), getDefRegMask(MO, *MRI), + *MRI); + } + + MaxPressure = max(MaxPressure, CurPressure + ECDefPressure); + + // Retire the uses that die here before adding the defs, mirroring + // RegPressureTracker::advance(). + // + // A PHI is skipped since its operands do not die here. + if (!CurrMI->isPHI()) { + SlotIndex SI = LIS.getInstructionIndex(*CurrMI).getDeadSlot(); + assert(SI.isValid()); + SmallSet<Register, 8> SeenRegs; + for (const MachineOperand &MO : CurrMI->all_uses()) { + Register Reg = MO.getReg(); + if (MO.readsReg() && Reg.isVirtual() && SeenRegs.insert(Reg).second) + retireVirtReg(Reg, SI); + } + } + } + // Add new registers or mask bits. for (const auto &MO : CurrMI->all_defs()) { Register Reg = MO.getReg(); diff --git a/llvm/lib/Target/AMDGPU/GCNRegPressure.h b/llvm/lib/Target/AMDGPU/GCNRegPressure.h index c17f8f33489ea..7fb60107367de 100644 --- a/llvm/lib/Target/AMDGPU/GCNRegPressure.h +++ b/llvm/lib/Target/AMDGPU/GCNRegPressure.h @@ -450,8 +450,11 @@ class GCNDownwardRPTracker : public GCNRPTracker { /// it is assumed that the tracker is using an externally managed iterator, /// and advance* calls will not update the state of the iterator. In such /// cases, the tracker will move to the state at the provided \p MI . + /// If \p RetireDeadUses is false the uses that die at the instruction are + /// kept live. Callers that do this must restore the tracked state afterwards. void advanceToNext(MachineInstr *MI = nullptr, - bool UseInternalIterator = true); + bool UseInternalIterator = true, + bool RetireDeadUses = true); /// Move to the state at the next MI. \p returns false if reached end of /// block. If \p UseInternalIterator is true, then internal iterators are used diff --git a/llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp b/llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp index 905c2afa9fcd8..c0c46be61576b 100644 --- a/llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp +++ b/llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp @@ -189,11 +189,12 @@ bool SIFormMemoryClausesImpl::canBundle(const MachineInstr &MI, // bundled into a memory clause. bool SIFormMemoryClausesImpl::checkPressure(const MachineInstr &MI, GCNDownwardRPTracker &RPT) { - // NB: skip advanceBeforeNext() call. Since all defs will be marked - // early-clobber they will all stay alive at least to the end of the - // clause. Therefor we should not decrease pressure even if load + // NB: skip advanceBeforeNext() call and keep the dying uses live. Since all + // defs will be marked early-clobber they will all stay alive at least to the + // end of the clause. Therefor we should not decrease pressure even if load // pointer becomes dead and could otherwise be reused for destination. - RPT.advanceToNext(); + RPT.advanceToNext(/*MI=*/nullptr, /*UseInternalIterator=*/true, + /*RetireDeadUses=*/false); GCNRegPressure MaxPressure = RPT.moveMaxPressure(); unsigned Occupancy = MaxPressure.getOccupancy( *ST, diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/fshl.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/fshl.ll index 7e4e9c6aeefe1..d5b230428fba5 100644 --- a/llvm/test/CodeGen/AMDGPU/GlobalISel/fshl.ll +++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/fshl.ll @@ -8157,88 +8157,88 @@ define <2 x i128> @v_fshl_v2i128(<2 x i128> %lhs, <2 x i128> %rhs, <2 x i128> %a ; GFX6-LABEL: v_fshl_v2i128: ; GFX6: ; %bb.0: ; GFX6-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) -; GFX6-NEXT: v_and_b32_e32 v19, 0x7f, v16 -; GFX6-NEXT: v_not_b32_e32 v18, 63 -; GFX6-NEXT: v_sub_i32_e32 v23, vcc, 64, v19 -; GFX6-NEXT: v_add_i32_e32 v27, vcc, v19, v18 -; GFX6-NEXT: v_lshr_b64 v[23:24], v[0:1], v23 -; GFX6-NEXT: v_lshl_b64 v[25:26], v[2:3], v19 -; GFX6-NEXT: v_lshl_b64 v[21:22], v[0:1], v19 -; GFX6-NEXT: v_lshl_b64 v[0:1], v[0:1], v27 -; GFX6-NEXT: v_cmp_gt_u32_e32 vcc, 64, v19 -; GFX6-NEXT: v_cmp_eq_u32_e64 s[4:5], 0, v19 -; GFX6-NEXT: v_or_b32_e32 v19, v23, v25 -; GFX6-NEXT: v_or_b32_e32 v23, v24, v26 -; GFX6-NEXT: v_cndmask_b32_e32 v0, v0, v19, vcc -; GFX6-NEXT: v_cndmask_b32_e32 v1, v1, v23, vcc -; GFX6-NEXT: v_cndmask_b32_e64 v19, v0, v2, s[4:5] -; GFX6-NEXT: v_cndmask_b32_e64 v23, v1, v3, s[4:5] -; GFX6-NEXT: v_lshr_b64 v[0:1], v[8:9], 1 +; GFX6-NEXT: v_and_b32_e32 v25, 0x7f, v16 +; GFX6-NEXT: v_sub_i32_e32 v17, vcc, 64, v25 +; GFX6-NEXT: v_lshr_b64 v[17:18], v[0:1], v17 +; GFX6-NEXT: v_lshl_b64 v[21:22], v[2:3], v25 +; GFX6-NEXT: v_or_b32_e32 v26, v17, v21 +; GFX6-NEXT: v_or_b32_e32 v27, v18, v22 ; GFX6-NEXT: v_mov_b32_e32 v17, 0x7f -; GFX6-NEXT: v_lshlrev_b32_e32 v2, 31, v10 -; GFX6-NEXT: v_or_b32_e32 v1, v1, v2 -; GFX6-NEXT: v_lshr_b64 v[2:3], v[10:11], 1 -; GFX6-NEXT: v_bfi_b32 v10, v16, 0, v17 -; GFX6-NEXT: v_cndmask_b32_e32 v24, 0, v21, vcc -; GFX6-NEXT: v_cndmask_b32_e32 v25, 0, v22, vcc -; GFX6-NEXT: v_add_i32_e32 v16, vcc, v10, v18 -; GFX6-NEXT: v_sub_i32_e32 v21, vcc, 64, v10 -; GFX6-NEXT: v_cmp_gt_u32_e32 vcc, 64, v10 -; GFX6-NEXT: v_cmp_eq_u32_e64 s[4:5], 0, v10 -; GFX6-NEXT: v_lshr_b64 v[8:9], v[2:3], v10 -; GFX6-NEXT: v_lshr_b64 v[10:11], v[0:1], v10 -; GFX6-NEXT: v_lshl_b64 v[21:22], v[2:3], v21 -; GFX6-NEXT: v_lshr_b64 v[2:3], v[2:3], v16 -; GFX6-NEXT: v_or_b32_e32 v10, v10, v21 -; GFX6-NEXT: v_or_b32_e32 v11, v11, v22 -; GFX6-NEXT: v_cndmask_b32_e32 v2, v2, v10, vcc -; GFX6-NEXT: v_cndmask_b32_e32 v3, v3, v11, vcc -; GFX6-NEXT: v_and_b32_e32 v16, 0x7f, v20 -; GFX6-NEXT: v_cndmask_b32_e64 v0, v2, v0, s[4:5] -; GFX6-NEXT: v_cndmask_b32_e64 v1, v3, v1, s[4:5] -; GFX6-NEXT: v_cndmask_b32_e32 v2, 0, v8, vcc -; GFX6-NEXT: v_cndmask_b32_e32 v3, 0, v9, vcc -; GFX6-NEXT: v_sub_i32_e32 v10, vcc, 64, v16 -; GFX6-NEXT: v_lshr_b64 v[10:11], v[4:5], v10 -; GFX6-NEXT: v_lshl_b64 v[21:22], v[6:7], v16 -; GFX6-NEXT: v_or_b32_e32 v2, v19, v2 -; GFX6-NEXT: v_add_i32_e32 v19, vcc, v16, v18 -; GFX6-NEXT: v_cmp_gt_u32_e32 vcc, 64, v16 -; GFX6-NEXT: v_cmp_eq_u32_e64 s[4:5], 0, v16 -; GFX6-NEXT: v_lshl_b64 v[8:9], v[4:5], v16 -; GFX6-NEXT: v_or_b32_e32 v16, v10, v21 -; GFX6-NEXT: v_or_b32_e32 v21, v11, v22 -; GFX6-NEXT: v_lshl_b64 v[10:11], v[4:5], v19 -; GFX6-NEXT: v_cndmask_b32_e32 v4, 0, v8, vcc -; GFX6-NEXT: v_cndmask_b32_e32 v5, 0, v9, vcc -; GFX6-NEXT: v_cndmask_b32_e32 v8, v10, v16, vcc -; GFX6-NEXT: v_cndmask_b32_e32 v9, v11, v21, vcc -; GFX6-NEXT: v_cndmask_b32_e64 v6, v8, v6, s[4:5] -; GFX6-NEXT: v_cndmask_b32_e64 v7, v9, v7, s[4:5] -; GFX6-NEXT: v_lshr_b64 v[8:9], v[12:13], 1 -; GFX6-NEXT: v_lshlrev_b32_e32 v10, 31, v14 -; GFX6-NEXT: v_or_b32_e32 v9, v9, v10 -; GFX6-NEXT: v_lshr_b64 v[10:11], v[14:15], 1 -; GFX6-NEXT: v_bfi_b32 v14, v20, 0, v17 -; GFX6-NEXT: v_add_i32_e32 v18, vcc, v14, v18 -; GFX6-NEXT: v_sub_i32_e32 v16, vcc, 64, v14 -; GFX6-NEXT: v_cmp_gt_u32_e32 vcc, 64, v14 -; GFX6-NEXT: v_cmp_eq_u32_e64 s[4:5], 0, v14 -; GFX6-NEXT: v_lshr_b64 v[12:13], v[10:11], v14 -; GFX6-NEXT: v_lshr_b64 v[14:15], v[8:9], v14 -; GFX6-NEXT: v_lshl_b64 v[16:17], v[10:11], v16 -; GFX6-NEXT: v_lshr_b64 v[10:11], v[10:11], v18 +; GFX6-NEXT: v_lshr_b64 v[18:19], v[8:9], 1 +; GFX6-NEXT: v_lshlrev_b32_e32 v8, 31, v10 +; GFX6-NEXT: v_lshr_b64 v[9:10], v[10:11], 1 +; GFX6-NEXT: v_bfi_b32 v11, v16, 0, v17 +; GFX6-NEXT: v_or_b32_e32 v19, v19, v8 +; GFX6-NEXT: v_sub_i32_e32 v8, vcc, 64, v11 +; GFX6-NEXT: v_lshl_b64 v[21:22], v[9:10], v8 +; GFX6-NEXT: v_lshr_b64 v[23:24], v[18:19], v11 +; GFX6-NEXT: v_not_b32_e32 v8, 63 +; GFX6-NEXT: v_or_b32_e32 v16, v23, v21 +; GFX6-NEXT: v_add_i32_e32 v21, vcc, v25, v8 +; GFX6-NEXT: v_or_b32_e32 v23, v24, v22 +; GFX6-NEXT: v_lshl_b64 v[21:22], v[0:1], v21 +; GFX6-NEXT: v_lshl_b64 v[0:1], v[0:1], v25 +; GFX6-NEXT: v_cmp_gt_u32_e32 vcc, 64, v25 +; GFX6-NEXT: v_cndmask_b32_e32 v24, 0, v0, vcc +; GFX6-NEXT: v_cndmask_b32_e32 v0, v21, v26, vcc +; GFX6-NEXT: v_cndmask_b32_e32 v21, v22, v27, vcc +; GFX6-NEXT: v_cndmask_b32_e32 v22, 0, v1, vcc +; GFX6-NEXT: v_cmp_eq_u32_e32 vcc, 0, v25 +; GFX6-NEXT: v_cndmask_b32_e32 v25, v0, v2, vcc +; GFX6-NEXT: v_cndmask_b32_e32 v21, v21, v3, vcc +; GFX6-NEXT: v_add_i32_e32 v0, vcc, v11, v8 +; GFX6-NEXT: v_lshr_b64 v[0:1], v[9:10], v0 +; GFX6-NEXT: v_lshr_b64 v[2:3], v[9:10], v11 +; GFX6-NEXT: v_cmp_gt_u32_e32 vcc, 64, v11 +; GFX6-NEXT: v_cmp_eq_u32_e64 s[4:5], 0, v11 +; GFX6-NEXT: v_and_b32_e32 v11, 0x7f, v20 +; GFX6-NEXT: v_cndmask_b32_e32 v0, v0, v16, vcc +; GFX6-NEXT: v_cndmask_b32_e32 v1, v1, v23, vcc +; GFX6-NEXT: v_cndmask_b32_e32 v2, 0, v2, vcc +; GFX6-NEXT: v_cndmask_b32_e32 v3, 0, v3, vcc +; GFX6-NEXT: v_sub_i32_e32 v9, vcc, 64, v11 +; GFX6-NEXT: v_cndmask_b32_e64 v0, v0, v18, s[4:5] +; GFX6-NEXT: v_cndmask_b32_e64 v1, v1, v19, s[4:5] +; GFX6-NEXT: v_lshr_b64 v[9:10], v[4:5], v9 +; GFX6-NEXT: v_lshl_b64 v[18:19], v[6:7], v11 +; GFX6-NEXT: v_add_i32_e32 v16, vcc, v11, v8 +; GFX6-NEXT: v_or_b32_e32 v1, v22, v1 +; GFX6-NEXT: v_or_b32_e32 v3, v21, v3 +; GFX6-NEXT: v_lshl_b64 v[21:22], v[4:5], v11 +; GFX6-NEXT: v_or_b32_e32 v18, v9, v18 +; GFX6-NEXT: v_or_b32_e32 v19, v10, v19 +; GFX6-NEXT: v_lshl_b64 v[9:10], v[4:5], v16 +; GFX6-NEXT: v_cmp_gt_u32_e32 vcc, 64, v11 +; GFX6-NEXT: v_cndmask_b32_e32 v4, 0, v21, vcc +; GFX6-NEXT: v_cndmask_b32_e32 v5, 0, v22, vcc +; GFX6-NEXT: v_cndmask_b32_e32 v9, v9, v18, vcc +; GFX6-NEXT: v_cndmask_b32_e32 v10, v10, v19, vcc +; GFX6-NEXT: v_cmp_eq_u32_e32 vcc, 0, v11 +; GFX6-NEXT: v_cndmask_b32_e32 v6, v9, v6, vcc +; GFX6-NEXT: v_cndmask_b32_e32 v7, v10, v7, vcc +; GFX6-NEXT: v_lshr_b64 v[9:10], v[12:13], 1 +; GFX6-NEXT: v_lshlrev_b32_e32 v11, 31, v14 +; GFX6-NEXT: v_or_b32_e32 v10, v10, v11 +; GFX6-NEXT: v_lshr_b64 v[11:12], v[14:15], 1 +; GFX6-NEXT: v_bfi_b32 v19, v20, 0, v17 +; GFX6-NEXT: v_sub_i32_e32 v15, vcc, 64, v19 +; GFX6-NEXT: v_add_i32_e32 v8, vcc, v19, v8 +; GFX6-NEXT: v_lshr_b64 v[13:14], v[9:10], v19 +; GFX6-NEXT: v_lshl_b64 v[15:16], v[11:12], v15 +; GFX6-NEXT: v_lshr_b64 v[17:18], v[11:12], v19 +; GFX6-NEXT: v_lshr_b64 v[11:12], v[11:12], v8 +; GFX6-NEXT: v_or_b32_e32 v13, v13, v15 ; GFX6-NEXT: v_or_b32_e32 v14, v14, v16 -; GFX6-NEXT: v_or_b32_e32 v15, v15, v17 -; GFX6-NEXT: v_cndmask_b32_e32 v10, v10, v14, vcc -; GFX6-NEXT: v_cndmask_b32_e32 v11, v11, v15, vcc -; GFX6-NEXT: v_cndmask_b32_e64 v8, v10, v8, s[4:5] -; GFX6-NEXT: v_cndmask_b32_e64 v9, v11, v9, s[4:5] -; GFX6-NEXT: v_cndmask_b32_e32 v10, 0, v12, vcc -; GFX6-NEXT: v_cndmask_b32_e32 v11, 0, v13, vcc +; GFX6-NEXT: v_cmp_gt_u32_e32 vcc, 64, v19 +; GFX6-NEXT: v_cndmask_b32_e32 v8, v11, v13, vcc +; GFX6-NEXT: v_cndmask_b32_e32 v11, v12, v14, vcc +; GFX6-NEXT: v_cmp_eq_u32_e64 s[4:5], 0, v19 +; GFX6-NEXT: v_cndmask_b32_e64 v8, v8, v9, s[4:5] +; GFX6-NEXT: v_cndmask_b32_e64 v9, v11, v10, s[4:5] +; GFX6-NEXT: v_cndmask_b32_e32 v10, 0, v17, vcc +; GFX6-NEXT: v_cndmask_b32_e32 v11, 0, v18, vcc ; GFX6-NEXT: v_or_b32_e32 v0, v24, v0 -; GFX6-NEXT: v_or_b32_e32 v1, v25, v1 -; GFX6-NEXT: v_or_b32_e32 v3, v23, v3 +; GFX6-NEXT: v_or_b32_e32 v2, v25, v2 ; GFX6-NEXT: v_or_b32_e32 v4, v4, v8 ; GFX6-NEXT: v_or_b32_e32 v5, v5, v9 ; GFX6-NEXT: v_or_b32_e32 v6, v6, v10 @@ -8248,88 +8248,88 @@ define <2 x i128> @v_fshl_v2i128(<2 x i128> %lhs, <2 x i128> %rhs, <2 x i128> %a ; GFX8-LABEL: v_fshl_v2i128: ; GFX8: ; %bb.0: ; GFX8-NEXT: s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0) -; GFX8-NEXT: v_and_b32_e32 v19, 0x7f, v16 -; GFX8-NEXT: v_not_b32_e32 v18, 63 -; GFX8-NEXT: v_sub_u32_e32 v23, vcc, 64, v19 -; GFX8-NEXT: v_add_u32_e32 v27, vcc, v19, v18 -; GFX8-NEXT: v_lshrrev_b64 v[23:24], v23, v[0:1] -; GFX8-NEXT: v_lshlrev_b64 v[25:26], v19, v[2:3] -; GFX8-NEXT: v_lshlrev_b64 v[21:22], v19, v[0:1] -; GFX8-NEXT: v_lshlrev_b64 v[0:1], v27, v[0:1] -; GFX8-NEXT: v_cmp_gt_u32_e32 vcc, 64, v19 -; GFX8-NEXT: v_cmp_eq_u32_e64 s[4:5], 0, v19 -; GFX8-NEXT: v_or_b32_e32 v19, v23, v25 -; GFX8-NEXT: v_or_b32_e32 v23, v24, v26 -; GFX8-NEXT: v_cndmask_b32_e32 v0, v0, v19, vcc -; GFX8-NEXT: v_cndmask_b32_e32 v1, v1, v23, vcc -; GFX8-NEXT: v_cndmask_b32_e64 v19, v0, v2, s[4:5] -; GFX8-NEXT: v_cndmask_b32_e64 v23, v1, v3, s[4:5] -; GFX8-NEXT: v_lshrrev_b64 v[0:1], 1, v[8:9] +; GFX8-NEXT: v_and_b32_e32 v25, 0x7f, v16 +; GFX8-NEXT: v_sub_u32_e32 v17, vcc, 64, v25 +; GFX8-NEXT: v_lshrrev_b64 v[17:18], v17, v[0:1] +; GFX8-NEXT: v_lshlrev_b64 v[21:22], v25, v[2:3] +; GFX8-NEXT: v_or_b32_e32 v26, v17, v21 +; GFX8-NEXT: v_or_b32_e32 v27, v18, v22 ; GFX8-NEXT: v_mov_b32_e32 v17, 0x7f -; GFX8-NEXT: v_lshlrev_b32_e32 v2, 31, v10 -; GFX8-NEXT: v_or_b32_e32 v1, v1, v2 -; GFX8-NEXT: v_lshrrev_b64 v[2:3], 1, v[10:11] -; GFX8-NEXT: v_bfi_b32 v10, v16, 0, v17 -; GFX8-NEXT: v_cndmask_b32_e32 v24, 0, v21, vcc -; GFX8-NEXT: v_cndmask_b32_e32 v25, 0, v22, vcc -; GFX8-NEXT: v_add_u32_e32 v16, vcc, v10, v18 -; GFX8-NEXT: v_sub_u32_e32 v21, vcc, 64, v10 -; GFX8-NEXT: v_cmp_gt_u32_e32 vcc, 64, v10 -; GFX8-NEXT: v_cmp_eq_u32_e64 s[4:5], 0, v10 -; GFX8-NEXT: v_lshrrev_b64 v[8:9], v10, v[2:3] -; GFX8-NEXT: v_lshrrev_b64 v[10:11], v10, v[0:1] -; GFX8-NEXT: v_lshlrev_b64 v[21:22], v21, v[2:3] -; GFX8-NEXT: v_lshrrev_b64 v[2:3], v16, v[2:3] -; GFX8-NEXT: v_or_b32_e32 v10, v10, v21 -; GFX8-NEXT: v_or_b32_e32 v11, v11, v22 -; GFX8-NEXT: v_cndmask_b32_e32 v2, v2, v10, vcc -; GFX8-NEXT: v_cndmask_b32_e32 v3, v3, v11, vcc -; GFX8-NEXT: v_and_b32_e32 v16, 0x7f, v20 -; GFX8-NEXT: v_cndmask_b32_e64 v0, v2, v0, s[4:5] -; GFX8-NEXT: v_cndmask_b32_e64 v1, v3, v1, s[4:5] -; GFX8-NEXT: v_cndmask_b32_e32 v2, 0, v8, vcc -; GFX8-NEXT: v_cndmask_b32_e32 v3, 0, v9, vcc -; GFX8-NEXT: v_sub_u32_e32 v10, vcc, 64, v16 -; GFX8-NEXT: v_lshrrev_b64 v[10:11], v10, v[4:5] -; GFX8-NEXT: v_lshlrev_b64 v[21:22], v16, v[6:7] -; GFX8-NEXT: v_or_b32_e32 v2, v19, v2 -; GFX8-NEXT: v_add_u32_e32 v19, vcc, v16, v18 -; GFX8-NEXT: v_cmp_gt_u32_e32 vcc, 64, v16 -; GFX8-NEXT: v_cmp_eq_u32_e64 s[4:5], 0, v16 -; GFX8-NEXT: v_lshlrev_b64 v[8:9], v16, v[4:5] -; GFX8-NEXT: v_or_b32_e32 v16, v10, v21 -; GFX8-NEXT: v_or_b32_e32 v21, v11, v22 -; GFX8-NEXT: v_lshlrev_b64 v[10:11], v19, v[4:5] -; GFX8-NEXT: v_cndmask_b32_e32 v4, 0, v8, vcc -; GFX8-NEXT: v_cndmask_b32_e32 v5, 0, v9, vcc -; GFX8-NEXT: v_cndmask_b32_e32 v8, v10, v16, vcc -; GFX8-NEXT: v_cndmask_b32_e32 v9, v11, v21, vcc -; GFX8-NEXT: v_cndmask_b32_e64 v6, v8, v6, s[4:5] -; GFX8-NEXT: v_cndmask_b32_e64 v7, v9, v7, s[4:5] -; GFX8-NEXT: v_lshrrev_b64 v[8:9], 1, v[12:13] -; GFX8-NEXT: v_lshlrev_b32_e32 v10, 31, v14 -; GFX8-NEXT: v_or_b32_e32 v9, v9, v10 -; GFX8-NEXT: v_lshrrev_b64 v[10:11], 1, v[14:15] -; GFX8-NEXT: v_bfi_b32 v14, v20, 0, v17 -; GFX8-NEXT: v_add_u32_e32 v18, vcc, v14, v18 -; GFX8-NEXT: v_sub_u32_e32 v16, vcc, 64, v14 -; GFX8-NEXT: v_cmp_gt_u32_e32 vcc, 64, v14 -; GFX8-NEXT: v_cmp_eq_u32_e64 s[4:5], 0, v14 -; GFX8-NEXT: v_lshrrev_b64 v[12:13], v14, v[10:11] -; GFX8-NEXT: v_lshrrev_b64 v[14:15], v14, v[8:9] -; GFX8-NEXT: v_lshlrev_b64 v[16:17], v16, v[10:11] -; GFX8-NEXT: v_lshrrev_b64 v[10:11], v18, v[10:11] +; GFX8-NEXT: v_lshrrev_b64 v[18:19], 1, v[8:9] +; GFX8-NEXT: v_lshlrev_b32_e32 v8, 31, v10 +; GFX8-NEXT: v_lshrrev_b64 v[9:10], 1, v[10:11] +; GFX8-NEXT: v_bfi_b32 v11, v16, 0, v17 +; GFX8-NEXT: v_or_b32_e32 v19, v19, v8 +; GFX8-NEXT: v_sub_u32_e32 v8, vcc, 64, v11 +; GFX8-NEXT: v_lshlrev_b64 v[21:22], v8, v[9:10] +; GFX8-NEXT: v_lshrrev_b64 v[23:24], v11, v[18:19] +; GFX8-NEXT: v_not_b32_e32 v8, 63 +; GFX8-NEXT: v_or_b32_e32 v16, v23, v21 +; GFX8-NEXT: v_add_u32_e32 v21, vcc, v25, v8 +; GFX8-NEXT: v_or_b32_e32 v23, v24, v22 +; GFX8-NEXT: v_lshlrev_b64 v[21:22], v21, v[0:1] +; GFX8-NEXT: v_lshlrev_b64 v[0:1], v25, v[0:1] +; GFX8-NEXT: v_cmp_gt_u32_e32 vcc, 64, v25 +; GFX8-NEXT: v_cndmask_b32_e32 v24, 0, v0, vcc +; GFX8-NEXT: v_cndmask_b32_e32 v0, v21, v26, vcc +; GFX8-NEXT: v_cndmask_b32_e32 v21, v22, v27, vcc +; GFX8-NEXT: v_cndmask_b32_e32 v22, 0, v1, vcc +; GFX8-NEXT: v_cmp_eq_u32_e32 vcc, 0, v25 +; GFX... [truncated] `````````` </details> https://github.com/llvm/llvm-project/pull/228294 _______________________________________________ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
