> The expression `(A & B) ^ B` is algebraically equivalent to `~A & B`, which > maps directly to the AArch64 `BIC` (bit clear) instruction. > > Now the expression is lowered as two operations `AndV` and `XorV`, this patch > adds C2 match rules to fold this pattern into a single `BIC` for both > integral vectors and vector masks on AArch64 (NEON and SVE). JMH > micro-benchmarks demonstrate a significant performance improvement. > > On a Nvidia Grace (Neoverse-V2) machine with 128-bit SVE2: > <html xmlns:v="urn:schemas-microsoft-com:vml" > xmlns:o="urn:schemas-microsoft-com:office:office" > xmlns:x="urn:schemas-microsoft-com:office:excel" > xmlns="http://www.w3.org/TR/REC-html40";> > > <head> > > <meta name=ProgId content=Excel.Sheet> > <meta name=Generator content="Microsoft Excel 15"> > <link id=Main-File rel=Main-File > href="file:////Users/erfang/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip.htm"> > <link rel=File-List > href="file:////Users/erfang/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip_filelist.xml"> > > </head> > > <body link="#467886" vlink="#96607D"> > > > Benchmark | size | Unit | Before | Error | After | Error | Uplift > -- | -- | -- | -- | -- | -- | -- | -- > LogicOperationsBenchmark.byteNotAnd | 1024 | ops/ms | 20769.0 | 49.6 | > 52065.1 | 684.0 | 2.51 > LogicOperationsBenchmark.intNotAnd | 1024 | ops/ms | 3576.6 | 0.4 | 7980.8 | > 4.0 | 2.23 > LogicOperationsBenchmark.longNotAnd | 1024 | ops/ms | 1699.1 | 0.6 | 3575.1 | > 0.6 | 2.10 > LogicOperationsBenchmark.shortNotAnd | 1024 | ops/ms | 7986.0 | 3.4 | 20523.8 > | 13.6 | 2.57 > MaskLogicOperationsBenchmark.byteMaskNotAnd | 256 | ops/ms | 82165.6 | 70.1 | > 108802.4 | 169.4 | 1.32 > MaskLogicOperationsBenchmark.byteMaskNotAnd | 512 | ops/ms | 42859.7 | 27.1 | > 59531.1 | 65.5 | 1.39 > MaskLogicOperationsBenchmark.byteMaskNotAnd | 1024 | ops/ms | 21703.1 | 9.3 | > 31475.2 | 5.5 | 1.45 > MaskLogicOperationsBenchmark.intMaskNotAnd | 256 | ops/ms | 15467.4 | 50.9 | > 27138.2 | 76.1 | 1.75 > MaskLogicOperationsBenchmark.intMaskNotAnd | 512 | ops/ms | 7951.4 | 14.9 | > 13962.2 | 4.9 | 1.76 > MaskLogicOperationsBenchmark.intMaskNotAnd | 1024 | ops/ms | 3988.2 | 4.1 | > 6876.1 | 2.1 | 1.72 > MaskLogicOperationsBenchmark.longMaskNotAnd | 256 | ops/ms | 7933.8 | 6.4 | > 13933.2 | 6.6 | 1.76 > MaskLogicOperationsBenchmark.longMaskNotAnd | 512 | ops/ms | 3977.9 | 4.0 | > 6865.8 | 4.2 | 1.73 > MaskLogicOperationsBenchmark.longMaskNotAnd | 1024 | ops/ms | 1992.7 | 1.2 | > 3400.5 | 1.7 | 1.71 > MaskLogicOperationsBenchmark.shortMaskNotAnd | 256 | ...
Eric Fang has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains three additional commits since the last revision: - Canonicalize "(A & B) ^ B" to "~A & B" in midend - Merge branch 'master' into JDK-8388918-not-and-opt - 8388918: AArch64: Optimize vector and vector mask (A & B) ^ B to BIC instruction The expression `(A & B) ^ B` is algebraically equivalent to `~A & B`, which maps directly to the AArch64 `BIC` (bit clear) instruction. Now the expression is lowered as two operations `AndV` and `XorV`, this patch adds C2 match rules to fold this pattern into a single `BIC` for both integral vectors and vector masks on AArch64 (NEON and SVE). JMH micro-benchmarks demonstrate a significant performance improvement. On a Nvidia Grace (Neoverse-V2) machine with 128-bit SVE2: ``` Benchmark size Unit Before Error After Error Uplift LogicOperationsBenchmark.byteNotAnd 1024 ops/ms 20769.0 49.6 52065.1 684.0 2.51 LogicOperationsBenchmark.intNotAnd 1024 ops/ms 3576.6 0.4 7980.8 4.0 2.23 LogicOperationsBenchmark.longNotAnd 1024 ops/ms 1699.1 0.6 3575.1 0.6 2.10 LogicOperationsBenchmark.shortNotAnd 1024 ops/ms 7986.0 3.4 20523.8 13.6 2.57 MaskLogicOperationsBenchmark.byteMaskNotAnd 256 ops/ms 82165.6 70.1 108802 169.4 1.32 MaskLogicOperationsBenchmark.byteMaskNotAnd 512 ops/ms 42859.7 27.1 59531.1 65.5 1.39 MaskLogicOperationsBenchmark.byteMaskNotAnd 1024 ops/ms 21703.1 9.3 31475.2 5.5 1.45 MaskLogicOperationsBenchmark.intMaskNotAnd 256 ops/ms 15467.4 50.9 27138.2 76.1 1.75 MaskLogicOperationsBenchmark.intMaskNotAnd 512 ops/ms 7951.4 14.9 13962.2 4.9 1.76 MaskLogicOperationsBenchmark.intMaskNotAnd 1024 ops/ms 3988.2 4.1 6876.1 2.1 1.72 MaskLogicOperationsBenchmark.longMaskNotAnd 256 ops/ms 7933.8 6.4 13933.2 6.6 1.76 MaskLogicOperationsBenchmark.longMaskNotAnd 512 ops/ms 3977.9 4.0 6865.8 4.2 1.73 MaskLogicOperationsBenchmark.longMaskNotAnd 1024 ops/ms 1992.7 1.2 3400.5 1.7 1.71 MaskLogicOperationsBenchmark.shortMaskNotAnd 256 ops/ms 30460.9 35.2 52855.5 30.4 1.74 MaskLogicOperationsBenchmark.shortMaskNotAnd 512 ops/ms 15430.4 37.9 26508.9 33.5 1.72 MaskLogicOperationsBenchmark.shortMaskNotAnd 1024 ops/ms 8064.8 14.7 13846.5 4.9 1.72 ``` On an AWS Graviton3 (Neoverse-V1) machine with 256-bit SVE1: ``` Benchmark size Unit Before Error After Error Uplift LogicOperationsBenchmark.byteNotAnd 1024 ops/ms 35793.9 6.5 64358.5 42.6 1.80 LogicOperationsBenchmark.intNotAnd 1024 ops/ms 5892.5 1.8 14042.4 1.5 2.38 LogicOperationsBenchmark.longNotAnd 1024 ops/ms 2722.2 0.4 5889.9 1.1 2.16 LogicOperationsBenchmark.shortNotAnd 1024 ops/ms 14096.6 6.8 30413.2 17.7 2.16 MaskLogicOperationsBenchmark.byteMaskNotAnd 256 ops/ms 53505.2 83.4 72209.9 493.7 1.35 MaskLogicOperationsBenchmark.byteMaskNotAnd 512 ops/ms 28833.3 129.2 37787.6 29.0 1.31 MaskLogicOperationsBenchmark.byteMaskNotAnd 1024 ops/ms 15157.6 23.0 19666.3 22.7 1.30 MaskLogicOperationsBenchmark.intMaskNotAnd 256 ops/ms 14977.2 65.2 19980.3 33.6 1.33 MaskLogicOperationsBenchmark.intMaskNotAnd 512 ops/ms 7737.0 3.8 10200.4 15.7 1.32 MaskLogicOperationsBenchmark.intMaskNotAnd 1024 ops/ms 3921.8 1.6 5195.8 11.2 1.32 MaskLogicOperationsBenchmark.longMaskNotAnd 256 ops/ms 7796.8 15.0 10178.0 11.2 1.31 MaskLogicOperationsBenchmark.longMaskNotAnd 512 ops/ms 3884.9 0.4 5174.7 3.7 1.33 MaskLogicOperationsBenchmark.longMaskNotAnd 1024 ops/ms 1865.3 2.7 2556.7 8.3 1.37 MaskLogicOperationsBenchmark.shortMaskNotAnd 256 ops/ms 28326.7 37.3 37997.8 406.9 1.34 MaskLogicOperationsBenchmark.shortMaskNotAnd 512 ops/ms 14864.5 29.7 20060.8 76.7 1.35 MaskLogicOperationsBenchmark.shortMaskNotAnd 1024 ops/ms 7600.4 9.8 10281.7 21.5 1.35 ``` On a Nvidia Grace (Neoverse-V2) machine with NEON: ``` Benchmark size Unit Before Error After Error Uplift LogicOperationsBenchmark.byteNotAnd 1024 ops/ms 20835.0 118.2 52623.3 926.7 2.53 LogicOperationsBenchmark.intNotAnd 1024 ops/ms 3557.0 2.4 7942.8 6.6 2.23 LogicOperationsBenchmark.longNotAnd 1024 ops/ms 1691.3 1.8 3561.8 3.7 2.11 LogicOperationsBenchmark.shortNotAnd 1024 ops/ms 7953.2 8.5 20472.2 16.5 2.57 MaskLogicOperationsBenchmark.byteMaskNotAnd 256 ops/ms 132525 612.7 139916 6617.6 1.06 MaskLogicOperationsBenchmark.byteMaskNotAnd 512 ops/ms 69445.0 35.0 78484.9 402.9 1.13 MaskLogicOperationsBenchmark.byteMaskNotAnd 1024 ops/ms 35659.9 181.1 38741.8 235.2 1.09 MaskLogicOperationsBenchmark.intMaskNotAnd 256 ops/ms 16553.8 16.1 18701.7 82.4 1.13 MaskLogicOperationsBenchmark.intMaskNotAnd 512 ops/ms 8312.9 5.3 9371.1 23.7 1.13 MaskLogicOperationsBenchmark.intMaskNotAnd 1024 ops/ms 4174.3 1.0 4639.7 17.3 1.11 MaskLogicOperationsBenchmark.longMaskNotAnd 256 ops/ms 6442.5 8.6 6768.9 3.6 1.05 MaskLogicOperationsBenchmark.longMaskNotAnd 512 ops/ms 3239.7 3.9 3390.9 1.2 1.05 MaskLogicOperationsBenchmark.longMaskNotAnd 1024 ops/ms 1620.9 1.7 1690.5 2.0 1.04 MaskLogicOperationsBenchmark.shortMaskNotAnd 256 ops/ms 46164.5 236.9 56235.9 212.7 1.22 MaskLogicOperationsBenchmark.shortMaskNotAnd 512 ops/ms 23326.0 57.0 28611.9 31.3 1.23 MaskLogicOperationsBenchmark.shortMaskNotAnd 1024 ops/ms 11711.3 9.9 14438.7 13.5 1.23 ``` The PR was tested on AArch64 (with SVE2, SVE1 and NEON) and X86 (AVX3, AVX2) platforms, all tests (tier1, tier2 and tier3) passed. ------------- Changes: - all: https://git.openjdk.org/jdk/pull/32095/files - new: https://git.openjdk.org/jdk/pull/32095/files/b7d755c8..c40310f1 Webrevs: - full: https://webrevs.openjdk.org/?repo=jdk&pr=32095&range=01 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=32095&range=00-01 Stats: 225501 lines in 2208 files changed: 207614 ins; 5988 del; 11899 mod Patch: https://git.openjdk.org/jdk/pull/32095.diff Fetch: git fetch https://git.openjdk.org/jdk.git pull/32095/head:pull/32095 PR: https://git.openjdk.org/jdk/pull/32095
