On Fri, 13 Nov 2026 06:56:45 GMT, Larry <[email protected]> wrote: >> ### Summary >> This PR adds a non-temporal store path to the x86_64 AVX2 forward primitive >> arraycopy stubs, extending the existing AVX-512 streaming-store approach to >> the AVX2 implementation and improving throughput for large streaming copies. >> >> ### Problem >> The existing AVX2 arraycopy implementation suffers from two major >> performance issues: >> >> - Unnecessary write-allocate traffic: temporal destination stores for >> streaming copies introduce write-allocate traffic and cache pollution >> - Missing streaming path in AVX2: the AVX-512 implementation already >> provides a streaming store path for large copies, but the AVX2 >> implementation does not >> >> ### Changes >> Added a non-temporal store path for large primitive array copies in the AVX2 >> forward stubs >> >> The new loop aligns the destination to 32 bytes and copies 64 bytes per >> iteration using two 256-bit loads and two VMOVNTDQ stores, followed by an >> SFENCE. Alignment and trailing elements use temporal stores, with each >> source element copied once to preserve correctness for forward overlapping >> copies. >> >> `ArrayCopyNonTemporalThreshold` specifies the selection threshold in bytes. >> When unspecified, it is initialized from the detected L3 cache size: >> standard CPUID leaf 4 on Intel and other applicable x86 processors, and >> extended leaf 0x80000006 on AMD/Hygon. An explicit nonzero value overrides >> detection; explicit zero disables the new path; failed detection leaves the >> default at zero. The threshold is included in AOT code-cache configuration >> checks to prevent reuse of stubs generated with a different setting. >> >> The path requires AVX2 and `UseUnalignedLoadStores`. Reference array copies, >> backward copies, and the existing AVX-512 paths retain their current >> implementations. >> >> ### Testing >> JMH benchmarks >> >> Measurements were taken on a two-socket Intel Xeon Platinum 8558 (Sapphire >> Rapids) system with a reported 260 MiB L3 cache per socket, running CentOS >> 8. The benchmark repeatedly copies a source byte array into two alternating >> destination arrays and measures throughput in operations per second at 0.5x, >> 1x, 2x, and 4x the reported per-socket L3 size. >> >> The machine supports AVX-512 and defaults to `UseAVX=3`; these runs >> explicitly use `-XX:UseAVX=2` to exercise the AVX2 implementation, with >> `ArrayCopyNonTemporalThreshold` automatically detected at 272629760 bytes >> (260 MiB). >> >> | Copy size | Baseline, ops/s | Patched, ops/s | Speedup | >> |-----------|----------------:|---------------:|--------:| >> ... > > Larry has updated the pull request incrementally with one additional commit > since the last revision: > > fix the product flag and avoid uintx
Thank you for adding the microbenchmark and the detailed answers. > The UseUnalignedLoadStores check was inherited from the existing AVX2 > arraycopy path. UseAVX >= 2 is sufficient for the instruction set. We kept > the check for consistency and to respect an explicit > -XX:-UseUnalignedLoadStores, but can remove it if preferred. I don't mind it. I just did not understand why it is there. > However, once the copy size exceeds the L3 cache capacity, the entire > destination can no longer remain cached. In this case, the benefit of cache > residency is reduced, making NT stores more likely to be beneficial even with > an immediate read. That makes sense. I guess as a follow-up the AVX-512 path might profit from this heuristic. In order to avoid bit-rot of these, could you add a test that performs large arraycopies with different intrinsic paths forced and result validation? Perhaps there already exists such a test that you could reuse, but having these code paths regularly run in testing is crucial. ------------- PR Comment: https://git.openjdk.org/jdk/pull/33437#issuecomment-6858386077
