On Fri, Mar 28, 2014 at 10:46 PM, Joseph S. Myers <jos...@codesourcery.com> wrote:
> In addition to the failures fixed by this patch, there are many > gcc.dg/vect tests where having additional vector extensions enabled > breaks their expectations; I'm not sure of the best way to handle > those. And you get > > FAIL: gcc.target/i386/avx512f-vfmaddXXXpd-2.c (test for excess errors) > FAIL: gcc.target/i386/avx512f-vfmaddXXXps-2.c (test for excess errors) > FAIL: gcc.target/i386/avx512f-vfmaddsubXXXpd-2.c (test for excess errors) > FAIL: gcc.target/i386/avx512f-vfmaddsubXXXps-2.c (test for excess errors) > FAIL: gcc.target/i386/avx512f-vfmsubXXXpd-2.c (test for excess errors) > FAIL: gcc.target/i386/avx512f-vfmsubXXXps-2.c (test for excess errors) > FAIL: gcc.target/i386/avx512f-vfmsubaddXXXpd-2.c (test for excess errors) > FAIL: gcc.target/i386/avx512f-vfmsubaddXXXps-2.c (test for excess errors) > FAIL: gcc.target/i386/avx512f-vfnmaddXXXpd-2.c (test for excess errors) > FAIL: gcc.target/i386/avx512f-vfnmaddXXXps-2.c (test for excess errors) > FAIL: gcc.target/i386/avx512f-vfnmsubXXXpd-2.c (test for excess errors) > FAIL: gcc.target/i386/avx512f-vfnmsubXXXps-2.c (test for excess errors) > > which are assembler errors such as "operand type mismatch for > `vfmaddpd'" - it looks like the compiler isn't really prepared for the > -mavx512f -mfma4 combination, but I'm not sure what the best way to > handle it is (producing invalid output doesn't seem right, however). Attached patch splits AVX512F modes out of existing FMA patterns. These modes are not supported by patterns that also support FMA4 insns. 2014-03-30 Uros Bizjak <ubiz...@gmail.com> * config/i386/sse.md (FMAMODE_NOVF512): New mode iterator. (<sd_mask_codefor>fma_fmadd_<mode><sd_maskz_name><round_name>): Split out <sd_mask_codefor>fma_fmadd_<VF_512:mode><sd_maskz_name><round_name>. Use FMAMODE_NOVF512 mode iterator. (<sd_mask_codefor>fma_fmsub_<mode><sd_maskz_name><round_name>): Ditto. (<sd_mask_codefor>fma_fnmadd_<mode><sd_maskz_name><round_name>): Ditto. (<sd_mask_codefor>fma_fnmsub_<mode><sd_maskz_name><round_name>): Ditto. (<sd_mask_codefor>fma_fmaddsub_<mode><sd_maskz_name><round_name>): Split out <sd_mask_codefor>fma_fmaddsub_<VF_512:mode><sd_maskz_name><round_name>. Use VF_128_256 mode iterator. (<sd_mask_codefor>fma_fmsubadd_<mode><sd_maskz_name><round_name>): Ditto. Bootstrapped and regression tested on x86_64-pc-linux-gnu {,-m32}. I have also eyeballed the asm - no FMA4 insn was emitted for AVX512F modes. Committed to mainline SVN. Uros.
Index: config/i386/sse.md =================================================================== --- config/i386/sse.md (revision 208944) +++ config/i386/sse.md (working copy) @@ -2712,8 +2712,7 @@ (fma:FMAMODEM (match_operand:FMAMODEM 1 "nonimmediate_operand") (match_operand:FMAMODEM 2 "nonimmediate_operand") - (match_operand:FMAMODEM 3 "nonimmediate_operand")))] - "") + (match_operand:FMAMODEM 3 "nonimmediate_operand")))]) (define_expand "fms<mode>4" [(set (match_operand:FMAMODEM 0 "register_operand") @@ -2720,8 +2719,7 @@ (fma:FMAMODEM (match_operand:FMAMODEM 1 "nonimmediate_operand") (match_operand:FMAMODEM 2 "nonimmediate_operand") - (neg:FMAMODEM (match_operand:FMAMODEM 3 "nonimmediate_operand"))))] - "") + (neg:FMAMODEM (match_operand:FMAMODEM 3 "nonimmediate_operand"))))]) (define_expand "fnma<mode>4" [(set (match_operand:FMAMODEM 0 "register_operand") @@ -2728,8 +2726,7 @@ (fma:FMAMODEM (neg:FMAMODEM (match_operand:FMAMODEM 1 "nonimmediate_operand")) (match_operand:FMAMODEM 2 "nonimmediate_operand") - (match_operand:FMAMODEM 3 "nonimmediate_operand")))] - "") + (match_operand:FMAMODEM 3 "nonimmediate_operand")))]) (define_expand "fnms<mode>4" [(set (match_operand:FMAMODEM 0 "register_operand") @@ -2736,18 +2733,18 @@ (fma:FMAMODEM (neg:FMAMODEM (match_operand:FMAMODEM 1 "nonimmediate_operand")) (match_operand:FMAMODEM 2 "nonimmediate_operand") - (neg:FMAMODEM (match_operand:FMAMODEM 3 "nonimmediate_operand"))))] - "") + (neg:FMAMODEM (match_operand:FMAMODEM 3 "nonimmediate_operand"))))]) ;; The builtins for intrinsics are not constrained by SSE math enabled. -(define_mode_iterator FMAMODE [(SF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F") - (DF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F") - (V4SF "TARGET_FMA || TARGET_FMA4") - (V2DF "TARGET_FMA || TARGET_FMA4") - (V8SF "TARGET_FMA || TARGET_FMA4") - (V4DF "TARGET_FMA || TARGET_FMA4") - (V16SF "TARGET_AVX512F") - (V8DF "TARGET_AVX512F")]) +(define_mode_iterator FMAMODE + [(SF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F") + (DF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F") + (V4SF "TARGET_FMA || TARGET_FMA4") + (V2DF "TARGET_FMA || TARGET_FMA4") + (V8SF "TARGET_FMA || TARGET_FMA4") + (V4DF "TARGET_FMA || TARGET_FMA4") + (V16SF "TARGET_AVX512F") + (V8DF "TARGET_AVX512F")]) (define_expand "fma4i_fmadd_<mode>" [(set (match_operand:FMAMODE 0 "register_operand") @@ -2754,8 +2751,7 @@ (fma:FMAMODE (match_operand:FMAMODE 1 "nonimmediate_operand") (match_operand:FMAMODE 2 "nonimmediate_operand") - (match_operand:FMAMODE 3 "nonimmediate_operand")))] - "") + (match_operand:FMAMODE 3 "nonimmediate_operand")))]) (define_expand "avx512f_fmadd_<mode>_maskz<round_expand_name>" [(match_operand:VF_512 0 "register_operand") @@ -2771,12 +2767,20 @@ DONE; }) +(define_mode_iterator FMAMODE_NOVF512 + [(SF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F") + (DF "TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F") + (V4SF "TARGET_FMA || TARGET_FMA4") + (V2DF "TARGET_FMA || TARGET_FMA4") + (V8SF "TARGET_FMA || TARGET_FMA4") + (V4DF "TARGET_FMA || TARGET_FMA4")]) + (define_insn "<sd_mask_codefor>fma_fmadd_<mode><sd_maskz_name><round_name>" - [(set (match_operand:FMAMODE 0 "register_operand" "=v,v,v,x,x") - (fma:FMAMODE - (match_operand:FMAMODE 1 "<round_nimm_predicate>" "%0,0,v,x,x") - (match_operand:FMAMODE 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m") - (match_operand:FMAMODE 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x")))] + [(set (match_operand:FMAMODE_NOVF512 0 "register_operand" "=v,v,v,x,x") + (fma:FMAMODE_NOVF512 + (match_operand:FMAMODE_NOVF512 1 "<round_nimm_predicate>" "%0,0,v,x,x") + (match_operand:FMAMODE_NOVF512 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m") + (match_operand:FMAMODE_NOVF512 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x")))] "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>" "@ vfmadd132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>} @@ -2788,6 +2792,21 @@ (set_attr "type" "ssemuladd") (set_attr "mode" "<MODE>")]) +(define_insn "<sd_mask_codefor>fma_fmadd_<mode><sd_maskz_name><round_name>" + [(set (match_operand:VF_512 0 "register_operand" "=v,v,v") + (fma:VF_512 + (match_operand:VF_512 1 "<round_nimm_predicate>" "%0,0,v") + (match_operand:VF_512 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>") + (match_operand:VF_512 3 "<round_nimm_predicate>" "v,<round_constraint>,0")))] + "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>" + "@ + vfmadd132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>} + vfmadd213<ssemodesuffix>\t{<round_sd_mask_op4>%3, %2, %0<sd_mask_op4>|%0<sd_mask_op4>, %2, %3<round_sd_mask_op4>} + vfmadd231<ssemodesuffix>\t{<round_sd_mask_op4>%2, %1, %0<sd_mask_op4>|%0<sd_mask_op4>, %1, %2<round_sd_mask_op4>}" + [(set_attr "isa" "fma_avx512f,fma_avx512f,fma_avx512f") + (set_attr "type" "ssemuladd") + (set_attr "mode" "<MODE>")]) + (define_insn "avx512f_fmadd_<mode>_mask<round_name>" [(set (match_operand:VF_512 0 "register_operand" "=v,v") (vec_merge:VF_512 @@ -2821,12 +2840,12 @@ (set_attr "mode" "<MODE>")]) (define_insn "<sd_mask_codefor>fma_fmsub_<mode><sd_maskz_name><round_name>" - [(set (match_operand:FMAMODE 0 "register_operand" "=v,v,v,x,x") - (fma:FMAMODE - (match_operand:FMAMODE 1 "<round_nimm_predicate>" "%0, 0, v, x,x") - (match_operand:FMAMODE 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m") - (neg:FMAMODE - (match_operand:FMAMODE 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x"))))] + [(set (match_operand:FMAMODE_NOVF512 0 "register_operand" "=v,v,v,x,x") + (fma:FMAMODE_NOVF512 + (match_operand:FMAMODE_NOVF512 1 "<round_nimm_predicate>" "%0,0,v,x,x") + (match_operand:FMAMODE_NOVF512 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m") + (neg:FMAMODE_NOVF512 + (match_operand:FMAMODE_NOVF512 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x"))))] "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>" "@ vfmsub132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>} @@ -2838,6 +2857,22 @@ (set_attr "type" "ssemuladd") (set_attr "mode" "<MODE>")]) +(define_insn "<sd_mask_codefor>fma_fmsub_<mode><sd_maskz_name><round_name>" + [(set (match_operand:VF_512 0 "register_operand" "=v,v,v") + (fma:VF_512 + (match_operand:VF_512 1 "<round_nimm_predicate>" "%0,0,v") + (match_operand:VF_512 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>") + (neg:VF_512 + (match_operand:VF_512 3 "<round_nimm_predicate>" "v,<round_constraint>,0"))))] + "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>" + "@ + vfmsub132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>} + vfmsub213<ssemodesuffix>\t{<round_sd_mask_op4>%3, %2, %0<sd_mask_op4>|%0<sd_mask_op4>, %2, %3<round_sd_mask_op4>} + vfmsub231<ssemodesuffix>\t{<round_sd_mask_op4>%2, %1, %0<sd_mask_op4>|%0<sd_mask_op4>, %1, %2<round_sd_mask_op4>}" + [(set_attr "isa" "fma_avx512f,fma_avx512f,fma_avx512f") + (set_attr "type" "ssemuladd") + (set_attr "mode" "<MODE>")]) + (define_insn "avx512f_fmsub_<mode>_mask<round_name>" [(set (match_operand:VF_512 0 "register_operand" "=v,v") (vec_merge:VF_512 @@ -2873,12 +2908,12 @@ (set_attr "mode" "<MODE>")]) (define_insn "<sd_mask_codefor>fma_fnmadd_<mode><sd_maskz_name><round_name>" - [(set (match_operand:FMAMODE 0 "register_operand" "=v,v,v,x,x") - (fma:FMAMODE - (neg:FMAMODE - (match_operand:FMAMODE 1 "<round_nimm_predicate>" "%0,0,v,x,x")) - (match_operand:FMAMODE 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m") - (match_operand:FMAMODE 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x")))] + [(set (match_operand:FMAMODE_NOVF512 0 "register_operand" "=v,v,v,x,x") + (fma:FMAMODE_NOVF512 + (neg:FMAMODE_NOVF512 + (match_operand:FMAMODE_NOVF512 1 "<round_nimm_predicate>" "%0,0,v,x,x")) + (match_operand:FMAMODE_NOVF512 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m") + (match_operand:FMAMODE_NOVF512 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x")))] "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>" "@ vfnmadd132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>} @@ -2890,6 +2925,22 @@ (set_attr "type" "ssemuladd") (set_attr "mode" "<MODE>")]) +(define_insn "<sd_mask_codefor>fma_fnmadd_<mode><sd_maskz_name><round_name>" + [(set (match_operand:VF_512 0 "register_operand" "=v,v,v") + (fma:VF_512 + (neg:VF_512 + (match_operand:VF_512 1 "<round_nimm_predicate>" "%0,0,v")) + (match_operand:VF_512 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>") + (match_operand:VF_512 3 "<round_nimm_predicate>" "v,<round_constraint>,0")))] + "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>" + "@ + vfnmadd132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>} + vfnmadd213<ssemodesuffix>\t{<round_sd_mask_op4>%3, %2, %0<sd_mask_op4>|%0<sd_mask_op4>, %2, %3<round_sd_mask_op4>} + vfnmadd231<ssemodesuffix>\t{<round_sd_mask_op4>%2, %1, %0<sd_mask_op4>|%0<sd_mask_op4>, %1, %2<round_sd_mask_op4>}" + [(set_attr "isa" "fma_avx512f,fma_avx512f,fma_avx512f") + (set_attr "type" "ssemuladd") + (set_attr "mode" "<MODE>")]) + (define_insn "avx512f_fnmadd_<mode>_mask<round_name>" [(set (match_operand:VF_512 0 "register_operand" "=v,v") (vec_merge:VF_512 @@ -2925,13 +2976,13 @@ (set_attr "mode" "<MODE>")]) (define_insn "<sd_mask_codefor>fma_fnmsub_<mode><sd_maskz_name><round_name>" - [(set (match_operand:FMAMODE 0 "register_operand" "=v,v,v,x,x") - (fma:FMAMODE - (neg:FMAMODE - (match_operand:FMAMODE 1 "<round_nimm_predicate>" "%0,0,v,x,x")) - (match_operand:FMAMODE 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m") - (neg:FMAMODE - (match_operand:FMAMODE 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x"))))] + [(set (match_operand:FMAMODE_NOVF512 0 "register_operand" "=v,v,v,x,x") + (fma:FMAMODE_NOVF512 + (neg:FMAMODE_NOVF512 + (match_operand:FMAMODE_NOVF512 1 "<round_nimm_predicate>" "%0,0,v,x,x")) + (match_operand:FMAMODE_NOVF512 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m") + (neg:FMAMODE_NOVF512 + (match_operand:FMAMODE_NOVF512 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x"))))] "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>" "@ vfnmsub132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>} @@ -2943,6 +2994,23 @@ (set_attr "type" "ssemuladd") (set_attr "mode" "<MODE>")]) +(define_insn "<sd_mask_codefor>fma_fnmsub_<mode><sd_maskz_name><round_name>" + [(set (match_operand:VF_512 0 "register_operand" "=v,v,v") + (fma:VF_512 + (neg:VF_512 + (match_operand:VF_512 1 "<round_nimm_predicate>" "%0,0,v")) + (match_operand:VF_512 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>") + (neg:VF_512 + (match_operand:VF_512 3 "<round_nimm_predicate>" "v,<round_constraint>,0"))))] + "<sd_mask_mode512bit_condition> && <round_mode512bit_condition>" + "@ + vfnmsub132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>} + vfnmsub213<ssemodesuffix>\t{<round_sd_mask_op4>%3, %2, %0<sd_mask_op4>|%0<sd_mask_op4>, %2, %3<round_sd_mask_op4>} + vfnmsub231<ssemodesuffix>\t{<round_sd_mask_op4>%2, %1, %0<sd_mask_op4>|%0<sd_mask_op4>, %1, %2<round_sd_mask_op4>}" + [(set_attr "isa" "fma_avx512f,fma_avx512f,fma_avx512f") + (set_attr "type" "ssemuladd") + (set_attr "mode" "<MODE>")]) + (define_insn "avx512f_fnmsub_<mode>_mask<round_name>" [(set (match_operand:VF_512 0 "register_operand" "=v,v") (vec_merge:VF_512 @@ -3014,11 +3082,11 @@ }) (define_insn "<sd_mask_codefor>fma_fmaddsub_<mode><sd_maskz_name><round_name>" - [(set (match_operand:VF 0 "register_operand" "=v,v,v,x,x") - (unspec:VF - [(match_operand:VF 1 "<round_nimm_predicate>" "%0,0,v,x,x") - (match_operand:VF 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m") - (match_operand:VF 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x")] + [(set (match_operand:VF_128_256 0 "register_operand" "=v,v,v,x,x") + (unspec:VF_128_256 + [(match_operand:VF_128_256 1 "<round_nimm_predicate>" "%0,0,v,x,x") + (match_operand:VF_128_256 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m") + (match_operand:VF_128_256 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x")] UNSPEC_FMADDSUB))] "(TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F) && <sd_mask_mode512bit_condition> && <round_mode512bit_condition>" "@ @@ -3031,6 +3099,22 @@ (set_attr "type" "ssemuladd") (set_attr "mode" "<MODE>")]) +(define_insn "<sd_mask_codefor>fma_fmaddsub_<mode><sd_maskz_name><round_name>" + [(set (match_operand:VF_512 0 "register_operand" "=v,v,v") + (unspec:VF_512 + [(match_operand:VF_512 1 "<round_nimm_predicate>" "%0,0,v") + (match_operand:VF_512 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>") + (match_operand:VF_512 3 "<round_nimm_predicate>" "v,<round_constraint>,0")] + UNSPEC_FMADDSUB))] + "TARGET_AVX512F && <sd_mask_mode512bit_condition> && <round_mode512bit_condition>" + "@ + vfmaddsub132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>} + vfmaddsub213<ssemodesuffix>\t{<round_sd_mask_op4>%3, %2, %0<sd_mask_op4>|%0<sd_mask_op4>, %2, %3<round_sd_mask_op4>} + vfmaddsub231<ssemodesuffix>\t{<round_sd_mask_op4>%2, %1, %0<sd_mask_op4>|%0<sd_mask_op4>, %1, %2<round_sd_mask_op4>}" + [(set_attr "isa" "fma_avx512f,fma_avx512f,fma_avx512f") + (set_attr "type" "ssemuladd") + (set_attr "mode" "<MODE>")]) + (define_insn "avx512f_fmaddsub_<mode>_mask<round_name>" [(set (match_operand:VF_512 0 "register_operand" "=v,v") (vec_merge:VF_512 @@ -3066,12 +3150,12 @@ (set_attr "mode" "<MODE>")]) (define_insn "<sd_mask_codefor>fma_fmsubadd_<mode><sd_maskz_name><round_name>" - [(set (match_operand:VF 0 "register_operand" "=v,v,v,x,x") - (unspec:VF - [(match_operand:VF 1 "<round_nimm_predicate>" "%0,0,v,x,x") - (match_operand:VF 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m") - (neg:VF - (match_operand:VF 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x"))] + [(set (match_operand:VF_128_256 0 "register_operand" "=v,v,v,x,x") + (unspec:VF_128_256 + [(match_operand:VF_128_256 1 "<round_nimm_predicate>" "%0,0,v,x,x") + (match_operand:VF_128_256 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>,x,m") + (neg:VF_128_256 + (match_operand:VF_128_256 3 "<round_nimm_predicate>" "v,<round_constraint>,0,xm,x"))] UNSPEC_FMADDSUB))] "(TARGET_FMA || TARGET_FMA4 || TARGET_AVX512F) && <sd_mask_mode512bit_condition> && <round_mode512bit_condition>" "@ @@ -3084,6 +3168,23 @@ (set_attr "type" "ssemuladd") (set_attr "mode" "<MODE>")]) +(define_insn "<sd_mask_codefor>fma_fmsubadd_<mode><sd_maskz_name><round_name>" + [(set (match_operand:VF_512 0 "register_operand" "=v,v,v") + (unspec:VF_512 + [(match_operand:VF_512 1 "<round_nimm_predicate>" "%0,0,v") + (match_operand:VF_512 2 "<round_nimm_predicate>" "<round_constraint>,v,<round_constraint>") + (neg:VF_512 + (match_operand:VF_512 3 "<round_nimm_predicate>" "v,<round_constraint>,0"))] + UNSPEC_FMADDSUB))] + "TARGET_AVX512F && <sd_mask_mode512bit_condition> && <round_mode512bit_condition>" + "@ + vfmsubadd132<ssemodesuffix>\t{<round_sd_mask_op4>%2, %3, %0<sd_mask_op4>|%0<sd_mask_op4>, %3, %2<round_sd_mask_op4>} + vfmsubadd213<ssemodesuffix>\t{<round_sd_mask_op4>%3, %2, %0<sd_mask_op4>|%0<sd_mask_op4>, %2, %3<round_sd_mask_op4>} + vfmsubadd231<ssemodesuffix>\t{<round_sd_mask_op4>%2, %1, %0<sd_mask_op4>|%0<sd_mask_op4>, %1, %2<round_sd_mask_op4>}" + [(set_attr "isa" "fma_avx512f,fma_avx512f,fma_avx512f") + (set_attr "type" "ssemuladd") + (set_attr "mode" "<MODE>")]) + (define_insn "avx512f_fmsubadd_<mode>_mask<round_name>" [(set (match_operand:VF_512 0 "register_operand" "=v,v") (vec_merge:VF_512