https://github.com/lhmouse created https://github.com/llvm/llvm-project/pull/237057
This commits implement the three options for arm64ec-w64-mingw32. `-mlong-double-80` is made default to fix the interoperability bug with x86_64-w64-mingw32. This also introduces an ABI break with old ARM64EC binaries. `-mlong-double-64` is the old behavior that is compativle with aarch64-w64-mingw32 and MSVC. `-mlong-double-128` enables float128 in software, same as x86-64. The new, default option `-mlong-double-80` changes `long double` to a pseudo floating-point type: The type exists in the frontend only; for the middle-end and backend, it's an 128-bit integer that provides storage for a value. The way how the frontend emit IR for an operation varies: 1. The ABI of such `long double` is the same with an 128-bit integer. For x86-64 it's passed and returned in memory. 2. Load/store operations are performed with integer instructions, and are exact. 3. Bitwise operations, such as `fabsl` and `isnan`, are performed directly in the frontend with integer instructions. 4. Arithmetic operations of builtin operators are performed by first converting `long double` to `double`, doing the arithmetic, then converting the result back. This matches how x86-64 emulation works by default on Windows on ARM64 with only 53-bit precision, and should be more efficient than a strict softmath implementation, which the AArch64 backend doesn't support yet. 5. Other builtin functions result in external calls to mingw-w64 CRT directly. These changes have been verified on my ARM64 tablet with a patched mingw-w64 CRT. No regressions are introduced. Assisted-by: DeepSeek V4.1 Flash >From 372b06826209447b18a2872da2d973614653ac28 Mon Sep 17 00:00:00 2001 From: LIU Hao <[email protected]> Date: Thu, 29 Oct 2026 13:18:23 +0800 Subject: [PATCH] [ARM64EC] Implement `-mlong-double-{64,80,128}` options This commits implement the three options for arm64ec-w64-mingw32. `-mlong-double-80` is made default to fix the interoperability bug with x86_64-w64-mingw32. This also introduces an ABI break with old ARM64EC binaries. `-mlong-double-64` is the old behavior that is compativle with aarch64-w64-mingw32 and MSVC. `-mlong-double-128` enables float128 in software, same as x86-64. The new, default option `-mlong-double-80` changes `long double` to a pseudo floating-point type: The type exists in the frontend only; for the middle-end and backend, it's an 128-bit integer that provides storage for a value. The way how the frontend emit IR for an operation varies: 1. The ABI of such `long double` is the same with an 128-bit integer. For x86-64 it's passed and returned in memory. 2. Load/store operations are performed with integer instructions, and are exact. 3. Bitwise operations, such as `fabsl` and `isnan`, are performed directly in the frontend with integer instructions. 4. Arithmetic operations of builtin operators are performed by first converting `long double` to `double`, doing the arithmetic, then converting the result back. This matches how x86-64 emulation works by default on Windows on ARM64 with only 53-bit precision, and should be more efficient than a strict softmath implementation, which the AArch64 backend doesn't support yet. 5. Other builtin functions result in external calls to mingw-w64 CRT directly. These changes have been verified on my ARM64 tablet with a patched mingw-w64 CRT. No regressions are introduced. Assisted-by: DeepSeek V4.1 Flash --- clang/include/clang/Basic/TargetInfo.h | 6 + clang/lib/Basic/Targets/AArch64.cpp | 17 + clang/lib/Basic/Targets/AArch64.h | 5 + clang/lib/CodeGen/CGBuiltin.cpp | 428 ++++++++++++++++++ clang/lib/CodeGen/CGExpr.cpp | 15 +- clang/lib/CodeGen/CGExprComplex.cpp | 107 ++++- clang/lib/CodeGen/CGExprConstant.cpp | 27 +- clang/lib/CodeGen/CGExprScalar.cpp | 383 +++++++++++++++- clang/lib/CodeGen/CodeGenFunction.h | 17 + clang/lib/CodeGen/CodeGenTypes.cpp | 7 + clang/lib/Driver/ToolChains/Clang.cpp | 3 + .../test/CodeGen/arm64ec-long-double-modes.c | 31 ++ .../CodeGen/arm64ec-soft-x87-long-double.c | 160 +++++++ clang/test/Driver/arm64ec-long-double.c | 21 + 14 files changed, 1197 insertions(+), 30 deletions(-) create mode 100644 clang/test/CodeGen/arm64ec-long-double-modes.c create mode 100644 clang/test/CodeGen/arm64ec-soft-x87-long-double.c create mode 100644 clang/test/Driver/arm64ec-long-double.c diff --git a/clang/include/clang/Basic/TargetInfo.h b/clang/include/clang/Basic/TargetInfo.h index a6523e8eaaa437..32aeb5f645b3db 100644 --- a/clang/include/clang/Basic/TargetInfo.h +++ b/clang/include/clang/Basic/TargetInfo.h @@ -709,6 +709,12 @@ class TargetInfo : public TransferrableTargetInfo, /// Whether half args and returns are supported. virtual bool allowHalfArgsAndReturns() const { return HalfArgsAndReturns; } + /// Determine whether the target has the x87-specific `long double` type + /// without a physical x87. The target arm64ec-w64-mingw32 has this property. + /// On such a target, `long double` is an integer image and is not a true + /// floating-point type. + virtual bool hasSoftX87LongDouble() const { return false; } + /// Determine whether the __float128 type is supported on this target. virtual bool hasFloat128Type() const { return HasFloat128; } diff --git a/clang/lib/Basic/Targets/AArch64.cpp b/clang/lib/Basic/Targets/AArch64.cpp index 7ddf9d87ad163e..01da8b879df053 100644 --- a/clang/lib/Basic/Targets/AArch64.cpp +++ b/clang/lib/Basic/Targets/AArch64.cpp @@ -206,6 +206,10 @@ AArch64TargetInfo::AArch64TargetInfo(const llvm::Triple &Triple, this->MCountName = "\01_mcount"; else if (Triple.getOS() == llvm::Triple::UnknownOS) this->MCountName = Triple.isGNUEnvironment() ? "\01_mcount" : "mcount"; + + // This should be compatible with x86-64. + if (Triple.isWindowsArm64EC() && !Triple.isWindowsMSVCEnvironment()) + HasFloat128 = true; } StringRef AArch64TargetInfo::getABI() const { return ABI; } @@ -450,6 +454,11 @@ void AArch64TargetInfo::getTargetDefines(const LangOptions &Opts, if (T.isLFI()) Builder.defineMacro("__LFI__"); + if (HasFloat128) { + Builder.defineMacro("__FLOAT128__"); + Builder.defineMacro("__SIZEOF_FLOAT128__", "16"); + } + // Inline assembly supports AArch64 flag outputs. Builder.defineMacro("__GCC_ASM_FLAG_OUTPUTS__"); @@ -1831,6 +1840,14 @@ WindowsARM64TargetInfo::WindowsARM64TargetInfo(const llvm::Triple &Triple, SizeType = UnsignedLongLong; PtrDiffType = SignedLongLong; IntPtrType = SignedLongLong; + + if (Triple.isWindowsArm64EC() && !Triple.isWindowsMSVCEnvironment()) { + // arm64ec-w64-mingw32 should be interoperable with x86_64-w64-mingw32, + // which by default has `long double` as the x87-specific extended + // precision format. + LongDoubleWidth = LongDoubleAlign = SuitableAlign = 128; + LongDoubleFormat = &llvm::APFloat::x87DoubleExtended(); + } } TargetInfo::BuiltinVaListKind diff --git a/clang/lib/Basic/Targets/AArch64.h b/clang/lib/Basic/Targets/AArch64.h index 00e84cc4864399..c17a092265ddb1 100644 --- a/clang/lib/Basic/Targets/AArch64.h +++ b/clang/lib/Basic/Targets/AArch64.h @@ -295,6 +295,11 @@ class LLVM_LIBRARY_VISIBILITY WindowsARM64TargetInfo BuiltinVaListKind getBuiltinVaListKind() const override; CallingConvCheckResult checkCallingConvention(CallingConv CC) const override; + + bool hasSoftX87LongDouble() const override { + return Triple.isWindowsArm64EC() && + LongDoubleFormat == &llvm::APFloat::x87DoubleExtended(); + } }; // Windows ARM, MS (C++) ABI diff --git a/clang/lib/CodeGen/CGBuiltin.cpp b/clang/lib/CodeGen/CGBuiltin.cpp index e154b9e752fd5c..9b3b28c85a03e3 100644 --- a/clang/lib/CodeGen/CGBuiltin.cpp +++ b/clang/lib/CodeGen/CGBuiltin.cpp @@ -2714,12 +2714,440 @@ static void ClearPadding(CodeGenFunction &CGF, Address Src, } } +/// Lower builtins that operate on ARM64EC's software x87 long double. The +/// ones that are required to be exact (fabs, copysign, signbit, isnan, isinf, +/// isfinite, isnormal, fpclassify) are implemented with integer operations on +/// the image itself; everything else is evaluated in `double` exactly like the +/// C library does for its own long double entry points. Return false when the +/// builtin does not involve such a value. +bool CodeGenFunction::EmitSoftX87LongDoubleBuiltin(unsigned BuiltinID, + const CallExpr *E, + RValue &Result) { + // The 'nan' and 'inf' families produce a `long double`, so they do not + // consume one but do need the integer image constructed below: they are the + // only builtins that yield a constant. Match the bare name, since some of + // them are also reachable with a `__builtin_` prefix. + std::string Name = getContext().BuiltinInfo.getName(BuiltinID); + StringRef BaseName = Name; + BaseName.consume_front("__builtin_"); + bool ProducesLongDouble = false; + for (const char *Str : {"nanl", "nansl", "infl", "huge_vall"}) + ProducesLongDouble |= BaseName == Str; + + // Everything below rewrites a call into its binary64 equivalent, which only + // makes sense when a `long double` is *consumed*. + bool ConsumesLongDouble = false; + for (const Expr *Arg : E->arguments()) { + QualType Ty = Arg->getType(); + if (const auto *CTy = Ty->getAs<ComplexType>()) + Ty = CTy->getElementType(); + ConsumesLongDouble |= Ty->isSpecificBuiltinType(BuiltinType::LongDouble); + } + + if (!ProducesLongDouble && !ConsumesLongDouble) + return false; + + // `fabsl`: Set the sign bit to zero. + if (BaseName == "fabsl") { + if (!E->getType()->isSpecificBuiltinType(BuiltinType::LongDouble)) + return false; + llvm::Value *V = EmitScalarExpr(E->getArg(0)); + llvm::Constant *MagnitudeMask = llvm::ConstantInt::get( + V->getType(), ~llvm::APInt::getOneBitSet(128, 79)); + Result = RValue::get(Builder.CreateAnd(V, MagnitudeMask)); + return true; + } + + // `copysignl`: Take the magnitude from the first operand and the sign bit + // from the second one. + if (BaseName == "copysignl") { + if (!E->getType()->isSpecificBuiltinType(BuiltinType::LongDouble)) + return false; + llvm::Value *X = EmitScalarExpr(E->getArg(0)); + llvm::Value *Y = EmitScalarExpr(E->getArg(1)); + llvm::Constant *MagnitudeMask = llvm::ConstantInt::get( + X->getType(), ~llvm::APInt::getOneBitSet(128, 79)); + llvm::Value *Magnitude = Builder.CreateAnd(X, MagnitudeMask); + llvm::Value *Sign = Builder.CreateAnd(Y, Builder.CreateNot(MagnitudeMask)); + Result = RValue::get(Builder.CreateOr(Magnitude, Sign)); + return true; + } + + // `signbit`: Take the sign bit and normalize it to boolean. + if (Name == "__builtin_signbit" || BaseName == "signbitl") { + if (!E->getArg(0)->getType()->isSpecificBuiltinType( + BuiltinType::LongDouble)) + return false; + llvm::Value *V = EmitScalarExpr(E->getArg(0)); + llvm::Value *Sign = + Builder.CreateTrunc(Builder.CreateLShr(V, 79), Builder.getInt1Ty()); + Result = RValue::get(Builder.CreateZExt(Sign, ConvertType(E->getType()))); + return true; + } + + // `__builtin_isnan`: Check if the value is a NaN. An unnormal or pseudo + // denormal image is undefined behavior, since it will never be produced in a + // constant expression. + if (Name == "__builtin_isnan") { + if (!E->getArg(0)->getType()->isSpecificBuiltinType( + BuiltinType::LongDouble)) + return false; + llvm::Value *V = EmitScalarExpr(E->getArg(0)); + llvm::Constant *MagnitudeMask = llvm::ConstantInt::get( + V->getType(), ~llvm::APInt::getOneBitSet(128, 79)); + llvm::Value *Magnitude = Builder.CreateAnd(V, MagnitudeMask); + llvm::APInt InfinityImage(128, 0x8000000000000000ull); + InfinityImage |= llvm::APInt(128, 0x7fffull) << 64; + llvm::Constant *Infinity = + llvm::ConstantInt::get(V->getType(), InfinityImage); + llvm::Value *IsNan = Builder.CreateICmpUGT(Magnitude, Infinity); + Result = RValue::get(Builder.CreateZExt(IsNan, ConvertType(E->getType()))); + return true; + } + + // `__builtin_isinf`: Check if the value is a (positive or negative) infinity. + if (Name == "__builtin_isinf") { + if (!E->getArg(0)->getType()->isSpecificBuiltinType( + BuiltinType::LongDouble)) + return false; + llvm::Value *V = EmitScalarExpr(E->getArg(0)); + llvm::Constant *MagnitudeMask = llvm::ConstantInt::get( + V->getType(), ~llvm::APInt::getOneBitSet(128, 79)); + llvm::Value *Magnitude = Builder.CreateAnd(V, MagnitudeMask); + llvm::APInt InfinityImage(128, 0x8000000000000000ull); + InfinityImage |= llvm::APInt(128, 0x7fffull) << 64; + llvm::Constant *Infinity = + llvm::ConstantInt::get(V->getType(), InfinityImage); + llvm::Value *IsInf = Builder.CreateICmpEQ(Magnitude, Infinity); + Result = RValue::get(Builder.CreateZExt(IsInf, ConvertType(E->getType()))); + return true; + } + + // `__builtin_isinf_sign`: Return 1 or -1 depending on the sign of an + // infinity, and zero for everything else. + if (Name == "__builtin_isinf_sign") { + if (!E->getArg(0)->getType()->isSpecificBuiltinType( + BuiltinType::LongDouble)) + return false; + llvm::Value *V = EmitScalarExpr(E->getArg(0)); + llvm::Constant *MagnitudeMask = llvm::ConstantInt::get( + V->getType(), ~llvm::APInt::getOneBitSet(128, 79)); + llvm::Value *Magnitude = Builder.CreateAnd(V, MagnitudeMask); + llvm::APInt InfinityImage(128, 0x8000000000000000ull); + InfinityImage |= llvm::APInt(128, 0x7fffull) << 64; + llvm::Constant *Infinity = + llvm::ConstantInt::get(V->getType(), InfinityImage); + llvm::Type *IntTy = ConvertType(E->getType()); + llvm::Value *Sign = + Builder.CreateTrunc(Builder.CreateLShr(V, 79), Builder.getInt1Ty()); + llvm::Value *Signed = Builder.CreateSelect( + Sign, llvm::ConstantInt::get(IntTy, -1, /*isSigned=*/true), + llvm::ConstantInt::get(IntTy, 1)); + llvm::Value *IsInf = Builder.CreateICmpEQ(Magnitude, Infinity); + Result = RValue::get( + Builder.CreateSelect(IsInf, Signed, llvm::ConstantInt::get(IntTy, 0))); + return true; + } + + // `__builtin_isfinite`: Check if the value is neither an infinity nor a NaN. + if (Name == "__builtin_isfinite") { + if (!E->getArg(0)->getType()->isSpecificBuiltinType( + BuiltinType::LongDouble)) + return false; + llvm::Value *V = EmitScalarExpr(E->getArg(0)); + llvm::Constant *MagnitudeMask = llvm::ConstantInt::get( + V->getType(), ~llvm::APInt::getOneBitSet(128, 79)); + llvm::Value *Magnitude = Builder.CreateAnd(V, MagnitudeMask); + llvm::APInt InfinityImage(128, 0x8000000000000000ull); + InfinityImage |= llvm::APInt(128, 0x7fffull) << 64; + llvm::Constant *Infinity = + llvm::ConstantInt::get(V->getType(), InfinityImage); + llvm::Value *IsFinite = Builder.CreateICmpULT(Magnitude, Infinity); + Result = + RValue::get(Builder.CreateZExt(IsFinite, ConvertType(E->getType()))); + return true; + } + + // `__builtin_isnormal`: Check if the value is finite and nonzero. A + // canonical image has the explicit integer bit set when the exponent is + // nonzero, so the exponent alone decides; an unnormal is undefined behaviour. + if (Name == "__builtin_isnormal") { + if (!E->getArg(0)->getType()->isSpecificBuiltinType( + BuiltinType::LongDouble)) + return false; + llvm::Value *V = EmitScalarExpr(E->getArg(0)); + llvm::Value *Exp = Builder.CreateAnd( + Builder.CreateTrunc(Builder.CreateLShr(V, 64), Builder.getInt16Ty()), + Builder.getInt16(0x7FFF)); + llvm::Value *IsNormal = + Builder.CreateAnd(Builder.CreateICmpNE(Exp, Builder.getInt16(0)), + Builder.CreateICmpNE(Exp, Builder.getInt16(0x7FFF))); + Result = + RValue::get(Builder.CreateZExt(IsNormal, ConvertType(E->getType()))); + return true; + } + + // `__builtin_fpclassify`: Pick the first five arguments that correspond to a + // NaN, an infinity, a normal, a subnormal and a zero respectively. + if (Name == "__builtin_fpclassify") { + if (!E->getArg(5)->getType()->isSpecificBuiltinType( + BuiltinType::LongDouble)) + return false; + llvm::Value *V = EmitScalarExpr(E->getArg(5)); + llvm::Value *Exp = Builder.CreateAnd( + Builder.CreateTrunc(Builder.CreateLShr(V, 64), Builder.getInt16Ty()), + Builder.getInt16(0x7FFF)); + llvm::Value *Man = Builder.CreateTrunc(V, Builder.getInt64Ty()); + llvm::Value *ExpZero = Builder.CreateICmpEQ(Exp, Builder.getInt16(0)); + llvm::Value *ExpAllOnes = + Builder.CreateICmpEQ(Exp, Builder.getInt16(0x7FFF)); + llvm::Value *ManZero = Builder.CreateICmpEQ(Man, Builder.getInt64(0)); + // Infinity is the all-ones exponent with nothing but the integer bit set; + // any other value with that exponent is a NaN. An unnormal, whose integer + // bit is clear, is undefined behaviour. + llvm::Value *IsInf = Builder.CreateAnd( + ExpAllOnes, + Builder.CreateICmpEQ(Man, Builder.getInt64(0x8000000000000000ull))); + llvm::Value *IsNaN = + Builder.CreateAnd(ExpAllOnes, Builder.CreateNot(IsInf)); + llvm::Value *IsNormal = Builder.CreateAnd(Builder.CreateNot(ExpZero), + Builder.CreateNot(ExpAllOnes)); + llvm::Value *IsSubnormal = + Builder.CreateAnd(ExpZero, Builder.CreateNot(ManZero)); + llvm::Value *R = EmitScalarExpr(E->getArg(4)); // zero + R = Builder.CreateSelect(IsSubnormal, EmitScalarExpr(E->getArg(3)), R); + R = Builder.CreateSelect(IsNormal, EmitScalarExpr(E->getArg(2)), R); + R = Builder.CreateSelect(IsInf, EmitScalarExpr(E->getArg(1)), R); + R = Builder.CreateSelect(IsNaN, EmitScalarExpr(E->getArg(0)), R); + Result = RValue::get(R); + return true; + } + + // `infl` and `huge_vall`: Infinity has the significand 0x8000000000000000 + // with an all-ones exponent, and the six padding bytes are zeroes. They + // take no argument at all, so there is nothing to check. + if (BaseName == "infl" || BaseName == "huge_vall") { + llvm::APInt Image(128, 0x8000000000000000ull); + Image |= llvm::APInt(128, 0x7fffull) << 64; + llvm::Type *ImageTy = llvm::IntegerType::get(getLLVMContext(), 128); + Result = RValue::get(llvm::ConstantInt::get(ImageTy, Image)); + return true; + } + + // `nanl` and `nansl`: The quiet NaN has the significand 0xc000000000000000 + // and the signalling one has 0xa000000000000000, both with an all-ones + // exponent. The payload of the tag argument is not propagated, but the + // argument is still evaluated, since it may have side effects. + if (BaseName == "nanl" || BaseName == "nansl") { + if (!E->getArg(0)->getType()->isPointerType()) + return false; + EmitScalarExpr(E->getArg(0)); + llvm::APInt Image(128, BaseName == "nansl" ? 0xa000000000000000ull + : 0xc000000000000000ull); + Image |= llvm::APInt(128, 0x7fffull) << 64; + llvm::Type *ImageTy = llvm::IntegerType::get(getLLVMContext(), 128); + Result = RValue::get(llvm::ConstantInt::get(ImageTy, Image)); + return true; + } + + // `__builtin_isunordered`: True if either operand is a NaN. + // `__builtin_isgreater`, `__builtin_isgreaterequal`, `__builtin_isless` and + // `__builtin_islessequal`: Perform ordered comparisons of two operands. + // `__builtin_islessgreater`: True if the operands are ordered and unequal; + // that is, if either of `x < y` and `x > y` holds. + struct BuiltinComparison { + const char *Name; + CmpInst::Predicate Predicate; + }; + static constexpr BuiltinComparison BuiltinComparisons[] = { + {"__builtin_isunordered", CmpInst::FCMP_UNO}, + {"__builtin_isgreater", CmpInst::FCMP_OGT}, + {"__builtin_isgreaterequal", CmpInst::FCMP_OGE}, + {"__builtin_isless", CmpInst::FCMP_OLT}, + {"__builtin_islessequal", CmpInst::FCMP_OLE}, + {"__builtin_islessgreater", CmpInst::FCMP_ONE}, + }; + for (const auto &BC : BuiltinComparisons) + if (Name == BC.Name) { + if (!E->getArg(0)->getType()->isSpecificBuiltinType( + BuiltinType::LongDouble) || + !E->getArg(1)->getType()->isSpecificBuiltinType( + BuiltinType::LongDouble)) + return false; + llvm::Value *A = EmitScalarExpr(E->getArg(0)); + llvm::Value *B = EmitScalarExpr(E->getArg(1)); + llvm::Value *Cmp = EmitSoftX87LongDoubleComparison(A, B, BC.Predicate); + Result = RValue::get(Builder.CreateZExt(Cmp, ConvertType(E->getType()))); + return true; + } + + // `__builtin_isfpclass`: Test the image against a mask of class bits. The + // second argument is an integer constant expression built out of the + // `__FPCLASS_*` values, whose bits number the classes in this order: + // signalling NaN, quiet NaN, -inf, negative normal, negative subnormal, + // -0, +0, positive subnormal, positive normal and +inf. So the value only + // has to be mapped to the ordinal of the class it belongs to, and the bit + // of the mask with that index decides. + if (Name == "__builtin_isfpclass") { + if (!E->getArg(0)->getType()->isSpecificBuiltinType( + BuiltinType::LongDouble)) + return false; + Expr::EvalResult MaskResult; + if (!E->getArg(1)->EvaluateAsInt(MaskResult, getContext())) + return false; + const uint64_t Mask = MaskResult.Val.getInt().getLimitedValue(); + llvm::Value *V = EmitScalarExpr(E->getArg(0)); + llvm::Value *Exp = Builder.CreateAnd( + Builder.CreateTrunc(Builder.CreateLShr(V, 64), Builder.getInt16Ty()), + Builder.getInt16(0x7FFF)); + llvm::Value *Man = Builder.CreateTrunc(V, Builder.getInt64Ty()); + llvm::Value *Negative = + Builder.CreateTrunc(Builder.CreateLShr(V, 79), Builder.getInt1Ty()); + llvm::Value *ExpZero = Builder.CreateICmpEQ(Exp, Builder.getInt16(0)); + llvm::Value *ExpAllOnes = + Builder.CreateICmpEQ(Exp, Builder.getInt16(0x7FFF)); + llvm::Value *ManZero = Builder.CreateICmpEQ(Man, Builder.getInt64(0)); + // Infinity is the all-ones exponent with nothing but the integer bit + // set, and a NaN is any other image with that exponent, quiet when bit + // 62 of the significand is set. An unnormal, whose integer bit is + // clear, is undefined behaviour. + llvm::Value *Infinity = Builder.CreateAnd( + ExpAllOnes, + Builder.CreateICmpEQ(Man, Builder.getInt64(0x8000000000000000ull))); + llvm::Value *Quiet = Builder.CreateICmpNE( + Builder.CreateAnd(Man, Builder.getInt64(0x4000000000000000ull)), + Builder.getInt64(0)); + // The magnitudes are numbered 2 to 5 from the largest downwards, and the + // positive classes continue the sequence in the reverse order, which makes + // them `11 - Magnitude`. A NaN takes 0 or 1 instead of a magnitude. + llvm::Value *Magnitude = Builder.getInt64(2); + Magnitude = Builder.CreateSelect(Builder.CreateNot(ExpAllOnes), + Builder.getInt64(3), Magnitude); + Magnitude = Builder.CreateSelect(ExpZero, Builder.getInt64(4), Magnitude); + Magnitude = Builder.CreateSelect(Builder.CreateAnd(ExpZero, ManZero), + Builder.getInt64(5), Magnitude); + Magnitude = Builder.CreateSelect( + Negative, Magnitude, + Builder.CreateSub(Builder.getInt64(11), Magnitude)); + llvm::Value *Ordinal = + Builder.CreateSelect(Quiet, Builder.getInt64(1), Builder.getInt64(0)); + Ordinal = Builder.CreateSelect( + Builder.CreateAnd(ExpAllOnes, Builder.CreateNot(Infinity)), Ordinal, + Magnitude); + llvm::Value *Bit = + Builder.CreateAnd(Builder.CreateLShr(Builder.getInt64(Mask), Ordinal), + Builder.getInt64(1)); + Result = RValue::get( + Builder.CreateZExt(Builder.CreateICmpNE(Bit, Builder.getInt64(0)), + ConvertType(E->getType()))); + return true; + } + + // `__builtin_complex`: Assemble a complex value out of a real and an + // imaginary part, which become the low and high halves of the image pair. + // Its operands are "custom type checked", so they are not promoted: both + // have the same real floating type already, and they have to be the `long + // double` one for this lowering to apply. + if (Name == "__builtin_complex") { + if (!E->getArg(0)->getType()->isSpecificBuiltinType( + BuiltinType::LongDouble) || + !E->getArg(1)->getType()->isSpecificBuiltinType( + BuiltinType::LongDouble)) + return false; + llvm::Value *Real = EmitScalarExpr(E->getArg(0)); + llvm::Value *Imag = EmitScalarExpr(E->getArg(1)); + Result = RValue::getComplex(Real, Imag); + return true; + } + + // `conjl`: Conjugation only flips the sign bit of the imaginary part, which + // is the high half of the image pair, and leaves the real part alone. + if (BaseName == "conjl") { + const auto *CTy = E->getArg(0)->getType()->getAs<ComplexType>(); + if (!CTy || + !CTy->getElementType()->isSpecificBuiltinType(BuiltinType::LongDouble)) + return false; + ComplexPairTy Z = EmitComplexExpr(E->getArg(0)); + llvm::Constant *SignMask = llvm::ConstantInt::get( + Z.second->getType(), llvm::APInt::getOneBitSet(128, 79)); + llvm::Value *Imag = Builder.CreateXor(Z.second, SignMask); + Result = RValue::getComplex(Z.first, Imag); + return true; + } + + // `creall` and `cimagl`: They only pick the low or the high half of the + // image pair, which is the real or the imaginary part. + if (BaseName == "creall" || BaseName == "cimagl") { + const auto *CTy = E->getArg(0)->getType()->getAs<ComplexType>(); + if (!CTy || + !CTy->getElementType()->isSpecificBuiltinType(BuiltinType::LongDouble)) + return false; + ComplexPairTy Z = EmitComplexExpr(E->getArg(0)); + Result = RValue::get(BaseName == "creall" ? Z.first : Z.second); + return true; + } + + // Everything else is a function of the C library, which on this target works + // on the 80-bit image itself: a `long double` *is* the image, so its own + // entry point takes and returns it untouched, and the builtin is nothing + // but a call to that entry point. Nothing is evaluated in `double` and + // rounded back, so that not a single bit is lost. The variadic `printf` + // and `scanf` families go the same way: their ellipsis arguments are + // forwarded untouched, since the x64 calling convention that ARM64EC + // reproduces passes a 16-byte `long double` by reference. + + // `powil` is the only one left that has no entry point of that spelling in + // the C library: there is no `powi` function, and `llvm.powi` only accepts + // real floating-point values. mingw-w64 CRT provides `__powil` for it. + if (Name == "__builtin_powil") { + llvm::Value *X = EmitScalarExpr(E->getArg(0)); + llvm::Value *N = EmitScalarExpr(E->getArg(1)); + llvm::FunctionType *FT = llvm::FunctionType::get( + ConvertType(E->getType()), {X->getType(), N->getType()}, false); + Result = RValue::get( + Builder.CreateCall(CGM.CreateRuntimeFunction(FT, "__powil"), {X, N})); + return true; + } + + // Both spellings arrive here: `__builtin_sinl` and the `sinl` that the + // header declares are separate builtin identifiers, and they resolve to the + // same entry point. + const FunctionDecl *FD = E->getDirectCallee(); + if (FD && getContext().BuiltinInfo.isLibFunction(BuiltinID)) { + Result = + emitLibraryCall(*this, FD, E, CGM.getBuiltinLibFunction(FD, BuiltinID)); + return true; + } + + if (FD && getContext().BuiltinInfo.isPredefinedLibFunction(BuiltinID)) { + Result = emitLibraryCall(*this, FD, E, CGM.getRawFunctionPointer(FD)); + return true; + } + + return false; +} + RValue CodeGenFunction::EmitBuiltinExpr(const GlobalDecl GD, unsigned BuiltinID, const CallExpr *E, ReturnValueSlot ReturnValue) { assert(!getContext().BuiltinInfo.isImmediate(BuiltinID) && "Should not codegen for consteval builtins"); + // A software x87 long double is a 16-byte integer image and is not a true + // floating-point value. We must take extreme caution not to pass such a + // value to constant folding below. In no case shall builtins that take or + // return such values be lowered to floating-point intrinsics. + // The C++ standard library has some builtins (`std::move`, `std::forward` + // and friends) that are declared in the `std` namespace as well; but those + // are lowered to a plain address computation and never touch the value, so + // they are none of our business. + if (getTarget().hasSoftX87LongDouble() && + !getContext().BuiltinInfo.isInStdNamespace(BuiltinID)) { + RValue X87Result; + if (EmitSoftX87LongDoubleBuiltin(BuiltinID, E, X87Result)) + return X87Result; + } + const FunctionDecl *FD = GD.getDecl()->getAsFunction(); // See if we can constant fold this builtin. If so, don't emit it at all. // TODO: Extend this handling to all builtin calls that we can constant-fold. diff --git a/clang/lib/CodeGen/CGExpr.cpp b/clang/lib/CodeGen/CGExpr.cpp index 98133ecf1b48fc..aec31e114458a7 100644 --- a/clang/lib/CodeGen/CGExpr.cpp +++ b/clang/lib/CodeGen/CGExpr.cpp @@ -1424,14 +1424,25 @@ EmitComplexPrePostIncDec(const UnaryOperator *E, LValue LV, ComplexPairTy InVal = EmitLoadOfComplex(LV, E->getExprLoc()); llvm::Value *NextVal; - if (isa<llvm::IntegerType>(InVal.first->getType())) { + QualType ElemTy = E->getType()->castAs<ComplexType>()->getElementType(); + if (getTarget().hasSoftX87LongDouble() && + ElemTy->isSpecificBuiltinType(BuiltinType::LongDouble)) { + SourceLocation Loc = E->getExprLoc(); + + // Convert the real part to `double`, add the inc/dec, then convert it + // back to `long double`. + llvm::Value *Real = EmitX87LongDoubleToDouble(InVal.first, Loc); + llvm::Value *Amount = llvm::ConstantFP::get( + llvm::Type::getDoubleTy(getLLVMContext()), isInc ? 1.0 : -1.0); + NextVal = EmitDoubleToX87LongDouble( + Builder.CreateFAdd(Real, Amount, isInc ? "inc" : "dec"), Loc); + } else if (isa<llvm::IntegerType>(InVal.first->getType())) { uint64_t AmountVal = isInc ? 1 : -1; NextVal = llvm::ConstantInt::get(InVal.first->getType(), AmountVal, true); // Add the inc/dec to the real part. NextVal = Builder.CreateAdd(InVal.first, NextVal, isInc ? "inc" : "dec"); } else { - QualType ElemTy = E->getType()->castAs<ComplexType>()->getElementType(); llvm::APFloat FVal(getContext().getFloatTypeSemantics(ElemTy), 1); if (!isInc) FVal.changeSign(); diff --git a/clang/lib/CodeGen/CGExprComplex.cpp b/clang/lib/CodeGen/CGExprComplex.cpp index 30d693fb371cb6..54d8c870390b45 100644 --- a/clang/lib/CodeGen/CGExprComplex.cpp +++ b/clang/lib/CodeGen/CGExprComplex.cpp @@ -258,6 +258,7 @@ class ComplexExprEmitter ComplexPairTy RHS; QualType Ty; // Computation Type. FPOptions FPFeatures; + SourceLocation Loc; // Location of the expression. }; BinOpInfo EmitBinOps(const BinaryOperator *E, @@ -692,7 +693,14 @@ ComplexPairTy ComplexExprEmitter::VisitMinus(const UnaryOperator *E, Op = Visit(E->getSubExpr()); llvm::Value *ResR, *ResI; - if (Op.first->getType()->isFloatingPointTy()) { + QualType ElemTy = E->getType()->castAs<ComplexType>()->getElementType(); + if (CGF.getTarget().hasSoftX87LongDouble() && + ElemTy->isSpecificBuiltinType(BuiltinType::LongDouble)) { + llvm::Constant *SignMask = llvm::ConstantInt::get( + CGF.getLLVMContext(), llvm::APInt::getOneBitSet(128, 79)); + ResR = Builder.CreateXor(Op.first, SignMask, "neg.r"); + ResI = Builder.CreateXor(Op.second, SignMask, "neg.i"); + } else if (Op.first->getType()->isFloatingPointTy()) { ResR = Builder.CreateFNeg(Op.first, "neg.r"); ResI = Builder.CreateFNeg(Op.second, "neg.i"); } else { @@ -708,7 +716,13 @@ ComplexPairTy ComplexExprEmitter::VisitUnaryNot(const UnaryOperator *E) { // ~(a+ib) = a + i*-b ComplexPairTy Op = Visit(E->getSubExpr()); llvm::Value *ResI; - if (Op.second->getType()->isFloatingPointTy()) + QualType ElemTy = E->getType()->castAs<ComplexType>()->getElementType(); + if (CGF.getTarget().hasSoftX87LongDouble() && + ElemTy->isSpecificBuiltinType(BuiltinType::LongDouble)) { + llvm::Constant *SignMask = llvm::ConstantInt::get( + CGF.getLLVMContext(), llvm::APInt::getOneBitSet(128, 79)); + ResI = Builder.CreateXor(Op.second, SignMask, "conj.i"); + } else if (Op.second->getType()->isFloatingPointTy()) ResI = Builder.CreateFNeg(Op.second, "conj.i"); else ResI = Builder.CreateNeg(Op.second, "conj.i"); @@ -718,8 +732,26 @@ ComplexPairTy ComplexExprEmitter::VisitUnaryNot(const UnaryOperator *E) { ComplexPairTy ComplexExprEmitter::EmitBinAdd(const BinOpInfo &Op) { llvm::Value *ResR, *ResI; - - if (Op.LHS.first->getType()->isFloatingPointTy()) { + QualType ElemTy = Op.Ty->castAs<ComplexType>()->getElementType(); + if (CGF.getTarget().hasSoftX87LongDouble() && + ElemTy->isSpecificBuiltinType(BuiltinType::LongDouble)) { + // When using software x87 `long double`, both the real and imaginary parts + // are converted to `double` for calculation, and the result parts are + // converted back. + BinOpInfo DblOp = Op; + DblOp.LHS = ComplexPairTy( + CGF.EmitX87LongDoubleToDouble(Op.LHS.first, Op.Loc), nullptr); + if (Op.LHS.second) + DblOp.LHS.second = CGF.EmitX87LongDoubleToDouble(Op.LHS.second, Op.Loc); + DblOp.RHS = ComplexPairTy( + CGF.EmitX87LongDoubleToDouble(Op.RHS.first, Op.Loc), nullptr); + if (Op.RHS.second) + DblOp.RHS.second = CGF.EmitX87LongDoubleToDouble(Op.RHS.second, Op.Loc); + DblOp.Ty = CGF.getContext().getComplexType(CGF.getContext().DoubleTy); + ComplexPairTy DblRes = EmitBinAdd(DblOp); + return ComplexPairTy(CGF.EmitDoubleToX87LongDouble(DblRes.first, Op.Loc), + CGF.EmitDoubleToX87LongDouble(DblRes.second, Op.Loc)); + } else if (Op.LHS.first->getType()->isFloatingPointTy()) { CodeGenFunction::CGFPOptionsRAII FPOptsRAII(CGF, Op.FPFeatures); ResR = Builder.CreateFAdd(Op.LHS.first, Op.RHS.first, "add.r"); if (Op.LHS.second && Op.RHS.second) @@ -738,6 +770,7 @@ ComplexPairTy ComplexExprEmitter::EmitBinAdd(const BinOpInfo &Op) { ComplexPairTy ComplexExprEmitter::EmitBinSub(const BinOpInfo &Op) { llvm::Value *ResR, *ResI; + QualType ElemTy = Op.Ty->castAs<ComplexType>()->getElementType(); if (Op.LHS.first->getType()->isFloatingPointTy()) { CodeGenFunction::CGFPOptionsRAII FPOptsRAII(CGF, Op.FPFeatures); ResR = Builder.CreateFSub(Op.LHS.first, Op.RHS.first, "sub.r"); @@ -747,6 +780,24 @@ ComplexPairTy ComplexExprEmitter::EmitBinSub(const BinOpInfo &Op) { ResI = Op.LHS.second ? Op.LHS.second : Builder.CreateFNeg(Op.RHS.second, "sub.i"); assert(ResI && "Only one operand may be real!"); + } else if (CGF.getTarget().hasSoftX87LongDouble() && + ElemTy->isSpecificBuiltinType(BuiltinType::LongDouble)) { + // When using software x87 `long double`, both the real and imaginary parts + // are converted to `double` for calculation, and the result parts are + // converted back. + BinOpInfo DblOp = Op; + DblOp.LHS = ComplexPairTy( + CGF.EmitX87LongDoubleToDouble(Op.LHS.first, Op.Loc), nullptr); + if (Op.LHS.second) + DblOp.LHS.second = CGF.EmitX87LongDoubleToDouble(Op.LHS.second, Op.Loc); + DblOp.RHS = ComplexPairTy( + CGF.EmitX87LongDoubleToDouble(Op.RHS.first, Op.Loc), nullptr); + if (Op.RHS.second) + DblOp.RHS.second = CGF.EmitX87LongDoubleToDouble(Op.RHS.second, Op.Loc); + DblOp.Ty = CGF.getContext().getComplexType(CGF.getContext().DoubleTy); + ComplexPairTy DblRes = EmitBinSub(DblOp); + return ComplexPairTy(CGF.EmitDoubleToX87LongDouble(DblRes.first, Op.Loc), + CGF.EmitDoubleToX87LongDouble(DblRes.second, Op.Loc)); } else { ResR = Builder.CreateSub(Op.LHS.first, Op.RHS.first, "sub.r"); assert(Op.LHS.second && Op.RHS.second && @@ -823,9 +874,30 @@ static StringRef getComplexMultiplyLibCallName(const llvm::Triple &T, ComplexPairTy ComplexExprEmitter::EmitBinMul(const BinOpInfo &Op) { using llvm::Value; Value *ResR, *ResI; + QualType ElemTy = Op.Ty->castAs<ComplexType>()->getElementType(); llvm::MDBuilder MDHelper(CGF.getLLVMContext()); - if (Op.LHS.first->getType()->isFloatingPointTy()) { + if (CGF.getTarget().hasSoftX87LongDouble() && + ElemTy->isSpecificBuiltinType(BuiltinType::LongDouble)) { + // When using software x87 `long double`, both the real and imaginary parts + // are converted to `double` for calculation, and the result parts are + // converted back. + BinOpInfo DblOp = Op; + DblOp.LHS = ComplexPairTy( + CGF.EmitX87LongDoubleToDouble(Op.LHS.first, Op.Loc), nullptr); + if (Op.LHS.second) + DblOp.LHS.second = CGF.EmitX87LongDoubleToDouble(Op.LHS.second, Op.Loc); + DblOp.RHS = ComplexPairTy( + CGF.EmitX87LongDoubleToDouble(Op.RHS.first, Op.Loc), nullptr); + if (Op.RHS.second) + DblOp.RHS.second = CGF.EmitX87LongDoubleToDouble(Op.RHS.second, Op.Loc); + // The ABI lowering of the operands and of the '__muldc3' libcall is driven + // by this type, which has to follow the components. + DblOp.Ty = CGF.getContext().getComplexType(CGF.getContext().DoubleTy); + ComplexPairTy DblRes = EmitBinMul(DblOp); + return ComplexPairTy(CGF.EmitDoubleToX87LongDouble(DblRes.first, Op.Loc), + CGF.EmitDoubleToX87LongDouble(DblRes.second, Op.Loc)); + } else if (Op.LHS.first->getType()->isFloatingPointTy()) { // The general formulation is: // (a + ib) * (c + id) = (a * c - b * d) + i(a * d + b * c) // @@ -1034,7 +1106,28 @@ ComplexPairTy ComplexExprEmitter::EmitBinDiv(const BinOpInfo &Op) { llvm::Value *LHSr = Op.LHS.first, *LHSi = Op.LHS.second; llvm::Value *RHSr = Op.RHS.first, *RHSi = Op.RHS.second; llvm::Value *DSTr, *DSTi; - if (LHSr->getType()->isFloatingPointTy()) { + QualType ElemTy = Op.Ty->castAs<ComplexType>()->getElementType(); + if (CGF.getTarget().hasSoftX87LongDouble() && + ElemTy->isSpecificBuiltinType(BuiltinType::LongDouble)) { + // When using software x87 `long double`, both the real and imaginary parts + // are converted to `double` for calculation, and the result parts are + // converted back. + BinOpInfo DblOp = Op; + DblOp.LHS = ComplexPairTy( + CGF.EmitX87LongDoubleToDouble(Op.LHS.first, Op.Loc), nullptr); + if (Op.LHS.second) + DblOp.LHS.second = CGF.EmitX87LongDoubleToDouble(Op.LHS.second, Op.Loc); + DblOp.RHS = ComplexPairTy( + CGF.EmitX87LongDoubleToDouble(Op.RHS.first, Op.Loc), nullptr); + if (Op.RHS.second) + DblOp.RHS.second = CGF.EmitX87LongDoubleToDouble(Op.RHS.second, Op.Loc); + // The ABI lowering of the operands and of the '__divdc3' libcall is driven + // by this type, which has to follow the components. + DblOp.Ty = CGF.getContext().getComplexType(CGF.getContext().DoubleTy); + ComplexPairTy DblRes = EmitBinDiv(DblOp); + return ComplexPairTy(CGF.EmitDoubleToX87LongDouble(DblRes.first, Op.Loc), + CGF.EmitDoubleToX87LongDouble(DblRes.second, Op.Loc)); + } else if (LHSr->getType()->isFloatingPointTy()) { CodeGenFunction::CGFPOptionsRAII FPOptsRAII(CGF, Op.FPFeatures); if (!RHSi) { assert(LHSi && "Can have at most one non-complex operand!"); @@ -1214,6 +1307,7 @@ ComplexExprEmitter::EmitBinOps(const BinaryOperator *E, else Ops.Ty = E->getType(); Ops.FPFeatures = E->getFPFeaturesInEffect(CGF.getLangOpts()); + Ops.Loc = E->getExprLoc(); return Ops; } @@ -1226,6 +1320,7 @@ LValue ComplexExprEmitter::EmitCompoundAssignLValue( BinOpInfo OpInfo; OpInfo.FPFeatures = E->getFPFeaturesInEffect(CGF.getLangOpts()); + OpInfo.Loc = E->getExprLoc(); CodeGenFunction::CGFPOptionsRAII FPOptsRAII(CGF, OpInfo.FPFeatures); const bool IsComplexDivisor = E->getOpcode() == BO_DivAssign && diff --git a/clang/lib/CodeGen/CGExprConstant.cpp b/clang/lib/CodeGen/CGExprConstant.cpp index e2ec07bbae711a..228b92412c820a 100644 --- a/clang/lib/CodeGen/CGExprConstant.cpp +++ b/clang/lib/CodeGen/CGExprConstant.cpp @@ -56,6 +56,20 @@ llvm::Constant *getPadding(const CodeGenModule &CGM, CharUnits PadSize) { return llvm::UndefValue::get(Ty); } +llvm::Constant *emitFloatConstantMaybeSoft(CodeGenModule &CGM, + const llvm::APFloat &Value) { + if (CGM.getTarget().hasSoftX87LongDouble() && + &Value.getSemantics() == &llvm::APFloat::x87DoubleExtended()) { + // This should be emitted as its 128-bit integer image, with padding bits + // filled with zeroes. + llvm::IntegerType *ImageTy = llvm::cast<llvm::IntegerType>( + CGM.getTypes().ConvertType(CGM.getContext().LongDoubleTy)); + return llvm::ConstantInt::get( + ImageTy, Value.bitcastToAPInt().zextOrTrunc(ImageTy->getBitWidth())); + } else + return llvm::ConstantFP::get(CGM.getLLVMContext(), Value); +} + struct ConstantAggregateBuilderUtils { CodeGenModule &CGM; @@ -2649,14 +2663,12 @@ ConstantEmitter::tryEmitPrivate(const APValue &Value, QualType DestType, return llvm::ConstantStruct::get(STy, Complex); } case APValue::Float: - return llvm::ConstantFP::get(CGM.getLLVMContext(), Value.getFloat()); + return emitFloatConstantMaybeSoft(CGM, Value.getFloat()); case APValue::ComplexFloat: { llvm::Constant *Complex[2]; - Complex[0] = llvm::ConstantFP::get(CGM.getLLVMContext(), - Value.getComplexFloatReal()); - Complex[1] = llvm::ConstantFP::get(CGM.getLLVMContext(), - Value.getComplexFloatImag()); + Complex[0] = emitFloatConstantMaybeSoft(CGM, Value.getComplexFloatReal()); + Complex[1] = emitFloatConstantMaybeSoft(CGM, Value.getComplexFloatImag()); // FIXME: the target may want to specify that this is packed. llvm::StructType *STy = @@ -2672,7 +2684,7 @@ ConstantEmitter::tryEmitPrivate(const APValue &Value, QualType DestType, if (Elt.isInt()) Inits[I] = llvm::ConstantInt::get(CGM.getLLVMContext(), Elt.getInt()); else if (Elt.isFloat()) - Inits[I] = llvm::ConstantFP::get(CGM.getLLVMContext(), Elt.getFloat()); + Inits[I] = emitFloatConstantMaybeSoft(CGM, Elt.getFloat()); else if (Elt.isIndeterminate()) Inits[I] = llvm::UndefValue::get(CGM.getTypes().ConvertType( DestType->castAs<VectorType>()->getElementType())); @@ -2696,8 +2708,7 @@ ConstantEmitter::tryEmitPrivate(const APValue &Value, QualType DestType, Inits[Idx] = llvm::ConstantInt::get(CGM.getLLVMContext(), Elt.getInt()); else if (Elt.isFloat()) - Inits[Idx] = - llvm::ConstantFP::get(CGM.getLLVMContext(), Elt.getFloat()); + Inits[Idx] = emitFloatConstantMaybeSoft(CGM, Elt.getFloat()); else if (Elt.isIndeterminate()) Inits[Idx] = llvm::PoisonValue::get( CGM.getTypes().ConvertType(MT->getElementType())); diff --git a/clang/lib/CodeGen/CGExprScalar.cpp b/clang/lib/CodeGen/CGExprScalar.cpp index 0e2e29f0745d27..83d516e344648a 100644 --- a/clang/lib/CodeGen/CGExprScalar.cpp +++ b/clang/lib/CodeGen/CGExprScalar.cpp @@ -47,6 +47,7 @@ #include "llvm/IR/GEPNoWrapFlags.h" #include "llvm/IR/GetElementPtrTypeIterator.h" #include "llvm/IR/GlobalVariable.h" +#include "llvm/IR/Instructions.h" #include "llvm/IR/Intrinsics.h" #include "llvm/IR/IntrinsicsPowerPC.h" #include "llvm/IR/IntrinsicsWebAssembly.h" @@ -70,6 +71,20 @@ extern cl::opt<bool> EnableSingleByteCoverage; namespace { +llvm::Constant *emitFloatConstantMaybeSoft(CodeGenModule &CGM, + const llvm::APFloat &Value) { + if (CGM.getTarget().hasSoftX87LongDouble() && + &Value.getSemantics() == &llvm::APFloat::x87DoubleExtended()) { + // This should be emitted as its 128-bit integer image, with padding bits + // filled with zeroes. + llvm::IntegerType *ImageTy = llvm::cast<llvm::IntegerType>( + CGM.getTypes().ConvertType(CGM.getContext().LongDoubleTy)); + return llvm::ConstantInt::get( + ImageTy, Value.bitcastToAPInt().zextOrTrunc(ImageTy->getBitWidth())); + } else + return llvm::ConstantFP::get(CGM.getLLVMContext(), Value); +} + /// Determine whether the given binary operation may overflow. /// Sets \p Result to the value of the operation for BO_Add, BO_Sub, BO_Mul, /// and signed BO_{Div,Rem}. For these opcodes, and for unsigned BO_{Div,Rem}, @@ -460,7 +475,18 @@ class ScalarExprEmitter Value *EmitNullValue(QualType Ty); /// EmitFloatToBoolConversion - Perform an FP to boolean conversion. - Value *EmitFloatToBoolConversion(Value *V) { + Value *EmitFloatToBoolConversion(Value *V, QualType Ty) { + // For software x87 long double, the value is zero iff both the + // exponent and mantissa are zeroes. The padding bits at the top are + // unspecified, so they have to be masked off first. + if (CGF.getTarget().hasSoftX87LongDouble() && + Ty->isSpecificBuiltinType(BuiltinType::LongDouble)) + return CGF.Builder.CreateICmpNE( + CGF.Builder.CreateAnd( + V, llvm::ConstantInt::get(V->getType(), + llvm::APInt::getLowBitsSet(128, 79))), + llvm::ConstantInt::get(V->getType(), 0)); + // Compare against 0.0 for fp scalars. llvm::Value *Zero = llvm::Constant::getNullValue(V->getType()); return Builder.CreateFCmpUNE(V, Zero, "tobool"); @@ -555,6 +581,9 @@ class ScalarExprEmitter return Builder.getInt(E->getValue()); } Value *VisitFloatingLiteral(const FloatingLiteral *E) { + if (CGF.getTarget().hasSoftX87LongDouble() && + E->getType()->isSpecificBuiltinType(BuiltinType::LongDouble)) + return emitFloatConstantMaybeSoft(CGF.CGM, E->getValue()); return llvm::ConstantFP::get(VMContext, E->getValue()); } Value *VisitCharacterLiteral(const CharacterLiteral *E) { @@ -838,6 +867,9 @@ class ScalarExprEmitter // Binary Operators. Value *EmitMul(const BinOpInfo &Ops) { + if (CGF.getTarget().hasSoftX87LongDouble() && + Ops.Ty->isSpecificBuiltinType(BuiltinType::LongDouble)) + return EmitSoftX87LongDoubleBinOp(Ops, &ScalarExprEmitter::EmitMul); if (Ops.Ty->isSignedIntegerOrEnumerationType() || Ops.Ty->isUnsignedIntegerType()) { const bool isSigned = Ops.Ty->isSignedIntegerOrEnumerationType(); @@ -985,10 +1017,23 @@ class ScalarExprEmitter Value *EmitCompare(const BinaryOperator *E, llvm::CmpInst::Predicate UICmpOpc, llvm::CmpInst::Predicate SICmpOpc, llvm::CmpInst::Predicate FCmpOpc, bool IsSignaling); -#define VISITCOMP(CODE, UI, SI, FP, SIG) \ - Value *VisitBin##CODE(const BinaryOperator *E) { \ - return EmitCompare(E, llvm::ICmpInst::UI, llvm::ICmpInst::SI, \ - llvm::FCmpInst::FP, SIG); } + Value *EmitSoftX87LongDoubleComp(const BinaryOperator *E, + llvm::CmpInst::Predicate FCmpOpc); + Value *EmitSoftX87LongDoubleBinOp( + const BinOpInfo &Ops, + Value *(ScalarExprEmitter::*Emit)(const BinOpInfo &)); +#define VISITCOMP(CODE, UI, SI, FP, SIG) \ + Value *VisitBin##CODE(const BinaryOperator *E) { \ + if (CGF.getTarget().hasSoftX87LongDouble() && \ + E->getLHS()->getType()->isSpecificBuiltinType( \ + BuiltinType::LongDouble) && \ + E->getRHS()->getType()->isSpecificBuiltinType( \ + BuiltinType::LongDouble)) \ + return EmitSoftX87LongDoubleComp(E, llvm::FCmpInst::FP); \ + else \ + return EmitCompare(E, llvm::ICmpInst::UI, llvm::ICmpInst::SI, \ + llvm::FCmpInst::FP, SIG); \ + } VISITCOMP(LT, ICMP_ULT, ICMP_SLT, FCMP_OLT, true) VISITCOMP(GT, ICMP_UGT, ICMP_SGT, FCMP_OGT, true) VISITCOMP(LE, ICMP_ULE, ICMP_SLE, FCMP_OLE, true) @@ -1045,7 +1090,7 @@ Value *ScalarExprEmitter::EmitConversionToBool(Value *Src, QualType SrcType) { assert(SrcType.isCanonical() && "EmitScalarConversion strips typedefs"); if (SrcType->isRealFloatingType()) - return EmitFloatToBoolConversion(Src); + return EmitFloatToBoolConversion(Src, SrcType); if (const MemberPointerType *MPT = dyn_cast<MemberPointerType>(SrcType)) return CGF.CGM.getCXXABI().EmitMemberPointerIsNotNull(CGF, Src, MPT); @@ -1629,6 +1674,35 @@ Value *ScalarExprEmitter::EmitScalarCast(Value *Src, QualType SrcType, return Builder.CreateFPExt(Src, DstTy, "conv"); } +/// Evaluate a binary operation on soft x87 long double operands: both sides +/// are converted to `double`, the operation is carried out in `double`, and the +/// result is converted back into the x87 image. +Value *ScalarExprEmitter::EmitSoftX87LongDoubleBinOp( + const BinOpInfo &Ops, + Value *(ScalarExprEmitter::*Emit)(const BinOpInfo &)) { + SourceLocation Loc = Ops.E ? Ops.E->getExprLoc() : SourceLocation(); + BinOpInfo DoubleOps = Ops; + DoubleOps.Ty = CGF.getContext().DoubleTy; + DoubleOps.LHS = CGF.EmitX87LongDoubleToDouble(Ops.LHS, Loc); + DoubleOps.RHS = CGF.EmitX87LongDoubleToDouble(Ops.RHS, Loc); + return CGF.EmitDoubleToX87LongDouble((this->*Emit)(DoubleOps), Loc); +} + +// The comparison is carried out on the 80-bit images themselves: narrowing +// the operands to `double` first would make distinct values compare equal +// (everything below the smallest `double` becomes zero, and values that only +// differ in the low significand bits collapse into one). +Value * +ScalarExprEmitter::EmitSoftX87LongDoubleComp(const BinaryOperator *E, + llvm::CmpInst::Predicate FCmpOpc) { + BinOpInfo Ops = EmitBinOps(E); + CodeGenFunction::CGFPOptionsRAII FPOptsRAII(CGF, Ops.FPFeatures); + llvm::Value *Result = + CGF.EmitSoftX87LongDoubleComparison(Ops.LHS, Ops.RHS, FCmpOpc); + return EmitScalarConversion(Result, CGF.getContext().BoolTy, E->getType(), + E->getExprLoc()); +} + /// Emit a conversion from the specified type to the specified destination type, /// both of which are LLVM scalar types. Value *ScalarExprEmitter::EmitScalarConversion(Value *Src, QualType SrcType, @@ -1674,14 +1748,43 @@ Value *ScalarExprEmitter::EmitScalarConversion(Value *Src, QualType SrcType, if (DstType->isVoidType()) return nullptr; - llvm::Value *OrigSrc = Src; - QualType OrigSrcType = SrcType; - llvm::Type *SrcTy = Src->getType(); - // Handle conversions to bool first, they are special: comparisons against 0. if (DstType->isBooleanType()) return EmitConversionToBool(Src, SrcType); + // If either source type is software x87 long double, conversion will have to + // go through an intermediate conversion to double, which seems to match + // x86-64 emulation on ARM64. + if (CGF.getTarget().hasSoftX87LongDouble()) { + QualType USrcType = SrcType; + if (const auto *AT = USrcType->getAs<AtomicType>()) + USrcType = AT->getValueType(); + QualType UDstType = DstType; + if (const auto *AT = UDstType->getAs<AtomicType>()) + UDstType = AT->getValueType(); + + // Only short-circuit `long double`, because `_Atomic bool` and `bool` are + // not the same type. + if (USrcType == UDstType && + USrcType->isSpecificBuiltinType(BuiltinType::LongDouble)) + return Src; + + if (UDstType->isSpecificBuiltinType(BuiltinType::LongDouble)) { + if (!USrcType->isSpecificBuiltinType(BuiltinType::Double)) + Src = EmitScalarConversion(Src, SrcType, CGF.getContext().DoubleTy, Loc, + Opts); + return CGF.EmitDoubleToX87LongDouble(Src, Loc); + } + + if (USrcType->isSpecificBuiltinType(BuiltinType::LongDouble)) { + Src = CGF.EmitX87LongDoubleToDouble(Src, Loc); + SrcType = CGF.getContext().DoubleTy; + } + } + + llvm::Value *OrigSrc = Src; + QualType OrigSrcType = SrcType; + llvm::Type *SrcTy = Src->getType(); llvm::Type *DstTy = ConvertType(DstType); // Determine whether an overflow behavior of 'trap' has been specified for @@ -3131,7 +3234,7 @@ Value *ScalarExprEmitter::VisitCastExpr(CastExpr *CE) { return EmitPointerToBoolConversion(Visit(E), E->getType()); case CK_FloatingToBoolean: { CodeGenFunction::CGFPOptionsRAII FPOptsRAII(CGF, CE); - return EmitFloatToBoolConversion(Visit(E)); + return EmitFloatToBoolConversion(Visit(E), E->getType()); } case CK_MemberPointerToBoolean: { llvm::Value *MemPtr = Visit(E); @@ -3561,6 +3664,10 @@ ScalarExprEmitter::EmitScalarPrePostIncDec(const UnaryOperator *E, LValue LV, llvm::Value *amt; CodeGenFunction::CGFPOptionsRAII FPOptsRAII(CGF, E); + if (CGF.getTarget().hasSoftX87LongDouble() && + type->isSpecificBuiltinType(BuiltinType::LongDouble)) + value = CGF.EmitX87LongDoubleToDouble(value, E->getExprLoc()); + // Another special case: half FP increment should be done via float. if (type->isHalfType() && !CGF.getContext().getLangOpts().NativeHalfType) value = Builder.CreateFPExt(value, CGF.CGM.FloatTy, "incdec.conv"); @@ -3597,6 +3704,10 @@ ScalarExprEmitter::EmitScalarPrePostIncDec(const UnaryOperator *E, LValue LV, if (type->isHalfType() && !CGF.getContext().getLangOpts().NativeHalfType) value = Builder.CreateFPTrunc(value, CGF.CGM.HalfTy, "incdec.conv"); + if (CGF.getTarget().hasSoftX87LongDouble() && + type->isSpecificBuiltinType(BuiltinType::LongDouble)) + value = CGF.EmitDoubleToX87LongDouble(value, E->getExprLoc()); + // Fixed-point types. } else if (type->isFixedPointType()) { // Fixed-point types are tricky. In some cases, it isn't possible to @@ -3708,6 +3819,14 @@ Value *ScalarExprEmitter::VisitMinus(const UnaryOperator *E, else Op = Visit(E->getSubExpr()); + // For software x87 long double, flip the sign bit in the integer image. + if (CGF.getTarget().hasSoftX87LongDouble() && + E->getType()->isSpecificBuiltinType(BuiltinType::LongDouble)) + return Builder.CreateXor( + Op, + llvm::ConstantInt::get(VMContext, llvm::APInt::getOneBitSet(128, 79)), + "fneg"); + // Generate a unary FNeg for FP ops. if (Op->getType()->isFPOrFPVectorTy()) { CodeGenFunction::CGFPOptionsRAII FPOptsRAII(CGF, E); @@ -4110,8 +4229,10 @@ LValue ScalarExprEmitter::EmitCompoundAssignLValue( llvm::Type *IRTy = CGF.ConvertType(AtomicValueTy); uint64_t StoreBits = CGF.CGM.getDataLayout().getTypeStoreSizeInBits(IRTy); // Floating atomicrmw operations cannot model constrained FP semantics. + // A type that is floating-point in the source but not in IR (`long + // double` on ARM64EC is an integer image) cannot use them at all. CanEmitAtomicRMW = - !OpInfo.FPFeatures.isFPConstrained() && + IRTy->isFloatingPointTy() && !OpInfo.FPFeatures.isFPConstrained() && CGF.getContext().hasSameUnqualifiedType(AtomicValueTy, ResultTy) && llvm::isPowerOf2_64(StoreBits); } else { @@ -4320,6 +4441,9 @@ void ScalarExprEmitter::EmitUndefinedBehaviorIntegerDivAndRemCheck( } Value *ScalarExprEmitter::EmitDiv(const BinOpInfo &Ops) { + if (CGF.getTarget().hasSoftX87LongDouble() && + Ops.Ty->isSpecificBuiltinType(BuiltinType::LongDouble)) + return EmitSoftX87LongDoubleBinOp(Ops, &ScalarExprEmitter::EmitDiv); { SanitizerDebugLocation SanScope(&CGF, {SanitizerKind::SO_IntegerDivideByZero, @@ -4784,6 +4908,9 @@ static Value* tryEmitFMulAdd(const BinOpInfo &op, } Value *ScalarExprEmitter::EmitAdd(const BinOpInfo &op) { + if (CGF.getTarget().hasSoftX87LongDouble() && + op.Ty->isSpecificBuiltinType(BuiltinType::LongDouble)) + return EmitSoftX87LongDoubleBinOp(op, &ScalarExprEmitter::EmitAdd); if (op.LHS->getType()->isPointerTy() || op.RHS->getType()->isPointerTy()) return emitPointerArithmetic(CGF, op, CodeGenFunction::NotSubtraction); @@ -4946,6 +5073,9 @@ Value *ScalarExprEmitter::EmitFixedPointBinOp(const BinOpInfo &op) { } Value *ScalarExprEmitter::EmitSub(const BinOpInfo &op) { + if (CGF.getTarget().hasSoftX87LongDouble() && + op.Ty->isSpecificBuiltinType(BuiltinType::LongDouble)) + return EmitSoftX87LongDoubleBinOp(op, &ScalarExprEmitter::EmitSub); // The LHS is always a pointer if either side is. if (!op.LHS->getType()->isPointerTy()) { if (op.Ty->isSignedIntegerOrEnumerationType() || @@ -5418,8 +5548,16 @@ Value *ScalarExprEmitter::EmitCompare(const BinaryOperator *E, if (CETy->isRealFloatingType()) { // As complex comparisons can only be equality comparisons, they // are never signaling comparisons. - ResultR = Builder.CreateFCmp(FCmpOpc, LHS.first, RHS.first, "cmp.r"); - ResultI = Builder.CreateFCmp(FCmpOpc, LHS.second, RHS.second, "cmp.i"); + if (CGF.getTarget().hasSoftX87LongDouble() && + CETy->isSpecificBuiltinType(BuiltinType::LongDouble)) { + ResultR = + CGF.EmitSoftX87LongDoubleComparison(LHS.first, RHS.first, FCmpOpc); + ResultI = CGF.EmitSoftX87LongDoubleComparison(LHS.second, RHS.second, + FCmpOpc); + } else { + ResultR = Builder.CreateFCmp(FCmpOpc, LHS.first, RHS.first, "cmp.r"); + ResultI = Builder.CreateFCmp(FCmpOpc, LHS.second, RHS.second, "cmp.i"); + } } else { // Complex comparisons can only be equality comparisons. As such, signed // and unsigned opcodes are the same. @@ -6556,3 +6694,220 @@ Address CodeGenFunction::EmitCheckedInBoundsGEP( IdxList, SignedIndices, IsSubtraction, Loc, Name), elementType, Align); } + +llvm::Value *CodeGenFunction::EmitX87LongDoubleToDouble(llvm::Value *Bits, + SourceLocation Loc) { + assert(getTarget().hasSoftX87LongDouble() && + "soft x87 long double not available"); + + // If this follows a conversion from `double` to `long double`, they cancel + // each other out. So restore the original argument, and then delete the + // previous call. + if (auto *CI = dyn_cast<llvm::CallInst>(Bits)) { + llvm::Function *Callee = CI->getCalledFunction(); + if (Callee && Callee->getName() == "__arm64ec_double_to_x87ld") { + llvm::Value *Orig = CI->getArgOperand(0); + if (CI->use_empty()) + CI->eraseFromParent(); + return Orig; + } + } + + llvm::Type *ImageTy = ConvertType(getContext().LongDoubleTy); + + // `double __arm64ec_x87ld_to_double(uint128 image)` + llvm::FunctionType *FT = llvm::FunctionType::get( + llvm::Type::getDoubleTy(getLLVMContext()), {ImageTy}, false); + llvm::AttrBuilder Attrs(getLLVMContext()); + Attrs.addMemoryAttr(llvm::MemoryEffects::none()); + Attrs.addAttribute(llvm::Attribute::NoUnwind); + llvm::FunctionCallee Fn = CGM.CreateRuntimeFunction( + FT, "__arm64ec_x87ld_to_double", + llvm::AttributeList::get(getLLVMContext(), + llvm::AttributeList::FunctionIndex, Attrs)); + + ApplyDebugLocation DL(*this, Loc); + return Builder.CreateCall(Fn, {Bits}, "x87ld.to.double"); +} + +llvm::Value *CodeGenFunction::EmitDoubleToX87LongDouble(llvm::Value *Value, + SourceLocation Loc) { + assert(getTarget().hasSoftX87LongDouble() && + "soft x87 long double not available"); + llvm::Type *ImageTy = ConvertType(getContext().LongDoubleTy); + + // `uint128 __arm64ec_double_to_x87ld(double value)` + llvm::FunctionType *FT = llvm::FunctionType::get( + ImageTy, {llvm::Type::getDoubleTy(getLLVMContext())}, false); + llvm::AttrBuilder Attrs(getLLVMContext()); + Attrs.addMemoryAttr(llvm::MemoryEffects::none()); + Attrs.addAttribute(llvm::Attribute::NoUnwind); + llvm::FunctionCallee Fn = CGM.CreateRuntimeFunction( + FT, "__arm64ec_double_to_x87ld", + llvm::AttributeList::get(getLLVMContext(), + llvm::AttributeList::FunctionIndex, Attrs)); + + ApplyDebugLocation DL(*this, Loc); + return Builder.CreateCall(Fn, {Value}, "double.to.x87ld"); +} + +llvm::Value *CodeGenFunction::EmitSoftX87LongDoubleComparison( + llvm::Value *LHS, llvm::Value *RHS, llvm::CmpInst::Predicate Pred) { + assert(getTarget().hasSoftX87LongDouble() && + "soft x87 long double not available"); + + llvm::Type *I128Ty = llvm::Type::getInt128Ty(getLLVMContext()); + assert(LHS->getType() == I128Ty && RHS->getType() == I128Ty && + "expected 16-byte x87 images"); + + // Bits 0..79 hold the value: sign, biased exponent, and a significand with + // an explicit integer bit. Bits 80..127 are padding whose contents are + // unspecified, so they are cleared up front and never take part in the + // comparison or in producing the ordering key. + llvm::Constant *Zero = llvm::ConstantInt::get(I128Ty, 0); + llvm::Constant *ValueMask = + llvm::ConstantInt::get(I128Ty, llvm::APInt::getLowBitsSet(128, 80)); + llvm::Constant *SignBit = + llvm::ConstantInt::get(I128Ty, llvm::APInt::getOneBitSet(128, 79)); + llvm::Constant *MagnitudeMask = + llvm::ConstantInt::get(I128Ty, llvm::APInt::getLowBitsSet(128, 79)); + llvm::Constant *ExpAllOnes = + llvm::ConstantInt::get(I128Ty, llvm::APInt(128, 0x7FFF)); + llvm::Constant *MantissaMask = + llvm::ConstantInt::get(I128Ty, llvm::APInt::getLowBitsSet(128, 64)); + llvm::Constant *IntegerBit = + llvm::ConstantInt::get(I128Ty, llvm::APInt::getOneBitSet(128, 63)); + + // Reduce each image to an integer whose unsigned order is the numeric order + // of the value it denotes, plus a flag telling whether it is a NaN, in which + // case there is no meaningful order at all. + llvm::Value *Images[2] = {LHS, RHS}; + llvm::Value *Keys[2]; + llvm::Value *IsNaNs[2]; + for (unsigned I = 0; I != 2; ++I) { + llvm::Value *Image = Builder.CreateAnd(Images[I], ValueMask, "x87.value"); + llvm::Value *Exp = + Builder.CreateAnd(Builder.CreateLShr(Image, 64), ExpAllOnes, "x87.exp"); + llvm::Value *Mantissa = + Builder.CreateAnd(Image, MantissaMask, "x87.mantissa"); + + // A significand of exactly the integer bit denotes infinity when the + // exponent is all ones, and a nonzero value otherwise. + llvm::Value *MantissaIsIntegerBit = + Builder.CreateICmpEQ(Mantissa, IntegerBit); + + // A NaN is an all-ones exponent with any other significand (quiet NaN, + // signalling NaN, and the pseudo-infinity encoding). + IsNaNs[I] = + Builder.CreateAnd(Builder.CreateICmpEQ(Exp, ExpAllOnes), + Builder.CreateNot(MantissaIsIntegerBit), "x87.isnan"); + + // Fold the two zeroes together by masking the sign bit off a zero, so that + // -0.0 and +0.0 compare equal. + llvm::Value *IsZero = Builder.CreateAnd( + Builder.CreateICmpEQ(Exp, Zero), Builder.CreateICmpEQ(Mantissa, Zero)); + llvm::Value *Magnitude = Builder.CreateSelect( + IsZero, Builder.CreateAnd(Image, MagnitudeMask), Image, "x87.mag"); + + // The extended-precision format carries an explicit integer bit, so besides + // the supported encodings there are "unsupported" ones (FXAM reports them + // as such); the x87 never produces them itself: + // + // 1. Unnormal: nonzero exponent (but not all ones) with the integer bit + // clear. FLD accepts it, but any arithmetic or comparison on it raises + // #IA, and with the exception masked the operand behaves as the + // indefinite NaN. + // 2. Pseudo-denormal: zero exponent with the integer bit set. This one is + // accepted and denotes the same value as a normal number with exponent + // one (the encoding itself is not rewritten). + // + // The pseudo-infinity and pseudo-NaN encodings (all-ones exponent, integer + // bit clear) are unsupported as well and are folded into the NaN test + // above, which is what the hardware effectively does. + // + // Neither of the two encodings below is ever generated by the compiler, so + // they are left unhandled and treated as undefined for now. + + // Order as unsigned integers: complement a negative value (within the 80 + // significant bits) and set the sign bit of a non-negative one, which + // lands both zeroes in the middle of the range. + llvm::Value *IsNegative = + Builder.CreateICmpNE(Builder.CreateAnd(Magnitude, SignBit), Zero); + Keys[I] = Builder.CreateSelect( + IsNegative, Builder.CreateXor(Magnitude, ValueMask), + Builder.CreateOr(Magnitude, SignBit), "x87.key"); + } + + // An unordered comparison ("a NaN is involved") is true for `!=` and false + // for every ordered predicate, so each case folds that in below instead of + // branching: the ordering keys of a NaN are garbage, but they are masked out + // by the ordered flag. + llvm::Value *HasNaN = Builder.CreateOr(IsNaNs[0], IsNaNs[1], "x87.has_nan"); + llvm::Value *Ordered = Builder.CreateNot(HasNaN, "x87.ordered"); + + switch (Pred) { + case llvm::CmpInst::FCMP_FALSE: + return Builder.getFalse(); + + case llvm::CmpInst::FCMP_OEQ: + return Builder.CreateAnd(Builder.CreateICmpEQ(Keys[0], Keys[1], "x87cmp"), + Ordered, "x87cmp.ord"); + + case llvm::CmpInst::FCMP_OGT: + return Builder.CreateAnd(Builder.CreateICmpUGT(Keys[0], Keys[1], "x87cmp"), + Ordered, "x87cmp.ord"); + + case llvm::CmpInst::FCMP_OGE: + return Builder.CreateAnd(Builder.CreateICmpUGE(Keys[0], Keys[1], "x87cmp"), + Ordered, "x87cmp.ord"); + + case llvm::CmpInst::FCMP_OLT: + return Builder.CreateAnd(Builder.CreateICmpULT(Keys[0], Keys[1], "x87cmp"), + Ordered, "x87cmp.ord"); + + case llvm::CmpInst::FCMP_OLE: + return Builder.CreateAnd(Builder.CreateICmpULE(Keys[0], Keys[1], "x87cmp"), + Ordered, "x87cmp.ord"); + + case llvm::CmpInst::FCMP_ONE: + return Builder.CreateAnd(Builder.CreateICmpNE(Keys[0], Keys[1], "x87cmp"), + Ordered, "x87cmp.ord"); + + case llvm::CmpInst::FCMP_ORD: + return Ordered; + + case llvm::CmpInst::FCMP_UNO: + return HasNaN; + + case llvm::CmpInst::FCMP_UEQ: + return Builder.CreateOr(Builder.CreateICmpEQ(Keys[0], Keys[1], "x87cmp"), + HasNaN, "x87cmp.unord"); + + case llvm::CmpInst::FCMP_UGT: + return Builder.CreateOr(Builder.CreateICmpUGT(Keys[0], Keys[1], "x87cmp"), + HasNaN, "x87cmp.unord"); + + case llvm::CmpInst::FCMP_UGE: + return Builder.CreateOr(Builder.CreateICmpUGE(Keys[0], Keys[1], "x87cmp"), + HasNaN, "x87cmp.unord"); + + case llvm::CmpInst::FCMP_ULT: + return Builder.CreateOr(Builder.CreateICmpULT(Keys[0], Keys[1], "x87cmp"), + HasNaN, "x87cmp.unord"); + + case llvm::CmpInst::FCMP_ULE: + return Builder.CreateOr(Builder.CreateICmpULE(Keys[0], Keys[1], "x87cmp"), + HasNaN, "x87cmp.unord"); + + case llvm::CmpInst::FCMP_UNE: + return Builder.CreateOr(Builder.CreateICmpNE(Keys[0], Keys[1], "x87cmp"), + HasNaN, "x87cmp.unord"); + + case llvm::CmpInst::FCMP_TRUE: + return Builder.getTrue(); + + default: + llvm_unreachable( + "unsupported comparison predicate for soft x87 long double"); + } +} diff --git a/clang/lib/CodeGen/CodeGenFunction.h b/clang/lib/CodeGen/CodeGenFunction.h index e17ed28c5d4f3a..cd1825328fa8b2 100644 --- a/clang/lib/CodeGen/CodeGenFunction.h +++ b/clang/lib/CodeGen/CodeGenFunction.h @@ -5200,6 +5200,11 @@ class CodeGenFunction : public CodeGenTypeCache { /// Emits a reference binding to the passed in expression. RValue EmitReferenceBindingToExpr(const Expr *E); + /// For ARM64EC, lower builtins that take or return x87 `long double` values. + /// Returns false if it's not handled. + bool EmitSoftX87LongDoubleBuiltin(unsigned BuiltinID, const CallExpr *E, + RValue &Result); + //===--------------------------------------------------------------------===// // Expression Emission //===--------------------------------------------------------------------===// @@ -5215,6 +5220,18 @@ class CodeGenFunction : public CodeGenTypeCache { llvm::Value *EmitScalarConversion(llvm::Value *Src, QualType SrcTy, QualType DstTy, SourceLocation Loc); + /// Emit intermediate conversions between software x86 `long double` and + /// hardware `double`. + llvm::Value *EmitX87LongDoubleToDouble(llvm::Value *Bits, SourceLocation Loc); + llvm::Value *EmitDoubleToX87LongDouble(llvm::Value *Value, + SourceLocation Loc); + + /// Compare two software x87 `long double` images, using only integer + /// operations. The operation is exact. + llvm::Value *EmitSoftX87LongDoubleComparison(llvm::Value *LHS, + llvm::Value *RHS, + llvm::CmpInst::Predicate Pred); + /// Emit a conversion from the specified complex type to the specified /// destination type, where the destination type is an LLVM scalar type. llvm::Value *EmitComplexToScalarConversion(ComplexPairTy Src, QualType SrcTy, diff --git a/clang/lib/CodeGen/CodeGenTypes.cpp b/clang/lib/CodeGen/CodeGenTypes.cpp index daed8559cc62f5..97a96e8fb1e82d 100644 --- a/clang/lib/CodeGen/CodeGenTypes.cpp +++ b/clang/lib/CodeGen/CodeGenTypes.cpp @@ -477,6 +477,13 @@ llvm::Type *CodeGenTypes::ConvertType(QualType T) { break; case BuiltinType::LongDouble: LongDoubleReferenced = true; + if (getTarget().hasSoftX87LongDouble()) { + // For ARM64EC, the software x87 `long double` is an integer image + // of the value in a 16-byte slot. All computation goes through + // `double` via runtime helper calls. + ResultType = llvm::IntegerType::get(getLLVMContext(), 128); + break; + } [[fallthrough]]; case BuiltinType::BFloat16: case BuiltinType::Float: diff --git a/clang/lib/Driver/ToolChains/Clang.cpp b/clang/lib/Driver/ToolChains/Clang.cpp index 0893b481f4ce64..614843b01d3439 100644 --- a/clang/lib/Driver/ToolChains/Clang.cpp +++ b/clang/lib/Driver/ToolChains/Clang.cpp @@ -6344,6 +6344,9 @@ void Clang::ConstructJob(Compilation &C, const JobAction &JA, else if (TC.getTriple().isPPC() && (A->getOption().getID() != options::OPT_mlong_double_80)) A->render(Args, CmdArgs); + else if (TC.getTriple().isWindowsArm64EC() && + !TC.getTriple().isWindowsMSVCEnvironment()) + A->render(Args, CmdArgs); else D.Diag(diag::err_drv_unsupported_opt_for_target) << A->getAsString(Args) << TripleStr; diff --git a/clang/test/CodeGen/arm64ec-long-double-modes.c b/clang/test/CodeGen/arm64ec-long-double-modes.c new file mode 100644 index 00000000000000..e63d3f6c1e1dbd --- /dev/null +++ b/clang/test/CodeGen/arm64ec-long-double-modes.c @@ -0,0 +1,31 @@ +// RUN: %clang_cc1 -triple arm64ec-w64-windows-gnu -mlong-double-64 \ +// RUN: -emit-llvm -o - %s | FileCheck %s --check-prefix=LD64 +// RUN: %clang_cc1 -triple arm64ec-w64-windows-gnu -mlong-double-80 \ +// RUN: -emit-llvm -o - %s | FileCheck %s --check-prefix=LD80 +// RUN: %clang_cc1 -triple arm64ec-w64-windows-gnu -mlong-double-128 \ +// RUN: -emit-llvm -o - %s | FileCheck %s --check-prefix=LD128 + +// The three widths select the format of `long double`; only the 80-bit one is +// the software x87 format that goes through the ARM64EC runtime helpers. + +long double g = 1.0L; +long double add(long double a, long double b) { return a + b; } +int cmp(long double a, long double b) { return a < b; } + +// LD64: @g = {{.*}}global double 1.000000e+00 +// LD64-LABEL: define {{.*}}double @add(double noundef %a, double noundef %b) +// LD64: fadd double +// LD64-LABEL: define {{.*}}i32 @cmp(double noundef %a, double noundef %b) +// LD64: fcmp olt double +// LD64-NOT: __arm64ec_ + +// LD80: @g = {{.*}}global i128 302222231531620438900736, align 16 +// LD80-LABEL: define {{.*}}i128 @add(i128 noundef %a, i128 noundef %b) +// LD80: call double @__arm64ec_x87ld_to_double( +// LD80-LABEL: define {{.*}}i32 @cmp(i128 noundef %a, i128 noundef %b) +// LD80: icmp ult i128 + +// LD128: @g = {{.*}}global fp128 +// LD128-LABEL: define {{.*}}fp128 @add(fp128 noundef %a, fp128 noundef %b) +// LD128-LABEL: define {{.*}}i32 @cmp(fp128 noundef %a, fp128 noundef %b) +// LD128-NOT: __arm64ec_ diff --git a/clang/test/CodeGen/arm64ec-soft-x87-long-double.c b/clang/test/CodeGen/arm64ec-soft-x87-long-double.c new file mode 100644 index 00000000000000..f1b9244695dbd7 --- /dev/null +++ b/clang/test/CodeGen/arm64ec-soft-x87-long-double.c @@ -0,0 +1,160 @@ +// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py UTC_ARGS: --check-globals --include-generated-funcs --global-value-regex "g" +// RUN: %clang_cc1 -triple arm64ec-w64-windows-gnu -emit-llvm -o - %s | FileCheck %s +// RUN: %clang_cc1 -triple arm64ec-w64-windows-gnu -O2 -emit-llvm -o - %s | FileCheck %s --check-prefix=OPT + +// ARM64EC keeps the x86_64 80-bit x87 format, in a 16-byte slot, but has no +// x87 hardware: the value is an integer image that only ever goes through +// `double` for computation. +_Static_assert(sizeof(long double) == 16, "the slot is 16 bytes"); +_Static_assert(_Alignof(long double) == 16, "the slot is 16-byte aligned"); +_Static_assert(__LDBL_MANT_DIG__ == 64, "80-bit x87 significand"); +_Static_assert(__LDBL_MAX_EXP__ == 16384, "80-bit x87 exponent range"); + +long double g = 1.0L; + +long double add(long double a, long double b) { return a + b; } +long double neg(long double a) { return -a; } + +int cmp(long double a, long double b) { return a < b; } +int cmp_eq(long double a, long double b) { return a == b; } + +int isinf_(long double a) { return __builtin_isinf(a); } +int isnan_(long double a) { return __builtin_isnan(a); } +int signbit_(long double a) { return __builtin_signbitl(a); } +int fpclass(long double a) { + return __builtin_fpclassify(1, 2, 4, 8, 16, a); +} +long double ilogb_(long double a) { return (long double)__builtin_ilogbl(a); } +long double powi_(long double a, int n) { return __builtin_powil(a, n); } +int pred(long double a, long double b) { + return __builtin_isgreater(a, b) || __builtin_islessequal(a, b) || + __builtin_isunordered(a, b); +} +int fpclassify_mask(long double a) { return __builtin_isfpclass(a, 0x200); } + +long double _Complex conj_(long double _Complex z) { return ~z; } +long double _Complex conj_b(long double _Complex z) { + return __builtin_conjl(z); +} +long double creal_b(long double _Complex z) { return __builtin_creall(z); } +long double cimag_b(long double _Complex z) { return __builtin_cimagl(z); } +long double cabs_b(long double _Complex z) { return __builtin_cabsl(z); } +long double _Complex cadd(long double _Complex a, long double _Complex b) { + return a + b; +} +long double _Complex cinc(long double _Complex z) { return ++z; } + +int varargs(const char *fmt, ...); +int callv(void) { + // A 16-byte `long double` is passed by reference, exactly as on x86_64. + return varargs("%Lg", 1.0L); +} + +// The image of 1.0L: significand 0x8000000000000000, sign/exponent 0x3FFF. +// CHECK: @g = {{.*}}global i128 302222231531620438900736, align 16 + +// CHECK-LABEL: define {{.*}}i128 @add(i128 noundef %a, i128 noundef %b) +// CHECK: call double @__arm64ec_x87ld_to_double( +// CHECK: call double @__arm64ec_x87ld_to_double( +// CHECK: call i128 @__arm64ec_double_to_x87ld( + +// Negation flips the sign bit of the image; it does not go through `double`. +// CHECK-LABEL: define {{.*}}i128 @neg(i128 noundef %a) +// CHECK: [[SIGN:%.*]] = xor i128 {{.*}}, 604462909807314587353088 +// CHECK-NOT: call double @__arm64ec_x87ld_to_double + +// Comparison is done on the images themselves, never with `fcmp`. +// CHECK-LABEL: define {{.*}}i32 @cmp(i128 noundef %a, i128 noundef %b) +// CHECK: [[CMP:%.*]] = icmp ult i128 +// CHECK: [[ORD:%.*]] = and i1 [[CMP]], +// CHECK: zext i1 [[ORD]] to i32 +// CHECK-NOT: fcmp + +// CHECK-LABEL: define {{.*}}i32 @cmp_eq( +// CHECK-NOT: fcmp + +// CHECK-LABEL: define {{.*}}i32 @isinf_( +// CHECK: icmp eq i128 +// CHECK-NOT: fcmp +// CHECK: ret i32 + +// CHECK-LABEL: define {{.*}}i32 @isnan_( +// CHECK-NOT: fcmp +// CHECK: ret i32 + +// The sign is extracted from the image rather than through a CRT `signbit`. +// CHECK-LABEL: define {{.*}}i32 @signbit_( +// CHECK: lshr i128 {{.*}}, 79 +// CHECK-NOT: call i32 @signbit +// CHECK: ret i32 + +// CHECK-LABEL: define {{.*}}i32 @fpclass( +// CHECK-NOT: fcmp +// CHECK: ret i32 + +// `ilogbl` works on the image, so it must not go through binary64, which +// reports INT_MAX for anything that overflows `double`. +// CHECK-LABEL: define {{.*}}i128 @ilogb_( +// CHECK: call i32 @ilogbl(i128 +// CHECK-NOT: call i32 @ilogb( + +// `__builtin_powil` has no C library entry point; the CRT provides `__powil`. +// CHECK-LABEL: define {{.*}}i128 @powi_( +// CHECK: call i128 @__powil(i128 +// CHECK-NOT: call double @powi( + +// The comparison predicates are lowered on the images. +// CHECK-LABEL: define {{.*}}i32 @pred( +// CHECK-NOT: call i32 @isgreater +// CHECK-NOT: call i32 @islessequal +// CHECK-NOT: call i32 @isunordered +// CHECK-NOT: call i32 @islessequa +// CHECK: ret i32 + +// CHECK-LABEL: define {{.*}}i32 @fpclassify_mask( +// CHECK-NOT: call i32 @isfpclass +// CHECK: ret i32 + +// Conjugation only flips the sign bit of the imaginary component. +// CHECK-LABEL: define {{.*}}@conj_( +// CHECK: xor i128 {{.*}}, 604462909807314587353088 +// CHECK-NOT: call double @__arm64ec_x87ld_to_double + +// The `conjl` builtin is the same operation, not a call to a CRT entry point. +// CHECK-LABEL: define {{.*}}@conj_b( +// CHECK: xor i128 {{.*}}, 604462909807314587353088 +// CHECK-NOT: call {{.*}}@conjl + +// `creall` and `cimagl` only pick a component of the image pair. +// CHECK-LABEL: define {{.*}}i128 @creal_b( +// CHECK-NOT: call {{.*}}@creall +// CHECK: ret i128 + +// CHECK-LABEL: define {{.*}}i128 @cimag_b( +// CHECK-NOT: call {{.*}}@cimagl +// CHECK: ret i128 + +// A complex magnitude is real mathematics, so it stays a library call. +// CHECK-LABEL: define {{.*}}i128 @cabs_b( +// CHECK: call i128 @cabsl( + +// CHECK-LABEL: define {{.*}}@cadd( +// CHECK: call double @__arm64ec_x87ld_to_double( +// CHECK: call i128 @__arm64ec_double_to_x87ld( + +// `++z` increments the value, not the integer image. +// CHECK-LABEL: define {{.*}}@cinc( +// CHECK: call double @__arm64ec_x87ld_to_double( +// CHECK: fadd double +// CHECK: call i128 @__arm64ec_double_to_x87ld( + +// CHECK-LABEL: define {{.*}}i32 @callv( +// CHECK: [[TMP:%.*]] = alloca i128, align 16 +// CHECK: store i128 302222231531620438900736, ptr [[TMP]], align 16 +// CHECK: call i32 (ptr, ...) @varargs(ptr {{.*}}, ptr {{.*}}[[TMP]]) + +// The optimized pipeline keeps the same properties. +// OPT-LABEL: define {{.*}}i128 @neg(i128 noundef %a) +// OPT-NEXT: entry: +// OPT-NEXT: [[NEG:%.*]] = xor i128 %a, 604462909807314587353088 +// OPT-NEXT: ret i128 [[NEG]] diff --git a/clang/test/Driver/arm64ec-long-double.c b/clang/test/Driver/arm64ec-long-double.c new file mode 100644 index 00000000000000..18df2a1858e1d1 --- /dev/null +++ b/clang/test/Driver/arm64ec-long-double.c @@ -0,0 +1,21 @@ +// RUN: %clang -### --target=arm64ec-w64-mingw32 -mlong-double-64 -c %s 2>&1 \ +// RUN: | FileCheck %s --check-prefix=LD64 +// RUN: %clang -### --target=arm64ec-w64-mingw32 -mlong-double-80 -c %s 2>&1 \ +// RUN: | FileCheck %s --check-prefix=LD80 +// RUN: %clang -### --target=arm64ec-w64-mingw32 -mlong-double-128 -c %s 2>&1 \ +// RUN: | FileCheck %s --check-prefix=LD128 +// RUN: not %clang --target=arm64ec-pc-windows-msvc -mlong-double-64 \ +// RUN: -fsyntax-only %s 2>&1 | FileCheck %s --check-prefix=MSVC +// RUN: not %clang --target=aarch64-w64-mingw32 -mlong-double-80 \ +// RUN: -fsyntax-only %s 2>&1 | FileCheck %s --check-prefix=AARCH64 + +// ARM64EC with a non-MSVC ABI defaults to the x87 80-bit format of x86_64, so +// all three widths are accepted and forwarded to the frontend. +// LD64: "-mlong-double-64" +// LD80: "-mlong-double-80" +// LD128: "-mlong-double-128" + +// The MSVC ABI keeps its binary64 `long double` for compatibility, and plain +// AArch64 has no reason to support the x87 format at all. +// MSVC: error: unsupported option '-mlong-double-64' for target 'arm64ec-pc-windows-msvc{{.*}}' +// AARCH64: error: unsupported option '-mlong-double-80' for target 'aarch64-w64-windows-gnu' _______________________________________________ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
