[
https://issues.apache.org/jira/browse/KUDU-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Serbin updated KUDU-3736:
--------------------------------
Description:
On RedHat9 OS, x86_64 architecture, after installing updates published by
RedHat on 2025-11-12, or just upgrading up to or installing RHEL9.7 fresh, Kudu
tablet servers might crash with SIGSEGV.
This happens once upgrading libgcc up to 11.5.0-10 and newer versions.
The workaround is to disable code generation for row projections by adding the
following customization into the set of flags that Kudu tablet servers start
with:
{noformat}
--mrs_use_codegen=false
{noformat}
Examples of the top of stack traces of crashed tablet servers on RHEL9 with
{{libgcc-11.5.0-11.el9.x86_64}} are below:
{noformat}
*** SIGSEGV (@0x0) received by PID 138135 (TID 0x7fc277d56640) from PID 0;
stack trace: ***
@ 0xb6a837 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7fc2bb63fc30 (unknown)
@ 0x7fc2bc446c5d (unknown)
@ 0x7fc2bc4473af (unknown)
@ 0x10b9f51 llvm::RTDyldMemoryManager::registerEHFrames()
@ 0x10cc2af llvm::RuntimeDyldELF::registerEHFrames()
@ 0x109fbe1 llvm::MCJIT::finalizeLoadedModules()
@ 0x10a0165 llvm::MCJIT::finalizeObject()
@ 0xe0b0c8 kudu::codegen::ModuleBuilder::Compile()
@ 0xe05a28 kudu::codegen::RowProjectorFunctions::Create()
@ 0xe07214 kudu::codegen::CodeGenerator::CompileRowProjector()
@ 0xdfe3a3 kudu::codegen::(anonymous
namespace)::CompilationTask::Run()
{noformat}
{noformat}
*** SIGSEGV (@0x0) received by PID 639256 (TID 0x7f4333432640) from PID 0;
stack trace: ***
@ 0xb6b2c7 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f4419e3fc30 (unknown)
@ 0x7f441ac23c5d (unknown)
@ 0x7f441ac24a08 (unknown)
@ 0x7f441ac25fec __deregister_frame_info_bases
@ 0x7f441ac26029 __deregister_frame
@ 0x10cc3af llvm::RuntimeDyldELF::deregisterEHFrames()
@ 0x10a26b9 llvm::MCJIT::~MCJIT()
@ 0x10a2e52 llvm::MCJIT::~MCJIT()
@ 0xe08f78 kudu::codegen::JITWrapper::~JITWrapper()
@ 0xe07654
kudu::codegen::RowProjectorFunctions::~RowProjectorFunctions()
{noformat}
was:
On RedHat9 OS, x86_64 architecture, after installing updates published by
RedHat on 2025-11-12, or just upgrading up to or installing RHEL9.7 fresh, Kudu
tablet servers might crash with SIGSEGV.
This happens once upgrading libgcc up to 11.5.0-10 and newer versions.
The workaround is to disable code generation for row projections by adding the
following customization into the set of flags that Kudu tablet servers start
with:
{noformat}
--mrs_use_codegen=false
{noformat}
Stack traces of crashed tablet servers look like below:
{noformat}
*** SIGSEGV (@0x0) received by PID 138135 (TID 0x7fc277d56640) from PID 0;
stack trace: ***
@ 0xb6a837 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7fc2bb63fc30 (unknown)
@ 0x7fc2bc446c5d (unknown)
@ 0x7fc2bc4473af (unknown)
@ 0x10b9f51 llvm::RTDyldMemoryManager::registerEHFrames()
@ 0x10cc2af llvm::RuntimeDyldELF::registerEHFrames()
@ 0x109fbe1 llvm::MCJIT::finalizeLoadedModules()
@ 0x10a0165 llvm::MCJIT::finalizeObject()
@ 0xe0b0c8 kudu::codegen::ModuleBuilder::Compile()
@ 0xe05a28 kudu::codegen::RowProjectorFunctions::Create()
@ 0xe07214 kudu::codegen::CodeGenerator::CompileRowProjector()
@ 0xdfe3a3 kudu::codegen::(anonymous
namespace)::CompilationTask::Run()
{noformat}
{noformat}
*** SIGSEGV (@0x0) received by PID 639256 (TID 0x7f4333432640) from PID 0;
stack trace: ***
@ 0xb6b2c7 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f4419e3fc30 (unknown)
@ 0x7f441ac23c5d (unknown)
@ 0x7f441ac24a08 (unknown)
@ 0x7f441ac25fec __deregister_frame_info_bases
@ 0x7f441ac26029 __deregister_frame
@ 0x10cc3af llvm::RuntimeDyldELF::deregisterEHFrames()
@ 0x10a26b9 llvm::MCJIT::~MCJIT()
@ 0x10a2e52 llvm::MCJIT::~MCJIT()
@ 0xe08f78 kudu::codegen::JITWrapper::~JITWrapper()
@ 0xe07654
kudu::codegen::RowProjectorFunctions::~RowProjectorFunctions()
{noformat}
> Kudu tablet server's codegen might crash with SIGSEGV on RedHat9 after
> upgrading libgcc to 11.5.0-10 and newer versions
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: KUDU-3736
> URL: https://issues.apache.org/jira/browse/KUDU-3736
> Project: Kudu
> Issue Type: Bug
> Components: codegen, tserver
> Affects Versions: 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.11.0, 1.12.0, 1.11.1,
> 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.17.1, 1.18.1
> Reporter: Alexey Serbin
> Priority: Critical
>
> On RedHat9 OS, x86_64 architecture, after installing updates published by
> RedHat on 2025-11-12, or just upgrading up to or installing RHEL9.7 fresh,
> Kudu tablet servers might crash with SIGSEGV.
> This happens once upgrading libgcc up to 11.5.0-10 and newer versions.
> The workaround is to disable code generation for row projections by adding
> the following customization into the set of flags that Kudu tablet servers
> start with:
> {noformat}
> --mrs_use_codegen=false
> {noformat}
> Examples of the top of stack traces of crashed tablet servers on RHEL9 with
> {{libgcc-11.5.0-11.el9.x86_64}} are below:
> {noformat}
> *** SIGSEGV (@0x0) received by PID 138135 (TID 0x7fc277d56640) from PID 0;
> stack trace: ***
>
> @ 0xb6a837 google::(anonymous
> namespace)::FailureSignalHandler() @ 0x7fc2bb63fc30 (unknown)
>
> @ 0x7fc2bc446c5d (unknown)
>
> @ 0x7fc2bc4473af (unknown)
>
> @ 0x10b9f51 llvm::RTDyldMemoryManager::registerEHFrames()
>
> @ 0x10cc2af llvm::RuntimeDyldELF::registerEHFrames()
>
> @ 0x109fbe1 llvm::MCJIT::finalizeLoadedModules()
>
> @ 0x10a0165 llvm::MCJIT::finalizeObject()
>
> @ 0xe0b0c8 kudu::codegen::ModuleBuilder::Compile()
>
> @ 0xe05a28 kudu::codegen::RowProjectorFunctions::Create()
>
> @ 0xe07214 kudu::codegen::CodeGenerator::CompileRowProjector()
>
> @ 0xdfe3a3 kudu::codegen::(anonymous
> namespace)::CompilationTask::Run()
> {noformat}
> {noformat}
> *** SIGSEGV (@0x0) received by PID 639256 (TID 0x7f4333432640) from PID 0;
> stack trace: ***
> @ 0xb6b2c7 google::(anonymous
> namespace)::FailureSignalHandler()
> @ 0x7f4419e3fc30 (unknown)
>
> @ 0x7f441ac23c5d (unknown)
>
> @ 0x7f441ac24a08 (unknown)
>
> @ 0x7f441ac25fec __deregister_frame_info_bases
>
> @ 0x7f441ac26029 __deregister_frame
>
> @ 0x10cc3af llvm::RuntimeDyldELF::deregisterEHFrames()
>
> @ 0x10a26b9 llvm::MCJIT::~MCJIT()
>
> @ 0x10a2e52 llvm::MCJIT::~MCJIT()
>
> @ 0xe08f78 kudu::codegen::JITWrapper::~JITWrapper()
>
> @ 0xe07654
> kudu::codegen::RowProjectorFunctions::~RowProjectorFunctions()
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
