[
https://issues.apache.org/jira/browse/KUDU-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18056246#comment-18056246
]
ASF subversion and git services commented on KUDU-3736:
-------------------------------------------------------
Commit 0efbebd487e7d0c7f21ab0d5a91e0fa70046af83 in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=0efbebd48 ]
KUDU-3736 fix SIGSEGV in codegen with libgcc-11.5.0-10+
Former libgcc versions used a mutex-protected linked list to store
information on EH frames. Addressing the scalability bottlenecks of
the __{register,deregister}_frame() API in the former implementation,
the new implementation in unwind-fde-dw2.c [1] and related headers
now uses the read-optimized b-tree approach [2]. The b-tree uses
the start address of a memory range as key.
While the lock and the related bottleneck are now gone, the b-tree-based
implementation is heavily reliant on a few invariants regarding the
properties of frames/sections being (de)registered. In particular,
there is a range invariant (a.k.a. the "non-overlapping rule") which
the new implementation heavily relies upon when updating and rebalancing
the b-tree when inserting/removing elements. At least, this rule is
referred to in the commit description at [3]:
* There must not be two frames that have overlapping address ranges
While the former implementation might have tolerated when this rule
wasn't honored, the new b-tree implementation is susceptible to UB
(undefined behavior) in such a case because the tree logic assumes
the presence of clear range boundaries for its separators.
As it turns out, this applies not only to a pair of particular frame
description entries (FDEs), but also to the whole range span of FDEs
that come along with corresponding CIE entry in the data structure
that is supplied to __{register,deregister}_frame() invocations. In
particular, in classify_object_over_fdes() the span is calculated by
going over each FDE of the supplied CIE/FDE chunk, and, as a result,
get_pc_range() returns an address range spanning from
min(beginning address of all FDEs) up to max(end address of all FDEs).
In its turn, the implementation of RuntimeDyld's SectionMemoryManager
in LLVM uses mmap() with MAP_ANON and MAP_PRIVATE flags to allocate
memory for jitted object sections, including the .eh_frame section.
That results in quite an arbitrary placement of the mmap-ed/allocated
memory ranges since the placement of an allocated range isn't controlled
by MCJIT execution engine beyond providing an 'address hint' for the
placement of a newly allocated memory range. The kernel is free to use
any range in the address space of the process that it finds appropriate
given the size of the requested memory range if its first attempt to
establish the new mapping at the closest memory page boundary fails
because there is an existing memory mapping at that address already [4].
The address space of a running Kudu tablet server process might become
fragmented when there are many jitted code references kept alive, while
other references might have been used and then purged out of the codegen
cache and other references dropped by already completed scanners,
compaction operations, etc. Eventually, it's possible to end up in a
memory layout that's illustrated below:
object A sections: [.....][...][...]
object B sections: [.....] [...][...]
.eh_frame section contents for the layout above wouldn't comply with
the "non-overlapping rule" for the FDEs span, and the new libgcc
implementation could end up with receiving SIGSEGV in an attempt to
register .eh_frame section for one of those objects or updating the
b-tree upon de-registering corresponding .eh_frame section.
The situation described above manifested itself in Kudu tablet servers
crashing with SIGSEGV on RHEL9 with libgcc 11.5.0-11 installed.
The libgcc package of 11.5.0-11 version came with the updates pushed
into the RedHat package repos along with newly released RHEL 9.7 on
2025-11-12 (November 11, 2025). From the libgcc's package changelog [5]
it's clear that RedHat's libgcc switched to the b-tree-based EH frame
implementation since version 11.5.0-10, and it's here to stay.
This patch addresses the issue of Kudu tablet servers crashing in the
codegen by providing a custom implementation of the section memory
manager for the LLVM's MCJIT execution engine. In essence, it reserves
the memory area for subsequent allocations of all the codegenned
objects' sections, so they are now guaranteed to be adjacent and
localized in the pre-allocated memory area. This approach ensures
that the ranges of all the FDEs provided to the libgcc's
__{register,deregister}_frame() API cannot interleave with ranges
of any other FDEs and range spans defined by their boundary entries
registered by a Kudu tablet server during its life cycle.
[1]
https://github.com/gcc-mirror/gcc/blob/1c0305d7aea53d788f3f74ca9a2bd9fb764c0109/libgcc/unwind-dw2-fde.c
[2]
https://github.com/gcc-mirror/gcc/blob/1c0305d7aea53d788f3f74ca9a2bd9fb764c0109/libgcc/unwind-dw2-btree.h
[3]
https://gcc.gnu.org/cgit/gcc/commit/libgcc/unwind-dw2-btree.h?id=21109b37e8585a7a1b27650fcbf1749380016108
[4] https://www.man7.org/linux/man-pages/man2/mmap.2.html
[5]
https://rhel.pkgs.org/9/red-hat-ubi-baseos-x86_64/libgcc-11.5.0-11.el9.x86_64.rpm.html
Change-Id: I691d2f442c3148f235847c4c8e56767577804b1a
Reviewed-on: http://gerrit.cloudera.org:8080/23925
Reviewed-by: Marton Greber <[email protected]>
Tested-by: Marton Greber <[email protected]>
Reviewed-by: Ashwani Raina <[email protected]>
> Kudu tablet server's codegen might crash with SIGSEGV on RedHat9 after
> upgrading libgcc to 11.5.0-10 and newer versions
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: KUDU-3736
> URL: https://issues.apache.org/jira/browse/KUDU-3736
> Project: Kudu
> Issue Type: Bug
> Components: codegen, tserver
> Affects Versions: 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.11.0, 1.12.0, 1.11.1,
> 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.17.1, 1.18.1
> Reporter: Alexey Serbin
> Assignee: Alexey Serbin
> Priority: Critical
>
> On RedHat9 OS, x86_64 architecture, after installing updates published by
> RedHat on 2025-11-12, or just upgrading up to or installing RHEL9.7 fresh,
> Kudu tablet servers might crash with SIGSEGV.
> This happens once upgrading libgcc up to 11.5.0-10 and newer versions.
> The workaround is to disable code generation for row projections by adding
> the following customization into the set of flags that Kudu tablet servers
> start with:
> {noformat}
> --mrs_use_codegen=false
> {noformat}
> Examples of the top of stack traces of crashed tablet servers on RHEL9 with
> {{libgcc-11.5.0-11.el9.x86_64}} are below:
> {noformat}
> *** SIGSEGV (@0x0) received by PID 138135 (TID 0x7fc277d56640) from PID 0;
> stack trace: ***
>
> @ 0xb6a837 google::(anonymous
> namespace)::FailureSignalHandler() @ 0x7fc2bb63fc30 (unknown)
>
> @ 0x7fc2bc446c5d (unknown)
>
> @ 0x7fc2bc4473af (unknown)
>
> @ 0x10b9f51 llvm::RTDyldMemoryManager::registerEHFrames()
>
> @ 0x10cc2af llvm::RuntimeDyldELF::registerEHFrames()
>
> @ 0x109fbe1 llvm::MCJIT::finalizeLoadedModules()
>
> @ 0x10a0165 llvm::MCJIT::finalizeObject()
>
> @ 0xe0b0c8 kudu::codegen::ModuleBuilder::Compile()
>
> @ 0xe05a28 kudu::codegen::RowProjectorFunctions::Create()
>
> @ 0xe07214 kudu::codegen::CodeGenerator::CompileRowProjector()
>
> @ 0xdfe3a3 kudu::codegen::(anonymous
> namespace)::CompilationTask::Run()
> {noformat}
> {noformat}
> *** SIGSEGV (@0x0) received by PID 639256 (TID 0x7f4333432640) from PID 0;
> stack trace: ***
> @ 0xb6b2c7 google::(anonymous
> namespace)::FailureSignalHandler()
> @ 0x7f4419e3fc30 (unknown)
>
> @ 0x7f441ac23c5d (unknown)
>
> @ 0x7f441ac24a08 (unknown)
>
> @ 0x7f441ac25fec __deregister_frame_info_bases
>
> @ 0x7f441ac26029 __deregister_frame
>
> @ 0x10cc3af llvm::RuntimeDyldELF::deregisterEHFrames()
>
> @ 0x10a26b9 llvm::MCJIT::~MCJIT()
>
> @ 0x10a2e52 llvm::MCJIT::~MCJIT()
>
> @ 0xe08f78 kudu::codegen::JITWrapper::~JITWrapper()
>
> @ 0xe07654
> kudu::codegen::RowProjectorFunctions::~RowProjectorFunctions()
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
