Re: [PATCH v3 00/20] Implement DWARF modversions
On Mon, Sep 23, 2024 at 2:19 PM Sami Tolvanen wrote:
>
> Hi,
>
> Here's v3 of the DWARF modversions series [1][2]. The main
> motivation remains modversions support for Rust, which is important
> for distributions like Android that are eager to ship Rust kernel
> modules. Per Luis' request [3], v2 dropped the Rust specific bits
> from the series and instead added the feature as an option for
> the entire kernel. Matt is still addressing Rust modversion_info
> compatibility issues in a separate series [4], and we'll follow up
> with a patch to actually allow CONFIG_MODVERSIONS with Rust once
> everything else has been sorted out.
>
> A short background recap: Unlike C, Rust source code doesn't
> have sufficient information about the final ABI, as the compiler
> has considerable freedom in adjusting structure layout, for
> example, which makes using a source code parser like genksyms a
> non-starter. Based on Matt's suggestion and previous feedback from
> maintainers, this series uses DWARF debugging information for
> computing versions. DWARF is an established and a relatively stable
> format, which includes all the necessary ABI details, and adding a
> CONFIG_DEBUG_INFO dependency for Rust symbol versioning seems like a
> reasonable trade-off.
>
> The first two patches add more list macros to scripts/include and
> move the genksyms CRC32 implementation to a shared header file. The
> next 15 patches add gendwarfksyms, a tool for computing symbol
> versions from DWARF. When passed a list of exported symbols and
> object files, the tool generates an expanded type string for each
> symbol and computes symbol CRCs similarly to genksyms. gendwarfksyms
> is written in C and uses libdw to process DWARF. Patch 18 ensures
> that debugging information is present where we need it, patch 19
> adds gendwarfksyms as an alternative to genksyms, and the last patch
> adds documentation.
>
> Note that v3 is based on next-20240923 as it depends on Masahiro's
> scripts/include changes. For x86, we also need a separate small
> patch to include asm/ptrace.h in asm/ftrace.h. [5] For your
> convenience, you can find this series with all the prerequisites
> here:
>
> https://github.com/samitolvanen/linux/commits/gendwarfksyms-v3
>
> If you also want to test the series with Rust modules, this branch
> adds Matt's modversion_info series and a small patch to enable Rust
> modversions:
>
> https://github.com/samitolvanen/linux/commits/rustmodversions-v3
>
> Looking forward to hearing your thoughts!
>
> Sami
>
>
> [1] v1:
> https://lore.kernel.org/lkml/20240617175818.58219-17-samitolva...@google.com/
> [2] v2:
> https://lore.kernel.org/lkml/20240815173903.4172139-21-samitolva...@google.com/
> [3] https://lore.kernel.org/lkml/znizetkkqweig...@bombadil.infradead.org/
> [4] https://lore.kernel.org/lkml/20240806212106.617164-1-mmau...@google.com/
> [5]
> https://lore.kernel.org/lkml/20240916221557.846853-2-samitolva...@google.com/
>
> ---
>
> Changes in v3:
> - Updated SPX license headers.
>
> - Squashed the first two patches in v2 and tried to reduce churn as
> much as reasonable.
>
> - Dropped patch 18 from v2 ("x86/asm-prototypes: Include
> ") as it's addressed by a separate patch. [5]
>
> - Changed the error handling code to immediately terminate instead
> of propagating the errors back to main, which cleaned up the code
> quite a bit.
>
> - Switched to the list and hashtable implementations in scripts and
> dropped the remaining tools/include dependencies. Added a couple
> missing list macros. (patch 1)
>
> - Moved the genksyms CRC32 implementation to scripts/include and
> dropped the duplicate code. (patches 2 and 14)
>
> - Switched from ad-hoc command line parsing to getopt_long (patch 3).
>
> - Added structure member and function parameter names to the DIE
> output to match genksyms behavior, and tweaked the symtypes format
> to be more parser-friendly in general based on Petr's suggestions.
>
> - Replaced the declaration-only struct annotations with more generic
> kABI stability rules that allow source code annotations to be used
> where #ifndef __GENKSYMS__ was previously used. Added support for
> rules that can be used to exclude enumerators from versioning.
> (patch 16)
>
> - Per Miroslav's suggestion, added an option to hide structure
> members from versioning when they're added to existing alignment
> holes, for example. (patch 16)
>
> - Per Greg's request, added documentation and example macros for the
> --stable features, and a couple of test cases. (patches 15, 16, and
> 20)
>
> - Fixed making symtypes files, which need to depend on .o files with
> gendwarfksyms. (patch 19)
>
> - Addressed several other smaller issues that Petr and Masahiro
> kindly pointed out during the v2 review.
>
> Changes in v2:
> - Per Luis' request, dropped Rust-specific patches and added
> gendwarfksyms as an alternative to genksyms for the entire
> kernel.
>
> - Added support for missing DWARF feature
[PATCH v3 00/20] Implement DWARF modversions
Hi,
Here's v3 of the DWARF modversions series [1][2]. The main
motivation remains modversions support for Rust, which is important
for distributions like Android that are eager to ship Rust kernel
modules. Per Luis' request [3], v2 dropped the Rust specific bits
from the series and instead added the feature as an option for
the entire kernel. Matt is still addressing Rust modversion_info
compatibility issues in a separate series [4], and we'll follow up
with a patch to actually allow CONFIG_MODVERSIONS with Rust once
everything else has been sorted out.
A short background recap: Unlike C, Rust source code doesn't
have sufficient information about the final ABI, as the compiler
has considerable freedom in adjusting structure layout, for
example, which makes using a source code parser like genksyms a
non-starter. Based on Matt's suggestion and previous feedback from
maintainers, this series uses DWARF debugging information for
computing versions. DWARF is an established and a relatively stable
format, which includes all the necessary ABI details, and adding a
CONFIG_DEBUG_INFO dependency for Rust symbol versioning seems like a
reasonable trade-off.
The first two patches add more list macros to scripts/include and
move the genksyms CRC32 implementation to a shared header file. The
next 15 patches add gendwarfksyms, a tool for computing symbol
versions from DWARF. When passed a list of exported symbols and
object files, the tool generates an expanded type string for each
symbol and computes symbol CRCs similarly to genksyms. gendwarfksyms
is written in C and uses libdw to process DWARF. Patch 18 ensures
that debugging information is present where we need it, patch 19
adds gendwarfksyms as an alternative to genksyms, and the last patch
adds documentation.
Note that v3 is based on next-20240923 as it depends on Masahiro's
scripts/include changes. For x86, we also need a separate small
patch to include asm/ptrace.h in asm/ftrace.h. [5] For your
convenience, you can find this series with all the prerequisites
here:
https://github.com/samitolvanen/linux/commits/gendwarfksyms-v3
If you also want to test the series with Rust modules, this branch
adds Matt's modversion_info series and a small patch to enable Rust
modversions:
https://github.com/samitolvanen/linux/commits/rustmodversions-v3
Looking forward to hearing your thoughts!
Sami
[1] v1:
https://lore.kernel.org/lkml/20240617175818.58219-17-samitolva...@google.com/
[2] v2:
https://lore.kernel.org/lkml/20240815173903.4172139-21-samitolva...@google.com/
[3] https://lore.kernel.org/lkml/znizetkkqweig...@bombadil.infradead.org/
[4] https://lore.kernel.org/lkml/20240806212106.617164-1-mmau...@google.com/
[5]
https://lore.kernel.org/lkml/20240916221557.846853-2-samitolva...@google.com/
---
Changes in v3:
- Updated SPX license headers.
- Squashed the first two patches in v2 and tried to reduce churn as
much as reasonable.
- Dropped patch 18 from v2 ("x86/asm-prototypes: Include
") as it's addressed by a separate patch. [5]
- Changed the error handling code to immediately terminate instead
of propagating the errors back to main, which cleaned up the code
quite a bit.
- Switched to the list and hashtable implementations in scripts and
dropped the remaining tools/include dependencies. Added a couple
missing list macros. (patch 1)
- Moved the genksyms CRC32 implementation to scripts/include and
dropped the duplicate code. (patches 2 and 14)
- Switched from ad-hoc command line parsing to getopt_long (patch 3).
- Added structure member and function parameter names to the DIE
output to match genksyms behavior, and tweaked the symtypes format
to be more parser-friendly in general based on Petr's suggestions.
- Replaced the declaration-only struct annotations with more generic
kABI stability rules that allow source code annotations to be used
where #ifndef __GENKSYMS__ was previously used. Added support for
rules that can be used to exclude enumerators from versioning.
(patch 16)
- Per Miroslav's suggestion, added an option to hide structure
members from versioning when they're added to existing alignment
holes, for example. (patch 16)
- Per Greg's request, added documentation and example macros for the
--stable features, and a couple of test cases. (patches 15, 16, and
20)
- Fixed making symtypes files, which need to depend on .o files with
gendwarfksyms. (patch 19)
- Addressed several other smaller issues that Petr and Masahiro
kindly pointed out during the v2 review.
Changes in v2:
- Per Luis' request, dropped Rust-specific patches and added
gendwarfksyms as an alternative to genksyms for the entire
kernel.
- Added support for missing DWARF features needed to handle
also non-Rust code.
- Changed symbol address matching to use the symbol table
information instead of relying on addresses in DWARF.
- Added __gendwarfksyms_ptr patches to ensure the compiler emits
the necessary type information in DWA
