> On Apr 26, 2023, at 7:12 PM, Konstantin Belousov wrote:
>
> On Wed, Apr 26, 2023 at 12:55:02PM +0200, Hans Petter Selasky wrote:
>> On 4/26/23 12:36, Zhenlei Huang wrote:
>>> Hi,
>>>
>>> I'm recently working on https://reviews.freebsd.org/D39638 (sysctl(9):
>>> Enable vnet sysctl variables be loader tunable),
>>> the changes to `sys/kern/link_elf_obj.c` are runtime tested, but not those
>>> to `sys/kern/link_elf.c` .
>>>
>>> After some hacking I realized that `link_elf.c` is for EXEC (Executable
>>> file) or DYN (Shared object file), and `link_elf_obj.c` is
>>> for REL (Relocatable file).
>>>
>>> ```
>>> /* link_elf.c */
>>> static int
>>> link_elf_load_file(linker_class_t cls, const char* filename,
>>> linker_file_t* result)
>>> {
>>> ...
>>> if (hdr->e_type != ET_EXEC && hdr->e_type != ET_DYN) {
>>> error = ENOSYS;
>>> goto out;
>>> }
>>> ...
>>> }
>>>
>>>
>>> /* link_elf_obj.c */
>>> static int
>>> link_elf_load_file(linker_class_t cls, const char *filename,
>>> linker_file_t *result)
>>> {
>>> ...
>>> if (hdr->e_type != ET_REL) {
>>> error = ENOSYS;
>>> goto out;
>>> }
>>> ...
>>> }
>>> ```
>>>
>>> Run the following snip:
>>> ```
>>> # find /boot/kernel -type f -name "*.ko" -exec readelf -h {} \; | grep Type
>>> ```
>>> shows that all the kernel modules' types are `REL (Relocatable file)`.
>>>
>>> I guess if some module such as if_bridge is linked to DYN type, then I can
>>> do runtime for the changes to `sys/kern/link_elf.c`.
>>>
>>> I'm not familiar with elf and linkers, is that ( compile module and link it
>>> to DYN type ) possible ?
>
> Module file type (shared object vs. object file) depends on architecture.
> For amd64 modules are objects, while kernel is shared library.
> For arm64 (and all other arches, I believe) modules and kernels are shared
> libraries.
>
> I think you can link amd64 module as shared object, but this require enough
> hacking of the build infrastructure. At least I am not aware of a simple
> knob to switch the produced type.
I did some hack on `sys/conf/kmod.mk` and finally produced DYN kernel modules.
The good news is I do some basic sysctl tests, but the bad news is the module
does not function correctly.
For exampe the if_disc.c
```
static void
vnet_disc_init(const void *unused __unused)
{
/* Reference V_disc_cloner will immediately trigger page fault panic */
V_disc_cloner = if_clone_simple(discname, disc_clone_create,
disc_clone_destroy, 0);
}
VNET_SYSINIT(vnet_disc_init, SI_SUB_PSEUDO, SI_ORDER_ANY,
vnet_disc_init, NULL);
```
I suspect the relocation is not done correctly for DYN elf kmod on amd64.
My local patch to kmod.mk:
```
diff --git a/sys/conf/kmod.mk b/sys/conf/kmod.mk
index 134b150af1d9..1fc5386204a5 100644
--- a/sys/conf/kmod.mk
+++ b/sys/conf/kmod.mk
@@ -84,6 +84,7 @@ __KLD_SHARED=yes
.else
__KLD_SHARED=no
.endif
+__KLD_SHARED=yes
.if !empty(CFLAGS:M-O[23s]) && empty(CFLAGS:M-fno-strict-aliasing)
CFLAGS+= -fno-strict-aliasing
@@ -167,6 +168,7 @@ CFLAGS+=-fno-omit-frame-pointer
-mno-omit-leaf-frame-pointer
${MACHINE_CPUARCH} == "powerpc"
CFLAGS+= -fPIC
.endif
+CFLAGS+= -fPIC
# Temporary workaround for PR 196407, which contains the fascinating details.
# Don't allow clang to use fpu instructions or registers in kernel modules.
```
As for https://reviews.freebsd.org/D39638 ,
for other platform such as arm, I think the `link_elf_propagate_vnets()` should
work if `parse_vnet()` works.
I'll appreciate if someone can test it on platforms those have DYN type kernel
modules.
>
>
>>>
>>
>> Hi,
>>
>> I don't have an answer for you either, but I have seen in the past, loading
>> kernel modules behaves a bit like libraries, in the following regard:
>>
>> If two kernel modules define the same global symbol, then no warning is
>> given and the first loaded symbol definition (I think) is used to resolve
>> that symbol for all kernel modules, regardless of the prototype. Probably we
>> should not allow this. That's why building LINT is a good thing, to avoid
>> this issue.
> No, in-kernel linker does not behave this way.
> Modules need to contain explicit reference to all modules they depend upon,
> using the MODULE_DEPEND() macro. Only symbols from the dependencies are
> resolved.
>
> All modules get an implicit reference to kernel.
>
>>
>> Even if we don't have C++ support in the FreeBSD kernel, defining symbol
>> names the way C++ does for C could be nice for the kernel too, also with
>> regards to debugging systems.
>>
>> Many times when I don't know what is going on, I do like this:
>>
>> #include
>>
>>
>>
>> if (not too fast or my sysctl debug) {
>> printf("My tracer\n");
>> kdb_backtrace();
>> }
>>
>> Dtrace can also do this, but not during boot. Just track who is calling
>> those functions, and you'll probably find the answer
