Re: request_module DoS
On Thu, May 12, 2022 at 10:07:26PM +1000, Michael Ellerman wrote: > Michael Ellerman writes: > > Luis Chamberlain writes: > ... > > > >> Can someone try this on ppc64le system? At this point I am not convinced > >> this issue is generic. > > > > Does your x86 system have at least 784 CPUs? > > > > I don't know where the original report came from, but the trace shows > > "CPU 784", which would usually indicate a system with at least that many > > CPUs. > > Update, apparently the report originally came from IBM, so I'll chase it > up internally. > > I think you're right that there's probably no issue in the module code, > sorry to waste your time. It gives me testing happiness to know that may be the case :) Luis
Re: request_module DoS
Michael Ellerman writes: > Luis Chamberlain writes: ... > >> Can someone try this on ppc64le system? At this point I am not convinced >> this issue is generic. > > Does your x86 system have at least 784 CPUs? > > I don't know where the original report came from, but the trace shows > "CPU 784", which would usually indicate a system with at least that many > CPUs. Update, apparently the report originally came from IBM, so I'll chase it up internally. I think you're right that there's probably no issue in the module code, sorry to waste your time. cheers
Re: request_module DoS
Luis Chamberlain writes: > On Mon, May 09, 2022 at 09:13:03AM -0700, Luis Chamberlain wrote: >> On Mon, May 09, 2022 at 09:23:39PM +1000, Michael Ellerman wrote: >> > Herbert Xu writes: >> > > Hi: >> > > >> > > There are some code paths in the kernel where you can reliably >> > > trigger a request_module of a non-existant module. For example, >> > > if you attempt to load a non-existent crypto algorithm, or create >> > > a socket of a non-existent network family, it will result in a >> > > request_module call that is guaranteed to fail. >> > > >> > > As user-space can do this repeatedly, it can quickly overwhelm >> > > the concurrency limit in kmod. This in itself is expected, >> > > however, at least on some platforms this appears to result in >> > > a live-lock. Here is an example triggered by stress-ng on ppc64: >> > > >> > > [ 529.853264] request_module: kmod_concurrent_max (0) close to 0 >> > > (max_modprobes: 50), for module crypto-aegis128l, throttling... >> > ... >> > > [ 580.414590] __request_module: 25 callbacks suppressed >> > > [ 580.414597] request_module: kmod_concurrent_max (0) close to 0 >> > > (max_modprobes: 50), for module crypto-aegis256-all, throttling... >> > > [ 580.423082] watchdog: CPU 784 self-detected hard LOCKUP @ >> > > plpar_hcall_norets_notrace+0x18/0x2c >> > > [ 580.423097] watchdog: CPU 784 TB:1297691958559475, last heartbeat >> > > TB:1297686321743840 (11009ms ago) >> > > [ 580.423099] Modules linked in: cast6_generic cast5_generic >> > > cast_common camellia_generic blowfish_generic blowfish_common tun >> > > nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet >> > > nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat >> > > nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill bonding tls ip_set >> > > nf_tables nfnetlink pseries_rng binfmt_misc drm >> > > drm_panel_orientation_quirks xfs libcrc32c sd_mod t10_pi sg ibmvscsi >> > > ibmveth scsi_transport_srp vmx_crypto dm_mirror dm_region_hash dm_log >> > > dm_mod fuse >> > > [ 580.423136] CPU: 784 PID: 77071 Comm: stress-ng Kdump: loaded Not >> > > tainted 5.14.0-55.el9.ppc64le #1 >> > > [ 580.423139] NIP: c00f8ff4 LR: c01f7c38 CTR: >> > > >> > > [ 580.423140] REGS: c043fdd7bd60 TRAP: 0900 Not tainted >> > > (5.14.0-55.el9.ppc64le) >> > > [ 580.423142] MSR: 8280b033 >> > > CR: 28008202 XER: 2004 >> > > [ 580.423148] CFAR: 0c00 IRQMASK: 1 >> > >GPR00: 28008202 c044c46b3850 c2a46f00 >> > > >> > >GPR04: 0010 >> > > c2a83060 >> > >GPR08: 0001 0001 >> > > >> > >GPR12: c01b9530 c043ffe16700 00020117 >> > > 10185ea8 >> > >GPR16: 10212150 10186198 101863a0 >> > > 1021b3c0 >> > >GPR20: 0001 0001 >> > > 00ff >> > >GPR24: c043f4a00e14 c043fafe0e00 0c44 >> > > >> > >GPR28: c043f4a00e00 c043f4a00e00 c21e0e00 >> > > c2561aa0 >> > > [ 580.423166] NIP [c00f8ff4] >> > > plpar_hcall_norets_notrace+0x18/0x2c >> > > [ 580.423168] LR [c01f7c38] >> > > __pv_queued_spin_lock_slowpath+0x528/0x530 >> > > [ 580.423173] Call Trace: >> > > [ 580.423174] [c044c46b3850] [00016b60] 0x16b60 >> > > (unreliable) >> > > [ 580.423177] [c044c46b3910] [c0ea6948] >> > > _raw_spin_lock_irqsave+0xa8/0xc0 >> > > [ 580.423182] [c044c46b3940] [c01dd7c0] >> > > prepare_to_wait_event+0x40/0x200 >> > > [ 580.423185] [c044c46b39a0] [c019e9e0] >> > > __request_module+0x320/0x510 >> > > [ 580.423188] [c044c46b3ac0] [c06f1a14] >> > > crypto_alg_mod_lookup+0x1e4/0x2e0 >> > > [ 580.423192] [c044c46b3b60] [c06f2178] >> > > crypto_alloc_tfm_node+0xa8/0x1a0 >> > > [ 580.423194] [c044c46b3be0] [c06f84f8] >> > > crypto_alloc_aead+0x38/0x50 >> > > [ 580.423196] [c044c46b3c00] [c072cba0] aead_bind+0x70/0x140 >> > > [ 580.423199] [c044c46b3c40] [c0727824] alg_bind+0xb4/0x210 >> > > [ 580.423201] [c044c46b3cc0] [c0bc2ad4] >> > > __sys_bind+0x114/0x160 >> > > [ 580.423205] [c044c46b3d90] [c0bc2b48] sys_bind+0x28/0x40 >> > > [ 580.423207] [c044c46b3db0] [c0030880] >> > > system_call_exception+0x160/0x300 >> > > [ 580.423209] [c044c46b3e10] [c000c168] >> > > system_call_vectored_common+0xe8/0x278 >> > > [ 580.423213] --- interrupt: 3000 at 0x7fff9b824464 >> > > [ 580.423214] NIP: 7fff9b824464 LR: CTR: >> > > >> > > [ 580.423215] REGS: c044c46b3e80 TRAP: 3000 Not t
Re: request_module DoS
On Mon, May 09, 2022 at 09:13:03AM -0700, Luis Chamberlain wrote: > On Mon, May 09, 2022 at 09:23:39PM +1000, Michael Ellerman wrote: > > Herbert Xu writes: > > > Hi: > > > > > > There are some code paths in the kernel where you can reliably > > > trigger a request_module of a non-existant module. For example, > > > if you attempt to load a non-existent crypto algorithm, or create > > > a socket of a non-existent network family, it will result in a > > > request_module call that is guaranteed to fail. > > > > > > As user-space can do this repeatedly, it can quickly overwhelm > > > the concurrency limit in kmod. This in itself is expected, > > > however, at least on some platforms this appears to result in > > > a live-lock. Here is an example triggered by stress-ng on ppc64: > > > > > > [ 529.853264] request_module: kmod_concurrent_max (0) close to 0 > > > (max_modprobes: 50), for module crypto-aegis128l, throttling... > > ... > > > [ 580.414590] __request_module: 25 callbacks suppressed > > > [ 580.414597] request_module: kmod_concurrent_max (0) close to 0 > > > (max_modprobes: 50), for module crypto-aegis256-all, throttling... > > > [ 580.423082] watchdog: CPU 784 self-detected hard LOCKUP @ > > > plpar_hcall_norets_notrace+0x18/0x2c > > > [ 580.423097] watchdog: CPU 784 TB:1297691958559475, last heartbeat > > > TB:1297686321743840 (11009ms ago) > > > [ 580.423099] Modules linked in: cast6_generic cast5_generic cast_common > > > camellia_generic blowfish_generic blowfish_common tun nft_fib_inet > > > nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 > > > nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack > > > nf_defrag_ipv6 nf_defrag_ipv4 rfkill bonding tls ip_set nf_tables > > > nfnetlink pseries_rng binfmt_misc drm drm_panel_orientation_quirks xfs > > > libcrc32c sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp vmx_crypto > > > dm_mirror dm_region_hash dm_log dm_mod fuse > > > [ 580.423136] CPU: 784 PID: 77071 Comm: stress-ng Kdump: loaded Not > > > tainted 5.14.0-55.el9.ppc64le #1 > > > [ 580.423139] NIP: c00f8ff4 LR: c01f7c38 CTR: > > > > > > [ 580.423140] REGS: c043fdd7bd60 TRAP: 0900 Not tainted > > > (5.14.0-55.el9.ppc64le) > > > [ 580.423142] MSR: 8280b033 > > > CR: 28008202 XER: 2004 > > > [ 580.423148] CFAR: 0c00 IRQMASK: 1 > > >GPR00: 28008202 c044c46b3850 c2a46f00 > > > > > >GPR04: 0010 > > > c2a83060 > > >GPR08: 0001 0001 > > > > > >GPR12: c01b9530 c043ffe16700 00020117 > > > 10185ea8 > > >GPR16: 10212150 10186198 101863a0 > > > 1021b3c0 > > >GPR20: 0001 0001 > > > 00ff > > >GPR24: c043f4a00e14 c043fafe0e00 0c44 > > > > > >GPR28: c043f4a00e00 c043f4a00e00 c21e0e00 > > > c2561aa0 > > > [ 580.423166] NIP [c00f8ff4] plpar_hcall_norets_notrace+0x18/0x2c > > > [ 580.423168] LR [c01f7c38] > > > __pv_queued_spin_lock_slowpath+0x528/0x530 > > > [ 580.423173] Call Trace: > > > [ 580.423174] [c044c46b3850] [00016b60] 0x16b60 > > > (unreliable) > > > [ 580.423177] [c044c46b3910] [c0ea6948] > > > _raw_spin_lock_irqsave+0xa8/0xc0 > > > [ 580.423182] [c044c46b3940] [c01dd7c0] > > > prepare_to_wait_event+0x40/0x200 > > > [ 580.423185] [c044c46b39a0] [c019e9e0] > > > __request_module+0x320/0x510 > > > [ 580.423188] [c044c46b3ac0] [c06f1a14] > > > crypto_alg_mod_lookup+0x1e4/0x2e0 > > > [ 580.423192] [c044c46b3b60] [c06f2178] > > > crypto_alloc_tfm_node+0xa8/0x1a0 > > > [ 580.423194] [c044c46b3be0] [c06f84f8] > > > crypto_alloc_aead+0x38/0x50 > > > [ 580.423196] [c044c46b3c00] [c072cba0] aead_bind+0x70/0x140 > > > [ 580.423199] [c044c46b3c40] [c0727824] alg_bind+0xb4/0x210 > > > [ 580.423201] [c044c46b3cc0] [c0bc2ad4] > > > __sys_bind+0x114/0x160 > > > [ 580.423205] [c044c46b3d90] [c0bc2b48] sys_bind+0x28/0x40 > > > [ 580.423207] [c044c46b3db0] [c0030880] > > > system_call_exception+0x160/0x300 > > > [ 580.423209] [c044c46b3e10] [c000c168] > > > system_call_vectored_common+0xe8/0x278 > > > [ 580.423213] --- interrupt: 3000 at 0x7fff9b824464 > > > [ 580.423214] NIP: 7fff9b824464 LR: CTR: > > > > > > [ 580.423215] REGS: c044c46b3e80 TRAP: 3000 Not tainted > > > (5.14.0-55.el9.ppc64le) > > > [ 580.423216] MSR: 8280f033 > > > CR: 42004802 XER: > >
Re: request_module DoS
On Mon, May 09, 2022 at 09:23:39PM +1000, Michael Ellerman wrote: > Herbert Xu writes: > > Hi: > > > > There are some code paths in the kernel where you can reliably > > trigger a request_module of a non-existant module. For example, > > if you attempt to load a non-existent crypto algorithm, or create > > a socket of a non-existent network family, it will result in a > > request_module call that is guaranteed to fail. > > > > As user-space can do this repeatedly, it can quickly overwhelm > > the concurrency limit in kmod. This in itself is expected, > > however, at least on some platforms this appears to result in > > a live-lock. Here is an example triggered by stress-ng on ppc64: > > > > [ 529.853264] request_module: kmod_concurrent_max (0) close to 0 > > (max_modprobes: 50), for module crypto-aegis128l, throttling... > ... > > [ 580.414590] __request_module: 25 callbacks suppressed > > [ 580.414597] request_module: kmod_concurrent_max (0) close to 0 > > (max_modprobes: 50), for module crypto-aegis256-all, throttling... > > [ 580.423082] watchdog: CPU 784 self-detected hard LOCKUP @ > > plpar_hcall_norets_notrace+0x18/0x2c > > [ 580.423097] watchdog: CPU 784 TB:1297691958559475, last heartbeat > > TB:1297686321743840 (11009ms ago) > > [ 580.423099] Modules linked in: cast6_generic cast5_generic cast_common > > camellia_generic blowfish_generic blowfish_common tun nft_fib_inet > > nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 > > nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack > > nf_defrag_ipv6 nf_defrag_ipv4 rfkill bonding tls ip_set nf_tables nfnetlink > > pseries_rng binfmt_misc drm drm_panel_orientation_quirks xfs libcrc32c > > sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp vmx_crypto dm_mirror > > dm_region_hash dm_log dm_mod fuse > > [ 580.423136] CPU: 784 PID: 77071 Comm: stress-ng Kdump: loaded Not > > tainted 5.14.0-55.el9.ppc64le #1 > > [ 580.423139] NIP: c00f8ff4 LR: c01f7c38 CTR: > > > > [ 580.423140] REGS: c043fdd7bd60 TRAP: 0900 Not tainted > > (5.14.0-55.el9.ppc64le) > > [ 580.423142] MSR: 8280b033 > > CR: 28008202 XER: 2004 > > [ 580.423148] CFAR: 0c00 IRQMASK: 1 > >GPR00: 28008202 c044c46b3850 c2a46f00 > > > >GPR04: 0010 > > c2a83060 > >GPR08: 0001 0001 > > > >GPR12: c01b9530 c043ffe16700 00020117 > > 10185ea8 > >GPR16: 10212150 10186198 101863a0 > > 1021b3c0 > >GPR20: 0001 0001 > > 00ff > >GPR24: c043f4a00e14 c043fafe0e00 0c44 > > > >GPR28: c043f4a00e00 c043f4a00e00 c21e0e00 > > c2561aa0 > > [ 580.423166] NIP [c00f8ff4] plpar_hcall_norets_notrace+0x18/0x2c > > [ 580.423168] LR [c01f7c38] > > __pv_queued_spin_lock_slowpath+0x528/0x530 > > [ 580.423173] Call Trace: > > [ 580.423174] [c044c46b3850] [00016b60] 0x16b60 > > (unreliable) > > [ 580.423177] [c044c46b3910] [c0ea6948] > > _raw_spin_lock_irqsave+0xa8/0xc0 > > [ 580.423182] [c044c46b3940] [c01dd7c0] > > prepare_to_wait_event+0x40/0x200 > > [ 580.423185] [c044c46b39a0] [c019e9e0] > > __request_module+0x320/0x510 > > [ 580.423188] [c044c46b3ac0] [c06f1a14] > > crypto_alg_mod_lookup+0x1e4/0x2e0 > > [ 580.423192] [c044c46b3b60] [c06f2178] > > crypto_alloc_tfm_node+0xa8/0x1a0 > > [ 580.423194] [c044c46b3be0] [c06f84f8] > > crypto_alloc_aead+0x38/0x50 > > [ 580.423196] [c044c46b3c00] [c072cba0] aead_bind+0x70/0x140 > > [ 580.423199] [c044c46b3c40] [c0727824] alg_bind+0xb4/0x210 > > [ 580.423201] [c044c46b3cc0] [c0bc2ad4] __sys_bind+0x114/0x160 > > [ 580.423205] [c044c46b3d90] [c0bc2b48] sys_bind+0x28/0x40 > > [ 580.423207] [c044c46b3db0] [c0030880] > > system_call_exception+0x160/0x300 > > [ 580.423209] [c044c46b3e10] [c000c168] > > system_call_vectored_common+0xe8/0x278 > > [ 580.423213] --- interrupt: 3000 at 0x7fff9b824464 > > [ 580.423214] NIP: 7fff9b824464 LR: CTR: > > > > [ 580.423215] REGS: c044c46b3e80 TRAP: 3000 Not tainted > > (5.14.0-55.el9.ppc64le) > > [ 580.423216] MSR: 8280f033 > > CR: 42004802 XER: > > [ 580.423221] IRQMASK: 0 > >GPR00: 0147 7fffdcff2780 7fff9b917100 > > 0004 > >GPR04: 7fffdcff27e0 0058 > > > >GPR08: 00
Re: request_module DoS
On Sat, May 07, 2022 at 12:14:47PM -0700, Luis Chamberlain wrote:
> On Sat, May 07, 2022 at 01:02:20AM -0700, Luis Chamberlain wrote:
> > You can try to reproduce by using adding a new test type for crypto-aegis256
> > on lib/test_kmod.c. These tests however can try something similar but other
> > modules.
> >
> > /tools/testing/selftests/kmod/kmod.sh -t 0008
> > /tools/testing/selftests/kmod/kmod.sh -t 0009
> >
> > I can't decipher this yet.
>
> Without testing it... but something like this might be an easier
> reproducer:
>
> + config_set_driver crypto-aegis256
If the module is not present though nothing really happens, and so
is it possible this is another issue?
Below a bogus module request.
diff --git a/tools/testing/selftests/kmod/kmod.sh
b/tools/testing/selftests/kmod/kmod.sh
index afd42387e8b2..a747ad549940 100755
--- a/tools/testing/selftests/kmod/kmod.sh
+++ b/tools/testing/selftests/kmod/kmod.sh
@@ -65,6 +66,7 @@ ALL_TESTS="$ALL_TESTS 0010:1:1"
ALL_TESTS="$ALL_TESTS 0011:1:1"
ALL_TESTS="$ALL_TESTS 0012:1:1"
ALL_TESTS="$ALL_TESTS 0013:1:1"
+ALL_TESTS="$ALL_TESTS 0014:150:1"
# Kselftest framework requirement - SKIP code is 4.
ksft_skip=4
@@ -504,6 +506,17 @@ kmod_test_0013()
"cat /sys/module/${DEFAULT_KMOD_DRIVER}/sections/.*text | head
-n1"
}
+kmod_test_0014()
+{
+ kmod_defaults_driver
+ MODPROBE_LIMIT=$(config_get_modprobe_limit)
+ let EXTRA=$MODPROBE_LIMIT/6
+ config_set_driver bogus_module_does_not_exist
+ config_num_thread_limit_extra $EXTRA
+ config_trigger ${FUNCNAME[0]}
+ config_expect_result ${FUNCNAME[0]} MODULE_NOT_FOUND
+}
+
list_tests()
{
echo "Test ID list:"
@@ -525,6 +538,7 @@ list_tests()
echo "0011 x $(get_test_count 0011) - test completely disabling module
autoloading"
echo "0012 x $(get_test_count 0012) - test /proc/modules address
visibility under CAP_SYSLOG"
echo "0013 x $(get_test_count 0013) - test /sys/module/*/sections/*
visibility under CAP_SYSLOG"
+ echo "0014 x $(get_test_count 0014) - multithreaded - push
kmod_concurrent over max_modprobes for request_module() for a missing module"
}
usage()
Re: request_module DoS
On Sat, May 07, 2022 at 01:02:20AM -0700, Luis Chamberlain wrote:
> You can try to reproduce by using adding a new test type for crypto-aegis256
> on lib/test_kmod.c. These tests however can try something similar but other
> modules.
>
> /tools/testing/selftests/kmod/kmod.sh -t 0008
> /tools/testing/selftests/kmod/kmod.sh -t 0009
>
> I can't decipher this yet.
Without testing it... but something like this might be an easier
reproducer:
diff --git a/tools/testing/selftests/kmod/kmod.sh
b/tools/testing/selftests/kmod/kmod.sh
index afd42387e8b2..48b6b5ec6c1e 100755
--- a/tools/testing/selftests/kmod/kmod.sh
+++ b/tools/testing/selftests/kmod/kmod.sh
@@ -41,6 +41,7 @@ set -e
TEST_NAME="kmod"
TEST_DRIVER="test_${TEST_NAME}"
TEST_DIR=$(dirname $0)
+PROC_CONFIG="/proc/config.gz"
# This represents
#
@@ -65,6 +66,7 @@ ALL_TESTS="$ALL_TESTS 0010:1:1"
ALL_TESTS="$ALL_TESTS 0011:1:1"
ALL_TESTS="$ALL_TESTS 0012:1:1"
ALL_TESTS="$ALL_TESTS 0013:1:1"
+ALL_TESTS="$ALL_TESTS 0014:150:1"
# Kselftest framework requirement - SKIP code is 4.
ksft_skip=4
@@ -79,6 +81,19 @@ test_modprobe()
fi
}
+kconfig_has()
+{
+ if [ -f $PROC_CONFIG ]; then
+ if zgrep -q $1 $PROC_CONFIG 2>/dev/null; then
+ echo "yes"
+ else
+ echo "no"
+ fi
+ else
+ echo "no"
+ fi
+}
+
function allow_user_defaults()
{
if [ -z $DEFAULT_KMOD_DRIVER ]; then
@@ -106,6 +121,8 @@ function allow_user_defaults()
fi
MODPROBE_LIMIT_FILE="${PROC_DIR}/kmod-limit"
+ HAS_CRYPTO_AEGIS256_MOD="$(kconfig_has CONFIG_CRYPTO_AEGIS256=m)"
+ HAS_CRYPTO_AEGIS256_BUILTIN="$(kconfig_has CONFIG_CRYPTO_AEGIS256=y)"
}
test_reqs()
@@ -504,6 +521,21 @@ kmod_test_0013()
"cat /sys/module/${DEFAULT_KMOD_DRIVER}/sections/.*text | head
-n1"
}
+kmod_test_0014()
+{
+ kmod_defaults_driver
+ MODPROBE_LIMIT=$(config_get_modprobe_limit)
+ let EXTRA=$MODPROBE_LIMIT/6
+ config_set_driver crypto-aegis256
+ config_num_thread_limit_extra $EXTRA
+ config_trigger ${FUNCNAME[0]}
+ if [[ "$HAS_CRYPTO_AEGIS256_MOD" == "yes" ||
"$HAS_CRYPTO_AEGIS256_BUILTIN" == "yes" ]]; then
+ config_expect_result ${FUNCNAME[0]} SUCCESS
+ else
+ config_expect_result ${FUNCNAME[0]} MODULE_NOT_FOUND
+ fi
+}
+
list_tests()
{
echo "Test ID list:"
@@ -525,6 +557,7 @@ list_tests()
echo "0011 x $(get_test_count 0011) - test completely disabling module
autoloading"
echo "0012 x $(get_test_count 0012) - test /proc/modules address
visibility under CAP_SYSLOG"
echo "0013 x $(get_test_count 0013) - test /sys/module/*/sections/*
visibility under CAP_SYSLOG"
+ echo "0014 x $(get_test_count 0014) - multithreaded - push
kmod_concurrent over max_modprobes for request_module() for crypto-aegis256"
}
usage()
Re: request_module DoS
On Sat, May 07, 2022 at 07:10:23AM +, Christophe Leroy wrote: > > There are some code paths in the kernel where you can reliably > > trigger a request_module of a non-existant module. For example, > > if you attempt to load a non-existent crypto algorithm, or create > > a socket of a non-existent network family, it will result in a > > request_module call that is guaranteed to fail. > > > > As user-space can do this repeatedly, it can quickly overwhelm > > the concurrency limit in kmod. This in itself is expected, > > however, at least on some platforms this appears to result in > > a live-lock. Here is an example triggered by stress-ng on ppc64: > > > > [ 579.845320] request_module: modprobe crypto-aegis256 cannot be > > processed, kmod busy with 50 threads for more than 5 seconds now > > [ 580.414590] __request_module: 25 callbacks suppressed > > [ 580.414597] request_module: kmod_concurrent_max (0) close to 0 > > (max_modprobes: 50), for module crypto-aegis256-all, throttling... > > [ 580.423082] watchdog: CPU 784 self-detected hard LOCKUP @ > > plpar_hcall_norets_notrace+0x18/0x2c > > [ 580.423097] watchdog: CPU 784 TB:1297691958559475, last heartbeat > > TB:1297686321743840 (11009ms ago) > > [ 580.423099] Modules linked in: cast6_generic cast5_generic cast_common > > camellia_generic blowfish_generic blowfish_common tun nft_fib_inet > > nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 > > nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack > > nf_defrag_ipv6 nf_defrag_ipv4 rfkill bonding tls ip_set nf_tables nfnetlink > > pseries_rng binfmt_misc drm drm_panel_orientation_quirks xfs libcrc32c > > sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp vmx_crypto dm_mirror > > dm_region_hash dm_log dm_mod fuse > > [ 580.423136] CPU: 784 PID: 77071 Comm: stress-ng Kdump: loaded Not > > tainted 5.14.0-55.el9.ppc64le #1 > > [ 580.423139] NIP: c00f8ff4 LR: c01f7c38 CTR: > > > > [ 580.423140] REGS: c043fdd7bd60 TRAP: 0900 Not tainted > > (5.14.0-55.el9.ppc64le) > > [ 580.423142] MSR: 8280b033 > > CR: 28008202 XER: 2004 > > [ 580.423148] CFAR: 0c00 IRQMASK: 1 > > GPR00: 28008202 c044c46b3850 c2a46f00 > > > > GPR04: 0010 > > c2a83060 > > GPR08: 0001 0001 > > > > GPR12: c01b9530 c043ffe16700 00020117 > > 10185ea8 > > GPR16: 10212150 10186198 101863a0 > > 1021b3c0 > > GPR20: 0001 0001 > > 00ff > > GPR24: c043f4a00e14 c043fafe0e00 0c44 > > > > GPR28: c043f4a00e00 c043f4a00e00 c21e0e00 > > c2561aa0 > > [ 580.423166] NIP [c00f8ff4] plpar_hcall_norets_notrace+0x18/0x2c > > [ 580.423168] LR [c01f7c38] > > __pv_queued_spin_lock_slowpath+0x528/0x530 > > [ 580.423173] Call Trace: > > [ 580.423174] [c044c46b3850] [00016b60] 0x16b60 > > (unreliable) > > [ 580.423177] [c044c46b3910] [c0ea6948] > > _raw_spin_lock_irqsave+0xa8/0xc0 > > [ 580.423182] [c044c46b3940] [c01dd7c0] > > prepare_to_wait_event+0x40/0x200 > > [ 580.423185] [c044c46b39a0] [c019e9e0] > > __request_module+0x320/0x510 > > [ 580.423188] [c044c46b3ac0] [c06f1a14] > > crypto_alg_mod_lookup+0x1e4/0x2e0 > > [ 580.423192] [c044c46b3b60] [c06f2178] > > crypto_alloc_tfm_node+0xa8/0x1a0 > > [ 580.423194] [c044c46b3be0] [c06f84f8] > > crypto_alloc_aead+0x38/0x50 > > [ 580.423196] [c044c46b3c00] [c072cba0] aead_bind+0x70/0x140 > > [ 580.423199] [c044c46b3c40] [c0727824] alg_bind+0xb4/0x210 > > [ 580.423201] [c044c46b3cc0] [c0bc2ad4] __sys_bind+0x114/0x160 > > [ 580.423205] [c044c46b3d90] [c0bc2b48] sys_bind+0x28/0x40 > > [ 580.423207] [c044c46b3db0] [c0030880] > > system_call_exception+0x160/0x300 > > [ 580.423209] [c044c46b3e10] [c000c168] > > system_call_vectored_common+0xe8/0x278 > > [ 580.423213] --- interrupt: 3000 at 0x7fff9b824464 > > [ 580.423214] NIP: 7fff9b824464 LR: CTR: > > > > [ 580.423215] REGS: c044c46b3e80 TRAP: 3000 Not tainted > > (5.14.0-55.el9.ppc64le) > > [ 580.423216] MSR: 8280f033 > > CR: 42004802 XER: > > [ 580.423221] IRQMASK: 0 > > GPR00: 0147 7fffdcff2780 7fff9b917100 > > 0004 > > GPR04: 7fffdcff27e0 0058 > > > > GPR08: 00
Re: request_module DoS
+ linuxppc list Le 07/05/2022 à 05:08, Herbert Xu a écrit : > Hi: > > There are some code paths in the kernel where you can reliably > trigger a request_module of a non-existant module. For example, > if you attempt to load a non-existent crypto algorithm, or create > a socket of a non-existent network family, it will result in a > request_module call that is guaranteed to fail. > > As user-space can do this repeatedly, it can quickly overwhelm > the concurrency limit in kmod. This in itself is expected, > however, at least on some platforms this appears to result in > a live-lock. Here is an example triggered by stress-ng on ppc64: > > [ 529.853264] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis128l, throttling... > [ 529.854329] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis128l, throttling... > [ 529.854341] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis128l, throttling... > [ 529.854419] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis128l, throttling... > [ 529.925327] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis128l, throttling... > [ 529.925328] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis128l, throttling... > [ 529.925328] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis128l, throttling... > [ 529.925356] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis128, throttling... > [ 529.925373] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis128l, throttling... > [ 529.925397] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis128l, throttling... > [ 534.863623] __request_module: 572 callbacks suppressed > [ 534.863632] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256, throttling... > [ 534.863642] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256, throttling... > [ 534.864113] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256, throttling... > [ 534.864989] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256, throttling... > [ 534.865908] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256, throttling... > [ 534.873626] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256, throttling... > [ 534.873682] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis128l-all, throttling... > [ 534.874487] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256, throttling... > [ 534.875200] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-rfc4106(gcm(aes))-all, throttling... > [ 534.88] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256, throttling... > [ 539.903506] __request_module: 604 callbacks suppressed > [ 539.903514] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256-all, throttling... > [ 539.923693] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-anubis-all, throttling... > [ 539.985508] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-rsa-all, throttling... > [ 540.005381] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256-all, throttling... > [ 540.033224] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256-all, throttling... > [ 540.035282] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256-all, throttling... > [ 540.044614] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256-all, throttling... > [ 540.045344] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256-all, throttling... > [ 540.063380] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256-all, throttling... > [ 540.073839] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50), for module crypto-aegis256-all, throttling... > [ 545.013451] __request_module: 364 callbacks suppressed > [ 545.013463] request_module: kmod_concurrent_max (0) close to 0 > (max_modprobes: 50
