Fix two memory issues in rsa_gen_key_prop(): - Correct n[]/rr[] sizing to prevent out-of-bounds writes. - Reject all-zero modulus to prevent out-of-bounds reads.
Changes in v2: - Patch 1: use DIV_ROUND_UP() instead of open-coded rounding for the n[]/rr[] allocation sizes. - Patch 2: new, addressing the num_bits == 0 case. UT: Sandbox: => ut lib lib_rsa_keyprop_oob Test: lib_rsa_keyprop_oob: rsa.c rsa_gen_key_prop: n_sz=258 leading_zeros=1 stripped_bytes=257 num_bits=2056 rsa_gen_key_prop: n_words=66 rr_words=130 rrtmp_words=130 n=000000001c135890 rr=000000001c1359a0 rrtmp=000000001c135bb0 (FIXED sizing) rsa_gen_key_prop: br_i32_decode(n) done, n[0] (bit_length)=2056 n0inv_input n[1]=0x968b6a4d rsa_gen_key_prop: rlen=4112 rr zero-init loop wrote indices 1..129 rsa_gen_key_prop: br_i32_reduce(rr) done, rr[0] (bit_length)=2056 Tests run: 1, 0 ms, average: 0 ms, failures: 0 => ut lib lib_rsa_keyprop_zero_modulus Test: lib_rsa_keyprop_zero_modulus: rsa.c rsa_gen_key_prop: n_sz=3 leading_zeros=3 num_bits=0, rejecting (-EINVAL) Tests run: 1, 0 ms, average: 0 ms, failures: 0 => Prasad Kummari (2): rsa: fix undersized modulus buffers in rsa_gen_key_prop() rsa: reject zero-bit modulus in rsa_gen_key_prop() lib/rsa/rsa-keyprop.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) -- 2.43.0
