Re: [PATCH v7 1/1] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()

2020-07-08 Thread Tom Rini
On Tue, Jul 07, 2020 at 10:57:26PM +0200, Heinrich Schuchardt wrote: > From: Heiko Stuebner > > When calculating rrtmp/rr rsa_gen_key_prop() tries to make > (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and > (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[] > with rlen being

[PATCH v7 1/1] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()

2020-07-07 Thread Heinrich Schuchardt
From: Heiko Stuebner When calculating rrtmp/rr rsa_gen_key_prop() tries to make (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[] with rlen being num_bits * 2 On a 4096bit key this comes down to to 257 uint32_t elements in rr and