I took a quick look and attempted to compile your Crypto library in Visual Studio.
I got nowhere: Probably for good reasons; Microsoft C doesn't allow dynamic array size like this: bigint_word_t d_b[a->length_W + b->length_W]; This is about as far as I'm going to get on this, there is just far too much work to get it to compile to run on the Desktop. Wouldn't it be funny if thiat line was the reason for your crash. -- Nigel On Fri, 28 May 2021 at 18:24, BERTRAND Joël <joel.bertr...@systella.fr> wrote: > Hello, > > I beg your pardon as my question is not related to gcc itself. > > I have added to my firmware a subset of avr-crypto-lib (AES and > RSA). > If AES128/192/256 run as expected, RSA causes a memory corruption. > Of course, I don't generate RSA key on AVR. I create this key (2048) on > a workstation with openssl and I write this key in EEPROM. > > My code reads public key from EEPROM and converts inernal > pointers. It > sends this key on serial console after it has converted EEPROM data into > rsa_publickey_t struct: > > RSA public key: 0xde1 <- address of exponent > BIGINT: 0xde1 > L=3 > I=0 > 01 00 01 > > BIGINT: 0xde6 <- address of modulus > L=256 > I=0 > FD 90 D5 83 32 44 55 07 5E AE F1 EF D5 64 4C EB > EC F1 51 C9 39 BC 78 AE 07 4F 34 95 FA 9D 04 64 > FF 7D 2F 5A CD 43 33 BE FD 69 29 0B 2B 3C 92 34 > 0D B2 E0 A0 E6 46 B6 B0 B0 58 95 2C D8 00 9F 8A > D3 C5 07 83 91 83 2B 04 71 AA 01 15 D2 C4 CF 7C > 9A 17 46 20 7B 17 EE 4E 79 5B 9D C0 35 6E B7 89 > F8 39 7C 06 95 DA 15 5A AB 2C 0F FD 4F 56 01 EE > EB 47 90 DA 7A 7F BE A3 05 20 AF 5B B4 4D 92 BA > BB B2 8D EB 1B 0C 18 5E 1D 78 3D AC 8F EB 0E 91 > 2A 89 B3 40 EF D8 13 A7 DA 3B 9C C4 F7 19 A5 BE > E5 9B C7 D2 A1 D0 16 9E A6 8D B6 A2 A1 B0 BC CD > 9B CE E0 6A 63 9E B0 C4 81 1A D4 3A EA 0E B7 50 > CC 49 8A F2 1C 60 A2 01 75 27 DE 20 FA 91 B4 FA > 41 0D 0F 66 21 E8 33 22 28 E9 F8 9F DA A9 D8 58 > BB 10 98 A3 6E CF B6 41 15 72 1B 10 0A D4 E1 D4 > 9C FA 8E 52 75 D2 2C 9B C9 2C CA 7F 3C C7 4A 9B > > Value written on serial console is expected value (I have the same > value in eeprom.hex file). > > Now, I want to use rsa_enc() function. This function waits for a > bigint_t. No problem, I create a bigint_t with my message: > > BIGINT: 0x3f62 <- address on stack > L=16 > I=0 > E6 4C E7 31 3D 1C 48 D0 5F 5C 2B 73 8A 5D 64 3E > > But when rsa_enc() is called, firmware crashes. I have searched a > buffer overflow somewhere without success. Is there a tool like electric > fence to find memory corruption on avr ? > > My firmware is downlodable at > https://hilbert.systella.fr/public/firmware-antivol.tar.gz > > Best regards, > > JB > >