Hello Niels,
Thank you for merging it and the suggestions.
Niels Möller writes:
> Thanks to the doc update, I now noticed the possibility of failure from
> the encryption functions. Failure is propagated from _oaep_encode_mgf1,
> which does
>
> assert (key_size >= 2 * hash->digest_size - 2);
>
> if (message_length > key_size - 2 * hash->digest_size - 2)
> return 0;
>
> Why is the first an assert (it could be triggered by using an unusually
> small RSA key with a large hash function, say rsa_oaep_sha512_encrypt
> with an old 512-bit RSA key, which from the docs isn't an obviously
> invalid usage), and the other a return value to indicate failure?
I think I split these conditions that way, as the first condition is
more about the algorithm setup, i.e., the user should choose RSA key
size and hash digest size appropriately, before using any of the
rsa_oaep_*_encrypt functions; otherwise treat it as a programming error.
That said, I agree that it would be more user friendly to combine them
and treat it as a regular error, as we do in pss_encode_mgf1.
> One alternative could be to instead check
>
> if (message_length > key_size
> || message_length + 2 + 2*hash->digest_size > key_size)
> return 0;
>
> (with two tests, to not trigger overflow in case message_length is close
> to the maximum size_t value; maybe that is more defensive than necessary
> since message_length large enough to trigger overflow can't be the size of
> properly allocated memory area).
I made this change in the attached patch.
> The opposite alternative would to be to have a documented way for the
> application to get the maximum message size, and have an assertion
> failure for both cases. That would have the advantage that no return
> value is needed, simplifying the api (at least very locally).
It is tempting, though for now I just documented the size restriction.
> Another doc detail: The docs for the decrypt functions don't say
> explicitly that *length is both an input and output argument. The text
> for the older function rsa_decrypt could be reused (or possibly
> improved).
Updated the documentation.
Regards,
--
Daiki Ueno
>From 04a7eb300a555617fd6f0503fdd48e8cf389e790 Mon Sep 17 00:00:00 2001
From: Daiki Ueno
Date: Fri, 16 Feb 2024 14:47:18 +0900
Subject: [PATCH] Clarify message length limitation in RSA-OAEP
Signed-off-by: Daiki Ueno
---
nettle.texinfo | 13 ++---
oaep.c | 5 ++---
2 files changed, 12 insertions(+), 6 deletions(-)
diff --git a/nettle.texinfo b/nettle.texinfo
index 25bce6fb..95e89971 100644
--- a/nettle.texinfo
+++ b/nettle.texinfo
@@ -5202,8 +5202,12 @@ is done with one of the following functions:
@deftypefunx int rsa_oaep_sha384_encrypt (const struct rsa_public_key *@var{key}, void *@var{random_ctx}, nettle_random_func *@var{random}, size_t @var{label_length}, const uint8_t *@var{label}, size_t @var{length}, const uint8_t *@var{message}, uint8_t *@var{ciphertext})
@deftypefunx int rsa_oaep_sha512_encrypt (const struct rsa_public_key *@var{key}, void *@var{random_ctx}, nettle_random_func *@var{random}, size_t @var{label_length}, const uint8_t *@var{label}, size_t @var{length}, const uint8_t *@var{message}, uint8_t *@var{ciphertext})
Returns 1 on success, 0 on failure. The label is optional and if
-omitted, @var{label_length} and @var{label} can be set to 0 and
-@code{NULL} respectively.
+omitted, @var{label_length} and @var{label} is set to 0 and
+@code{NULL} respectively. The maximum size of @var{message} can be
+calculated with @code{k - 2 * hLen - 2}, where @code{k} is the key
+size in octets obtained with @var{key}->size, and @code{hLen} is the
+output size of the underlying hash function (e.g.,
+@code{SHA256_DIGEST_SIZE} for @code{rsa_oaep_sha256_encrypt}).
@end deftypefun
Decrypting a cipher text message using RSA with the OAEP padding
@@ -5213,7 +5217,10 @@ scheme is done with one of the following functions:
@deftypefunx int rsa_oaep_sha384_decrypt (const struct rsa_public_key *@var{pub}, const struct rsa_private_key *@var{key}, void *@var{random_ctx}, nettle_random_func *@var{random}, size_t @var{label_length}, const uint8_t *@var{label}, size_t *@var{length}, uint8_t *@var{message}, const uint8_t *@var{ciphertext})
@deftypefunx int rsa_oaep_sha512_decrypt (const struct rsa_public_key *@var{pub}, const struct rsa_private_key *@var{key}, void *@var{random_ctx}, nettle_random_func *@var{random}, size_t @var{label_length}, const uint8_t *@var{label}, size_t *@var{length}, uint8_t *@var{message}, const uint8_t *@var{ciphertext})
Returns 1 on success, 0 on failure. These function utilize randomized
-RSA blinding similarly to @code{rsa_decrypt_tr}.
+RSA blinding similarly to @code{rsa_decrypt_tr}. The message buffer
+pointed to by @var{cleartext} must be of size *@var{length}. After
+decryption, *@var{length} will be updated with the size of the
+message.
@end deftypefun
diff --git a/oaep.c b/oaep.c
index c504fbb9..5ebf8ba0 100644
--- a/oaep.c
