Hi Pádraig,
What do you think of the proposed patch?
Arguably the setting of errno belongs in Gnulib. But I still think it is
wrong to have DIGEST_STREAM return an error, but continue the program if
errno is not set. That seems like it is prone to printing uninitialized
memory. ENOMEM seems like a reasonable fallback.
Collin
>From 0de361eca4f4444630b6c5f00dbae626fca7d1e4 Mon Sep 17 00:00:00 2001
Message-ID: <0de361eca4f4444630b6c5f00dbae626fca7d1e4.1757183936.git.collin.fu...@gmail.com>
From: Collin Funk <collin.fu...@gmail.com>
Date: Sat, 6 Sep 2025 11:35:01 -0700
Subject: [PATCH] cksum: don't leak memory using -a sha3 with OpenSSL
* gnulib: Update to latest commit.
* src/digest.c (digest_file): If getting the digest of the file fails
but errno is not set, assume that it is caused by a failed allocation.
---
gnulib | 2 +-
src/digest.c | 8 +++++++-
2 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/gnulib b/gnulib
index 06e7da510..745cf7d95 160000
--- a/gnulib
+++ b/gnulib
@@ -1 +1 @@
-Subproject commit 06e7da510bd055f524c1baff0890c861bef2b6a4
+Subproject commit 745cf7d95dbe64bcea7d4eed9c81369e1898bb70
diff --git a/src/digest.c b/src/digest.c
index c9f6106b9..cad82aa26 100644
--- a/src/digest.c
+++ b/src/digest.c
@@ -1094,7 +1094,13 @@ digest_file (char const *filename, int *binary, unsigned char *bin_result,
#else
err = DIGEST_STREAM (fp, bin_result);
#endif
- err = err ? errno : 0;
+ /* If an error occurred but errno is not set, we can assume that we
+ ran out of memory. I.e., OpenSSL functions that do not set errno. */
+ if (err)
+ {
+ if (! errno)
+ err = ENOMEM;
+ }
if (is_stdin)
clearerr (fp);
else if (fclose (fp) != 0 && !err)
--
2.51.0
