With an ASAN build we can trigger the issue like:
$ src/factor 22486472939460651857
src/factor.c:888:36: runtime error:
index 675 out of bounds for type 'int_least16_t [675]'
* src/factor.c (factor_using_division): Guard the final unrolled block
before reading a subsequent prime.
* tests/factor/factor.pl: Add a test case.
* NEWS: Mention the bug fix.
Problem caught by Coverity analysis.
---
NEWS | 3 +++
src/factor.c | 2 ++
tests/factor/factor.pl | 2 ++
3 files changed, 7 insertions(+)
diff --git a/NEWS b/NEWS
index 39967eab1..d57a6441c 100644
--- a/NEWS
+++ b/NEWS
@@ -16,6 +16,9 @@ GNU coreutils NEWS -*-
outline -*-
if N is negative. Previously it behaved as if N were zero.
[bug introduced in coreutils-9.4]
+ 'factor' avoids a buffer over-read (CWE-126) for certain values.
+ [bug introduced in coreutils-9.8]
+
'head' and 'tail' now quote names in file headers when needed.
[This bug was present in "the beginning".]
diff --git a/src/factor.c b/src/factor.c
index e9c373207..615cf6ada 100644
--- a/src/factor.c
+++ b/src/factor.c
@@ -885,6 +885,8 @@ factor_using_division (struct factors *factors, mp_limb_t
t1, mp_limb_t t0,
t0 = divblock (factors, t0, pd, i, 6);
t0 = divblock (factors, t0, pd, i, 7);
+ if (PRIMES_PTAB_ENTRIES <= i + 8)
+ break;
int_least32_t p = primes_ptab[i + 8];
if (p * p > t0)
break;
diff --git a/tests/factor/factor.pl b/tests/factor/factor.pl
index 2b8e54086..524c57baa 100755
--- a/tests/factor/factor.pl
+++ b/tests/factor/factor.pl
@@ -73,6 +73,8 @@ my @Tests =
['bug-2012-c', '6635692801', {OUT => '57601 115201'}],
['bug-2012-d', '17709149503', {OUT => '94099 188197'}],
['bug-2012-e', '17754345703', {OUT => '94219 188437'}],
+ ['bug-prime-table-end', '22486472939460651857',
+ {OUT => '4993 4503599627370449'}],
# Infinite loop bugs in v8.20 to 8.26 inclusive
['bug-2016-a', '158909489063877810457',
{OUT => '3401347 3861211 12099721'}],
--
2.55.0
