Dan Mclaughlin created CODEC-348:
------------------------------------
Summary: Base64 STRICT decoding in 1.23.0 rejects valid input when
the codec has a line length (breaking change from 1.22.1)
Key: CODEC-348
URL: https://issues.apache.org/jira/browse/CODEC-348
Project: Commons Codec
Issue Type: Bug
Affects Versions: 1.23.0
Environment: OpenJDK 21.0.12
Reporter: Dan Mclaughlin
Component: {{org.apache.commons.codec.binary.Base64}} / {{BaseNCodec}}
Works in: 1.22.1 and earlier (STRICT since 1.15)
h2. Summary
Commit {{8b506a1c}} ("Base32 and Base64 STRICT decoding now require the
encoder's canonical alphabet, padding, and line separators, and validate
streams through EOF") changed what {{CodecPolicy.STRICT}} means for decoding. A
STRICT {{Base64}} instance constructed with a line length now accepts only
byte-for-byte what that instance would encode, including a trailing line
separator. Previously valid input that decoded correctly in 1.22.1 now throws
{{IllegalArgumentException}}.
This is a behavior change in a minor release that breaks existing callers who
use one STRICT instance for both encoding and decoding. Shibboleth's
{{Base64Support}} (used by the Shibboleth IdP, SP integrations and OpenSAML for
all SAML/XML-DSig base64) is one such caller: with 1.23.0 it can no longer
decode its own encoded output. This is tracked downstream as Shibboleth
[JSSH-84|https://shibboleth.atlassian.net/browse/JSSH-84] (priority Highest),
where it breaks SAML message, signature and certificate decoding; in our
deployment it broke SAML metadata signature validation. Because the IdP does
not control which commons-codec version deployers' builds resolve, every
Shibboleth deployment whose dependency management moves to 1.23.0 is affected.
h2. Steps to reproduce
Attached {{CodecOnlyRepro.java}} (commons-codec only):
{code:java}
Base64 strict76 = new Base64(76, new byte[] {'\n'}, false, CodecPolicy.STRICT);
String exact = strict76.encodeToString(data);
strict76.decode(exact); // exact encoder output
strict76.decode(exact.trim()); // same, trailing '\n' removed
strict76.decode(singleLineEncodingOf(data)); // same bytes, no line breaks
strict76.decode(exact.replace("\n", "\r\n")); // same bytes, CRLF separators
{code}
h2. Expected
All four decode to the original bytes, as in 1.22.1. Each input is valid base64
for the same data, with canonical alphabet, padding and trailing bits.
h2. Actual (1.23.0)
||Input||20 bytes||256 bytes||
|Exact encoder output|OK|OK|
|Encoder output, trailing {{\n}} trimmed|Missing or incomplete final line
separator|Missing or incomplete final line separator|
|Single line, no separators|Missing or incomplete final line separator|Invalid
line separator or line length|
|76-char lines, CRLF|Unexpected character or data after padding|Invalid line
separator or line length|
In 1.22.1 every case returns the correct bytes.
h2. Why this is a compatibility problem
* Through 1.22.1, {{BaseNCodec.isStrictDecoding()}} documented STRICT narrowly:
decoding raises {{IllegalArgumentException}} "if trailing bits are not part of
a valid encoding." Callers chose STRICT for that guarantee, not for
line-structure validation.
* {{CodecPolicy.STRICT}}'s Javadoc is unchanged in 1.23.0 ("Data that causes a
codec to fail should throw an exception"), so nothing at the API level signals
the broader meaning.
* The changelog entry lists it under fixed bugs, and the release notes describe
1.23.0 as a "feature and maintenance release," so downstream consumers and
automated update tools treat it as a safe minor bump.
* Requiring a trailing separator even for input shorter than one line is
stricter than RFC 2045 and RFC 4648 practice, and rejects the common pattern of
trimming encoded output.
* In CODEC-346 (resolved 2026-10-03), a request to make default decoding
stricter was declined because "changing the default would introduce another
compatibility break." Changing what STRICT itself accepts is the same kind of
break for callers who already opted into STRICT.
h2. Requests (any one would resolve it)
# Make line-structure validation opt-in (for example a builder option such as
{{setStrictLineValidation(true)}}), keeping 1.22.x STRICT semantics as the
default.
# Apply line-structure validation only when explicitly requested, and continue
to accept input without separators (or with a missing final separator) for
instances that have a line length.
# If the new behavior stays, document it as a breaking change in the release
notes and in the {{CodecPolicy.STRICT}} and {{isStrictDecoding()}} Javadoc, and
state that an instance with a line length should not be used to decode
arbitrary input.
h2. Workaround for affected users
Decode with an instance that has no line length, after removing whitespace:
{code:java}
new Base64(0, new byte[] {'\n'}, false,
CodecPolicy.STRICT).decode(input.replaceAll("\\s+", ""));
{code}
This decodes all the inputs above in both 1.22.1 and 1.23.0, and still rejects
non-canonical trailing bits.
h2. Attachments
{{commons-codec-1.23-strict-repro}} project: {{CodecOnlyRepro.java}},
{{run.sh}}, {{run-output.txt}} (output for 1.22.1 and 1.23.0). Also includes
the Shibboleth {{Base64Support}} and OpenSAML reproductions showing downstream
impact.
Related: Shibboleth [JSSH-84|https://shibboleth.atlassian.net/browse/JSSH-84]
(downstream impact), CODEC-346 (compatibility rationale), commit {{8b506a1c}}
(the change).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
