[
https://issues.apache.org/jira/browse/CODEC-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130891#comment-18130891
]
Gary D. Gregory commented on CODEC-348:
---------------------------------------
Thank you for the detailed reproducer. We have chosen {*}option 3{*}: retain
the decoding behavior introduced in 1.23.0 and explicitly document the
behavioral incompatibility and migration guidance. We don't want to flip-flop
behavior from release to release.
See [commit
0e856487|https://github.com/apache/commons-codec/commit/0e8564870d206fac24ea9df1b0c607cf6570c7b9]:
Clarify Base64 decoding policies and compatibility changes. The commit expands
the Base64 class Javadoc with policy explanations and examples, adds an
explicit compatibility warning to the 1.23.0 release notes, and records the
documentation improvements under 1.23.1. It does not change decoding behavior
or binary compatibility.
Base32/Base64 STRICT now validates the exact representation produced by the
configured encoder, including alphabet, padding, wrapping, and the final line
separator for nonempty chunked output. Unchanged encoder output still
round-trips, but trimming the final separator or changing the wrapping can
cause rejection. A strict instance configured for chunked output is therefore
unsuitable for externally supplied input whose wrapping may differ.
This exact-layout requirement is a Commons Codec contract, not a general Base64
requirement. [RFC 2045 section
6.8|https://datatracker.ietf.org/doc/html/rfc2045#section-6.8] instructs
decoders to ignore line breaks, while [XML Schema’s base64Binary
definition|https://www.w3.org/TR/xmlschema-2/#base64Binary] permits XML
whitespace and prohibits enforcing MIME’s line-length restriction.
For XML Schema Base64 values, use a separate unchunked STRICT decoder after
removing only XML whitespace from the extracted value:
{code:java}
Base64 decoder = Base64.builder()
.setLineLength(0)
.setDecodingPolicy(CodecPolicy.STRICT)
.get();
byte[] decoded = decoder.decode(input.replaceAll("[ \\t\\r\\n]", ""));
{code}
(Feel free to provide a PR if you want to improve the API)
This accepts the wrapping variations in the reproducer while retaining
trailing-bit validation. In 1.23.0 it also retains the stricter alphabet,
padding, and trailing-data checks. Apply this normalization at the Base64
decoding stage, not by rewriting signed XML. XML Signature’s Base64 transform
separately specifies RFC 2045 decoding.
Switching to LENIENT is another option when broader permissive decoding is
intended, but it does not preserve the trailing-bit validation that existing
STRICT callers selected.
We acknowledge that changing an explicitly selected policy broke existing
callers. Retaining the released behavior now avoids another change to STRICT’s
acceptance rules and establishes a documented contract going forward.
> Base64 STRICT decoding in 1.23.0 rejects valid input when the codec has a
> line length (breaking change from 1.22.1)
> -------------------------------------------------------------------------------------------------------------------
>
> Key: CODEC-348
> URL: https://issues.apache.org/jira/browse/CODEC-348
> Project: Commons Codec
> Issue Type: Bug
> Affects Versions: 1.23.0
> Environment: OpenJDK 21.0.12
> Reporter: Dan Mclaughlin
> Priority: Major
> Attachments: commons-codec-1.23-strict-repro.zip
>
>
> Component: {{org.apache.commons.codec.binary.Base64}} / {{BaseNCodec}}
> Works in: 1.22.1 and earlier (STRICT since 1.15)
> h2. Summary
> Commit {{8b506a1c}} ("Base32 and Base64 STRICT decoding now require the
> encoder's canonical alphabet, padding, and line separators, and validate
> streams through EOF") changed what {{CodecPolicy.STRICT}} means for decoding.
> A STRICT {{Base64}} instance constructed with a line length now accepts only
> byte-for-byte what that instance would encode, including a trailing line
> separator. Previously valid input that decoded correctly in 1.22.1 now throws
> {{IllegalArgumentException}}.
> This is a behavior change in a minor release that breaks existing callers who
> use one STRICT instance for both encoding and decoding. Shibboleth's
> {{Base64Support}} (used by the Shibboleth IdP, SP integrations and OpenSAML
> for all SAML/XML-DSig base64) is one such caller: with 1.23.0 it can no
> longer decode its own encoded output. This is tracked downstream as
> Shibboleth [JSSH-84|https://shibboleth.atlassian.net/browse/JSSH-84]
> (priority Highest), where it breaks SAML message, signature and certificate
> decoding; in our deployment it broke SAML metadata signature validation.
> Because the IdP does not control which commons-codec version deployers'
> builds resolve, every Shibboleth deployment whose dependency management moves
> to 1.23.0 is affected.
> h2. Steps to reproduce
> Attached {{CodecOnlyRepro.java}} (commons-codec only):
> {code:java}
> Base64 strict76 = new Base64(76, new byte[] {'\n'}, false,
> CodecPolicy.STRICT);
> String exact = strict76.encodeToString(data);
> strict76.decode(exact); // exact encoder output
> strict76.decode(exact.trim()); // same, trailing '\n'
> removed
> strict76.decode(singleLineEncodingOf(data)); // same bytes, no line breaks
> strict76.decode(exact.replace("\n", "\r\n")); // same bytes, CRLF
> separators
> {code}
> h2. Expected
> All four decode to the original bytes, as in 1.22.1. Each input is valid
> base64 for the same data, with canonical alphabet, padding and trailing bits.
> h2. Actual (1.23.0)
> ||Input||20 bytes||256 bytes||
> |Exact encoder output|OK|OK|
> |Encoder output, trailing {{\n}} trimmed|Missing or incomplete final line
> separator|Missing or incomplete final line separator|
> |Single line, no separators|Missing or incomplete final line
> separator|Invalid line separator or line length|
> |76-char lines, CRLF|Unexpected character or data after padding|Invalid line
> separator or line length|
> In 1.22.1 every case returns the correct bytes.
> h2. Why this is a compatibility problem
> * Through 1.22.1, {{BaseNCodec.isStrictDecoding()}} documented STRICT
> narrowly: decoding raises {{IllegalArgumentException}} "if trailing bits are
> not part of a valid encoding." Callers chose STRICT for that guarantee, not
> for line-structure validation.
> * {{CodecPolicy.STRICT}}'s Javadoc is unchanged in 1.23.0 ("Data that causes
> a codec to fail should throw an exception"), so nothing at the API level
> signals the broader meaning.
> * The changelog entry lists it under fixed bugs, and the release notes
> describe 1.23.0 as a "feature and maintenance release," so downstream
> consumers and automated update tools treat it as a safe minor bump.
> * Requiring a trailing separator even for input shorter than one line is
> stricter than RFC 2045 and RFC 4648 practice, and rejects the common pattern
> of trimming encoded output.
> * In CODEC-346 (resolved 2026-10-03), a request to make default decoding
> stricter was declined because "changing the default would introduce another
> compatibility break." Changing what STRICT itself accepts is the same kind of
> break for callers who already opted into STRICT.
> h2. Requests (any one would resolve it)
> # Make line-structure validation opt-in (for example a builder option such as
> {{setStrictLineValidation(true)}}), keeping 1.22.x STRICT semantics as the
> default.
> # Apply line-structure validation only when explicitly requested, and
> continue to accept input without separators (or with a missing final
> separator) for instances that have a line length.
> # If the new behavior stays, document it as a breaking change in the release
> notes and in the {{CodecPolicy.STRICT}} and {{isStrictDecoding()}} Javadoc,
> and state that an instance with a line length should not be used to decode
> arbitrary input.
> h2. Workaround for affected users
> Decode with an instance that has no line length, after removing whitespace:
> {code:java}
> new Base64(0, new byte[] {'\n'}, false,
> CodecPolicy.STRICT).decode(input.replaceAll("\\s+", ""));
> {code}
> This decodes all the inputs above in both 1.22.1 and 1.23.0, and still
> rejects non-canonical trailing bits.
> h2. Attachments
> {{commons-codec-1.23-strict-repro}} project: {{CodecOnlyRepro.java}},
> {{run.sh}}, {{run-output.txt}} (output for 1.22.1 and 1.23.0). Also includes
> the Shibboleth {{Base64Support}} and OpenSAML reproductions showing
> downstream impact.
> Related: Shibboleth [JSSH-84|https://shibboleth.atlassian.net/browse/JSSH-84]
> (downstream impact), CODEC-346 (compatibility rationale), commit {{8b506a1c}}
> (the change).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
