garydgregory commented on PR #1809:
URL: https://github.com/apache/commons-lang/pull/1809#issuecomment-6224967368
Thank you for the proposal. The change is consistent with
`Character.isWhitespace('\u00A0')` returning `false`, but we will retain the
established NBSP handling and will not merge this PR.
The confusion is understandable: the Javadoc referenced XPath's
`normalize-space()` and default Java regex `\s`, without explaining all the
implementation's differences. XPath preserves NBSP, and default Java regex `\s`
does not match it. However, neither defines the current Commons Lang behavior.
The relevant history is:
- [LANG-640](https://issues.apache.org/jira/browse/LANG-640), implemented in
[823f2603a](https://github.com/apache/commons-lang/commit/823f2603a),
introduced `normalizeSpace()` with a regex implementation. The Javadoc
described it as similar to XPath, with additional Java whitespace and trimming
behavior.
- [LANG-910](https://issues.apache.org/jira/browse/LANG-910), implemented in
[f9b486a17](https://github.com/apache/commons-lang/commit/f9b486a17) and
released in **Commons Lang 3.2**, explicitly added NBSP support to the regex
implementation.
- [LANG-1020](https://issues.apache.org/jira/browse/LANG-1020), implemented
in [bc8e23808](https://github.com/apache/commons-lang/commit/bc8e23808) through
[PR #27](https://github.com/apache/commons-lang/pull/27), replaced regex
processing with manual normalization. This introduced the current
`Character.isWhitespace()` check and explicit conversion of each U+00A0 to
U+0020.
- [LANG-1226](https://issues.apache.org/jira/browse/LANG-1226), implemented
in [7e85d1cf5](https://github.com/apache/commons-lang/commit/7e85d1cf5) through
[PR #150](https://github.com/apache/commons-lang/pull/150), restored final
trimming.
Consequently, removing the NBSP conversion in [PR
#1809](https://github.com/apache/commons-lang/pull/1809) changes longstanding
observable behavior, including at string boundaries:
| Input | Established behavior | Proposed behavior |
|---|---|---|
| `"a\u00A0b"` | `"a b"` | `"a\u00A0b"` |
| `"\u00A0"` | `""` | `"\u00A0"` |
| `"\u00A0a\u00A0"` | `"a"` | `"\u00A0a\u00A0"` |
| `"a\u00A0\u00A0b"` | `"a b"` | `"a\u00A0\u00A0b"` |
The boundary behavior results from converting NBSP to an ordinary space
before the final `trim()`.
The JDK itself uses several definitions: `Character.isWhitespace()` excludes
NBSP, `Character.isSpaceChar()` includes it, and regex `\s` includes it when
`UNICODE_CHARACTER_CLASS` or `(?U)` is enabled. `String.trim()` preserves NBSP
unless it has first been converted to an ordinary space. See the [Character
documentation](https://docs.oracle.com/en/java/javase/25/docs/api/java.base/java/lang/Character.html),
[String
documentation](https://docs.oracle.com/en/java/javase/25/docs/api/java.base/java/lang/String.html#trim()),
and [Pattern
documentation](https://docs.oracle.com/en/java/javase/25/docs/api/java.base/java/util/regex/Pattern.html).
Our decision is to preserve compatibility and correct the documentation.
XPath's
[`normalize-space()`](https://www.w3.org/TR/1999/REC-xpath-19991116/#function-normalize-space)
was an inspiration; this method does not reproduce its behavior. The
documentation should explicitly describe Java whitespace normalization, U+00A0
conversion, final trimming, and the fact that adjacent converted NBSPs are not
collapsed. Regression tests should protect those behaviors.
This is a compatibility decision, not a claim that the previous Javadoc
accurately described the implementation. Thank you for bringing that
discrepancy to our attention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
