pjfanning opened a new issue, #1410:
URL: https://github.com/apache/poi/issues/1410
## Problem
XSSF decodes `_xHHHH_` escape sequences on read but never encodes them on
write, so some literal strings do not survive a round trip.
- **Read:** `XSSFRichTextString.utfDecode` (pattern `_x([0-9A-Fa-f]{4})_`)
is applied to shared and inline strings in the usermodel and event APIs. It
turns `_x0041_` into `A`.
- **Write:** neither `XSSFCell.setCellValue(String)` / `XSSFRichTextString`
nor SXSSF's `SheetDataWriter` escapes a literal `_xHHHH_`. Following ECMA-376
(`ST_Xstring`), Excel stores a literal `_x0041_` as `_x005F_x0041_`, escaping
the leading underscore. POI stores it unchanged.
Result:
```java
cell.setCellValue("_x0041_ col"); // XSSF or SXSSF
// save, re-open with POI or Excel
cell.getStringCellValue(); // "A col"
```
The data is silently corrupted. It mostly hits product codes, identifiers
and user-entered text that happen to match `_xHHHH_`.
Downstream impact: Apache Fesod (incubating) ships a default
`EscapeHexCellWriteHandler` to work around this. It is currently being extended
path by path: to in-memory `XSSFCell` writes (apache/fesod#1223), and to header
cells and template-fill cells (apache/fesod#1226). Each write path that
bypasses the handler is a new corruption bug, which is why fixing this at the
source would help.
## Proposed fix
Add the inverse of `utfDecode`: escape `_` as `_x005F_` when it starts a
`_xHHHH_` match. Then apply it where strings are written to the sheet XML or
shared strings table (XSSF `XSSFRichTextString`/`SharedStringsTable`, and SXSSF
`SheetDataWriter` inline strings).
## Behaviour-change risk for existing users
Turning on encoding unconditionally would break some existing usage:
- **Deliberate escapes.** Some users write `_x000D_`, `_x0009_` etc. on
purpose so that Excel and POI decode them into control characters that XML 1.0
cannot carry. Some characters (`\u0000`–`\u001F` apart from tab, LF and CR) are
replaced with `?` by SXSSF or rejected by XMLBeans, so this has been the
working trick for years. Encoding would turn those into literal text.
- **Users who already pre-escape.** Fesod and others escape strings before
passing them to POI. They would get `_x005F_x005F_...` (double escaping) unless
they remove their workaround at the same time.
- **Byte-level output changes.** Anything comparing the generated XML/XLSX
with golden files, or non-POI consumers that do not decode `_xHHHH_`, would see
different stored text.
- **Rich text runs.** Encoding has to be applied per run, and run
offsets/lengths must stay based on the decoded string.
## Suggestions for making it opt-in
1. **Public helpers (no behaviour change):** make `utfDecode` public and add
a public `utfEncode(String)`, for example on `XSSFRichTextString` or a small
`XSSFStringUtil`. Libraries can then do this correctly without copying the
regex. This can ship in 5.x with no risk.
2. **Workbook-level flag:** add something like
`XSSFWorkbook#setEscapeLiteralUnicodeSequences(boolean)` and the same on
`SXSSFWorkbook` (passed to `SheetDataWriter`), default `false`. When it is set,
every string write path escapes, including cell values, rich text and shared
strings, so callers don't have to cover each path themselves. A per-cell API
would recreate the coverage-gap problem described above.
3. **Default in the next major:** consider making the flag default to `true`
in POI 6, with a release note about deliberate `_x000D_`-style usage (which can
then opt out).
I would suggest doing 1 and 2 together: the helper for people who need
fine-grained control, and the flag for people who just want round trips to be
lossless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
