[
https://issues.apache.org/jira/browse/TIKA-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116159#comment-18116159
]
ASF GitHub Bot commented on TIKA-4897:
--------------------------------------
Copilot commented on code in PR #3184:
URL: https://github.com/apache/tika/pull/3184#discussion_r4029621402
##########
tika-core/src/main/java/org/apache/tika/metadata/Metadata.java:
##########
@@ -445,7 +446,18 @@ public void add(final String name, final String value) {
* reserved key by name rather than by its {@link Property}.
*/
public void addTrusted(final String name, final String value) {
- writeLimiter.add(name, value, metadata);
+ writeLimiter.add(name, wellFormed(name, value), metadata);
Review Comment:
This only normalizes values at the write call, but `Metadata.getValues`
exposes the backing `String[]` (Metadata.java:405-415), so a caller can mutate
a value after this check and leave `\uD800` in the object. `MetadataSerializer`
and the plain `PipesWorker` metadata map then write that raw value and can
still fail the JSON/Smile serialization; sanitize at the serialization
boundaries or make the returned arrays defensive.
##########
tika-core/src/main/java/org/apache/tika/metadata/Metadata.java:
##########
@@ -445,7 +446,18 @@ public void add(final String name, final String value) {
* reserved key by name rather than by its {@link Property}.
*/
public void addTrusted(final String name, final String value) {
- writeLimiter.add(name, value, metadata);
+ writeLimiter.add(name, wellFormed(name, value), metadata);
+ }
+
+ /**
+ * Every string value is stored well-formed: a lone surrogate becomes
U+FFFD at the write,
+ * so no reader (a Smile encoder, for one) meets one. {@code tk:content}
is exempt: it came
+ * through {@link org.apache.tika.sax.SafeContentHandler}, which already
did this, and it
+ * is the one value large enough for the scan to cost anything.
+ */
+ private static String wellFormed(String name, String value) {
+ return TikaCoreProperties.TIKA_CONTENT.getName().equals(name) ? value
+ : StringUtils.wellFormed(value);
Review Comment:
This normalizes only metadata values, not field names.
`Metadata.add(KeyPrefix, ...)` accepts document-derived suffixes verbatim, so a
suffix containing an unpaired surrogate still becomes a malformed name that
`MetadataSerializer` writes directly and Jackson can reject. Normalize the name
before passing it to the limiter as well (while keeping the `tk:content`
exemption scoped to the value).
> Handle unpaired surrogates more robustly
> ----------------------------------------
>
> Key: TIKA-4897
> URL: https://issues.apache.org/jira/browse/TIKA-4897
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Minor
>
> An unpaired surrogate causes less than ideal behavior during serialization.
> We should handle this more robustly.
>
> After a bit of agentic digging, this is a smile-only issue, so it only hits
> in IPC between tika-pipes client and server. This does not affect our
> standard json serdes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
