rzo1 opened a new issue, #2105:
URL: https://github.com/apache/stormcrawler/issues/2105
## What happens
`MetadataRecordFormat.format()` writes one line per metadata value as `key:
value` followed by CR LF. Neither the key nor the value is checked for CR or
LF. A value that contains CR LF therefore becomes two or more field lines in
the `application/warc-fields` payload, and the extra lines look exactly like
fields the crawler wrote. Framing stays valid, because `Content-Length` is
computed from the finished payload, so no record is split.
## Where
`external/warc/src/main/java/org/apache/stormcrawler/warc/MetadataRecordFormat.java:84`,
config key `warc.metadata.keys`.
```java
payload.append(key).append(": ").append(value).append(CRLF);
```
The same lack of escaping applies to the resource record Content-Type at
`external/warc/src/main/java/org/apache/stormcrawler/warc/WARCRecordFormat.java:443-448`,
where the server-supplied content type is appended verbatim into the WARC
header block:
```java
String ct = metadata.getFirstValue(HttpHeaders.CONTENT_TYPE,
this.protocolMDprefix);
...
buffer.append("Content-Type: ").append(ct).append(CRLF);
```
## Why it matters
Values that reach this sink often come from parsed content. `FeedParserBolt`
sets `feed.description` from the feed item without trimming
(core/src/main/java/org/apache/stormcrawler/bolt/FeedParserBolt.java:223), and
`parse.*` values from the XPath, LDJson and Tika filters keep their newlines.
If an operator lists such a key in `warc.metadata.keys`, a crawled page or feed
can add field lines to its own metadata record, for example a fabricated
`hopsFromSeed` or `via`. The effect is limited: nothing in this repository
reads warc-fields back, metadata records are opt-in, and the harm lands on a
downstream tool that ranks or filters captures by those fields. The
Content-Type case in `WARCRecordFormat` is worse in principle, since it would
inject into the WARC header block itself, and is only out of reach because the
sibling formats happen not to feed it such values today.
## Reproduction
Save as
`external/warc/src/test/java/org/apache/stormcrawler/warc/MetadataRecordFormatCRLFTest.java`.
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to you under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.stormcrawler.warc;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.List;
import org.apache.storm.tuple.Tuple;
import org.apache.stormcrawler.Metadata;
import org.junit.jupiter.api.Test;
import org.netpreserve.jwarc.MessageHeaders;
import org.netpreserve.jwarc.WarcMetadata;
import org.netpreserve.jwarc.WarcReader;
import org.netpreserve.jwarc.WarcRecord;
/**
* A metadata value that contains CR LF becomes extra field lines in the
application/warc-fields
* payload of the metadata record.
*/
class MetadataRecordFormatCRLFTest {
private byte[] record(String value) {
Metadata metadata = new Metadata();
metadata.addValue("feed.description", value);
Tuple tuple = mock(Tuple.class);
when(tuple.getStringByField("url")).thenReturn("https://www.example.org/";);
when(tuple.getValueByField("metadata")).thenReturn(metadata);
MetadataRecordFormat format = new
MetadataRecordFormat(List.of("feed.description"));
return format.format(tuple);
}
@Test
void valueWithCRLFDoesNotCreateExtraFields() {
byte[] warcBytes = record("some text\r\nhopsFromSeed: 1");
String warcString = new String(warcBytes, StandardCharsets.UTF_8);
System.out.println(warcString);
assertFalse(
warcString.contains("\r\nhopsFromSeed: 1\r\n"),
"a metadata value must not introduce a new warc-fields
line");
try (WarcReader reader = new WarcReader(new
ByteArrayInputStream(warcBytes))) {
for (WarcRecord rec : reader) {
MessageHeaders fields = ((WarcMetadata) rec).fields();
System.out.println("parsed fields: " + fields.map());
assertFalse(
fields.contains("hopsFromSeed", "1"),
"parsers must not see a field the crawler did not
write");
}
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
```
Run it:
```
mvn -pl external/warc test -Dtest=MetadataRecordFormatCRLFTest
```
It fails on main.
```
WARC/1.0
WARC-Type: metadata
WARC-Record-ID: <urn:uuid:636926c9-5d8e-4fbe-8a5e-2c9a58f05ed6>
Content-Length: 46
WARC-Date: 2026-08-18T18:06:12Z
WARC-Target-URI: https://www.example.org/
Content-Type: application/warc-fields
WARC-Block-Digest: sha1:AMUWFMBWN4RTNHSWLQITPH74IXYYDDSD
feed.description: some text
hopsFromSeed: 1
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[ERROR]
MetadataRecordFormatCRLFTest.valueWithCRLFDoesNotCreateExtraFields:57 a
metadata value must not introduce a new warc-fields line ==> expected: <false>
but was: <true>
```
## Suggested fix
In `MetadataRecordFormat.format`, replace CR and LF in both the key and the
value before appending, or fold long values using the continuation form allowed
by the WARC spec, and drop a key that is not a valid field name. Apply the same
treatment to the Content-Type append in `WARCRecordFormat.format`. Existing
archives are unaffected; new records will show a sanitised value where they
previously showed a broken one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
