The GitHub Actions job "Java CI with Maven" on stormcrawler.git/fix/warc-crlf-escaping-2105 has succeeded. Run started by GitHub user abhinav-phi (triggered by dpol1).
Head commit for run: d263b91f932c7477ca739e59db7a5da75ae1fe7c / abhinav-phi <[email protected]> Escape CR and LF in WARC metadata records and resource Content-Type MetadataRecordFormat.format() wrote one line per metadata value into the application/warc-fields payload without checking for CR or LF. A value containing CR LF (e.g. feed.description set by FeedParserBolt, or values of parse.* filters such as the XPath, LDJson and Tika filters) therefore became additional field lines that look exactly like fields written by the crawler - for example a fabricated hopsFromSeed or via. Framing stayed valid because Content-Length is computed from the finished payload, so WARC readers had no way to detect the injected fields. - replace CR and LF by spaces in metadata values written into the warc-fields payload, and drop metadata keys that are not valid WARC field names (printable ASCII without colon, RFC 5322 section 2.2) - sanitise the server-supplied Content-Type used for resource records in WARCRecordFormat.format(), which was appended verbatim into the WARC header block - log MetadataRecordFormat messages under MetadataRecordFormat instead of WARCRequestRecordFormat Fixes #2105 Report URL: https://github.com/apache/stormcrawler/actions/runs/33264094117 With regards, GitHub Actions via GitBox
