rzo1 opened a new issue, #2106:
URL: https://github.com/apache/stormcrawler/issues/2106
## What happens
`WARCRecordFormat.getDigestSha1()` computes a SHA-1 over the bytes and
returns it as `sha1:<base32>`. That value is used for `WARC-Payload-Digest` and
`WARC-Block-Digest` in response, request, resource and metadata records. There
is no configuration key for the algorithm, so an operator who wants SHA-256
digests in their archives, which WARC 1.1 allows, cannot have them.
## Where
`external/warc/src/main/java/org/apache/stormcrawler/warc/WARCRecordFormat.java:105-112`,
used at `:402`, `:404`, `:409`, and reused by `MetadataRecordFormat.java:106`
and `WARCRequestRecordFormat.java:85`.
```java
public static String getDigestSha1(byte[] bytes) {
return "sha1:" + base32.encodeAsString(DigestUtils.sha1(bytes));
}
```
## Why it matters
`sha1:` base32 is the convention across the WARC ecosystem, and the field
identifies content for deduplication rather than authenticating it, so this is
not urgent. It is still a fixed choice in a place where the bytes being
digested come from the network, and operators whose own policy rules out SHA-1,
or whose downstream tooling wants SHA-256, have no option other than rewriting
the records. Making the algorithm a setting costs little and removes the
argument.
## Reproduction
Save as
`external/warc/src/test/java/org/apache/stormcrawler/warc/WARCDigestAlgorithmTest.java`.
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to you under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.stormcrawler.warc;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import java.nio.charset.StandardCharsets;
import java.util.List;
import org.apache.storm.tuple.Tuple;
import org.apache.stormcrawler.Metadata;
import org.junit.jupiter.api.Test;
/**
* Documents the current behaviour: the digest algorithm is hard coded to
SHA-1 in
* WARCRecordFormat.getDigestSha1 and there is no configuration key to pick
another one. Once the
* algorithm is configurable, this test should be replaced by one that sets
the key and asserts the
* prefix changes.
*/
class WARCDigestAlgorithmTest {
@Test
void digestPrefixIsAlwaysSha1() {
String digest = WARCRecordFormat.getDigestSha1("some crawled
bytes".getBytes(StandardCharsets.UTF_8));
System.out.println("digest: " + digest);
assertTrue(digest.startsWith("sha1:"), "digest algorithm is fixed to
sha1");
}
@Test
void metadataRecordBlockDigestIsAlwaysSha1() {
Metadata metadata = new Metadata();
metadata.addValue("source", "a source");
Tuple tuple = mock(Tuple.class);
when(tuple.getStringByField("url")).thenReturn("https://www.example.org/";);
when(tuple.getValueByField("metadata")).thenReturn(metadata);
MetadataRecordFormat format = new
MetadataRecordFormat(List.of("source"));
String warcString = new String(format.format(tuple),
StandardCharsets.UTF_8);
System.out.println(warcString);
assertTrue(
warcString.contains("WARC-Block-Digest: sha1:"),
"no configuration can change the algorithm used here");
}
}
```
Run it:
```
mvn -pl external/warc test -Dtest=WARCDigestAlgorithmTest
```
It passes on main and pins the current behaviour, since there is no setting
to assert against yet. Replace it after the fix with one that sets the key and
checks the prefix changes.
```
digest: sha1:B7VTSBZ5WRVHOA4OQPESX36RFAJF2URB
...
Content-Type: application/warc-fields
WARC-Block-Digest: sha1:2O53BWA6NLGDTA4665VJAM6IJBA4C5DK
source: a source
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
```
## Suggested fix
Add a configuration key, for example `warc.digest.algorithm`, accepting
`sha1` and `sha256`, and thread it through `WARCRecordFormat` so
`MetadataRecordFormat` and `WARCRequestRecordFormat` pick it up too. Replace
the static `getDigestSha1` helpers with an instance method that emits the
matching `sha1:` or `sha256:` prefix, keeping the static ones for compatibility
if anything outside the module calls them. Keep `sha1` as the default: CDX
indexes and revisit tooling downstream assume `sha1:` base32, and changing the
default would break them. Decide the default separately from adding the option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
