abhinav-phi opened a new pull request, #2110:
URL: https://github.com/apache/stormcrawler/pull/2110
This PR addresses #2106 and fixes it.
## Problem
`WARCRecordFormat.getDigestSha1()` computed a SHA-1 digest over the bytes
and returned it as `sha1:<base32>`. That fixed value was used for the
`WARC-Payload-Digest` and `WARC-Block-Digest` headers of response, resource,
request and metadata records. There was no configuration key for the algorithm,
so an operator who wants SHA-256 digests in their archives — which WARC 1.1
allows — could not have them without rewriting the records afterwards.
SHA-1 digests are the convention across the WARC ecosystem and identify
content for deduplication rather than authenticate it, so this is not urgent —
but it was still a fixed choice in a place where the digested bytes come from
the network. Operators whose own policy rules out SHA-1, or whose downstream
tooling wants SHA-256, had no option.
## Solution
Add a configuration key `warc.digest.algorithm` and thread the configured
algorithm through all record formats:
```
warc.digest.algorithm: sha256
```
* **Accepted values**: `sha1` and `sha256`. The value is matched
case-insensitively, an optional hyphen is ignored (`sha256`, `SHA-256`,
`SHA256` are all accepted) and surrounding whitespace is trimmed.
* **`sha1` remains the default**: CDX indexes and revisit tooling downstream
assume `sha1:` base32, and changing the default would break them. Deciding on a
different default can be done separately from adding the option.
* **Fail fast on invalid values**: an unsupported value (e.g. `md5`) raises
an `IllegalArgumentException` with a descriptive message when the bolt is
prepared, instead of silently producing digests with a different algorithm than
the one configured.
### Changes
**`external/warc/src/main/java/org/apache/stormcrawler/warc/WARCRecordFormat.java`**
* New configuration key constants: `DIGEST_ALGORITHM_PARAM`
(`warc.digest.algorithm`) and the supported values `DIGEST_ALGORITHM_SHA1` /
`DIGEST_ALGORITHM_SHA256`.
* New constructor `WARCRecordFormat(String protocolMDprefix, String
digestAlgorithm)`; the existing single-argument constructor is unchanged in
behaviour and keeps defaulting to SHA-1.
* New instance methods `getDigest(byte[])` and `getDigest(byte[], byte[])`
emit the matching `sha1:` / `sha256:` prefix. The digest used for records
without content (`digestNoContent`) is now derived from the configured
algorithm instead of a static SHA-1 constant.
* The static helpers `getDigestSha1(byte[])` and `getDigestSha1(byte[],
byte[])` are kept for compatibility with code outside the module that calls
them, now marked `@Deprecated` in favour of the instance methods.
**`external/warc/src/main/java/org/apache/stormcrawler/warc/WARCRequestRecordFormat.java`**
and **`MetadataRecordFormat.java`**
* New constructors accepting the digest algorithm; both use the configurable
`getDigest` instance methods so request and metadata records use the same
algorithm as response and resource records.
**`external/warc/src/main/java/org/apache/stormcrawler/warc/WARCHdfsBolt.java`**
* Reads `warc.digest.algorithm` from the topology configuration (default
`sha1`) and passes it to every record format it instantiates, so a single
setting governs the whole WARC output.
### Documentation
* `external/warc/README.md`: documents the new key, the accepted values and
the compatibility considerations around SHA-1.
* `docs/src/main/asciidoc/configuration.adoc`: adds `warc.digest.algorithm`
to the WARC section of the configuration reference.
### Note on #2034
As requested in the issue, I checked PR #2034 ("WARC writer: WARC-Protocol
header to follow WARC field proposals"): it is about the `WARC-Protocol` /
cipher-suite headers and does not touch digest computation, so this is not a
duplicate. It modifies the same file, so whichever merges second may need a
trivial rebase.
## Testing
* New `WARCDigestAlgorithmTest` (9 tests):
* SHA-1 is the default, including for a `null` algorithm value;
* SHA-256 digests for `getDigest(byte[])` and `getDigest(byte[], byte[])`,
including empty content;
* acceptance of value variants (`SHA256`, `SHA-256`, ` sha256 `, `SHA-1`);
* `IllegalArgumentException` for unsupported values (`md5`, `sha512`) on
all three record format classes;
* full response, resource, request and metadata records verifying that
`WARC-Payload-Digest` / `WARC-Block-Digest` carry the `sha256:` prefix and the
correct digest value, computed independently of the code under test.
* `WARCHdfsBoltTest.testDigestAlgorithmConfig` prepares the bolt with
`warc.digest.algorithm: sha256` and verifies that the written records
(warcinfo, request, response) use `sha256:` digests.
* `mvn -pl external/warc test`: 22 tests run, 0 failures, 0 errors;
`checkstyle:check` reports 0 violations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
