Lewis John McGibbney created NUTCH-3209:
-------------------------------------------
Summary: Replace SHA-1 URL fingerprints in dump tools with SHA-256
Key: NUTCH-3209
URL: https://issues.apache.org/jira/browse/NUTCH-3209
Project: Nutch
Issue Type: Improvement
Components: dumpers
Affects Versions: 1.23
Reporter: Lewis John McGibbney
Assignee: Lewis John McGibbney
Fix For: 1.24
SonarCloud flags two uses of DigestUtils.sha1Hex (java:S4790, "Weak hashing
algorithms should not be used") on master:
- org.apache.nutch.util.DumpFileUtil#createFileNameFromUrl
- org.apache.nutch.tools.CommonCrawlDataDumper (reverse-key construction)
Both hash a URL to name dump directories / Common Crawl reverse-key values.
This is fingerprinting for filesystem and JSON layout, not password storage or
cryptographic integrity of crawl data.
Replace SHA-1 with SHA-256 (DigestUtils.sha256Hex), which satisfies S4790 for
non-sensitive fingerprinting. commons-codec is already a dependency.
Compatibility: dump path leaves and reverse-key strings will change (40-char
hex -> 64-char hex). Re-running a dump into an existing output tree will not
collocate files with older SHA-1 directories. CrawlDb signatures (MD5Signature
/ db.signature.class) are unchanged.
Out of scope: Hadoop MD5Hash usage (DumpFileUtil.getUrlMD5, MD5Signature,
TextMD5Signature, TextProfileSignature) is not reported by this Sonar rule and
would be a separate, more compatibility-sensitive change.
Test: extend DumpFileUtilTest to cover createFileNameFromUrl.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
