[
https://issues.apache.org/jira/browse/NUTCH-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on NUTCH-3209 started by Lewis John McGibbney.
---------------------------------------------------
> Replace SHA-1 URL fingerprints in dump tools with SHA-256
> ---------------------------------------------------------
>
> Key: NUTCH-3209
> URL: https://issues.apache.org/jira/browse/NUTCH-3209
> Project: Nutch
> Issue Type: Improvement
> Components: dumpers
> Affects Versions: 1.23
> Reporter: Lewis John McGibbney
> Assignee: Lewis John McGibbney
> Priority: Minor
> Fix For: 1.24
>
>
> SonarCloud flags two uses of DigestUtils.sha1Hex (java:S4790, "Weak hashing
> algorithms should not be used") on master:
> - org.apache.nutch.util.DumpFileUtil#createFileNameFromUrl
> - org.apache.nutch.tools.CommonCrawlDataDumper (reverse-key construction)
> Both hash a URL to name dump directories / Common Crawl reverse-key values.
> This is fingerprinting for filesystem and JSON layout, not password storage
> or cryptographic integrity of crawl data.
> Replace SHA-1 with SHA-256 (DigestUtils.sha256Hex), which satisfies S4790 for
> non-sensitive fingerprinting. commons-codec is already a dependency.
> Compatibility: dump path leaves and reverse-key strings will change (40-char
> hex -> 64-char hex). Re-running a dump into an existing output tree will not
> collocate files with older SHA-1 directories. CrawlDb signatures
> (MD5Signature / db.signature.class) are unchanged.
> Out of scope: Hadoop MD5Hash usage (DumpFileUtil.getUrlMD5, MD5Signature,
> TextMD5Signature, TextProfileSignature) is not reported by this Sonar rule
> and would be a separate, more compatibility-sensitive change.
> Test: extend DumpFileUtilTest to cover createFileNameFromUrl.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
