Lewis John McGibbney created NUTCH-3195:
-------------------------------------------
Summary: Make Apache Yetus a trustworthy precommit signal for Nutch
Key: NUTCH-3195
URL: https://issues.apache.org/jira/browse/NUTCH-3195
Project: Nutch
Issue Type: Task
Components: ci/cd
Reporter: Lewis John McGibbney
Assignee: Lewis John McGibbney
Fix For: 1.24
NUTCH-3163 put Apache Yetus into GitHub Actions so pull requests get automated
precommit feedback (spelling, whitespace, shell/Dockerfile lint, secret
scanning,
markdown, and related checks). On pushes to master, Yetus also scans the
*entire*
tree. That full-tree run currently fails hard: thousands of findings, many of
them
historical noise or intentional test content, mixed with a smaller set of real
problems worth fixing.
As long as master stays red for the wrong reasons, Yetus cannot be a reliable
signal. Reviewers learn to ignore it; contributors get drowned in unrelated
output; we cannot honestly make the check required. This parent tracks the work
to separate noise from signal, fix what is actually wrong, pay down style debt
in reviewable slices, and only then treat Yetus as a gate we trust.
*What went wrong on master?*
A recent full-tree run (Yetus 0.15.1) voted -1 with large counts in codespell,
blanks/tabs, markdownlint, @author, shellcheck, detsecrets, hadolint, shelldocs,
and jshint. Much of that is expected for a long-lived codebase the first time a
whole-tree linter is applied (for example, language-identifier test corpora are
full of non-English words that codespell treats as typos; auth test fixtures
trip detect-secrets). A meaningful remainder is real: misspellings in product
code, fragile patterns in bin/crawl and bin/nutch, Dockerfile structure, missing
shelldocs annotations, and one jshint nit.
PR patch mode already focuses on changed lines. The initiative is about making
*master* full-tree results honest and eventually green—without forcing a single
unreviewable mega-PR.
*How we will attack it?*
Work is split into linked child issues so each PR has one job and can be
reviewed thoroughly. Early children are allowed to leave master Yetus red;
getting the diff right matters more than painting the build green too soon.
Yetus should remain informational until the cleanup is done; only then should
we consider making it a required PR check.
* Child A — Teach Yetus what “not a bug” looks like (excludes, codespell
ignores, reviewed detsecrets hash baseline). No behavior changes.
* Child B — Fix defects the gate is right about (product typos, shellcheck
*errors*, Dockerfile structure, shelldocs, jshint). Behavior-preserving where
scripts change; manual smoke checklist for bin tools.
* Child C — Pay down historical style debt (@author removal, blanks/tabs,
markdownlint) in small PRs; hybrid markdown rule tuning so we do not rewrite
every README for list-marker fashion.
* Child D — Finish shellcheck on bin/crawl and bin/nutch (warnings/notes)
after the error-level fixes in B; can proceed in parallel with C.
Related earlier work: NUTCH-3163, NUTCH-3192.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
