[
https://issues.apache.org/jira/browse/NUTCH-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lewis John McGibbney updated NUTCH-3195:
----------------------------------------
Description:
NUTCH-3163 put Apache Yetus into GitHub Actions so pull requests get automated
precommit feedback (spelling, whitespace, shell/Dockerfile lint, secret
scanning, markdown, and related checks). On pushes to master, Yetus also scans
the *entire* tree. That full-tree run currently fails hard: thousands of
findings, many of them historical noise or intentional test content, mixed with
a smaller set of real problems worth fixing.
As long as master stays red for the wrong reasons, Yetus cannot be a reliable
signal. Reviewers learn to ignore it; contributors get drowned in unrelated
output; we cannot honestly make the check required. This parent tracks the work
to separate noise from signal, fix what is actually wrong, pay down style debt
in reviewable slices, and only then treat Yetus as a gate we trust.
*What went wrong on master?*
A recent full-tree run (Yetus 0.15.1) voted -1 with large counts in codespell,
blanks/tabs, markdownlint, @author, shellcheck, detsecrets, hadolint,
shelldocs, and jshint. Much of that is expected for a long-lived codebase the
first time a
whole-tree linter is applied (for example, language-identifier test corpora are
full of non-English words that codespell treats as typos; auth test fixtures
trip detect-secrets). A meaningful remainder is real: misspellings in product
code, fragile patterns in bin/crawl and bin/nutch, Dockerfile structure,
missing shelldocs annotations, and one jshint nit.
PR patch mode already focuses on changed lines. The initiative is about making
master full-tree results honest and eventually green—without forcing a single
unreviewable mega-PR.
*How we will attack it?*
Work is split into linked child issues so each PR has one job and can be
reviewed thoroughly. Early children are allowed to leave master Yetus red;
getting the diff right matters more than painting the build green too soon.
Yetus should remain informational until the cleanup is done; only then should
we consider making it a required PR check.
* Child A — Teach Yetus what “not a bug” looks like (excludes, codespell
ignores, reviewed detsecrets hash baseline). No behavior changes.
* Child B — Fix defects the gate is right about (product typos, shellcheck
{*}errors{*}, Dockerfile structure, shelldocs, jshint). Behavior-preserving
where scripts change; manual smoke checklist for bin tools.
* Child C — Pay down historical style debt (@author removal, blanks/tabs,
markdownlint) in small PRs; hybrid markdown rule tuning so we do not rewrite
every README for list-marker fashion.
* Child D — Finish shellcheck on bin/crawl and bin/nutch (warnings/notes)
after the error-level fixes in B; can proceed in parallel with C.
Related earlier work: NUTCH-3163, NUTCH-3192.
was:
NUTCH-3163 put Apache Yetus into GitHub Actions so pull requests get automated
precommit feedback (spelling, whitespace, shell/Dockerfile lint, secret
scanning,
markdown, and related checks). On pushes to master, Yetus also scans the
*entire*
tree. That full-tree run currently fails hard: thousands of findings, many of
them
historical noise or intentional test content, mixed with a smaller set of real
problems worth fixing.
As long as master stays red for the wrong reasons, Yetus cannot be a reliable
signal. Reviewers learn to ignore it; contributors get drowned in unrelated
output; we cannot honestly make the check required. This parent tracks the work
to separate noise from signal, fix what is actually wrong, pay down style debt
in reviewable slices, and only then treat Yetus as a gate we trust.
*What went wrong on master?*
A recent full-tree run (Yetus 0.15.1) voted -1 with large counts in codespell,
blanks/tabs, markdownlint, @author, shellcheck, detsecrets, hadolint, shelldocs,
and jshint. Much of that is expected for a long-lived codebase the first time a
whole-tree linter is applied (for example, language-identifier test corpora are
full of non-English words that codespell treats as typos; auth test fixtures
trip detect-secrets). A meaningful remainder is real: misspellings in product
code, fragile patterns in bin/crawl and bin/nutch, Dockerfile structure, missing
shelldocs annotations, and one jshint nit.
PR patch mode already focuses on changed lines. The initiative is about making
*master* full-tree results honest and eventually green—without forcing a single
unreviewable mega-PR.
*How we will attack it?*
Work is split into linked child issues so each PR has one job and can be
reviewed thoroughly. Early children are allowed to leave master Yetus red;
getting the diff right matters more than painting the build green too soon.
Yetus should remain informational until the cleanup is done; only then should
we consider making it a required PR check.
* Child A — Teach Yetus what “not a bug” looks like (excludes, codespell
ignores, reviewed detsecrets hash baseline). No behavior changes.
* Child B — Fix defects the gate is right about (product typos, shellcheck
*errors*, Dockerfile structure, shelldocs, jshint). Behavior-preserving where
scripts change; manual smoke checklist for bin tools.
* Child C — Pay down historical style debt (@author removal, blanks/tabs,
markdownlint) in small PRs; hybrid markdown rule tuning so we do not rewrite
every README for list-marker fashion.
* Child D — Finish shellcheck on bin/crawl and bin/nutch (warnings/notes)
after the error-level fixes in B; can proceed in parallel with C.
Related earlier work: NUTCH-3163, NUTCH-3192.
> Make Apache Yetus a trustworthy precommit signal for Nutch
> ----------------------------------------------------------
>
> Key: NUTCH-3195
> URL: https://issues.apache.org/jira/browse/NUTCH-3195
> Project: Nutch
> Issue Type: Task
> Components: ci/cd
> Reporter: Lewis John McGibbney
> Assignee: Lewis John McGibbney
> Priority: Major
> Fix For: 1.24
>
>
> NUTCH-3163 put Apache Yetus into GitHub Actions so pull requests get
> automated precommit feedback (spelling, whitespace, shell/Dockerfile lint,
> secret scanning, markdown, and related checks). On pushes to master, Yetus
> also scans the *entire* tree. That full-tree run currently fails hard:
> thousands of findings, many of them historical noise or intentional test
> content, mixed with a smaller set of real problems worth fixing.
> As long as master stays red for the wrong reasons, Yetus cannot be a reliable
> signal. Reviewers learn to ignore it; contributors get drowned in unrelated
> output; we cannot honestly make the check required. This parent tracks the
> work to separate noise from signal, fix what is actually wrong, pay down
> style debt in reviewable slices, and only then treat Yetus as a gate we trust.
> *What went wrong on master?*
> A recent full-tree run (Yetus 0.15.1) voted -1 with large counts in
> codespell, blanks/tabs, markdownlint, @author, shellcheck, detsecrets,
> hadolint, shelldocs, and jshint. Much of that is expected for a long-lived
> codebase the first time a
> whole-tree linter is applied (for example, language-identifier test corpora
> are full of non-English words that codespell treats as typos; auth test
> fixtures
> trip detect-secrets). A meaningful remainder is real: misspellings in product
> code, fragile patterns in bin/crawl and bin/nutch, Dockerfile structure,
> missing shelldocs annotations, and one jshint nit.
> PR patch mode already focuses on changed lines. The initiative is about
> making master full-tree results honest and eventually green—without forcing a
> single unreviewable mega-PR.
> *How we will attack it?*
> Work is split into linked child issues so each PR has one job and can be
> reviewed thoroughly. Early children are allowed to leave master Yetus red;
> getting the diff right matters more than painting the build green too soon.
> Yetus should remain informational until the cleanup is done; only then should
> we consider making it a required PR check.
> * Child A — Teach Yetus what “not a bug” looks like (excludes, codespell
> ignores, reviewed detsecrets hash baseline). No behavior changes.
> * Child B — Fix defects the gate is right about (product typos, shellcheck
> {*}errors{*}, Dockerfile structure, shelldocs, jshint). Behavior-preserving
> where scripts change; manual smoke checklist for bin tools.
> * Child C — Pay down historical style debt (@author removal, blanks/tabs,
> markdownlint) in small PRs; hybrid markdown rule tuning so we do not rewrite
> every README for list-marker fashion.
> * Child D — Finish shellcheck on bin/crawl and bin/nutch (warnings/notes)
> after the error-level fixes in B; can proceed in parallel with C.
> Related earlier work: NUTCH-3163, NUTCH-3192.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
