On Sat, 17 Jan 2026 at 04:25, Xiyue Deng <[email protected]> wrote: > > > I have been working on a new Perl-based unit tests and I think it's now > in a usable form. It runs dh-make-elpa in a temp directory and compares > the contents of its output with the expected ones.
> [1] https://salsa.debian.org/emacsen-team/dh-make-elpa/-/merge_requests/6 (i cant review the perl parts in detail though) I've used this approach in a couple of other packages, so thought i'd offer some thoughts based on my experiences The pros include - the test is what the underlying code is actually producing, so you get confidence that the test is actually testing what a user sees - you can add tests without needing to understand the code at all, and the underlying code doesnt need to have any support for "testing" - it's esp good when you want to know if the output has changed -- i think this is a good fit for something like dh-make-elpa where if the output changes it may be unintentional, and checking "by hand" is tedious The cons include - someone has to keep the test data up-to-date - i think a lot of people in debian commit things before bothering to test - more git commits fixing test failures - rebasing branches and MRs that change output can be a HUGE pain - most test failures will be "the test suite is out of date" rather than "there is a bug in the package", this can de-sensitize you from investigating, as once the "expected" is updated, any issues are hidden. - you are exposed to changes in other packages (*)see below - (this may not apply for this package?) you are exposed to differences in the build environment - it may pass in a local sbuild but fail on salsa, or in the debian build system, or debusine, or in stable vs unstable - generally anything that makes the output non-reproducible, like timestamps are a pain. - everyone making changes to the package will need to understand how it works - in every case ive tried, yu end up needing a way to "filter" the 'actual' output before comparison to avoid all the unreproducible bits! (i think the benefits are more than the costs for a "stable", slow-moving package with an active maintainer!) *i did notice that you are hard-coding the contents of a "new git repos" including all the hook templates (eg t/data/github_team/repo/git/hooks/commit-msg.sample )- this means when a new version of git makes a change (even changing the comments in a template, the test will fail (and maybe block a new git from migrating?). I dont know if you can instead do a fresh "git init" and copy the .git into the "expected" directory before doing the comparison, rather than holding it in the repo?
