On 02/06/2021 12.35, sebb wrote:
On Wed, 2 Jun 2021 at 04:20, Daniel Gruno <[email protected]> wrote:
On 01/06/2021 23.26, sebb wrote:
I just discovered that the generator test harness only processed the
first generator type within each file. Oops.
Having fixed this, lots of test failures are now revealed.
It looks like all versions of PonyMail have some failures.
Most of them are related to the medium generator.
The dkim failures are presumably due to recent changes in the Foal
generator, and can be fixed by updating the test data.
The other failures are more concerning, as it suggests there have been
incompatible changes since the tests were generated.
Thanks for looking into this.
I dug a little further:
- there are 12 parsing errors in foal that are not present in pony, all
of which are due to format=flowed not converting at all in older pony
(returns a body of None). 10 return None, one fails to decode
quoted-printable, and one seems to not be format=flowed at all.
- when accounting for the above, there are 118 generator issues with
medium and 12 with cluster and none with cluster (the 12 reported with
The previous line does not make sense to me
cluster are the exact same as with the parser and the format=flowed
issue). All of these issues were where no Date: header was present
(sigh, old gitbox issue mainly...). As medium inserts a fake date when
that happens, this is to be expected.
The solution to that is to provide a fixed date, so that other aspects
of the generation are still tested.
There is already a way to do this, by defining the MOCK_GMTIME
environment variable.
This can be defined as an argument in the relevant yaml file -- indeed
several already do.
Rather than disabling such tests, the variable needs to be defined as needed.
Right, that's a useful tool for many situations.
What I'm looking at right now is the gens-maven_duplicates_2017.yaml
which attempts to compare all four mbox copies as if they were the same.
This fails with the medium generator across both pony and foal, as
medium will default to using the archived-at header when the date header
is missing. This is a very wrong thing to do for many reasons, but is
nonetheless what it does. MOCK_GTMTIME=0 does not solve it in this case,
as the default behavior is hardcoded into the generator and uses the
mbox data. I'll work around this for now with a MOCK_AAT env variable
for these tests, which resets the archived-at header to 0
There are about 1000 dkim failures, I suspect - as you do - that this is
changes to the dkim generator that has caused this, and this is probably
fine so long as we hit stability before we do a release of foal.
Agreed.
