On Wed, 2 Jun 2021 at 04:20, Daniel Gruno <[email protected]> wrote: > > On 01/06/2021 23.26, sebb wrote: > > I just discovered that the generator test harness only processed the > > first generator type within each file. Oops. > > > > Having fixed this, lots of test failures are now revealed. > > It looks like all versions of PonyMail have some failures. > > > > Most of them are related to the medium generator. > > > > The dkim failures are presumably due to recent changes in the Foal > > generator, and can be fixed by updating the test data. > > > > The other failures are more concerning, as it suggests there have been > > incompatible changes since the tests were generated. > > > > Thanks for looking into this. > I dug a little further: > > - there are 12 parsing errors in foal that are not present in pony, all > of which are due to format=flowed not converting at all in older pony > (returns a body of None). 10 return None, one fails to decode > quoted-printable, and one seems to not be format=flowed at all. > > - when accounting for the above, there are 118 generator issues with > medium and 12 with cluster and none with cluster (the 12 reported with
The previous line does not make sense to me > cluster are the exact same as with the parser and the format=flowed > issue). All of these issues were where no Date: header was present > (sigh, old gitbox issue mainly...). As medium inserts a fake date when > that happens, this is to be expected. The solution to that is to provide a fixed date, so that other aspects of the generation are still tested. There is already a way to do this, by defining the MOCK_GMTIME environment variable. This can be defined as an argument in the relevant yaml file -- indeed several already do. Rather than disabling such tests, the variable needs to be defined as needed. > There are about 1000 dkim failures, I suspect - as you do - that this is > changes to the dkim generator that has caused this, and this is probably > fine so long as we hit stability before we do a release of foal. Agreed.
