On 03/06/2021 00.21, sebb wrote:
On Wed, 2 Jun 2021 at 18:32, Daniel Gruno <[email protected]> wrote:
I've made it so that the tests will pass with 56 specific format=flowed
issues in mind.
Wrong, so wrong.
Depends on what the goal of the test is. We seem to disagree on what
they are there for, and that's fair. I'm not hung up on pony and foal
being exactly equal in their results, I use the tests to guard against
unintended changes. When the change is very much intended, I expect the
tests to allow for that.
We have the option of:
- Be strict and let foal fail forever
- Be flexible and have foal and pony differ in these specific cases
- Create a separate set of tests for both
Or, fix foal so it generates the same results for the same generators.
Fixing here would be "fixing" by breaking improvements - I am not
willing to do that. I'd rather we yank out the legacy, medium and
cluster generators and then get sean's DKIM patches merged in and settle
on DKIM and full as the new standard generators.
I'll look at those patches as time allows today.
This won't solve the parsing tests however, where the two versions
produce different results due to how they parse format=flowed and how
some character encodings are handled. Here we either need to keep the
alternate flag, or split it into tests for pony and tests for foal, and
have runall figure that out somehow.
I realise that foal has deprecated the old generators, however it
still offers them as options.
There is no point in doing so if the results are different, which is
why the tests were set up.
Fixing the tests does not make the code correct, it just hides the problems.
We have to make allowances for v 0.10, because incompatible changes
were made previously.
It's not possible to change the past, so we have to live with it.
However, for new code, it is possible to get it right -- or at least
not make it more wrong.
I am leaning towards option 2, which is how it is right now. Option 1
means we can't use the tests for foal, and option 3 would mean we risk
not noticing when the two diverge.
For the places where the tests diverge (and need alternates), I've put a
top comment in each YAML specifying who/why. I think this is currently
the best compromise.
There is no need to compromise.
Foal needs to generate the same output for the same input for all the
historic generators that it supports.
Anything else just makes an already bad situation worse.
