On Sun, Apr 15, 2018 at 01:49:29PM +0200, Richard Levitte wrote: > In message <ac42d34d-4ad9-3646-19a9-e9e3521d6...@openssl.org> on Sun, 15 Apr > 2018 13:27:15 +0200, Andy Polyakov <ap...@openssl.org> said: > > appro> To summarize, failing tests in 110 should be revisited to see if they > appro> are actually representative, before one can consider as drastic > measures > appro> as #5945. > > At this point, I agree. Viktor's look at several applications and > Kurt's report of successful did convince me that I might by crying > wolf a bit much. I've started looking a bit deeper at the failing > tests, for exactly the reasons you mention.
I'll echo the thanks going around for looking into the tests more closely, and note that I'm happy to see that you are stepping back a bit from the initial report -- it seems good be concerned, but I don't think we have reason to panic quite yet. Having said that, I'll step back a bit and try to speak to the "philosophy" question from the start of the thread. I think it is good philosophy to worry about this sort of ABI stability, and agree with Tim that it's good release engineering practice to have automated tests for it (to the extent that we can). What I'd like us to think more about, though, is our release engineering in general. Don't get me wrong; we've made huge progess since the 1.0.2 era already, adding much more comprehensive test suite (including external tests) and insisting that new features come with tests and documentation. But I still wonder if we could/should be more aware of the release engineering consequences of our more daily changes, at least when master is intended to be on a "stable ABI" branch. In the following examples, I'm not trying to point fingers or say that we made mistakes; so far as I know these were all consistent with our current policy and procedures. That said, I have to wonder if doing things like reverting old changes that we don't have CLAs for, or introducing additional churn into the RNG, are the best idea in this stage of the release cycle. True, we're only in feature freeze and not some final release freeze, but there is risk/reward analysis to be done, and I just don't know how much we've been thinking about that sort of tradeoff. (I believe there are other examples in a similar vein, but don't remember them off the top of my head.) I guess this can be summarized as "I don't have a good understanding for what scale/size of change we as a group are comfortable making during various stages of ABI compatibility and release planning" -- I don't want to dictate policy, just start a discussion (or at least gain a better understanding of existing consensus). I will note, though, that if we want to be very contemplative about the release engineering and ABI stability consequences of changes landing on an "ABI-stable" branch, that may end up leading to a stronger argument for having master never be the "ABI-stable" branch, so that it's always open for development work and we need not do exhaustive analysis for *all* commits during some months+-long period of time. (There would then be a separate stable branch, of course, and we'd do the extra "thinking" when merging to it.) -Ben _______________________________________________ openssl-project mailing list firstname.lastname@example.org https://mta.openssl.org/mailman/listinfo/openssl-project