On Dec 6, 2014 9:13 PM, "Justin Mclean" <jus...@classsoftware.com> wrote: > > Hi, > > > You are doing it again :-) > > IMO It was a fairly accurate representation of the state of affairs. You do realise when I write "PMC" I'm including myself? For instance look at the amount of feedback from the the "test" RC of TourDeFlex (RC0) and then the issues found in RC1 and RC2. Other releases have has similar issues, ie that important issues or regressions are only found a few RCs in. Our CI doesn't catch all issues. The RC0 step was added to try and get people to try stuff out before the first proper RC and cut down on the number of RC cycles - and IMO it's had some positive effect but not a huge amount.
'Fairly' accurate is not good enough. The point is, by painting with such broad strokes, you are discounting the efforts of those who take time to contribute to the project. I request you to avoid these kind of blanket statements. > > > What are the alternatives to CI and Mustella tests? > > There no real good alternative to CI as manual testing take up a lot of time. With the installer and several other projects don't have tests that the CI can run *, so someone would need to step forward and write tests for those project. (Perhaps that not seen as important enough?) This has happened with TLF (which was donated with no tests) which is great. On that subject (and while it is manual testing) TourDeFlex is a good way of testing a new SDK, particularly in a few areas where we have limited or no tests eg OSMF or newer Apache SDK features. > > Having a set of FlexUnit tests rather than Mustella tests would probably get more people involved in helping out with tests, and would make testing easier, but that would be a (very) large undertaking. Having a set of tests that could be run more easily and that didn't take 8 hours would also help, a few people has suggested some solutions (ie run tests in parallel) but also hard to implement. > Exactly. We must be happy that we have at least the Mustella tests to keep the regression bugs away. Anything more than that is definitely welcome. But there is no point in blaming the CI system or the Mustella tests for our difficult release process. Thanks, Om > Thanks, > Justin > > * Ant for AIR used in the installer does have tests but the installer itself doesn't
