On Sun, Feb 2, 2014 at 2:43 PM, Marcus Denker <[email protected]>wrote:
> > On 02 Feb 2014, at 14:31, [email protected] wrote: > > > For diligence and curiosity leading up to the Pharo3 release, I > downloaded build image 30733 (with PharoLauncher) > > Getting an image green is a lot of little, trivial, hard work... and green > always means "Green in this context". Just headless vs. non-headless > changes a lot, for example. > > Another wonderful thing is that tests are fine when run individually, but > fail when running all tests. > > We need to think about automatic UI testing... > It will be awesome to have automatic UI tests. Currently, I can go as far as invoking the UI but then to complete the fields in another process/thread, I haven't really spent enough time. I think that's already discussed on the mailing list but some time I should have a look. Automated tests before a release/roll-out will be a dream coming true ;-). Usman > > Loading an external package changes *everything*. We do not have a a > release criterium that all tests are green after loading external packages. > > > and immediately ran all tests, which reported 7 failures and 1 error > (see attached snapshot). Now presumably these did not occur in the CI > build since otherwise it would not have been available to PharoLauncher, so > this raises a few questions/ideas: > > > > 1. Why the discrepancy / how did these slip through ? Perhaps some > difference in local environment / difference in VM? Can anyone else > reproduce this for 30733? I've logged an new case [1] for this where I > plan to either link to existing cases or add a subcase for each failure. > Please add your results there, particularly if you see a known issue. > > > > Here are my failures... > > BlockClosureTest>>#testTrace > > Not failing for me > > > JobTest>>#testCurrent > Not failing for me > > > MetacelloRepositorySqueakCommonTestCase>>#testAsRepositorySpecFor > > MetacelloRepositorySqueakCommonTestCase>>#testDirectoryRepository > > MetacelloRepositorySqueakCommonTestCase>>#testFileTreeRepository > Not failing fo rme > > > ReleaseTest>>#testObsoleteClasses > > Not failing for me > > SimmulateMouseSpecification>>#testSimulateRightClick > yes, that is failing > > > and errors... > > OpenToolTest>>#testOpenBrowseOnInstalledTraitMethod > > > yes. > > The last two are examples of UI tests that are not run on the Build server > in a way that they could fail. > > > 3. Assuming Pharo3 will go through a Release Candidate phase, > > Normally the idea is that we will just release. In the past we used an > elaborate process, but the reality is that nobody looks at release > candidates. > You can tell 5 times that the release candidate will be released unchanged > and people should check: they will not. They will download > the release, though, and then complain that the "obvious" thing X is not > fixed that that the people who managed the release process did > everything wrong. Because "release" means bug free. By magic. > > Marcus >
