11/05/14 13:50, intrigeri wrote: > Hi, > > intrigeri wrote (10 May 2014 15:22:54 GMT) : >> Test suite run in progress, I'll inspect results and >> report back later (likely tomorrow). > > Here we go! > > I think my first failure may be a blocker, as it's a regression. > For the others, I don't know. > > So, the failures I have seen are: > > 1. I got this failure once out of two tries in the "Writing files to > a read/write-enabled persistent partition" scenario, once out of > two tries in the "Writing files to a read/write-enabled persistent > partition with the old Tails USB installation" scenario: > > And I completely shutdown Tails # > features/step_definitions/common_steps.rb:398 > Then only the expected files should persist on USB drive "current" # > features/step_definitions/usb.rb:433 > FindFailed: can not find TailsEmergencyShutdownHalt.png on the screen. > > Looking at the video, it seems as if TailsEmergencyShutdownHalt.png > was never clicked. There's a notification left on screen at this > time, so I wonder if it may be interfering somehow (but perhaps I'm > only trying to find justifications for my doubt wrt. dropping the > notification handling code ;) -- in any case, that's a regression, > since I have never seen this before.
Note that that scenario does the And all notifications have disappeared step. The notification killing I removed in commit 13d0ae9 doesn't affect usb_install.feature in any way. I think this just shows that the "all notifications have disappeared" step is problematic (and ). That step will immediately complete once it sees no notifications, so it really depends on being run after the last notification is has been shown... I really don't know how to improve that step. My suggestion would be that we don't use the emergency shitdown, and simply sends a `halt`, which we did before, and which was much more reliable. I know you want us to the same ways we except our users to use, and I agree, but well... can't we just make a dedicated test for the emergency shutdown instead? > 2. Scenario: Booting Tails from a USB drive upgraded from DVD with > persistence enabled # features/usb_install.feature:182 > [...] > And the boot device has safe access rights > # features/step_definitions/usb.rb:326 > And the expected persistent files are present in the filesystem > # features/step_definitions/usb.rb:423 > Could not find expected file in persistent directory > /etc/NetworkManager/system-connections (RuntimeError) > > Same in "Booting Tails from a USB drive upgraded from USB with > persistence enabled" and "Booting a USB drive upgraded from ISO > with persistence enabled". > > And on next run, I cannot reproduce this. Weird. It's notable that that particular persistence preset's directory was changed in feature/wheezy. What was your --old-iso when you ran the first test vs the second? Could it have been a Wheezy-based image from before the persistence preset was changed? > 3. Scenario: Iceweasel should not have any plugins enabled # > features/torified_browsing.feature:26 > When I run "iceweasel" # > features/step_definitions/common_steps.rb:340 > And Iceweasel has started and is not loading a web page # > features/step_definitions/common_steps.rb:247 > And I open the address "about:plugins" in Iceweasel # > features/step_definitions/torified_browsing.rb:7 > Then I see "IceweaselNoPlugins.png" after at most 60 seconds # > features/step_definitions/common_steps.rb:302 > FindFailed: can not find IceweaselNoPlugins.png on the screen. > > I've seen this once out of three tries, without --keep-snapshots. > Unfortunately, I have no screenshot nor video anymore. I just had a look (with --view) at what's actually happening in these scenarios. I noticed that the Iceweasel window, after appearing, is minimized and than maximized, which is strange. It would also explain the randomness of this error as the following key presses surely can be lost while that's in motion. The reason is that we end up doing two @screen.wait_and_click("IceweaselRunning.png", ...) and IceweaselRunning.png was changed from being the title to the task icon in commit 13dce7f (introduced in the wheezy branch). The `wait_and_click` was introduced to ensure focus of the window, so it has to be e.g. the title. This is fixed now. > 4. Scenario: Memory erasure on an old computer # > features/erase_memory.fe > [...] > And I shutdown and wait for Tails to finish wiping the memory # > features/step_definitions/erase_memory.rb:164 > Then I find very few patterns in the guest's memory # > features/step_definitions/erase_memory.rb:140 > Pattern coverage: 0.314% (11 MiB) > 0.314% of the memory is filled with the pattern, but less than 0.250% > was expected (RuntimeError) > > I got this once out of two tries. Is it an acceptable drawback of > how the test suite works, or a real problem? To me it just shows that with the particular kernel (or whatever) that we happen to use now require us to bump the highly arbitrary 0.25% to 0.5%, perhaps. Without some more rigorous guideline to what we think is acceptable, arbitrary is what we've got. What do you think? Cheers! _______________________________________________ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.