Nick, thanks for clear response. Actually, sync_integration_tests don't mix tests (that's good), but they don't use correct launcher (that's bad and will induce flakiness). See chrome/test/browser/browser_test_launcher_* files, and also how browser_tests binary is built in chrome.gyp. sync_integration_tests should use the same launcher. Maybe you can just include them in browser_tests? There is an easy way to add Win-specific tests to them. Here's my best try to clarify issues with interactive_ui_tests. They also need to use correct launcher for browser_tests to properly isolate them. However, they also contain UI tests. I'm not sure if it's possible to mix the two when using the browser_tests launcher, and if it won't make debugging harder (on Linux the launcher forks a process, and debugging launcher -> forked UI test binary -> browser process is going to be hard). I'm looking for some advice, generally from people who wrote the browser_tests launcher and maintain interactive_ui_tests.
On Wed, Sep 2, 2009 at 10:24, Nick Carter <[email protected]> wrote: > sync_integration_tests should be only in proc browser tests. If it > includes other types of tests, that's a mistake and they should be moved to > the unit tests target. Could you share some details of what you've found, > and what it would take to make these tests as robust as browser_tests? > > As background, the sync integration tests run against a live sync server, > possibly a test server running the same code as the Google production > servers, using real Google accounts. Currently the tests expect an external > force (probably the python script we have) to provide it wish an sync server > URL, user account, and password combo that is in a clean state. As a > result, we run the tests one at a time currently. So carried state bugs in > memory between individual tests are not presently risky. Eventually, I'd > like to see a more normal setup with gtest driving execution of multiple > tests, calling out to some external provider to get the sync > server/account/password info, instead of having it passed in. Probably, the > provider would reset/restart the server instance through some backdoor. > > The dependence on an external server is why we put these tests in their own > executable; they should all be in-proc browser tests. > > - nick > > On Wed, Sep 2, 2009 at 9:44 AM, Paweł Hajdan Jr. > <[email protected]>wrote: > >> So, here are the choices I currently see: >> - let each binary only use one kind of tests (probably both UI) >> - split each binary to only one kind of tests binaries (probably too much >> test binaries anyway) >> >> What do you think? I can help with the switch, just need a decision what >> kind of tests/split would be better. >> >> >> On Mon, Aug 31, 2009 at 11:44, Paweł Hajdan Jr. >> <[email protected]>wrote: >> >>> I don't quite know how to solve the problem. When IN_PROC_BROWSER_TEST is >>> used, a launcher that will isolate the test cases should be used (like >>> running them in forked processes or reloading a dll). >>> However, currently only browser_tests binary does that. Other users, >>> sync_integration_tests and interactive_ui_tests, don't. This is a fantastic >>> recipe for flakiness. In-process browser test carries a lot of state, all >>> the singletons, etc. It does cause problems if not cleaned up properly, and >>> I removed that kind of problem from unit_tests (and introduced >>> ALLOW_IN_PROC_BROWSER_TEST to prevent the problem from growing). >>> >>> The problem is, that the mentioned binaries mix UI tests with browser >>> tests. So it's not trivially possible to use browser_tests launcher for them >>> (or maybe it is?). What do you think? >>> >> >> >> >> >> > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
