If we can't mix the two types, then separate targets is going to be the only way to go.
-Scott On Wed, Sep 2, 2009 at 10:47 AM, Paweł Hajdan Jr.<[email protected]> wrote: > Nick, thanks for clear response. Actually, sync_integration_tests don't mix > tests (that's good), but they don't use correct launcher (that's bad and > will induce flakiness). See chrome/test/browser/browser_test_launcher_* > files, and also how browser_tests binary is built in chrome.gyp. > sync_integration_tests should use the same launcher. Maybe you can just > include them in browser_tests? There is an easy way to add Win-specific > tests to them. > Here's my best try to clarify issues with interactive_ui_tests. They also > need to use correct launcher for browser_tests to properly isolate them. > However, they also contain UI tests. I'm not sure if it's possible to mix > the two when using the browser_tests launcher, and if it won't make > debugging harder (on Linux the launcher forks a process, and debugging > launcher -> forked UI test binary -> browser process is going to be hard). > I'm looking for some advice, generally from people who wrote the > browser_tests launcher and maintain interactive_ui_tests. > > On Wed, Sep 2, 2009 at 10:24, Nick Carter <[email protected]> wrote: >> >> sync_integration_tests should be only in proc browser tests. If it >> includes other types of tests, that's a mistake and they should be moved to >> the unit tests target. Could you share some details of what you've found, >> and what it would take to make these tests as robust as browser_tests? >> >> As background, the sync integration tests run against a live sync server, >> possibly a test server running the same code as the Google production >> servers, using real Google accounts. Currently the tests expect an external >> force (probably the python script we have) to provide it wish an sync server >> URL, user account, and password combo that is in a clean state. As a >> result, we run the tests one at a time currently. So carried state bugs in >> memory between individual tests are not presently risky. Eventually, I'd >> like to see a more normal setup with gtest driving execution of multiple >> tests, calling out to some external provider to get the sync >> server/account/password info, instead of having it passed in. Probably, the >> provider would reset/restart the server instance through some backdoor. >> >> The dependence on an external server is why we put these tests in their >> own executable; they should all be in-proc browser tests. >> >> - nick >> >> On Wed, Sep 2, 2009 at 9:44 AM, Paweł Hajdan Jr. <[email protected]> >> wrote: >>> >>> So, here are the choices I currently see: >>> - let each binary only use one kind of tests (probably both UI) >>> - split each binary to only one kind of tests binaries (probably too much >>> test binaries anyway) >>> What do you think? I can help with the switch, just need a decision what >>> kind of tests/split would be better. >>> >>> On Mon, Aug 31, 2009 at 11:44, Paweł Hajdan Jr. <[email protected]> >>> wrote: >>>> >>>> I don't quite know how to solve the problem. When IN_PROC_BROWSER_TEST >>>> is used, a launcher that will isolate the test cases should be used (like >>>> running them in forked processes or reloading a dll). >>>> However, currently only browser_tests binary does that. Other users, >>>> sync_integration_tests and interactive_ui_tests, don't. This is a fantastic >>>> recipe for flakiness. In-process browser test carries a lot of state, all >>>> the singletons, etc. It does cause problems if not cleaned up properly, and >>>> I removed that kind of problem from unit_tests (and introduced >>>> ALLOW_IN_PROC_BROWSER_TEST to prevent the problem from growing). >>>> The problem is, that the mentioned binaries mix UI tests with browser >>>> tests. So it's not trivially possible to use browser_tests launcher for >>>> them >>>> (or maybe it is?). What do you think? >>> >>> >> > > > > > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
