I'll have a look at the browser test launcher and see if we can reuse it with the sync integration tests. I think we're better off keeping the sync tests segregated, so long as they have a dependency on live Google services and accounts (e.g. they do real authentication with Google Accounts). The external setup that these currently tests require make them difficult to run, and including them in browser_tests would either force them to be disabled by default (making them possibly even harder to use/comprehend) or else they'd fail all the time, wasting people's time. Right now, it's mostly just the buildbot that runs these tests.
If we had a reference sync server implementation that could be run on the local machine, then we could definitely consider integrating these tests, but of course then they'd be testing something different. - nick On Wed, Sep 2, 2009 at 10:47 AM, Paweł Hajdan Jr. <[email protected]>wrote: > Nick, thanks for clear response. Actually, sync_integration_tests don't mix > tests (that's good), but they don't use correct launcher (that's bad and > will induce flakiness). See chrome/test/browser/browser_test_launcher_* > files, and also how browser_tests binary is built in chrome.gyp. > sync_integration_tests should use the same launcher. Maybe you can just > include them in browser_tests? There is an easy way to add Win-specific > tests to them. > Here's my best try to clarify issues with interactive_ui_tests. They also > need to use correct launcher for browser_tests to properly isolate them. > However, they also contain UI tests. I'm not sure if it's possible to mix > the two when using the browser_tests launcher, and if it won't make > debugging harder (on Linux the launcher forks a process, and debugging > launcher -> forked UI test binary -> browser process is going to be hard). > I'm looking for some advice, generally from people who wrote the > browser_tests launcher and maintain interactive_ui_tests. > > > On Wed, Sep 2, 2009 at 10:24, Nick Carter <[email protected]> wrote: > >> sync_integration_tests should be only in proc browser tests. If it >> includes other types of tests, that's a mistake and they should be moved to >> the unit tests target. Could you share some details of what you've found, >> and what it would take to make these tests as robust as browser_tests? >> >> As background, the sync integration tests run against a live sync server, >> possibly a test server running the same code as the Google production >> servers, using real Google accounts. Currently the tests expect an external >> force (probably the python script we have) to provide it wish an sync server >> URL, user account, and password combo that is in a clean state. As a >> result, we run the tests one at a time currently. So carried state bugs in >> memory between individual tests are not presently risky. Eventually, I'd >> like to see a more normal setup with gtest driving execution of multiple >> tests, calling out to some external provider to get the sync >> server/account/password info, instead of having it passed in. Probably, the >> provider would reset/restart the server instance through some backdoor. >> >> The dependence on an external server is why we put these tests in their >> own executable; they should all be in-proc browser tests. >> >> - nick >> >> On Wed, Sep 2, 2009 at 9:44 AM, Paweł Hajdan Jr. <[email protected] >> > wrote: >> >>> So, here are the choices I currently see: >>> - let each binary only use one kind of tests (probably both UI) >>> - split each binary to only one kind of tests binaries (probably too much >>> test binaries anyway) >>> >>> What do you think? I can help with the switch, just need a decision what >>> kind of tests/split would be better. >>> >>> >>> On Mon, Aug 31, 2009 at 11:44, Paweł Hajdan Jr. <[email protected] >>> > wrote: >>> >>>> I don't quite know how to solve the problem. When IN_PROC_BROWSER_TEST >>>> is used, a launcher that will isolate the test cases should be used (like >>>> running them in forked processes or reloading a dll). >>>> However, currently only browser_tests binary does that. Other users, >>>> sync_integration_tests and interactive_ui_tests, don't. This is a fantastic >>>> recipe for flakiness. In-process browser test carries a lot of state, all >>>> the singletons, etc. It does cause problems if not cleaned up properly, and >>>> I removed that kind of problem from unit_tests (and introduced >>>> ALLOW_IN_PROC_BROWSER_TEST to prevent the problem from growing). >>>> >>>> The problem is, that the mentioned binaries mix UI tests with browser >>>> tests. So it's not trivially possible to use browser_tests launcher for >>>> them >>>> (or maybe it is?). What do you think? >>>> >>> >>> >>> >>> >>> >> > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
