After a talk with jcampan I think that the plan is to convert all the interactive_ui_tests to browser_tests and make the test launcher reusable.
On Wed, Sep 2, 2009 at 12:27, Scott Violet <[email protected]> wrote: > If we can't mix the two types, then separate targets is going to be > the only way to go. > > -Scott > > On Wed, Sep 2, 2009 at 10:47 AM, Paweł Hajdan > Jr.<[email protected]> wrote: > > Nick, thanks for clear response. Actually, sync_integration_tests don't > mix > > tests (that's good), but they don't use correct launcher (that's bad and > > will induce flakiness). See chrome/test/browser/browser_test_launcher_* > > files, and also how browser_tests binary is built in chrome.gyp. > > sync_integration_tests should use the same launcher. Maybe you can just > > include them in browser_tests? There is an easy way to add Win-specific > > tests to them. > > Here's my best try to clarify issues with interactive_ui_tests. They also > > need to use correct launcher for browser_tests to properly isolate them. > > However, they also contain UI tests. I'm not sure if it's possible to mix > > the two when using the browser_tests launcher, and if it won't make > > debugging harder (on Linux the launcher forks a process, and debugging > > launcher -> forked UI test binary -> browser process is going to be > hard). > > I'm looking for some advice, generally from people who wrote the > > browser_tests launcher and maintain interactive_ui_tests. > > > > On Wed, Sep 2, 2009 at 10:24, Nick Carter <[email protected]> wrote: > >> > >> sync_integration_tests should be only in proc browser tests. If it > >> includes other types of tests, that's a mistake and they should be moved > to > >> the unit tests target. Could you share some details of what you've > found, > >> and what it would take to make these tests as robust as browser_tests? > >> > >> As background, the sync integration tests run against a live sync > server, > >> possibly a test server running the same code as the Google production > >> servers, using real Google accounts. Currently the tests expect an > external > >> force (probably the python script we have) to provide it wish an sync > server > >> URL, user account, and password combo that is in a clean state. As a > >> result, we run the tests one at a time currently. So carried state bugs > in > >> memory between individual tests are not presently risky. Eventually, > I'd > >> like to see a more normal setup with gtest driving execution of multiple > >> tests, calling out to some external provider to get the sync > >> server/account/password info, instead of having it passed in. Probably, > the > >> provider would reset/restart the server instance through some backdoor. > >> > >> The dependence on an external server is why we put these tests in their > >> own executable; they should all be in-proc browser tests. > >> > >> - nick > >> > >> On Wed, Sep 2, 2009 at 9:44 AM, Paweł Hajdan Jr. < > [email protected]> > >> wrote: > >>> > >>> So, here are the choices I currently see: > >>> - let each binary only use one kind of tests (probably both UI) > >>> - split each binary to only one kind of tests binaries (probably too > much > >>> test binaries anyway) > >>> What do you think? I can help with the switch, just need a decision > what > >>> kind of tests/split would be better. > >>> > >>> On Mon, Aug 31, 2009 at 11:44, Paweł Hajdan Jr. < > [email protected]> > >>> wrote: > >>>> > >>>> I don't quite know how to solve the problem. When IN_PROC_BROWSER_TEST > >>>> is used, a launcher that will isolate the test cases should be used > (like > >>>> running them in forked processes or reloading a dll). > >>>> However, currently only browser_tests binary does that. Other users, > >>>> sync_integration_tests and interactive_ui_tests, don't. This is a > fantastic > >>>> recipe for flakiness. In-process browser test carries a lot of state, > all > >>>> the singletons, etc. It does cause problems if not cleaned up > properly, and > >>>> I removed that kind of problem from unit_tests (and introduced > >>>> ALLOW_IN_PROC_BROWSER_TEST to prevent the problem from growing). > >>>> The problem is, that the mentioned binaries mix UI tests with browser > >>>> tests. So it's not trivially possible to use browser_tests launcher > for them > >>>> (or maybe it is?). What do you think? > >>> > >>> > >> > > > > > > > > > > > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
