Reading the FlexUnit docs, I see that there’s a lot of advanced features (that I never even knew about).
I’m going to start with just the metadata tags that Josh wrote support for (i.e. Before, Text and After). Maybe I’ll add Ignore, BeforeClass and AfterClass as well. A question I have for those who have used FlexUnit is how extensively these advanced features are used? I have to figure out the best way to run tests on both node and Flash. It might be simplest to have two separate runner classes… Harbs > On Nov 4, 2017, at 7:20 PM, Harbs <harbs.li...@gmail.com> wrote: > > Wow. I don’t think I ever noticed the tutorial for FlexUnit on the Flex site: > https://flex.apache.org/flexunit/tutorial/ > <https://flex.apache.org/flexunit/tutorial/> > > Who put that together? Looks like a lot of work went into that. > > Harbs > >> On Nov 4, 2017, at 6:59 PM, Harbs <harbs.li...@gmail.com >> <mailto:harbs.li...@gmail.com>> wrote: >> >> By “Unit Tests”, I really meant tests which are UI agnostic. I’m not >> concerned by size of “Unit” tests, I’m more concerned about feature >> coverage. When I think of unit tests in this context I mean FlexUnit-type >> tests. The problem with FlexUnit, is that it only tests in Flash. Josh’s >> work solves that problem. Ideally, we’d have tests which are run on both >> platforms. I think We can use FlexUnit on Flash and Josh’s implementation to >> run the exact same tests in node. >> >> To me, it’s easier to tackle this problem if we first get “unit” tests >> working that are platform independent. >> >> Once that’s working, we can figure out how much the Unit-type tests can be >> extended to test UI features and/or Mustella can be used. Mustella is a big >> mystery to me. >> >> Whatever the strategy, the tests need to be easy to write. Maybe it already >> is easy to write mustella tests, but we need more documentation. I don’t >> know. >> >> Either way, I will try to add Josh’s code and write some tests and see how >> it goes. >> >> Harbs >> >>> On Nov 3, 2017, at 6:26 PM, Alex Harui <aha...@adobe.com.INVALID >>> <mailto:aha...@adobe.com.INVALID>> wrote: >>> >>> I was unable to tell what it is Josh has from that link. >>> >>> Right now, we have 3 test subsystems: >>> 1) Java-based tests using Selenium that Chris Dutz wrote >>> 2) FlexUnit tests (written in AS3) that only work in Flash (in the Basic >>> project) >>> 3) Mustella tests that are written in MXML that run on both Flash and in >>> the browser using Selenium. >>> >>> This thread is titled "Unit Tests" Do we have agreement on what the size >>> of the unit is? >>> >>> Is there any advantage to trying to get FlexUnit to work without Flash? >>> For example, would folks migrating have old FlexUnit tests they want to >>> run? >>> >>> I agree with Piotr that it would be better if tests didn't have to be >>> written in Java. Using a declarative language like MXML makes you more >>> output-language agnostic. Mustella works by having each Mustella command >>> implemented in Java for Selenium. It constrains what your test can do to >>> things we know we can do on all platforms, and is agnostic about when the >>> next command run, which is helpful when there are differences in the way >>> the runtimes execute code and make results available. I'd have to dig >>> into it again, but I think there are some issues in failure reporting in >>> Selenium of Selenium thinks there is only one test step which is Java code >>> that shoved a pile of JS into the browser to be executed. >>> >>> It would be nice if we can leverage what we have instead of building a >>> whole new test system, but maybe Josh has something that is closer to what >>> we are looking for. >>> >>> My 2 cents, >>> -Alex >>> >>> On 11/3/17, 8:37 AM, "Josh Tynjala" <joshtynj...@apache.org >>> <mailto:joshtynj...@apache.org>> wrote: >>> >>>> Please feel free to use my test runner. I'm happy to commit it somewhere >>>> to make it official Apache code, if necessary. >>>> >>>> - Josh >>>> >>>> On 2017-11-03 04:19, Harbs <harbs.li...@gmail.com >>>> <mailto:harbs.li...@gmail.com>> wrote: >>>>> One topic which keeps coming up is better test coverage for Royale. >>>>> >>>>> I think this is becoming a critical issue for a few reasons: >>>>> 1. As we get close to version 1.0 it’s necessary to have good test >>>>> coverage for confidence of quality and confidence that we don’t >>>>> introduce recessive bugs. >>>>> 2. It’s really hard to accept Github pull requests without examining >>>>> the code VERY well that it does not introduce recessive bugs. CI which >>>>> runs automated tests could give a preliminary test on pull requests to >>>>> ensure that they don’t break anything. If the pull requests do break >>>>> something, it allows the requester to fix the problem with confidence >>>>> without taking others’ time. >>>>> >>>>> I think we need to break up testing into pieces and figure out a >>>>> strategy to implement automated tests in a way that they are >>>>> maintainable. >>>>> >>>>> Some points: >>>>> 1. I think integration into something like Travis would be very helpful. >>>>> 2. I think there’s a Jenkins plugin for building pull requests. Not >>>>> sure how useful it is.[1] >>>>> 3. Josh has created a Node.js-compatible test-runner architecture which >>>>> could be useful for unit tests on parts of the framework which don’t >>>>> rely on browser features. (i.e. models and the like) He mentioned the >>>>> possibility of donating it, and I think that would be a very useful >>>>> feature. >>>>> 4. For UI and integration tests, there seem to be some pretty cool >>>>> integrations using Selenium.[2][3] >>>>> 5. I think the main testing effort needs to be using JS and something >>>>> like Josh’s testing framework for non-UI pieces and some easy-to-use >>>>> Selenium integration for visual pieces and integration tests. >>>>> 6. We probably also want some API endpoints we can test off of for >>>>> integration testing. >>>>> >>>>> I’m willing to invest time into this. >>>>> >>>>> It’s going to be a lot of work building out the tests and I think we >>>>> need a plan for that. My thoughts: >>>>> >>>>> 1. Step one is to make it easy to write meaningful automated tests and >>>>> establish a clear pattern. >>>>> 2. Step two is to start writing tests starting from the >>>>> most-used/easiest to beak pieces and work out from there. >>>>> 3. Once the pattern is established, any new pieces MUST have testing >>>>> coverage. >>>>> 4. When fixing bugs, attention should be paid to adding testing for >>>>> that component. >>>>> 5. When a pull request comes in on a piece which does not have unit >>>>> test, a test must be written before accepting the pull request. The test >>>>> does not need to be written by the requester. Before examining the >>>>> request, the test should be written to pass for expected behavior and >>>>> fail for the bug that the pull request is attempting to fix (assuming >>>>> the pull request is to fix a bug). >>>>> >>>>> Thoughts? >>>>> Harbs >>>>> >>>>> P.S. I’m thinking of coming to the US in late December/early January. >>>>> I would be interested in getting together for a hacking session with >>>>> folks who are available. >>>>> >>>>> >>>>> [1]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki >>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki>. >>>>> jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull%2Brequest%2Bbuilder%2Bplug >>>>> in&data=02%7C01%7C%7C36d681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b3443879 >>>>> 4aed2c178decee1%7C0%7C0%7C636453202314829220&sdata=dqT0Nao8HHzlkWmk6zy%2F >>>>> eDEGi6nXYkM8P0%2Bbibe4NA4%3D&reserved=0 >>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.je >>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.je> >>>>> nkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull%2Brequest%2Bbuilder%2Bplugin >>>>> &data=02%7C01%7C%7C36d681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794a >>>>> ed2c178decee1%7C0%7C0%7C636453202314829220&sdata=dqT0Nao8HHzlkWmk6zy%2FeD >>>>> EGi6nXYkM8P0%2Bbibe4NA4%3D&reserved=0> >>>>> >>>>> [2]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs >>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs>. >>>>> travis-ci.com >>>>> <http://travis-ci.com/>%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C36d >>>>> 681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0% >>>>> 7C636453202314829220&sdata=9D9riTJ29rqNonT81v%2FRtdpC6J8titW7LIUb07LLXa4% >>>>> 3D&reserved=0 >>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.tr >>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.tr> >>>>> avis-ci.com >>>>> <http://avis-ci.com/>%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C36d68 >>>>> 1ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C >>>>> 636453202314829220&sdata=9D9riTJ29rqNonT81v%2FRtdpC6J8titW7LIUb07LLXa4%3D >>>>> &reserved=0> >>>>> >>>>> [3]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsauce >>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsauce> >>>>> labs.com >>>>> <http://labs.com/>%2Fproducts%2Fintegrations&data=02%7C01%7C%7C36d681ea88ba4315312f >>>>> 08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636453202314829 >>>>> 220&sdata=WNzEofopPA6YaHS51O%2BCx6Zrsc2qLOFcSWMUUXssD3M%3D&reserved=0 >>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsaucela >>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsaucela> >>>>> bs.com >>>>> <http://bs.com/>%2Fproducts%2Fintegrations&data=02%7C01%7C%7C36d681ea88ba4315312f08 >>>>> d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C63645320231482922 >>>>> 0&sdata=WNzEofopPA6YaHS51O%2BCx6Zrsc2qLOFcSWMUUXssD3M%3D&reserved=0> >>> >> >
