On Fri, Dec 16, 2016 at 2:43 AM, Gustavo Sverzut Barbieri <barbi...@gmail.com> wrote: > On Fri, Dec 16, 2016 at 7:54 AM, Stefan Schmidt <ste...@osg.samsung.com> > wrote: >> On 13/12/16 16:32, Gustavo Sverzut Barbieri wrote: >>> On Tue, Dec 13, 2016 at 11:54 AM, Stefan Schmidt <ste...@osg.samsung.com> >>> wrote: >>>> On 13/12/16 14:09, Felipe Magno de Almeida wrote: >>>>> On Tue, Dec 13, 2016 at 8:28 AM, Stefan Schmidt <ste...@osg.samsung.com> >>>>> wrote:
[snip] >>>> but a test that fails on a machine that doesn't >>>>> handle IPv6 seems fine to me as long as it is rare enough. >>>> >>>> We assume IPv6 now, we assume a working internet connection, we assume >>>> udev for eeze testing, we assume a working dbus setup, in some cases we >>>> assume a file system which supports extended attributes, etc... >>>> >>>> If it is complicated to run make check we will have even less people >>>> running it. It should be the other way around. I guess I could look at >>>> anybody here who contributed a few patches this year and see someone who >>>> broke make check. If it is to complicated or fails for some reason >>>> people will just stop using it. >>> >>> our test suite is unusable the way it is... >> >> While I agree that the test suite needs improvements calling it unusable >> is over the top. > > ok, so improved... particularly for developers to run them... also > something that simplifies to write them, and how to do it properly. > > For instance, very few tests uses correct macros in check.h, like > ck_assert_int_eq(), that will highlight expected values and required > values, compared to failif(a == b). Some tests miss granularity, so > they do bunch of tests at once, missing the "unit test" aspect... and > while we can manually select the test suite to run (calling it > directly), and sometimes the component (ie: ecore_con_suite -h to list > them), it would be very helpful to select individual functions to > test, particularly under development. And some helpers to debug > failing tests (I always need to check for CK_FORK=no and things like > that). I agree with you on the missing granularity. Some tests are awful in that regard. Overall I like most of your point here, it is just so massive and our ressource are spread so thinly that it is not something anyone working on the core as time to work on. But this task are pretty simple and easy, they are quite an easy training to learn our API, environment and contribute their first patches. We could have a beginner/new comer section in our wiki for developers to know about this. As for the helper, a simple shell script that set CK_RUN_CASE, CK_FORK and start the test suite under valgrind + gdb shouldn't be to hard I think. I personnaly always rely on CK_RUN_CASE + valgrind --trace-children and do not bother with gdb, but I can see its use. > what happens in most of the cases is that the suite is painful to deal > with so it's easier to write the "other required" bit that is the > example and test from there, rather than do it from the test suite... > which is what I did, and some guys follow the same pattern (or just > commit to enlightenment an usage). > >> really it lacks >>> granularity, it takes lots of time and most of the time it looks like >>> it failed since people forget to trap EINA_LOG to capture errors, then >>> you get tons of error messages (although the test reports success, >>> messages confuses users). >> >> In the normal run you see a green PASS or red FAILED. The eina_log >> errors are only in the log file itself. > > still, this is for build bot or random users running the tests. Not to > us, the developers... you're going to run the full test suite after > each development step... becomes costly and painful :-/ I agree with Gustavo here I think. Having all our tests hidden behind PASS/FAILED make it a problem to deal with when running make check as you have to read somewhere else what is going on. Would be neat if they where not directed to a file actually... but then we really should be more careful with our printf/log ! -- Cedric BAIL ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ enlightenment-devel mailing list enlightenment-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/enlightenment-devel
