Hi, > Hi, > > I think the direction of this project should also be discussed with > your mentor, but among the summer of code mentors I think we agreed > that it should be python based and not really fine grained unit tests > for C code. Unit testing specifically is not necessarily what we think > is the best idea, if it is the kind of thing that requires a lot of > effort to write and maintain as code changes. We want something that > allows us test as much as possible with as little effort as possible. > > I think there might have been a slight misunderstanding. The proposal was listed as Unit Tests even on our suggestion page and while I have said that we will evaluate at which level and to what extent they can be added these are definitely different from end-to-end (regression) tests. We will not add unit tests to each and every function (way over the scope of a GSOC project anyway ;) and certainly add tests only to parts where the code is currently not very volatile and the API is mostly stable. > Instead you can think of things like: > * Regression testing of renders compared to existing version (compare pixels) > * Same for e.g. mesh tools (compare mesh topology) > * Importers/exporters testing if the result is the same > * Maybe even ecording user operations and then verifying if the result > is the same? > * Verify operators don't crash when run in different contexts > * Testing that using API functions/properties don't crash with various inputs > * Testing for NaN/Inf number results. > * Testing for memory leak error prints. > While I agree that end-to-end tests like comparing renders are useful too, I don't think there is enough substance in them for a GSOC project, as it's more installing a few tools, creating some .blend files and comparing the output with previous version.
Unit tests are useful *during* development, to make sure new code doesn't break existing functionality. They help pointing out to exactly the function/part of the code that doesn't run as expected in contrast to a more coarse end-to-end test that tests the whole pipeline. Unit tests are meant to run fast (it is definitely a goal to keep the running time of *all* the tests definitely under a minute and single test suites under 10 secs or so) and thus can be run constantly like to make sure nothing breaks before I commit. The end-to-end tests also run *much* more slowly than unit tests (at least as soon as some kind of render is involved), so I don't believe they will be picked up by developers unless maybe shortly before a release. > In fact just based on renders you can test if most modifiers, nodes, > constraints, particle options still work as before. Clearly this is > not as useful as fine-grained unit testing as you usually do with > toolkits like Google Test, but only a testing framework that is simple > and requires little work to maintain will actually be picked up by > developers. This leads a bit more to 'regression testing' maybe, but > many of the things that break can't be expressed easily as assertions > unless you write hundreds of thousands of lines of testing code. > > Brecht. > I understand that for developers who are not familiar with unit testing the fact that there is code required to write the unit tests and some effort to maintain them might seem too much at first. I do hope to show with this project that the effort to maintain them isn't really that much, since if you think about it, the only time a test needs to change is when the API of a function changes. And this is something that we are moving to anyway - to have much more stable internal APIs that likely won't change *that* much and I wouldn't propose to test every single internal function, but test through the API that is exposed to other parts of Blender. In this light, if the API changes, you'd probably have to change a few places anyway and updating the tests is then not that much more compared to the normal development work. From my experience, having unit tests in place leads to code that is more bug free and stable and greatly reduces time spent debugging. So in the end the tests should *save* time. I actually don't think writing a small test takes more time than to add debug printf in strategic places and remove them later, which so far has been probably the most used debugging tool. The only difference is that the unit tests would remain as a permanent safety net and also act as a documentation on how the original coder expected the code to be used. The tests will be kept separate from the "productive code", so as to not clutter it, though we will work out how to structure this exactly as part of the GSOC project. To allow all developers to gain experience, I suggested to Leif to first look into specific areas that would benefit most from testing, such as mesh tools or modifiers and write a small number of tests initially to get the framework and automatic running of the tests in place. We would also probably not add tests to areas that are currently under heavy development. As tests are added, my hope is that more developers would see the benefits and add tests for their area of code. Of course tests will be discussed with the maintainers for their area of code and reviewed. The only thing that would be asked of the developers is to not break any tests in code they are not maintaining. I also would count it as a huge success, if during the GSOC project we even get a test coverage of 10% of the code. The second result I hope to get from the GSOC project is to find out how difficult it is to test the different areas of Blender and where we could probably benefit from cleaning up the architecture and dependencies a bit. Lastly Leif also planned for some code documentation towards the end of the project, which I think would also benefit the whole development process and make life easier for new developers. The reason why we are currently looking into C or C++ testing frameworks is that Leif is more familiar with C/C++ than with Python and as far as operators or tools are concerned, the tests can theoretically test the same level of the code, the calls would just not go through the Python API. As a mentor I would like to allow the student to choose his preferred language for this to make things easier. I hope that I could clarifiy things a bit and alleviate some fears, I honestly don't think that it will be such a big issue to maintain the tests. - Andrea - _______________________________________________ Bf-committers mailing list [email protected] http://lists.blender.org/mailman/listinfo/bf-committers
