Re: Unit testing: The Small Plan
On Wed, May 8, 2013 at 2:59 AM, Elmar Hinz t3el...@googlemail.com wrote: To get started, I would need to do some more simple cases first. Hi Elmar, Did you make any progress on this? I know little about unit tests so I won't be able to help much here, but I'm interested in any progress you feel like sharing. Scott
Re: Unit testing: The Small Plan
On Wed, May 8, 2013 at 2:59 AM, Elmar Hinzwrote: > To get started, I would need to do some more simple cases first. Hi Elmar, Did you make any progress on this? I know little about unit tests so I won't be able to help much here, but I'm interested in any progress you feel like sharing. Scott
Re: Unit testing: The Small Plan
Hello Cyrille, On Wed, May 8, 2013 at 1:28 AM, Cyrille Artho c.ar...@aist.go.jp wrote: Hi Elmar, I think your plan covers the question HOW do we want to unit test the software well. However, we have not thought much about the WHAT do we want to test? question. Essentially, we need to think about which classes/functions to test first. I think it is not realistic to aim for a high test coverage with software as large as LyX. Unit tests make sense in certain cases and less in others. So we should identify these use cases first, and start with a few selective unit tests. We can then grow them as we see fit. Yes, I share this vision of let the tests grow. That's the reason, why it's only a small plan. A plan can't tell the people what they want to test. It's the decision of the people themself, driven by different motivations. A plan can only leverage. There are at least two questions: * What do I want to test? * What can I test at all, at reasonable costs. When I write new software in other languages, I write code and tests side by side, resulting in a test coverage of approximately 100%, without extra costs, as I need to run the code anyway. Writing tests for existing software, is much more difficult. In the beginning that are extra costs. They pay on the long run by fewer bugs. I have to weight. For user-driven actions (LFUNs), LyX already has a test framework. However, these tests do not cover internals of LyX. Right, unit testing is only one class of testing. That's why it has it's own path tests/unit. Gui testing needs completely different approaches and more support by testing framewoks resulting in tests/gui or similar. Probably there are also tools to cover memory leaks or performance. Which internals do not require a complex sequence of actions (or a complex internal state) to be tested? (Those that do are maybe better covered with other types of tests.) Where has the code changed often in the past, and is expected to change often in the future? What kinds of (internal) functions are often covered by bug reports and thus warrant better test automation? Likely the most error-prone corners are the haredest to tests, because of the same reasons. To get started, I would need to do some more simple cases first. I hope some of the veteran developers can help answer these questions. \Elmar -- Elmar Hinz Freiherr-vom-Stein-Str. 1 33014 Bad Driburg TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m
Re: Unit testing: The Small Plan
Hello Cyrille, On Wed, May 8, 2013 at 1:28 AM, Cyrille Arthowrote: > Hi Elmar, > I think your plan covers the question "HOW do we want to unit test the > software" well. However, we have not thought much about the "WHAT do we > want to test?" question. Essentially, we need to think about which > classes/functions to test first. > > I think it is not realistic to aim for a high test coverage with software > as large as LyX. Unit tests make sense in certain cases and less in others. > So we should identify these use cases first, and start with a few selective > unit tests. We can then grow them as we see fit. > > Yes, I share this vision of let the tests grow. That's the reason, why it's only a small plan. A plan can't tell the people what they want to test. It's the decision of the people themself, driven by different motivations. A plan can only leverage. There are at least two questions: * What do I want to test? * What can I test at all, at reasonable costs. When I write new software in other languages, I write code and tests side by side, resulting in a test coverage of approximately 100%, without extra costs, as I need to run the code anyway. Writing tests for existing software, is much more difficult. In the beginning that are extra costs. They pay on the long run by fewer bugs. I have to weight. > For user-driven actions (LFUNs), LyX already has a test framework. > However, these tests do not cover internals of LyX. > > Right, unit testing is only one class of testing. That's why it has it's own path "tests/unit". Gui testing needs completely different approaches and more support by testing framewoks resulting in "tests/gui" or similar. Probably there are also tools to cover memory leaks or performance. > Which internals do not require a complex sequence of actions (or a complex > internal state) to be tested? (Those that do are maybe better covered with > other types of tests.) Where has the code changed often in the past, and is > expected to change often in the future? What kinds of (internal) functions > are often covered by bug reports and thus warrant better test automation? > Likely the most error-prone corners are the haredest to tests, because of the same reasons. To get started, I would need to do some more simple cases first. > > I hope some of the veteran developers can help answer these questions. > > \Elmar -- Elmar Hinz Freiherr-vom-Stein-Str. 1 33014 Bad Driburg TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m
Unit testing: The Small Plan
Hello list, I'd like to come up with a small plan for getting started with unit testing: == 1.) Directory structure: tests/unit/ == * Unit tests stay their own directory separated from src/. * Below tests/unit the directory structure mirrors the structure of src/. * The reason for the subfolder unit is, that unit tests are not the only type of tests. Unit tests test the single units of the source. == 2.) Mocking: googlemock == This weekend I researched for alternatives, but there was none that made a stable impression to me apart from googlemock. With C++ mocking can take two approaches: 2.a) Turning classes into Templates, to be able to replace members by mocks. http://programmaticallyspeaking.blogspot.de/2010/04/beautiful-dependency-injection-in-c.html This seems to invasive into the existing sources for me. 2.b) Inheriting the mocks from the real objects, so that they are of the same type. This is the approach of googlemock. Googlemock can be used with any mocking framework. It best integrates with googletest. == 3.) Testing framework: any == Unit tests are independent units, so any framework or none can be used. For the reason of mocks, googletest is recommended. == 4.) Runnig all tests == To be able to run all tests at once they need to be detected. * The naming pattern of the binary to support this: whateverTest * A successful test binary returns 0, the others an Error. * The global test detector and runner is a Python script. == 5.) Dependency injection == Dependency injection (the mock objects) is more difficult with C++ for multiple reasons like the missing garbage collection or reflection. The approach that looks most forward and less intrusive for the sources to me, is to directly access the private members by making them public during testing and only during testing. That is not really kosher, but at least a simple way to get started directly. It is done by use of a simple preprocessor directive: #public_on_testing Once a wall of tests is created to ensure the behaviour of the given API, more skilled approaches can be introduced. \Elmar -- Elmar Hinz Freiherr-vom-Stein-Str. 1 33014 Bad Driburg TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m
Re: Unit testing: The Small Plan
On 05/07/2013 04:57 AM, Elmar Hinz wrote: Hello list, I'd like to come up with a small plan for getting started with unit testing: I am a total ignoramus when it comes to unit testing, so I will leave it to others who actually know something to express a view. Richard
Re: Unit testing: The Small Plan
Elmar Hinz wrote: If somebody can give improvements to the plan, it's welcome. I guess people will let you do almost anything what you like in test/* but they will become much more picky when it comes to changes in src/. Perhaps the best way is to try example, post patch here and and look what people think. Pavel
Re: Unit testing: The Small Plan
Op 7-5-2013 10:57, Elmar Hinz schreef: Hello list, I'd like to come up with a small plan for getting started with unit testing: I would like to see some examples of mocking and injection. I tried to write some tests using the google framework, and started with the Buffer class. This immediately gives you the problem that it is dependent on a large number of other classes. So, this would mean that you have to fake a large part of the LyX codebase. Then I tried to link against all libraries of LyX, but then I ran into problems that the application was not initialized (e.g., Package, translations etc.) I'm afraid we can't do much testing until the above is solved. My attempt thus far can be found at: git://git.lyx.org/developers/vfr/lyx.git That is not really kosher, but at least a simple way to get started directly. It is done by use of a simple preprocessor directive: #public_on_testing Once a wall of tests is created to ensure the behaviour of the given API, more skilled approaches can be introduced. Ideally, one would not need to care about private variables because we are only interested in that the public interface does what it is supposed to do. Right ? Vincent
Re: Unit testing: The Small Plan
On Tue, May 7, 2013 at 9:20 PM, Pavel Sanda sa...@lyx.org wrote: Elmar Hinz wrote: If somebody can give improvements to the plan, it's welcome. I guess people will let you do almost anything what you like in test/* but they will become much more picky when it comes to changes in src/. Perhaps the best way is to try example, post patch here and and look what people think. Pavel Hi Pavel, as long as people don't become picky it's a good sign. They even shouldn't become picky, during big refactoring of the sources. It works in two steps. In the first step you write tests, that prove the existing API is fully functioning. Then you start with new functions or other kind of refactoring. The test bell should immediately ring when something is broken, long before people get picky at all. The drawback may be that people don't notice that you are working until you come out with the new feature. Fotunatly the world is never that perfect. \Elmar -- Elmar Hinz Freiherr-vom-Stein-Str. 1 33014 Bad Driburg TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m
Re: Unit testing: The Small Plan
I would like to see some examples of mocking and injection. Thank you, Vincent, I tried to write some tests using the google framework, and started with the Buffer class. This immediately gives you the problem that it is dependent on a large number of other classes. So, this would mean that you have to fake a large part of the LyX codebase. Well yes, actually I would start with the smallest class and do the Buffer class as the last one. I guess it requires a lot of experiance to tame that. Then I tried to link against all libraries of LyX, but then I ran into problems that the application was not initialized (e.g., Package, translations etc.) I'm afraid we can't do much testing until the above is solved. So a few tests already brought up a vision of what need to be solved. That's exactly their most important strength. They are a perfect teacher. Here we find out how everything currently depends on everything. We couldn't even take a little piece out, to use it as a library for something completly different. Regards \Elmar -- Elmar Hinz Freiherr-vom-Stein-Str. 1 33014 Bad Driburg TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m
Re: Unit testing: The Small Plan
Ideally, one would not need to care about private variables because we are only interested in that the public interface does what it is supposed to do. Right ? Yes, at least as far as it concerns the testing. Denpendency Injection has other aspects. As an example, if there is a class that prints to stdout you mayby would not test that, taking it as an interal behaviour. With DI you inject the output stream, with the result that you can use different printers in future. Lyx already does this in many caes. Exactly that is dependancy injection. Now you could drop in a MockStream to prove that the class uses the stream object like expacted. Regards \Elmar -- Elmar Hinz Freiherr-vom-Stein-Str. 1 33014 Bad Driburg TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m
Re: Unit testing: The Small Plan
Hi Elmar, I think your plan covers the question HOW do we want to unit test the software well. However, we have not thought much about the WHAT do we want to test? question. Essentially, we need to think about which classes/functions to test first. I think it is not realistic to aim for a high test coverage with software as large as LyX. Unit tests make sense in certain cases and less in others. So we should identify these use cases first, and start with a few selective unit tests. We can then grow them as we see fit. For user-driven actions (LFUNs), LyX already has a test framework. However, these tests do not cover internals of LyX. Which internals do not require a complex sequence of actions (or a complex internal state) to be tested? (Those that do are maybe better covered with other types of tests.) Where has the code changed often in the past, and is expected to change often in the future? What kinds of (internal) functions are often covered by bug reports and thus warrant better test automation? I hope some of the veteran developers can help answer these questions. Elmar Hinz wrote: Hello list, I'd like to come up with a small plan for getting started with unit testing: == 1.) Directory structure: tests/unit/ == * Unit tests stay their own directory separated from src/. * Below tests/unit the directory structure mirrors the structure of src/. * The reason for the subfolder unit is, that unit tests are not the only type of tests. Unit tests test the single units of the source. == 2.) Mocking: googlemock == This weekend I researched for alternatives, but there was none that made a stable impression to me apart from googlemock. With C++ mocking can take two approaches: 2.a) Turning classes into Templates, to be able to replace members by mocks. http://programmaticallyspeaking.blogspot.de/2010/04/beautiful-dependency-injection-in-c.html This seems to invasive into the existing sources for me. 2.b) Inheriting the mocks from the real objects, so that they are of the same type. This is the approach of googlemock. Googlemock can be used with any mocking framework. It best integrates with googletest. == 3.) Testing framework: any == Unit tests are independent units, so any framework or none can be used. For the reason of mocks, googletest is recommended. == 4.) Runnig all tests == To be able to run all tests at once they need to be detected. * The naming pattern of the binary to support this: whateverTest * A successful test binary returns 0, the others an Error. * The global test detector and runner is a Python script. == 5.) Dependency injection == Dependency injection (the mock objects) is more difficult with C++ for multiple reasons like the missing garbage collection or reflection. The approach that looks most forward and less intrusive for the sources to me, is to directly access the private members by making them public during testing and only during testing. That is not really kosher, but at least a simple way to get started directly. It is done by use of a simple preprocessor directive: #public_on_testing Once a wall of tests is created to ensure the behaviour of the given API, more skilled approaches can be introduced. \Elmar -- Elmar Hinz Freiherr-vom-Stein-Str. 1 33014 Bad Driburg TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m -- Regards, Cyrille Artho - http://artho.com/ Love is the delusion that one woman differs from another. -- H.L. Mencken
Unit testing: The Small Plan
Hello list, I'd like to come up with a small plan for getting started with unit testing: == 1.) Directory structure: "tests/unit/" == * Unit tests stay their own directory separated from src/. * Below "tests/unit" the directory structure mirrors the structure of "src/". * The reason for the subfolder "unit" is, that unit tests are not the only type of tests. Unit tests test the single units of the source. == 2.) Mocking: googlemock == This weekend I researched for alternatives, but there was none that made a stable impression to me apart from googlemock. With C++ mocking can take two approaches: 2.a) Turning classes into Templates, to be able to replace members by mocks. http://programmaticallyspeaking.blogspot.de/2010/04/beautiful-dependency-injection-in-c.html This seems to invasive into the existing sources for me. 2.b) Inheriting the mocks from the real objects, so that they are of the same type. This is the approach of googlemock. Googlemock can be used with any mocking framework. It best integrates with googletest. == 3.) Testing framework: any == Unit tests are independent units, so any framework or none can be used. For the reason of mocks, googletest is recommended. == 4.) Runnig all tests == To be able to run all tests at once they need to be detected. * The naming pattern of the binary to support this: whateverTest * A successful test binary returns 0, the others an Error. * The global test detector and runner is a Python script. == 5.) Dependency injection == Dependency injection (the mock objects) is more difficult with C++ for multiple reasons like the missing garbage collection or reflection. The approach that looks most forward and less intrusive for the sources to me, is to directly access the private members by making them public during testing and only during testing. That is not really kosher, but at least a simple way to get started directly. It is done by use of a simple preprocessor directive: #public_on_testing Once a wall of tests is created to ensure the behaviour of the given API, more skilled approaches can be introduced. \Elmar -- Elmar Hinz Freiherr-vom-Stein-Str. 1 33014 Bad Driburg TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m
Re: Unit testing: The Small Plan
On 05/07/2013 04:57 AM, Elmar Hinz wrote: Hello list, I'd like to come up with a small plan for getting started with unit testing: I am a total ignoramus when it comes to unit testing, so I will leave it to others who actually know something to express a view. Richard
Re: Unit testing: The Small Plan
Elmar Hinz wrote: > If somebody can give improvements to the plan, it's welcome. I guess people will let you do almost anything what you like in test/* but they will become much more picky when it comes to changes in src/. Perhaps the best way is to try example, post patch here and and look what people think. Pavel
Re: Unit testing: The Small Plan
Op 7-5-2013 10:57, Elmar Hinz schreef: Hello list, I'd like to come up with a small plan for getting started with unit testing: I would like to see some examples of mocking and injection. I tried to write some tests using the google framework, and started with the Buffer class. This immediately gives you the problem that it is dependent on a large number of other classes. So, this would mean that you have to fake a large part of the LyX codebase. Then I tried to link against all libraries of LyX, but then I ran into problems that the application was not initialized (e.g., Package, translations etc.) I'm afraid we can't do much testing until the above is solved. My attempt thus far can be found at: git://git.lyx.org/developers/vfr/lyx.git That is not really kosher, but at least a simple way to get started directly. It is done by use of a simple preprocessor directive: #public_on_testing Once a wall of tests is created to ensure the behaviour of the given API, more skilled approaches can be introduced. Ideally, one would not need to care about private variables because we are only interested in that the public interface does what it is supposed to do. Right ? Vincent
Re: Unit testing: The Small Plan
On Tue, May 7, 2013 at 9:20 PM, Pavel Sandawrote: > Elmar Hinz wrote: > > If somebody can give improvements to the plan, it's welcome. > > I guess people will let you do almost anything what you like in test/* > but they will become much more picky when it comes to changes in src/. > Perhaps the best way is to try example, post patch here and > and look what people think. > > Pavel > Hi Pavel, as long as people don't become picky it's a good sign. They even shouldn't become picky, during big refactoring of the sources. It works in two steps. In the first step you write tests, that prove the existing API is fully functioning. Then you start with new functions or other kind of refactoring. The test bell should immediately ring when something is broken, long before people get picky at all. The drawback may be that people don't notice that you are working until you come out with the new feature. Fotunatly the world is never that perfect. \Elmar -- Elmar Hinz Freiherr-vom-Stein-Str. 1 33014 Bad Driburg TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m
Re: Unit testing: The Small Plan
> I would like to see some examples of mocking and injection. > > Thank you, Vincent, > I tried to write some tests using the google framework, and started with > the Buffer class. This immediately gives you the problem that it is > dependent on a large number of other classes. So, this would mean that you > have to fake a large part of the LyX codebase. > Well yes, actually I would start with the smallest class and do the Buffer class as the last one. I guess it requires a lot of experiance to tame that. > Then I tried to link against all libraries of LyX, but then I ran into > problems that the application was not initialized (e.g., Package, > translations etc.) > > I'm afraid we can't do much testing until the above is solved. > So a few tests already brought up a vision of what need to be solved. That's exactly their most important strength. They are a perfect teacher. Here we find out how everything currently depends on everything. We couldn't even take a little piece out, to use it as a library for something completly different. Regards \Elmar -- Elmar Hinz Freiherr-vom-Stein-Str. 1 33014 Bad Driburg TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m
Re: Unit testing: The Small Plan
> Ideally, one would not need to care about private variables because we are only interested in that the public interface does what it is supposed to do. Right ? Yes, at least as far as it concerns the testing. Denpendency Injection has other aspects. As an example, if there is a class that prints to stdout you mayby would not test that, taking it as an "interal" behaviour. With DI you inject the output stream, with the result that you can use different printers in future. Lyx already does this in many caes. Exactly that is dependancy injection. Now you could drop in a MockStream to prove that the class uses the stream object like expacted. Regards \Elmar -- Elmar Hinz Freiherr-vom-Stein-Str. 1 33014 Bad Driburg TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m
Re: Unit testing: The Small Plan
Hi Elmar, I think your plan covers the question "HOW do we want to unit test the software" well. However, we have not thought much about the "WHAT do we want to test?" question. Essentially, we need to think about which classes/functions to test first. I think it is not realistic to aim for a high test coverage with software as large as LyX. Unit tests make sense in certain cases and less in others. So we should identify these use cases first, and start with a few selective unit tests. We can then grow them as we see fit. For user-driven actions (LFUNs), LyX already has a test framework. However, these tests do not cover internals of LyX. Which internals do not require a complex sequence of actions (or a complex internal state) to be tested? (Those that do are maybe better covered with other types of tests.) Where has the code changed often in the past, and is expected to change often in the future? What kinds of (internal) functions are often covered by bug reports and thus warrant better test automation? I hope some of the veteran developers can help answer these questions. Elmar Hinz wrote: Hello list, I'd like to come up with a small plan for getting started with unit testing: == 1.) Directory structure: "tests/unit/" == * Unit tests stay their own directory separated from src/. * Below "tests/unit" the directory structure mirrors the structure of "src/". * The reason for the subfolder "unit" is, that unit tests are not the only type of tests. Unit tests test the single units of the source. == 2.) Mocking: googlemock == This weekend I researched for alternatives, but there was none that made a stable impression to me apart from googlemock. With C++ mocking can take two approaches: 2.a) Turning classes into Templates, to be able to replace members by mocks. http://programmaticallyspeaking.blogspot.de/2010/04/beautiful-dependency-injection-in-c.html This seems to invasive into the existing sources for me. 2.b) Inheriting the mocks from the real objects, so that they are of the same type. This is the approach of googlemock. Googlemock can be used with any mocking framework. It best integrates with googletest. == 3.) Testing framework: any == Unit tests are independent units, so any framework or none can be used. For the reason of mocks, googletest is recommended. == 4.) Runnig all tests == To be able to run all tests at once they need to be detected. * The naming pattern of the binary to support this: whateverTest * A successful test binary returns 0, the others an Error. * The global test detector and runner is a Python script. == 5.) Dependency injection == Dependency injection (the mock objects) is more difficult with C++ for multiple reasons like the missing garbage collection or reflection. The approach that looks most forward and less intrusive for the sources to me, is to directly access the private members by making them public during testing and only during testing. That is not really kosher, but at least a simple way to get started directly. It is done by use of a simple preprocessor directive: #public_on_testing Once a wall of tests is created to ensure the behaviour of the given API, more skilled approaches can be introduced. \Elmar -- Elmar Hinz Freiherr-vom-Stein-Str. 1 33014 Bad Driburg TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m -- Regards, Cyrille Artho - http://artho.com/ Love is the delusion that one woman differs from another. -- H.L. Mencken