Re: Unit testing

[Abdelrazak Younes]

On 03/11/2015 14:28, Jean-Marc Lasgouttes wrote:

Le 02/11/2015 21:36, Vincent van Ravesteijn a écrit :

Dear all,

I have prepared a unit test framework based on google-test (gtest). You
can see the commits at
http://git.lyx.org/?p=developers/vfr/lyx.git;a=shortlog;h=refs/heads/tests. 


It includes gtest-1.7.0 (permitted by the license).


Hi Vincent,

What is the difference with the tests we current use (I mean the "make 
check" sort). The same but with better interface?


gtest is about unit testing. Current test framework from Georg is more 
about Blackbox testing.


In my experience blackbox tests are much easier to create, deploy and 
maintain and they are testing real use case. Blackbox testing catches 
regressions which is very good. All developers should care about 
Blackbox testing and ideally you should not create a single feature 
without the associated test.


Unit tests are good for testing all possible corner cases, focusing on a 
single function or class. In general Unit testing catches corner cases 
which are difficult to find with common use cases. The downside is that 
lazy developpers don't care about unit testing as it takes in general 
twice as much time or more to create the unit test than the time to 
create the function.


Both are good to have IMO and gtest is a good framework.

my 2 cents :-)

Abdel

Re: Unit testing

[Abdelrazak Younes]

On 04/11/2015 14:30, Vincent van Ravesteijn wrote:

However, I would also count as advantage, if there is someone familiar with
the framework and willing to do the setup and lend a helping hand to others.
(Maybe, the toolkit with the most developers willing to work on should win.)


We have discussed this topic several times, and no one ever stood up
with a strong
preference for, or claimed to have a lot of experience with one of the
frameworks.


I have some experience with gtest, as I said, a good framework, easy to 
install and maintain.


Abdel.

Re: Unit testing

[Vincent van Ravesteijn]

> If there is no known advantage of gtest, I suggest using one of
> boost::test or QTest, as these come from sources we already rely on.
>

The major advantage of "gtest" is that I only have to type "git push"
and we have the
framework up and running.

A downside of boost::test is, according someone on the internet, that
it heavily relies
on templates. This makes it slightly more difficult to debug, and
maybe slightly more
advanced usage.

All living souls on the planet already rely on Google anyhow... ;)

> However, I would also count as advantage, if there is someone familiar with
> the framework and willing to do the setup and lend a helping hand to others.
> (Maybe, the toolkit with the most developers willing to work on should win.)
>

We have discussed this topic several times, and no one ever stood up
with a strong
preference for, or claimed to have a lot of experience with one of the
frameworks.

Vincent

Re: Unit testing

[Guenter Milde]

On 2015-11-03, Georg Baum wrote:
> Vincent van Ravesteijn wrote:
>> On Tue, Nov 3, 2015 at 2:28 PM, Jean-Marc Lasgouttes 
>> wrote:

> Regarding the test framework there is also boost::test (which would fit our 
> needs as well). Unfortunately I have neither experience with gtest nor with 
> QTest, so I don't know how it compares to these frameworks. Anyway, I think 
> the most important thing is to have a working setup, so I don't care much 
> which framework is used in the end.


If there is no known advantage of gtest, I suggest using one of
boost::test or QTest, as these come from sources we already rely on.

However, I would also count as advantage, if there is someone familiar with
the framework and willing to do the setup and lend a helping hand to others.
(Maybe, the toolkit with the most developers willing to work on should win.)

Günter

Re: Unit testing

[Jean-Marc Lasgouttes]

Le 02/11/2015 21:36, Vincent van Ravesteijn a écrit :

Dear all,

I have prepared a unit test framework based on google-test (gtest). You
can see the commits at
http://git.lyx.org/?p=developers/vfr/lyx.git;a=shortlog;h=refs/heads/tests.
It includes gtest-1.7.0 (permitted by the license).


Hi Vincent,

What is the difference with the tests we current use (I mean the "make 
check" sort). The same but with better interface?


JMarc

Re: Unit testing

[Vincent van Ravesteijn]

>> Any comments ? Shall I proceed to push this to master ?
>>
>> Vincent
>>
>
> Having a unit test framework integrated is a very good idea!
>
> Why have you chosen gtest and not QTest? Does gtest has interesting
> features which QTest does not have?

I'm not very experienced with unit testing, so I cannot tell any
details. Someone proposed gtest a long time ago on the list, and I've
seen it being used in other projects. So, it must be good enough for
us.

>
> One benefit of QTest is that it ships with the Qt installation and
> no code has to be added to the repository.

True. Most of the added code is actually not used
(tests,samples,unused autotools/MacOs/msvc code, etc.).

>
> But knowing gtest (also integrating into a existing project)
> is also good for other non-Qt projects.

Yes, as I said, I've seen it around, more than QTest (no, I don't have
usage statistics).

Vincent

Re: Unit testing

[Peter Kümmel]

Am 02.11.2015 um 21:36 schrieb Vincent van Ravesteijn:

Dear all,

I have prepared a unit test framework based on google-test (gtest). You can see 
the commits at
http://git.lyx.org/?p=developers/vfr/lyx.git;a=shortlog;h=refs/heads/tests. It 
includes gtest-1.7.0 (permitted by the
license).

Unit testing can be enabled by running CMake with -DLYX_ENABLE_UNIT_TESTS. This 
will make a target gtest-main and
unit-tests. It also create LyX.lib against which the unit testing executable is 
linked so that the functionality in the
core can be tested.

I have created a first (example) test case in tests/src/Buffer.cpp, called 
"BufferTest", which has three tests and which
is supposed to test the functionality of "lyx::Buffer".

A CMake target "unit-tests" now builds an executable running all unit-tests and 
gives an output as below. For each test
case, there is also automatically a CTest created. This test runs "unit-tests 
--gtest_filter=BufferTest" and gives a
summary. For more details, it is adviced to run the unit-tests executable 
directly (or improve CMake to capture the
output somehow).

Any comments ? Shall I proceed to push this to master ?

Vincent



Having a unit test framework integrated is a very good idea!

Why have you chosen gtest and not QTest? Does gtest has interesting
features which QTest does not have?

One benefit of QTest is that it ships with the Qt installation and
no code has to be added to the repository.

But knowing gtest (also integrating into a existing project)
is also good for other non-Qt projects.

Peter


#ctest -N
Test project C:/Users/Vincent/Documents/LyX/build/lyx/tests
   Test #1: unit_test/src/BufferTest

Total Tests: 1

#ctest -R
Test project C:/Users/Vincent/Documents/LyX/build/lyx/tests
 Start 1: unit_test/src/BufferTest
1/1 Test #1: unit_test/src/BufferTest .***Failed0.27 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.28 sec

The following tests FAILED:
   1 - unit_test/src/BufferTest (Failed)
Errors while running CTest

#unit-tests
Running main() from gtest_main.cc
[==] Running 3 tests from 1 test case.
[--] Global test environment set-up.
[--] 3 tests from BufferTest
[ RUN  ] BufferTest.NonExistingFile
[   OK ] BufferTest.NonExistingFile (0 ms)
[ RUN  ] BufferTest.NonExistingFileOperations
[   OK ] BufferTest.NonExistingFileOperations (0 ms)
[ RUN  ] BufferTest.UnicodeCharacters
..\..\..\source\gitlyx\tests\src\Buffer.cpp(45): error: Value of: string("C:/tés
t.lyx")
   Actual: "C:/t\xA8\xA6st.lyx"
Expected: fn1.absFileName()
Which is: "C:/t\xEF\xBF\xBD\xEF\xBF\xBDst.lyx"
[  FAILED  ] BufferTest.UnicodeCharacters (0 ms)
[--] 3 tests from BufferTest (0 ms total)

[--] Global test environment tear-down
[==] 3 tests from 1 test case ran. (0 ms total)
[  PASSED  ] 2 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] BufferTest.UnicodeCharacters

  1 FAILED TEST

Re: Unit testing

[Georg Baum]

Vincent van Ravesteijn wrote:

> On Tue, Nov 3, 2015 at 2:28 PM, Jean-Marc Lasgouttes 
> wrote:
>>
>> Hi Vincent,
>>
>> What is the difference with the tests we current use (I mean the "make
>> check" sort). The same but with better interface?

The overhead to add a test with the current setup is far too high. Using 
unit tests massively does only work if you do not need to do much more than 
adding your test function somewhere.


> The current tests are all different executables, which is not really
> optimal for speed in compilation and execution and has some overhead.
> Now they all appear as different targets here and there. Using a test
> framework also gives you a bunch of tools for writing tests (automatic
> preparation and cleanup of objects, mocking, etc.). Then it gives you
> the statistics and the ability to easily select tests you want to run,
> etc.
> 
> Maybe, more importantly, having a proper framework may ignite the
> spirit of creating tests, which then might become a new habit.

I hope so;-) Thank you very much for working on the tests.

Regarding the test framework there is also boost::test (which would fit our 
needs as well). Unfortunately I have neither experience with gtest nor with 
QTest, so I don't know how it compares to these frameworks. Anyway, I think 
the most important thing is to have a working setup, so I don't care much 
which framework is used in the end.


Georg

Re: Unit testing

[Vincent van Ravesteijn]

On Tue, Nov 3, 2015 at 2:28 PM, Jean-Marc Lasgouttes  wrote:
> Le 02/11/2015 21:36, Vincent van Ravesteijn a écrit :
>>
>> Dear all,
>>
>> I have prepared a unit test framework based on google-test (gtest). You
>> can see the commits at
>>
>> http://git.lyx.org/?p=developers/vfr/lyx.git;a=shortlog;h=refs/heads/tests.
>> It includes gtest-1.7.0 (permitted by the license).
>
>
> Hi Vincent,
>
> What is the difference with the tests we current use (I mean the "make
> check" sort). The same but with better interface?
>
> JMarc
>

The current tests are all different executables, which is not really
optimal for speed in compilation and execution and has some overhead.
Now they all appear as different targets here and there. Using a test
framework also gives you a bunch of tools for writing tests (automatic
preparation and cleanup of objects, mocking, etc.). Then it gives you
the statistics and the ability to easily select tests you want to run,
etc.

Maybe, more importantly, having a proper framework may ignite the
spirit of creating tests, which then might become a new habit.

Vincent

Unit testing

[Vincent van Ravesteijn]

Dear all,

I have prepared a unit test framework based on google-test (gtest). You 
can see the commits at 
http://git.lyx.org/?p=developers/vfr/lyx.git;a=shortlog;h=refs/heads/tests. 
It includes gtest-1.7.0 (permitted by the license).


Unit testing can be enabled by running CMake with 
-DLYX_ENABLE_UNIT_TESTS. This will make a target gtest-main and 
unit-tests. It also create LyX.lib against which the unit testing 
executable is linked so that the functionality in the core can be tested.


I have created a first (example) test case in tests/src/Buffer.cpp, 
called "BufferTest", which has three tests and which is supposed to test 
the functionality of "lyx::Buffer".


A CMake target "unit-tests" now builds an executable running all 
unit-tests and gives an output as below. For each test case, there is 
also automatically a CTest created. This test runs "unit-tests 
--gtest_filter=BufferTest" and gives a summary. For more details, it is 
adviced to run the unit-tests executable directly (or improve CMake to 
capture the output somehow).


Any comments ? Shall I proceed to push this to master ?

Vincent

#ctest -N
Test project C:/Users/Vincent/Documents/LyX/build/lyx/tests
  Test #1: unit_test/src/BufferTest

Total Tests: 1

#ctest -R
Test project C:/Users/Vincent/Documents/LyX/build/lyx/tests
Start 1: unit_test/src/BufferTest
1/1 Test #1: unit_test/src/BufferTest .***Failed0.27 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.28 sec

The following tests FAILED:
  1 - unit_test/src/BufferTest (Failed)
Errors while running CTest

#unit-tests
Running main() from gtest_main.cc
[==] Running 3 tests from 1 test case.
[--] Global test environment set-up.
[--] 3 tests from BufferTest
[ RUN  ] BufferTest.NonExistingFile
[   OK ] BufferTest.NonExistingFile (0 ms)
[ RUN  ] BufferTest.NonExistingFileOperations
[   OK ] BufferTest.NonExistingFileOperations (0 ms)
[ RUN  ] BufferTest.UnicodeCharacters
..\..\..\source\gitlyx\tests\src\Buffer.cpp(45): error: Value of: 
string("C:/tés

t.lyx")
  Actual: "C:/t\xA8\xA6st.lyx"
Expected: fn1.absFileName()
Which is: "C:/t\xEF\xBF\xBD\xEF\xBF\xBDst.lyx"
[  FAILED  ] BufferTest.UnicodeCharacters (0 ms)
[--] 3 tests from BufferTest (0 ms total)

[--] Global test environment tear-down
[==] 3 tests from 1 test case ran. (0 ms total)
[  PASSED  ] 2 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] BufferTest.UnicodeCharacters

 1 FAILED TEST

Re: LyX unit testing

[Vincent van Ravesteijn]

Op 19-9-2013 8:37, Hashini Senaratne schreef:

Hello Vincent,


But in general, I would like to introduce unit testing, yes.


Currently, is there any guide that I can follow, how the suggested testframe
work can be set to the LyX environment?

Thanks
Hashini



Hi Hashini,

I'm not sure what you're asking for.

If you want to write a test using the framework I set up, you can merge 
my branch.


git remote add vfr g...@git.lyx.org:developers/vfr/lyx.git,

 and then write tests like the one I did as an example: t0001_Buffer.cpp.

Hope this helps.

Vincent

Re: LyX unit testing

[Vincent van Ravesteijn]

Op 19-9-2013 8:37, Hashini Senaratne schreef:

Hello Vincent,


But in general, I would like to introduce unit testing, yes.


Currently, is there any guide that I can follow, how the suggested testframe
work can be set to the LyX environment?

Thanks
Hashini



Hi Hashini,

I'm not sure what you're asking for.

If you want to write a test using the framework I set up, you can merge 
my branch.


git remote add vfr g...@git.lyx.org:developers/vfr/lyx.git,

 and then write tests like the one I did as an example: t0001_Buffer.cpp.

Hope this helps.

Vincent

Re: LyX unit testing

[Hashini Senaratne]

Hello Vincent,

 But in general, I would like to introduce unit testing, yes.
 

Currently, is there any guide that I can follow, how the suggested testframe
work can be set to the LyX environment?

Thanks
Hashini

Re: LyX unit testing

[Hashini Senaratne]

Hello Vincent,

> But in general, I would like to introduce unit testing, yes.
> 

Currently, is there any guide that I can follow, how the suggested testframe
work can be set to the LyX environment?

Thanks
Hashini

Re: LyX unit testing

[Vincent van Ravesteijn]

Op 16-9-2013 8:16, Scott Kostyshak schreef:

Hi Vincent,

Do you have any plans to merge the gtest code
http://git.lyx.org/?p=developers/vfr/lyx.git;a=shortlog;h=refs/heads/tests
into master after the 2.1 release? Does it still need a lot of work
before it should be merged?

Hashini has expressed interest in implementing unit tests, and a
couple of questions have popped up on the list over the months so I
just wanted to get an idea of how feasible it is to have in LyX's main
repo.

Scott


Hi Scott,

I was interested in getting some feedback on whether we want to proceed 
with the google-test framework, or not ? Do we need a 3rd-party testing 
framework at all ?


Lastly, there are huge problems in implementing unit tests for existing 
pieces of code. This is because of the large number of dependencies 
between classes, the fact that a lot of functionalities are packed 
together in some large classe, etc.


But in general, I would like to introduce unit testing, yes.

Vincent

Vincent

Re: LyX unit testing

[Scott Kostyshak]

On Tue, Sep 17, 2013 at 3:33 AM, Vincent van Ravesteijn v...@lyx.org wrote:
 Op 16-9-2013 8:16, Scott Kostyshak schreef:

 Hi Vincent,

 Do you have any plans to merge the gtest code
 http://git.lyx.org/?p=developers/vfr/lyx.git;a=shortlog;h=refs/heads/tests
 into master after the 2.1 release? Does it still need a lot of work
 before it should be merged?

 Hashini has expressed interest in implementing unit tests, and a
 couple of questions have popped up on the list over the months so I
 just wanted to get an idea of how feasible it is to have in LyX's main
 repo.

 Scott


 Hi Scott,

 I was interested in getting some feedback on whether we want to proceed with
 the google-test framework, or not ? Do we need a 3rd-party testing framework
 at all ?

I have only researched the google-test framework briefly but it looks
nice. I don't write much of the core code though so it would be nice
to get feedback from others.

How much of a burden would introducing this framework be for the
developers that aren't interested in unit testing? Would their
workflow have to change or could they just ignore everything to do
with unit testing if they want?

 Lastly, there are huge problems in implementing unit tests for existing
 pieces of code. This is because of the large number of dependencies between
 classes, the fact that a lot of functionalities are packed together in some
 large classe, etc.

Sounds complicated.

 But in general, I would like to introduce unit testing, yes.

Great.

Thanks for the quick reply and explanations,

Scott

Re: LyX unit testing

[Vincent van Ravesteijn]

Op 16-9-2013 8:16, Scott Kostyshak schreef:

Hi Vincent,

Do you have any plans to merge the gtest code
http://git.lyx.org/?p=developers/vfr/lyx.git;a=shortlog;h=refs/heads/tests
into master after the 2.1 release? Does it still need a lot of work
before it should be merged?

Hashini has expressed interest in implementing unit tests, and a
couple of questions have popped up on the list over the months so I
just wanted to get an idea of how feasible it is to have in LyX's main
repo.

Scott


Hi Scott,

I was interested in getting some feedback on whether we want to proceed 
with the google-test framework, or not ? Do we need a 3rd-party testing 
framework at all ?


Lastly, there are huge problems in implementing unit tests for existing 
pieces of code. This is because of the large number of dependencies 
between classes, the fact that a lot of functionalities are packed 
together in some large classe, etc.


But in general, I would like to introduce unit testing, yes.

Vincent

Vincent

Re: LyX unit testing

[Scott Kostyshak]

On Tue, Sep 17, 2013 at 3:33 AM, Vincent van Ravesteijn <v...@lyx.org> wrote:
> Op 16-9-2013 8:16, Scott Kostyshak schreef:
>
>> Hi Vincent,
>>
>> Do you have any plans to merge the gtest code
>> http://git.lyx.org/?p=developers/vfr/lyx.git;a=shortlog;h=refs/heads/tests
>> into master after the 2.1 release? Does it still need a lot of work
>> before it should be merged?
>>
>> Hashini has expressed interest in implementing unit tests, and a
>> couple of questions have popped up on the list over the months so I
>> just wanted to get an idea of how feasible it is to have in LyX's main
>> repo.
>>
>> Scott
>
>
> Hi Scott,
>
> I was interested in getting some feedback on whether we want to proceed with
> the google-test framework, or not ? Do we need a 3rd-party testing framework
> at all ?

I have only researched the google-test framework briefly but it looks
nice. I don't write much of the core code though so it would be nice
to get feedback from others.

How much of a burden would introducing this framework be for the
developers that aren't interested in unit testing? Would their
workflow have to change or could they just ignore everything to do
with unit testing if they want?

> Lastly, there are huge problems in implementing unit tests for existing
> pieces of code. This is because of the large number of dependencies between
> classes, the fact that a lot of functionalities are packed together in some
> large classe, etc.

Sounds complicated.

> But in general, I would like to introduce unit testing, yes.

Great.

Thanks for the quick reply and explanations,

Scott

LyX unit testing

[Scott Kostyshak]

Hi Vincent,

Do you have any plans to merge the gtest code
http://git.lyx.org/?p=developers/vfr/lyx.git;a=shortlog;h=refs/heads/tests
into master after the 2.1 release? Does it still need a lot of work
before it should be merged?

Hashini has expressed interest in implementing unit tests, and a
couple of questions have popped up on the list over the months so I
just wanted to get an idea of how feasible it is to have in LyX's main
repo.

Scott

LyX unit testing

[Scott Kostyshak]

Hi Vincent,

Do you have any plans to merge the gtest code
http://git.lyx.org/?p=developers/vfr/lyx.git;a=shortlog;h=refs/heads/tests
into master after the 2.1 release? Does it still need a lot of work
before it should be merged?

Hashini has expressed interest in implementing unit tests, and a
couple of questions have popped up on the list over the months so I
just wanted to get an idea of how feasible it is to have in LyX's main
repo.

Scott

Re: Unit testing: The Small Plan

[Scott Kostyshak]

On Wed, May 8, 2013 at 2:59 AM, Elmar Hinz t3el...@googlemail.com wrote:

 To get started, I would need to do some more simple cases first.

Hi Elmar,

Did you make any progress on this? I know little about unit tests so I
won't be able to help much here, but I'm interested in any progress
you feel like sharing.

Scott

Re: Unit testing: The Small Plan

[Scott Kostyshak]

On Wed, May 8, 2013 at 2:59 AM, Elmar Hinz  wrote:

> To get started, I would need to do some more simple cases first.

Hi Elmar,

Did you make any progress on this? I know little about unit tests so I
won't be able to help much here, but I'm interested in any progress
you feel like sharing.

Scott

Re: Unit testing: The Small Plan

[Elmar Hinz]

Hello Cyrille,

On Wed, May 8, 2013 at 1:28 AM, Cyrille Artho c.ar...@aist.go.jp wrote:

 Hi Elmar,
 I think your plan covers the question HOW do we want to unit test the
 software well. However, we have not thought much about the WHAT do we
 want to test? question. Essentially, we need to think about which
 classes/functions to test first.

 I think it is not realistic to aim for a high test coverage with software
 as large as LyX. Unit tests make sense in certain cases and less in others.
 So we should identify these use cases first, and start with a few selective
 unit tests. We can then grow them as we see fit.


Yes, I share this vision of let the tests grow. That's the reason, why it's
only a small plan. A plan can't tell the people what they want to test.
It's the decision of the people themself, driven by different motivations.
A plan can only leverage.

There are at least two questions:

* What do I want to test?
* What can I test at all, at reasonable costs.

When I write new software in other languages, I write code and tests side
by side, resulting in a test coverage of approximately 100%, without extra
costs, as I need to run the code anyway.

Writing tests for existing software, is much more difficult. In the
beginning that are extra costs. They pay on the long run by fewer bugs. I
have to weight.


 For user-driven actions (LFUNs), LyX already has a test framework.
 However, these tests do not cover internals of LyX.


Right, unit testing is only one class of testing. That's why it has it's
own path tests/unit.

Gui testing needs completely different approaches and more support by
testing framewoks resulting in tests/gui or similar.

Probably there are also tools to cover memory leaks or performance.


 Which internals do not require a complex sequence of actions (or a complex
 internal state) to be tested? (Those that do are maybe better covered with
 other types of tests.) Where has the code changed often in the past, and is
 expected to change often in the future? What kinds of (internal) functions
 are often covered by bug reports and thus warrant better test automation?


Likely the most error-prone corners are the haredest to tests, because of
the same reasons.

To get started, I would need to do some more simple cases first.



 I hope some of the veteran developers can help answer these questions.


\Elmar

-- 
Elmar Hinz
Freiherr-vom-Stein-Str. 1
33014 Bad Driburg

TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m
personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m

Re: Unit testing: The Small Plan

[Elmar Hinz]

Hello Cyrille,

On Wed, May 8, 2013 at 1:28 AM, Cyrille Artho <c.ar...@aist.go.jp> wrote:

> Hi Elmar,
> I think your plan covers the question "HOW do we want to unit test the
> software" well. However, we have not thought much about the "WHAT do we
> want to test?" question. Essentially, we need to think about which
> classes/functions to test first.
>
> I think it is not realistic to aim for a high test coverage with software
> as large as LyX. Unit tests make sense in certain cases and less in others.
> So we should identify these use cases first, and start with a few selective
> unit tests. We can then grow them as we see fit.
>
>
Yes, I share this vision of let the tests grow. That's the reason, why it's
only a small plan. A plan can't tell the people what they want to test.
It's the decision of the people themself, driven by different motivations.
A plan can only leverage.

There are at least two questions:

* What do I want to test?
* What can I test at all, at reasonable costs.

When I write new software in other languages, I write code and tests side
by side, resulting in a test coverage of approximately 100%, without extra
costs, as I need to run the code anyway.

Writing tests for existing software, is much more difficult. In the
beginning that are extra costs. They pay on the long run by fewer bugs. I
have to weight.


> For user-driven actions (LFUNs), LyX already has a test framework.
> However, these tests do not cover internals of LyX.
>
>
Right, unit testing is only one class of testing. That's why it has it's
own path "tests/unit".

Gui testing needs completely different approaches and more support by
testing framewoks resulting in "tests/gui" or similar.

Probably there are also tools to cover memory leaks or performance.


> Which internals do not require a complex sequence of actions (or a complex
> internal state) to be tested? (Those that do are maybe better covered with
> other types of tests.) Where has the code changed often in the past, and is
> expected to change often in the future? What kinds of (internal) functions
> are often covered by bug reports and thus warrant better test automation?
>

Likely the most error-prone corners are the haredest to tests, because of
the same reasons.

To get started, I would need to do some more simple cases first.


>
> I hope some of the veteran developers can help answer these questions.
>
>
\Elmar

-- 
Elmar Hinz
Freiherr-vom-Stein-Str. 1
33014 Bad Driburg

TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m
personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m

Unit testing: The Small Plan

[Elmar Hinz]

Hello list,

I'd like to come up with a small plan for getting started with unit testing:

==
1.) Directory structure: tests/unit/
==

* Unit tests stay their own directory separated from src/.
* Below tests/unit the directory structure mirrors the structure of
src/.
* The reason for the subfolder unit is, that unit tests are not the
only type of tests. Unit tests test the single units of the source.

==
2.) Mocking: googlemock
==

This weekend I researched for alternatives, but there was none
that made a stable impression to me apart from googlemock.

With C++ mocking can take two approaches:

2.a) Turning classes into Templates, to be able to replace members by mocks.

http://programmaticallyspeaking.blogspot.de/2010/04/beautiful-dependency-injection-in-c.html

This seems to invasive into the existing sources for me.

2.b) Inheriting the mocks from the real objects, so that they are of the
same type.

This is the approach of googlemock.

Googlemock can be used with any mocking framework. It best integrates with
googletest.

==
3.) Testing framework: any
==

Unit tests are independent units, so any framework or none can be used.
For the reason of mocks, googletest is recommended.

==
4.) Runnig all tests
==

To be able to run all tests at once they need to be detected.

* The naming pattern of the binary to support this: whateverTest
* A successful test binary returns 0, the others an Error.
* The global test detector and runner is a Python script.

==
5.) Dependency injection
==

Dependency injection (the mock objects) is more difficult with C++
for multiple reasons like the missing garbage collection or reflection.

The approach that looks most forward and less intrusive for the sources
to me, is to directly access the private members by making them public
during testing and only during testing.

That is not really kosher, but at least a simple way to get started
directly.

It is done by use of a simple preprocessor directive:

#public_on_testing

Once a wall of tests is created to ensure the behaviour of the given API,
more skilled approaches can be introduced.

\Elmar

-- 
Elmar Hinz
Freiherr-vom-Stein-Str. 1
33014 Bad Driburg

TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m
personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m

Re: Unit testing: The Small Plan

[Richard Heck]

On 05/07/2013 04:57 AM, Elmar Hinz wrote:

Hello list,

I'd like to come up with a small plan for getting started with unit 
testing:


I am a total ignoramus when it comes to unit testing, so I will leave it 
to others

who actually know something to express a view.

Richard

Re: Unit testing: The Small Plan

[Pavel Sanda]

Elmar Hinz wrote:
 If somebody can give improvements to the plan, it's welcome.

I guess people will let you do almost anything what you like in test/*
but they will become much more picky when it comes to changes in src/.
Perhaps the best way is to try example, post patch here and
and look what people think.

Pavel

Re: Unit testing: The Small Plan

[Vincent van Ravesteijn]

Op 7-5-2013 10:57, Elmar Hinz schreef:

Hello list,

I'd like to come up with a small plan for getting started with unit 
testing:


I would like to see some examples of mocking  and injection.

I tried to write some tests using the google framework, and started with 
the Buffer class. This immediately gives you the problem that it is 
dependent on a large number of other classes. So, this would mean that 
you have to fake a large part of the LyX codebase.


Then I tried to link against all libraries of LyX, but then I ran into 
problems that the application was not initialized (e.g., Package, 
translations etc.)


I'm afraid we can't do much testing until the above is solved.

My attempt thus far can be found at: 
git://git.lyx.org/developers/vfr/lyx.git


That is not really kosher, but at least a simple way to get started 
directly.


It is done by use of a simple preprocessor directive:

#public_on_testing

Once a wall of tests is created to ensure the behaviour of the given API,
more skilled approaches can be introduced.


Ideally, one would not need to care about private variables because we 
are only interested in that the public interface does what it is 
supposed to do. Right ?


Vincent

Re: Unit testing: The Small Plan

[Elmar Hinz]

On Tue, May 7, 2013 at 9:20 PM, Pavel Sanda sa...@lyx.org wrote:

 Elmar Hinz wrote:
  If somebody can give improvements to the plan, it's welcome.

 I guess people will let you do almost anything what you like in test/*
 but they will become much more picky when it comes to changes in src/.
 Perhaps the best way is to try example, post patch here and
 and look what people think.

 Pavel


Hi Pavel,

as long as people don't become picky it's a good sign.
They even shouldn't become picky, during big refactoring of the sources.

It works in two steps. In the first step you write tests, that prove the
existing API is fully functioning.

Then you start with new functions or other kind of refactoring.  The test
bell should immediately ring
when something is broken, long before people get picky at all.

The drawback may be that people don't notice that you are working until you
come out with the new feature.

Fotunatly the world is never that perfect.

\Elmar

-- 
Elmar Hinz
Freiherr-vom-Stein-Str. 1
33014 Bad Driburg

TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m
personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m

Re: Unit testing: The Small Plan

[Elmar Hinz]

 I would like to see some examples of mocking  and injection.


Thank you, Vincent,


 I tried to write some tests using the google framework, and started with
 the Buffer class. This immediately gives you the problem that it is
 dependent on a large number of other classes. So, this would mean that you
 have to fake a large part of the LyX codebase.


Well yes, actually I would start with the smallest class and do the Buffer
class as the last one. I guess it requires a lot of experiance to tame that.


 Then I tried to link against all libraries of LyX, but then I ran into
 problems that the application was not initialized (e.g., Package,
 translations etc.)

 I'm afraid we can't do much testing until the above is solved.


So a few tests already brought up a vision of what need to be solved.
That's exactly their most important strength. They are a perfect teacher.

Here we find out how everything currently depends on everything. We
couldn't even take a little piece out, to use it as a library for something
completly different.

Regards

\Elmar


-- 
Elmar Hinz
Freiherr-vom-Stein-Str. 1
33014 Bad Driburg

TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m
personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m

Re: Unit testing: The Small Plan

[Elmar Hinz]

 Ideally, one would not need to care about private variables because we
are only interested in that the public interface does what it is supposed
to do. Right ?

Yes, at least as far as it concerns the testing.

Denpendency Injection has other aspects.

As an example, if there is a class that prints to stdout you mayby would
not test that, taking it as an interal behaviour.
With DI you inject the output stream, with the result that you can use
different printers in future.

Lyx already does this in many caes. Exactly that is dependancy injection.

Now you could drop in a MockStream to prove that the class uses the stream
object like expacted.

Regards

\Elmar


-- 
Elmar Hinz
Freiherr-vom-Stein-Str. 1
33014 Bad Driburg

TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m
personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m

Re: Unit testing: The Small Plan

[Cyrille Artho]

Hi Elmar,
I think your plan covers the question HOW do we want to unit test the 
software well. However, we have not thought much about the WHAT do we 
want to test? question. Essentially, we need to think about which 
classes/functions to test first.


I think it is not realistic to aim for a high test coverage with software 
as large as LyX. Unit tests make sense in certain cases and less in others. 
So we should identify these use cases first, and start with a few selective 
unit tests. We can then grow them as we see fit.


For user-driven actions (LFUNs), LyX already has a test framework. However, 
these tests do not cover internals of LyX.


Which internals do not require a complex sequence of actions (or a complex 
internal state) to be tested? (Those that do are maybe better covered with 
other types of tests.) Where has the code changed often in the past, and is 
expected to change often in the future? What kinds of (internal) functions 
are often covered by bug reports and thus warrant better test automation?


I hope some of the veteran developers can help answer these questions.

Elmar Hinz wrote:

Hello list,

I'd like to come up with a small plan for getting started with unit testing:

==
1.) Directory structure: tests/unit/
==

* Unit tests stay their own directory separated from src/.
* Below tests/unit the directory structure mirrors the structure of src/.
* The reason for the subfolder unit is, that unit tests are not the
only type of tests. Unit tests test the single units of the source.

==
2.) Mocking: googlemock
==

This weekend I researched for alternatives, but there was none
that made a stable impression to me apart from googlemock.

With C++ mocking can take two approaches:

2.a) Turning classes into Templates, to be able to replace members by mocks.

http://programmaticallyspeaking.blogspot.de/2010/04/beautiful-dependency-injection-in-c.html

This seems to invasive into the existing sources for me.

2.b) Inheriting the mocks from the real objects, so that they are of the
same type.

This is the approach of googlemock.

Googlemock can be used with any mocking framework. It best integrates with
googletest.

==
3.) Testing framework: any
==

Unit tests are independent units, so any framework or none can be used.
For the reason of mocks, googletest is recommended.

==
4.) Runnig all tests
==

To be able to run all tests at once they need to be detected.

* The naming pattern of the binary to support this: whateverTest
* A successful test binary returns 0, the others an Error.
* The global test detector and runner is a Python script.

==
5.) Dependency injection
==

Dependency injection (the mock objects) is more difficult with C++
for multiple reasons like the missing garbage collection or reflection.

The approach that looks most forward and less intrusive for the sources
to me, is to directly access the private members by making them public
during testing and only during testing.

That is not really kosher, but at least a simple way to get started directly.

It is done by use of a simple preprocessor directive:

#public_on_testing

Once a wall of tests is created to ensure the behaviour of the given API,
more skilled approaches can be introduced.

\Elmar

--
Elmar Hinz
Freiherr-vom-Stein-Str. 1
33014 Bad Driburg

TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m
personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m



--
Regards,
Cyrille Artho - http://artho.com/
Love is the delusion that one woman differs from another.
-- H.L. Mencken

Unit testing: The Small Plan

[Elmar Hinz]

Hello list,

I'd like to come up with a small plan for getting started with unit testing:

==
1.) Directory structure: "tests/unit/"
==

* Unit tests stay their own directory separated from src/.
* Below "tests/unit" the directory structure mirrors the structure of
"src/".
* The reason for the subfolder "unit" is, that unit tests are not the
only type of tests. Unit tests test the single units of the source.

==
2.) Mocking: googlemock
==

This weekend I researched for alternatives, but there was none
that made a stable impression to me apart from googlemock.

With C++ mocking can take two approaches:

2.a) Turning classes into Templates, to be able to replace members by mocks.

http://programmaticallyspeaking.blogspot.de/2010/04/beautiful-dependency-injection-in-c.html

This seems to invasive into the existing sources for me.

2.b) Inheriting the mocks from the real objects, so that they are of the
same type.

This is the approach of googlemock.

Googlemock can be used with any mocking framework. It best integrates with
googletest.

==
3.) Testing framework: any
==

Unit tests are independent units, so any framework or none can be used.
For the reason of mocks, googletest is recommended.

==
4.) Runnig all tests
==

To be able to run all tests at once they need to be detected.

* The naming pattern of the binary to support this: whateverTest
* A successful test binary returns 0, the others an Error.
* The global test detector and runner is a Python script.

==
5.) Dependency injection
==

Dependency injection (the mock objects) is more difficult with C++
for multiple reasons like the missing garbage collection or reflection.

The approach that looks most forward and less intrusive for the sources
to me, is to directly access the private members by making them public
during testing and only during testing.

That is not really kosher, but at least a simple way to get started
directly.

It is done by use of a simple preprocessor directive:

#public_on_testing

Once a wall of tests is created to ensure the behaviour of the given API,
more skilled approaches can be introduced.

\Elmar

-- 
Elmar Hinz
Freiherr-vom-Stein-Str. 1
33014 Bad Driburg

TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m
personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m

Re: Unit testing: The Small Plan

[Richard Heck]

On 05/07/2013 04:57 AM, Elmar Hinz wrote:

Hello list,

I'd like to come up with a small plan for getting started with unit 
testing:


I am a total ignoramus when it comes to unit testing, so I will leave it 
to others

who actually know something to express a view.

Richard

Re: Unit testing: The Small Plan

[Pavel Sanda]

Elmar Hinz wrote:
> If somebody can give improvements to the plan, it's welcome.

I guess people will let you do almost anything what you like in test/*
but they will become much more picky when it comes to changes in src/.
Perhaps the best way is to try example, post patch here and
and look what people think.

Pavel

Re: Unit testing: The Small Plan

[Vincent van Ravesteijn]

Op 7-5-2013 10:57, Elmar Hinz schreef:

Hello list,

I'd like to come up with a small plan for getting started with unit 
testing:


I would like to see some examples of mocking  and injection.

I tried to write some tests using the google framework, and started with 
the Buffer class. This immediately gives you the problem that it is 
dependent on a large number of other classes. So, this would mean that 
you have to fake a large part of the LyX codebase.


Then I tried to link against all libraries of LyX, but then I ran into 
problems that the application was not initialized (e.g., Package, 
translations etc.)


I'm afraid we can't do much testing until the above is solved.

My attempt thus far can be found at: 
git://git.lyx.org/developers/vfr/lyx.git


That is not really kosher, but at least a simple way to get started 
directly.


It is done by use of a simple preprocessor directive:

#public_on_testing

Once a wall of tests is created to ensure the behaviour of the given API,
more skilled approaches can be introduced.


Ideally, one would not need to care about private variables because we 
are only interested in that the public interface does what it is 
supposed to do. Right ?


Vincent

Re: Unit testing: The Small Plan

[Elmar Hinz]

On Tue, May 7, 2013 at 9:20 PM, Pavel Sanda  wrote:

> Elmar Hinz wrote:
> > If somebody can give improvements to the plan, it's welcome.
>
> I guess people will let you do almost anything what you like in test/*
> but they will become much more picky when it comes to changes in src/.
> Perhaps the best way is to try example, post patch here and
> and look what people think.
>
> Pavel
>

Hi Pavel,

as long as people don't become picky it's a good sign.
They even shouldn't become picky, during big refactoring of the sources.

It works in two steps. In the first step you write tests, that prove the
existing API is fully functioning.

Then you start with new functions or other kind of refactoring.  The test
bell should immediately ring
when something is broken, long before people get picky at all.

The drawback may be that people don't notice that you are working until you
come out with the new feature.

Fotunatly the world is never that perfect.

\Elmar

-- 
Elmar Hinz
Freiherr-vom-Stein-Str. 1
33014 Bad Driburg

TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m
personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m

Re: Unit testing: The Small Plan

[Elmar Hinz]

> I would like to see some examples of mocking  and injection.
>
>
Thank you, Vincent,


> I tried to write some tests using the google framework, and started with
> the Buffer class. This immediately gives you the problem that it is
> dependent on a large number of other classes. So, this would mean that you
> have to fake a large part of the LyX codebase.
>

Well yes, actually I would start with the smallest class and do the Buffer
class as the last one. I guess it requires a lot of experiance to tame that.


> Then I tried to link against all libraries of LyX, but then I ran into
> problems that the application was not initialized (e.g., Package,
> translations etc.)
>
> I'm afraid we can't do much testing until the above is solved.
>

So a few tests already brought up a vision of what need to be solved.
That's exactly their most important strength. They are a perfect teacher.

Here we find out how everything currently depends on everything. We
couldn't even take a little piece out, to use it as a library for something
completly different.

Regards

\Elmar


-- 
Elmar Hinz
Freiherr-vom-Stein-Str. 1
33014 Bad Driburg

TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m
personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m

Re: Unit testing: The Small Plan

[Elmar Hinz]

> Ideally, one would not need to care about private variables because we
are only interested in that the public interface does what it is supposed
to do. Right ?

Yes, at least as far as it concerns the testing.

Denpendency Injection has other aspects.

As an example, if there is a class that prints to stdout you mayby would
not test that, taking it as an "interal" behaviour.
With DI you inject the output stream, with the result that you can use
different printers in future.

Lyx already does this in many caes. Exactly that is dependancy injection.

Now you could drop in a MockStream to prove that the class uses the stream
object like expacted.

Regards

\Elmar


-- 
Elmar Hinz
Freiherr-vom-Stein-Str. 1
33014 Bad Driburg

TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m
personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m

Re: Unit testing: The Small Plan

[Cyrille Artho]

Hi Elmar,
I think your plan covers the question "HOW do we want to unit test the 
software" well. However, we have not thought much about the "WHAT do we 
want to test?" question. Essentially, we need to think about which 
classes/functions to test first.


I think it is not realistic to aim for a high test coverage with software 
as large as LyX. Unit tests make sense in certain cases and less in others. 
So we should identify these use cases first, and start with a few selective 
unit tests. We can then grow them as we see fit.


For user-driven actions (LFUNs), LyX already has a test framework. However, 
these tests do not cover internals of LyX.


Which internals do not require a complex sequence of actions (or a complex 
internal state) to be tested? (Those that do are maybe better covered with 
other types of tests.) Where has the code changed often in the past, and is 
expected to change often in the future? What kinds of (internal) functions 
are often covered by bug reports and thus warrant better test automation?


I hope some of the veteran developers can help answer these questions.

Elmar Hinz wrote:

Hello list,

I'd like to come up with a small plan for getting started with unit testing:

==
1.) Directory structure: "tests/unit/"
==

* Unit tests stay their own directory separated from src/.
* Below "tests/unit" the directory structure mirrors the structure of "src/".
* The reason for the subfolder "unit" is, that unit tests are not the
only type of tests. Unit tests test the single units of the source.

==
2.) Mocking: googlemock
==

This weekend I researched for alternatives, but there was none
that made a stable impression to me apart from googlemock.

With C++ mocking can take two approaches:

2.a) Turning classes into Templates, to be able to replace members by mocks.

http://programmaticallyspeaking.blogspot.de/2010/04/beautiful-dependency-injection-in-c.html

This seems to invasive into the existing sources for me.

2.b) Inheriting the mocks from the real objects, so that they are of the
same type.

This is the approach of googlemock.

Googlemock can be used with any mocking framework. It best integrates with
googletest.

==
3.) Testing framework: any
==

Unit tests are independent units, so any framework or none can be used.
For the reason of mocks, googletest is recommended.

==
4.) Runnig all tests
==

To be able to run all tests at once they need to be detected.

* The naming pattern of the binary to support this: whateverTest
* A successful test binary returns 0, the others an Error.
* The global test detector and runner is a Python script.

==
5.) Dependency injection
==

Dependency injection (the mock objects) is more difficult with C++
for multiple reasons like the missing garbage collection or reflection.

The approach that looks most forward and less intrusive for the sources
to me, is to directly access the private members by making them public
during testing and only during testing.

That is not really kosher, but at least a simple way to get started directly.

It is done by use of a simple preprocessor directive:

#public_on_testing

Once a wall of tests is created to ensure the behaviour of the given API,
more skilled approaches can be introduced.

\Elmar

--
Elmar Hinz
Freiherr-vom-Stein-Str. 1
33014 Bad Driburg

TYPO3 community contact: t.3.e.l.m.a...@.g.m.a.i.l.dot.c.o.m
personal contact: e.l.m.a.r.dot.h.i.n...@.g.m.a.i.l.dot.c.o.m



--
Regards,
Cyrille Artho - http://artho.com/
Love is the delusion that one woman differs from another.
-- H.L. Mencken

Re: Unit-Testing

[Lars Gullik Bjønnes]

Allan Rae [EMAIL PROTECTED] writes:

| Okay so we have a situation where different implementations of std::string
| may return different sizes.

The ones that don't allow '\0' chars is bogus. The ones that only
count until the first '\0' is met is bogus.

|  I haven't got an STL reference handy but
| isn't there a string member that returns the length excluding any null
| characters?

No.

| Besides, the point of the test is to make sure that the
| string length is correct in our implementation of lyxstring not to check
| the STL supplied strings.

the test might be ok after methods that work in C-type strings, but
not as a general inveriant.

|  I admit I still haven't gotten around to
| checking out the new strand so I'm still assuming we have our own string
| class.  It's likely to be the end of the month before I have much time for
| LyX again.  I'm flatout keeping up with the mailing list.

We are waiting :-)

Lgb

Re: Unit-Testing

[Lars Gullik Bjønnes]

Allan Rae <[EMAIL PROTECTED]> writes:

| Okay so we have a situation where different implementations of std::string
| may return different sizes.

The ones that don't allow '\0' chars is bogus. The ones that only
count until the first '\0' is met is bogus.

|  I haven't got an STL reference handy but
| isn't there a string member that returns the length excluding any null
| characters?

No.

| Besides, the point of the test is to make sure that the
| string length is correct in our implementation of lyxstring not to check
| the STL supplied strings.

the test might be ok after methods that work in C-type strings, but
not as a general inveriant.

|  I admit I still haven't gotten around to
| checking out the new strand so I'm still assuming we have our own string
| class.  It's likely to be the end of the month before I have much time for
| LyX again.  I'm flatout keeping up with the mailing list.

We are waiting :-)

Lgb

Re: Unit-Testing

[Allan Rae]

On 10 Oct 1999, Lars Gullik Bjønnes wrote:

 Allan Rae [EMAIL PROTECTED] writes:
 
 | the new strand yet but I already have some doubts about the recent removal
 | of one of those tests.
 
 That test was completely bogus, since it demanded that a std::string
 cannont contain '\0', so the test had to go.

Okay so we have a situation where different implementations of std::string
may return different sizes.  I haven't got an STL reference handy but
isn't there a string member that returns the length excluding any null
characters?  Besides, the point of the test is to make sure that the
string length is correct in our implementation of lyxstring not to check
the STL supplied strings.  I admit I still haven't gotten around to
checking out the new strand so I'm still assuming we have our own string
class.  It's likely to be the end of the month before I have much time for
LyX again.  I'm flatout keeping up with the mailing list.

Allan. (ARRae)

Re: Unit-Testing

[Allan Rae]

On 10 Oct 1999, Lars Gullik Bjønnes wrote:

> Allan Rae <[EMAIL PROTECTED]> writes:
> 
> | the new strand yet but I already have some doubts about the recent removal
> | of one of those tests.
> 
> That test was completely bogus, since it demanded that a std::string
> cannont contain '\0', so the test had to go.

Okay so we have a situation where different implementations of std::string
may return different sizes.  I haven't got an STL reference handy but
isn't there a string member that returns the length excluding any null
characters?  Besides, the point of the test is to make sure that the
string length is correct in our implementation of lyxstring not to check
the STL supplied strings.  I admit I still haven't gotten around to
checking out the new strand so I'm still assuming we have our own string
class.  It's likely to be the end of the month before I have much time for
LyX again.  I'm flatout keeping up with the mailing list.

Allan. (ARRae)

Re: Unit-Testing

[Lars Gullik Bjønnes]

Allan Rae [EMAIL PROTECTED] writes:

| the new strand yet but I already have some doubts about the recent removal
| of one of those tests.

That test was completely bogus, since it demanded that a std::string
cannont contain '\0', so the test had to go.

Lgb

Re: Unit-Testing

[Lars Gullik Bjønnes]

Allan Rae <[EMAIL PROTECTED]> writes:

| the new strand yet but I already have some doubts about the recent removal
| of one of those tests.

That test was completely bogus, since it demanded that a std::string
cannont contain '\0', so the test had to go.

Lgb

Re: Unit-Testing

[John Weiss]

On Thu, Oct 07, 1999 at 07:19:27PM +0200, Lars Gullik Bjønnes wrote:
 Allan Rae [EMAIL PROTECTED] writes:
 
 | Unit-testing is certainly a good thing -- tests all those promises the
 | code makes and keeps Arndt happy about programming by contract since the
 | interface can't change without everybody finding out.
 
 Unit-testing and independant modules is easy...unfortunately most of
 the modules in LyX depends on a multitude of other mudules makeing
 unit-testing very cumbersome and a lot of additional work.

Agreed.  Unit testing a class in a vacuum can be highly cumbersome.
That's why you unit test the fundamental ones first, by themselves.
Unit tests of more complex classes simply use the fundamental ones and
assume that they are correct.

Also, you can't unit test *every* class.  Where one can test, however,
one should.


 modules in the support dir could easily benefit form unit
 testing...but it is a lot of work to create the tests...
 
 And tests can be wrong too...

True.  But once they're working, they save a great deal of time
finding and fixing future bugs...

This is one of those cases where the cost of the investment pays
itself back over time.

-- 
John Weiss

Re: Unit-Testing

[John Weiss]

On Thu, Oct 07, 1999 at 08:47:31PM +0200, Asger K. Alstrup Nielsen wrote:
 Instead, it's much better to simply address each file in order,
 document the header file, rename functions and variables to have 
 better names, in general clean it up, and try to make things better 
 small piece by piece.

I did not realize code cleanup was in such an imperfect state!  ;)

Seriously, I wasn't suggesting that we all drop what we're doing and
unit-test every class in the LyX source.  Some of us who can code but
have been daunted by the size of the source code, however, can build
unit tests.  Unit tests for cleaned-up classes and/or fundamental ones
should be feasable.

-- 
John Weiss

Re: Unit-Testing

[John Weiss]

On Thu, Oct 07, 1999 at 08:47:31PM +0200, Asger K. Alstrup Nielsen wrote:
 Instead, it's much better to simply address each file in order,
 document the header file, rename functions and variables to have 
 better names, in general clean it up, and try to make things better 
 small piece by piece.
 And keep in mind that everything should be stable all along.
 
 This, I believe, is the way to proceed with LyX.
 This, I believe, is also the best way to learn the LyX source code:

No, no it's not.  If this were, as you say, the best way to learn the
LyX source code, about 4-5 devlist members who know C++ (myself
included) could have learned everything about LyX over the past couple
of years *and* simultaneously completely cleaned up the code.  That
never happened.  I, for one, took one look at the LyX source in June
of 1998 and got a bd headache!  Hell --- I've wanted a
"transpose-char" function in LyX for 4 years now, and I *still* have
no clue where it would go or how to add it.

Picking a class that *has* been cleaned up and documented, however,
and writing a test to see that it works isn't nearly as difficult.  If
you had asked me to build a test for LString a year ago, I could have
done it.  The only reason I haven't done so on my own is time.  I
barely read my email anymore.

-- 
John Weiss

Re: Unit-Testing

[John Weiss]

On Thu, Oct 07, 1999 at 05:25:36PM +0100, Arnd Hanses wrote:

 Else we would found the 'HardCore testing LyX never come to terms
 gang', making Core Developers live impossible. 
 
 Then the real testing experts can by and by implement the rest on their
 own account and risk and report the results.

I only suggested that the author of a class not also author the unit
test for three reasons:  A) Lack of time; B) Conflict of interest; and
C) Lack of time.

In any case, unit tests aren't about making anybody's life
impossible. They're about putting each function through its paces,
making sure it does what the author intended, and also making sure it
handles bad input gracefully.  (Many a coredump was born when a
function met input it didn't expect...)

-- 
John Weiss

Re: Unit-Testing

[John Weiss]

Well, it seems that everyone thinks unit testing is a nice thing, but
nobody has the time to do it.

-- 
John Weiss

Re: Unit-Testing

[Allan Rae]

On Fri, 8 Oct 1999, John Weiss wrote:

 On Thu, Oct 07, 1999 at 07:19:27PM +0200, Lars Gullik Bjønnes wrote:
  Allan Rae [EMAIL PROTECTED] writes:
  
  | Unit-testing is certainly a good thing -- tests all those promises the
  | code makes and keeps Arndt happy about programming by contract since the
  | interface can't change without everybody finding out.
  
  Unit-testing and independant modules is easy...unfortunately most of
  the modules in LyX depends on a multitude of other mudules makeing
  unit-testing very cumbersome and a lot of additional work.
 
 Agreed.  Unit testing a class in a vacuum can be highly cumbersome.
 That's why you unit test the fundamental ones first, by themselves.
 Unit tests of more complex classes simply use the fundamental ones and
 assume that they are correct.
 
 Also, you can't unit test *every* class.  Where one can test, however,
 one should.

In a way we've already started some limited unit-testing: lyxstring.
The new implementation includes a set of tests that admittedly aren't
complete but are reasonably comprehensive.  I haven't managed to checkout
the new strand yet but I already have some doubts about the recent removal
of one of those tests.

The Bullet class also includses an almost-useless test that I need to
update.

Allan. (ARRae)

Re: Unit-Testing

[John Weiss]

On Thu, Oct 07, 1999 at 07:19:27PM +0200, Lars Gullik Bjønnes wrote:
> Allan Rae <[EMAIL PROTECTED]> writes:
> 
> | Unit-testing is certainly a good thing -- tests all those promises the
> | code makes and keeps Arndt happy about programming by contract since the
> | interface can't change without everybody finding out.
> 
> Unit-testing and independant modules is easy...unfortunately most of
> the modules in LyX depends on a multitude of other mudules makeing
> unit-testing very cumbersome and a lot of additional work.

Agreed.  Unit testing a class in a vacuum can be highly cumbersome.
That's why you unit test the fundamental ones first, by themselves.
Unit tests of more complex classes simply use the fundamental ones and
assume that they are correct.

Also, you can't unit test *every* class.  Where one can test, however,
one should.


> modules in the support dir could easily benefit form unit
> testing...but it is a lot of work to create the tests...
> 
> And tests can be wrong too...

True.  But once they're working, they save a great deal of time
finding and fixing future bugs...

This is one of those cases where the cost of the investment pays
itself back over time.

-- 
John Weiss

Re: Unit-Testing

[John Weiss]

On Thu, Oct 07, 1999 at 08:47:31PM +0200, Asger K. Alstrup Nielsen wrote:
> Instead, it's much better to simply address each file in order,
> document the header file, rename functions and variables to have 
> better names, in general clean it up, and try to make things better 
> small piece by piece.

I did not realize code cleanup was in such an imperfect state!  ;)

Seriously, I wasn't suggesting that we all drop what we're doing and
unit-test every class in the LyX source.  Some of us who can code but
have been daunted by the size of the source code, however, can build
unit tests.  Unit tests for cleaned-up classes and/or fundamental ones
should be feasable.

-- 
John Weiss

Re: Unit-Testing

[John Weiss]

On Thu, Oct 07, 1999 at 08:47:31PM +0200, Asger K. Alstrup Nielsen wrote:
> Instead, it's much better to simply address each file in order,
> document the header file, rename functions and variables to have 
> better names, in general clean it up, and try to make things better 
> small piece by piece.
> And keep in mind that everything should be stable all along.
> 
> This, I believe, is the way to proceed with LyX.
> This, I believe, is also the best way to learn the LyX source code:

No, no it's not.  If this were, as you say, the best way to learn the
LyX source code, about 4-5 devlist members who know C++ (myself
included) could have learned everything about LyX over the past couple
of years *and* simultaneously completely cleaned up the code.  That
never happened.  I, for one, took one look at the LyX source in June
of 1998 and got a bd headache!  Hell --- I've wanted a
"transpose-char" function in LyX for 4 years now, and I *still* have
no clue where it would go or how to add it.

Picking a class that *has* been cleaned up and documented, however,
and writing a test to see that it works isn't nearly as difficult.  If
you had asked me to build a test for LString a year ago, I could have
done it.  The only reason I haven't done so on my own is time.  I
barely read my email anymore.

-- 
John Weiss

Re: Unit-Testing

[John Weiss]

On Thu, Oct 07, 1999 at 05:25:36PM +0100, Arnd Hanses wrote:
>
> Else we would found the 'HardCore testing LyX never come to terms
> gang', making Core Developers live impossible. 
> 
> Then the real testing experts can by and by implement the rest on their
> own account and risk and report the results.

I only suggested that the author of a class not also author the unit
test for three reasons:  A) Lack of time; B) Conflict of interest; and
C) Lack of time.

In any case, unit tests aren't about making anybody's life
impossible. They're about putting each function through its paces,
making sure it does what the author intended, and also making sure it
handles bad input gracefully.  (Many a coredump was born when a
function met input it didn't expect...)

-- 
John Weiss

Re: Unit-Testing

[John Weiss]

Well, it seems that everyone thinks unit testing is a nice thing, but
nobody has the time to do it.

-- 
John Weiss

Re: Unit-Testing

[Allan Rae]

On Fri, 8 Oct 1999, John Weiss wrote:

> On Thu, Oct 07, 1999 at 07:19:27PM +0200, Lars Gullik Bjønnes wrote:
> > Allan Rae <[EMAIL PROTECTED]> writes:
> > 
> > | Unit-testing is certainly a good thing -- tests all those promises the
> > | code makes and keeps Arndt happy about programming by contract since the
> > | interface can't change without everybody finding out.
> > 
> > Unit-testing and independant modules is easy...unfortunately most of
> > the modules in LyX depends on a multitude of other mudules makeing
> > unit-testing very cumbersome and a lot of additional work.
> 
> Agreed.  Unit testing a class in a vacuum can be highly cumbersome.
> That's why you unit test the fundamental ones first, by themselves.
> Unit tests of more complex classes simply use the fundamental ones and
> assume that they are correct.
> 
> Also, you can't unit test *every* class.  Where one can test, however,
> one should.

In a way we've already started some limited unit-testing: lyxstring.
The new implementation includes a set of tests that admittedly aren't
complete but are reasonably comprehensive.  I haven't managed to checkout
the new strand yet but I already have some doubts about the recent removal
of one of those tests.

The Bullet class also includses an almost-useless test that I need to
update.

Allan. (ARRae)

Re: Unit-Testing

[Arnd Hanses]

On Thu, 7 Oct 1999 15:47:30 +1000 (GMT+1000), Allan Rae wrote:


Unit-testing is certainly a good thing -- tests all those promises the
code makes and keeps Arndt happy about programming by contract since the
interface can't change without everybody finding out.

I've learned my suggestions are damn dangerous ;-)

For starters I'd propose only some very simple assert() macros to
become obligatory for public functions, testing arguments/globals
(preconditions) and return values/side effects (postconditions) for
obvious non-sense. Moreover each public method should document these
conditions (and what it is supposed to do, por supuesto).

Else we would found the 'HardCore testing LyX never come to terms
gang', making Core Developers live impossible. 

Then the real testing experts can by and by implement the rest on their
own account and risk and report the results.

AH

Re: Unit-Testing

[Lars Gullik Bjønnes]

Allan Rae [EMAIL PROTECTED] writes:

| Unit-testing is certainly a good thing -- tests all those promises the
| code makes and keeps Arndt happy about programming by contract since the
| interface can't change without everybody finding out.

Unit-testing and independant modules is easy...unfortunately most of
the modules in LyX depends on a multitude of other mudules makeing
unit-testing very cumbersome and a lot of additional work.

modules in the support dir could easily benefit form unit
testing...but it is a lot of work to create the tests...

And tests can be wrong too...

Lgb

Re: Unit-Testing

[Asger K. Alstrup Nielsen]

 Unit-testing and independant modules is easy...unfortunately most of
 the modules in LyX depends on a multitude of other mudules makeing
 unit-testing very cumbersome and a lot of additional work.
 
 modules in the support dir could easily benefit form unit
 testing...but it is a lot of work to create the tests...
 
 And tests can be wrong too...

I agree with Lgb that unit testing is not the best use of efforts
right now.  Unit tests are nice when you have a well structured
and well designed program, and they will help you to keep such
a thing.

However, LyX is neither well structured or well designed, so the
benefit of unit testing will not be grand.  We will just learn that
a lot of code is broken, and we already know that.

Instead, it's much better to simply address each file in order,
document the header file, rename functions and variables to have 
better names, in general clean it up, and try to make things better 
small piece by piece.
And keep in mind that everything should be stable all along.

This, I believe, is the way to proceed with LyX.
This, I believe, is also the best way to learn the LyX source code:
By rewriting it in a better way.

Greets,

Asger

Re: Unit-Testing

[Arnd Hanses]

On Thu, 7 Oct 1999 15:47:30 +1000 (GMT+1000), Allan Rae wrote:

>
>Unit-testing is certainly a good thing -- tests all those promises the
>code makes and keeps Arndt happy about programming by contract since the
>interface can't change without everybody finding out.

I've learned my suggestions are damn dangerous ;-)

For starters I'd propose only some very simple assert() macros to
become obligatory for public functions, testing arguments/globals
(preconditions) and return values/side effects (postconditions) for
obvious non-sense. Moreover each public method should document these
conditions (and what it is supposed to do, por supuesto).

Else we would found the 'HardCore testing LyX never come to terms
gang', making Core Developers live impossible. 

Then the real testing experts can by and by implement the rest on their
own account and risk and report the results.

AH

Re: Unit-Testing

[Lars Gullik Bjønnes]

Allan Rae <[EMAIL PROTECTED]> writes:

| Unit-testing is certainly a good thing -- tests all those promises the
| code makes and keeps Arndt happy about programming by contract since the
| interface can't change without everybody finding out.

Unit-testing and independant modules is easy...unfortunately most of
the modules in LyX depends on a multitude of other mudules makeing
unit-testing very cumbersome and a lot of additional work.

modules in the support dir could easily benefit form unit
testing...but it is a lot of work to create the tests...

And tests can be wrong too...

Lgb

Re: Unit-Testing

[Asger K. Alstrup Nielsen]

> Unit-testing and independant modules is easy...unfortunately most of
> the modules in LyX depends on a multitude of other mudules makeing
> unit-testing very cumbersome and a lot of additional work.
> 
> modules in the support dir could easily benefit form unit
> testing...but it is a lot of work to create the tests...
> 
> And tests can be wrong too...

I agree with Lgb that unit testing is not the best use of efforts
right now.  Unit tests are nice when you have a well structured
and well designed program, and they will help you to keep such
a thing.

However, LyX is neither well structured or well designed, so the
benefit of unit testing will not be grand.  We will just learn that
a lot of code is broken, and we already know that.

Instead, it's much better to simply address each file in order,
document the header file, rename functions and variables to have 
better names, in general clean it up, and try to make things better 
small piece by piece.
And keep in mind that everything should be stable all along.

This, I believe, is the way to proceed with LyX.
This, I believe, is also the best way to learn the LyX source code:
By rewriting it in a better way.

Greets,

Asger

Unit-Testing

[John Weiss]

[I've been meaning to mention this for some time now.  But I've been
eaten alive by my house, and have been spending my train rides lately
hacking together new emacs programming modes.]

At work, just about every library has its own set of unit-tests.  If
you've never heard of a unit-test, the idea is this:  take a lib
you've written, and put it through its paces.  

One should attempt to test every method in a class (yues, even the
protected and private ones), making sure it behaves as expected.  One
should also make sure it correctly handles oddball conditions (bad
parameters, weird values, etc).

Unit tests are wonderful things to run through profilers (such as
Purify) to check for memory leaks.

Need to handle a large set of possible options?  Use a Monte-Carlo
simulation.  I can provide you with a class to generate basic random
entities (Including random pseudo-text that's verbally pronouncable).
I can also help out with creating Monte-Carlo simulations (that was a
main tool in my doctoral research).

Unit tests have an additional advantage:  they catch code-breaking
patches *before* you run the full program.  The only time you change a
unit test is if you make fundamental changes to the underlying
methods. 

Example: int foo(N) returns an index (don't ask me to what) between
0...N-1.  Someone submits a patch to foo() that breaks the unit test.
The patch is therefore bad.  Someone else submits a patch that changes
the returned index range to 1...N.  This also breaks the unit test,
but it should - the patch intentionally alters the fundamental
behavior of foo().

Who initially creates the unit test?  Well, for new code, whoever
wrote the new method/class/feature.  For old code, we should divide up
the work.  Writing unit tests is also a good way for new devvies to
cut their teeth on code in the CVS repository.

Who maintains/changes the unit tests?  Well, if a patch changes
fundamental behavior in a method or class, the patch author should
*NOT* be the one to change the unit test.  Someone else should do
that, just to maintain an independent perspective.

-- 
John Weiss

Re: Unit-Testing

[Allan Rae]

On Wed, 6 Oct 1999, John Weiss wrote:

 [I've been meaning to mention this for some time now.  But I've been
 eaten alive by my house, and have been spending my train rides lately
 hacking together new emacs programming modes.]

So your house is haunted?   What possible emacs programming modes are
there left to hack?

 At work, just about every library has its own set of unit-tests.  If
 you've never heard of a unit-test, the idea is this:  take a lib
 you've written, and put it through its paces.  
[...more on unit tests...]

Aegis is apparently a very capable tool to support this but I'm not sure
how well it would work on components of a project rather than an entire
project.  Do you have any suggestions for other tools to provide automated
support for unit-testing?  After all,  programmers are lazy and if we have
to do extra work just to get the testing done then we probably won't use
it.  I suppose the makefiles could be extended.

Unit-testing is certainly a good thing -- tests all those promises the
code makes and keeps Arndt happy about programming by contract since the
interface can't change without everybody finding out.

Allan. (ARRae)

Unit-Testing

[John Weiss]

[I've been meaning to mention this for some time now.  But I've been
eaten alive by my house, and have been spending my train rides lately
hacking together new emacs programming modes.]

At work, just about every library has its own set of unit-tests.  If
you've never heard of a unit-test, the idea is this:  take a lib
you've written, and put it through its paces.  

One should attempt to test every method in a class (yues, even the
protected and private ones), making sure it behaves as expected.  One
should also make sure it correctly handles oddball conditions (bad
parameters, weird values, etc).

Unit tests are wonderful things to run through profilers (such as
Purify) to check for memory leaks.

Need to handle a large set of possible options?  Use a Monte-Carlo
simulation.  I can provide you with a class to generate basic random
entities (Including random pseudo-text that's verbally pronouncable).
I can also help out with creating Monte-Carlo simulations (that was a
main tool in my doctoral research).

Unit tests have an additional advantage:  they catch code-breaking
patches *before* you run the full program.  The only time you change a
unit test is if you make fundamental changes to the underlying
methods. 

Example: int foo(N) returns an index (don't ask me to what) between
0...N-1.  Someone submits a patch to foo() that breaks the unit test.
The patch is therefore bad.  Someone else submits a patch that changes
the returned index range to 1...N.  This also breaks the unit test,
but it should - the patch intentionally alters the fundamental
behavior of foo().

Who initially creates the unit test?  Well, for new code, whoever
wrote the new method/class/feature.  For old code, we should divide up
the work.  Writing unit tests is also a good way for new devvies to
cut their teeth on code in the CVS repository.

Who maintains/changes the unit tests?  Well, if a patch changes
fundamental behavior in a method or class, the patch author should
*NOT* be the one to change the unit test.  Someone else should do
that, just to maintain an independent perspective.

-- 
John Weiss

Re: Unit-Testing

[Allan Rae]

On Wed, 6 Oct 1999, John Weiss wrote:

> [I've been meaning to mention this for some time now.  But I've been
> eaten alive by my house, and have been spending my train rides lately
> hacking together new emacs programming modes.]

So your house is haunted?   What possible emacs programming modes are
there left to hack?

> At work, just about every library has its own set of unit-tests.  If
> you've never heard of a unit-test, the idea is this:  take a lib
> you've written, and put it through its paces.  
[...more on unit tests...]

Aegis is apparently a very capable tool to support this but I'm not sure
how well it would work on components of a project rather than an entire
project.  Do you have any suggestions for other tools to provide automated
support for unit-testing?  After all,  programmers are lazy and if we have
to do extra work just to get the testing done then we probably won't use
it.  I suppose the makefiles could be extended.

Unit-testing is certainly a good thing -- tests all those promises the
code makes and keeps Arndt happy about programming by contract since the
interface can't change without everybody finding out.

Allan. (ARRae)