Re: [viff-devel] VIFF unit tests
Janus Dam Nielsen <[EMAIL PROTECTED]> writes: >> But I asked since I am not sure I completely understand how you want >> this implemented? > > Neither do I. I just want to be naive and stupid and run the entire > test suite with a set of parameters without having to remember or even > know that a certain testcase does not support my parameter. How to do > this, I don't know. I just like it. Then all I can say is that I think we have an idea for a possible solution below... :-) > Den 15/02/2008 kl. 10.21 skrev Martin Geisler: > >> Janus Dam Nielsen <[EMAIL PROTECTED]> writes: >> > I think that having parametrized tests is good, however I just > wanted to point out that defining the parameters in the Runtime > class/object might not be suffienciently expressive to what we > want. > We might would like a kind of grouping/system of tests so that it > is > easy to run the tests without any particular knowledge of which > protocols support which parameters. > > Those tests for which a given set of parameters is invalid, the > test > could return an undefined value, or the test could be elided from > the set of tests since it doesn't make any sense for these > parameters anyhow. The test suite is implemented using Trial, a Twisted tool which extends the standard Python unittest module with support for Deferreds. The Python unittest module is modelled after JUnit. In Trial there is support for marking a test as skipped, and that might be useful for what you are describing -- we could query the tests for their requirements and if they do not match the parameters of the current test, then we skip that test. Something like that could work, but I don't know if it is the best way... Have you looked at the Trial documentation to see how it could be done? There is a tutorial here: http://twistedmatrix.com/trac/browser/branches/trial- tutorial-2443/doc/core/howto/trial.xhtml?format=raw and the API documentation is here: http://twistedmatrix.com/documents/current/api/ twisted.trial.unittest.TestCase.html Trial is not so well documented as the rest of Twisted, so looking at the source code has helped me a bit until I found the above tutorial. >>> >>> I haven't looked at the code. I just wanted to make sure you didn't >>> shoot yourself in the foot unintentionally :) >> >> Yeah, thanks, that would be bad :-) so it is really nice to get some >> input in these design questions! >> >> But I asked since I am not sure I completely understand how you want >> this implemented? >> >> You want some set of parameters that are specified on the command line >> when one starts Trial, or should they be put in the unit tests >> directly? >> >> The latter is easy, and we can already now generate many unit tests >> with >> small differences such as the number of players. Adding this to >> test_active_runtime.py: >> >> class Active5(ActiveRuntimeTest): >> num_players = 5 >> >> class Active6(ActiveRuntimeTest): >> num_players = 6 >> >> class Active7(ActiveRuntimeTest): >> num_players = 7 >> >> makes the Bracha broadcast test be run three more times, but with more >> players. It even works, I just tested! :-) >> >> That might be a good way to do things: code a TestCase with some unit >> tests, but be sure to make them generic in the sense that they can be >> run with any number of players (or threshold). Then create several >> subclasses like above. The classes can even be created at runtime: >> >> for n in range(3,6): >> code = """ >> class Active%d(ActiveRuntimeTest): >> num_players = %d >> """ % (n, n) >> exec code >> >> Trial never notices the difference and runs our tests as normal! >> >> This way we have lots of power over how the tests are generated: we >> can >> easily test, say, all n<10, 10 randomly chosen n between 10 and 50, >> and >> 5 random n's between 50 and 100. Something like that... >> >> -- >> Martin Geisler >> ___ >> viff-devel mailing list (http://viff.dk/) >> viff-devel@viff.dk >> http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk > -- Martin Geisler pgp3Wq7tZo8nZ.pgp Description: PGP signature ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
Re: [viff-devel] VIFF unit tests
> But I asked since I am not sure I completely understand how you want > this implemented? Neither do I. I just want to be naive and stupid and run the entire test suite with a set of parameters without having to remember or even know that a certain testcase does not support my parameter. How to do this, I don't know. I just like it. -- Janus Den 15/02/2008 kl. 10.21 skrev Martin Geisler: > Janus Dam Nielsen <[EMAIL PROTECTED]> writes: > I think that having parametrized tests is good, however I just wanted to point out that defining the parameters in the Runtime class/object might not be suffienciently expressive to what we want. We might would like a kind of grouping/system of tests so that it is easy to run the tests without any particular knowledge of which protocols support which parameters. Those tests for which a given set of parameters is invalid, the test could return an undefined value, or the test could be elided from the set of tests since it doesn't make any sense for these parameters anyhow. >>> >>> The test suite is implemented using Trial, a Twisted tool which >>> extends the standard Python unittest module with support for >>> Deferreds. The Python unittest module is modelled after JUnit. >>> >>> In Trial there is support for marking a test as skipped, and that >>> might be useful for what you are describing -- we could query the >>> tests for their requirements and if they do not match the parameters >>> of the current test, then we skip that test. >>> >>> Something like that could work, but I don't know if it is the best >>> way... Have you looked at the Trial documentation to see how it >>> could >>> be done? There is a tutorial here: >>> >>> http://twistedmatrix.com/trac/browser/branches/trial- >>> tutorial-2443/doc/core/howto/trial.xhtml?format=raw >>> >>> and the API documentation is here: >>> >>> http://twistedmatrix.com/documents/current/api/ >>> twisted.trial.unittest.TestCase.html >>> >>> Trial is not so well documented as the rest of Twisted, so >>> looking at >>> the source code has helped me a bit until I found the above >>> tutorial. >> >> I haven't looked at the code. I just wanted to make sure you didn't >> shoot yourself in the foot unintentionally :) > > Yeah, thanks, that would be bad :-) so it is really nice to get some > input in these design questions! > > But I asked since I am not sure I completely understand how you want > this implemented? > > You want some set of parameters that are specified on the command line > when one starts Trial, or should they be put in the unit tests > directly? > > The latter is easy, and we can already now generate many unit tests > with > small differences such as the number of players. Adding this to > test_active_runtime.py: > > class Active5(ActiveRuntimeTest): > num_players = 5 > > class Active6(ActiveRuntimeTest): > num_players = 6 > > class Active7(ActiveRuntimeTest): > num_players = 7 > > makes the Bracha broadcast test be run three more times, but with more > players. It even works, I just tested! :-) > > That might be a good way to do things: code a TestCase with some unit > tests, but be sure to make them generic in the sense that they can be > run with any number of players (or threshold). Then create several > subclasses like above. The classes can even be created at runtime: > > for n in range(3,6): > code = """ > class Active%d(ActiveRuntimeTest): > num_players = %d > """ % (n, n) > exec code > > Trial never notices the difference and runs our tests as normal! > > This way we have lots of power over how the tests are generated: we > can > easily test, say, all n<10, 10 randomly chosen n between 10 and 50, > and > 5 random n's between 50 and 100. Something like that... > > -- > Martin Geisler > ___ > viff-devel mailing list (http://viff.dk/) > viff-devel@viff.dk > http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
Re: [viff-devel] VIFF unit tests
Janus Dam Nielsen <[EMAIL PROTECTED]> writes: >>> I think that having parametrized tests is good, however I just >>> wanted to point out that defining the parameters in the Runtime >>> class/object might not be suffienciently expressive to what we want. >>> We might would like a kind of grouping/system of tests so that it is >>> easy to run the tests without any particular knowledge of which >>> protocols support which parameters. >>> >>> Those tests for which a given set of parameters is invalid, the test >>> could return an undefined value, or the test could be elided from >>> the set of tests since it doesn't make any sense for these >>> parameters anyhow. >> >> The test suite is implemented using Trial, a Twisted tool which >> extends the standard Python unittest module with support for >> Deferreds. The Python unittest module is modelled after JUnit. >> >> In Trial there is support for marking a test as skipped, and that >> might be useful for what you are describing -- we could query the >> tests for their requirements and if they do not match the parameters >> of the current test, then we skip that test. >> >> Something like that could work, but I don't know if it is the best >> way... Have you looked at the Trial documentation to see how it could >> be done? There is a tutorial here: >> >> http://twistedmatrix.com/trac/browser/branches/trial- >> tutorial-2443/doc/core/howto/trial.xhtml?format=raw >> >> and the API documentation is here: >> >> http://twistedmatrix.com/documents/current/api/ >> twisted.trial.unittest.TestCase.html >> >> Trial is not so well documented as the rest of Twisted, so looking at >> the source code has helped me a bit until I found the above tutorial. > > I haven't looked at the code. I just wanted to make sure you didn't > shoot yourself in the foot unintentionally :) Yeah, thanks, that would be bad :-) so it is really nice to get some input in these design questions! But I asked since I am not sure I completely understand how you want this implemented? You want some set of parameters that are specified on the command line when one starts Trial, or should they be put in the unit tests directly? The latter is easy, and we can already now generate many unit tests with small differences such as the number of players. Adding this to test_active_runtime.py: class Active5(ActiveRuntimeTest): num_players = 5 class Active6(ActiveRuntimeTest): num_players = 6 class Active7(ActiveRuntimeTest): num_players = 7 makes the Bracha broadcast test be run three more times, but with more players. It even works, I just tested! :-) That might be a good way to do things: code a TestCase with some unit tests, but be sure to make them generic in the sense that they can be run with any number of players (or threshold). Then create several subclasses like above. The classes can even be created at runtime: for n in range(3,6): code = """ class Active%d(ActiveRuntimeTest): num_players = %d """ % (n, n) exec code Trial never notices the difference and runs our tests as normal! This way we have lots of power over how the tests are generated: we can easily test, say, all n<10, 10 randomly chosen n between 10 and 50, and 5 random n's between 50 and 100. Something like that... -- Martin Geisler pgpqDrd8ZCeHb.pgp Description: PGP signature ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
Re: [viff-devel] VIFF unit tests
I haven't looked at the code. I just wanted to make sure you didn't shoot yourself in the foot unintentionally :) -- Janus Den 15/02/2008 kl. 1.57 skrev Martin Geisler: > Janus Dam Nielsen <[EMAIL PROTECTED]> writes: > >> I think that having parametrized tests is good, however I just >> wanted to point out that defining the parameters in the Runtime >> class/object might not be suffienciently expressive to what we want. >> We might would like a kind of grouping/system of tests so that it is >> easy to run the tests without any particular knowledge of which >> protocols support which parameters. >> >> Those tests for which a given set of parameters is invalid, the test >> could return an undefined value, or the test could be elided from >> the set of tests since it doesn't make any sense for these >> parameters anyhow. > > The test suite is implemented using Trial, a Twisted tool which > extends the standard Python unittest module with support for > Deferreds. The Python unittest module is modelled after JUnit. > > In Trial there is support for marking a test as skipped, and that > might be useful for what you are describing -- we could query the > tests for their requirements and if they do not match the parameters > of the current test, then we skip that test. > > Something like that could work, but I don't know if it is the best > way... Have you looked at the Trial documentation to see how it could > be done? There is a tutorial here: > > http://twistedmatrix.com/trac/browser/branches/trial- > tutorial-2443/doc/core/howto/trial.xhtml?format=raw > > and the API documentation is here: > > http://twistedmatrix.com/documents/current/api/ > twisted.trial.unittest.TestCase.html > > Trial is not so well documented as the rest of Twisted, so looking at > the source code has helped me a bit until I found the above tutorial. > > -- > Martin Geisler > ___ > viff-devel mailing list (http://viff.dk/) > viff-devel@viff.dk > http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
Re: [viff-devel] VIFF unit tests
Janus Dam Nielsen <[EMAIL PROTECTED]> writes: > I think that having parametrized tests is good, however I just > wanted to point out that defining the parameters in the Runtime > class/object might not be suffienciently expressive to what we want. > We might would like a kind of grouping/system of tests so that it is > easy to run the tests without any particular knowledge of which > protocols support which parameters. > > Those tests for which a given set of parameters is invalid, the test > could return an undefined value, or the test could be elided from > the set of tests since it doesn't make any sense for these > parameters anyhow. The test suite is implemented using Trial, a Twisted tool which extends the standard Python unittest module with support for Deferreds. The Python unittest module is modelled after JUnit. In Trial there is support for marking a test as skipped, and that might be useful for what you are describing -- we could query the tests for their requirements and if they do not match the parameters of the current test, then we skip that test. Something like that could work, but I don't know if it is the best way... Have you looked at the Trial documentation to see how it could be done? There is a tutorial here: http://twistedmatrix.com/trac/browser/branches/trial-tutorial-2443/doc/core/howto/trial.xhtml?format=raw and the API documentation is here: http://twistedmatrix.com/documents/current/api/twisted.trial.unittest.TestCase.html Trial is not so well documented as the rest of Twisted, so looking at the source code has helped me a bit until I found the above tutorial. -- Martin Geisler ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
Re: [viff-devel] VIFF unit tests
I think that having parametrized tests is good, however I just wanted to point out that defining the parameters in the Runtime class/object might not be suffienciently expressive to what we want. We might would like a kind of grouping/system of tests so that it is easy to run the tests without any particular knowledge of which protocols support which parameters. Those tests for which a given set of parameters is invalid, the test could return an undefined value, or the test could be elided from the set of tests since it doesn't make any sense for these parameters anyhow. -- Janus Den 14/02/2008 kl. 4.22 skrev Martin Geisler: > "Thomas Jakobsen" <[EMAIL PROTECTED]> writes: > >> As Martin points out, it will be inconvenient or impossible to >> parameterize some tests like this. But for others, it will be just >> as easy, and in these cases we get a lot of extra testing for "free" >> by doing it, since it will enable us to run them with many >> parameters. > > We just have to remember to be very careful with the random seeds used > in these tests -- we must be able to reproduce the tests exactly. > > But if we get that right, I think random testing would be great way to > expand the scope of the current tests to make sure that everything > still holds together with more players, larger threshold, etc. > > -- > Martin Geisler > ___ > viff-devel mailing list (http://viff.dk/) > viff-devel@viff.dk > http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
Re: [viff-devel] VIFF unit tests
"Thomas Jakobsen" <[EMAIL PROTECTED]> writes: > As Martin points out, it will be inconvenient or impossible to > parameterize some tests like this. But for others, it will be just > as easy, and in these cases we get a lot of extra testing for "free" > by doing it, since it will enable us to run them with many > parameters. We just have to remember to be very careful with the random seeds used in these tests -- we must be able to reproduce the tests exactly. But if we get that right, I think random testing would be great way to expand the scope of the current tests to make sure that everything still holds together with more players, larger threshold, etc. -- Martin Geisler ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
Re: [viff-devel] VIFF unit tests
Jakob Illeborg Pagter <[EMAIL PROTECTED]> writes: > Thomas Jakobsen skrev: > >> As Martin points out, it will be inconvenient or impossible to >> parameterize some tests like this. But for others, it will be just >> as easy, and in these cases we get a lot of extra testing for >> "free" by doing it, since it will enable us to run them with many >> parameters. > > Why don't you just parameterize tests by default and then let the > test throw an error if the parameters are not within a defined legal > range? No reason, I guess... By grouping tests in different classes we can already obtains some of the wanted flexibility: the test_broadcast method is in the ActiveRuntimeTest class because this class specifies that we need 4 players. We could change this number to be a threshold factor, something like threshold_factor = 3 which would mean that we need num_playes > threshold_factor * threshold which with threshold = 1 gives the four players we had before. But given such a factor, it would be easy to generate test cases with varying number of players and threshold. -- Martin Geisler ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
Re: [viff-devel] VIFF unit tests
Why don't you just parameterize tests by default and then let the test throw an error if the parameters are not within a defined legal range? -- Jakob Thomas Jakobsen skrev: > Exactly. > > As Martin points out, it will be inconvenient or impossible to > parameterize some tests like this. But for others, it will be just as > easy, and in these cases we get a lot of extra testing for "free" by > doing it, since it will enable us to run them with many parameters. > > -- Thomas > > ___ > viff-devel mailing list (http://viff.dk/) > viff-devel@viff.dk > http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
Re: [viff-devel] VIFF unit tests
Exactly. As Martin points out, it will be inconvenient or impossible to parameterize some tests like this. But for others, it will be just as easy, and in these cases we get a lot of extra testing for "free" by doing it, since it will enable us to run them with many parameters. -- Thomas ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
Re: [viff-devel] VIFF unit tests
Janus Dam Nielsen <[EMAIL PROTECTED]> writes: > Does it make sense to run all unit tests with the same configuations > of players and thresholds. That is, for all protocols p, is p > executed with x players and threshold t is p welldefined? Well, no, not in general. If you write a protocol for seven people Shamir sharing would look like this in VIFF: a, b, c, d, e, f, g = rt.shamir_share(range(1, 8), input) and that breaks down (I guess quite spectacularly...) when you run it with fewer than seven players. I think Thomas wants us to write protocols that are generic in the sense that they don't assume a particular number of players. So the above line becomes: shares = rt.shamir_share(range(1, len(rt.players)+1), input) If the next step is to add all inputs, then this is easily changed from sum = a + b + c + d + e + f + g to import operator sum = reduce(operator.add, shares) which works any number of shares. So it is only carefully written protocols that can be adapted like that. Examples could be: * Find the minimum input, the median input, the sum, or the average. * Sort the inputs (using a simple bubble-sort or similar). The Yao millionaires example already sorts three elements via three comparisons. -- Martin Geisler ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
Re: [viff-devel] Viff unit tests
I thing to consider. Does it make sense to run all unit tests with the same configuations of players and thresholds. That is, for all protocols p, is p executed with x players and threshold t is p welldefined? I suspect, out of the blue air, that this is not the case. I am I right or am I wrong? -- Janus Den 13/02/2008 kl. 8.43 skrev Martin Geisler: > "Thomas Jakobsen" <[EMAIL PROTECTED]> writes: > >> I suggest that we - as far as it's reasonable - write unit tests >> that don't depend on a specific number of players and threshold, but >> instead use number of players and threshold as defined by >> runtime.threshold and runtime.players. This will allow us to >> automatically run these tests with many combinations of (no of >> players, threshold), say, (3,1), (5,2), (5,1), (7,3). > > That is an excellent idea! You're right that the current unit tests > are hard-coded to use a particular number of players (3 player for > most of tests and 4 for the Bracha broadcast test). We should change > this and make a setup where the same test is run with different > parameters. > > Changing it is quite easy (but somewhat tedious): > > a, b, c = runtime.shamir_share([1, 2, 3], self.Zp, 42 + > runtime.id) > > becomes > > everybody = range(1, len(runtime.players)+1) > shares = runtime.shamir_share(everybody, self.Zp, 42 + > runtime.id) > > and we can then loop over shares as we see fit. I think I will make a > num_players field in Runtime so that we can stop writing > len(runtime.players) so often... > > One can almost just use runtime.players.keys(), but since > runtime.players is a dictionary we don't really know in which order > the keys will come out, or more importantly, if the order is the same > for all players. > >> My experience from developing protocols for the previous SIMAP >> prototype was that testing all protocols with (3,1), (5,1) (5,2) and >> (7,3) sometimes caught subtle bugs that wouldn't have been caught >> when testing only (3,1) and, say, (5,2). > > Yeah, there might be some strange bugs hiding there... > > I think we should do this after a 0.4 release. There has been quite a > lot of new features since 0.3: some asymmetric protocols, Bracha > broadcast, xor overloading. When we have asymmetric open and > prss_share, then I think we should release the code as VIFF 0.4. > > -- > Martin Geisler > ___ > viff-devel mailing list (http://viff.dk/) > viff-devel@viff.dk > http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
Re: [viff-devel] Viff unit tests
"Thomas Jakobsen" <[EMAIL PROTECTED]> writes: > I suggest that we - as far as it's reasonable - write unit tests > that don't depend on a specific number of players and threshold, but > instead use number of players and threshold as defined by > runtime.threshold and runtime.players. This will allow us to > automatically run these tests with many combinations of (no of > players, threshold), say, (3,1), (5,2), (5,1), (7,3). That is an excellent idea! You're right that the current unit tests are hard-coded to use a particular number of players (3 player for most of tests and 4 for the Bracha broadcast test). We should change this and make a setup where the same test is run with different parameters. Changing it is quite easy (but somewhat tedious): a, b, c = runtime.shamir_share([1, 2, 3], self.Zp, 42 + runtime.id) becomes everybody = range(1, len(runtime.players)+1) shares = runtime.shamir_share(everybody, self.Zp, 42 + runtime.id) and we can then loop over shares as we see fit. I think I will make a num_players field in Runtime so that we can stop writing len(runtime.players) so often... One can almost just use runtime.players.keys(), but since runtime.players is a dictionary we don't really know in which order the keys will come out, or more importantly, if the order is the same for all players. > My experience from developing protocols for the previous SIMAP > prototype was that testing all protocols with (3,1), (5,1) (5,2) and > (7,3) sometimes caught subtle bugs that wouldn't have been caught > when testing only (3,1) and, say, (5,2). Yeah, there might be some strange bugs hiding there... I think we should do this after a 0.4 release. There has been quite a lot of new features since 0.3: some asymmetric protocols, Bracha broadcast, xor overloading. When we have asymmetric open and prss_share, then I think we should release the code as VIFF 0.4. -- Martin Geisler ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
[viff-devel] Viff unit tests
Hi I suggest that we - as far as it's reasonable - write unit tests that don't depend on a specific number of players and threshold, but instead use number of players and threshold as defined by runtime.threshold and runtime.players. This will allow us to automatically run these tests with many combinations of (no of players, threshold), say, (3,1), (5,2), (5,1), (7,3). My experience from developing protocols for the previous SIMAP prototype was that testing all protocols with (3,1), (5,1) (5,2) and (7,3) sometimes caught subtle bugs that wouldn't have been caught when testing only (3,1) and, say, (5,2). Any opinions about this? Regards, Thomas ___ viff-devel mailing list (http://viff.dk/) viff-devel@viff.dk http://lists.viff.dk/listinfo.cgi/viff-devel-viff.dk
