On 8/16/06, Anton Luht wrote:
IMHO analysis of exception stacktrace output is a bad idea. Tests should not produce uncatched exceptions unless they fail. Test authors should explicitly catch an exception and print some text to stdout.
See [1] for testing details. Reasons are:
- exception messages are not standard, different JVMs can produce different output - localized messages can be printed, the test may fail because of locales - if test author expects IOException but subclass of this exception is thrown and printed, the test will fail while if the test was written in the way "catch IOException; print "PASSED" the test would pass
For example, how you will test that VM is aborted during start-up with NoClassDefFoundError (not with IllegalArgumentException or any other exception) if it is impossible to load requested class? Thanks, Stepan. [1] http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200608.mbox/[EMAIL PROTECTED] On 8/16/06, Stepan Mishura <[EMAIL PROTECTED]> wrote:
> On 8/16/06, Anton Luht wrote: > > > > Cool idea - rethrow exceptions from exec'ed java :) You exec jvm that > > dies because some exception (say, NPE) not catched, support_exec > > analyzes output, understands that NPE was thrown and rethrows it :) > > People who will look at those tests will certainly go mad. > > > You misunderstoond the idea for using support_exec - no exceptions > re-throwing should be made. A test will verify only that VM fails in > expected way (i.e. with expected error message). > > Do you have a better idea how to test that? > > Thanks, > Stepan. > > On 8/16/06, Jimmy, Jing Lv <[EMAIL PROTECTED]> wrote: > > > Paulex Yang wrote: > > > > Jimmy, Jing Lv wrote: > > > >> Hi, > > > >> > > > >> I'm trying to write some unit tests of instrument. As discussed, > > > >> I'm using exec. > > > >> Currently I've tried Support_Exec and find it fairly easy to > > > >> use(Thanks Stepan :) ). > > > >> > > > >> For resources (jar files and utility classes), I make a new > > > >> directory: > > > >> > > $Harmony/modules/instrument/src/test/resource/org/apache/harmony/tests/instrument > > > >> and put them there. > > > >> > > > >> And I meet a problem here. Currently the default classpath in > > > >> running test is "$Harmony/modules/instrument/bin", so it meets > > problem > > > >> in loading those utility classes (it can load jar files though). > > > >> I remember Mark has plan to change the classpath in running > > > >> testcase, it'll be great if he make some progress. > > > >> > > > >> An another problem is checking, currently the result is checked > > by > > > >> searching and compare in string(e.g, find "someException" in result). > > > >> But the framework seems fails if any exception is thrown out. Can it > > > >> make a little change that it passes and returns the error message in > > > >> exec? > > > > Didn't catch up here, did you mean that you need to check the > > exception > > > > message as well as exception type? > > > > > > Not necessary to check messages. However we can not catch those > > > exceptions so we have to analysis output. For an example: > > > > > > run "java -javaagent:test.jar myclass" > > > > > > so get output: > > > Exception in thread "main" java.lang.reflect.InvocationTargetException > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > Source) > > > at java.lang.reflect.Method.invoke(Unknown Source) > > > at > > > sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown So > > > urce) > > > Caused by: java.lang.IllegalAccessError > > > ... > > > > > > In this case we can search for "InvocationTargetException" in the output > > > to ensure the expected exception. Sometimes we may also check "cause by" > > > exceptions. > > > > > > This approach seems not perfect. Any comments/suggestions? Thanks! > > > > > > <snip> > > > -- > > > > > > Best Regards! > > > > > > Jimmy, Jing Lv > > > China Software Development Lab, IBM ------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
