On Mon, Apr 9, 2018 at 1:38 PM, Vincent Massol <vinc...@massol.net> wrote: > Hi Thomas, > >> On 6 Apr 2018, at 11:57, Thomas Mortagne <thomas.morta...@xwiki.com> wrote: >> >> Not sure what to think of those examples. >> >> The theory sounds nice and it probably produce nice result when >> talking about one method with a few inputs and an output but for >> anything more complex (like anything involving components) I feel you >> end up with a generated test which is a lot harder to read than simply >> looking at the code. > > As I said in my original mail, I think that really depends about who handles > the issue (a long time committer who’s being working for years on the code > and knows it very well) or someone new who doesn’t know the code well. > > Also, it really depends on the context. Imagine the following example:you > have a NPE stack trace in production and the line where the NPE is has > several variables that can be null. Just looking at the code will not tell > you what the problem is. But the test will and you can even put a breakpoint > to know which variable is actually null. > >> * https://jira.xwiki.org/browse/XWIKI-13031: does not produce the same >> error at all. Looks like evocash just try to produce any >> ClassCastException but does not care about the actual types in it >> which is quite useless (plus reading that test requires a lot of >> effort just to conclude that the type of the field in the document is >> not String…) > > I’ve not tested them all but all the ones I’ve tested do reproduce correctly > the stack trace (that’s the point of evocrash so there’s no way it’ll > generate a test that doesn’t reproduce the stack trace! ;)). > > To give you more info, you run evocrash by giving it a stacktrace and a frame > number (the number of stack trace lines you want it to reproduce). > > For ex, on: > > Caused by: java.lang.ClassCastException: org.apache.lucene.document.Field > cannot be cast to java.lang.String > at > org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getWikiReference(SolrEntityReferenceResolver.java:93) > ~[xwiki-platform-search-solr-api-7.4.jar:na] > at > org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getEntityReference(SolrEntityReferenceResolver.java:70) > ~[xwiki-platform-search-solr-api-7.4.jar:na] > at > org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:63) > ~[xwiki-platform-search-solr-api-7.4.jar:na] > at > org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:46) > ~[xwiki-platform-search-solr-api-7.4.jar:na] > at > org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:48) > ~[xwiki-platform-search-solr-api-7.4.jar:na] > at > org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:38) > ~[xwiki-platform-search-solr-api-7.4.jar:na] > […] > > If you run evocrash with frame = 5, it’ll generate a test that, when executed > lead to: > > Caused by: java.lang.ClassCastException: org.apache.lucene.document.Field > cannot be cast to java.lang.String > at > org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getWikiReference(SolrEntityReferenceResolver.java:93) > ~[xwiki-platform-search-solr-api-7.4.jar:na] > at > org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getEntityReference(SolrEntityReferenceResolver.java:70) > ~[xwiki-platform-search-solr-api-7.4.jar:na] > at > org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:63) > ~[xwiki-platform-search-solr-api-7.4.jar:na] > at > org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:46) > ~[xwiki-platform-search-solr-api-7.4.jar:na] > at > org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:48) > ~[xwiki-platform-search-solr-api-7.4.jar:na]
If you execute the test indicated in https://jira.xwiki.org/browse/XWIKI-13031?focusedCommentId=98082&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-98082 you will get the following exception: java.lang.ClassCastException: org.xwiki.model.internal.reference.ExplicitStringEntityReferenceResolver cannot be cast to java.lang.String Sure you get the same stack trace from Exception type and methods calls point of view but what the message describe is a very different use case. > >> * https://jira.xwiki.org/browse/XWIKI-14475: totally unreadable > > Actually this test was not that hard to read for me. It just sets up some > mocks. Evocrash can make it even simpler by presenting the mocks in a simpler > way but that’s cosmetic and not the focus of evocrash just right now (it’s > not the hard part). I’ve also raised something a bit related: > https://github.com/STAMP-project/EvoCrash/issues/7 > > Feel free to raise new issues at > https://github.com/STAMP-project/EvoCrash/issues/ > >> * https://jira.xwiki.org/browse/XRENDERING-422: not really useful but >> I doubt evocrash can do much about this kind of issue > > Yes I agree. > >> * https://jira.xwiki.org/browse/XWIKI-13196: this one could help a bit >> I guess but not sure it's really faster to execute evocash than >> looking at the code :) > > Sure but it also give you the test which is nice. And depending on the line > generating the NPE, it’s not always easy to know which variable is null by > looking at the code. > >> * https://jira.xwiki.org/browse/XWIKI-13916: that's a more complex way >> than it should to express that we call addTextAreaField for an already >> existing field yes it's useful > > Thanks for the feedback. > > There are plenty of limitations/issues with evocrash ATM. It’s good if we can > identify cases where it generates stuff that could be useful. One direction > that the devs are pursuing is looking at existing test classes to mimick how > they’re written to produce the new tests. I don’t know how they’d be able to > pull this off (that seems really hard to me ;)) but I know they’re working in > this direction. > > Thanks > -Vincent > >> >> On Fri, Apr 6, 2018 at 10:44 AM, Vincent Massol <vinc...@massol.net> wrote: >>> Hi devs, >>> >>> Context >>> ====== >>> >>> Evocrash is a tool developed as part of the STAMP research project >>> (https://www.stamp-project.eu/). Its goal is to take a stack trace (the use >>> case is production stack traces) and generate a test that, when executed, >>> reproduces the stack trace ;) >>> >>> It’s using Guided Genetic Algorithm to simplify the search space (which is >>> pretty cool, isn’t it? ;)). More info at >>> https://github.com/STAMP-project/EvoCrash and http://www.evocrash.org/ >>> >>> Results >>> ====== >>> >>> So far evocrash was able to generate tests for the following issues (I’ve >>> pasted the generated test code as comment): >>> * https://jira.xwiki.org/browse/XRENDERING-422 >>> * https://jira.xwiki.org/browse/XWIKI-14475 >>> * https://jira.xwiki.org/browse/XWIKI-13031 >>> * https://jira.xwiki.org/browse/XWIKI-13196 >>> * https://jira.xwiki.org/browse/XWIKI-13916 >>> >>> Using >>> ===== >>> >>> The use case I see is the following: >>> * A user reports a problem when using XWiki and create a jira issue with a >>> stack trace >>> * An XWiki dev finds it and runs evocrash to generate a test reproducing >>> the problem >>> * The XWiki dev uses the test to understand the reason (breakpoints can be >>> put in the IDE) of the problem and writes a better test (which can be >>> inspired by the test generated by evocrash or be completely different). >>> >>> Questions >>> ======== >>> >>> The question we need to answer is whether we think it coud be useful or >>> not. Are there cases where it can be useful? >>> >>> For example if we imagine someone not knowing the xwiki code base well, >>> maybe this can help him/her understand the bug in a simpler way than just >>> having to read/find where the problem is in the code base? >>> >>> I’ve started the work of trying to evaluate how useful it could be on the >>> XWiki project by analyzing the examples above. I've only analyzed 2 ATM >>> (see the links in the jira comments) and I’d be interested to have your >>> analysis too. >>> >>> WDYT? >>> >>> Thanks >>> -Vincent >> >> >> >> -- >> Thomas Mortagne > -- Thomas Mortagne
