> On 9 Apr 2018, at 15:34, Thomas Mortagne <thomas.morta...@xwiki.com> wrote:
> 
> On Mon, Apr 9, 2018 at 1:38 PM, Vincent Massol <vinc...@massol.net> wrote:
>> Hi Thomas,
>> 
>>> On 6 Apr 2018, at 11:57, Thomas Mortagne <thomas.morta...@xwiki.com> wrote:
>>> 
>>> Not sure what to think of those examples.
>>> 
>>> The theory sounds nice and it probably produce nice result when
>>> talking about one method with a few inputs and an output but for
>>> anything more complex (like anything involving components) I feel you
>>> end up with a generated test which is a lot harder to read than simply
>>> looking at the code.
>> 
>> As I said in my original mail, I think that really depends about who handles 
>> the issue (a long time committer who’s being working for years on the code 
>> and knows it very well) or someone new who doesn’t know the code well.
>> 
>> Also, it really depends on the context. Imagine the following example:you 
>> have a NPE stack trace in production and the line where the NPE is has 
>> several variables that can be null. Just looking at the code will not tell 
>> you what the problem is. But the test will and you can even put a breakpoint 
>> to know which variable is actually null.
>> 
>>> * https://jira.xwiki.org/browse/XWIKI-13031: does not produce the same
>>> error at all. Looks like evocash just try to produce any
>>> ClassCastException but does not care about the actual types in it
>>> which is quite useless (plus reading that test requires a lot of
>>> effort just to conclude that the type of the field in the document is
>>> not String…)
>> 
>> I’ve not tested them all but all the ones I’ve tested do reproduce correctly 
>> the stack trace (that’s the point of evocrash so there’s no way it’ll 
>> generate a test that doesn’t reproduce the stack trace! ;)).
>> 
>> To give you more info, you run evocrash by giving it a stacktrace and a 
>> frame number (the number of stack trace lines you want it to reproduce).
>> 
>> For ex, on:
>> 
>> Caused by: java.lang.ClassCastException: org.apache.lucene.document.Field 
>> cannot be cast to java.lang.String
>>        at 
>> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getWikiReference(SolrEntityReferenceResolver.java:93)
>>  ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at 
>> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getEntityReference(SolrEntityReferenceResolver.java:70)
>>  ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at 
>> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:63)
>>  ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at 
>> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:46)
>>  ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at 
>> org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:48)
>>  ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at 
>> org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:38)
>>  ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        […]
>> 
>> If you run evocrash with frame = 5, it’ll generate a test that, when 
>> executed lead to:
>> 
>> Caused by: java.lang.ClassCastException: org.apache.lucene.document.Field 
>> cannot be cast to java.lang.String
>>        at 
>> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getWikiReference(SolrEntityReferenceResolver.java:93)
>>  ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at 
>> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getEntityReference(SolrEntityReferenceResolver.java:70)
>>  ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at 
>> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:63)
>>  ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at 
>> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:46)
>>  ~[xwiki-platform-search-solr-api-7.4.jar:na]
>>        at 
>> org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:48)
>>  ~[xwiki-platform-search-solr-api-7.4.jar:na]
> 
> If you execute the test indicated in
> https://jira.xwiki.org/browse/XWIKI-13031?focusedCommentId=98082&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-98082
> you will get the following exception:
> 
> java.lang.ClassCastException:
> org.xwiki.model.internal.reference.ExplicitStringEntityReferenceResolver
> cannot be cast to java.lang.String
> 
> Sure you get the same stack trace from Exception type and methods
> calls point of view but what the message describe is a very different
> use case.

As I said evocrash replicates exactly the same stack trace, including the 
message. If you get something different it’s either because you’re executing on 
different code or whoever generated the test made a mistake copying it.

Thanks
-Vincent

> 
>> 
>>> * https://jira.xwiki.org/browse/XWIKI-14475: totally unreadable
>> 
>> Actually this test was not that hard to read for me. It just sets up some 
>> mocks. Evocrash can make it even simpler by presenting the mocks in a 
>> simpler way but that’s cosmetic and not the focus of evocrash just right now 
>> (it’s not the hard part). I’ve also raised something a bit related: 
>> https://github.com/STAMP-project/EvoCrash/issues/7
>> 
>> Feel free to raise new issues at 
>> https://github.com/STAMP-project/EvoCrash/issues/
>> 
>>> * https://jira.xwiki.org/browse/XRENDERING-422: not really useful but
>>> I doubt evocrash can do much about this kind of issue
>> 
>> Yes I agree.
>> 
>>> * https://jira.xwiki.org/browse/XWIKI-13196: this one could help a bit
>>> I guess but not sure it's really faster to execute evocash than
>>> looking at the code :)
>> 
>> Sure but it also give you the test which is nice. And depending on the line 
>> generating the NPE, it’s not always easy to know which variable is null by 
>> looking at the code.
>> 
>>> * https://jira.xwiki.org/browse/XWIKI-13916: that's a more complex way
>>> than it should to express that we call addTextAreaField for an already
>>> existing field yes it's useful
>> 
>> Thanks for the feedback.
>> 
>> There are plenty of limitations/issues with evocrash ATM. It’s good if we 
>> can identify cases where it generates stuff that could be useful. One 
>> direction that the devs are pursuing is looking at existing test classes to 
>> mimick how they’re written to produce the new tests. I don’t know how they’d 
>> be able to pull this off (that seems really hard to me ;)) but I know 
>> they’re working in this direction.
>> 
>> Thanks
>> -Vincent
>> 
>>> 
>>> On Fri, Apr 6, 2018 at 10:44 AM, Vincent Massol <vinc...@massol.net> wrote:
>>>> Hi devs,
>>>> 
>>>> Context
>>>> ======
>>>> 
>>>> Evocrash is a tool developed as part of the STAMP research project 
>>>> (https://www.stamp-project.eu/). Its goal is to take a stack trace (the 
>>>> use case is production stack traces) and generate a test that, when 
>>>> executed, reproduces the stack trace ;)
>>>> 
>>>> It’s using Guided Genetic Algorithm to simplify the search space (which is 
>>>> pretty cool, isn’t it? ;)). More info at 
>>>> https://github.com/STAMP-project/EvoCrash and http://www.evocrash.org/
>>>> 
>>>> Results
>>>> ======
>>>> 
>>>> So far evocrash was able to generate tests for the following issues (I’ve 
>>>> pasted the generated test code as comment):
>>>> * https://jira.xwiki.org/browse/XRENDERING-422
>>>> * https://jira.xwiki.org/browse/XWIKI-14475
>>>> * https://jira.xwiki.org/browse/XWIKI-13031
>>>> * https://jira.xwiki.org/browse/XWIKI-13196
>>>> * https://jira.xwiki.org/browse/XWIKI-13916
>>>> 
>>>> Using
>>>> =====
>>>> 
>>>> The use case I see is the following:
>>>> * A user reports a problem when using XWiki and create a jira issue with a 
>>>> stack trace
>>>> * An XWiki dev finds it and runs evocrash to generate a test reproducing 
>>>> the problem
>>>> * The XWiki dev uses the test to understand the reason (breakpoints can be 
>>>> put in the IDE) of the problem and writes a better test (which can be 
>>>> inspired by the test generated by evocrash or be completely different).
>>>> 
>>>> Questions
>>>> ========
>>>> 
>>>> The question we need to answer is whether we think it coud be useful or 
>>>> not. Are there cases where it can be useful?
>>>> 
>>>> For example if we imagine someone not knowing the xwiki code base well, 
>>>> maybe this can help him/her understand the bug in a simpler way than just 
>>>> having to read/find where the problem is in the code base?
>>>> 
>>>> I’ve started the work of trying to evaluate how useful it could be on the 
>>>> XWiki project by analyzing the examples above. I've only analyzed 2 ATM 
>>>> (see the links in the jira comments) and I’d be interested to have your 
>>>> analysis too.
>>>> 
>>>> WDYT?
>>>> 
>>>> Thanks
>>>> -Vincent
>>> 
>>> 
>>> 
>>> --
>>> Thomas Mortagne
>> 
> 
> 
> 
> -- 
> Thomas Mortagne

Reply via email to