Dear Ratis developers,
My name is Helge, I am a researcher at IT University of Copenhagen [1].
I am currently conducting a study on the impact of continuous code quality
assessment tools (SonarQube) on defects.
I am writing to you -the Ratis developers-, since I found that Ratis is the
only ASF project, for which the amount of reported defects, i.e., bugs in Jira,
are significantly reduced after SonarCloud was introduced to development.
The first SonarCloud assessment of Ratis is from 2020-05-28 [2].
In Jira, on average 4 defect reports per week (median 3) in the year before the
first SonarCloud assessment drop to on average 2 defect reports per week
(median 2) in the year after. That is impressive. I would like to understand if
the reduction of defect reports is caused by introduction of SonarCloud to
development or if this is just a coincidence.
Therefore, I have some questions for which I would really appreciate your
feedback:
1) Do you -the Ratis developers- believe that the reduction of defect reports
on Jira is actually related to your work on remediation of SonarCloud issues
(code smells, vulnerabilities, or bugs)?
1.2) If not, what do you believe is the reason for decreasing defect
reports?
2) Is it correct that the Ratis project did not use SonarQube (the one hosted
by the Apache Infrastructure Team) before using SonarCloud?
3) SonarCloud issues (code smells, vulnerabilities, or bugs) are generally
decreasing (https://sonarcloud.io/project/activity?id=apache-ratis). Is that
due to:
3.1) Your active work on remedying these SonarCloud issues?
3.2) Changes of the applied SonarQube quality profile?
3.3) A change of your coding to respect SonarQube rules proactively?
Thank you in advance for your feedback and consideration. I will share the
results of my work with you as soon they are written down in a presentable
format. Below, I present preliminary insights of my current analysis.
Best regards,
Helge
## Some insights...
I find 16 Jira issues that mention `[Ss]onar` and that are resolved[3]. For
these, I find 15 commits[4] that address and reference these Jira issues. In
between the first SonarCloud assessment (2020-05-28) and yesterday (2021-06-14)
there are 315 commits to the repositories main branch (up to version
[https://github.com/apache/ratis/commit/a7315511bddcc12dc4acea25f2d0d71608a8591c](a73155)).
That is, 15 out of 315 commits (ca. 5%) are related to SonarCloud and its code
quality assessments.
I am hesitant to believe that that these 15 commits can have such a strong
impact to the decreasing amount of defect reports in Jira, in particular since
four of them are about configuring and setting-up SonarCloud.
Also, I believe that these commits cannot explain the generally decreasing
trends of code smells, bugs, and vulnerabilities in SonarCloud's dashboard[5].
Especially, since Ratis is growing with 9905 lines from 2020-05-28 to
2021-06-14 and more lines in software usually mean higher chance of
introduction of defects and violation of SonarCloud issues.
I find no commits that mention code `[Ss]mell`s or `[Vv]ulnerabilities` in the
period from 2020-05-28 to 2021-06-14. That is, I believe there are no commits
that address SonarCloud code smells or vulnerabilities. From 2020-05-28 to
2021-06-14, I find 10 commits that mention the term `[Bb]ug`[6]. After
inspecting all of these and the respective Jira issues, I believe that they do
not fix bugs that are reported by SonarCloud but bugs in the sense of defects.
-------------------
[1] https://www.itu.dk/people/ropf/ and
https://www.researchgate.net/profile/Helge-Pfeiffer-2
[2]
https://sonarcloud.io/project/activity?id=apache-ratis&selected_date=2020-05-28T10%3A30%3A20%2B0000
[3] Resolved issues that mention `[Ss]onar` or that are a sub-task of one of
these (ordered by Jira ID):
* https://issues.apache.org/jira/browse/RATIS-940
* https://issues.apache.org/jira/browse/RATIS-948
* https://issues.apache.org/jira/browse/RATIS-950
* https://issues.apache.org/jira/browse/RATIS-953
* https://issues.apache.org/jira/browse/RATIS-1051
* https://issues.apache.org/jira/browse/RATIS-1052
* https://issues.apache.org/jira/browse/RATIS-1053
* https://issues.apache.org/jira/browse/RATIS-1054
* https://issues.apache.org/jira/browse/RATIS-1055
* https://issues.apache.org/jira/browse/RATIS-1075
* https://issues.apache.org/jira/browse/RATIS-1306
* https://issues.apache.org/jira/browse/RATIS-1311
* https://issues.apache.org/jira/browse/RATIS-1364
* https://issues.apache.org/jira/browse/RATIS-1365
* https://issues.apache.org/jira/browse/RATIS-1366
* https://issues.apache.org/jira/browse/RATIS-1367
[4] Commits that refer to a resolved `[Ss]onar` related issue:
* RATIS-940:
https://github.com/apache/ratis/commit/00f1747a1915f42fd256e9a5b7b8e0131cd06c2c
* RATIS-948:
https://github.com/apache/ratis/commit/a2f3895396a81ee6e31d6fd1a8a6c8a7bf121dd6
* RATIS-950:
https://github.com/apache/ratis/commit/43a042a8bbe123bcb5e567af0aeced12eb299290
* RATIS-953:
https://github.com/apache/ratis/commit/02caace296f4414de3eda9f4469dbd806ca594b1
* RATIS-1075:
https://github.com/apache/ratis/commit/9b1d2c18f5677f3d443344e69f98e6c3ac953835
* RATIS-1051:
https://github.com/apache/ratis/commit/cf9ed0864615e82083f6be5f4f958563dbf242ad
* RATIS-1052:
https://github.com/apache/ratis/commit/0d964950e8fdf6887e53e8eb53695a14d29314c9
* RATIS-1052:
https://github.com/apache/ratis/commit/6dd51454994a843fde10933d2ec3daedd6fb36de
* RATIS-1055:
https://github.com/apache/ratis/commit/61f038b6bea2934e910e8504df0d8c79d6d34799
* RATIS-1306:
https://github.com/apache/ratis/commit/9d1b711f9d4606145e19a28270b0c50b04dc869b
* RATIS-1311:
https://github.com/apache/ratis/commit/87bd1fd1df9f02e83b973291d507ad002bd9d3f4
* RATIS-1364:
https://github.com/apache/ratis/commit/9577d564eeac36cf449843d34b7010b33d634818
* RATIS-1365:
https://github.com/apache/ratis/commit/ff8aa668f1a0569ba5e6b0f30dbd51a673913344
* RATIS-1366:
https://github.com/apache/ratis/commit/d65ca26a0291fc6067f860eff4ff3092d25c0aec
* RATIS-1367:
https://github.com/apache/ratis/commit/040bc52e19a5e36f5710ccd4fc1981e862e691e8
[5] https://sonarcloud.io/project/activity?id=apache-ratis
[6] Commits that mention `[Bb]ug`s in the period 2020-05-28 to 2021-06-14:
*
https://github.com/apache/ratis/commit/af358415cc8ebc70665d73e5fc4812b7cb669c75
*
https://github.com/apache/ratis/commit/32016e4a40ae44bb3c3718880327bfb085bff509
*
https://github.com/apache/ratis/commit/86dd7fa68081e33428ed65dfdb613032b1378560
*
https://github.com/apache/ratis/commit/e5a052c18766293b94cc89e3a34a469193009537
*
https://github.com/apache/ratis/commit/79223f89054109d2c499d81a149e60c15fc90453
*
https://github.com/apache/ratis/commit/17a2198b0016a66494da4344d218593ed2c4d7de
*
https://github.com/apache/ratis/commit/f51196455679443be438a1ddb04bd6860f4ad6dc
*
https://github.com/apache/ratis/commit/cfa6c4f70e8dc296c921b7c8b33511d58270ab39
*
https://github.com/apache/ratis/commit/195c572024fc5fd88d00c3c0985c01215d2719aa
*
https://github.com/apache/ratis/commit/9fc6a1163b993177fd884bba4212326bb6abbf73
