[jira] [Commented] (KAFKA-1722) static analysis code coverage for pci audit needs
[ https://issues.apache.org/jira/browse/KAFKA-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291266#comment-14291266 ] Ashish Kumar Singh commented on KAFKA-1722: --- How to run: ./gradlew sonarRunner -PscalaVersion=2.11 Note that if you do not have sonarqube running on your system. The sonarRunner task will fail, but it would have generated coverage reports for core and clients at core/build/reports/scoverage/ and clients/build/reports/jacocoHtml respectively. Open index.html in any of those dirs to see the coverage. Once gradle-scoverage starts publishing scoverage report, a single report generated from sonar will be available. static analysis code coverage for pci audit needs - Key: KAFKA-1722 URL: https://issues.apache.org/jira/browse/KAFKA-1722 Project: Kafka Issue Type: Bug Components: security Reporter: Joe Stein Assignee: Ashish Kumar Singh Fix For: 0.9.0 Attachments: KAFKA-1722.patch Code coverage is a measure used to describe the degree to which the source code of a product is tested. A product with high code coverage has been more thoroughly tested and has a lower chance of containing software bugs than a product with low code coverage. Apart from PCI audit needs, increasing user base of Kafka makes it important to increase code coverage of Kafka. Something just can not be improved without being measured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1722) static analysis code coverage for pci audit needs
[ https://issues.apache.org/jira/browse/KAFKA-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14282051#comment-14282051 ] Ashish Kumar Singh commented on KAFKA-1722: --- Coverity does not support scala at all. Coverall only has a sbt plugin for scala. Using Coverall would have been really nice had their been a way to get scala coverage. Also I am not sure if coverall provides a way to manage multi module project with projects in different languages. As Kafka uses Gradle as a build tool and has most of its code in Scala, I do not think Coverall or Coverity will serve the purpose here. For the scope of this JIRA, I believe having a way to generate coverage manually should suffice. Automating it should not be a big deal once we have this. Instrumentation and scanning will definitely take extra time, but I do not think its significant. I am not sure how review becomes hard if you get additional info on code coverage. If a piece of code is optimized and is tested code coverage can only increase. static analysis code coverage for pci audit needs - Key: KAFKA-1722 URL: https://issues.apache.org/jira/browse/KAFKA-1722 Project: Kafka Issue Type: Bug Components: security Reporter: Joe Stein Assignee: Ashish Kumar Singh Fix For: 0.9.0 Code coverage is a measure used to describe the degree to which the source code of a product is tested. A product with high code coverage has been more thoroughly tested and has a lower chance of containing software bugs than a product with low code coverage. Apart from PCI audit needs, increasing user base of Kafka makes it important to increase code coverage of Kafka. Something just can not be improved without being measured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1722) static analysis code coverage for pci audit needs
[ https://issues.apache.org/jira/browse/KAFKA-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281995#comment-14281995 ] Ashish Kumar Singh commented on KAFKA-1722: --- [~bosco] coverall supports scala, but it will also have the limitations I mentioned above. For automative coverage report, I was planning to put it as part of preCommit patch testing. For each patch contributor can know if the patch is decreasing/ increasing code coverage. If the patch decreases code coverage more than a threshold value, preCommit patch testing bot will give it a -1. static analysis code coverage for pci audit needs - Key: KAFKA-1722 URL: https://issues.apache.org/jira/browse/KAFKA-1722 Project: Kafka Issue Type: Bug Components: security Reporter: Joe Stein Assignee: Ashish Kumar Singh Fix For: 0.9.0 Code coverage is a measure used to describe the degree to which the source code of a product is tested. A product with high code coverage has been more thoroughly tested and has a lower chance of containing software bugs than a product with low code coverage. Apart from PCI audit needs, increasing user base of Kafka makes it important to increase code coverage of Kafka. Something just can not be improved without being measured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1722) static analysis code coverage for pci audit needs
[ https://issues.apache.org/jira/browse/KAFKA-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14282002#comment-14282002 ] Don Bosco Durai commented on KAFKA-1722: There few things to note here: - Instrumentation and scanning takes significant amount time (at least in java) - There is a upfront cost to review and write rules to eliminate false positives - There is routine cost to eliminate false positives If we can setup this process, then it will be very ideal and beneficial. It would be good to have an build option to optionally run the scanning before committing the code. Also, by increase/decrease code coverage, do you mean by number of lines or issues? Because number of lines can decrease if a piece of code is optimized. static analysis code coverage for pci audit needs - Key: KAFKA-1722 URL: https://issues.apache.org/jira/browse/KAFKA-1722 Project: Kafka Issue Type: Bug Components: security Reporter: Joe Stein Assignee: Ashish Kumar Singh Fix For: 0.9.0 Code coverage is a measure used to describe the degree to which the source code of a product is tested. A product with high code coverage has been more thoroughly tested and has a lower chance of containing software bugs than a product with low code coverage. Apart from PCI audit needs, increasing user base of Kafka makes it important to increase code coverage of Kafka. Something just can not be improved without being measured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1722) static analysis code coverage for pci audit needs
[ https://issues.apache.org/jira/browse/KAFKA-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281977#comment-14281977 ] Ashish Kumar Singh commented on KAFKA-1722: --- After elaborate trials of various available tools and compatibility plugins, below is a brief summary. *We need to measure code coverage of following modules* # Core (in Scala, with a little Java code) # Clients (in Java) Other modules do not have tests. *Lang specific coverage tools* # Java, [JaCoCo|http://www.eclemma.org/jacoco/] appears to be a decent tool, which provides line and branch coverage. # Scala, [Scoverage|http://scoverage.org/] provides line and branch coverage. *Coverage summary* [SonarQube|http://www.sonarqube.org/] is a widely used tool that provides the capability to merge compatibility reports form various modules and present an overall report. Sonar uses plugins to parse and understand coverage report of an underlying sub-module of a project. A project can have sub-modules with different coverage tools, i.e., in different languages. We need following plugins for Kafka. # Sonar-Jacoco (v2.1) # Sonar-scoverage-plugin *Issues* # Sonar-socverage-plugin depends on [scalac-scoverage-plugin|https://github.com/scoverage/scalac-scoverage-plugin]. scalac-scoverage-plugin can be used in a gradle project using [gradle-scoverage|https://github.com/scoverage/gradle-scoverage]. gradle-scoverage,as of now, only publishes html and cobertura report. However, scalac-scoverage-plugin needs scoverage report to be able to parse it. In short, sonar can not report coverage for scala project as of now. A full coverage report does get generated for scala project, but it would not show up in overall report. I have discussed this with the collaborators of gradle-scoverage and they are working on it. # Scala 2.10 is not supported by scalac-scoverage-plugin, [detailed discussion|https://github.com/scoverage/scalac-scoverage-plugin/blob/master/2.10.md]. *OK, so where do we stand* We can generate coverage reports, with line and branch coverage included, for core and clients sub modules. We can generate a sonar summary report for the project, but that will only include coverage of clients sub module. Coverage report, web report, for core module will have to be browsed separately. As soon as gradle-scoverage start publishing scoverage report, we can see core's coverage as well in the sonar summary report. If this sounds ok then I can provide a patch. static analysis code coverage for pci audit needs - Key: KAFKA-1722 URL: https://issues.apache.org/jira/browse/KAFKA-1722 Project: Kafka Issue Type: Bug Components: security Reporter: Joe Stein Assignee: Ashish Kumar Singh Fix For: 0.9.0 Code coverage is a measure used to describe the degree to which the source code of a product is tested. A product with high code coverage has been more thoroughly tested and has a lower chance of containing software bugs than a product with low code coverage. Apart from PCI audit needs, increasing user base of Kafka makes it important to increase code coverage of Kafka. Something just can not be improved without being measured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1722) static analysis code coverage for pci audit needs
[ https://issues.apache.org/jira/browse/KAFKA-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281981#comment-14281981 ] Don Bosco Durai commented on KAFKA-1722: Ashish, Coverity is another option. They are free for open source projects. I have been scanning for most of the Hadoop projects. There is already a project created for Kafka (https://scan.coverity.com/projects/1340). I am not sure who is the owner, but if you want I can investigate that path. I had checked with Coverity before and they don't support Scala yet. So it will be only for the java components. static analysis code coverage for pci audit needs - Key: KAFKA-1722 URL: https://issues.apache.org/jira/browse/KAFKA-1722 Project: Kafka Issue Type: Bug Components: security Reporter: Joe Stein Assignee: Ashish Kumar Singh Fix For: 0.9.0 Code coverage is a measure used to describe the degree to which the source code of a product is tested. A product with high code coverage has been more thoroughly tested and has a lower chance of containing software bugs than a product with low code coverage. Apart from PCI audit needs, increasing user base of Kafka makes it important to increase code coverage of Kafka. Something just can not be improved without being measured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1722) static analysis code coverage for pci audit needs
[ https://issues.apache.org/jira/browse/KAFKA-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281984#comment-14281984 ] Ashish Kumar Singh commented on KAFKA-1722: --- If we are open to use something like that, then I guess [coverall|https://coveralls.io/] is a better option. static analysis code coverage for pci audit needs - Key: KAFKA-1722 URL: https://issues.apache.org/jira/browse/KAFKA-1722 Project: Kafka Issue Type: Bug Components: security Reporter: Joe Stein Assignee: Ashish Kumar Singh Fix For: 0.9.0 Code coverage is a measure used to describe the degree to which the source code of a product is tested. A product with high code coverage has been more thoroughly tested and has a lower chance of containing software bugs than a product with low code coverage. Apart from PCI audit needs, increasing user base of Kafka makes it important to increase code coverage of Kafka. Something just can not be improved without being measured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1722) static analysis code coverage for pci audit needs
[ https://issues.apache.org/jira/browse/KAFKA-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281992#comment-14281992 ] Don Bosco Durai commented on KAFKA-1722: coverall also seems to be good. It says on it's website that it supports scala. Not sure to what level. Have you thought about automating the build and submission? Coverity can be be integrated with Travis CI, so it is easy to schedule the build and have results shared with everyone. static analysis code coverage for pci audit needs - Key: KAFKA-1722 URL: https://issues.apache.org/jira/browse/KAFKA-1722 Project: Kafka Issue Type: Bug Components: security Reporter: Joe Stein Assignee: Ashish Kumar Singh Fix For: 0.9.0 Code coverage is a measure used to describe the degree to which the source code of a product is tested. A product with high code coverage has been more thoroughly tested and has a lower chance of containing software bugs than a product with low code coverage. Apart from PCI audit needs, increasing user base of Kafka makes it important to increase code coverage of Kafka. Something just can not be improved without being measured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1722) static analysis code coverage for pci audit needs
[ https://issues.apache.org/jira/browse/KAFKA-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268377#comment-14268377 ] Ashish Kumar Singh commented on KAFKA-1722: --- [~joestein], the growing user and contributor community of Kafka makes this Jira very important. If it is OK with you, I would like to take a stab at it. static analysis code coverage for pci audit needs - Key: KAFKA-1722 URL: https://issues.apache.org/jira/browse/KAFKA-1722 Project: Kafka Issue Type: Bug Components: security Reporter: Joe Stein Fix For: 0.9.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
