On Fri, 14 Jul 2017 17:19:36 +0300
Andrey Karpov <kar...@viva64.com> wrote:
Hello Carsten,
After publishing my article, several unreasonable news appeared on
the Internet. That's why, I'd like to note that in my article I
didn’t write about the bad/good quality of Tizen code or that
PVS-Studio analyzer is magic, best of the best. I only gave those
numbers that I’d got. I can't talk about the quality of Tizen code,
since I have insufficient data for this purpose. I understand clearly
what you are talking about and agree with you.
Indeed that is one reason I ask for numbers that can be compared
because when you publish some number, and that number is large (in this
case due to the fact that codebase you extrapolated for is large) ... a
lot of people don't see the forest from the trees.
Don't get me wrong. Code quality is important. Fixing bugs is
important. Tools to point out "here is a potential issue" are very
useful. But you do need to stand back and see the possibly urgency in
light of everything else and that requires comparative numbers.
My objective was to show that despite the already used techniques,
PVS-Studio analyzer can help to make Tizen code better and more
reliable. As I think, I managed to demonstrate this by pointing to
the 900 fragments of code, which, in my opinion, deserve attention,
fixing and refactoring.
At this point, for me at least, the jury is out on PVS Studio. We have
internal tools and we're using those first at this point on Tizen's
codebase. If PVS Studio is better or not than those tools, I'm not
going to comment, as I am not in a good position to do so.
For the upstream projects tizen depends on that I look after, I'm VERY
familiar with Coverity scan and to a lesser extent clang's static
analyser. I can say the text reports PVS studio produce pale in
comparison to what Coverity provides in the Web UI. It provides code
flow analysis (what code path was used to get there) which is very
useful in learning the "why", A full browsable view of the code (so I
don't keep having to match it back to the source file/line when
reading), and a collaborative environment where many deveopers can work
together live on the issue list and see what is triaged by who, whenand
a log about it, BUT it offers something I have not seen so far in the
information you provided for PVS Studio on the blog references here,
... the ability to say "No. False positive. Tool - you're wrong. Here's
why". Coverity lets you do this... which lets you move on and not
modify code that works fine and is correct.
My experience shows that a decent percentage of "fixes for warnings
from coverity" lead to ADDING bugs. I've seen it happen many times. It
made things WORSE. It went from a "theoretical bug but actually due to
environment will never happen" to "I wake up, update code and suddenly
app x, y, z are broken in some way and it was a fix for a warning".
This is also why I'm wary of having people unfamiliar with code "fix
it" based on such warnings. Even people who know the code well mess up.
Unfortunately, the question "Include this as a bugs per 1 k lines of
code or similar metric?" is not very clear.
It's simple. "This code has an average of 0.123 bugs per 1k lines of
code". But measure that on everything you analyse. You at least have a
ROUGH measuring stick. They may be all major exploitable issues, maybe
all minor unlikely ones or maybe some combination so a rating system
ultimately would score them based on that, BUT let's keep it simple. If
you quote a number, quote numbers for everything you look at and to
make it comparable... make it proportional to size.
In my opinion, the article presents all the necessary data. I've got:
* The density of detected errors in code (c) 2015 Samsung
Electronics: 0.41 errors on 1000 lines of code.
Your numbers say it's 0.375. 900 in 2.4 million lines of code.
900/2400 = 0.375. The 0.41 is including false positives? Now we have a
report disagreeing with itself. It's small, but important.
OK - found it now. I missed that paragraph towards the end (it is a
long article), but these numbers disagrees with others you give right at
the top and in the headline (the headline number is an extrapolation).
Why do your numbers differ?
* The density of detected errors in the third-party libraries: 0.36
errors on 1000 lines of code.
(I did not consider comments as the lines of code).
Can these data be incorrect? Yes, they can. This is not a scientific
research, this is a demonstration on practice that the tool may be
useful.
Moreover, some errors, in Tizen developers’ opinion, canbe not that
erroneous. Well, at least, there is no sense to fix them. Then the
density of detected errors will diminish.
This is another issue entirely - if the error is "worth worrying about".
On the other hand, I might highlight not all the errors. I approached
to the study of the report very carefully, but without bigotry. For
example, I was a bit lazy to study the warnings V730 -
https://www.viva64.com/en/w/V730/. This is a very time consuming and
thankless work, when you work with someone else's code. All the time
it is unclear, if it is dangerous or not, that some class member has
been left uninitialized. It is a tedious long labour that needs to be
done carefully. So, perhaps, with more careful reviewing of the log,
other errors can be found.
Indeed it is time consuming to examine them all. I think you did a good
job at explaining what errors are there and why in the general sense
these are potential issues.
About the comparison with the quality of other projects... Difficult
question. I ask to understand, that while writing the articles we
don’t have the aim to compare which code is better or
worse.Therefore, we usually stop when found enough of quite
The problem is - when you publish numbers like you did, it becomes a
comparison thing. That's how people respond. Headlines like "27000
bugs found in Tizen!" are very much designed to catch attention with
numbers but when there is no context provided... the only thing people
see is a headline.
interesting bugs for writing an article. Performing the careful
analysis of all warnings for a large project will take a lot of time.
I also ask to consider that it is difficult and long to deal with
unknown code. Therefore, we sometimes mention about density of errors
only for small projects, since is not very difficult to view the
whole report. Examples:
* Notepad++: we detect about 2 errors per 1000 lines of code.
https://www.viva64.com/en/b/0511/
WOW. that's a lot. Thanks for the number.
* Far Manager for Linux: we detect about 0.464 errors per 1000 lines
of code. https://www.viva64.com/en/b/0478/
Thanks. your numbers are in line with what coverity indicates too.
* Tor project: we do not find anything. Density
0.https://www.viva64.com/en/b/0507/
<https://www.viva64.com/en/b/0507/>
Indeed that is good. I also noted openssl last i checked on coverity
scan had a density of 0 too.
As we can see, the results are different. However, it seems to me
they are not worth to concentrate on. Static analyzer is a tool of
They are a tool for prioritization. As I said - bugs are bugs and
should be fixed. Knowing if your bug count is particularly bad or now
compared to "the industry at large" etc. lets you know how much effort
or money or time to spend on these issues.
I have taken a position in upstream projects to just slowly fix the
reported issues over a long period of time. every release - fix some
more so every release has the bug rate go down. So you mix feature
addition and static analysis triage, but focus on the feature
development etc. and less so on triage.
finding bugs in fresh code. Yes, old mistakes are also worth to be
fixed, but generally they are not as critical as new ones. Actually,
if the error is in the code for several years,it means that it rarely
reveals itself or interferes no one. That’s why it is interesting to
look to the future rather than the past. Sure, the PVS-Studio
analyzer can be a good assistant for a programmer.
Indeed I agree there.
About the percent of false positives. It makes no sense to talk about
them without first configuring the analyzer. It's a lot of work,
which we are ready to engage, if a cooperation begins someday. Can we
deal with it? Yes, we can:
That sounds like a fair bit of work.
*
https://www.unrealengine.com/en-US/blog/how-pvs-studio-team-improved-unreal-engines-code
*
https://www.unrealengine.com/en-US/blog/static-analysis-as-part-of-the-process
I think when it comes to projects of such a large size as Tizen, it
makes sense to speak not only on product licensing, but also on a
great support, carried out by our team.
P.S. On Monday I will demonstrate that the analyzer can be useful not
only for finding bugs, but also regarding micro-optimizations. :)
Actually ... that does interest me. That's a new use of static
analysis. I think that this would be generally well received in Tizen
too. If PVS Studio can do this too... it just bumped up in value for
me. I'd like to see what these reports are and what they point out etc.
----
Best regards,
Andrey Karpov, Microsoft MVP,
Ph.D. in Mathematics, CTO
"Program Verification Systems" Co Ltd.
On 14.07.2017 4:35, Carsten Haitzler wrote:
On Thu, 13 Jul 2017 14:26:35 +0300
Andrey Karpov <kar...@viva64.com> wrote:
Could you:
1. Include this as a bugs per 1k lines of code or similar metric?
Total bugs is not that useful without knowing total size of code
looked at. At least in the summary.
2. Include metrics calculated similarly for other major projects
(Linux kernel, etc. etc.).
Why? The below is like saying "you're doing 120km/h!!!!!!" ... but
if it's on a freeway and the speed limit is 130km/h ... in context
it's very different. This here lacks context.
As I haven't used PVS studio before (it's on a list of things to try
out and see if it's good), but I do know Coverity's scan service
very well, I'll do some back of a napkin numbers:
1. In my experience about ~10-15% of bugs are false positives etc.
with coverity.
2. Coverity says Linux kernel gets 0.48 issues peer 1k lines of
code. applying the above false positive rate, let's call that 0.40.
Qt gets 0.72, so lets call that 0.61 adjusting for false positives.
Glib gets 0.45, so 0.38 accounting for false positives. So:
With your numbers, Tizen sees 900 issues in 2.4 million lines of
code. that comes out at 0.38.
Linux kernel = 0.40
Qt = 0.61
Glib = 0.38
Tizen = 0.38
Yes PVS studio is a different tool to coverity. I'm making an
assumption (much like you do too in many ways) that these two tools
are in the same ballpark and will report similar issues and
numbers, but may be disjoint sets. I'm going with this assumption
because you didn't provide other numbers to go by, and it'd be nice
to.
My conclusion is that Tizen code quality is pretty decent in the
scheme of things. It's bug rate is pretty low-ish.
Now on the other side, it';s always great to have tools point out
possible errors. Another tool is another weapon in a war chest to
improve code quality. That's a good thing. Bugs should be looked
into and addressed accordingly based on actual severity and
context. just blindly fixing issues will result in misallocation of
time and resources because it may be an issue in a debug tool that
is rarely used and only for gathering quick information by a
developer when something goes wrong... it may be a seriously
exploitable bug in code that is always able to be triggered
remotely. So context is important. Knowing issues are there and
what a tool thinks they are is a great speedup vs full code review.
PVS Studio is indeed such a tool. There are others too. We have
tools of our own we're using more and more.
Hello All,
This article will demonstrate that during the development of large
projects static analysis is not just a useful, but a completely
necessary part of the development process. This article is the
first one in a series of posts, devoted to the ability to use
PVS-Studio static analyzer to improve the quality and reliability
of the Tizen operating system. For a start, I checked a small part
of the code of the operating system (3.3%) and noted down about
900 warnings pointing to real errors. If we extrapolate the
results, we will see that our team is able to detect and fix about
27000 errors in Tizen. Using the results of the conducted study, I
made a presentation for the demonstration to the Samsung
representatives with the offers about possible cooperation. The
meeting was postponed, that is why I decided not to waste time and
transform the material of the presentation to an article:
https://www.viva64.com/en/b/0519/
----
Best regards,
Andrey Karpov, Microsoft MVP,
Ph.D. in Mathematics, CTO
"Program Verification Systems" Co Ltd.
_______________________________________________
Dev mailing list
Dev@lists.tizen.org
https://lists.tizen.org/listinfo/dev
