Re: About static analyzers on some various projects

[Josh Matthews]

On 2015-09-25 10:06 AM, Robert O'Callahan wrote:

On Sat, Sep 26, 2015 at 1:46 AM, Ehsan Akhgari 
wrote:


Our static analysis builds can be easily triggered from the try server
(although I have been unable to get anyone interested to fix bug 1116518 to
make those builds happen on the try server by default, which makes it all
too easy for people to forget to turn on these builds from their trychooser
syntax) so as soon as the code is pushed to try (perhaps through autoland)
the check failures will be visible as build failures.  I'm not quite sure
what it would take to get those build failures to appear in MozReview but
it should be possible.



The tricky bit is to determine which failures were introduced by the patch,
and just display those, and display them in the context of the patch in
some way that makes sense.

Rob



If the static checkers are not a big resource drain, we could just run 
it on the parent revision as well and diff the output. Alternatively, 
store the results of such runs against our non-try branches somewhere 
where the results are easily accessible, then again fetch and diff.


Cheers,
JOsh
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: About static analyzers on some various projects

[Justin Dolske]

On 9/25/15 7:06 AM, Robert O'Callahan wrote:


[...]I'm not quite sure
what it would take to get those build failures to appear in MozReview but
it should be possible.



The tricky bit is to determine which failures were introduced by the patch,
and just display those, and display them in the context of the patch in
some way that makes sense.


Yeah, I think this needs a solution to be effective.

*begin thousand yard stare*

In a previous life at Sun, all new code was required to pass lint and 
cstyle checks, and it was strictly enforced. Of course, tons of existing 
code did not pass, and so it became an enormous PITA to diff piles of 
output.


At Mozilla, it seems like previous discussions on this kind of thing 
(style and warnings come to mind) have dealt with this at a 
file/directory/module level... Someone fixes up a thing completely, adds 
it to a whitelist, and then it's a simple pass/fail. Could that work 
here too?


Justin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: About static analyzers on some various projects

[Ehsan Akhgari]

On Fri, Sep 25, 2015 at 10:06 AM, Robert O'Callahan 
wrote:

> On Sat, Sep 26, 2015 at 1:46 AM, Ehsan Akhgari 
> wrote:
>
> Our static analysis builds can be easily triggered from the try server
>> (although I have been unable to get anyone interested to fix bug 1116518 to
>> make those builds happen on the try server by default, which makes it all
>> too easy for people to forget to turn on these builds from their trychooser
>> syntax) so as soon as the code is pushed to try (perhaps through autoland)
>> the check failures will be visible as build failures.  I'm not quite sure
>> what it would take to get those build failures to appear in MozReview but
>> it should be possible.
>>
>
> The tricky bit is to determine which failures were introduced by the
> patch, and just display those, and display them in the context of the patch
> in some way that makes sense.
>

The way that the existing static analysis bots work, those will simply be
all failures.  :-)

-- 
Ehsan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: git-bz-moz and Bugzilla 2 factor authentication

[Andrew McCreight]

I have now updated the git-bz-moz and moz-git-tools repositories with my
REST API rewrite. Let me know if you have any problems or feedback (I'm
mccr8 on IRC). Thanks to everybody who gave me feedback, and to froydnj who
looked over the many patches.

Andrew

On Wed, Sep 23, 2015 at 1:49 PM, Andrew McCreight 
wrote:

> I've updated my repository for my git-bz REST API branch:
>   https://github.com/amccreight/git-bz-moz/tree/RestAPI
>
> I think everything is fixed now, so please try it out and let me know if
> you encounter problems or something seems slow. (There's some initial
> caching thing that takes a while, but it should show a message and doesn't
> happen very often.) You have to set up an API key, but thanks to feedback
> from people it now spams your console with instructions on how to do that
> if you don't have one set.
>
> I removed |git bz edit --fix| and |git bz push|, as well as |git edit
> --pushed| though as far as I can tell the latter doesn't actually do
> anything. Hopefully nobody uses those given that we have Pulsebot.
>
> One nice feature of the new version (inherited with little effort by me
> from bzexport) is that if the user you have set for a r? etc. flag is
> ambiguous it will present a menu for you to select a user rather than just
> exiting immediately.
>
> I'll submit a pull request in the next few days to the main git-bz-moz
> repo in the next few days barring anybody finding some problems. Nathan
> Froyd has graciously offered to review this stack of 28 patches. (Though he
> volunteered before he knew it was that many patches.)
>
> The current set of patches copies over a few files from the
> version-control-tools repository. The next step will be to properly
> integrate those files, maybe via a subrepo, until git-bz-moz gets moved
> into that repo per gps's plan. [1]
>
> [1]
> https://groups.google.com/forum/#!topic/mozilla.dev.version-control/z9z9N7_vWiY
>
> Andrew
>
>
>
> On Tue, Sep 15, 2015 at 2:37 PM, Andrew McCreight 
> wrote:
>
>> Apparently Bugzilla 2fa breaks the weird cookie authentication method
>> that git-bz-moz and bzexport use. I think I've read that this is a bugzilla
>> bug, but in the meanwhile I've been working on making git-bz-moz use the
>> Bugzilla backend of bexport, which is less hacky and supports using API
>> keys for authentication. Setting up an API key is extremely easy.
>>
>> If you want to try it out, it is available from my github repo here:
>>   https://github.com/amccreight/git-bz-moz/tree/RestAPI
>>
>> git bz edit and git bz push don't work, but the rest should, so please
>> file issues at https://github.com/mozilla/git-bz-moz or email me if you
>> notice them. (Oh, I think setting reviewer flags for Firefox product bugs
>> doesn't work, is a known issue.) The patches are still a little hacky and
>> probably don't cache as much as they should.
>>
>> Andrew
>>
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: NPAPI plug-in use case: live video broadcast

[lietz]

On Saturday, September 19, 2015 at 8:15:50 PM UTC+2, Eric Rescorla wrote:
> On Sat, Sep 19, 2015 at 11:09 AM, Oliver Lietz  wrote:
> 
> > Hi,
> > our nanoStream plugin supports live encoding and streaming with
> > h264/aac/rtmp from live camera sources and capture devices.
> > We needed to replace this with a native extension on Chrome.
> > WebRTC is a possible future option but not a suitable replacement for all
> > use cases.
> >
> 
> One or both of WebRTC and MSE is intended to cover precisely these
> use cases. What are they missing here?
> 
> -Ekr
> 
> 
> NPAPI stills works on Firefox, but how long will it remain?
> > What options do we have in the future for implementing native extensions?
> > Thanks, Oliver
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >

MSE is playback only, so no option for live broadcast.
WebRTC is VP9 and a peer protocol (RTP) not compatible to existing streaming 
environments (RTMP, HLS). It requires a proxy/transcoder. It also does not 
allow local recording in mp4 properly, and the codec details cannot be 
accessed. Many other features lacking which would be possible in native 
capture/encoding/streaming code.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: About static analyzers on some various projects

[Nicholas Nethercote]

On Fri, Sep 25, 2015 at 11:46 PM, Ehsan Akhgari  wrote:
>
> Our static analysis builds can be easily triggered from the try server
> (although I have been unable to get anyone interested to fix bug 1116518 to
> make those builds happen on the try server by default, which makes it all
> too easy for people to forget to turn on these builds from their trychooser
> syntax)

But static analysis does run if you do a |-p all| build. Speaking for
myself, almost every try push I do is |-p all| because I've broken the
tree enough times to have learned my lesson :)

Nick
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: NPAPI plug-in use case: live video broadcast

[Robert O'Callahan]

On Sat, Sep 26, 2015 at 10:20 AM,  wrote:

> MSE is playback only, so no option for live broadcast.
>

What do you mean? There doesn't seem to be any reason why MSE can't do live
broadcast.

If you're using HLS, you can use MSE.
https://github.com/dailymotion/hls.js


> WebRTC is VP9 and a peer protocol (RTP) not compatible to existing
> streaming environments (RTMP, HLS).

It requires a proxy/transcoder. It also does not allow local recording in
> mp4 properly,


FWIW Firefox for Android and FirefoxOS support recording WebRTC streams to
MP4 locally. Desktop Firefox supports recording to WebM. Do you have a
use-case for recording locally to MP4 on desktop?

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: About static analyzers on some various projects

[Robert O'Callahan]

On Sat, Sep 26, 2015 at 7:34 AM, Ehsan Akhgari 
wrote:

> On 2015-09-25 12:01 PM, Justin Dolske wrote:
>
>> At Mozilla, it seems like previous discussions on this kind of thing
>> (style and warnings come to mind) have dealt with this at a
>> file/directory/module level... Someone fixes up a thing completely, adds
>> it to a whitelist, and then it's a simple pass/fail. Could that work
>> here too?
>>
>
> Yes, that is another option for checks that tons of existing code don't
> pass.


That'll help some.

The problem of attributing new errors to the correct part of the diff
remains, and is quite interesting.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: NPAPI plug-in use case: live video broadcast

[Eric Rescorla]

On Fri, Sep 25, 2015 at 3:20 PM,  wrote:

> On Saturday, September 19, 2015 at 8:15:50 PM UTC+2, Eric Rescorla wrote:
> > On Sat, Sep 19, 2015 at 11:09 AM, Oliver Lietz  wrote:
> >
> > > Hi,
> > > our nanoStream plugin supports live encoding and streaming with
> > > h264/aac/rtmp from live camera sources and capture devices.
> > > We needed to replace this with a native extension on Chrome.
> > > WebRTC is a possible future option but not a suitable replacement for
> all
> > > use cases.
> > >
> >
> > One or both of WebRTC and MSE is intended to cover precisely these
> > use cases. What are they missing here?
> >
> > -Ekr
> >
> >
> > NPAPI stills works on Firefox, but how long will it remain?
> > > What options do we have in the future for implementing native
> extensions?
> > > Thanks, Oliver
> > > ___
> > > dev-platform mailing list
> > > dev-platform@lists.mozilla.org
> > > https://lists.mozilla.org/listinfo/dev-platform
> > >
>
> MSE is playback only, so no option for live broadcast.
> WebRTC is VP9 and a peer protocol (RTP) not compatible to existing
> streaming environments (RTMP, HLS).


Firefox WebRTC supports H.264.



> It requires a proxy/transcoder. It also does not allow local recording in
> mp4 properly, and the codec details cannot be accessed.


I don't know what this means about codec details.



> Many other features lacking which would be possible in native
> capture/encoding/streaming code.
>

Can you elaborate what those features are.

In any case, this is the set of use cases WebRTC is intended to support and
so
the right direction to be going is to work with WebRTC (potentially helping
us
enhance WebRTC to support your applications) rather than doubling
down on NPAPI.

-Ekr
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: NPAPI plug-in use case: live video broadcast

[Jonas Sicking]

On Sep 26, 2015 09:33, "Eric Rescorla"  wrote:
>
> On Fri, Sep 25, 2015 at 3:20 PM,  wrote:
>
> > On Saturday, September 19, 2015 at 8:15:50 PM UTC+2, Eric Rescorla
wrote:
> > > On Sat, Sep 19, 2015 at 11:09 AM, Oliver Lietz 
wrote:
> > >
> > > > Hi,
> > > > our nanoStream plugin supports live encoding and streaming with
> > > > h264/aac/rtmp from live camera sources and capture devices.
> > > > We needed to replace this with a native extension on Chrome.
> > > > WebRTC is a possible future option but not a suitable replacement
for
> > all
> > > > use cases.
> > > >
> > >
> > > One or both of WebRTC and MSE is intended to cover precisely these
> > > use cases. What are they missing here?
> > >
> > > -Ekr
> > >
> > >
> > > NPAPI stills works on Firefox, but how long will it remain?
> > > > What options do we have in the future for implementing native
> > extensions?
> > > > Thanks, Oliver
> > > > ___
> > > > dev-platform mailing list
> > > > dev-platform@lists.mozilla.org
> > > > https://lists.mozilla.org/listinfo/dev-platform
> > > >
> >
> > MSE is playback only, so no option for live broadcast.

MSE can be used to play live broadcast. I'm willing to bet that that is
what YouTube uses for their live broadcast support.

/ Jonas
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: About static analyzers on some various projects

[Ehsan Akhgari]

On 2015-09-25 5:35 AM, Sylvestre Ledru wrote:

Le 24/09/2015 23:29, Ehsan Akhgari a écrit :

On 2015-09-24 1:41 PM, Sylvestre Ledru wrote:

= Static analyzers =
For now, we are running:
* Coverity, a proprietary tool with a great (but slow) web interface. As
Firefox is Free software, the service is provided for free
but with a restriction in term of number of build. Now, the analysis is
launched once a week on Monday. Supports C, C++ & Java.
A few improvements will be made to silent some of the defects.


Does anybody look at these regularly?

I am looking at the weekly reports. I am reporting the issue I can confirm.
However, to be honest, I am not technically able to analyze every one of them.
I am also tagging false positive to keep a clean database.

FYI, at some point, we might have someone to help on this full time.


That would be nice, although I'm not sure if we need one full time 
position just for this purpose!  But having someone who tracks these 
issues is definitely good.



I would be interested to know if they produce high quality results these days.  
My past experience with Coverity has been that it's full of false positivies.

Several answers:
* I think the results are still pretty much the same
* false positives can be silent. This is a work to be done either in our code 
(you reviewed some of my patches for this in the past)
or in coverity
* some checkers have a small false positives ratio, some other, an higher.


Yeah.  My point was that if we find out that for example codechecker 
gives us higher quality results, we should focus on that more.



* scan-build (aka clang-analyzer), a static analyzer integrated into
Clang. This tool is executed every day. Support C & C++.
The main issue with scan-build is that here is no history management and
it is not really possible to ignore false positive.
Ericsson started to work on a new (Python) tool based on clang-analyzer
called Code Checker - https://github.com/Ericsson/codechecker
to address that.


FWIW I am planning to stand this up for us at some point (hopefully soon.)


Could you share some details? I am on the process of deploying code checker.


I haven't started any actual work yet, that was on my list.  If you're 
in the process of doing this, I will eagerly wait.  :-)  Is there a bug 
# or something else that I can use to keep track of this work?  (And 
thanks for making it happen!)



== Infer ==

Firefox (just C code):
https://people.mozilla.org/~sledru/reports/firefox-infer/bugs.txt

Fennec (Java code):
https://people.mozilla.org/~sledru/reports/fennec-infer/bugs.txt


Neat!  I did not know about this one.  Has anyone looked at the results?

This bug https://bugzilla.mozilla.org/show_bug.cgi?id=1175203 has been reported
but no activity.


Those are Java issues which should probably be discussed more with the 
Android team.  After having a cursory look over a few of the C++ ones, 
most looked like false positives (the leak in 
xpcom/tests/TestJemalloc.cpp was real but that's a test...)

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: About static analyzers on some various projects

[Sylvestre Ledru]

Le 24/09/2015 23:29, Ehsan Akhgari a écrit :
> On 2015-09-24 1:41 PM, Sylvestre Ledru wrote:
>> = Static analyzers =
>> For now, we are running:
>> * Coverity, a proprietary tool with a great (but slow) web interface. As
>> Firefox is Free software, the service is provided for free
>> but with a restriction in term of number of build. Now, the analysis is
>> launched once a week on Monday. Supports C, C++ & Java.
>> A few improvements will be made to silent some of the defects.
>
> Does anybody look at these regularly?  
I am looking at the weekly reports. I am reporting the issue I can confirm.
However, to be honest, I am not technically able to analyze every one of them.
I am also tagging false positive to keep a clean database.

FYI, at some point, we might have someone to help on this full time.

> I would be interested to know if they produce high quality results these 
> days.  My past experience with Coverity has been that it's full of false 
> positivies.
Several answers:
* I think the results are still pretty much the same
* false positives can be silent. This is a work to be done either in our code 
(you reviewed some of my patches for this in the past)
or in coverity
* some checkers have a small false positives ratio, some other, an higher.
>
>> * scan-build (aka clang-analyzer), a static analyzer integrated into
>> Clang. This tool is executed every day. Support C & C++.
>> The main issue with scan-build is that here is no history management and
>> it is not really possible to ignore false positive.
>> Ericsson started to work on a new (Python) tool based on clang-analyzer
>> called Code Checker - https://github.com/Ericsson/codechecker
>> to address that.
>
> FWIW I am planning to stand this up for us at some point (hopefully soon.)
>
Could you share some details? I am on the process of deploying code checker.
>> == Infer ==
>>
>> Firefox (just C code):
>> https://people.mozilla.org/~sledru/reports/firefox-infer/bugs.txt
>>
>> Fennec (Java code):
>> https://people.mozilla.org/~sledru/reports/fennec-infer/bugs.txt
>
> Neat!  I did not know about this one.  Has anyone looked at the results?
This bug https://bugzilla.mozilla.org/show_bug.cgi?id=1175203 has been reported
but no activity.

Sylvestre
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Considering dropping Talos support for Linux32

[Ehsan Akhgari]

On 2015-09-24 10:55 PM, Justin Dolske wrote:

On 9/24/15 6:55 PM, Robert O'Callahan wrote:


FWIW I agree that Talos results for Linux32 are unimportant. Even if
there
was a Linux-32-only regression, I don't think it'd be worth our
developers
spending time on it.


I agree. Kill them.


+1.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: About static analyzers on some various projects

[Robert O'Callahan]

On Sat, Sep 26, 2015 at 1:46 AM, Ehsan Akhgari 
wrote:

> Our static analysis builds can be easily triggered from the try server
> (although I have been unable to get anyone interested to fix bug 1116518 to
> make those builds happen on the try server by default, which makes it all
> too easy for people to forget to turn on these builds from their trychooser
> syntax) so as soon as the code is pushed to try (perhaps through autoland)
> the check failures will be visible as build failures.  I'm not quite sure
> what it would take to get those build failures to appear in MozReview but
> it should be possible.
>

The tricky bit is to determine which failures were introduced by the patch,
and just display those, and display them in the context of the patch in
some way that makes sense.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: About static analyzers on some various projects

[Gregory Szorc]

On Fri, Sep 25, 2015 at 12:19 AM, Robert O'Callahan 
wrote:

> On Fri, Sep 25, 2015 at 5:41 AM, Sylvestre Ledru 
> wrote:
>
> > Any questions, comments?
> >
>
> This whitepaper on Infer is an interesting read:
>
> https://fbcdn-dragon-a.akamaihd.net/hphotos-ak-xap1/t39.2365-6/10935986_985284008163608_74391_n/Moving_Fast_with_Software_Verification.pdf
> (misleading title though).
> (Apart from the comments about the tool, I like the observation that mobile
> app development is a problem for Facebook compared to Web because they have
> to support old versions of their app.)
> One of the key things in that paper, which I've claimed for a while, is
> that the real value of static analysis tools for developers like us is to
> apply them at code review time. That's when developers are motivated to
> clean up, explain, and fix their code, and when it's cheapest to do so. So
> I'm looking forward to having support in Mozreview for automated drive-by
> reviews, and then it would be really valuable to adapt these tools to do
> such reviews --- at least as valuable as running them over the whole
> codebase.


We are actively working to integrate automatic review bots into MozReview.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform