Re: reducing high try macosx pending counts

2017-08-02 Thread Bobby Holley
Thanks to everyone involved for accommodating the extra load from the stylo
side - it's a necessary part of our ramp-up to shipping. Luckily, we should
only be running the two configurations side-by-side for a month or two,
after which point load should drop back down to normal levels.

Thanks Kim for staying on top of this and coming up with ways to keep it
manageable!
bholley

On Wed, Aug 2, 2017 at 2:30 PM, Kim Moir  wrote:

> You may have noticed that the time to wait for macosx test results on try
> has been very long (>1day) this week.
>
> [tracking] macosx test load is unsustainable
> https://bugzilla.mozilla.org/show_bug.cgi?id=1386625
>
> There are several factors that contributed to this situation.
> 1) Additional macosx stylo tests were enabled on trunk which added
> significant test load to a system already running at capacity
> 2) These tests were enabled before some optimizations were in place to
> reduce the number of tests run per push
> 3) Since jobs on try have a lower priority than jobs on trunk, the trunk
> jobs continue to consume the the mac test machine capacity before it can be
> used by try jobs
>
> We have taken the following steps to address this problem
> 1) I have disabled the mac stylo tests on m-i to reduce load (
> https://bugzilla.mozilla.org/show_bug.cgi?id=1386625)
> 2)  The optimizations have been enabled for autoland and m-i branches so
> all the tests are not run on every push for the new stylo tests (
> https://bugzilla.mozilla.org/show_bug.cgi?id=1386405)
> 3) Patches are in progress to disable some non-e10s tests to further reduce
> load (https://bugzilla.mozilla.org/show_bug.cgi?id=1386689)
>
> We will continue to monitor the situation and implement additional measures
> to allow macosx test runs on try to complete in a more reasonable interval.
>
> Please ping us in #releng or follow the tracking bug if you have questions
> https://bugzilla.mozilla.org/show_bug.cgi?id=1386625
>
> Kim
> ___
> dev-planning mailing list
> dev-plann...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


reducing high try macosx pending counts

2017-08-02 Thread Kim Moir
You may have noticed that the time to wait for macosx test results on try
has been very long (>1day) this week.

[tracking] macosx test load is unsustainable
https://bugzilla.mozilla.org/show_bug.cgi?id=1386625

There are several factors that contributed to this situation.
1) Additional macosx stylo tests were enabled on trunk which added
significant test load to a system already running at capacity
2) These tests were enabled before some optimizations were in place to
reduce the number of tests run per push
3) Since jobs on try have a lower priority than jobs on trunk, the trunk
jobs continue to consume the the mac test machine capacity before it can be
used by try jobs

We have taken the following steps to address this problem
1) I have disabled the mac stylo tests on m-i to reduce load (
https://bugzilla.mozilla.org/show_bug.cgi?id=1386625)
2)  The optimizations have been enabled for autoland and m-i branches so
all the tests are not run on every push for the new stylo tests (
https://bugzilla.mozilla.org/show_bug.cgi?id=1386405)
3) Patches are in progress to disable some non-e10s tests to further reduce
load (https://bugzilla.mozilla.org/show_bug.cgi?id=1386689)

We will continue to monitor the situation and implement additional measures
to allow macosx test runs on try to complete in a more reasonable interval.

Please ping us in #releng or follow the tracking bug if you have questions
https://bugzilla.mozilla.org/show_bug.cgi?id=1386625

Kim
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: security problems [WAS: Intent to remove: sensor APIs]

2017-08-02 Thread Enrico Weigelt, metux IT consult

On 02.08.2017 15:53, Blair MacIntyre wrote:


FWIW, I wouldn’t mind being involved in a discussion about this,

> if people want to seriously consider putting it behind a
> "user-permission prompt"  (similar to geolocation) or

"user-action requirement”


I'd even go further and move it to an extra package, that may not even
be deployed in the first place.

One major reason is that browsers are often used on system where we
can't always trust the user to always take the right decisions.
Scenarios could be my 90rs old grandpa (who's already annoyed by all
that web-2.0 stuff) or business machines where companies wanna protect
themselves from espionage etc (similar to banning smartphones w/
cameras from their facilities, etc.

The more of those things are added to the browser, the harder it gets
to manage this. Sooner or later we'll get to a point where FF is just
banned (or forked). I doubt that this is your intention.


There has been discussion of this issue in the WebVR community, for

> example, noting that in WebVR, you don’t get any device reports
> without a user action requesting the “VR”.

By device reports you mean calling home ?


On top of that, there is very likely a need to not just “ask once at

> the start” but toggle access to sensitive info on/off as the user uses
> a web app

And there should be a manual control (eg. via keys or mouse gestures),
w/o the web app noticing that it's manual.

There's even more: the user also needs control over where the data
is actually coming from (eg. which device exactly). Otherwise that
fancy feature will only be usable in some specific usecases.


I think as we move toward exposing AR technology (like Tango, ARKit,

> Windows Holographic) in web user-agents, we may need to rethink how
> we obtain and manage the data user’s give to pages.

Yes, that's a very vital issue. And I'd also suggest which parts of that
are implemented *inside* the browser at all (vs external applications)

> I believe that respecting user privacy and supporting their ability
> to control information flow may actually be the thing that makes the

web a preferred platform for AR/VR,


Perhaps we should also rethink what "the web" *actually* means here.
Does everything that the web might offer need to run inside the
browser ? Does that mean the browser has to become an kind of own
operating system ?


since the various platforms are giving all data to apps automatically,

> which create a “take it or leave it” attitude regarding privacy and
> sensor information.

An important point here is that it's easy to leave it. If you don't want
to run any proprietary code, just don't do it. Period. And it doesn't
seem to be easy making great number of people dependent on it.

OTOH, if these things are already integrated in the browser, it isn't
so easy anymore. It quickly becomes an all or nothing decision.


Anyway, if folks want to discuss this, let me know.  We should probably move 
off this thread?


Agreed, for the WebVR stuff (maybe should be even discussed on a
separate list). In general, I'd just like to highlight that
security is a vital aspect.


--mtx

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: sccache as ccache

2017-08-02 Thread Ted Mielczarek
On Wed, Aug 2, 2017, at 12:26 PM, Ben Kelly wrote:
> On Wed, Jul 26, 2017 at 9:05 AM, Ted Mielczarek
>  wrote:>> Yesterday I published sccache 0.2 to 
> crates.io, so you can now `cargo>>  install sccache` and get the latest 
> version (it'll install to
>>  ~/.cargo/bin).
> 
> I tried this on my linux build machine today and got:
> 
> error: failed to run custom build command for `openssl-sys v0.9.15`
<...>

You need to install the `libssl-dev` package (on Ubuntu) or the
equivalent on other distros. Sorry, wish this was clearer!
-Ted

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: sccache as ccache

2017-08-02 Thread Ben Kelly
On Wed, Aug 2, 2017 at 12:26 PM, Ben Kelly  wrote:

> On Wed, Jul 26, 2017 at 9:05 AM, Ted Mielczarek 
> wrote:
>
>> Yesterday I published sccache 0.2 to crates.io, so you can now `cargo
>> install sccache` and get the latest version (it'll install to
>> ~/.cargo/bin).
>>
>
> I tried this on my linux build machine today and got:
>
> error: failed to run custom build command for `openssl-sys v0.9.15`
> process didn't exit successfully: `/tmp/cargo-install.
> FAs9llrjaqDW/release/build/openssl-sys-a543e0ede317714a/build-script-build`
> (exit code: 101)
> --- stdout
> cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
> cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
> cargo:rerun-if-env-changed=OPENSSL_DIR
> run pkg_config fail: "`\"pkg-config\" \"--libs\" \"--cflags\" \"openssl\"`
> did not exit successfully: exit code: 1\n--- stderr\nPackage openssl was
> not found in the pkg-config search path.\nPerhaps you should add the
> directory containing `openssl.pc\'\nto the PKG_CONFIG_PATH environment
> variable\nNo package \'openssl\' found\n"
>
> --- stderr
> thread 'main' panicked at '
>
> Could not find directory of OpenSSL installation, and this `-sys` crate
> cannot
> proceed without this knowledge. If OpenSSL is installed and this crate had
> trouble finding it,  you can set the `OPENSSL_DIR` environment variable
> for the
> compilation process.
>
> If you're in a situation where you think the directory *should* be found
> automatically, please open a bug at https://github.com/sfackler/
> rust-openssl
> and include information about your system as well as this message.
>
> $HOST = x86_64-unknown-linux-gnu
> $TARGET = x86_64-unknown-linux-gnu
> openssl-sys = 0.9.15
>
> ', /home/bkelly/.cargo/registry/src/github.com-
> 1ecc6299db9ec823/openssl-sys-0.9.15/build.rs:198
> note: Run with `RUST_BACKTRACE=1` for a backtrace.
>
> Build failed, waiting for other jobs to finish...
> error: failed to compile `sccache v0.2.0`, intermediate artifacts can be
> found at `/tmp/cargo-install.FAs9llrjaqDW`
>

Looks like I needed to run this as a pre-req on ubuntu:

sudo apt-get install pkg-config libssl-dev
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Upcoming C++ standards meeting in Toronto, Canada

2017-08-02 Thread Botond Ballo
On Wed, Jul 5, 2017 at 7:55 PM, Botond Ballo  wrote:
> I will be there, hanging out in the Evolution Working
> Group as usual, and blogging about the meeting afterwards.

If you're interested in what happened at this meeting, my blog post
about it is now live:

https://botondballo.wordpress.com/2017/08/02/trip-report-c-standards-meeting-in-toronto-july-2017/

Cheers,
Botond
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Firefox startup code coverage (ts_paint talos test)

2017-08-02 Thread Mike Conley
This is great, thanks so much Marco! This will likely be a useful way
for us to find more opportunities to make start-up faster.

Thanks for your work!

On 2017-08-02 10:28 AM, Marco Castelluccio wrote:
> Hello all,
> at https://marco-c.github.io/ts_paint_startup_coverage_report/ you can
> find a (C/C++-only for now) code coverage report of Firefox startup. The
> report was generated using the ts_paint talos test, considering code
> executed until the "MozAfterPaint" event and after a setTimeout(, 0). It
> also contains the number of times a line was executed ("Line data" in
> the source view).
> 
> The report could be useful to identify code that is currently being
> executed at startup but shouldn't.
> 
> The build is a debug build with all optimizations disabled, to have more
> precise line coverage.
> 
> - Marco.
> 
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
> 
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: sccache as ccache

2017-08-02 Thread Ben Kelly
On Wed, Jul 26, 2017 at 9:05 AM, Ted Mielczarek  wrote:

> Yesterday I published sccache 0.2 to crates.io, so you can now `cargo
> install sccache` and get the latest version (it'll install to
> ~/.cargo/bin).
>

I tried this on my linux build machine today and got:

error: failed to run custom build command for `openssl-sys v0.9.15`
process didn't exit successfully:
`/tmp/cargo-install.FAs9llrjaqDW/release/build/openssl-sys-a543e0ede317714a/build-script-build`
(exit code: 101)
--- stdout
cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
cargo:rerun-if-env-changed=OPENSSL_DIR
run pkg_config fail: "`\"pkg-config\" \"--libs\" \"--cflags\" \"openssl\"`
did not exit successfully: exit code: 1\n--- stderr\nPackage openssl was
not found in the pkg-config search path.\nPerhaps you should add the
directory containing `openssl.pc\'\nto the PKG_CONFIG_PATH environment
variable\nNo package \'openssl\' found\n"

--- stderr
thread 'main' panicked at '

Could not find directory of OpenSSL installation, and this `-sys` crate
cannot
proceed without this knowledge. If OpenSSL is installed and this crate had
trouble finding it,  you can set the `OPENSSL_DIR` environment variable for
the
compilation process.

If you're in a situation where you think the directory *should* be found
automatically, please open a bug at https://github.com/sfackler/rust-openssl
and include information about your system as well as this message.

$HOST = x86_64-unknown-linux-gnu
$TARGET = x86_64-unknown-linux-gnu
openssl-sys = 0.9.15

',
/home/bkelly/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-sys-0.9.15/
build.rs:198
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Build failed, waiting for other jobs to finish...
error: failed to compile `sccache v0.2.0`, intermediate artifacts can be
found at `/tmp/cargo-install.FAs9llrjaqDW`
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Actually-Infallible Fallible Allocations

2017-08-02 Thread Boris Zbarsky

On 8/2/17 11:18 AM, Nathan Froyd wrote:

In particular, the API of Sequence<> is constrained because it
inherits from FallibleTArray, which *only* exposes fallible
operations.


We should consider just fixing this.

The history here is that FallibleTArray and InfallibleTArray used to 
bake the allocator into the class type.  There was no way to do fallible 
operations with an InfallibleTArray.


Because Sequence is used for both in parameters and return values in the 
DOM, and in parameters _must_ be fallible (web content trivially 
controls the allocation size: "var arr = []; arr.length = 1e9;"), 
Sequence inherited from FallibleTArray.


But for return values, where _we_ may control the allocation size, it 
may be reasonable to use infallible array bits if we know the array is 
small.


I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1386711 on maybe 
changing this.


-Boris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Intent to remove: sensor APIs

2017-08-02 Thread Enrico Weigelt, metux IT consult

On 02.08.2017 14:39, Blair MacIntyre wrote:


It’s used for panoramic image viewing (orient the pano with the camera
movement), and google street view uses it for similar motion control.


Okay, why not adding a generic interface for controlling the virtual
view direction ? So, the user/operator could decide how to control it
(keys, mouse gestures, external 3d positioning devices, etc).

Anyways, I wonder how the current approach would work at all with
non-portable devices. And the idea of having to physically turn around
just to see different perspectives seems really weird to me.


Regarding security:  perhaps it is, I have seen discussions of this
sort.


Allowing webites to track individual movements, IMHO qualifies more
than just an "perhaps". Do you feel well with the idea of being tracked
with every single footstep ?

> But, it would seem that ship sailed when the W3C approved it, and

now it’s common and assumed and relied upon. Removing it in Firefox
would render Firefox incompatible with a growing use of the web,


Okay, we now know that was a wrong assumption - but even it was
really was approved: does that mean we have to support anything that
some beaurocrats find a good idea ?

In the past, there have been lots of standards that peopel stopped
supporting / complying to, because they turned out to be a bad idea.
I still remember when leaded fuel was standard - meanwhile it's even
prohibited (except for a few old timers). I also remember when IPX
or DecNet have been de-facto standards. Flash has been a de-facto
standard, it was a security nightmare, and it took a long time to get
rid of it.

What would you do if W3C decided that web applications shall be allowed
to execute arbitrary binaries with user's full privileges ?
(actually, I wouldn't be surprised if Goverments making such laws ...)


especially mobile (including Windows tablets).


What would be the actual impact ? A few websites that rely on that
won't work completely anymore. Actually, haven't seen single one yet.

Is it okay, to sacrify the security of all users just for the joy of
that small minority ?


This might be a discussion the security team wants to have, I guess:

> is the Firefox team worried enough about the threats opened by this
> API to justify breaking a large class of applications, and making

Firefoxunusable for AR/VR moving forward.


Suggestive question. And implies that this "large class of applications"
actually exists already.

I'd prefer asking: is it okay to sacrifice security for some niche
features ?

OTOH, I'd also question whether that AR/VR really belongs into a
browser, and what's the costs and risks of that. (some countries
already started legislation against AR, most notably Pokemon Go, for
good reasons ...).

Certainly, AR/VR has it's use in some industrial applications or
entertainment machines, but I doubt Browsers and HTML are particular
well suited for those jobs.



--mtx
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: security problems [WAS: Intent to remove: sensor APIs]

2017-08-02 Thread Blair MacIntyre
> At least these things should be purely optional and providing an
> *easy* way to filter that data. (same for the geolocation stuff).


FWIW, I wouldn’t mind being involved in a discussion about this, if people want 
to seriously consider putting it behind a "user-permission prompt"  (similar to 
geolocation) or "user-action requirement” (similar to webvr and some aspects of 
mobile video playing) of some sort.  

There has been discussion of this issue in the WebVR community, for example, 
noting that in WebVR, you don’t get any device reports without a  user action 
requesting the “VR”.  But, there is the tension between making the APIs usable, 
permission fatigue on the part of users, etc.  On top of that, there is very 
likely a need to not just “ask once at the start” but toggle access to 
sensitive info on/off as the user uses a web app (e.g., in the experimental 
Argon4 “AR-enabled” web browser, we have the ability to toggle location data 
on/off at any time without having to reload).

I think as we move toward exposing AR technology (like Tango, ARKit, Windows 
Holographic) in web user-agents, we may need to rethink how we obtain and 
manage the data user’s give to pages.  

I want the web to work well in these new application areas;  but I also want 
the characteristics of the web we love (i.e., the ability to feel relatively 
safe as you move around and follow links) to survive as well.  I believe that 
respecting user privacy and supporting their ability to control information 
flow may actually be the thing that makes the web a preferred platform for 
AR/VR,  since the various platforms are giving all data to apps automatically, 
which create a “take it or leave it” attitude regarding privacy and sensor 
information.   This is a major driver for me for how a WebAR api may structured.

Anyway, if folks want to discuss this, let me know.  We should probably move 
off this thread?
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Intent to ship: CSS font-display descriptor

2017-08-02 Thread Jonathan Kew
For Firefox 57, I intend to turn the CSS 'font-display' descriptor for 
@font-face rules on by default. This feature was developed behind the 
layout.css.font-display.enabled preference.


Blink has already shipped this feature as of Chrome 60:
  https://developers.google.com/web/updates/2017/07/nic60#fontdisplay

This feature was developed in:
  https://bugzilla.mozilla.org/show_bug.cgi?id=1157064
and landed (preffed off) for Firefox 46.

Bug to turn on by default:
  https://bugzilla.mozilla.org/show_bug.cgi?id=1317445

Link to [draft] standard:
  https://drafts.csswg.org/css-fonts-4/#font-display-desc

Testing:
  https://github.com/w3c/web-platform-tests/tree/master/css-font-display


JK
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Actually-Infallible Fallible Allocations

2017-08-02 Thread Nathan Froyd
On Tue, Aug 1, 2017 at 12:31 PM, Alexis Beingessner
 wrote:
> I was recently searching through our codebase to look at all the ways we
> use fallible allocations, and was startled when I came across several lines
> that looked like this:
>
> dom::SocketElement  = *sockets.AppendElement(fallible);
>
> For those who aren't familiar with how our allocating APIs work:
>
> * by default we hard abort on allocation failure
> * but if you add `fallible` (or use the Fallible template), you will get
> back nullptr on allocation failure
>
> So in isolation this code is saying "I want to handle allocation failure"
> and then immediately not doing that and just dereferencing the result. This
> turns allocation failure into Undefined Behaviour, rather than a process
> abort.
>
> Thankfully, all the cases where I found this were preceded by something
> like the following:
>
> uint32_t length = socketData->mData.Length();if
> (!sockets.SetCapacity(length, fallible)) {   JS_ReportOutOfMemory(cx);
>   return NS_ERROR_OUT_OF_MEMORY;}for (uint32_t i = 0; i <
> socketData->mData.Length(); i++) {dom::SocketElement  =
> *sockets.AppendElement(fallible);
>
> So really, the fallible AppendElement *is* infallible, but we're just
> saying "fallible" anyway.

Stating this upfront: I think we should use infallible AppendElement
here, or at the very least use a comment.  But it's worth looking at
the larger context of one of these examples:

https://dxr.mozilla.org/mozilla-central/source/netwerk/base/Dashboard.cpp#426-435

(AFAICT, the only such examples of this pattern come from the above
file, and for the reasons outlined below.)

In particular, the API of Sequence<> is constrained because it
inherits from FallibleTArray, which *only* exposes fallible
operations.  One can argue that FallibleTArray shouldn't do this, but
for Sequence, which is used for DOM bindings code, I believe the
intent is to nudge people writing DOM-exposed code to consider how to
recover from allocation failures, and to not blindly assume that
everything succeeds.

-Nathan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


security problems [WAS: Intent to remove: sensor APIs]

2017-08-02 Thread Enrico Weigelt, metux IT consult

On 02.08.2017 14:29, Michael Hoye wrote:


You need to dial this rhetoric back about 100%. It is not acceptable to
bring even an implied accusation like that to a technical discussion, or
indeed any conversation at all, at Mozilla.


Who did I accuse of what exactly ?

All I'd like to say here is that those features add yet another tool
for mass sourveillance.

I've grown up in the GDR regime - I've learned what it means when your
privacy is invaded or you get punished because somebody in your family
or a neighbor said a wrong word.

And we're strongly marching towards the same again, but now with the
oppressors having much better technology, in our bedrooms. It's not
fiction, it's fact - it's already there. Spying phone apps everywhere,
even spying TV sets, remote controllable cards, etc, etc.

Quite recently, the German parliament voted yet another enabling act
for mass sourveillance (eg. wiretapping people just because some
neighbour or colleque *might* possibly have done a tax fraud, etc).
And they did what w/ only a small minority of the representatives
even present (reminds me to 1933).

So, the problem is very immanent. We need to be very careful here,
which information to send out (or even aquire in the first place).
Various fingerprinting techniques already impose a big problem
(IMHO, generic cookies should never have been introduced in the
first place).


We're always happy to listen to honest criticism and walk back our
mistakes, but we are going have those discussions without demeaning the
work or comparing the people doing that work to volkscryptopolitzei
collaborators.


Please. I didn't imply anybody here collaborating with dark forces.
I'm just warning about the danger of these features.

At least these things should be purely optional and providing an
*easy* way to filter that data. (same for the geolocation stuff).


--mtx

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Intent to remove: sensor APIs

2017-08-02 Thread Blair MacIntyre
> On Wed, Aug 2, 2017 at 4:39 PM, Blair MacIntyre  
> wrote:
>> Are we still talking about deviceorientation?
> 
> As I said twice and Frederik repeated, we're not, other than asking if
> anyone has a plan for how to make it interoperable.

Yes, I know;  I was just responding to Enrico’s question. :)

> Note that it's far
> from a W3C standard: https://www.w3.org/TR/orientation-event/. Doesn't
> seem like anything got approved there.

Fair enough, sorry for my incorrect assertion, I should have checked!   So, 
this is an example of the danger, perhaps, of everybody implementing a working 
proposal before it’s approved, and then having it adopted and used widely.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Actually-Infallible Fallible Allocations

2017-08-02 Thread Ehsan Akhgari

On 08/01/2017 08:16 PM, Jim Blandy wrote:

I have to ask: does anyone recall the benchmarks that showed that bounds
checks or vector reallocations were a measurable performance hit in this
code?
Extra needless bounds checks certainly have shown up in profiles, for an 
example see https://bugzilla.mozilla.org/show_bug.cgi?id=1374033 where 
we were accidentally doing two bounds checks when using nsTObserverArray 
iterators on release builds as opposed to one.


Vector reallocations show up in profiles all the time, literally in more 
than half of the profiles I look at every day.  If you examine for 
example the large dependency graph of 
https://bugzilla.mozilla.org/show_bug.cgi?id=Speedometer_V2, a large 
number of optimizations that are landing every day are merely removing 
various types of extra needless buffer reallocations we have in profiles.



If we don't, we should just write simple, obviously correct code.

If we do, there should be a comment to that effect, or something to convey
to readers the performance constraints this code is trying to satisfy.


Where do you propose for such a comment to go?

Generic code such as arrays and vectors must be free of additional 
needless overhead.  They are bound to be used in performance sensitive 
areas, and these issues will show up sooner or later.  I believe we have 
tons of practical evidence to suggest that these are far from 
theoretical concerns.


Cheers,
Ehsan


On Tue, Aug 1, 2017 at 2:10 PM, Kris Maglione  wrote:


On Tue, Aug 01, 2017 at 01:28:37PM -0700, Eric Rahm wrote:


Both AppendElements and SetLength default initialize the values, there's
no
intermediate uninitialzed state. SetCapacity doesn't initialize the
values,
but they're also not indexable -- we have release bounds assertions --
unless you try really hard.


For a lot of cases, default initialized isn't much better than completely
uninitialized. And it has an additional, unnecessary performance penalty
for the pattern that you suggested.

SetCapacity brings us back to the same problem where we either have
unnecessary allocation checks for every element we append, or we skip
sanity checks entirely and hope things stay sane if we refactor.

With the infallible*() methods of the Vector class, though, we don't have
to worry about any of that, and the code you suggested below becomes
something like:

  Vector sockets;
  if (!sockets.reserve(inputs.length()))
return false;

  for (auto input : inputs)
sockets.infallibleEmplaceBack(input);

and we still get automatic allocation sanity checks and bounds accounting,
but without any release overhead from default initialization, allocation
checks, or unnecessary reallocations.

And since that's the natural pattern for appending a fixed number of
elements to a Vector, it doesn't really require any thought when writing
it. The safe and efficient approach is basically the default option.


nsTArray doesn't support emplace although it does have AppendElement(T&&),

but that wouldn't really help in this case. It's possible we could add
that
of course!

-e

On Tue, Aug 1, 2017 at 1:11 PM, Kris Maglione 
wrote:

On Tue, Aug 01, 2017 at 12:57:31PM -0700, Eric Rahm wrote:

nsTArray has various Append methods, in this case just using the

infallible
AppendElements w/o out a SetCapacity call would do the job. Another
option
would be to use SetLength which would default initialize the new
elements.

If we're trying to make things fast-but-safeish in this case, the
preferred
way is probably:

auto itr = AppendElements(length, fallible);
if (!itr) ...

// no bounds checking
for (...; i++, itr++)
   auto& mSocket = *itr;

// bounds checking
for (...)
auto& mSocket = *sockets[i];

In general I agree the pattern of fallibly allocating and then fallibly
appending w/o checking the return value is a bit silly. Perhaps we
should
just mark the fallible version MUST_USE? It looks like it's commented
out
for some reason (probably a ton of bustage).



Honestly, I think the Vector infallible* methods are a much cleaner way
to
handle this, especially when we need something like
infallibleEmplaceBack.
They tend to encourage writing more efficient code, with fewer
reallocations and allocation checks. And I think the resulting code tends
to be safer than AppendElements followed by unchecked raw pointer access,
placement `new`s, and an intermediate state where we have a bunch of
indexable but uninitialized elements in the array.

That's been my experience when reading and writing code using the various
approaches, anyway.


On Tue, Aug 1, 2017 at 12:18 PM, Kris Maglione 


wrote:

On Tue, Aug 01, 2017 at 12:31:24PM -0400, Alexis Beingessner wrote:


I was recently searching through our codebase to look at all the ways
we


use fallible allocations, and was startled when I came across several
lines
that looked like this:

dom::SocketElement  = *sockets.AppendElement(fallible);
...
So in 

Re: Intent to remove: sensor APIs

2017-08-02 Thread Anne van Kesteren
On Wed, Aug 2, 2017 at 4:39 PM, Blair MacIntyre  wrote:
> Are we still talking about deviceorientation?

As I said twice and Frederik repeated, we're not, other than asking if
anyone has a plan for how to make it interoperable. Note that it's far
from a W3C standard: https://www.w3.org/TR/orientation-event/. Doesn't
seem like anything got approved there.


-- 
https://annevankesteren.nl/
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Intent to remove: sensor APIs

2017-08-02 Thread Blair MacIntyre
Are we still talking about deviceorientation?

It’s used to determine the 3D orientation of the device, so that we can tell 
the direction it is facing.  Developers use it to render 3D graphics (WebGL or 
CSS3D using perspective DIV) around the user.  e.g., look at one of my project 
samples, like https://samples.argonjs.io/directionsWebGL 
, which uses device location and 
deviceorientation (this simple samples puts 3D labels in the cardinal 
directions, and uses the position to illuminate them based on the current sun 
location).  The WebVR polyfill uses it to determine viewing direction, to 
simulate 3D device orientation.

It’s used for panoramic image viewing (orient the pano with the camera 
movement), and google street view uses it for similar motion control. 

Regarding security:  perhaps it is, I have seen discussions of this sort.  But, 
it would seem that ship sailed when the W3C approved it, and now it’s common 
and assumed and relied upon. Removing it in Firefox would render Firefox 
incompatible with a growing use of the web, especially mobile (including 
Windows tablets).  This might be a discussion the security team wants to have, 
I guess:  is the Firefox team worried enough about the threats opened by this 
API to justify breaking a large class of applications, and making Firefox 
unusable for AR/VR moving forward.

> On Aug 2, 2017, at 9:54 AM, Enrico Weigelt, metux IT consult 
>  wrote:
> 
> On 02.08.2017 13:01, Salvador de la Puente wrote:
>> I strongly encourage you to take a look at the telemetry stats regarding
>> the usage of deviceorientation API and other. I don't know the penetration
>> of proximity and ambient light APIs but deviceorientation is definitively
>> used.
> 
> Just curious: for what exactly is it used ?
> 
> For rendering / GUI, I'd assume that the job of the windowing system /
> compositor - the application would just see a window geometry change.
> 
> Making that information visible to websites (even worse: movement
> tracking via g-sensor, etc), definitively looks like security nightmare
> which even the Stasi never dared dreaming of.
> 
> --mtx
> 

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: refcounting [WAS: More Rust code]

2017-08-02 Thread Joshua Cranmer 

On 8/2/2017 6:37 AM, Enrico Weigelt, metux IT consult wrote:

On 31.07.2017 13:53, smaug wrote:




Reference counting is needed always if both JS and C++ can have a
pointer to the object.


By the way - just curious:

Anybody already thought about garbage collection ?


Yes. About a decade ago, Mozilla invested some resources in being able 
to automatically rewrite the codebase to use GC instead of reference 
counting: .


Some conclusions:
1. Automated rewriting of C++ code is possible (this is when the only 
significant open-source C++ compiler relied on horribly inaccurate 
position tracking, so it actually was a big deal).
2. Converting from reference-counting to conservative GC is barely 
possible. (I recall bsmedberg saying that the resulting build could go 
for a few minutes before crashing)
3. It's not the performance win you think it is. The main performance 
wins are a compacting GC, which minimizes memory use over time and makes 
heap allocation basically a pointer bump. On the other hand, now you 
have to have threadsafe reference counting on every object.



That wouldn't have the problem w/ circular graphs, and should make the
whole code smaller and faster.


See the results on the linked page. It was neither smaller, nor faster.

We already have a cycle-collector, which is basically an opt-in garbage 
collector (it only looks at a subset of the total ownership graph). The 
main difficulty in using it is having to annotate classes in cycles, but 
this is exactly the sort of thing that's easy to write in a #[derive()] 
attr in Rust.


--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


./mach try fuzzy: A Try Syntax Alternative

2017-08-02 Thread Andrew Halberstadt
I'm pleased to announce an alternative method for scheduling tasks on try
is now landed on mozilla-central. It makes use of the awesome fzf [1]
project to filter down the list of all task labels with a fuzzy matching
algorithm.

It works both with mercurial or git. If using mercurial, you'll need to
make sure you're updated to the latest version-control-tools:

$ ./mach mercurial-setup --update

To push to try, run:

$ ./mach try fuzzy

This will prompt you to install fzf. After bootstrapping is complete, it'll
generate the task list. This takes around ~10-20 seconds, but will be
cached so subsequent invocations won't incur this penalty again.

Now you'll be in the fzf interface. Basic usage is to start typing to
filter down the task list. You can use the arrow keys to move the cursor up
or down,  to select a task,  to select all tasks and 
to schedule the current selection (and their dependencies) on try.

There are a ton more keyboard shortcuts and features you can use to tweak
fzf just to your liking. For extra help, see:

$ ./mach try fuzzy --help
or
$ man fzf

For a demo and more information on implementation details, see:
https://ahal.ca/blog/2017/mach-try-fuzzy/

I expect this to work on all platforms including Windows for both mercurial
(with push-to-try) and git (with git-cinnabar). But it's a new tool without
a lot of real world testing, so a few bumps are likely. If you find any
bugs or bad UX, please file a bug under Testing :: General and let me know
(we should find a better component for this).

Cheers,
Andrew

p.s When running the fzf bootstrap, you'll be prompted to install some
shell extensions. These aren't necessary for |mach try fuzzy| to work so
feel free to hit 'N' at each prompt. That being said, the fzf shell
extensions are awesome. I highly encourage you to look into setting it up
for your shell. The fzf vim plugin is also great. See the project [1] for
more details.

[1] https://github.com/junegunn/fzf
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Intent to remove: sensor APIs

2017-08-02 Thread Michael Hoye
On Aug 2, 2017 15:54, "Enrico Weigelt, metux IT consult" <
enrico.weig...@gr13.net> wrote:



Making that information visible to websites (even worse: movement
tracking via g-sensor, etc), definitively looks like security nightmare
which even the Stasi never dared dreaming of.


You need to dial this rhetoric back about 100%. It is not acceptable to
bring even an implied accusation like that to a technical discussion, or
indeed any conversation at all, at Mozilla.

We're always happy to listen to honest criticism and walk back our
mistakes, but we are going have those discussions without demeaning the
work or comparing the people doing that work to volkscryptopolitzei
collaborators.

I encourage you to read our community participation guidelines carefully,
and to take them to heart before continuing.

Thank you.

-mhoye
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Firefox startup code coverage (ts_paint talos test)

2017-08-02 Thread Marco Castelluccio
Hello all,
at https://marco-c.github.io/ts_paint_startup_coverage_report/ you can
find a (C/C++-only for now) code coverage report of Firefox startup. The
report was generated using the ts_paint talos test, considering code
executed until the "MozAfterPaint" event and after a setTimeout(, 0). It
also contains the number of times a line was executed ("Line data" in
the source view).

The report could be useful to identify code that is currently being
executed at startup but shouldn't.

The build is a debug build with all optimizations disabled, to have more
precise line coverage.

- Marco.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Intent to remove: sensor APIs

2017-08-02 Thread Enrico Weigelt, metux IT consult

On 02.08.2017 13:01, Salvador de la Puente wrote:

I strongly encourage you to take a look at the telemetry stats regarding
the usage of deviceorientation API and other. I don't know the penetration
of proximity and ambient light APIs but deviceorientation is definitively
used.


Just curious: for what exactly is it used ?

For rendering / GUI, I'd assume that the job of the windowing system /
compositor - the application would just see a window geometry change.

Making that information visible to websites (even worse: movement
tracking via g-sensor, etc), definitively looks like security nightmare
which even the Stasi never dared dreaming of.

--mtx

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: refcounting [WAS: More Rust code]

2017-08-02 Thread Ted Mielczarek
On Wed, Aug 2, 2017, at 08:32 AM, Nathan Froyd wrote:
> On Wed, Aug 2, 2017 at 7:37 AM, Enrico Weigelt, metux IT consult
>  wrote:
> > On 31.07.2017 13:53, smaug wrote:
> >> Reference counting is needed always if both JS and C++ can have a
> >> pointer to the object.
> >
> > Anybody already thought about garbage collection ?
> 
> Reference counting is a garbage collection technique.  See
> https://en.wikipedia.org/wiki/Reference_counting where the
> introductory paragraphs and the first section specifically refer to it
> as a garbage collection technique.  Or consult _The Garbage Collection
> Handbook_ by Jones, Hosking, and Moss, which has an entire chapter
> devoted to reference counting.
> 
> Note also that Gecko's reference counting tends to be cheaper than the
> reference counting assumed in the literature, since many of Gecko's
> reference-counted objects can use non-thread-safe reference counting,
> as said objects are only ever accessed on a single thread.  (Compare
> http://robert.ocallahan.org/2012/06/computer-science-in-beijing.html)
> 
> Changing the garbage collection technique used by our C++ code to
> something other than reference counting would be a large project of
> dubious worth.

Also we tried that once and it didn't work well for various reasons:
https://wiki.mozilla.org/XPCOMGC

-Ted
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Intent to remove: sensor APIs

2017-08-02 Thread Frederik Braun
As mentioned in thread, we will not disable deviceorientation.
Please see below.

On 02.08.2017 15:01, Salvador de la Puente wrote:
> I strongly encourage you to take a look at the telemetry stats regarding
> the usage of deviceorientation API and other. I don't know the penetration
> of proximity and ambient light APIs but deviceorientation is definitively
> used.
> 
> Please, consider twice before taking a final decision.
> 
> El 31 jul. 2017 3:44 p. m., "Anne van Kesteren"  escribió:
> 
>> On Mon, Jul 24, 2017 at 6:11 PM, Anne van Kesteren 
>> wrote:
>>> Please consider the request to remove device orientation retracted for
>>> now. We'll still need to figure out some kind of long term plan for
>>> that API though. WebVR building on it through libraries that abstract
>>> away the browser incompatibilities will just make it harder to fix the
>>> underpinnings going forward. (And there's always the risk that folks
>>> don't use libraries and code directly against what Chrome ships. Seems
>>> likely even.)
>>
>> Small update: we'll start by just disabling proximity. Disabling
>> ambient light will follow soon after, but is a little trickier as we
>> use the web-facing API in the Firefox for Android frontend.
>> (Suggestions for fixing the orientation interoperability mess are
>> still welcome!)
>>
>>
>> --
>> https://annevankesteren.nl/
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
> 
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: Intent to remove: sensor APIs

2017-08-02 Thread Salvador de la Puente
I strongly encourage you to take a look at the telemetry stats regarding
the usage of deviceorientation API and other. I don't know the penetration
of proximity and ambient light APIs but deviceorientation is definitively
used.

Please, consider twice before taking a final decision.

El 31 jul. 2017 3:44 p. m., "Anne van Kesteren"  escribió:

> On Mon, Jul 24, 2017 at 6:11 PM, Anne van Kesteren 
> wrote:
> > Please consider the request to remove device orientation retracted for
> > now. We'll still need to figure out some kind of long term plan for
> > that API though. WebVR building on it through libraries that abstract
> > away the browser incompatibilities will just make it harder to fix the
> > underpinnings going forward. (And there's always the risk that folks
> > don't use libraries and code directly against what Chrome ships. Seems
> > likely even.)
>
> Small update: we'll start by just disabling proximity. Disabling
> ambient light will follow soon after, but is a little trickier as we
> use the web-facing API in the Firefox for Android frontend.
> (Suggestions for fixing the orientation interoperability mess are
> still welcome!)
>
>
> --
> https://annevankesteren.nl/
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: refcounting [WAS: More Rust code]

2017-08-02 Thread Nathan Froyd
On Wed, Aug 2, 2017 at 7:37 AM, Enrico Weigelt, metux IT consult
 wrote:
> On 31.07.2017 13:53, smaug wrote:
>> Reference counting is needed always if both JS and C++ can have a
>> pointer to the object.
>
> Anybody already thought about garbage collection ?

Reference counting is a garbage collection technique.  See
https://en.wikipedia.org/wiki/Reference_counting where the
introductory paragraphs and the first section specifically refer to it
as a garbage collection technique.  Or consult _The Garbage Collection
Handbook_ by Jones, Hosking, and Moss, which has an entire chapter
devoted to reference counting.

Note also that Gecko's reference counting tends to be cheaper than the
reference counting assumed in the literature, since many of Gecko's
reference-counted objects can use non-thread-safe reference counting,
as said objects are only ever accessed on a single thread.  (Compare
http://robert.ocallahan.org/2012/06/computer-science-in-beijing.html)

Changing the garbage collection technique used by our C++ code to
something other than reference counting would be a large project of
dubious worth.

-Nathan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Re: nodejs for extensions ?

2017-08-02 Thread Enrico Weigelt, metux IT consult

On 31.07.2017 20:30, David Teller wrote:

Node dependency trees tend to be pretty large, so I'm a little concerned
here. Has the memory footprint be measured?


That would also concern me (OTOH, moz's footprint already is as large as
a small skyscraper anyways ;-)).

We also have similar problems jscript-intensive websites.
Maybe there could be a way of sharing node modules between extensions ?

In the optional case, I'd like to have an extension actually being a
node modules, managed via npm or distro's package manager.

But I've got the feeling, that would require a fresh start w/ moving
that completely out to separate server processes: gecko exports some
IPC interface which extensions can call into, eg. request becoming a
filter for url pattern, retrieve and change a DOM subtree, trigger
certan actions, etc. When we even add docking windows to a dom element,
we'd also have the most vital parts for generic plugins.
This approach isn't even bound to javascript or nodejs - these
extensions could be written in any language.


--mtx

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


refcounting [WAS: More Rust code]

2017-08-02 Thread Enrico Weigelt, metux IT consult

On 31.07.2017 13:53, smaug wrote:




Reference counting is needed always if both JS and C++ can have a
pointer to the object.


By the way - just curious:

Anybody already thought about garbage collection ?

That wouldn't have the problem w/ circular graphs, and should make the
whole code smaller and faster.


--mtx

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform


Stack walking in Gecko

2017-08-02 Thread Nicholas Nethercote
Hi,

We have multiple ways of getting stack traces, and it is hard to keep track
of
them all. I wrote down some notes that reflect my understanding, and I
thought
I'd share it in case it's useful, and in case people have ideas on how to
improve things.

Nick


-
mozglue/misc/StackWalk.{h,cpp}
-
The main stackwalk function is MozStackWalk:

> MFBT_API bool
> MozStackWalk(MozWalkStackCallback aCallback,
>  uint32_t aSkipFrames,
>  uint32_t aMaxFrames,
>  void* aClosure,

Here's how it is implemented on different platforms.

- Win32   Custom code using StackWalk64()
- Win64   Custom code using RtlVirtualUnwind()
- Linux32 FramePointerStackWalk?[1]
- Linux64 _Unwind_Backtrace
- Mac32?  FramePointerStackWalk?[1]
- Mac64   _Unwind_Backtrace[2]
- Android _Unwind_Backtrace?
- Other   (unimplemented, mostly)

[1] FramePointerStackWalk() is a straightforward FP-chasing unwind. It is a
public function and so can be called directly, too.

[2] _Unwind_Backtrace is really slow on some recent Mac versions, alas.

On Windows there is also MozStackWalkThread(), which differs from
MozStackWalk() by being able to walk the stack of a different thread.

-
Gecko Profiler
-
The Gecko Profiler has a couple of other stack walkers used on some
platforms.

- LUL: Linux-only stack walker that uses DWARF info to unwind.

- EHABI: Android-only stack walker that uses the ARM exception handling ABI.

-
Places where stack walking is used.
-
Lots of places use MozStackWalk() for all platforms: telemetry,
ah_crap_handler, HangMonitor, BlockingResourceBase,
TlsAllocationTracker.cpp,
nsTraceRefCnt.cpp, etc. These are all places where the number of stack walks
performed is small, so speed isn't that important.

The Gecko Profiler and DMD use a mixture of stack walkers, as per this
table.

Gecko Profiler  DMD
--  ---
Win32   FramePointerStackWalk   FramePointerStackWalk
Win64   MozStackWalkMozStackWalk
Linux32 LUL MozStackWalk
Linux64 LUL MozStackWalk
Mac32?  FramePointerStackWalk[1]FramePointerStackWalk[1]
Mac64   FramePointerStackWalk[1]FramePointerStackWalk[1]
Android EHABI   MozStackWalk

[1] requires MOZ_PROFILING to be set so that frame pointers are used.

The Gecko Profiler and DMD do *lots* of stack walks, so speed is important
for
them.

Note that the Gecko Profiler needs to stackwalk other threads than the
current
one.

- Windows' MozStackWalkThread() allows that by naming the thread.

- FramePointerStackWalk() allows that, because you just give it the
  to-be-walked thread's registers.

- _Unwind_Backtrace() doesn't allow it.

HangMonitor also requires off-thread stackwalking, but it's Windows-only
(perhaps exactly because it requires off-thread stackwalking). It could
potentially use profiler_suspend_and_sample() instead (see next paragraph).

Finally, the Gecko Profiler also has a public function
profiler_suspend_and_sample_thread() that walks the stack of another thread
(using the abovementioned stackwalkers), and interleaves that with
pseudostacks
and JS stacks. BHR uses this function.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform