Re: New JavaScript engine module owner

[Gabriele Svelto]

On 10/03/2021 02:12, Jason Orendorff wrote:
> Hi, everyone.
> 
> I'm pleased to announce that Jan De Mooij has agreed to take ownership of
> the JavaScript engine module.
> 
> Following a Mozilla tradition that was venerable when I got here, Jan has
> been doing the job already for quite some time. Please join me in
> congratulating Jan and thanking him for his ongoing leadership.

Congrats Jan, that's one big module!

 Gabriele
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Build on IllumOS, where can I find the version of external dependencies

[Gabriele Svelto]

Il 04/03/2021 15.53, Carsten Grzemba ha scritto:
> Is there a recommendation for the gcc version to use for building the 78esr 
> release?

You need GCC 7.1 or newer:

https://hg.mozilla.org/releases/mozilla-esr78/file/tip/build/moz.configure/toolchain.configure#l890

 Gabriele
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Gecko performance with newer x86_64 levels

[Gabriele Svelto]

On 10/02/21 10:21, Henri Sivonen wrote:
> Chrome is moving to SSE3 as the unconditional baseline, which I
> personally find surprising:
> https://docs.google.com/document/d/1QUzL4MGNqX4wiLvukUwBf6FdCL35kCDoEJTm2wMkahw/edit#
> 
> A quick and very unscientific look at Searchfox suggests that
> unconditional SSE3 would mainly eliminate conditional/dynamic dispatch
> on YUV conversion code paths when it comes to explicit SSE3 usage. No
> idea how LLVM would insert SSE3 usage on its own.

SSE3 instructions were very specialized - mostly for video processing -
I doubt that LLVM can make use of them in regular code. It's unclear to
me why the Chrome devs decided to jump to SSE3 given it should give very
little benefit over SSE2. It would have made more sense if they jumped
to SSSE3 instead, but that would have cut out all the users still on
early Athlon 64 processors.

 Gabriele



OpenPGP_signature
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Oxydized minidump writer in Linux builds

[Gabriele Svelto]

[cross-posting to stability]

 Hiya all,
this is a heads-up that we just landed a patch that rewrites the code we
use to write minidumps for content process crashes in Rust [1] for our
Linux builds. The resulting dumps should be identical to what we
obtained before, so if you spot strange crashes in the next few nightly
builds please report them.

Note that this change only affects content process crashes; main process
crashes are still using the old code and will be switched over only in
the coming months. Android is not affected by this change ATM and so are
the non-x86 Linux platforms.

I'd like to thank Martin Sirringhaus who contributed this change. It was
a very significant chunk of work in an area of Gecko that is both
complex and obscure.

 Gabriele

[1] Rewrite the Linux-specific minidump writer code in Rust
https://bugzilla.mozilla.org/show_bug.cgi?id=1620993
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Improvement to build times through cleanup of C++ include dependencies

[Gabriele Svelto]

 Thanks for this work Simon, this is awesome!
There's also plenty of side effects to this that will make life better
for developers, just a few off the top of my mind:

- When sccache is in use files are preprocessed before being sent to
sccache, so even when hitting the cache we pay the price of running the
preprocessor so less includes will speed up sccache hot builds!

- All our static analysis passes need to go through the preprocessed
sources too, so less includes means less code to churn through when we
run them.

- When templates are involved the linker needs to de-duplicate identical
instantiations in different compilation units, with less of them linking
will be faster too.

- And also less I/O which might seem academic if you've got a very fast
SSD but not everybody has one (think about contributors who often don't
have access to beefy boxes).

 Gabriele

On 14/12/20 12:23, Simon Giesecke wrote:
>  tl;dr Build times on all platforms, in particular for incremental builds,
> have decreased in the last weeks by landing several cleanups to C++ include
> dependencies that reduce the aggregated number of included files by about
> 30%.
> 
> Hi,
> 
> Did you notice a reduction in build times lately? This might not be a
> coincidence. In this mail, I want to provide some details on the
> improvements made. I want to thank everyone who contributed to this through
> up-front discussions and reviews.
> 
> Recently I landed a number of patches on Bug 1676346 [1] that in various
> ways clean up C++ include directives in our codebase. Some landed ca. 3
> weeks ago, some landed last week. Overall, these reduce the aggregated
> number of included non-system header files in a full build by about 30% on
> Linux. I don't have numbers for other platforms, but they should benefit as
> well. On my machine, this reduced the time for a clobber build by about
> 10%. While that might go unnoticed, incremental builds are often sped up
> even a lot more, since the number of translation units that need to be
> rebuilt decreases. E.g. the number of translation units that include
> dom/base/Document.h reduced by ca. 52%, resulting in a build time reduction
> of 48% on my machine after modifying that. Your mileage may vary.
> 
> While this might not spare you from buying new hardware, it will make
> builds faster regardless of the hardware you are running on, and hopefully
> increase productivity. If you want to share your experiences with me,
> please get in touch!
> 
> You might be curious what I did to achieve that, or how you can contribute
> to reducing build times yourself. It's a combination of things, most
> importantly three things:
> 1. Remove unused include directives
> 2. Split some headers
> 3. Use forward declarations more
> 4. Hide definitions to allow using forward declarations even more
> 
> About 1: I found there are several include directives that are not needed
> at all. They could simply be removed. However, the majority of cases were a
> bit more complex, because of a lot of missing include directives. When
> removing an include for X.h from a header file Y.h that doesn't need it,
> another file that included Y.h might need X.h. Or, Y.h itself might need
> something from a header indirectly included from X.h. Or similar cases.
> This meant quite a lot of more include directives for more basic things
> needed to be added to ensure the build doesn't break.
> 
> About 2: Some headers have a lot of dependencies, but only relatively few
> users of that header need them. One example was IPCMessageUtils.h, which is
> included by all files generated by IPDL, which also contained a lot of
> specializations of the ParamTraits template that are needed only by few
> files. Apart from some very basic specializations, these were moved to the
> new EnumSerializers.h and IPCMessageUtilsSpecializations.h as well as some
> existing headers, so that the remaining IPCMessageUtils.h header has only
> much more limited dependencies.
> 
> About 3: Our coding style favors forward declarations over inclusion of the
> full definition of types where possible. I replaced the inclusion of header
> files containing full definitions of types by forward declarations at a
> number of places where this is sufficient, e.g. because they are only used
> in function signatures. It's worth noting that there were also a number of
> cases where a forward declaration was present, but actually the full
> definition is required, e.g. when a type is used as a base class or as the
> value type of a data member, or an inline function body dereferences into a
> member.
> 
> About 4: As mentioned in the last point, inline function bodies often
> require the inclusion of additional headers because they dereference into
> types of which otherwise only forward declarations were necessary. Similar
> considerations apply to private (nested) classes. Some of those were moved
> to the corresponding implementation files to hide these dependencies, and
> reduce 

Re: Improved stacks in Linux crash reports

[Gabriele Svelto]

On 10/11/20 05:57, ISHIKAWA,chiaki wrote:
> Does this support split dwarf object files?

Not yet, but it should be possible to add support easily because the
crates we depend on for parsing DWARF debuginfo have just been updated
to support split DWARF [1].

 Gabriele

[1] https://github.com/gimli-rs/gimli/blob/master/CHANGELOG.md#added


OpenPGP_signature
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Improved stacks in Linux crash reports

[Gabriele Svelto]

[cross-posting to dev-platform]

 Hello all,
I'm glad to announce that we switched our symbol scrapers for Linux
distributions to our new and improved Rust-based tooling [1]. This means
that when looking at crash reports or profiling you will not find stack
traces with missing or omitted function names anymore.

If you're tracking Linux crashes this might cause signature changes
starting with release version 83; the new signatures should be
significantly improved.

The impact of this change will be more visible to users on Fedora and
OpenSUSE.

 Gabriele

[1] https://github.com/mozilla/dump_syms/
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Software Fallback for Web Render Dog Fooding Phase

[Gabriele Svelto]

On 27/10/20 21:18, Jim Mathies wrote:
> 3) Detect noticeable performance issues with page interaction and scrolling 
> that normally don't occur, we'd appreciate bug reports with Firefox profiles.

Is the software fallback expected to use all available CPU cores/threads
for rendering? I enabled it and played around a bit on Google Maps.
Performance is visibly lower than with hardware support - something I
did expect - but I could see only one core being loaded at 100%, the
rest were sitting idle. That surprised me because most software
fallbacks (e.g. LLVMpipe on Mesa) tend to use all available CPU resources.

So I was wondering if this was a deliberate choice (to conserve power on
laptops maybe?) or if I should file a bug.

 Gabriele


OpenPGP_signature
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

More volunteers needed for nightly crash triage rotation

[Gabriele Svelto]

[cross posting to firefox-dev and stability]

 Hello all,
the crash triage team needs your help in keeping Firefox stable! The
role of the triagers is to have a look at the crashes for a certain
version of nightly a few days after it has been released and file bugs
for crashes that haven't been identified yet. This is usually done to
quickly identify regressions but it often also involves looking at
low-volume crashes that flew under the radar or platform-specific issue
that only appear with specific configurations.

Triage is usually done on a daily basis: every triager looks at a
certain day worth of crashes, for example on Monday morning I analyze
the crashes for the previous Friday's nightly build.

Currently we need help for the following days:

* Monday nightlies (to be triaged on Wednesday or Thursday)
* Thursday nightlies (to be triaged on Friday or Monday)
* Saturday nightlies (to be triaged on Monday or Tuesday)

The days in which the triage will happen is a suggestion, if you have
time another day it's fine given it's at least 24-48h after the build
was released.

In case you're interested please contact me and I'll be happy to walk
you through the process and also mentor you for your first sessions if
you're unfamiliar with crash analysis.

For further information also please have a look at our wiki page, fairly
up-to-date docs are linked from there:

https://wiki.mozilla.org/NightlyCrashTriage

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: What hardware is needed to make a Firefox OS phone and can you list them?

[Gabriele Svelto]

 Hi,
this is probably not the right channel for this question, you might want
to ask on dev-f...@lists.mozilla.org or on the #b2g channel on Matrix.
Also there's no Firefox OS devices at the moment, but you can build a
B2G emulator using the latest mozilla-central tree. Let's follow up on
this on the appropriate lists.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New and improved stack-fixing

[Gabriele Svelto]

On 06/05/20 17:01, Markus Stange wrote:
> I see. What about crashes during regular optimized builds? I'd hope
> those print stacks.

You mean local builds? Unless you have ac_add_options --enable-debug in
your mozconfig you won't see stack traces. What you can do in that case
is build with the crash reporter enabled and point the
MINIDUMP_STACKWALK env variable to a build of the stackwalker. When
you'll hit an assertion a minidump will be generated, the test harness
will detect it and print out a stack-trace by feeding it into the stack
walker. Not optimal but better than nothing.

All of the above already happens on automation when running tests for
optimized builds: you won't have inline stack traces for crashes but at
the end of the test you'll see the crash dump with the complete stack trace.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New and improved stack-fixing

[Gabriele Svelto]

On 05/05/20 23:38, Nicholas Nethercote wrote:
> As for why that check is there... do opt builds produce any stack traces in
> tests? Normal assertions aren't enabled on opt builds, but
> diagnostic/release assertions are. I can't remember off the top of my head
> if they produce stacks traces in opt builds, and likewise with crashes.

Stack traces are printed out only on debug builds:

https://searchfox.org/mozilla-central/rev/7908ce29657cfd623993046bd8e38664e1c0b28e/mfbt/Assertions.h#451-454

So optimized debug builds will print assertions, but regular optimized
builds we have on automation will not.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Engineering Effectiveness Newsletter #1

[Gabriele Svelto]

g> as well as a list
of fuzzing tools
<https://mana.mozilla.org/wiki/pages/viewpage.action?spaceKey=FIREFOX=Fuzzing+tools>.

 = Stability =

- Calixte Denizet rewrote the Windows dump_syms tool in Rust we use to
extract symbols from Firefox debuginfo. The new tool is dramatically
faster than the old one and produces better symbol information
<https://github.com/mozilla/dump_syms/>.

- Nicholas Nethercote rewrote the stack-fixing scripts we used to make
stack traces readable on automation in Rust. The resulting scripts are
two order of magnitudes faster leading to test runtimes being shortened
by up to 40 minutes <https://github.com/mozilla/fix-stacks/>.

- Gabriele Svelto wrote scripts to scrape symbols for the most popular
Linux distros, this gives us better Linux crash reports and visibility
into crashes from distro-packaged builds of Firefox
<https://github.com/gabrielesvelto/symbol-scrapers/>.

 = Release management =

- Following Julien Cristau’s suggestion, Calixte Denizet added a new
autonag rule to find missing potential beta uplifts in Bugzilla
<https://github.com/mozilla/relman-auto-nag/pull/912>.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New and improved stack-fixing

[Gabriele Svelto]

On 06/03/20 16:46, Andrew Sutherland wrote:
> Thank you both very much for the clarifications and your excellent work
> here!
> 
> In terms of the Sentry crates, I presume that means the crates in
> https://github.com/getsentry/symbolic repo?  Are there still reasons to
> use/pay attention to Ted's https://github.com/luser/rust-minidump repo? 
> For example, I use slightly modified versions of
> `get-minidump-instructions` and `minidump_dump` from the latter, but I
> want to make sure I'm hitching my metaphorical tooling wagon to the
> right metaphorical tooling horse.

Yes, we will soon be improving over that to replace the
minidump-specific functionality of Breakpad. Sentry still relies on
Breakpad for that stuff and I intend to replace that functionality with
an all-Rust implementation based on Ted's stuff. Bug 1588530 [1] is the
meta that tracks all this work.

 Gabriele

[1] [meta] Crash reporting Rust rewrite
https://bugzilla.mozilla.org/show_bug.cgi?id=1588530



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New and improved stack-fixing

[Gabriele Svelto]

Thanks a lot for doing this work Nicholas.

I would like to add that once this hits automation the perf and memory
usage improvements will have a measurable impact on the tests runtime.
That's because we feed the output of all our tests on automation to the
scripts, and if they hit even a single function in libxul they will load
the entire symbol file and keep it in memory in a very
memory-inefficient format. This can take minutes and consume gigabytes
of memory. So much that at some point we even ran into OOM scenarios
because of it (bug 1493365 [1] for example).

 Gabriele

[1] Intermittent PLDHashTableTest.GrowToMaxCapacity | Value of: false
https://bugzilla.mozilla.org/show_bug.cgi?id=1493365

bug 1493365

On 05/03/2020 23:52, Nicholas Nethercote wrote:
> Hi all,
> 
> I have written a new stack-fixing tool, called `fix-stacks`, which
> symbolizes stack traces produced by Firefox.
> 
> `fix-stacks` is intended to replace our existing stack-fixing scripts,
> `fix_{linux,macosx}_stack.py` and `fix_stack_using_bpsyms.py`, which are
> used (a) on many test outputs, both locally and on automation, and (b) by
> DMD, Firefox's built-in heap profiler.
> 
> `fix-stacks` is now installed by `mach bootstrap` on Windows, Mac, and
> Linux. It is usable from Python code via the `fix_stacks.py` wrapper
> script. Its code is at https://github.com/mozilla/fix-stacks/.
> 
> In bug 1604095 I replaced the use of `fix_{linux,macosx}_stack.py` with
> `fix_stacks.py` for DMD, with the following benefits.
> 
> * On Linux, stack-fixing of DMD output files (which occurs when you run
> `dmd.py`) is roughly 100x faster. On my Linux box I saw reductions from
> 20-something minutes to ~13 seconds.
> 
> * On Mac, stack-fixing of DMD output files on Mac is roughly 10x faster. On
> my Mac laptop I saw reductions from ~7 minutes to ~45 seconds.
> 
> * On Windows, stack-fixing of DMD output files now occurs. (It previously
> did not because there is no `fix_window_stacks.py` script.) This means that
> DMD is now realistically usable on Windows without jumping through hoops to
> use breakpad symbols.
> 
> There is more work to be done. Soon, I plan to:
> 
> * use `fix-stacks` on test outputs (in `utils.py` and `automation.py.in`);
> 
> * re-enable stack fixing on Mac test runs on local builds, which is
> currently disabled because it is so slow;
> 
> * add breakpad symbol support to `fix-stacks`;
> 
> * remove the old scripts.
> 
> The tree of relevant bugs can be seen at
> https://bugzilla.mozilla.org/showdependencytree.cgi?id=1596292_resolved=1
> .
> 
> The stack traces produced by `fix-stacks` are sometimes different to those
> produced by the old stack-fixers. In my experience these differences are
> minor and you won't notice them if you aren't looking for them. But let me
> know if you have any problems.
> 
> Nick
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
> 
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New and improved stack-fixing

[Gabriele Svelto]

 Hi Andrew,

On 06/03/2020 01:27, Andrew Sutherland wrote:
> Does this eliminate the need documented at
> https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Mochitest#stacks
> to acquire a `minidump_stackwalk` binary and then expose it via
> MINIDUMP_STACKWALK environment variable to get symbolicated backtraces
> when local test runs crash?  Or is that part of the future work to "use
> `fix-stacks` on test outputs"?

the short answer is no. We use `minidump_stackwalk` to pull out the
stack trace from a crash and print it, the stack-fixing scripts can't do
that. However, if the frames printed out by `minidump_stackwalk` are not
symbolicated (it can happen) then the stack-fixing scripts will try to
symbolicate the un-symbolicated lines in the output.

Generally speaking the stack-fixing scripts are needed because we often
print stack traces from inside Gecko - by using MozStackWalk() for
example - and those don't have symbols. So the stack-fixing scripts scan
Gecko's output and when they find something that looks like an
unsymbolicated entry in a stack-trace they replace it with the
appropriate symbol.

Now for the long answer: we're leveraging the Sentry crates to replace
our current crash reporting machinery. Not only they're faster and
better than what we have now but their functionality is far richer. So
it will be possible - and possibly even desirable - to fold all of this
functionality into a single tool that knows how to do both stack-walking
and symbolication.

For now we're just building 1-to-1 replacements of what we have because
that's the path of least resistance to replace our aging tools and it
yields very tangible perf and memory consumption improvements as
Nicholas highlighted in his post.

 Gabriele
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Volunteers needed for nightly crash triage rotation

[Gabriele Svelto]

[cross-posting to firefox-dev and stability]

We still have an open slot to triage crashes from Firefox nightly Monday
builds. This should take place on a Tuesday or a Wednesday. If somebody
has some spare cycles please shoot me a message or fill the roster entry
yourself here:

https://wiki.mozilla.org/NightlyCrashTriage#Roster

I'd also like to thank everybody who has responded to this. The roster
is now almost full and we've been quickly finding regressions in
nightly, that's no small feat.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

IPCError-browser | ShutDownKill signatures are about to change

[Gabriele Svelto]

[cross-posting to dev-platform]

IPCError-browser | ShutDownKill crashes are the second most common ones
after OOMs and generally not actionable because we've been lumping
together a bunch of different stacks under the same signature.

These crashes represent a snapshot of a content process taking too long
to shut down. The affected content process is killed right after the
crash report is generated.

Bug 1612569 [1] will change those signatures soon. The new signatures
will include one or more function names after `IPCError-browser |
ShutDownKill` coming from the stack of the content process.

This will allow us to find common causes of slowness (or deadlocks) that
prevent content processes from shutting down rapidly and hopefully act
on them.

 Gabriele

[1] Improve IPCError-browser | ShutDownKill signatures
https://bugzilla.mozilla.org/show_bug.cgi?id=1612569



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Volunteers needed for nightly crash triage rotation

[Gabriele Svelto]

[cross-posting to firefox-dev and stability]

FYI quite a few people followed up with me by mail and in person so the
roster will soon be full again but more volunteers are still welcome. If
we have more volunteers than the slots I'm very happy to do rotations so
that we have more eyes on nightly and a lighter workload for everybody.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Volunteers needed for nightly crash triage rotation

[Gabriele Svelto]

[cross-posting to firefox-dev and stability]

Due to the recent layoffs the nightly crash triage roster has a lot of
holes so if you have some spare cycles please come and help out!

Nightly crash triage consists in looking at the crashes for a specific
version of nightly (e.g. the previous week Monday build), catching
regressions and filing them. Usually it takes no more than 30 minutes
per week and is a good way to learn more about Firefox internals and
interact with code you're not necessarily familiar with.

We've got detailed documentation [1] to get you started. We'll be adding
more docs about tools that make the job easier.

We've got four slots to fill ATM [2] but in the lucky case there's more
candidates than slots we're happy to rotate. Please contact me if you're
interested.

 Gabriele

[1] https://wiki.mozilla.org/NightlyCrashTriage
[2] https://wiki.mozilla.org/NightlyCrashTriage#Roster



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: To what extent is sccache's distributed compilation usable?

[Gabriele Svelto]

 Hi all,
when dealing with builds on slower machines I usually run a build every
morning (4AM) so that when I wake up I've got it ready (together with a
warm cache) and following builds won't be as bad.

This can be done easily on a Mac by setting up a cronjob that will run a
build on a freshly updated mozilla-central clone (you might want to
stash your changes before updating). I then set a schedule with the
energy saver preferences to wake up my Mac from sleep a few minutes
before the cronjob is supposed to run.

This is not a solution for changes that end up affecting large parts of
the build (such as touching a common header) but usually saves me a lot
of time.

 Gabriele

On 29/10/19 18:53, Steve Fink wrote:
> On 10/28/19 9:17 PM, Marcos Caceres wrote:
>> On Tuesday, October 29, 2019 at 3:27:52 AM UTC+11, smaug wrote:
>>> Quite often one has just a laptop. Not compiling tons of Rust stuff
>>> all the time would be really nice.
>>> (I haven't figured out when stylo decides to recompile itself - it
>>> seems to be somewhat random.)
>> Probably a gross misunderstanding on my part, but the sccache project
>> page states [1]: "It is used as a compiler wrapper and avoids
>> compilation when possible, storing a cache in a remote storage using
>> the Amazon Simple Cloud Storage Service (S3) API, the Google Cloud
>> Storage (GCS) API, or Redis."
>>
>> I'm still (possibly naively) imagining that we will leverage the "the
>> cloud"™️ to speed up compiles? Or am I totally misreading what the
>> above is saying?
>>
>> [1] https://github.com/mozilla/sccache#sccache---shared-compilation-cache
> 
> My experience with other distributed compilation tools (distcc, icecc)
> indicates that cloud resources are going to be of very limited use here.
> Compiles are just way too sensitive to network bandwidth and latency,
> especially when compiling with debuginfo which tends to be extremely
> large. Even if the network transfer takes way less time than the
> compile, the sending/receiving scheduling never seems to work out very
> well and things collapse down to a trickle.
> 
> Also, I've had very limited luck with using slow local machines. A CPU
> is not a CPU  -- even on a local gigabit network, farming off compiles
> to slow machines is more likely to slow things down than speed them up.
> Despite the fancy graphical tools, I was never completely satisfied with
> my understanding of exactly why that is. It could be that a lack of
> parallelism meant that everything ended up repeatedly waiting on the
> slow machine to finish the last file in a directory (or whatever your
> boundary of parallelism is). Or it could be network contention,
> especially when your object files have massive debuginfo portions. (I
> always wanted to have a way to generate split debuginfo, and not block
> on the debuginfo transfers.) The tools tended to show things working
> great for a while, and then slowing down to a snail's pace.
> 
> I've long thought [1] that predictive prefetching would be cool: when
> you do something (eg pull from mozilla-central), a background task
> starts prefetching cached build results that were generated remotely.
> Your local compile would use them if they were available, or generate
> them locally if not. That would at least do no harm (if you don't count
> network bandwidth).
> 
> sccache's usage of S3 makes sense when running from within AWS. I'm
> skeptical of its utility when running remotely. But I haven't tried
> setting up sccache on my local network, and my internet connectivity
> isn't great anyway.
> 
> I really ought to put my decade-old desktop into action again. My last
> attempt was with icecc, and though it worked pretty well when it worked,
> the pain in keeping it alive wasn't worth the benefit.
> 
> 
> [1] Ancient history -
> https://wiki.mozilla.org/Sfink/Thought_Experiment_-_One_Minute_Builds
> 
> 
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Crash reporting flow documentation

[Gabriele Svelto]

[cross-posting to dev-platform]

 Hello all,
I just wanted to point out an excellent blog post by :willkg that
describes the whole crash reporting flow from the exception handler and
all the way to crash-stats.mozilla.com. You can find it here:

https://bluesock.org/~willkg/blog/mozilla/crash_pings_crash_reports.html

AFAIK this is the first write-up that covers the whole flow, so if you
ever need some information on crashes that's an excellent starting point
and it contains pointers to pretty much every other resource we have on
the topic.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Building dav1d 0.2.1 - nasm error

[Gabriele Svelto]

 Hi Gangadharan,

On 22/03/19 05:25, gangadhara...@gmail.com wrote:
> I've been trying to build dav1d 0.2 but I always end with meson error
> 
> Unusable script '/usr/bin/nasm'
> Program nasm found: NO
> 
> meson.build:324:4: ERROR:  Program(s) ['nasm'] not found or not executable
> 
> Eventhough the path /usr/bin/nasm has nasm in it(latest version). Maybe am I 
> missing something here? please suggest me to get this error.

nasm is now part of the toolchain that can be obtained using ./mach
bootstrap. Run ./mach bootstrap again in your mozilla-central directory
and it will be installed under ~/.mozbuild like the rest of the
toolchain. Once that's done try building again, it should work
out-of-the box.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Type-based alias analysis and Gecko C++

[Gabriele Svelto]

Adding my 2p

On 19/02/19 17:54, Jason Orendorff wrote:
>> My understanding is that the original purpose is that if you in a loop
>> read from T* and write on each iteration via U* (where T and U differ
>> by more than signedness and neither is char), without
>> -fno-strict-aliasing the compiler doesn't need to pessimize on the
>> assumption that a write via U* might change the next read via T*. With
>> -fno-strict-aliasing, the compiler has to pessimize on the assumption
>> that a write via U* might affect the next read via T*. The problem is
>> that if accessing a T-typed value via U* is UB *in general*, who knows
>> what *else* the compiler might do if -fno-strict-aliasing is not used
>> and there are reinterpret_casts between pointer types, especially with
>> LTO giving it a broader view to spot UB.

The most common issue I ran into when not using -fno-strict-aliasing on
code with strict aliasing violations is that at high optimization levels
the compiler reordered the memory accesses to the pointers that it
assumed were not aliasing each other. Depending on the code and the
types involved this can result in anything between an immediate (but
confusing) crash or subtle data-corruption bugs.

On the reverse I've seen performance regressions from using
-fno-strict-aliasing only in tight loops where the inability to move
accesses around was lengthening the critical path through the loop.
However this was on processors with very limited memory reordering
capabilities; my guess is that on today's hardware
-fno-strict-aliasing's impact is lost in the noise.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Process Priority Manager enabled on Nightly for Windows (only)

[Gabriele Svelto]

On 30/01/19 17:26, Mike Conley wrote:
> That's a good question. I haven't yet noticed any difference yet, but
> I'm hoping people can keep an eye out to see if there's appreciable lag
> when switching back to a tab with video, or if background audio starts
> to stutter.
> 
> (I suspect that even if we don't hear any reports, it might be better to
> be safe than sorry, and mark any background tabs that are playing media
> as "active" - I'll file a bug.)

The process priority manager already has a concept of "perceivable tabs"
(i.e. ones that are playing some form of audio) and assigns them a
higher priority compared to non-playing background ones. Unfortunately
the Windows-specific mapping of those priorities lumps them together
because Windows has only one background scheduling class. However if we
run into problems like skipping or jittering audio it will be trivial to
put them back in the normal priority class.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Proposal to adjust our clang-format rules to include spaces after the hash for nested preprocessor directives

[Gabriele Svelto]

On 11/01/19 02:01, Ehsan Akhgari wrote:
> The common way to deal with this problem is to indent nested preprocessor
> directives, very much similarly to how we indent normal code, for example:
> 
> #if foo
> #  if bar
> #define x 1
> #  else
> #define x 2
> #  endif
> #endif

this would be much better than what we have now and I recently ran into
an issue with a complex block of #ifdef's that I would have spotted
easily if they had been indented. The first iteration of a patch I wrote
"accidentally" cut out a chunk of a function when built on Windows but
still compiled and even executed most of our test suite just fine.
Fortunately a couple of xpchsell tests failed, but if it hadn't been
covered by those I might not have spotted the error at all.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: PSA: Building Firefox for arm64 windows got a lot easier

[Gabriele Svelto]

 Hi all,

On 11/01/2019 01.24, Mike Hommey wrote:
> In the MSVC installer, choose the following extra components:
> - Visual Studio C++ compiler and libraries for ARM64
> - Visual C++ ATL for ARM64
> - Visual C++ MFC for ARM64

I don't have the MFC for ARM64 libraries installed and I didn't
encounter problems during the build. I guess they're not needed? I'm
still building with WebRTC disabled though, not sure if it might require
them.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Dropping support for compiling with MSVC in the near future

[Gabriele Svelto]

On 06/12/18 15:39, Kris Maglione wrote:
> As it stands, we need to remain compatible with at least GCC and Clang,
> because some of our static analysis code still depends on GCC plugins.

Some Linux distros will keep building Firefox with GCC so there's going
to be at least some external users of that particular combination. MSVC
is a different story since nobody else would be using it.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: error while building a release(official) version on window error: Gecko exception wrapping doesn't understand your your MSVC/SDK. Please file a bug describing this error and your build configurati

[Gabriele Svelto]

 Hi Amirhossein,

On 06/11/2018 16.28, Amirhossein Ghafari wrote:
>  0:43.67 DEBUG: configure:3005: cl.exe -c  -TP -nologo -w15038 -wd5026 
> -wd5027 -Zc:sizedDealloc- -wd4091 -wd4577 -D_HAS_EXCEPTIONS=0  conftest.C 1>&5
>  0:43.67 DEBUG: conftest.C
>  0:43.67 DEBUG: 
> C:\PROGRA~2\MICROS~1\2017\COMMUN~1\VC\Tools\MSVC\1415~1.267\include\yvals.h(512):
>  warning C4005: '_RAISE': macro redefinition
>  0:43.67 DEBUG: configure(2997): note: see previous definition of '_RAISE'
>  0:43.67 DEBUG: configure: error: Gecko exception wrapping doesn't understand 
> your your MSVC/SDK.  Please file a bug describing this error and your build 
> configuration.
>  0:43.67 ERROR: old-configure failed
>  0:43.75 *** Fix above errors and then restart with\
>  0:43.75"c:/mozilla-build/bin/mozmake.EXE -f client.mk build"
>  0:43.77 mozmake.EXE: *** [client.mk:149: configure] Error 1

That error usually crops up when something's wrong with the Windows SDK,
the most likely explanation is that you don't have the correct version
installed. IIRC you need either SDK 10.0.15063.0 or 10.0.17134.0.

Additionally I'd suggest you to try building the current mozilla-central
codebase instead of a release version and use ./mach bootstrap to get
all your dependencies installed.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Workaround for building with Visual Studio 15.7+

[Gabriele Svelto]

 Hello all,
being among those unfortunate enough to have updated Visual Studio
before realizing that the most recent version cannot be used to build
Firefox, I was faced with the prospect of reinstalling the whole
monstrosity - which takes forever - or finding a workaround. As it turns
out I found a clunky yet simple one:

- Launch the Visual Studio Installer, the click on the "modify" button

- Go into the "Individual components" and select the "VC++ 2017 version
15.6 v14.13 toolset" component

- Launch the installation

- Modify build/moz.configure/toolchain.configure by manually setting the
tools_version string in [1] to the value '14.13.26128'

- You're done! Both a clang-cl and a msvc should now pick up the old
tooling and work properly (provided you have the right Windows SDK).

I couldn't find a way to override this via mozconfig options or
environment variables but if there is a way to do this without modifying
the sources I'd like to hear it.

 Gabriele

[1]
https://searchfox.org/mozilla-central/rev/5dbfd833bbb114afe758db4d4bdbc5b13bcc33ef/build/moz.configure/toolchain.configure#584



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Changes in crash annotations

[Gabriele Svelto]

Bug 1348273 [1] just landed and with it how we deal with crash
annotations. All annotations are now declared in
toolkit/crashreporter/CrashAnnotations.yaml which contains both a
description, type and whitelisting/blacklisting information.

To add an annotation just add the entry to that list. By default the
annotations will only be sent alongside a crash report. If the
annotation needs to be included in the crash ping mark it with the
'ping: true' attribute (after getting data review).

The C++ API has been changed to accept an enum value generated from
CrashAnnotations.yaml so the call sites have changed from this:

AnnotateCrashReport(
  NS_LITERAL_CSTRING("MyAnnotation"),
  NS_LITERAL_CSTRING("data for my annotation"));

To this:

AnnotateCrashReport(
  CrashReporter::Annotations::MyAnnotation,,
  NS_LITERAL_CSTRING("data for my annotation"));

Overloaded functions are also provided for annotations taking integers,
unsigned integers and boolean values. The types listed under
CrashAnnotations.yaml are not enforced yet but will be soon. A dedicated
function has also been added to remove an annotation.

JavaScript callers need no modification, they still take the annotation
name as a string but will throw if given an unknown one. On this topic,
it's unfortunately not possible to add an annotation to an artifact
build; a full build is required to re-generate the relevant code.

 Gabriele

[1] Make AnnotateCrashReport() more robust by turning annotations into a
well known list of constants
https://bugzilla.mozilla.org/show_bug.cgi?id=1348273



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Fission MemShrink Newsletter #1: What (it is) and Why (it matters to you)

[Gabriele Svelto]

Just another bit of info to raise awareness on a thorny issue we have to
face if we want to significantly raise the number of content processes.
On 64-bit Windows we often consume significantly more commit space than
physical memory. This consumption is currently unaccounted for in
about:memory though I've seen hints of it being cause by the GPU driver
(or other parts of the graphics pipeline). I've filed bug 1475518 [1] so
that I don't forget and I encourage anybody with Windows experience to
have a look because it's something we _need_ to solve to reduce content
process memory usage.

 Gabriele

[1] Commit-space usage investigation
https://bugzilla.mozilla.org/show_bug.cgi?id=1475518



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Fission MemShrink Newsletter #1: What (it is) and Why (it matters to you)

[Gabriele Svelto]

On 13/07/2018 04:55, Randell Jesup wrote:
> Correct - we need to have observers/what-have-you for
> background/foreground state (and we may want an intermediate state or
> two - foreground-but-not-focused (for example a visible window that
> isn't the focused window); recently-in-foreground (switching back and
> forth); background-for-longer-than-delta, etc.
> 
> Modules can use these to drop caches, shut down unnecessary threads,
> change strategies, force GCs/CCs, etc.
> 
> Some of this certainly already exists, but may need to be extended (and
> used a lot more).

We already had most of this stuff in the ProcessPriorityManager [1]
which has be only ever used in Firefox OS. Since we had
one-process-per-tab there it was designed that way so it might need some
reworking to deal with one tab consisting of multiple content processes.

Also note that dealing with the "importance" of a page is not just a
matter of visibility and focus. There are other factors to take into
account such as if the page is playing audio or video (like listening to
music on YouTube), if it's self-updating and so on.

The only mechanism to reduce memory consumption we have now is
memory-pressure events which while functional are still under-used. We
might also need more fine grained mechanisms than "drop as much memory
as you can".

 Gabriele

[1]
https://searchfox.org/mozilla-central/rev/46292b1212d2d61d7b5a7df184406774727085b8/dom/ipc/ProcessPriorityManager.cpp



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Fission MemShrink Newsletter #1: What (it is) and Why (it matters to you)

[Gabriele Svelto]

On 12/07/2018 22:19, Kris Maglione wrote:
> I've actually been thinking on filing a bug to do something similar, to
> measure cumulative effects of excess padding in certain types since I
> began looking into bug 1460674, and Sylvestre mentioned that
> clang-analyzer can generate reports on excess padding.

I've encountered at least one structure where a boolean flag is 64-bits
in size on 64-bit builds. If we really want to go to the last mile we
might want to also evaluate things like tagged pointers; there's
probably some KiB's to be saved there too.

There's also more than one place where we're using strings to identify
stuff where we could use enums/integers instead. And yeah, my much
delayed refactoring of the observer service got a lot higher on my
priority list after reading this thread.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: mozilla-inbound backout policy subject to change (become similar to autoland)

[Gabriele Svelto]

On 19/06/2018 15:04, Sebastian Hengst wrote:
> TL;DR: We would like to change the mozilla-inbound backout policy to be
> like autoland’s.

Sounds like a very good idea. When I screw up with a patch what worries
me most is wasting other peoples' time with a broken tree. Having
problematic patches backed out quickly means saving time for everybody.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Coding style: brace initialization syntax

[Gabriele Svelto]

On 13/04/2018 16:49, Nathan Froyd wrote:
> I lean towards the former here.  I think the former is more common in
> the code I've seen, but apparently the latter is "preferred C++" or
> something?

Yes, let's have a solid rationale if we're doing sweeping changes of
this sort. Blindly following the "C++ flavor du jour" is going to lead
to a lot of unnecessary churn.

 Gabriele
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: skip-if(verify)

[Gabriele Svelto]

I've also seen TV failures caused by flaws in the test harness which
weren't visible before. See bug 1410165 [1] for example, it was caused
by an observer being unregistered after the first iteration of a test
had finished, causing the following iterations to fail.

 Gabriele

[1] Permaorange test-verify
dom/plugins/test/mochitest/test_crash_notify.xul |
expected-crash-dump-missing - This test did not leave any crash dumps
behind, but we were expecting some!
https://bugzilla.mozilla.org/show_bug.cgi?id=1410165

On 06/03/18 20:05, Kyle Machulis wrote:
> In my experience, many tests were written pre-test-verify, and don't clean
> up correctly to deal with multiple runs in the same process. They work fine
> when running as a single session, but blow up in TV. Having skip-if(verify)
> means that we can at least mark those tests as known broken on TV without
> taking them out of the test suites completely, especially if we find them
> in places that we don't know how to fix them and need to file to someone
> else who may not currently have time to fix.
> 
> On Tue, Mar 6, 2018 at 10:49 AM, Emilio Cobos Álvarez 
> wrote:
> 
>> On 03/06/2018 06:04 PM, Geoffrey Brown wrote:
>>> It is now possible to skip tests in test-verify. Simplify annotate the
>>> manifest for your test:
>>>
>>> [test]
>>> skip-if = verify
>>>
>>> or, for reftests:
>>>
>>> skip-if(verify) ...
>>>
>>> and the test-verify (TV) test task will not try to verify the annotated
>>> test.
>>>
>>> Please don't abuse this feature! Most TV failures indicate a weakness in
>>> the test.
>>
>> Could you point to an example of a "good" use of this feature? Is it
>> just to avoid TV failing too intermittently? Is TV unable to run some
>> tests? When does TV provide no value?
>>
>> Thanks!
>>
>>  -- Emilio
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
> 



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Revised proposal to refactor the observer service

[Gabriele Svelto]

On 29/01/2018 10:54, Gijs Kruitbosch wrote:
> FWIW, this plan seems great to me. Thanks for taking the time to revise
> based on everybody's feedback!

You're welcome! When doing changes that touch the entire codebase I
believe it's critical to have everybody's feedback because it's
impossible to be aware of all the use-cases that we have.

> Would it be possible for this to use WebIDL instead of xpidl?

Frankly I hadn't thought about that because I believed we only used
.webidl for interfaces we exposed on the web but now that you made me
look into it, it seems that we're using it for internal interfaces too.
WebIDL does support enumerations so unless there are drawbacks I might
not be aware of then I could use it just fine.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Revised proposal to refactor the observer service

[Gabriele Svelto]

 Hello all,
I've tried to take into account all the suggestions from the discussion
of my previous proposal [1] and I've come up with a new plan which
should cover all bases. Don't hesitate to point out things that might
still be problematic, since this is going to be a large refactoring it's
better to get it right from the start.

The biggest sticking point of my previous proposal was that it would
prevent any modifications to the observer list from artifact builds.
Addressing that particular issue effectively requires to keep the
existing service in place so instead of replacing it altogether I'd do
the following:

1) Introduce a new observer service that would live alongside the
current one (nsIObserverService2?). This will use a machine-generated
list of topics that will be held within the interface itself instead of
a separate file as I originally proposed. This will become possible
thanks to bug 1428775 [2]. The only downside of this is that the C++
code will not use an enum but just integer constants. The upside is that
this will need only one code generator and only one output file (the
IDL) versus two generators and three files in my original proposal.

2) Migrate all C++-only and mixed C++/JS users to use the new service.
Since the original service would still be there this can be done
incrementally. Leave JS-only users alone.

3) Consider writing a JS-only pub/sub service that would be a better fit
than the current observer service. If we can come up with something
that's better than the observer service for JS then it can be used to
retire the old service for good.

So, how does this sound?

 Gabriele

[1]
https://lists.mozilla.org/pipermail/dev-platform/2018-January/020935.html
[2] Make it possible to generate IDL files
https://bugzilla.mozilla.org/buglist.cgi?quicksearch=1428775



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Refactoring proposal for the observer service

[Gabriele Svelto]

On 04/01/18 22:47, Nathan Froyd wrote:
> This is very doable, it just requires some build system hackery: we
> accept preprocessed/generated WebIDL files, and generated IDL files
> would require basically the same approach.  I can help with the build
> system hackery if you want to continue pursuing this approach.

If it's possible then I think it would be worth pursuing. With this
approach the constants would be held together with the methods within
the interface object/class versus three separate entities of my current
approach (the interface object/class, a C++ enum and a JS object).
Having everything in one place is definitely a better approach from a
code design perspective (and would prevent potential issues such as
including the interface definition but not the header with the enum
declaration). We'd also retain the other advantages such as the ability
to add further metadata to the declaration, having the constants
generated instead of written by hand, etc...

The only drawback I can think of is that we'd lose the ability to use an
enum in C++ for extra type safety. But that really isn't a big deal. For
crash annotations it mattered because the JS interface is built on top
of an already exposed C++ one which could be made type-safe. For
observers the interface used is always the generated one which - being
the same for JS and C++ - cannot use C++ enums directly anyway.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Refactoring proposal for the observer service

[Gabriele Svelto]

On 04/01/18 22:39, Ben Kelly wrote:
> We can't have a comment explaining "please add any new constants in
> sequential order in the following list"?

We could, but what about removing one? Also if we have hundreds (or
thousands) the risk that one is accidentally duplicated is significant.
My rationale for this is that managing topics the way we do know simply
doesn't scale so if we can avoid manual work we should.

>  Or make your "generator"
> create the idl which then creates the js/c++?

I tried as that could have worked! Unfortunately it doesn't seem to be
possible ATM. mach bailed out with a weird error when I tried to put an
IDL file among the generated ones. I didn't really dig into it but I
suspect that since we already generate code from IDL files they're not
expected to be generated in turn.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Refactoring proposal for the observer service

[Gabriele Svelto]

On 03/01/18 23:30, Ben Kelly wrote:
> Could we use our existing idl/webidl/ipdl for this?  It would be nice
> not to have to maintain another code generator in the tree if possible.


AFAIK there is no way in IDL to declare an enum. Constants can be
declared but one needs to manually assign them a value which makes it
unsuitable for this task. Also there's no way to attach other metadata
to the declaration (such as a mandatory comment).

Don't get me wrong though, I am convinced it would be better if this
could be done in IDL because it could then be attached to the relevant
interface (such as nsIObserverService) and we wouldn't need separate
generation of the C++ and JS code, but we would need to significantly
extend it for the purpose.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Refactoring proposal for the observer service

[Gabriele Svelto]

TL;DR this is a proposal to refactor the observer service to use a
machine-generated list of integers for the topics (disguised as enums/JS
constants) instead of arbitrary strings.

Long version:

I've been working on bug 1348273 [1] in an attempt to gather all our
crash annotations into a machine-readable list that could be used to
generate a list of (named) constants. Specifically an enumeration for
C++ and a frozen object holding constant fields for JS. These constants
would then be used instead of the strings when invoking the crash
annotation interface.

As I was hacking this stuff together it occurred to me that the observer
service has the same kind of interface and the same kind of issues:
figuring out what a particular topic is for is quite hard, and often
requires a fair amount of grep'ing and hg blame'ing to figure out where
it's coming from and what it does. There's no single place where it can
be documented. There's no way to tell what the subject is and what the
data string contains (and if it contains a string at all, I've seen
callers stuffing a pointer to a non-string object in there). Retiring or
renaming a topic is also a nightmare, as it's extremely hard to verify
that you've fixed all its uses.

So after validating my approach in that bug (which is almost ready) I've
thought that it might be time to give the observer service the same
treatment. First of all we'd have a list of topics (I've picked YAML for
the list but it could be anything that fits the bill):

QuitApplication:
  description: Notification sent when the application is about to quit

ProfileBeforeChange:
  description: Notification sent before the profile is lost

...

Which would generate a C++ enum:

// Generated enum
enum class ObserverTopic : uint32_t {
  QuitApplication = 0,
  ProfileBeforeChange = 1,
  ...
}

And a JS object:

const ObserverTopic = Object.freeze({
  QuitApplication: 0,
  ProfileBeforeChange: 1,
  ...
}

Callers would change so that they'd look like this:

os->NotifyObservers(nullptr, ObserverTopic::QuitApplication, nullptr);

Or this:

Services.obs.notifyObservers(null, ObserverTopic.QuitApplication);

Pretty straightforward stuff.

This would have quite a few coding benefits:

- It would make trivially simple to document the topics, their subjects
and payloads. Potentially this could even be integrated with our
generated documentation and code search tools making developer lives a
lot easier.

- It would make it far easier to retire/rename a topic, since C++ code
still using an unknown topic would fail to compile, and JS would throw.

- It would prevent typos from slipping in.

- It would make the naming consistent (compared to current things like
"apz:cancel-autoscroll", "APZ:FlushActiveCheckerboard" and
"apz-repaints-flushed").

It would also help with code size, performance and memory use:

- It makes the generated code a lot leaner, since the identifiers are
now integers under the hood, which require far less code to handle than
strings. For reference, on a Linux PGO build my patch for bug 1348273
shrinks libxul.so by ~35KiB while touching only ~100 call sites, the
observer service has thousands.

- We wouldn't need a hash-table anymore. A fixed-size array of topics,
each holding a vector of listeners, suffices and it can be indexed
directly. Besides the obvious performance improvements (no string
comparisons, no need to walk a sparse structure with all the cache/TLB
effects that come with it) it would also save memory.

While the above effects would be hard to measure (and possibly below the
noise threshold) they would definitely be there; and as we learned with
Quantum significant performance improvements in a codebase as large as
Firefox often come from plenty of small fixes.

This isn't without drawbacks though. The biggest ones I could think of
would be:

- We'd have a new header file that would be included in a lot of places,
and adding, removing or changing a topic would touch it, causing much
recompile.

- One would not be able to add/remove/change a topic in an artifact
build. I'm not sure how many JS-only users of the observer service there
are but if they're not many then this wouldn't be a big deal.

- This would most certainly need significant changes in Thunderbird too,
but I'd help with that, I promise.

So, what do you think? This is non-trivial work but I think that it
would be worth the fuss.

 Gabriele

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1348273
Make AnnotateCrashReport() more robust by turning annotations into a
well known list of constants



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: No more #ifdef MOZ_CRASHREPORTER directives needed when using crash reporter functions

[Gabriele Svelto]

On 24/11/2017 05:18, Frank-Rainer Grahl wrote:
> Hi,
> 
> I didn't see package-manifest changes in the bug.
> 
> I assume this needs to stay in as-is?

Yes, when we build with --disable-crashreporter we still don't want the
various other bits (including the crash reporter UI) to be built and
packaged. Only the C++ interface is stubbed with the dummy implementation.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

No more #ifdef MOZ_CRASHREPORTER directives needed when using crash reporter functions

[Gabriele Svelto]

[cross-posting to firefox-dev]

 Fellow mozillians,
bug 1402519 [1] just landed and it introduces a dummy implementation of
the crash reporter which is built when configuring with
--disable-crash-reporter. This removes the need for bracketing calls to
the crash reporter with #ifdef MOZ_CRASHREPORTER / #endif preprocessor
directives. You can now freely use the crash reporter methods without
worrying about it being enabled at compiler time or not.

I've also removed all the existing directives and only a few remain
where they are actually needed. JavaScript consumers should be
unaffected save for the following pattern which was used in a few places
to detect if the crash reporter had been compiled:

if ("nsICrashReporter" in Ci &&
Services.appinfo instanceof Ci.nsICrashReporter) {
// Crash reporter is enabled
}

This doesn't work anymore as the nsICrashReporter interface is always
present. You can replace it with:

if (AppConstants.MOZ_CRASHREPORTER) {
  // Crash reporter is enabled
}

This touched a lot of places in the code so if anything crash-related
seems wrong please let me know.

 Gabriele

[1] Use a dummy crashreporter implementation when building with
--disable-crashreporter
https://bugzilla.mozilla.org/show_bug.cgi?id=1402519




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: More ThinkStation P710 Nvidia tips (was Re: Faster gecko builds with IceCC on Mac and Linux)

[Gabriele Svelto]

On 06/11/2017 22:44, Jeff Gilbert wrote:
> Price matters, since every dollar we spend chasing ECC would be a
> dollar we can't allocate towards perf improvements, hardware refresh
> rate, or simply more machines for any build clusters we may want.

And every day our developers or IT staff waste chasing apparently random
issues is a waste of both money and time.

> The paper linked above addresses massive compute clusters, which seems
> to have limited implications for our use-cases.

The clusters are 6000 and 8500 nodes respectively, quite small by
today's standards. How many developers do we have? Hundreds for sure, it
could be a thousand looking at our current headcount so we're in the
same ballpark.

> Nearly every machine we do development on does not currently use ECC.
> I don't see why that should change now.

Not true. The current Xeon E5-based ThinkStation P710 available from
Service Now has ECC memory and so did the previous models in the last
five years. Having a workstation available w/o ECC would actually be a
step backwards.

> To me, ECC for desktop compute
> workloads crosses the line into jumping at shadows, since "restart
> your machine slightly more often than otherwise" is not onerous.
Do you have data to prove that this is not an issue?

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: More ThinkStation P710 Nvidia tips (was Re: Faster gecko builds with IceCC on Mac and Linux)

[Gabriele Svelto]

On 04/11/2017 01:10, Jeff Gilbert wrote:
> Clock speed and core count matter much more than ECC. I wouldn't chase
> ECC support for general dev machines.

The Xeon-W SKUs I posted in the previous thread all had identical or
higher clock speeds than equivalent Core i9 SKUs and ECC support with
the sole exception of the i9-7980XE which has slightly higher (100MHz)
peak turbo clock than the Xeon W-2195.

There is IMHO no performance-related reason to skimp on ECC support
especially for machines that will sport a significant amount of memory.

Importance of ECC memory is IMHO underestimated mostly because it's not
common and thus users do not realize they may be hitting memory errors
more frequently than they realize. My main workstation is now 5 years
old and has accumulated 24 memory errors; that may not seem much but if
it happens at a bad time, or in a bad place, they can ruin your day or
permanently corrupt your data.

As another example of ECC importance my laptop (obviously) doesn't have
ECC support and two years ago had a single bit that went bad in the
second DIMM. The issue manifested itself as internal compiler errors
while building Fennec. The first time I just pulled again from central
thinking it was a fluke, the second I updated the build dependencies
which I hadn't done in a while thinking that an old GCC might have been
the cause. It was not until the third day with a failure that I realized
what was happening. A 2-hours long memory test showed me the second DIMM
was bad so I removed it, ordered a new one and went on to check my
machine. I had to purge my compilation cache because garbage had
accumulated in there, run an hg verify on my repo as well as verifying
all the installed packages for errors. Since I didn't have access to my
main workstation at the time I had wasted 3 days chasing the issue and
my workflow was slowed down by a cold compilation cache and a gimped
machine (until I could replace the DIMM).

This is not common, but it's not rare either and we now have hundreds of
developers within Mozilla so people are going to run into issues that
can be easily prevented by having ECC memory.

That being said ECC memory also makes machines less susceptible to
Rowhammer-like attacks and makes them detectable while they are happening.

For a more in-depth reading on the matter I suggest reading "Memory
Errors in Modern Systems - The Good, The Bad, and The Ugly" [1] in which
the authors analyze memory errors on live systems over two years and
argue that SEC-DED ECC (the type of protection you usually get on
workstations) is often insufficient and even chipkill ECC (now common on
most servers) is not enough to catch all errors happening during real
world use.

 Gabriele

[1] https://www.cs.virginia.edu/~gurumurthi/papers/asplos15.pdf



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: More ThinkStation P710 Nvidia tips (was Re: Faster gecko builds with IceCC on Mac and Linux)

[Gabriele Svelto]

On 28/10/2017 01:08, Sophana "Soap" Aik wrote:
> Thanks Gabriele, that poses a problem then for the system build we have
> in mind here as the i9's do not support ECC memory. That may have to be
> a separate system with a Xeon.

Xeon-W processors are identical to the i9 but come with more
workstation/server-oriented features such as ECC memory support; they
are also offered with slightly higher peak clock speed to equivalent
i9s. Here's a side-by-side comparison of the top 4 SKUs in both families:

https://ark.intel.com/compare/123589,126709,123767,126707,125042,123613,126793,126699

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: More ThinkStation P710 Nvidia tips (was Re: Faster gecko builds with IceCC on Mac and Linux)

[Gabriele Svelto]

On 27/10/2017 01:02, Gregory Szorc wrote:
> Sophana (CCd) is working on a new system build right now. It will be based
> on the i9's instead of dual socket Xeons and should be faster and cheaper.

... and lacking ECC memory. Please whatever CPU is chosen make sure it
has ECC support and the machine comes loaded with ECC memory. Developer
boxes usually ship with plenty of memory, and they can stay on for days
without a reboot churning at builds and tests. Memory errors happen and
they can ruin days of work if they hit you at the wrong time.

 Gabriele




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: We need better canaries for JS code

[Gabriele Svelto]

On 18/10/2017 10:28, David Teller wrote:
> I have one proposal. Could we change the behavior of the JS VM as follows?
> 
> - The changes affect only Nightly.
> - The changes affect only mozilla chrome code (including system add-ons
> but not user add-ons or test code).
> - Any programmer error (e.g. SyntaxError) causes a crash a crash that
> displays (and attaches to the CrashReporter) both the JS stack in and
> the native stack.
> - Any SyntaxError is a programmer error.
> - Any TypeError is a programmer error.

I'd love to have that, even only on try it would catch a lot of stuff.
BTW the problem of errors in JS code not affecting test results in any
way is very widespread. We have plenty of tests spewing out actual
errors (often only when certain races occur and objects/functions become
undefined) that have no visible impact on the test results themselves
since they're ignored.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: C++ function that the optimizer won't eliminate

[Gabriele Svelto]

On 09/10/2017 13:47, Henri Sivonen wrote:
> I omitted volatile, because the GCC manual said it has no effect on
> basic asm. However, it turns out it still has an effect on extended
> asm, which is what this is. Oops.

Yes, volatile implies the statement has side effects and so it cannot be
moved around or eliminated (even though GCC might prove that it is
indeed useless). So in your example https://godbolt.org/g/iTBXYW
you're telling GCC you're doing something with the contents of ptr,
which indeed forces it to emit separate adds for every iteration. In the
case with volatile https://godbolt.org/g/35xcCL you're telling it you're
doing something with ptr and you're doing something else it doesn't know
about but which may touch other variables or memory locations. The
latter is a pretty big hammer, it works like a barrier in the code so if
GCC has promoted some stuff from memory to registers it will be forced
to spill them and reload them, statements will not be moved around it,
etc... If the only thing that interests you is pretending that the
output of the function is being consumed then the former non-volatile
option is preferable as it will not have such a negative impact on code
generation.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: C++ function that the optimizer won't eliminate

[Gabriele Svelto]

On 06/10/2017 11:00, Henri Sivonen wrote:
> Do we already have a C++ analog of Rust's test::black_box() function?
> I.e. a function that just passes through a value but taints it in such
> a way that the optimizer can't figure out that there are no side
> effects. (For the purpose of ensuring that the compiler can't
> eliminate computation that's being benchmarked.)
> 
> If we don't have one, how should one be written so that it works in
> GCC, clang and MSVC?
> 
> It's surprisingly hard to find an answer to this on Google or
> StackOverflow, and experimentation on godbolt.org suggests that easy
> answers that are found are also wrong.
> 
> Specifically, this isn't the answer for GCC:
> void* black_box(void* foo) {
>   asm ("":"=r" (foo): "r" (foo):"memory");
>   return foo;
> }

IIUC what you are looking for is the '+' constraint which implies the
parameter is both read and written in the asm statement, e.g.:

void* black_box(void* foo) {
  asm ("":"+r" (foo): "r" (foo):"memory");
  return foo;
}

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: How to configure mozilla Firefox build options to build a customized release/developer edition build?

[Gabriele Svelto]

On 28/08/2017 09:28, Erxin Shang wrote:
> I'm trying to build a release version Firefox. But after the build there 
> seems still a little bit different compare to the official build. Such as the 
> official build doesn't contain the folder chrome/, components/, gmp-fake/ etc 
> in the root of Firefox folder. 

A release version will have the MOZILLA_OFFICIAL variable set (i.e. add
"mk_add_options MOZILLA_OFFICIAL=1" to your .mozconfig file). As the
name implies however such a build cannot be redistributed as is, just so
that you know.

> My question is how to configure the options to build a version just like the 
> official Firefox but it's a nightly build? Where can I find the build 
> options, such as release/developer edition, which are used by Mozilla?
You will find the release mozconfig files under browser/config/mozconfigs

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: webidl: partial interfaces and build-time configuration

[Gabriele Svelto]

On 06/08/2017 21:56, Enrico Weigelt, metux IT consult wrote:
> For example, I'm currently working on making the whole wakelock
> stuff optional (eg. only built on android). Navigator.webidl
> references MozWakeLock, which wouldn't exist w/o wakelocks.

Do you mean navigator.requestWakeLock()? That's deprecated and should be
removed altogether. We already have bug 1369194 [1] for that, I haven't
had time to work on it yet so patches are welcome. You can needinfo me
for details about which parts of the implementation should also be removed.

 Gabriele

[1] Remove navigator.requestWakeLock()
https://bugzilla.mozilla.org/show_bug.cgi?id=1369194



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: C++ performance test harness?

[Gabriele Svelto]

On 05/07/2017 16:31, William Lachance wrote:
> We do run these tests in automation, you can track the results in
> perfherder under the platform_microbench framework:
> 
> https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bautoland,2aa20d42dc00d0a953f871f76056d8d217dc569e,1,6%5D=%5Bmozilla-inbound,2aa20d42dc00d0a953f871f76056d8d217dc569e,1,6%5D=%5Bmozilla-inbound,2aa20d42dc00d0a953f871f76056d8d217dc569e,220714,290981367,6%5D

There was talk about turning them off some time ago:

https://groups.google.com/forum/#!topic/mozilla.dev.platform/u4tUwHJg-Dc

Are they still maintained?

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Phabricator Update, July 2017

[Gabriele Svelto]

On 14/07/2017 05:39, Boris Zbarsky wrote:
> I should note that with GitHub what this means is that you get
> discussion on the PR that should have gone in the issue, with the result
> that people following the issue don't see half the relevant discussion.
> In particular, it's common to go off from "reviewing this line of code"
> to "raising questions about what the desired behavior is", which is
> squarely back in issue-land, not code-review land.
> 
> Unfortunately, I don't know how to solve that problem without
> designating a "central point where all discussion happens" and ensuring
> that anything outside it is mirrored there...

Yeah, we frequently had that problem with Gaia issues as part of Firefox
OS. Reviews were done on GitHub's PR so the bugzilla entries were often
empty (even for bugs with huge, complex patch-sets). To gather any kind
of information one had to go and check the comments on the PR itself
which not only was less-than-optimal but made life a lot harder for
those not directly involved with development.

It's not a dealbreaker for me but it's something that should be kept in
mind.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: More Rust code

[Gabriele Svelto]

On 13/07/2017 01:06, Eric Rahm wrote:
>   * *using breakpad* - was the problem that creating wrappers to access
> the c/c++ code was too tedious? Could bindgen help with that, if not
> it would be interesting gather some details about why it wouldn't
> work and file bugs against it.

There were multiple reasons: I had no experience in writing Rust
bindings and we wanted to have it ready quickly; additionally we're in
the process of rewriting this part of Breakpad functionality in Rust so
we decided that it would all become Rust at a later point and writing
bindings wouldn't have been a good use of the time.

>   * *pingsender* - was something like https://hyper.rs/ not around when
> you were working on it or is this a case of finding the things you
> want can be difficult in rust-land? Either way it might be a good
> idea for us to put together a list of "sanctioned" crates for
> various tasks.

hyper.rs was already available but it's a big dependency and I didn't
feel like pulling it into mozilla-central just for the pingsender. I
believe we'll have HTTP functionality available from Rust code at some
point in the future so when it's ready I might as well port the
pingsender to it. As Mike said though the program is very small and
self-contained at this point so it would be nice to keep it that way
even after a Rust rewrite if possible.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: More Rust code

[Gabriele Svelto]

On 10/07/2017 12:29, Nicholas Nethercote wrote:
> Hi,
> What are the obstacles? Here are some that I've heard.
> [...] 
> Anything else?

In the past year I wrote two tools (minidump-analyzer and pingsender)
that ship with Firefox but are separate executables so both were good
candidates for being written in Rust in the first place. I implemented
both in C++ because of different issues:

- The minidump-analyzer relies on Google Breakpad machinery that is
currently C++. It's being rewritten in Rust but it's not there yet and
since we needed it to quickly get more crash information going the Rust
route would have simply taken too long. It is a good candidate for being
rewritten in Rust in the coming 12 months though.

- The pingsender talks to our telemetry infrastructure so needs
HTTP/HTTPS functionality but without linking with libxul. In this case
if I had a crate that offered HTTP functionality I would have gone
straight to Rust but alas it wasn't there. So instead it's a blob of
platform-specific C++ that dlopen()s libcurl on Mac and Linux and relies
on WININET on Windows.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Faster gecko builds with IceCC on Mac and Linux

[Gabriele Svelto]

On 24/03/2017 05:39, Gregory Szorc wrote:
> The introduction of Ryzen has literally changed the landscape
> and the calculus that determines what hardware engineers should have.
> Before I disappeared for ~1 month, I was working with IT and management to
> define an optimal hardware load out for Firefox engineers. I need to resume
> that work and fully evaluate Ryzen...

The fact that with the appropriate motherboard they also support ECC
memory (*) made a lot of Xeon offerings a lot less appealing. Especially
the workstation-oriented ones.

 Gabriele

*) Which is useful to those of us who keep their machines on for weeks
w/o rebooting or just want to have a more reliable setup



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Please write good commit messages before asking for code review

[Gabriele Svelto]

On 09/03/2017 22:35, Eric Rescorla wrote:
> I'm in favor of good commit messages, but I would note that current m-c
> convention really pushes against this, because people seem to feel that
> commit messages should be one line. Not sure what to do about that,
> but thought I would mention it.

Yeah, I always thought we had a policy of one-line commit messages but
I think that originated back in the day when one would attach a patch on
Bugzilla and added a comment there describing it in detail. With
mozreview it seems more sensible to put that detailed description in the
commit message.

 Gabriele




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Is there a way to improve partial compilation times?

[Gabriele Svelto]

On 08/03/2017 01:11, Mike Hommey wrote:
> You probably want a desktop machine, not a new laptop.

I second that, modern laptops are usually thermally limited. I actually
drilled holes in the back of my Thinkpad to improve airflow (and it did
improve build times).

My main box is a not-so-recent Xeon (E3-1270v2) and a clobber build is
usually 20-25 minutes. A more modern machine would be measurably faster
than that.

Another trick you could use is to run builds before you start working to
warm ccache. I've got a cronjob that runs all the builds I care about on
clean trees in the early morning, so when I get to work builds with my
patches applied run mostly out of ccache.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Content process launch time distribution

[Gabriele Svelto]

On 07/02/2017 18:57, Gabor Krizsanits wrote:
> For a temporary workaround until we can speed up content process start up
> and initialization time, we might want to use the pre-allocated process
> manager for the e10s-multi case. Maybe even force to pick one from the
> existing processes unless there is one already available.

I was also thinking about it. I'm not sure in what shape pre-allocated
process support is since it's not been tested for a year already (only
Firefox OS used it AFAIK) but it's a good way to take process startup
out of the critical path at the cost of some extra memory consumption.

 Gabriele




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: PSA: sysconf(_SC_NPROCESSORS_ONLN) is apparently not reliable on Android / ARM

[Gabriele Svelto]

On 23/01/2017 18:44, Gian-Carlo Pascutto wrote:
> If only we had some crossplatform runtime that abstracted such system
> specifics away from us
> (https://bugzilla.mozilla.org/show_bug.cgi?id=663970) then maybe we
> wouldn't have to re-fix the same bugs every 5 years.

On this topic, I've heard multiple times from multiple sources to steer
away from NSPR for cross-platform stuff and use MFBT instead as much as
possible. But MFBT doesn't even come close to supporting all the
system-specific stuff that NSPR supports. Do we plan on extendind MFBT
to cover what's missing?

 Gabriele




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: forward declarations vs includes

[Gabriele Svelto]

On 12/01/2017 09:05, Mike Hommey wrote:
> +1
> 
> The sad part is that it's not followed enough.

The include hell [1] bug hasn't seem some action in a while. We might
set some time aside to do a bit of cleanup on the most commonly used
headers. I remember that the last time we did a significant push like
this we shed a lot of unnecessary stuff and improved build times measurably.

 Gabriele

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=includehell




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Introducing mozilla::Result for better error handling

[Gabriele Svelto]

On 22/12/2016 19:14, Bobby Holley wrote:
> We've had this debate several times already, culminating in the attempt to
> ban NS_ENSURE_* macros. It didn't work.

If we don't want to get rid of it then why not make the intent
immediately obvious from the name? Something like MOZ_RETURN_IF_FAILED()
or MOZ_RETURN_ON_ERROR().

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: [e10s] Higher priority handling for vsync in child processes

[Gabriele Svelto]

On 10/11/2016 18:12, smaug wrote:
> vsync handling happens usually right after a task (per HTML spec).
> Basically we have now two event queues with similar priority, and they
> are handled the same priority, but since
> the other one is used very rarely, events added to it get processed on
> average sooner than the ones added to the normal queue.
> (We can't starve the normal queue, so both need to be processed)
> 
> MemoryPressure seems to want to really processed before any new task, so
> the new system can't be used for that.

Thanks for clearing that up.

> Looks like memory pressure works only on some platforms.

Yes, only on Windows and Android AFAIK, and the former has some serious
issues when it kicks in (which almost never happens BTW, we tend to
OOM-crash well before it does).

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: [e10s] Higher priority handling for vsync in child processes

[Gabriele Svelto]

On 10/11/2016 00:22, smaug wrote:
> Parent process doesn't yet have higher priority handling [2].
> Need to fix racy tests first.
> Locally using higher priority also in parent process seems to make tab
> throbber smoother.

We have an ad-hoc mechanism for scheduling memory pressure event ahead
of all other pending tasks [1]. Do you think we could use this new
functionality to replace it? It would be nice to have this handled via a
generic mechanism too.

 Gabriele

[1]
https://dxr.mozilla.org/mozilla-central/rev/336759fad4621dfcd0a3293840edbed67018accd/xpcom/threads/nsThread.cpp#1444




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Removal of B2G from mozilla-central

[Gabriele Svelto]

Replying to myself since nobody thought it worthwhile to reply and have
an open, honest discussion on this ML about this.

Parts of the gonk widget have already been removed; so the decision to
do it was taken was not communicated publicly. Or maybe it was never up
for discussion. I don't know which.

Either way it seems to me that our culture of working in an open and
transparent way - which we proudly proclaimed only a few years back in a
video [1] - is disappearing.

I find this both sad and worrisome. The reason I find it worrisome is
that one thing that hurt Firefox OS the most was that all decisions
regarding the project's direction were taken internally. This opaqueness
made it impossible to attract contributors and build a community around
the project.

I can only hope this was an exception and not our general approach to
development because this attitude can hurt us, badly.

 Gabriele

[1] https://www.youtube.com/watch?v=AIKLdVEWPrE

"While most products and technologies are developed behind closed doors,
ours are cultivated out in the open, for everyone to see".



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Help ridding tests of unsafe CPOWs

[Gabriele Svelto]

 Hi Blake,

On 19/10/2016 00:28, Blake Kaplan wrote:
> I've been seeing a pattern of "unsafe" CPOWs causing our browser-chrome
> mochitests to go intermittently orange. Generally, it seems that a test
> randomly turns orange, gets starred and ignored until RyanVM or another one
> of our sheriffs gets tired of seeing it and needinfos somebody in the hopes
> of fixing it.

I remember we did a big push in the War on Orange maybe three (or four?)
years ago. We could do it again; calling out to everybody to take
intermittent tests in the modules they're familiar with and start fixing
them. Personally I'd happily dedicate a chunk of my time to doing it for
a couple of weeks; having to re-trigger stuff every single time I make a
try run drives me nuts.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Removal of B2G from mozilla-central

[Gabriele Svelto]

On 04/10/2016 01:22, Gregory Szorc wrote:
> https://dxr.mozilla.org/mozilla-central/search?q=gonk seems to
> contradict your assertion that gonk is well-contained.

I thought that the analysis in my first post was sufficiently detailed
but here's one with line numbers to get a more accurate idea:

- All gonk references and code in the following directory will be
removed entirely because it's needed only for B2G-specific APIs and
those are already being removed. This means that the following top-level
directories will be purged of all gonk-specific code:

* accessible
* addon-sdk
* browser
* caps
* chrome
* devtools
* docshell
* dom (save for a few lines in dom/media, see below)
* embedding
* extensions
* hal (save for a single file)
* image
* intl
* ipc
* layout
* memory
* modules
* mozglue
* netwerk
* parser
* rdf
* security
* storage
* testing
* toolkit
* tools
* uriloader (not 100% sure, but it's 3 files anyway)
* xpcom (what's being done could be handled externally from gecko)
* xpfe

What would remain is the following; first the actual code:

* widget/gonk - roughly ~50K lines but would drop to ~40K once we drop
suppport for older Android versions as well as APIs that have been removed

* b2g

  ~20K lines which would also drop considerably due to the removal of
the APIs, completely self-contained

* dom/media

  ~4K lines, which would be cut down once camera & overlay support are
removed as well as the phone-specific, non-standard video and audio
formats. The rest of the code is mostly self-contained within
gonk-specific files

* gfx

  ~2K lines a significant amount of which are related to camera and
screen recording code that can be removed

* hal

  Only one file would be kept in hal/gonk of ~2K lines (which are likely
to be reduced once support for non-standard APIs is removed). A handful
of lines would remain in hal/Hal.cpp and hal/Hal.h but if we drop the
process priority manager (which I would do) those would also go. ~10
lines in hal/linux/LinuxPower.cpp

* media

  Less than 1K lines, some of which are shared with Fennec and some
which can be easily removed (e.g. they were used only on certain
devices/versions)

Then the configuration bits

* build - 10 lines shared with Fennec
* config -  One line, could be possibly removed
* js - 2 lines
* nsprpub - 7 lines
* python - ~50 lines
* xpcom

All in all we're talking about less than ~70K lines which would likely
drop to less than ~50K lines once we've removed all the non-standard
APIs. Of these most are self-contained save for gfx, media and dom/media
which are arguably the ones where there's more intermingling between
gonk-specific code and the rest though it's only a handful of lines.

> There are
> literally references to gonk throughout the tree. Every reference that
> isn't self-contained introduces cognitive dissonance when someone
> encounters it. They have to consider the existence of gonk when reading
> and changing the code. This makes changing code harder and undermines
> the ability for Firefox/Gecko to "evolve quickly." Even the very
> presence of unused, self-contained code (like gonk widgets) adds
> overhead because it can make common operations like refactoring more
> time consuming. And if someone breaks the code (because it isn't used)
> and a bug gets filed to track that, now you've introduced overhead for
> people to triage said bugs. These problems don't exist when the code
> doesn't exist. That's why we should aggressively delete unused and
> unsupported code.

While I can relate to that this also discounts the advantage which we
would have by retaining the code which is the ability to deploy gecko on
Android-based devices directly. That's something that proved to be
useful outside of the phone domain.

Also we're talking about a tier 3 platform which means that if a
refactoring breaks it, it will be up to the people that are involved
with it to fix it. IIRC bugs for tier 3 platforms are not triaged either
so that shouldn't be a problem.

Besides there's unsupported code that's been living in our codebase for
ages and nobody seemed to be bothered by it. Searching for references to
long-defunct platforms such as Maemo, MeeGo or OS/2 still yields results
in our codebase.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Removal of B2G from mozilla-central

[Gabriele Svelto]

> Respectively, it seems like these requests were ultimately not included
> in the final decision.

I would like to know why; I think that's not much to ask. I would also
like to know why this decision was made without any public discussion.
As I've pointed out the removal of another widget was discussed on this
list only a few months ago.

The gonk widget is made of roughly 60K lines of code out of the 14
million lines of our codebase is made of[1]. It's not a large number
both in absolute and relative terms and also well contained so why can't
it be kept around?

I'd also like to point out that as far as I know Mozilla has four times
the staff as when it was introduced, so I doubt that the effort of a few
people keeping it around as tier 3 would impact the organization much.

 Gabriele

[1] https://www.openhub.net/p/firefox



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Removal of B2G from mozilla-central

[Gabriele Svelto]

On 30/09/2016 06:04, Chris Peterson wrote:
> Is Gonk used anywhere besides B2G? Can we remove all Gonk code, e.g.
> dom/camera/Gonk* and #ifdef MOZ_WIDGET_GONK?

Gonk is not used anywhere else, some of it's code was merged with
Fennec's code to reduce the maintenance burden but there's still quite a
bit that's B2G-specific.

Since gonk is a widget on its own, during the internal discussions about
it I - and others who worked on B2G - repeatedly asked for the gonk
widget to be left in the code even after the removal of all the
B2G-specific APIs (as a tier3 platform obviously).

This would allow a few things to happen:

- We would still have the functionality to render web content into the
low-levels of the Android graphics stack

- We could keep developing B2G OS using fully web APIs by implementing
all the B2G-specific bits in external daemons with web-facing interfaces
that would live out of mozilla-central

As far as mozilla-central is concerned this would mean that all gonk
directories for non-standard APIs would still be removed, this includes:

- dom/media/platforms/gonk
- dom/cellbroadcast/gonk
- dom/system/gonk
- dom/secureelement/gonk
- dom/nfc/gonk
- dom/telephony/gonk
- dom/mobilemessage/gonk
- dom/icc/gonk
- dom/mobileconnection/gonk
- dom/wappush/gonk
- dom/voicemail/gonk

All the code protected by MOZ_WIDGET_GONK but B2G-specific would still
be removed, that means pretty much all the code under netwerk, dom,
security, etc...

What would be left would be essentially:

hal/gonk
widget/gonk

The amount of code in both directories would be cut down significantly
though as all the code needed to support the non standard APIs would be
removed.

We'd still have some MOZ_WIDGET_GONK-specific stuff under gfx/ and
media/ but it's not much and I wouldn't mind removing all the
non-necessary stuff (e.g. screen recording, overlays, ...). Besides
being tier3 nobody should mind breaking it. We'd fix it up ourselves.

Since there's not been a public discussion on this like we had for the
Qt widget [1] I'd like to talk about it here. Note that I've been
personally taking part in the efforts of removing the B2G-specific APIs
mostly because I know how they work and what non-obvious parts of our
codebase they affect.

 Gabriele

[1] https://lists.mozilla.org/pipermail/dev-platform/2016-June/015447.html



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Intent to unship: SimplePush API

[Gabriele Svelto]

 Hello,
the non-standard SimplePush API provided a mechanism for Firefox OS to
receive push messages before the standard Push API was finalized. With
the standard Push API now implemented in Gecko and already enabled on
Android builds it should be easy to enable it on B2G too (see [1] if you
want to help out with that). We have a number of consumers of this API
but they're only Firefox OS-specific (e.g. Telegram) but none of them
works on the current B2G trunk because of the ongoing changes to remove
non-standard API. As such I will be removing the SimplePush API in bug
1296579 [2].

 Gabriele

[1] Implement Push API on FirefoxOS
https://bugzilla.mozilla.org/show_bug.cgi?id=1151180
[2] Remove the SimplePush API
https://bugzilla.mozilla.org/show_bug.cgi?id=1296579



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Return-value-optimization when return type is RefPtr

[Gabriele Svelto]

On 17/06/2016 16:57, Gerald Squelart wrote:
> From what *I* understand, RVO is guaranteed (or at least supported 
> everywhere?) when there is only one stack variable that is returned, e.g.:
> C foo()
> {
>   C rv;
>   // ... (put stuff in rv)
>   return rv;
> }
> In this case, the caller function stack can host 'rv' directly, no copies 
> needed.

Sounds like the compiler needs visibility into the called function for
this to work so it'll either be limited to functions compiled as part of
the same invocation or when LTO is enabled. Also what about non-leaf
calls? I.e.

C foo()
{
  C rv;
  // ... (put stuff in rv)
  return rv;
}

C bar()
{
  // ...
  return foo();
}

Would it still work? Do we care about it?

> (Any actual expert who knows the truth?)

That's most likely dependent on a combination of the actual code,
compiler, compiler version and flags used when compiling (and possibly
even target dependent if it interacts with other things like register
promotion of stack variables, etc...).

 Gabriele




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

[Gabriele Svelto]

On 01/06/2016 15:20, Jonathan Kew wrote:

Does this suggest that we're not sufficiently proactive about firing
memory-pressure notifications, so that we'll free up memory from various
caches, etc? It looks like we regard 128MB of available VM as "low" (see
[1]) on Windows 32-bit, but apparently we're liable to suffer small-OOM
crashes well before we reach that point. That doesn't seem like a
healthy balance.


Those values were set when the AvailableMemoryTracker was introduced 5 
years ago [1]. I'd say we should probably revisit them especially in the 
light of these findings.


That being said in Firefox OS we employed memory-pressure notifications 
quite successfully to keep processes alive when memory was running low 
but we didn't rely on a single threshold because it didn't prove very 
effective. Instead we used a floating trigger which started at a certain 
level of free memory which once passed would trigger memory-pressure 
events. Once that happened we'd lower the threshold. If we hit this new 
threshold we'd fire memory-pressure events again, if not we'd try to 
raise the threshold again to the higher level. Exponential back-off was 
used to avoid having the threshold fluctuate too much between the two 
values.


Something similar could be implemented on Windows. If there's consensus 
I'm happy to look into it.


 Gabriele

[1] On Windows, fire a memory-pressure event when the amount of 
available virtual address space or physical memory is low

https://bugzilla.mozilla.org/show_bug.cgi?id=670967

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

[Gabriele Svelto]

On 31/05/2016 13:26, Gijs Kruitbosch wrote:
> We could do a find/replace of no-arg calls to a new macro that uses
> MOZ_CRASH with a boilerplate message, and make the argument non-optional
> for new uses of MOZ_CRASH? That would avoid the problem for new
> MOZ_CRASH() additions, which seems like it would be wise so the problem
> doesn't get worse? Or is it not worth even that?

What about adding file/line number information? This way one could
always tell where it's coming from even if it doesn't have a descriptive
string.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Removing the Chromium event loop

[Gabriele Svelto]

On 31/03/2016 07:42, Kyle Huey wrote:
> I'll pose to you the same question I posed bsmedberg, is this actually
> worth fiddling with all the existing runnables.

I did some testing around a couple of years ago and the answer is the
same as usual: it depends. On modern x86 machines I doubt one would be
able to measure the difference; sure it's there, but it's small enough
that it's generally not worth the fuss.

On entry-level ARM stuff (Cortex A5, A7, etc...) it's definitely there,
but mostly because using nsIRunnable requires atomics for ref-counting
which are very expensive on those cores. So there's definitely a cost
over stuffing a raw (function) pointer into an array and calling it later.

So it comes down to a cost/benefit tradeoff. On the one hand it would be
nice to have a lighter alternative to nsIRunnable provided it doesn't
require massive changes in the code, on the other hand its impact would
be limited on the vast majority of our users.

On this topic, did anyone experiment with trying to lighten the
synchronization burden when handling nsEventQueues? Both nsThread and
nsThreadPool acquire a mutex each time we need to get the next runnable;
we could pull out all pending runnables every time we acquire the lock
(up to a predefined maximum) to amortize the synchronization cost. In my
measurements mutex-handling was also quite expensive on low-end ARM
cores, not so much on x86 as long as the mutex was not contended.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: An analysis of content process memory overhead

[Gabriele Svelto]

On 15/03/2016 04:34, Nicholas Nethercote wrote:
> - "heap-overhead" is 4 MiB per process. I've looked at this closely.
>   The numbers tend to be noisy.
> 
>   - "page-cache" is pages that jemalloc holds onto for fast recycling. It is
> capped at 4 MiB per process and we can reduce that with a jemalloc
> configuration, though this may make allocation slightly slower.

We aggressively got rid of that on B2G by sending memory-pressure events
to apps that were unused. We did have the advantage there that we had
only one page per process so establishing if one was not being used was
very easy. On desktop Firefox we might consider to try and minimize the
memory usage of processes which do not have active tabs (e.g. none of
the tabs is visible, or none of the tabs has received input for a while).

Besides the immediate memory usage reduction this had the important
side-effect of reducing steady-state consumption. A lot of the
structures and caches that were purged had often been bloated by
transient data required only during startup. Once minimized they would
start to grow again once a process would become active again but never
as much as before the minimization.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: On nsICancelableRunnable (or how bad decisions come back to haunt you)

[Gabriele Svelto]

On 13/03/2016 09:22, Kyle Huey wrote:
> Much of the "disease" you see is simply support for DOM worker threads,
> because every runnable that can run on those threads must be
> cancelable.  Workers do not guarantee one of the invariants that other
> XPCOM threads do: that every runnable is eventually run after being
> successfully dispatched.  Instead we guarantee that every runnable is
> either Run() or Cancel()ed.

Yeah, that's something I had noticed and I was trying to understand why.
So it seems that this interface actually turned out to have a proper use
after all.

> The disadvantage of pushing this into the event queue is that today
> runnables that are dispatched from other threads to workers *must*
> implement nsICancelableRunnable, which means authors of code that runs
> on workers must think about this.  We do already have an exception for
> NS_DispatchToCurrentThread though (via ExternalRunnableWrapper), so this
> may be something we can relax.  It would generally require us to write
> cleanup code (for the existing runnables and any future ones) to run
> from destructors, and historically destructors could run on either the
> dispatching thread or the executing thread (although that has changed
> somewhat.)

My gripes with its original design is that it's inherently
non-thread-safe and non-atomic (i.e. Cancel() can be called while Run()
is being executed) and it essentially drops the burden of implementing
proper cancellation on the user. My idea of handling cancellation
externally was motivated by having the operation be both thread-safe and
fully atomic (either something runs or it doesn't). This however implies
that if the task was fire-and-forget, then destruction will always be
done by the canceling thread and will happen immediately; in the current
scenario destruction will be done by the thread owning the queue, and it
will happen only when the runnable is drained from it.

> Regardless, your plan should probably involve me ;)

Yeah, this was my idea when nsICancelableRunnable was practically only
used in FxOS-specific code to support canceling potentially long-running
operations posted on the main thread (and which might have not executed
yet). Since it's use has evolved it might not be worth pursuing it
anymore, which is why I wanted to hear about how it's been used now.

> PS. Subscribe to the mailing list so you don't get caught in moderation.

I'm already subscribed, I just verified it, it's bizarre that the
message was held :-/

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

On nsICancelableRunnable (or how bad decisions come back to haunt you)

[Gabriele Svelto]

 Many moons ago, while we were frantically trying to make Firefox OS 1.0
run half-decently, I introduced a derived class of nsIRunnable that
could be canceled: nsICancelableRunnable. As with many other things that
led to the first release of FxOS this was done in a rush and without
much thinking. It's only goal at the time was to allow me to cancel a
single, simple, stateless runnable that could take a while to run.

Thinking about it afterwards I came to the conclusion that this was a
terrible approach: implementing the Cancel() method was up to the user
and there was no telling on which threads Run() and Cancel() could be
called making it potentially racy. So I thought it could be replaced by
adding a Cancel() method to the event queue that would just remove the
event in a thread-safe way. This seemed a cleaner, safer approach.

But then I looked into our code and discovered that
nsICancelableRunnable has spread like a disease through the codebase and
it's systematically used in workers to let us cancel events posted to
them. Ouch. Some of its uses highlight its problems: in some cases its
only purpose is to force calling the Run() method by the calling thread
so that certain objects get destroyed on that thread (possibly to
prevent leaks). In other cases we have patterns like this:

Run() {
  if (mDoSomething) {
DoSomething();
  }
}

Cancel() {
  mDoSomething = false;
}

Which is fine if both methods are called by the same thread, but most
likely not if they're called on different threads and the interface
itself does nothing to prevent you from doing it.

All in all I'd still like to get rid of it but I'm not sure if my
original approach of moving the cancel functionality into the EventQueue
is the right way to go. Which leads me to the point of this e-mail: to
gather feedback on my idea to get rid of it. Since I don't have enough
knowledge about all the affected areas I'd like to hear from those who
actually used it.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Moving FirefoxOS into Tier 3 support

[Gabriele Svelto]

On 26/01/2016 10:51, jma...@mozilla.com wrote:
> I would assume all gecko patches would get landed on either mozilla-inbound 
> or fx-team.  If you use mozreview and take advantage of the autoland feature, 
> then these specific patches will land on mozilla-inbound.

Thanks, I'll definitely have a look at mozreview/autoland.

 Gabriele




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Moving FirefoxOS into Tier 3 support

[Gabriele Svelto]

On 26/01/2016 10:19, Shawn Huang wrote:
> Hi Fabrice,
> Following this comment, I'm confused. Do we need to check-in into
> b2g-inbound by hand? "checked-in" keyboard no longer supports FirefoxOS
> project? Does this mean only people who have Level 3 access permission can
> land code?

Same here. I'm now unable to land the gecko dependencies for bug 1227980
[1] and that's annoying to say the least considering it took a month to
write and test that thing (not even mentioning the time spent by the
many reviewers and QA people involved). What are we supposed to do with
those kind of patches which are b2g-only? Shall we land them ourselves
on mozilla-central?

 Gabriele

[1] [Dialer] Merge the callscreen app back into the dialer and remove
the associated system app hacks
https://bugzilla.mozilla.org/show_bug.cgi?id=1227980



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: mozglue.dll crash

[Gabriele Svelto]

On 2015-07-24 7:37 AM, Meenakshi Shankar wrote:
 We are using Firefox SDKS for our FF extension (libs generated after
 compiling Firefox 40 beta 4 source using start-shell-msvc2013). As
 there are some differences between Firefox 39 and Firefox 40 binaries,
 instead of mozalloc.lib , the mozcrt.lib is used for building our
 binaries.

 Our extension is enabled. But While launching the browser we could see
 a crash at mozglue.dll when _AtlBaseModule.AddResourceInstance(...) 
 code is called.

 Can someone kindly let us know if we need to make anyother changes
 withe respect to Firefox 40 sdks.

I'm not sure if this is related but bug 858928 [1] slightly changed the
order in which DLLs are loaded during startup. Check [2] for more details.

 Gabriele

[1] Switch XRE_StartupTimelineRecord() from PRTime to mozilla::TimeStamp
https://bugzilla.mozilla.org/show_bug.cgi?id=858928
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=858928#c12




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Proposal to remove `aFoo` prescription from the Mozilla style guide for C and C++

[Gabriele Svelto]

On 07/07/2015 05:12, Jeff Gilbert wrote:
 Notable works or style guides [2] which do not recommend `aFoo`: [3]
[...]

To add another internal datapoint the FxOS gaia codebase is mostly
devoid of this style. There are some places using the m prefix for
pseudo member variables (really just JS attributes) but the a prefix for
arguments is quite rare, I could find only a hundred uses of it or so.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Excessive inbound bustage

[Gabriele Svelto]

On 21/04/2015 08:25, Gabor Krizsanits wrote:
 Maybe because I usually work on core, and such confidence is hard to reach
 there, but I'd like to think at least a try run that check if the patch
 builds on all platform and a full test run on at least one platform is not
 too much sacrifice of ones time.
 
 Personally I think that my time is cheaper than everyone's time. It is
 slow. It is annoying, but holding up ALL the other patches/developers is
 expensive hence a risky option. So I suggest everyone to be very
 conservative about that confident feeling.

I agree; since I work mostly on patches that are relevant for FxOS/B2G I
always run a try build across all architectures before to ensure it
doesn't break anything else. Somehow I was under the impression that
everybody did that.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Evaluating the performance of new features

[Gabriele Svelto]

On 01/02/2015 13:18, Howard Chu wrote:
 People may say I'm biased since I'm the author of LMDB but I have only
 ever posted objective, reproducible comparisons of LMDB to other
 alternatives. http://symas.com/mdb/#bench
 
 If your typical record sizes are smaller than 1KB and you have more
 writes than reads, LMDB may be a poor choice. If you need to support
 large DBs on 32-bit processors LMDB will be a bad choice.

I had come across LMDB myself some time ago and I remember thinking that
it looked like a very good fit for implementing IndexedDB. I didn't
really dig too much into it as my knowledge of our storage code is
limited and I know I wouldn't have the time to try and hack together a
prototype. I think it would be pretty interesting though.

 If you have heavy reads, LMDB may be an ideal choice. Nothing else
 matches it for small footprint, nothing else matches it for reliability,
 and nothing else matches it for read performance.

It sounds like it would be perfect for practically all the IndexedDB
databases I've come across while working on FxOS.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Evaluating the performance of new features

[Gabriele Svelto]

On 30/01/2015 08:45, Jonas Sicking wrote:
 However, it would be cool if we fixed our IndexedDB implementation
 rather than told our own developers not to use it. Web developers are
 not so lucky as to have other options.

Yeah and we're making some pretty heavy use of it within Firefox OS.
I've spent some time trying to reduce its memory footprint (e.g.
including sending PRAGMA shrink_memory commands when low on memory or
when an app is sent to the background) but we really have no way around
using it. Any optimization effort for small  very small databases would
be very welcome as those are rather common AFAIK.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Firefox 2.2.0 and everything.me

[Gabriele Svelto]

On 27/10/2014 08:08, Karl Dubost wrote:
 2. It seems to be a change of policy in terms of sharing data the user type 
 in the URL bar. Is there a possibility to put that off and not having 
 whatever you type up there to be sent somewhere the user does not expect?

Yes, suggestions can be disabled from the settings app: go into Settings
 Search and turn off Search Suggestions. This should stop search
suggestions from the URL/RocketBar while giving you the possibility of
manually starting a search.

 Gabriele




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Firefox 2.2.0 and everything.me

[Gabriele Svelto]

On 27/10/2014 09:55, Karl Dubost wrote:
 1. Being usability (performance on flickering) Bug number?

We've got bug 1027381 [1] though we might want to introduce a mechanism
to also throttle the requests we send (and drawing the spinner which
takes a huge amount of CPU time slowing down the user). I had
implemented a simple solution for the old browser search bar in bug
985369 [2], it could be used as an inspiration for improving the
Rocketbar search though as I mentioned above one would need to do more
there than simple throttling because the spinner alone is a huge drag on
performance.

 Gabriele

[1] [SCR][Rocketbar] Can we do a reduction of flashing via a comparison
with this found set versus last found set?
https://bugzilla.mozilla.org/show_bug.cgi?id=1027381
[2] Writing into the URL bar consumes 100% CPU time
https://bugzilla.mozilla.org/show_bug.cgi?id=985369



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: The worst piece of Mozilla code

[Gabriele Svelto]

On 17/10/2014 00:32, Nicholas Nethercote wrote:
 Thanks for the replies so far! I deliberately left this question vague
 to see what kind of responses people would give. But mostly I'm
 interested in code whose awfulness impacts users in a serious way.
 Ones where refactoring/rewriting efforts would be valuable. That
 excludes code that no longer exists :)

A lot of early FxOS code both in gecko and gaia was hacked together in
no time during caffeine-fueled work weeks by jet-lagged developers. Some
of it was cleaned up but some is still with us, mostly in gaia I'd say,
where it often does have an impact on end users. A few things that come
to mind:

- In some applications transitions between a panel or a similar UI state
are not atomic so if you tap fast enough funny things happen.

- We often store application data in the DOM again making for funny bugs
when we update the DOM asynchronously while the user is interacting with
the application.

- I'm not really sure all of our IndexedDB uses are doing transactions
properly. We had (and possibly still have) multiple places where we were
doing things that should have been atomic in different transactions
(because of async_storage.js for example) and we assumed that a
transaction succeeded if the last request succeeded (but not the
transaction itself).

- Dual-SIM support is still partially done. Some things work, some
things kinda work, some things don't work at all (e.g. use SIMs from two
different countries and only the first one's country prefix will be
recognized correctly when matching numbers).

- Last but not least issues in horribly buggy Android binary blobs were
worked around in gecko, some of those workarounds ended up being
duplicated in different pieces of code as refactorings came and went, e.g.:

http://dxr.mozilla.org/mozilla-central/source/gfx/layers/RotatedBuffer.cpp#403
http://dxr.mozilla.org/mozilla-central/source/gfx/layers/client/TextureClient.cpp#226

I know we blacklist drivers on desktops too; what makes this worse is
that we can't ask users to update their drivers on their phones and
these horrors keep creeping into our codebase and we don't know when
we'll be able to get rid of them (or if we'll remember all the places we
put them in).

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: GGC (generational garbage collection) has landed for B2G

[Gabriele Svelto]

Having previously worked on a couple of GCs I must say this is amazing
work! I don't know many other large, complex code bases such as Gecko
that were moved successfully from a conservative mark-and-sweep
collector to an accurate generational one. Bravo to everyone involved!

 Gabriele

On 16/09/2014 21:45, Steve Fink wrote:
 GGC was enabled yesterday (2014-Sep-15) on b2g-inbound with bug 1020751.
 It has not yet been merged into mozilla-central, but I expect it will be
 soon. GGC is already on desktop Firefox, and in fact just shipped with
 Firefox 32. Keep an eye out for regressions on FxOS. Or improvements,
 for that matter -- we'd be interested in any real-world workloads that
 see significant changes.




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Studying Lossy Image Compression Efficiency, July 2014

[Gabriele Svelto]

On 19/07/2014 22:40, Ralph Giles wrote:
 Probably not for Firefox OS, if you mean mozjpeg. Not necessarily
 because it uses hardware, but because mozjpeg is about spending more cpu
 power to compress images. It's more something you'd use server-side or
 in creating apps. The phone uses libjpeg-turbo for image decoding, which
 is fast, just not as good an compression.

It might be useful in Firefox OS development: we routinely re-compress
PNG assets in FxOS but we never tried re-compressing our JPEG assets
(which are mostly wallpapers IIRC).

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: fine-grained filtering of bugmail

[Gabriele Svelto]

On 14/07/2014 15:33, Ehsan Akhgari wrote:
 2. Can we get a field designating the creation of bugs, so that I can
 set things up so that I get bugmails for new bugs no matter what?

+1 for that. One thing I always do is check for new bugs in the
components I'm watching and then CC me on those of interest. Only
receiving successive messages for bugs I've explicitly CC'd me on would
greatly reduce the amount of bugmail I have to go through every day
while still allowing me to watch entire components for activity.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: OMTC on Windows

[Gabriele Svelto]

On 30/05/2014 14:19, Andreas Gal wrote:
 Now for Intel hardware being slow there could be a couple reasons, and APZ 
 might fix them actually. If I remember correctly Atom GPUs are PowerVR based, 
 which is a tile based rendering architecture. It splits the frame buffer in 
 small tiles and renders those. To do this efficiently it defers rendering for 
 as long as possible. Other GPUs start rendering as soon as possible, whereas 
 PowerVR waits until the entire frame is ready and then renders it then. We do 
 a couple operations while rendering that might force a pipeline flush, which 
 likely forces PowerVR to render right away, which is very bad for PowerVR’s 
 particular render model. If you can point us to some specific hardware we 
 really suck on we can definitely look into this.

The test in https://bugzilla.mozilla.org/show_bug.cgi?id=894128#c30 is
using an HD4000 iGPU which is an internal Intel design and not
PowerVR-based. Has anybody tried using a Intel's Graphics Performance
Analyzer [1] tools to see if we're hitting a slow path in the driver or
some other suboptimal scenario?

 Gabriele

[1] https://software.intel.com/en-us/vcsource/tools/intel-gpa



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Hal peers wanted

[Gabriele Svelto]

This morning I've been asked to do a review for bug 980027 [1] mostly
because I've been working actively on that code and I did some small
reviews in the past. Since the change was relatively large this time
I've asked the patch author to ask a review from a Hal peer in addition
to me which lead us to discover that none of them is actively working on
the project anymore (sadly):

https://wiki.mozilla.org/Modules/Core#HAL

So it seems that we needs new peers and possibly a new owner too. How do
we proceed in situations like this?

 Gabriele

[1] honor elevated thread priorities for libmedia and compositor threads
in gonk
https://bugzilla.mozilla.org/show_bug.cgi?id=980027



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Linux testing on single-core VMs nowadays

[Gabriele Svelto]

On 07/04/2014 23:13, Dave Hylands wrote:
 Personally, I think that the more ways we can test for threading issues the 
 better.
 It seems to me that we should do some amount of testing on single core and 
 multi-core.
 
 Then I suppose the question becomes how many cores? 2? 4? 8?
 
 Maybe we can cycle through some different number of cores so that we get 
 coverage without duplicating everything?

One configuration that is particularly good at catching threading errors
(especially narrow races) is constraining the software to run on two
hardware threads on the same SMT-enabled core. This effectively forces
the threads to share the L1 D$ which in turn can reveal some otherwise
very-hard-to-find data synchronization issues.

I don't know if we have that level of control on our testing hardware
but if we do then that's a scenario we might want to include.

 Gabriele




signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Wasted bytes due to padding (bug 492185)

[Gabriele Svelto]

On 20/02/2014 18:57, arpad.bor...@googlemail.com wrote:
 And yes, allocation counts would be awesome. Any idea how to get to those? Or 
 even better: number of average live objects, since short lived allocations do 
 not matter that much.

Just thinking out loud: some memory reporters should already cover some
of those objects, could we make them return object counts instead of
sizes? Then we might feed that list of wasted space to about:memory's
logic in order to display the amount of wasted space per class of objects.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Deciding whether to change the number of unified sources

[Gabriele Svelto]

On 03/12/2013 22:30, Jed Davis wrote:

At the risk of stating the obvious, localizing the size change should
be a simple matter of `readelf -WS`.  If we're seeing actual change
in .text size by way of per-directory sort-of-LTO, then that would be
interesting (and could be localized further with the symbol table).


It would be interesting to see if this had an impact on our performance 
profile too.


 Gabriele

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Recent build time improvements due to unified sources

[Gabriele Svelto]

On 28/11/2013 10:57, Neil wrote:

I often build in a VM. I allocate 2 CPUs and 2GB of RAM to it, which
seems to be enough even to link xul.dll with debug symbols, although it
takes a few minutes. (Linking it without symbols takes just seconds.)
Given the amount of memory needed to link I haven't considered the case
of running out of RAM during compiling, although I did manage to run out
of virtual disk space for temporary files when I accidentally built with
-j instead of -j3.


I assume this was a Firefox for Windows build, correct? This week I 
found an issue that causes FxOS's gecko to be built with -j16 
irrespective of the number of jobs you specify on the command-line or in 
the .config file. That's likely the cause for some of these issues; 
fortunately it's FxOS-specific. I'm fixing this problem as part of bug 
888698.


 Gabriele

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Recent build time improvements due to unified sources

[Gabriele Svelto]

On 28/11/2013 06:06, Gregory Szorc wrote:

On de5aa094b55f, we're now down to:

Wall   7:37   (457)
User  45:38  (2738)
Sys3:54   (234)
Total 49:32  (2972)

That's with WebRTC and ICU enabled.


Looking at my own stats while building I was wondering if anybody has 
looked at peak memory consumption with unified sources. Since now we're 
compiling up to 16 files per compiler instance I would imagine that peak 
memory consumption has increased, possibly significantly. This could be 
offset by the fact that most of the files will be sharing headers but I 
still wonder how much of an impact it has.


 Gabriele
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform