Re: MinGW Target on TaskCluster

[Tom Ritter]

On Tue, Feb 6, 2018 at 5:36 PM, Aaron Klotz  wrote:
> I'd like to follow up on this old thread to discuss what we can do about
> improving the mingw developer experience for people doing Windows-centric
> stuff.


To follow up on this, in
https://bugzilla.mozilla.org/show_bug.cgi?id=1444167 I landed a lint
job that will look for #includes that match one of MinGW's headers but
isn't all-lowercase.

If you're adding third-party code that (for whatever reason) doesn't
break the MinGW build but still has uppercase includes and the lint
test is failing, you can add file or directory exceptions here:
https://hg.mozilla.org/mozilla-central/file/tip/tools/lint/mingw-capitalization.yml

-tom
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: MinGW Target on TaskCluster

[Steve Fink]

On 2/6/18 6:27 PM, Tom Ritter wrote:

But when it comes to the header case mismatch issue, we could automate this.
Would it be possible for us to write some code to automatically detect these 
case
discrepancies and automatically land corrections? Where would be the right place
for something like that to live?

I'd be happy to build something that does this. I don't know what's
possible or where it should live though. It's probably easy for me to
add it to the lint job, wherever that is.  I might be able to
integrate it into the new MozReview Static Analysis thing. I don't
know if it's possible, or how, to automatically land corrections* but
maybe?



This came up in a less pressing situation recently within the js/src 
tree, where we were thinking that the nicest way for this to work would 
be to have some limited pre-push lint checks. In our case, it could be 
easily restricted to a particular portion of the tree. I don't know if 
it would be fast enough to not add friction if it had to check any 
source file anywhere in the tree, but it seems like the nicest developer 
experience -- you wouldn't be bothered by it until you tried to land, 
and then an opt-out pre-push hook would tell you that you're going to 
break something. There has been talk of hooking up mozlint in this way, 
so that might be the place to look if you're thinking of setting it up. 
I'm not sure what the story there is for a git workflow, but at least 
for mercurial it would seem to make sense to have the hooks installed 
via mach bootstrap / mach mercurial-setup.


The new phabricator workflow may have a place for this sort of thing as 
well?



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: MinGW Target on TaskCluster

[Ryan VanderMeulen]

On 2/6/2018 6:36 PM, Aaron Klotz wrote:

Furthermore, can we clarify whether mingw is currently a tier 1 or tier 2
platform? Earlier in this thread, mingw was denoted as tier 2 (and it still
shows up as tier 2 on treeherder), yet sheriffs are backing out patches
when mingw bustage occurs. Tor is important, so of course we want to make
our best effort to ensure that mingw isn't broken, but this "tier 2 but
really tier 1" state is not helpful.


Tier 2 job bustage shouldn't lead to an automatic backout under normal 
circumstances. Normal practice would be to file a bug for the bustage 
with a NI directed to the author of the patch that caused it. If you've 
been backed out without any such notice, feel free to raise your 
concerns directly with either myself or Sebastian.


Note that the sheriffing team does reserve the right to backout a patch 
for Tier 2 bustage if it's not addressed in a reasonable time frame 
(Tier 2 != unowned), but that would typically only happen in the 
situation where a bug Was filed and saw no further activity afterwards.


-Ryan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: MinGW Target on TaskCluster

[Tom Ritter]

Yes, let's fix it.

I'm very grateful to you and everyone else who has kept the MinGW
build running; and I'd like to keep you... well as happy as you can be
while supporting an esoteric build target that's inconvenient for you.

With regards to backouts, when sheriffs have asked, my response has
been that if something is directly attributable to a patch and it
doesn't look like it's a MinGW problem (like the recent missing
dwrite_3.h issue) - I'd appreciate a backout. This is mostly
selfishness, the times there was no backout and MinGW starts
perma-failing it's been up to me to diagnose the issue and fix it;
only once have MinGW patches been needed. If that needs to change,
alright.


As far as what we can do to make it easier. I think/hope MinGW isn't a
strain on TC; there are no tests associated, it even skips the normal
check-tests that run for other build jobs; and it runs on a small
Linux box so it should be the cheapest option for us. It could build
faster though; sccache doesn't work for it and I haven't been able to
figure out why (Bug 1411212).

> But when it comes to the header case mismatch issue, we could automate this.
> Would it be possible for us to write some code to automatically detect these 
> case
> discrepancies and automatically land corrections? Where would be the right 
> place
> for something like that to live?

I'd be happy to build something that does this. I don't know what's
possible or where it should live though. It's probably easy for me to
add it to the lint job, wherever that is.  I might be able to
integrate it into the new MozReview Static Analysis thing. I don't
know if it's possible, or how, to automatically land corrections* but
maybe?



Also, not that this is a solution for today but as I mentioned in the
first email - my hope is that some of this becomes easier with a move
to clang. It won't address everything, but I hope at least some of the
MinGW pain is lessened by a mingw-clang option. I could see us
patching our mingw-clang build to work with case insensitive includes
for example; and the __try/__except issues we have would go away also.
There's a few paths forward on this front. One using a mingw-clang,
one using clang-cl on Linux. The work on the first one is happening in
Bug 1390583.

-tom

* and I am nervous about giving some new system the ability to
automatically land code

On Tue, Feb 6, 2018 at 5:36 PM, Aaron Klotz  wrote:
> I'd like to follow up on this old thread to discuss what we can do about
> improving the mingw developer experience for people doing Windows-centric
> stuff.
>
> I've had several patches backed out over the past month because of mingw
> header files differing in case from the headers distributed by the official
> Windows SDK. Frankly, it's a waste of developer time to be dealing with
> this. This is particularly frustrating in the case of autoland, where we
> can't just push our own follow-up patches.
>
> Those of us who are trying to do Windows-specific stuff are effectively
> stuck simultaneously coding to two similar, yet still different, SDKs, being
> built across two distinct operating systems that differ in case sensitivity.
>
> In some cases, discrepancies between the SDKs are inevitable: Perhaps mingw
> hasn't kept up with the official Windows SDK and some headers/APIs are
> missing. We just have to take those as they come.
>
> But when it comes to the header case mismatch issue, we could automate this.
> Would it be possible for us to write some code to automatically detect these
> case discrepancies and automatically land corrections? Where would be the
> right place for something like that to live?
>
> Those of us locally building on Windows already have enough annoyances with
> the build as-is, so I don't consider "you should be building locally on both
> Windows/MSVC and Linux/mingw simultaneously" to be a reasonable option. Nor
> do I consider always pushing mingw jobs to try to be ideal: it delays the
> landing of patches and potentially wastes build machine time, compared to an
> automated lint.
>
> Furthermore, can we clarify whether mingw is currently a tier 1 or tier 2
> platform? Earlier in this thread, mingw was denoted as tier 2 (and it still
> shows up as tier 2 on treeherder), yet sheriffs are backing out patches when
> mingw bustage occurs. Tor is important, so of course we want to make our
> best effort to ensure that mingw isn't broken, but this "tier 2 but really
> tier 1" state is not helpful.
>
> I apologize for being curt, but this is becoming rather frustrating.
>
> Aaron
>
>
> On Mon, Oct 9, 2017 at 10:14 AM, Tom Ritter  wrote:
>>
>> On Mon, Oct 9, 2017 at 10:31 AM, Philipp Wagner 
>> wrote:
>>
>> > Am 09.10.2017 um 07:31 schrieb Tom Ritter:
>> > > As part of our work with Tor, we’ve been working on getting a
>> > > MinGW-based
>> > > build of Windows into TaskCluster.
>> >
>> > A maybe too obvious question from the side lines: Why is 

Re: MinGW Target on TaskCluster

[Aaron Klotz]

I'd like to follow up on this old thread to discuss what we can do about
improving the mingw developer experience for people doing Windows-centric
stuff.

I've had several patches backed out over the past month because of mingw
header files differing in case from the headers distributed by the official
Windows SDK. Frankly, it's a waste of developer time to be dealing with
this. This is particularly frustrating in the case of autoland, where we
can't just push our own follow-up patches.

Those of us who are trying to do Windows-specific stuff are effectively
stuck simultaneously coding to two similar, yet still different, SDKs,
being built across two distinct operating systems that differ in case
sensitivity.

In some cases, discrepancies between the SDKs are inevitable: Perhaps mingw
hasn't kept up with the official Windows SDK and some headers/APIs are
missing. We just have to take those as they come.

But when it comes to the header case mismatch issue, we could automate
this. Would it be possible for us to write some code to automatically
detect these case discrepancies and automatically land corrections? Where
would be the right place for something like that to live?

Those of us locally building on Windows already have enough annoyances with
the build as-is, so I don't consider "you should be building locally on
both Windows/MSVC and Linux/mingw simultaneously" to be a reasonable
option. Nor do I consider always pushing mingw jobs to try to be ideal: it
delays the landing of patches and potentially wastes build machine time,
compared to an automated lint.

Furthermore, can we clarify whether mingw is currently a tier 1 or tier 2
platform? Earlier in this thread, mingw was denoted as tier 2 (and it still
shows up as tier 2 on treeherder), yet sheriffs are backing out patches
when mingw bustage occurs. Tor is important, so of course we want to make
our best effort to ensure that mingw isn't broken, but this "tier 2 but
really tier 1" state is not helpful.

I apologize for being curt, but this is becoming rather frustrating.

Aaron


On Mon, Oct 9, 2017 at 10:14 AM, Tom Ritter  wrote:

> On Mon, Oct 9, 2017 at 10:31 AM, Philipp Wagner 
> wrote:
>
> > Am 09.10.2017 um 07:31 schrieb Tom Ritter:
> > > As part of our work with Tor, we’ve been working on getting a
> MinGW-based
> > > build of Windows into TaskCluster.
> >
> > A maybe too obvious question from the side lines: Why is the Tor browser
> > cross-compiled and not using MSVC?
> >
> >
> Building on Linux allows Tor Browser (including its entire toolchain and
> dependencies) to be built deterministically and thus reproducibly using an
> entirely open source toolchain. (There are a few small exceptions but
> they're quite small.)
>
> Reproducible builds significantly reduce the trust inherent in an
> organization's build infrastructure and when recreated independently ensure
> that nothing unexpected was inserted into the final executable. Having the
> entire toolchain open source ensures that anyone who wants to inspect the
> code or reason about its behavior can do so. (And as I've learned in the
> past year Tor actually has a good amount of anonymous contributors reading
> the source code of its toolchain and reporting things.)
>
> You can read more about the philosophy behind this movement at
> https://blog.torproject.org/deterministic-builds-part-one-
> cyberwar-and-global-compromise
> https://reproducible-builds.org/
> CCleaner is a good example of attackers backdooring compiled software.
>
> The next step, past reproducible builds, is Binary Transparency, which
> ensures that before an update is applied, it is publicly known, so
> attackers cannot surreptitiously subvert the update mechanism.  Tor is
> exploring some avenues there. FLAME is a good example of attacking the
> update mechanism.
>
> I would describe Mozilla as 'curious' about reproducible builds (see
> https://bugzilla.mozilla.org/show_bug.cgi?id=885777); but we are actively
> working on Binary Transparency (see
> https://bugzilla.mozilla.org/show_bug.cgi?id=1341395).
>
> -tom
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: MinGW Target on TaskCluster

[Tom Ritter]

On Mon, Oct 9, 2017 at 10:31 AM, Philipp Wagner 
wrote:

> Am 09.10.2017 um 07:31 schrieb Tom Ritter:
> > As part of our work with Tor, we’ve been working on getting a MinGW-based
> > build of Windows into TaskCluster.
>
> A maybe too obvious question from the side lines: Why is the Tor browser
> cross-compiled and not using MSVC?
>
>
Building on Linux allows Tor Browser (including its entire toolchain and
dependencies) to be built deterministically and thus reproducibly using an
entirely open source toolchain. (There are a few small exceptions but
they're quite small.)

Reproducible builds significantly reduce the trust inherent in an
organization's build infrastructure and when recreated independently ensure
that nothing unexpected was inserted into the final executable. Having the
entire toolchain open source ensures that anyone who wants to inspect the
code or reason about its behavior can do so. (And as I've learned in the
past year Tor actually has a good amount of anonymous contributors reading
the source code of its toolchain and reporting things.)

You can read more about the philosophy behind this movement at
https://blog.torproject.org/deterministic-builds-part-one-cyberwar-and-global-compromise
https://reproducible-builds.org/
CCleaner is a good example of attackers backdooring compiled software.

The next step, past reproducible builds, is Binary Transparency, which
ensures that before an update is applied, it is publicly known, so
attackers cannot surreptitiously subvert the update mechanism.  Tor is
exploring some avenues there. FLAME is a good example of attacking the
update mechanism.

I would describe Mozilla as 'curious' about reproducible builds (see
https://bugzilla.mozilla.org/show_bug.cgi?id=885777); but we are actively
working on Binary Transparency (see
https://bugzilla.mozilla.org/show_bug.cgi?id=1341395).

-tom
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: MinGW Target on TaskCluster

[Philipp Wagner]

Am 09.10.2017 um 07:31 schrieb Tom Ritter:
> As part of our work with Tor, we’ve been working on getting a MinGW-based
> build of Windows into TaskCluster. 

A maybe too obvious question from the side lines: Why is the Tor browser
cross-compiled and not using MSVC?

Philipp
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: MinGW Target on TaskCluster

[Ted Mielczarek]

On Mon, Oct 9, 2017, at 01:31 AM, Tom Ritter wrote:
> As part of our work with Tor, we’ve been working on getting a MinGW-based
> build of Windows into TaskCluster. Tor is currently using ESR releases,
> and
> every ESR they have to go through a large amount of work to get the build
> working under MinGW again; by continually building (and testing) that
> build
> we’ll be able to cut weeks to months of effort for them each ESR release.
> (Not breaking the MinGW build is also a necessity if they were ever to
> move
> off ESRs.)

Great work, Tom! I know this was a long slog, but keeping this build
working is going to lift a massive burden from the Tor Browser team.
Thanks for taking on this project and driving it to completion!

-Ted
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: MinGW Target on TaskCluster

[Georg Koppen]

Hi!

Tom Ritter:
> As part of our work with Tor, we’ve been working on getting a MinGW-based
> build of Windows into TaskCluster. Tor is currently using ESR releases, and
> every ESR they have to go through a large amount of work to get the build
> working under MinGW again; by continually building (and testing) that build
> we’ll be able to cut weeks to months of effort for them each ESR release.
> (Not breaking the MinGW build is also a necessity if they were ever to move
> off ESRs.)
> 
> Unlike our normal Windows builds with msvc or clang-cl, this is a
> cross-compile on Linux for Windows, as well as the first time in a long
> time we’ve built for Windows with gcc. Building with gcc for Windows has
> exposed a number of new warnings
>  to clean up, as well
> as some C++ spec incompatibilities that MSVC (and clang-cl) accept. All of
> the breaks have been resolved; and a lot of the warnings and similar are
> either resolved or in-progress.
> 
> Effective this weekend, MinGW is on Treeherder, with the single build
> target win32-mingw32 (debug). Its toolchain builds are under TMW (with a
> few under TL because they are generic Linux packages.)
> 
> The one-click loaner system works for MinGW, and we’re going to allow this
> to bake at Tier 2 for a while. Your help in keeping the build green is
> greatly appreciated, and if you hit a MinGW bug you can’t decipher, I’ll be
> happy to help, just send me an email or a needinfo. Some of the most common
> things that cause errors for MinGW:
> 
> 
>  - #include  instead of #include  and similar
> 
>  - _uuidof() instead of IID_* ref
> 
> 
>  - Casting nullptr to bool or int
> 
>  - Using the (in)correct assembly code
> 
>  - Breaking --disable-webrtc, --disable-sandbox, or --disable-accessibility
> 
>  - MinGW lagging behind on defining new constants Microsoft has defined
> 
> There are a few things left outstanding
> 
> for the MinGW build, including webrtc
> , stylo
>  (which may be the
> most difficult in this list), the sandbox
> ,
> --enable-accessibility
> , cleaning
> up warnings , and 
> running
> tests . In the
> future, there is hope (by me at least) to throw all of this work away.
> Compiling on Linux for Windows with clang (or clang-cl) would probably be a
> significant improvement for Tor’s builds, and would enable them to take
> advantage of CFI and SEH, as well as simplifying the number of platforms
> Mozilla supports. Chrome has been pushing towards this
> , so we’re
> keeping an eye on their progress.
> 
> Finally, I want to thank everyone who brought this long process to this
> point: everyone who wrote patches, explained things (sometimes over and
> over again), and reviewed. That’s including but not limited to: boklm,
> dmajor, froydnj, glandium, georg, jacek, jrmuizel, and ted; plus the dozen
> or so more people who have reviewed my patches all over the codebase.

That's great news, Tom! Thanks for working on this project and all its
numerous child bugs, greatly appreciated. I know from experience how
time-consuming and tedious this is.

To stress a point you made in your email above: your efforts and those
of other Mozilla engineers that helped with this project does indeed
save us weeks of work when preparing major Tor Browser releases based on
a new ESR. We can now use this valuable time to fix more privacy issues
faster which benefits both Tor Browser and Firefox users alike because
we upstream those patches as we think not only Tor users should benefit
from them.

As to your future plans, yes, we are happy if we can help with the
effort to move away from using GCC for Windows cross-builds and move to
clang instead. That's not only due to the SEH and CFI support you
mentioned. Rather, it would allow us to get rid of annoying GCC bugs,
like https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64384, as well and
would significantly simpify our cross-compile setups we need to maintain.

Thanks,
Georg



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

MinGW Target on TaskCluster

[Tom Ritter]

As part of our work with Tor, we’ve been working on getting a MinGW-based
build of Windows into TaskCluster. Tor is currently using ESR releases, and
every ESR they have to go through a large amount of work to get the build
working under MinGW again; by continually building (and testing) that build
we’ll be able to cut weeks to months of effort for them each ESR release.
(Not breaking the MinGW build is also a necessity if they were ever to move
off ESRs.)

Unlike our normal Windows builds with msvc or clang-cl, this is a
cross-compile on Linux for Windows, as well as the first time in a long
time we’ve built for Windows with gcc. Building with gcc for Windows has
exposed a number of new warnings
 to clean up, as well
as some C++ spec incompatibilities that MSVC (and clang-cl) accept. All of
the breaks have been resolved; and a lot of the warnings and similar are
either resolved or in-progress.

Effective this weekend, MinGW is on Treeherder, with the single build
target win32-mingw32 (debug). Its toolchain builds are under TMW (with a
few under TL because they are generic Linux packages.)

The one-click loaner system works for MinGW, and we’re going to allow this
to bake at Tier 2 for a while. Your help in keeping the build green is
greatly appreciated, and if you hit a MinGW bug you can’t decipher, I’ll be
happy to help, just send me an email or a needinfo. Some of the most common
things that cause errors for MinGW:


 - #include  instead of #include  and similar

 - _uuidof() instead of IID_* ref


 - Casting nullptr to bool or int

 - Using the (in)correct assembly code

 - Breaking --disable-webrtc, --disable-sandbox, or --disable-accessibility

 - MinGW lagging behind on defining new constants Microsoft has defined

There are a few things left outstanding

for the MinGW build, including webrtc
, stylo
 (which may be the
most difficult in this list), the sandbox
,
--enable-accessibility
, cleaning
up warnings , and running
tests . In the
future, there is hope (by me at least) to throw all of this work away.
Compiling on Linux for Windows with clang (or clang-cl) would probably be a
significant improvement for Tor’s builds, and would enable them to take
advantage of CFI and SEH, as well as simplifying the number of platforms
Mozilla supports. Chrome has been pushing towards this
, so we’re
keeping an eye on their progress.

Finally, I want to thank everyone who brought this long process to this
point: everyone who wrote patches, explained things (sometimes over and
over again), and reviewed. That’s including but not limited to: boklm,
dmajor, froydnj, glandium, georg, jacek, jrmuizel, and ted; plus the dozen
or so more people who have reviewed my patches all over the codebase.


-tom
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform