Re: Localized Repacks now visible on (most) pushes
Congratulations Callek! I know this was a tremendous amount of work on your side to implement and coordinate. Thank you for all your efforts to drive this forward, it has unblocked a lot of future l10n work. Kim On Wed, May 30, 2018 at 1:57 PM, Gregory Szorc wrote: > On Wed, May 30, 2018 at 10:08 AM, Justin Wood wrote: > > > Hello Everyone, > > > > tl;dr You should now see "L10n" jobs on treeherder with many pushes, > these > > are tier 1 and if they break they would also be breaking Nightly so your > > patch would need to be backed out. > > > > As many of you know, especially the old guard [1] here, Localized Repacks > > have frequently been known to fail in weird and interesting ways on > Nightly > > and Beta builds. > > > > Throughout the movement to taskcluster we have been reducing the > > differences in automation to make what we ship to release users happen > with > > the same process as what we ship to nightly users. We have recently > > achieved that parity now that we have finished our migration to > taskcluster > > [2] > > > > One straggler was on our implementation of L10n builds on try [3][4] > which > > had begun to frequently fail when users add/remove any localized file > (.dtd > > or .ftl). And similarly we have always lacked the ability to easily vet a > > change to central/inbound/autoland as "will this break l10n". > > > > With the work I've now done we have aligned this "try" l10n job with what > > we perform in the Nightly and Release Promotion process, as well as > allowed > > ourselves the ability to run these on every push. > > > > Implementation details: > > * For now these still run only when a subset of files change [5] but this > > list can be expanded easily, or we can rip it out and instead *always* > run > > these jobs. > > * These jobs are performed using live L10n repositories, but just a small > > set of our total localization, specifically: en-CA, he, it, ja, ja-JP-mac > > [6] > > * As part of doing this work, we needed to specify the STUB Installer > > differently, if we need it on any new channels/builds we need to specify > it > > in the build taskcluster kind, like [7]. We have a check in configure to > > error if its not set correctly [8] > > > > If you have any questions, feel free to reach out to me/releng. > > ~Justin Wood (Callek) > > > > Thank you, Justin and everyone else who worked on this! l10n packaging has > historically suffered from a lack of visibility in CI and lack of > understanding outside its small circle of maintainers. Moving the l10n > automation to Taskcluster and giving it visibility in Treeherder as part of > regular jobs that normal people see will go a very long way to increasing > understanding of l10n packaging. It also paves the road for overhauling the > technical underpinnings of l10n packaging. For those not aware, l10n > packaging has historically been a major burden for build system maintainers > because of the convoluted ways it interacts with the build system. Let's > just say that we have actively avoided touching code related to l10n out of > fear that it will break a convoluted system. Now that the l10n CI can more > easily be toggled and tested, it is substantially easier to iterate on and > we now have the confidence that our changes won't break things. This is a > game changer and will directly enable us to perform some long-overdue > refactoring of l10n code. Thank you! > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Update on rustc/clang goodness
On Thursday, 31 May 2018 02:44:51 UTC+12, Tom Ritter wrote: > Oh. Are we doing this rustc inlining development on a particular old > version of clang? I'm not even close to getting CFI ready but I'm > basically working off llvm trunk as I'm finding and filing llvm bugs > and working with llvm devs to get them fixed We don't have experience with cross language LTO. It is difficult to predict but relatively easy to try. Here's what we do know: In order for the linker to do LTO, it needs to be able to understand the llvm-ir from both clang and rustc. For the most part this means having an lld verison more recent than the compilers. In theory the newer linker can read old versions of llvm-ir. Unfortunately llvm-ir changes over time and while some effort goes into making it backward compatible, it seems reasonable to expect a large gap between clang and rustc versions may fall short of excellence. We may be able to get away with heterogenous clang/rustc/lld versions, but it may be more trouble than it is worth. Alternatively we may need to keep rustc/clang/lld in lock step with all that it entails. It is possible that in the end we decide the whole thing isn't worth the trouble. Time will tell. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Localized Repacks now visible on (most) pushes
On Wed, May 30, 2018 at 10:08 AM, Justin Wood wrote: > Hello Everyone, > > tl;dr You should now see "L10n" jobs on treeherder with many pushes, these > are tier 1 and if they break they would also be breaking Nightly so your > patch would need to be backed out. > > As many of you know, especially the old guard [1] here, Localized Repacks > have frequently been known to fail in weird and interesting ways on Nightly > and Beta builds. > > Throughout the movement to taskcluster we have been reducing the > differences in automation to make what we ship to release users happen with > the same process as what we ship to nightly users. We have recently > achieved that parity now that we have finished our migration to taskcluster > [2] > > One straggler was on our implementation of L10n builds on try [3][4] which > had begun to frequently fail when users add/remove any localized file (.dtd > or .ftl). And similarly we have always lacked the ability to easily vet a > change to central/inbound/autoland as "will this break l10n". > > With the work I've now done we have aligned this "try" l10n job with what > we perform in the Nightly and Release Promotion process, as well as allowed > ourselves the ability to run these on every push. > > Implementation details: > * For now these still run only when a subset of files change [5] but this > list can be expanded easily, or we can rip it out and instead *always* run > these jobs. > * These jobs are performed using live L10n repositories, but just a small > set of our total localization, specifically: en-CA, he, it, ja, ja-JP-mac > [6] > * As part of doing this work, we needed to specify the STUB Installer > differently, if we need it on any new channels/builds we need to specify it > in the build taskcluster kind, like [7]. We have a check in configure to > error if its not set correctly [8] > > If you have any questions, feel free to reach out to me/releng. > ~Justin Wood (Callek) > Thank you, Justin and everyone else who worked on this! l10n packaging has historically suffered from a lack of visibility in CI and lack of understanding outside its small circle of maintainers. Moving the l10n automation to Taskcluster and giving it visibility in Treeherder as part of regular jobs that normal people see will go a very long way to increasing understanding of l10n packaging. It also paves the road for overhauling the technical underpinnings of l10n packaging. For those not aware, l10n packaging has historically been a major burden for build system maintainers because of the convoluted ways it interacts with the build system. Let's just say that we have actively avoided touching code related to l10n out of fear that it will break a convoluted system. Now that the l10n CI can more easily be toggled and tested, it is substantially easier to iterate on and we now have the confidence that our changes won't break things. This is a game changer and will directly enable us to perform some long-overdue refactoring of l10n code. Thank you! ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Localized Repacks now visible on (most) pushes
Congratulations Justin! Excited to see this coming all together. With this change, we can now both improve our software quality and culture of paying attention to red L10n in treeherder :) Thank you! zb. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Localized Repacks now visible on (most) pushes
Hello Everyone, tl;dr You should now see "L10n" jobs on treeherder with many pushes, these are tier 1 and if they break they would also be breaking Nightly so your patch would need to be backed out. As many of you know, especially the old guard [1] here, Localized Repacks have frequently been known to fail in weird and interesting ways on Nightly and Beta builds. Throughout the movement to taskcluster we have been reducing the differences in automation to make what we ship to release users happen with the same process as what we ship to nightly users. We have recently achieved that parity now that we have finished our migration to taskcluster [2] One straggler was on our implementation of L10n builds on try [3][4] which had begun to frequently fail when users add/remove any localized file (.dtd or .ftl). And similarly we have always lacked the ability to easily vet a change to central/inbound/autoland as "will this break l10n". With the work I've now done we have aligned this "try" l10n job with what we perform in the Nightly and Release Promotion process, as well as allowed ourselves the ability to run these on every push. Implementation details: * For now these still run only when a subset of files change [5] but this list can be expanded easily, or we can rip it out and instead *always* run these jobs. * These jobs are performed using live L10n repositories, but just a small set of our total localization, specifically: en-CA, he, it, ja, ja-JP-mac [6] * As part of doing this work, we needed to specify the STUB Installer differently, if we need it on any new channels/builds we need to specify it in the build taskcluster kind, like [7]. We have a check in configure to error if its not set correctly [8] If you have any questions, feel free to reach out to me/releng. ~Justin Wood (Callek) [1] - https://en.wikipedia.org/wiki/Old_Guard [2] - https://atlee.ca/blog/posts/migration-status-3.html [3] - https://bugzilla.mozilla.org/show_bug.cgi?id=848284 [4] - https://bugzilla.mozilla.org/show_bug.cgi?id=1458378 [5] - https://hg.mozilla.org/integration/mozilla-inbound/annotate/d7472bf663bd/taskcluster/ci/l10n/kind.yml#l177 [6] - https://hg.mozilla.org/integration/mozilla-inbound/rev/d2f587986a3b [7] - https://hg.mozilla.org/integration/mozilla-inbound/diff/e250856b4688/taskcluster/ci/build/windows.yml [8] - https://hg.mozilla.org/integration/mozilla-inbound/rev/a562a809e8dc ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: [Update] Revert RESOLVED:INACTIVE State
The process finished this morning. The vast majority of bugs were successfully reverted. As intended, some were skipped due to having been updated since the first script ran. There were also a handful of unexpected errors that we will be cleaning up manually. See https://bug1465190.bmoattachments.org/attachment.cgi?id=8981848 for the full results and https://bugzilla.mozilla.org/show_bug.cgi?id=1464312 for discussion. Mark On Tuesday, 29 May 2018 16:12:18 UTC-4, Dylan Hardison wrote: > The process has begun. We'll send out another announcement when it is done > (or if there are issues). > > > > On May 28, 2018, at 23:57, Erin Lancaster wrote: > > > > Of course, happy to get us back to a good place and very sorry for all of > > the inconvenience. We haven't had any further requests for exclusions for > > the revert: https://bugzilla.mozilla.org/show_bug.cgi?id=1464312. It would > > be helpful to get any other requests in asap. > > > > Dylan will be checking the ticket first thing Tuesday morning ET and will > > then get going with final review/testing. Many thanks to Dylan and to Mark > > for all their support. > > > > Erin > > > > On Sat, May 26, 2018 at 7:43 PM, Boris Zbarsky wrote: > > On 5/25/18 5:07 PM, Erin Lancaster wrote: > > Here’s where we are with timing/process for the revert: > > > > Erin, > > > > Thank you for all the work you've put in on this. I know it hasn't been > > easy dealing with this, and I really appreciate it. > > > > -Boris > > > > ___ > > firefox-dev mailing list > > firefox-...@mozilla.org > > https://mail.mozilla.org/listinfo/firefox-dev ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Everything you always wanted to know about I18n/L10n/L12y, but....
Hi! I'm very proud to introduce the latest addition to our family of "explain ourselves" efforts - shiny, new developer docs about locale management, internationalization and localization practices at Mozilla. https://firefox-source-docs.mozilla.org/intl/index.html We put a lot of effort to make it a "cup of coffee per topic" size wise, and hope you'll get to easily identify the topics relevant to your work. The first topic - Locale Management - is for that time when you need to make your code work with languages, select them, manage, negotiate etc. The other two - UI Internationalization and Localization - are important if you ever work with any front-end code at Mozilla. We focused on Firefox and Gecko apps, but almost everything applies universally across all of our portfolio. I'd love to hear your feedback on whether we are making progress in turning complex and hermetic field of knowledge more approachable and easier to work with If you any questions, please let me know! We'll do our best to keep those documents, and our extensive in-source-code docs in sync and up to date. Enjoy your coffee :) zb. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Update on rustc/clang goodness
On Wed, May 30, 2018 at 2:16 PM, Mike Hommey wrote: > I'm sure the day we'll have to choose between not > doing cross-language inlining or upgrading clang for e.g. security > features is relatively close. Oh. Are we doing this rustc inlining development on a particular old version of clang? I'm not even close to getting CFI ready but I'm basically working off llvm trunk as I'm finding and filing llvm bugs and working with llvm devs to get them fixed -tom ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Update on rustc/clang goodness
On Wed, May 30, 2018 at 02:40:01PM +0300, Henri Sivonen wrote: > On Wed, May 30, 2018 at 8:06 AM, Dave Townsend wrote: > > On Tue, May 29, 2018 at 10:03 PM Jeff Gilbert wrote: > >> I get that, but it reminds me of the reasons people give for "our > >> website works best in $browser". > > > > I was concerned by this too but found myself swayed by the arguments in > > https://blog.mozilla.org/nfroyd/2018/05/29/when-implementation-monoculture-right-thing/and > > in particular the first comment there. > > Indeed, the first comment there (by roc) gets to the point. > > Additionally, the reasons for not supporting multiple browsers tend to > be closer to the "didn't bother" kind whereas we're looking to get a > substantial benefit from clang that MSVC and GCC don't offer to us but > clang likely will: Cross-language inlining across code compiled with > clang and code compiled with rustc. > > To the extent Mozilla runs the compiler, it makes sense to go for the > Open Source choice that allows us to deliver better on "performance as > a feature". We still have at least one static analysis running on GCC, > so I wouldn't expect GCC-compatibility to be dropped even if the app > wouldn't be "best compiled with" GCC. The Linux distro case is > trickier than Mozilla's compiler choice. For CPUs that are tier-3 for > Mozilla, we already tolerate less great performance attributes in > order to enable availability, so distros keeping using GCC for tier-3 > probably isn't a problem. x86_64 could be a problem, though. If > Firefox's performance becomes significantly dependent on having > cross-language inlining, and I expect it will, having a substantial > portion of the user base run without it while thinking they have a > top-tier build could be bad. I hope we can get x86_64 Linux distros to > track our compiler configuration closely. That part might end up more difficult than one could expect. Cross-language inlining is going to require rustc and clang having a compatible llvm ir, and that's pretty much guaranteed to be a problem, even for Mozilla. I'm sure the day we'll have to choose between not doing cross-language inlining or upgrading clang for e.g. security features is relatively close. Anyways, Mozilla builds today are built with PGO, and many downstreams don't even do that... Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Update on rustc/clang goodness
We've also been running Windows code coverage builds with clang on CI for a while (since December), with almost all tests. - Marco. On Friday, May 11, 2018 at 2:35:57 AM UTC+2, Anthony Jones wrote: > You may already know that the Low-Level Tools team support important tools > and code infrastructure. Lately we’ve also been improving our rustc/clang > (LLVM) story and I’d like bring everyone up to date. > > There are a lot of important and interesting things going on: > > * Michael Woerister and Nathan Froyd recently (mid-March) enabled Rust > incremental compilation for Firefox developers > * Michael is experimenting with cross language inlining[1] (rustc/clang) > using LTO > * Preliminary results show compiling LLVM with clang and using LTO on > rustc improves stylo compilation time by around 15% [on-my-machine > measurements] > * LTO requires more work to support Firefox > * Nick Nethercote has started speeding up rustc[2] and has already had a > number of wins > * David Major has got Windows clang builds working with green tests > * LTO is still buggy (but works well enough to run benchmarks) and > PGO won’t run on Windows > * Win32 - clang with LTO w/o PGO is marginally faster than MSVC with > LTO/PGO > * Win64 - clang with LTO w/o PGO is ~5-10% slower than MSVC with > LTO/PGO > * Windows users can build locally with clang[3] > * Mike Hommey is tracking improvements and regressions in the Rust > compiler (performance regression[4], crash reporting[5] and more crash > reporting[6]) > * Tom Tromey is continuing to work on first class Rust support in GDB and > LLDB > > Note: this a summary of things rustc/clang stuff that is happening only. > > Ultimately, I’d like to see us standardise on clang on all platforms because > it makes Rust/C++ integration better as well as making things simpler for > developers and build system maintainers. We’ll get more done if we make our > own lives simpler. > > I have some specific requests for you: > > Let me know if you have specific Firefox related cases where Rust is > slowing you down (thanks Jeff [7]) > Cross language inlining is coming - avoid duplication between Rust and > C++ in the name of performance > Do developer builds with clang > > Anthony > > [1] https://github.com/rust-lang/rust/issues/49879 > [2] > https://blog.mozilla.org/nnethercote/2018/04/30/how-to-speed-up-the-rust-compiler-in-2018/ > [3] > https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Building_Firefox_on_Windows_with_clang-cl > [4] https://github.com/rust-lang/rust/issues/49873 > [5] https://bugzilla.mozilla.org/show_bug.cgi?id=1456150 > [6] https://bugzilla.mozilla.org/show_bug.cgi?id=1448868 > [7] https://github.com/rust-lang/rust/issues/50584 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Update on rustc/clang goodness
On Wed, May 30, 2018 at 8:06 AM, Dave Townsend wrote: > On Tue, May 29, 2018 at 10:03 PM Jeff Gilbert wrote: >> I get that, but it reminds me of the reasons people give for "our >> website works best in $browser". > > I was concerned by this too but found myself swayed by the arguments in > https://blog.mozilla.org/nfroyd/2018/05/29/when-implementation-monoculture-right-thing/and > in particular the first comment there. Indeed, the first comment there (by roc) gets to the point. Additionally, the reasons for not supporting multiple browsers tend to be closer to the "didn't bother" kind whereas we're looking to get a substantial benefit from clang that MSVC and GCC don't offer to us but clang likely will: Cross-language inlining across code compiled with clang and code compiled with rustc. To the extent Mozilla runs the compiler, it makes sense to go for the Open Source choice that allows us to deliver better on "performance as a feature". We still have at least one static analysis running on GCC, so I wouldn't expect GCC-compatibility to be dropped even if the app wouldn't be "best compiled with" GCC. The Linux distro case is trickier than Mozilla's compiler choice. For CPUs that are tier-3 for Mozilla, we already tolerate less great performance attributes in order to enable availability, so distros keeping using GCC for tier-3 probably isn't a problem. x86_64 could be a problem, though. If Firefox's performance becomes significantly dependent on having cross-language inlining, and I expect it will, having a substantial portion of the user base run without it while thinking they have a top-tier build could be bad. I hope we can get x86_64 Linux distros to track our compiler configuration closely. I do feel bad for the GCC devs, but it's worth noting that this is a result of a deliberate decision not to modularize GCC for licensing strategy reasons while LLVM has been designed as a module, demand for which has had solid technical reasons. The modularity meant it made more sense to build rustc on LLVM than on GCC and now that technical design leads to better synergy with clang. -- Henri Sivonen hsivo...@hsivonen.fi https://hsivonen.fi/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
