Upcoming changes to the Linux 32-bits and Android builders
Hi, With the increasing number of reports of local builds for Linux and Android hitting the 32-bits address space limits when linking, and with the fact that gcc 4.7 on try does hit that limit on PGO builds, it has become necessary to switch the 32-bits Linux and Android builds to 64-bits environments and toolchains. This switch is going to happen some time today (May 15). In practice, this doesn't change much of anything, since the 64-bits toolchain just happily builds 32-bits binaries. However, since the Android NDK currently in use (r8c) doesn't have a 64-bits toolchain, we'll also switch to NDK r8e. We also arranged things such that pushing to try should work as long as the patch for bug 864262 is applied, which is the case on most branches tips, including aurora, beta, release and esr17. For Android, the previous 32-bits toolchain will be used if pushing branches without bug 859984 applied, which isn't a problem, as we made the 32-bits NDK available in the 64-bits environments. Following this, we're going to switch all Linux builds to gcc 4.7, later this week. Cheers, Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS.File and shutdown
On 5/14/13 8:35 PM, Felipe Gomes wrote: Should profile-before-change then be my call to stop accepting changes to the data and call writeAtomic to flush it? I've seen some code nearby doing it at quit-application-granted. Or perhaps there's no correct answer and it varies case by case (or anything goes that works and is early enough..) profile-before-change should be good. Any OS.File call posted before xpcom-shutdown will be completed before we exit Firefox. Cheers, David -- David Rajchenbach-Teller, PhD Performance Team, Mozilla ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Upcoming changes to the Linux 32-bits and Android builders
On 5/15/13 2:42 AM, Mike Hommey wrote: Following this, we're going to switch all Linux builds to gcc 4.7, later this week. Android NDK r8e also includes gcc 4.7. We may want to investigate updating to gcc 4.7 on Android, too. chris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Upcoming changes to the Linux 32-bits and Android builders
On Wed, May 15, 2013 at 11:36:01AM -0700, Chris Peterson wrote: On 5/15/13 2:42 AM, Mike Hommey wrote: Following this, we're going to switch all Linux builds to gcc 4.7, later this week. Android NDK r8e also includes gcc 4.7. We may want to investigate updating to gcc 4.7 on Android, too. We use stripped down ndks in tooltool that only contain one toolchain because of size concerns, however upgrading to gcc 4.7 on android shouldn't be hard. You can generate an appropriate ndk with the script in bug 859984 then get that uploaded by releng and you should be good to try gcc 4.7 on try and when you're ready change the mozconfigs in the tree. Doing that's on my list of things to do, but I don't mind if someone else does it first. Trev chris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Windows 7 test jobs running on iX
These jobs are now running across the board up to mozilla-aurora. It seems that dromaeo css on pgo is misbehaving: https://bugzilla.mozilla.org/show_bug.cgi?id=870453 cheers, Armen On 2013-05-13 11:23 AM, Armen Zambrano G. wrote: The intermittent failures on consecutive changesets were not as bad as I first thought. This week we should have the jobs be running across all branches. IT should have the machines ready between Tuesday to Wednesday. cheers, Armen On 2013-05-09 3:20 PM, Armen Zambrano G. wrote: Hi all, We have started running Windows 7 test jobs in our new iX hardware on the Cedar tree. These machines will replace our current Rev3 minis. The reason I'm reaching you out is because I have found some intermittent failures on those machines. I need some help investigating them. I'm afraid that they may happen too often that it will make triaging the tree and inconvenience for sheriffs and developers. * [1] Bug 820738 - Intermittent browser_newtab_block.js | grid status = 0,1,2,3,6,7,8,9, - Got 0,1,2,3,6,7,8,9,http://mochi.test:/browser/browser/base/content/test/test_wyciwyg_copying.html, expected 0,1,2,3,6,7,8,9, (and another) * [2] Bug 870485 - Intermittent reftest failure on Windows 7 on iX machines * [3] Bug 870490 - Intermittent mochitest-browser-chrome failure on Windows 7 on iX machines * [4] Bug 870488 - Intermittent xpcshell failure on Windows 7 on iX hardware I hope to have more machines next week and I will enable the jobs on Mozilla-Inbound and other branches as well. Thanks in advance, Armen [1] https://bugzilla.mozilla.org/show_bug.cgi?id=820738 [2] https://bugzilla.mozilla.org/show_bug.cgi?id=870485 [3] https://bugzilla.mozilla.org/show_bug.cgi?id=870490 [4] https://bugzilla.mozilla.org/show_bug.cgi?id=870488 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Ordering shutdown observers?
On 5/15/13 2:18 PM, Vladan Djeric wrote: Hi all, I recently came across a situation where it would have been useful to control the order in which components receive the same shutdown notification. Specifically, Telemetry writes out session data to the profile dir during the profile-before-change event, but it needs to include any measurements reported by other components while they are handling the same profile-before-change notification. So ideally, Telemetry would be the last observer to get the notification. As an additional example, OS.File needs to be the second-last observer of profile-before-change, since it needs to service new requests to OS.File from other components during their handling of profile-before-change, but it also needs to record measurements in Telemetry before Telemetry has handled the same event. I'd like to know if these use-cases are sufficiently rare that we should just add new shutdown events when needed (e.g. we added profile-before-change2 for Telemetry in bug 844331), or if we should come up with a general way to impose order of shutdown based on dependencies? There are race conditions all over the place. Another example is preferences. Preferences flush during one of the shutdown observers IIRC. If your observer mutates preferences after preferences has fired, pref writes are lost. So, it all comes down to relying on the order observers were registered on startup (since observers are notified in registration order). This all seems rather fragile to me. I think the more compelling use case is service startup. Proper dependencies should allow us to more intelligently start services on demand. This should lead to lower resource utilization and faster startup times. Shutdown times should also speed up if there are fewer services to shut down. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Ordering shutdown observers?
On 2013-05-15 5:18 PM, Vladan Djeric wrote: I'd like to know if these use-cases are sufficiently rare that we should just add new shutdown events when needed (e.g. we added profile-before-change2 for Telemetry in bug 844331), or if we should come up with a general way to impose order of shutdown based on dependencies? Do you have use cases besides these two? Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Ordering shutdown observers?
On Wed, May 15, 2013 at 05:18:09PM -0400, Vladan Djeric wrote: Hi all, I recently came across a situation where it would have been useful to control the order in which components receive the same shutdown notification. Specifically, Telemetry writes out session data to the profile dir during the profile-before-change event, but it needs to include any measurements reported by other components while they are handling the same profile-before-change notification. So ideally, Telemetry would be the last observer to get the notification. As an additional example, OS.File needs to be the second-last observer of profile-before-change, since it needs to service new requests to OS.File from other components during their handling of profile-before-change, but it also needs to record measurements in Telemetry before Telemetry has handled the same event. I'd like to know if these use-cases are sufficiently rare that we should just add new shutdown events when needed (e.g. we added profile-before-change2 for Telemetry in bug 844331), or if we should come up with a general way to impose order of shutdown based on dependencies? I believe roc proposed just having an explicit hard coded list of things to start up a while ago, and I'm tempted to say that's what we should do for shutdown too. So just add an explicit call to some os.file thing followd by a call to a telemetry function after profile-before-change but not after anything else. Trev Thanks, Vladan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Ordering shutdown observers?
I believe roc proposed just having an explicit hard coded list of things to start up a while ago, and I'm tempted to say that's what we should do for shutdown too. So just add an explicit call to some os.file thing followd by a call to a telemetry function after profile-before-change but not after anything else. Indeed, to implement ClearOnShutdown, I added an explicit call to KillClearOnShutdown during the XPCOM shutdown process, precisely because I needed to guarantee a particular ordering wrt shutdown observers. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Ordering shutdown observers?
On Wednesday 2013-05-15 14:32 -0700, Gregory Szorc wrote: I think the more compelling use case is service startup. Proper dependencies should allow us to more intelligently start services on demand. This should lead to lower resource utilization and faster startup times. Shutdown times should also speed up if there are fewer services to shut down. This is what we do already; we don't create an XPCOM service until somebody asks for it. Now, I'm not saying that all of our code is perfect about not *asking* for the service until it's needed. But in many cases that's more trouble than it's worth; there are many things we know we'll need during startup, and it's not worth the extra overhead of checking every time if we've already called getService. -David -- 턞 L. David Baron http://dbaron.org/ 턂 턢 Mozilla http://www.mozilla.org/ 턂 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Ordering shutdown observers?
On 5/15/13 3:31 PM, L. David Baron wrote: On Wednesday 2013-05-15 14:32 -0700, Gregory Szorc wrote: I think the more compelling use case is service startup. Proper dependencies should allow us to more intelligently start services on demand. This should lead to lower resource utilization and faster startup times. Shutdown times should also speed up if there are fewer services to shut down. This is what we do already; we don't create an XPCOM service until somebody asks for it. Now, I'm not saying that all of our code is perfect about not *asking* for the service until it's needed. But in many cases that's more trouble than it's worth; there are many things we know we'll need during startup, and it's not worth the extra overhead of checking every time if we've already called getService. Ahh, I was thinking more of JS services. In this world, your manifest adds an entry to the app-startup category and your service receives the app-startup notification. It is customary for it to register an observer for a later startup phase and finish initialization then. Many services also install one shot timers so they don't overburden app init. Typically if there are service dependencies, you have service A emit a notification and service B then starts upon receiving it. A problem with this model is you can't reliably and/or easily init service B by itself because B doesn't necessary know how to init A. The current model is essentially a giant domino reaction cascaded from app-startup. I think I'd prefer a pull based solution where the first consumer of service B (perhaps it's a timer or an observer if B is frequently-used) tries to grab an instance and this results in service A being instantiated on-demand. This is accomplished via a service management framework of some kind that knows about all services and their dependencies. This is comparable to how *NIX operating systems initialize. Gecko is currently using System-V style runlevels. But all the new hotness are replacements like sytemd and upstart which explicitly express service dependencies, can start services in parallel and on demand, and are much faster at starting the operating system as a result. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Ordering shutdown observers?
Gregory Szorc wrote: Ahh, I was thinking more of JS services. In this world, your manifest adds an entry to the app-startup category and your service receives the app-startup notification. It is customary for it to register an observer for a later startup phase and finish initialization then. Well, if it's still at startup then all you're doing is moving the service load to a different point in startup, but you can register with profile-after-change instead if that's early enough. (I don't think any other notifications have a category.) -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Ordering shutdown observers?
On Wed, May 15, 2013 at 4:00 PM, Gregory Szorc g...@mozilla.com wrote: Ahh, I was thinking more of JS services. In this world, your manifest adds an entry to the app-startup category and your service receives the app-startup notification. It is customary for it to register an observer for a later startup phase and finish initialization then. We're on a tangent here, but many (most?) such services only care to be instantiated so that they can in turn observe other events (or they need to be continuously observing events from other services). It's sometimes wasteful to load+initialize the entire service just to observe events, so a mechanism similar to the existing messageWakeupService.js (but for observer service notifications rather than message manager messages) could be useful in avoiding some of that waste. Gavin ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: OS.File and shutdown
Felipe Gomes schrieb: What prompted the question is that I'm working on a conversion from SQLite storage to a JSON file. OS.File.writeAtomic provides a good guarantee of data consistency against crashes etc, and I'm now looking how to guarantee a proper full flush of the data to disk during shutdown. Just make sure you do (lazily but still) also flush that data out to disk every now and then during normal operation. For one thing, people leave Firefox running for hours, days and even weeks and if they run into a crash, they shouldn't lose state over such long periods, and for the other, we want shutdown to be fast and the ideal situation is that you already have written everything when the shutdown request comes and you don't need to care. ;-) Robert Kaiser ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Storage in Gecko
David Rajchenbach-Teller schrieb: I'd even go as far as limiting it to 16kb. (possibly with a transition phase during which going above 16kb only prints warnings) I think most of us agree, but the problem is that apparently a number of add-ons rely on large prefs atm, so right now we did set to 1MB. Adding a warning for everything over 10KB or 16KB or something and targeting to move the limit down to that at some point would surely be a good idea, and I'd be happy about someone filing a bug and patch about this. For now, the important thing was to prohibit the case that caused crashes, but we surely can do better and reduce that kind of prefs mis-use in favor of indexedDB, etc. Robert Kaiser ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
