I no longer think a 50% stable experiment will be required to get small confidence intervals for all our metrics.
I sent an Intent to Ship: https://groups.google.com/a/chromium.org/g/blink-dev/c/5SZB2CFFGqE On Friday, November 4, 2022 at 1:30:33 PM UTC-4 fdo...@chromium.org wrote: > Hi Mike, > > With 1 week of ramped up data of stable, we know that this intervention > reduces overall CPU usage from Chrome by a few (single-digit) percents on > all desktop platforms. On ChromeOS, this reduces jank (it's typical for > CPU-reduction experiments to reduce jank on ChromeOS, due to tight resource > constraints). On Mac, it extends battery life. There is no regression for > any guiding metric. > > We're hoping to experiment on 50% Stable to get more clarity on the > battery life impact. It's important to get this data to prioritize future > investments. Confidence intervals for battery discharge rate are wide with > 7-days of 1% stable experiment. For other CPU reduction experiments, > ramping up to 50% Stable was the fastest way to obtain smaller confidence > intervals. > > Do you recommend sending an "Intent to Ship" asking to ramp up gradually: > 1% -> 50% -> 100%? > > François > > On Thu, Nov 3, 2022 at 8:32 AM Mike West <mk...@chromium.org> wrote: > >> Hey Jiahe, >> >> Can you summarize what you've learned from the 1% experiment? The design >> doc isn't exactly clear on what y'all would consider "success" from an >> experimental perspective, so I'd like to understand what you're evaluating. >> >> I'm also curious about the hop from 1% to 50%. What do you expect to >> learn from the 50:50 experiment that you're not learning from the 1%? We >> often do incremental rollouts, ramping from 1% to 5% to 10% and so on, but >> I think you could do that as part of an Intent to Ship, rather than >> extending this experiment. >> >> -mike >> >> >> On Thu, Nov 3, 2022 at 11:57 AM Jiahe Zhang <jiahe...@intel.com> wrote: >> >>> We've been experimenting this on 1% Stable on M107 for weeks , and the >>> results are quite encouraging. >>> So I'd like to request a larger scope of experiments to 50% Stable. >>> Please review. >>> >>> Best Regards, >>> Jiahe >>> >>> On Tuesday, October 4, 2022 at 4:45:42 AM UTC+8 François Doray wrote: >>> >>>> Update: We ended up experimenting with M106+, because there was a bug >>>> in the code in prior versions. The M106 Beta experiment has good results. >>>> We'll start the 1% Stable experiment this week. >>>> >>>> On Tue, Aug 9, 2022 at 9:47 AM François Doray <fdo...@google.com> >>>> wrote: >>>> >>>>> Thanks! I started the 1% Stable experiment. I will share an overview >>>>> of the results in ~3 weeks. >>>>> >>>>> On Tue, Aug 9, 2022 at 4:21 AM Mike West <mk...@chromium.org> wrote: >>>>> >>>>>> IMO, this is somewhere on the border between a web-visible experiment >>>>>> and a pure expression of user agent preference regarding flexibility >>>>>> explicitly carved out in a standard. >>>>>> >>>>>> Rather than debating the feature's philosophical state, I'd simply >>>>>> treat this email as an Intent to Experiment from M104 (current stable) >>>>>> to >>>>>> M107, and give you an explicit LGTM. >>>>>> >>>>>> Additionally: it would be ideal for the experience you gather in this >>>>>> experiment to fold back into the spec as an "Implementation >>>>>> Consideration" >>>>>> that might help other implementers determine how to use the flexibility >>>>>> the >>>>>> spec provides. >>>>>> >>>>>> -mike >>>>>> >>>>>> >>>>>> On Fri, Aug 5, 2022 at 9:24 PM 'François Doray' via blink-dev < >>>>>> blin...@chromium.org> wrote: >>>>>> >>>>>>> +Scott Haseley as an expert in this field. >>>>>>> >>>>>>> We would like to start experimenting with this intervention on 1% >>>>>>> Stable very soon. We've been experimenting on 50% of Beta for almost 2 >>>>>>> months. The results are encouraging and we aren't aware of negative Web >>>>>>> developer feedback. Do we need your LGTM to proceed? >>>>>>> >>>>>>> On Mon, Aug 1, 2022 at 3:45 AM Zhang, Jiahe <jiahe...@intel.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Contact emails >>>>>>>> >>>>>>>> jiahe...@intel.com, fdo...@chromium.org >>>>>>>> >>>>>>>> >>>>>>>> Specification >>>>>>>> >>>>>>>> https://html.spec.whatwg.org/multipage/timers-and-user-prompts.html >>>>>>>> >>>>>>>> >>>>>>>> Design docs >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> https://docs.google.com/document/d/1WFyfKUUxqM7uKxKOGhLiOjyY6T7QRduVcuHN0f6vJkk/edit >>>>>>>> >>>>>>>> >>>>>>>> Summary >>>>>>>> >>>>>>>> Enter Intensive Wake Up throttling after 10 seconds if the page is >>>>>>>> fully loaded when it becomes hidden. Currently, wake ups from JS >>>>>>>> timers >>>>>>>> with a nesting level >= 5 are throttled to 1 per minute after the page >>>>>>>> has >>>>>>>> spent 5 minutes in the background [1], which is very conservative and >>>>>>>> was >>>>>>>> chosen to allow a launch of Intensive Wake Up Throttling with minimal >>>>>>>> regression risk. We're now planning to reduce this timeout to 10 >>>>>>>> seconds if >>>>>>>> the page is fully loaded when hidden. [1] >>>>>>>> https://chromestatus.com/feature/4718288976216064 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Blink component >>>>>>>> >>>>>>>> Blink>Scheduling >>>>>>>> <https://bugs.chromium.org/p/chromium/issues/list?q=component:Blink%3EScheduling> >>>>>>>> >>>>>>>> >>>>>>>> TAG review >>>>>>>> >>>>>>>> Not applicable. This feature changes the behavior of an existing >>>>>>>> API, while remaining spec-compliant ("Optionally, wait a further >>>>>>>> implementation-defined length of time. >>>>>>>> <https://html.spec.whatwg.org/multipage/timers-and-user-prompts.html#run-steps-after-a-timeout> >>>>>>>> ") >>>>>>>> >>>>>>>> Risks >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Interoperability and Compatibility >>>>>>>> >>>>>>>> >>>>>>>> *Gecko*: No signal >>>>>>>> >>>>>>>> *WebKit*: No signal >>>>>>>> >>>>>>>> *Web developers*: No signals >>>>>>>> >>>>>>>> *Other signals*: The more conservative version of Intensive Wake >>>>>>>> Up Throttling shipped smoothly to 100% Stable more than 1 year ago. A >>>>>>>> few >>>>>>>> bugs were filed, but in all cases we've been able to propose >>>>>>>> workarounds >>>>>>>> which made apps more efficient (example >>>>>>>> <https://bugs.chromium.org/p/chromium/issues/detail?id=1186569#c16> >>>>>>>> ). >>>>>>>> >>>>>>>> >>>>>>>> WebView application risks >>>>>>>> >>>>>>>> *Does this intent deprecate or change behavior of existing APIs, >>>>>>>> such that it has potentially high risk for Android WebView-based >>>>>>>> applications?* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> No, this feature will only ship on desktop platforms. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Goals for experimentation >>>>>>>> >>>>>>>> We plan to experiment on 1% Stable to confirm whether we observe >>>>>>>> the same memory and power improvements as in the lab and on lower >>>>>>>> channels. >>>>>>>> We will decide whether this intervention ships based on the experiment >>>>>>>> data. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Ongoing technical constraints >>>>>>>> >>>>>>>> None >>>>>>>> >>>>>>>> >>>>>>>> Debuggability >>>>>>>> >>>>>>>> This is not a new Web Platform feature. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Will this feature be supported on all six Blink platforms (Windows, >>>>>>>> Mac, Linux, Chrome OS, Android, and Android WebView)? >>>>>>>> >>>>>>>> No >>>>>>>> >>>>>>>> This feature will only ship on desktop platforms. On Android, the >>>>>>>> system severely limits resource consumption from background renderers, >>>>>>>> which makes this feature unnecessary. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Is this feature fully tested by web-platform-tests >>>>>>>> <https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md> >>>>>>>> ? >>>>>>>> >>>>>>>> Yes >>>>>>>> >>>>>>>> >>>>>>>> Flag name >>>>>>>> >>>>>>>> quick-intensive-throttling-after-loading >>>>>>>> >>>>>>>> >>>>>>>> Requires code in //chrome? >>>>>>>> >>>>>>>> False >>>>>>>> >>>>>>>> >>>>>>>> Tracking bug >>>>>>>> >>>>>>>> https://bugs.chromium.org/p/chromium/issues/detail?id=1324656 >>>>>>>> >>>>>>>> >>>>>>>> Estimated milestones >>>>>>>> >>>>>>>> DevTrial on desktop >>>>>>>> >>>>>>>> 105 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Link to entry on the Chrome Platform Status >>>>>>>> >>>>>>>> https://chromestatus.com/feature/5580139453743104 >>>>>>>> >>>>>>>> This intent message was generated by Chrome Platform Status >>>>>>>> <https://chromestatus.com/>. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "blink-dev" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to blink-dev+...@chromium.org. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAGD3t5E9r%2BjOcMa5nR3ZgYjNykEyj8bUBmjvszgFYmiBJKP-dA%40mail.gmail.com >>>>>>> >>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAGD3t5E9r%2BjOcMa5nR3ZgYjNykEyj8bUBmjvszgFYmiBJKP-dA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/792b0400-0bca-4164-9015-a06c7657ff1fn%40chromium.org.
