I wonder if we can get enough confidence with less work than investigating 40 randomly chosen sites from UseCounter hits.
This is a population proportion problem, and https://sample-size.net/confidence-interval-proportion/ is a useful tool. If you check 40 cases and find no breakage (N=40, x=0) that gives us 95% confidence that breakage is less than 7.2% of samples in this data set. If it's useful to check that much depends on the value of the use counter. Is https://chromestatus.com/metrics/feature/timeline/popularity/4463 the right use counter, and has it reached stable yet? Why is marked as obsolete? For purposes of illustration, let's use 0.04% from earlier in the thread and say we want to be (95%) confident that real breakage is less than 0.01%. Then we just need to get below 25% in the linked tool, and checking 11 samples and finding nothing is enough to do this. On Wed, Apr 19, 2023 at 5:43 PM Mathias Bynens <m...@google.com> wrote: > Thanks for the guidance, Rick. I’ve prepared a CL moving the flag to > status=experimental > <https://chromium-review.googlesource.com/c/chromium/src/+/4447958> and > I can commit to investigating 40 unique UseCounter hits and summarizing my > findings. Fingers crossed the trend of “no actual breakage detected” > continues. I’ll keep you posted. > > On Wed, Apr 19, 2023 at 5:26 PM Rick Byers <rby...@chromium.org> wrote: > >> Thanks for doing a thorough compat analysis of this Mathias. I can >> totally see this being one where all the examples we can find don't seem to >> cause breakage in practice. I know it's a lot, but if we looked at 40 >> random examples and found none of them to break, that would suggest an >> upper bound of <0.001% of pages impacted (probably much lower) and I'd be >> OK giving this a shot with a finch killswitch ready in case of reports of >> serious breakage. Does that sound reasonable to you? >> >> Also feel free to set your flag >> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/platform/runtime_enabled_features.json5;l=1912?q=HTMLPatternRegExpUnicodeSets%20file:.json5&ss=chromium> >> to status=experimental, that'll get us some additional usage coverage (from >> the small population that runs with >> --enable-experimental-web-platform-features) and also signal that this is >> close to becoming shipping behavior. >> >> Rick >> >> On Mon, Apr 17, 2023 at 7:03 AM 'Mathias Bynens' via blink-dev < >> blink-dev@chromium.org> wrote: >> >>> So far, none of the UseCounter hits I investigated constitute any actual >>> breakage. The vast majority of hits seem to be login forms backed by >>> server-side validation. I’ll keep looking though. >>> >>> In the meantime, this feature is now >>> <https://chromium-review.googlesource.com/c/chromium/src/+/4414859> >>> available behind the `--enable-blink-features=HTMLPatternRegExpUnicodeSets` >>> flag (disabled by default). >>> >>> On Wednesday, April 5, 2023 at 5:53:10 PM UTC+2 Mathias Bynens wrote: >>> >>>> On Wed, Apr 5, 2023 at 5:23 PM Alex Russell <sligh...@chromium.org> >>>> wrote: >>>> >>>>> I don't understand why TAG review is not applicable for this intent. >>>>> >>>> >>>> Fair enough. I’ve filed a TAG review request here: >>>> https://github.com/w3ctag/design-reviews/issues/832 I’ll update the >>>> ChromeStatus entry to refer to it. >>>> >>>> On Tuesday, April 4, 2023 at 5:21:16 AM UTC-7 mt...@google.com wrote: >>>>> >>>> Thanks to the UseCounter + UKM + M112 hitting Stable, more results are >>>>>> starting to come in. I’ll be collecting public examples of potential >>>>>> incompatibilities here: >>>>>> https://bugs.chromium.org/p/chromium/issues/detail?id=1412729#c11 So >>>>>> far 0 out of the 2 examples cause any actual breakage — fingers crossed >>>>>> that trend continues. >>>>>> >>>>>> On Mon, Apr 3, 2023 at 10:26 AM Philip Jägenstedt < >>>>>> foo...@chromium.org> wrote: >>>>>> >>>>> I took a look at https://github.com/whatwg/html/pull/7908 and it >>>>>>> looks like there's agreement to merge it, but it's waiting on this >>>>>>> intent >>>>>>> to be approved. Normally we block in the other direction, but that's >>>>>>> fine, >>>>>>> as long as the spec change is merged. >>>>>>> >>>>>>> Looks like there's broad support for this change, and it's just a >>>>>>> question of the site compat risk. ~0.04% as an upper bound is quite >>>>>>> high. >>>>>>> Can we wait until the use counter is in stable and look at a random set >>>>>>> of >>>>>>> sites hitting the use counter to determine what the real-world breakage >>>>>>> looks like? >>>>>>> >>>>>>> On Fri, Mar 31, 2023 at 5:07 PM 'Mathias Bynens' via blink-dev < >>>>>>> blin...@chromium.org> wrote: >>>>>>> >>>>>> On Fri, Mar 31, 2023 at 4:35 PM Mike Taylor <mike...@chromium.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hey Mathias, >>>>>>>>> On 3/31/23 5:56 AM, Mathias Bynens wrote: >>>>>>>>> >>>>>>>>> Contact emails >>>>>>>>> >>>>>>>>> mat...@chromium.org >>>>>>>>> >>>>>>>>> Specification >>>>>>>>> >>>>>>>>> https://github.com/whatwg/html/pull/7908 >>>>>>>>> >>>>>>>>> Summary >>>>>>>>> >>>>>>>>> The <input pattern> attribute allows developers to specify a >>>>>>>>> regular expression pattern against which the input’s values are >>>>>>>>> checked for >>>>>>>>> validity. >>>>>>>>> >>>>>>>>> <label> >>>>>>>>> >>>>>>>>> Part number: >>>>>>>>> >>>>>>>>> <input pattern="[0-9][A-Z]{3}" name="part" >>>>>>>>> >>>>>>>>> title="A part number is a digit followed by three >>>>>>>>> uppercase letters."> >>>>>>>>> >>>>>>>>> </label> >>>>>>>>> >>>>>>>>> When the pattern attribute was first implemented, these regular >>>>>>>>> expressions were compiled without any RegExp flags. In 2014, the HTML >>>>>>>>> Standard changed this by implicitly enabling the u flag for the >>>>>>>>> pattern attribute, enabling better Unicode support (including support >>>>>>>>> for >>>>>>>>> Unicode character properties like \p{Letter}). This change >>>>>>>>> shipped in Chrome 53. >>>>>>>>> <https://chromestatus.com/feature/4753420745441280> >>>>>>>>> >>>>>>>>> Now, we’re taking this to the next level by enabling the new >>>>>>>>> RegExp v flag <https://v8.dev/features/regexp-v-flag> instead of u, >>>>>>>>> enabling the use of set notation, string literal syntax, and Unicode >>>>>>>>> properties of strings. >>>>>>>>> >>>>>>>>> (Context: The RegExp v flag is a JavaScript language feature >>>>>>>>> which previously went through the Blink Intents process and shipped >>>>>>>>> in Chrome 112 <https://chromestatus.com/feature/5144156542861312>. >>>>>>>>> This new ChromeStatus entry is specifically about integrating it with >>>>>>>>> the >>>>>>>>> HTML pattern attribute.) >>>>>>>>> >>>>>>>>> Blink component >>>>>>>>> >>>>>>>>> Blink>Forms >>>>>>>>> <https://bugs.chromium.org/p/chromium/issues/list?q=component:Blink%3EForms> >>>>>>>>> >>>>>>>>> Search tags >>>>>>>>> >>>>>>>>> unicode <https://chromestatus.com/features#tags:unicode>, regexp >>>>>>>>> <https://chromestatus.com/features#tags:regexp>, pattern >>>>>>>>> <https://chromestatus.com/features#tags:pattern>, validation >>>>>>>>> <https://chromestatus.com/features#tags:validation> >>>>>>>>> >>>>>>>>> TAG review >>>>>>>>> TAG review status >>>>>>>>> >>>>>>>>> Not applicable >>>>>>>>> >>>>>>>>> Risks >>>>>>>>> Interoperability and Compatibility >>>>>>>>> >>>>>>>>> The spec patch at https://github.com/whatwg/html/pull/7908 lists >>>>>>>>> the potentially breaking changes. Some patterns that previously would >>>>>>>>> compile, now throw an early error with the v flag — specifically >>>>>>>>> those with a character class including either an unescaped special >>>>>>>>> character or a double punctuator. >>>>>>>>> >>>>>>>>> We expect such patterns to be rare. To validate this assumption >>>>>>>>> we’ve added a UseCounter called >>>>>>>>> HTMLPatternRegExpUnicodeSetIncompatibilitiesWithUnicodeMode >>>>>>>>> <https://chromestatus.com/metrics/feature/popularity#HTMLPatternRegExpUnicodeSetIncompatibilitiesWithUnicodeMode> >>>>>>>>> in M112, which tracks patterns in any JavaScript u RegExps >>>>>>>>> generated via the HTML pattern attribute that would throw if they >>>>>>>>> were used with the v flag. >>>>>>>>> >>>>>>>>> Importantly, note that any throwing pattern gracefully degrades — >>>>>>>>> it simply behaves as if the pattern attribute wasn’t present, >>>>>>>>> resulting in inputElement.validity.valid === true for any input >>>>>>>>> value. Consequently, the only compatibility risk is that some >>>>>>>>> value/pattern >>>>>>>>> combinations that would previously result in >>>>>>>>> inputElement.validity.valid being false now result in it being >>>>>>>>> true. Thus, for every UseCounter hit, it could still be that >>>>>>>>> there is no actual breakage — the UseCounter just gives us the >>>>>>>>> upper bound. The currently available data from Beta suggests the >>>>>>>>> UseCounter hits for 0.0393% of Chrome page loads. >>>>>>>>> >>>>>>>>> I'm somewhat curious to see how much that UseCounter will grow (if >>>>>>>>> at all) when 112 goes to stable next week... >>>>>>>>> >>>>>>>> Me too, and FWIW I'd understand if you and the other API owners >>>>>>>> prefer to wait until there’s some data for Stable before responding to >>>>>>>> this >>>>>>>> Intent. >>>>>>>> >>>>>>>>> Do you have any concerns about certain inputs being sent to a >>>>>>>>> server that might not have any backend validation, that would >>>>>>>>> previously be >>>>>>>>> prevented by the u-vintage validation? >>>>>>>>> >>>>>>>> That’s indeed the only scenario in which there would be breakage. >>>>>>>> So far we haven’t heard of such cases in the wild. (Arguably, such web >>>>>>>> pages are already broken, since DevTools could easily be used to >>>>>>>> remove the >>>>>>>> `pattern` attribute, or requests could be made with tools like `curl`.) >>>>>>>> FWIW, there was a similar discussion in this old blink-dev thread: >>>>>>>> https://groups.google.com/a/chromium.org/g/blink-dev/c/XUNMtri0tI4/m/mjPkwXKNAQAJ >>>>>>>> >>>>>>>> I forgot to mention that we explicitly added a console warning in >>>>>>>> M112 for any `pattern` attribute values that would be affected by this >>>>>>>> change, to help developers prepare for the potential change. One >>>>>>>> developer >>>>>>>> reported seeing the warning and adjusting their `pattern` attribute >>>>>>>> values >>>>>>>> accordingly, but it’s unclear whether inaction would have really broken >>>>>>>> their web page: >>>>>>>> https://bugs.chromium.org/p/chromium/issues/detail?id=1412729#c7 >>>>>>>> >>>>>>>> >>>>>>> >>>>>>>>> Gecko: Positive (Mozilla standards position request >>>>>>>>> <https://github.com/mozilla/standards-positions/issues/745>, >>>>>>>>> implementation >>>>>>>>> tracking issue >>>>>>>>> <https://bugzilla.mozilla.org/show_bug.cgi?id=pattern-v>) >>>>>>>>> >>>>>>>>> WebKit: Positive (WebKit standards position request >>>>>>>>> <https://github.com/WebKit/standards-positions/issues/132>, >>>>>>>>> implementation >>>>>>>>> tracking issue <https://bugs.webkit.org/show_bug.cgi?id=pattern-v> >>>>>>>>> ) >>>>>>>>> >>>>>>>>> Web developers: No signals >>>>>>>>> >>>>>>>>> Other signals: >>>>>>>>> >>>>>>>>> Debuggability >>>>>>>>> >>>>>>>>> The pattern attribute is already well-supported in DevTools and >>>>>>>>> other tooling; no changes are necessary. >>>>>>>>> >>>>>>>>> Will this feature be supported on all six Blink platforms >>>>>>>>> (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? >>>>>>>>> >>>>>>>>> Yes >>>>>>>>> >>>>>>>>> Is this feature fully tested by web-platform-tests >>>>>>>>> <https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md> >>>>>>>>> ? >>>>>>>>> >>>>>>>>> Pull Request: https://github.com/web-platform-tests/wpt/pull/38547 >>>>>>>>> >>>>>>>>> Flag name >>>>>>>>> >>>>>>>>> N/A >>>>>>>>> >>>>>>>>> Requires code in //chrome? >>>>>>>>> >>>>>>>>> False >>>>>>>>> >>>>>>>>> Tracking bug >>>>>>>>> >>>>>>>>> https://bugs.chromium.org/p/chromium/issues/detail?id=1412729 >>>>>>>>> >>>>>>>>> Sample links >>>>>>>>> >>>>>>>>> https://mathiasbynens.be/demo/pattern-u-vs-v >>>>>>>>> >>>>>>>>> Estimated milestones >>>>>>>>> >>>>>>>>> M114 >>>>>>>>> >>>>>>>>> Link to entry on the Chrome Platform Status >>>>>>>>> >>>>>>>>> https://chromestatus.com/feature/5149507107422208 >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "blink-dev" group. >>>>>>>>> >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to blink-dev+...@chromium.org. >>>>>>>>> >>>>>>>>> >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADizRgaAq4FwzJbUqLQVo%2BQdd_V0PT7rBr510OGe8fenHA%3D3HQ%40mail.gmail.com >>>>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADizRgaAq4FwzJbUqLQVo%2BQdd_V0PT7rBr510OGe8fenHA%3D3HQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "blink-dev" group. >>>>>>>>> >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to blink-dev+...@chromium.org. >>>>>>>> >>>>>>>> >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c9571b2a-a35b-3824-0f37-c93a9bb522fc%40chromium.org >>>>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c9571b2a-a35b-3824-0f37-c93a9bb522fc%40chromium.org?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "blink-dev" group. >>>>>>>> >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to blink-dev+...@chromium.org. >>>>>>> >>>>>>> >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADizRgYxU2v2ANQgzNiLD%2B4P-qJHxzTYJfRDsKNCtY0Yb_0bdg%40mail.gmail.com >>>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADizRgYxU2v2ANQgzNiLD%2B4P-qJHxzTYJfRDsKNCtY0Yb_0bdg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "blink-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to blink-dev+unsubscr...@chromium.org. >>> To view this discussion on the web visit >>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/bf73fe5b-fde2-42df-90f0-582a905d1948n%40chromium.org >>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/bf73fe5b-fde2-42df-90f0-582a905d1948n%40chromium.org?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYdiAfn2t8%2BLaGwPPRxTxuh8Vj4vqnbM4Z%2B2exGdrVDBWw%40mail.gmail.com.
