Contact emails [email protected]
Explainer https://github.com/WebAudio/web-speech-api/blob/main/explainers/speech-recognition-result-timestamps.md Specification https://webaudio.github.io/web-speech-api/#speechreco-result Summary The Web Speech API currently does not expose the acoustic timing bounds of recognized speech segments, making it difficult to associate transcribed text with media timelines or generate synchronized subtitle cues. We propose adding speechStartTime and speechEndTime (double, in seconds relative to stream start) attributes on the SpeechRecognitionResult interface. Blink component Blink>Speech <https://issues.chromium.org/issues?q=customfield1222907:%22Blink%3ESpeech%22> Web Feature ID speech-recognition <https://webstatus.dev/features/speech-recognition> Motivation The Web Speech API currently does not expose the acoustic start and end timestamps of recognized speech utterances on SpeechRecognitionResult. While SpeechRecognitionEvent.timeStamp conveys when the DOM event occurred, it provides no information about the acoustic boundaries or duration of the speech segment within the source audio stream. Exposing speechStartTime and speechEndTime (in seconds as a double, matching HTMLMediaElement.currentTime, BaseAudioContext.currentTime, and VTTCue) addresses critical media timeline association capabilities that are currently impossible on the web: 1. Automated Subtitling & Caption Tracks: Constructing frame-accurate subtitle cues (VTTCue(start, end, text)) for recorded or streaming audio/video. 2. Interactive "Click-to-Seek" Navigation: Enabling users to click a phrase in a transcript and immediately seek playback to the exact moment it was spoken. 3. Conferencing & Media Recording Alignment: Synchronizing real-time captions with recorded media tracks or WebRTC playout delay buffers. 4. Text-Based Media Editing: Accurately cutting, splicing, or rearranging audio and video based on transcript text. Initial public proposal https://github.com/WebAudio/web-speech-api/issues/191 TAG review https://github.com/w3ctag/design-reviews/issues/1267 TAG review status Issues addressed Risks Interoperability and Compatibility Low interoperability and compatibility risk. This is a purely additive change introducing two read-only double attributes (speechStartTime and speechEndTime) on SpeechRecognitionResult, along with updating SpeechRecognitionEvent.timeStamp to reflect the end of the recognized speech segment on the session audio timeline. Existing applications using SpeechRecognition continue to work without modification, and developers can feature-detect support via 'speechStartTime' in SpeechRecognitionResult.prototype. The specification update ( https://github.com/WebAudio/web-speech-api/pull/192) was shaped by feedback from the W3C Audio Working Group / Community Group (including @padenot at Mozilla) and the W3C TAG ( https://github.com/w3ctag/design-reviews/issues/1267). Gecko: No signal (https://github.com/WebAudio/web-speech-api/pull/192) Unofficial approval from @padenot (Mozilla) on spec update. WebKit: No signal Web developers: Positive ( https://github.com/WebAudio/web-speech-api/issues/191) Requested by internal and external web developers building real-time captioning, media recording alignment, and latency-sensitive conferencing applications (including Google Meet). For example, speechStartTime and speechEndTime are used to generate synchronized VTTCue captions and interactive click-to-seek transcripts and developer specifically needed this feature to properly map results to the timeline to know when the speech that generated the text in the result started and ended for applications such as Subtitling & Closed Captions, Meeting Transcripts, Live WebRTC Video Sync (see crbug.com/551992211#comment3). Other signals: WebView application risks Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications? None. This feature only adds new read-only attributes to SpeechRecognitionResult and does not deprecate or break existing APIs. Furthermore, the Web Speech API (SpeechRecognition) is not supported on Android WebView. Debuggability No special DevTools support is needed. speechStartTime and speechEndTime are standard read-only IDL attributes exposed on SpeechRecognitionResult instances (and timeStamp on SpeechRecognitionEvent) and can be inspected directly in the DevTools Console, Scope pane, and watch expressions like any other DOM object properties. Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)? No Supported on desktop platforms (Windows, Mac, Linux, and ChromeOS). Not currently supported on Android or Android WebView: Android WebView does not support the SpeechRecognition API, and Chrome on Android delegates to the OS-level Android SpeechRecognizer service, which does not expose segment-level start and end timestamps. The WebSpeechTimestamps runtime feature flag will only be enabled on desktop platforms (Windows, Mac, Linux, and ChromeOS), leaving speechStartTime and speechEndTime unexposed on Android and Android WebView. Is this feature fully tested by web-platform-tests <https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md> ? No End-to-end Web Speech API recognition tests cannot currently run in automated Web Platform Tests (WPT) because WPT lacks a WebDriver mock speech recognition backend automation hook (and on-device recognition models are not bundled in standard WPT test runners). The IDL surface is covered by WPT idlharness tests, and the timestamp exposure, 2ms quantization (FuzzTimestamp), and SpeechRecognitionEvent.timeStamp rebasing are covered by Blink unit tests (third_party/blink/renderer/modules/speech/speech_recognition_test.cc, crrev.com/c/8409366 and crrev.com/c/8729884) as well as browser/service unit tests (content/browser/speech/soda_speech_recognition_engine_impl_unittest.cc, content/browser/speech/on_device_speech_recognition_engine_unittest.cc, chrome/services/speech/soda_speech_recognizer_impl_unittest.cc, chrome/services/speech/speech_recognition_recognizer_impl_unittest.cc, and services/on_device_model/on_device_model_service_unittest.cc). Flag name on about://flags enable-experimental-web-platform-features Finch feature name WebSpeechTimestamps Rollout plan Will ship enabled for all users Requires code in //chrome? True While the WebIDL surface and result plumbing live in //third_party/blink and //content/browser/speech, the on-device speech recognition backends and model management that supply segment timestamps (SODA in //chrome/services/speech and Optimization Guide / on-device model integration in //chrome/browser/speech) currently live in //chrome. Tracking bug https://crbug.com/528037568 Estimated milestones Shipping on desktop 159 DevTrial on desktop 153 Anticipated spec changes Open questions about a feature may be a source of future web compat or interop issues. Please list open issues (e.g. links to known github issues in the project for the feature specification) whose resolution may introduce web compat/interop risk (e.g., changing to naming or structure of the API in a non-backward-compatible way). None. Following W3C TAG ( https://github.com/w3ctag/design-reviews/issues/1267 ) and Audio WG/CG review on PR #192 (https://github.com/WebAudio/web-speech-api/pull/192 ), the attribute names (speechStartTime and speechEndTime), type and units, and SpeechRecognitionEvent.timeStamp behavior have already been updated to match reviewer consensus. No further backward-incompatible spec changes are anticipated. Link to entry on the Chrome Platform Status https://chromestatus.com/feature/5811907077472256?gate=4580112537354240 Links to previous Intent discussions Intent to Prototype: https://groups.google.com/a/chromium.org/g/blink-dev/c/r78IyGSnclg This intent message was generated by Chrome Platform Status <https://chromestatus.com/>. -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABqJGMcN2VT2kH7cVjACrzQdpSddqnEcqTQX5VLAH9aOYusngg%40mail.gmail.com.
