GitHub user Matthew-DWM created a discussion: Proposal: Audio Attribute Extractor, measuring audio files in a workflow
## Summary A new operator that reads audio files from a binary column, as File Scan produces them, and measures one property of the sound over time: its waveform, loudness, noisiness, pitch, onsets or beats. Each file becomes rows of `x` (time in seconds), `y` (the value) and `series` (the property's name), so the existing chart, aggregate and join operators work on it unchanged. The idea comes from issue #3762. A prototype of the feature exists at PR #9146 with a demo video and additional details. ## Motivation Texera can already load audio files as binary data, and the Hugging Face operator can send audio to a hosted model for speech recognition or classification. What's missing is a way to look at the sound itself. Today that takes a Python UDF that brings its own decoder and signal-processing code. Some uses: - Compare a folder of recordings by tempo, loudness or how busy each one is, joined with their metadata. - Follow the pitch of a voice or an instrument over time. - Find quiet, loud or noisy stretches, for example to check recordings before sending them to a speech model. - Count sound events, such as notes or taps, per file. ## Approach **For the user.** One operator in Utilities. Pick the audio column (WAV, MP3 or FLAC) and one attribute: | Attribute | `y` | |---|---| | `waveform` | the signal | | `rms` | loudness of each frame | | `zcr` | zero-crossing rate: high for noise, low for tones | | `pitch_hz` | pitch, 60 to 1000 Hz | | `onsets` | 1.0 at each new sound | | `beats` | 1.0 at each beat | The other settings are the sample rate, the frame and hop length, a cap on rows and seconds per file, keeping the input's other columns (such as the filename), and skipping unreadable files instead of stopping. **How it runs.** Like the other Python-based operators: the Scala descriptor checks the settings when the workflow compiles and generates the Python a worker runs. The worker decodes each file, mixes it to mono (single audio stream), resamples it, measures it and emits a bounded number of rows per file. **Dependencies.** One new Python package, `miniaudio` (MIT), which brings `cffi` (MIT-0) and `pycparser` (BSD-3). NumPy and SciPy are already there. ## Prototype #9146 has the operator (about 1,350 lines, most of it the Python the worker runs), 68 tests, a docs page and a recording of an example workflow. On my machine, a four-minute WAV takes at most about 1.2 s per attribute. ## Open questions 1. Would you find an audio operator useful? 2. Is a new Python dependency acceptable (`miniaudio`, plus `cffi` and `pycparser`)? 3. Should the scope of the current PR be reduced and split into multiple PRs? 4. Is use of the librosa-derived code acceptable? GitHub link: https://github.com/apache/texera/discussions/9276 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
