Hello everybody, Thanks so much for all your feedback!
I’d like to give a few general comments to clarify how we created this video and to respond to some of the comments that we received. I’ll respond a few individual comments further below. ReTiSAR does realtime processing, which means that what happens is independent of whether you plug in a microphone array and read the signals of the individual microphones from the AD converter or if you read the pre-recorded raw microphone signals from a storage medium. In the video, we first recorded the microphone signals. Afterwards, we rendered them with ReTiSAR while recording ReTiSAR’s output and while I was moving the head left and right in the right moment. We then added these signals to the video. Upon execution, ReTiSAR expands the captured sound field that is represented by the microphone signals block-by-block into spherical harmonics (SH). Some HRTF set will also be loaded and expanded into SH. There is then some processing the SH domain happening that virtually puts the head whose HRTFs are being used into the sound field that was captured by the array and computes the resulting ear signals. So, it’s microphone signals in -> binaural signals out. As all processing is performed live, we can rotate the sound field and the HRTFs against each other and thereby change the head orientation. Currently, ReTiSAR supports Polhemus Fastrack and some old Razor head tracker as tracking sensors. Please believe me, listening to this example with head tracking makes a world of a difference. There are several reasons why the output of the pipeline may sound different than the real thing: 1) Of course, the HRTFs are potentially non-individual. ReTiSAR loads SOFA files. So, you are more than welcome to try your own HRTF set. Ideally, it should have no gaps anywhere in the measured grid so that there is no ambiguity when expanding the HRTFs into SH. This is an involved topic. Note that the default setting of ReTiSAR will play an Eigenmike recording that is very similar to the one in the video (but less formal and in a drier room). So please give it a try! 2) The finite number of sensors forces us to limit the SH order to 4. This affects both timbre and spatial perception. Several of you have noted that there seems a difference in timbre between front/rear and the lateral source locations. Indeed, there are physical reasons for this related to the SH order limitation. Finding ways to mitigate this is actually an active research topic at the moment. I’ll append a list of example references on this. Please excuse if this one is not comprehensive. We actually did an extensive listening test in which we compared a selection of these methods. The article on it is currently under review at JAES. In a nutshell, the result was that all these methods achieve an improvement. By the way, the signals from the video happened to employ the method by Hold et al. (for no particular reason). A few authors have looked into how these order-limitation issues improve with increasing order. All found that around, say, order 10 is where things saturate. Then, for most source locations (at least as long as they are in the horizontal plane), the array rendering sounds indistinguishable from a direct dummy head recording of the same scenario (of course, when using the dummy head’s HRTFs in the array pipeline). Order 10 means at least (10+1)^2 microphones, though. BTW, using 100+ microphones is not a problem regarding the sensor self noise. The SNR in the binaural signals can even be higher than what it is in the signal of a given microphone. Stay tuned for Hannes Helmholz’s Forum Acusticum paper on this. Augustine rote: > Hey ! My cynicism alarms were buzzing away, its my default setting now - but that's actually really effective - well done ! I don't suppose you do a VST plugin / panner of some sort ? Sorry, no, we haven’t planned this. It will actually be difficult to wrap ReTiSAR in such a framework. I’d rather want to point you to other implementations of the same processing pipeline, which are presumably doing exactly the same like ReTiSAR: https://plugins.iem.at/ http://research.spa.aalto.fi/projects/sparta_vsts/ Greg wrote: > It would be interesting if there were other uploads of this rendered with different head models (or even just alternative audio only versions): This didn't image for me at all except for the part where the speaker gets close to the right ear, that part imaged well and caused me to jump in my seat a bit. > > IIRC only two of the HRTFs in the listen database images well for me, and kemar doesn't at all, so I'm not surprised. As you know, it’s hard to diagnose from the distance why it is such. I can only recommend that you try ReTiSAR with an HRTF set of which you know that it works for you (the default is Neumann KU100, which is also what we used in the video). As mentioned above, ReTiSAR comes together with an Eigenmike recording. (Note, though, that we have tested ReTiSAR on macOS only.) Fernando wrote: > Right and Left sounded less reverberated, although the distance does not seem > to change so much - and the tone quality was (seemed?) brighter. The timbre differences can be well explained with the effects of order truncation. We don’t really know what it is that changes the perception of the reverb. Best regards, Jens — Here’s a few references on the order limitation. Some include propositions for mitigating its effect: C. Hold, H. Gamper, V. Pulkki, N. Raghuvanshi, I. J. Tashev, “Improving Binaural Ambisonics Decodingby Spherical Harmonics Domain Tapering and ColorationCompensation,” presented at the International Conference on Acoustics, Speech and Signal Processing, vol. 2, pp. 261–265 (2019) M. Zaunschirm, C. Schörkhuber, R. Höldrich,“Binaural rendering of Ambisonic signals by head-relatedimpulse response time alignment and a diffuseness constraint,” The Journal of the Acoustical Society of America,vol. 143, no. 6, pp. 3616–3627 (2018). Z. Ben-Hur, F. Brinkmann, J. Sheaffer, S. Weinzierl, B. Rafaely, “Spectral equalization in binaural signals represented by order-truncated spherical harmonics,” The Journal of the Acoustical Society of America, vol.141, no. 6, pp. 4087–4096 (2017). J. Ahrens, C. Andersson, “Perceptual evaluation of headphone auralization of rooms captured with spherical microphone arrays with respect to spaciousness and timbre,” Journal of the Acoustical Society of America, vol.145, no. April, pp. 2783–2794 (2019). > On 22 May 2020, at 01:20, Jacob William Wolfe <[email protected]> wrote: > > Wow. Perception of distance especially - he sounds directly in my ear when he > gets close. > > On 5/21/20, 6:04 PM, "Sursound on behalf of mgraves mstvp.com" > <[email protected] on behalf of [email protected]> wrote: > > That's really good! > > Michael Graves > [email protected] > o: (713) 861-4005 > c: (713) 201-1262 > sip:[email protected] > > -----Original Message----- > From: Sursound <[email protected]> On Behalf Of Jens Ahrens > Sent: Thursday, May 21, 2020 11:04 AM > To: Sursound <[email protected]> > Subject: [Sursound] Binaural rendering of an Eigenmike recording > > Hello everyone, > > … and another post from me. > > Here’s a quick 2-min video of what binaural rendering of an Eigenmike > recording can sound like for those of you who haven’t heard this before: > https://youtu.be/qcqeygqjxZ4 It’s 4th order rendered directly in the > spherical harmonic domain (without a virtual discrete loudspeaker array). The > rendering was done with ReTiSAR > (https://github.com/AppliedAcousticsChalmers/ReTiSAR), which is generously > funded by Facebook Reality Labs. > > Best regards, > Jens > > -- > Jens Ahrens > Associate Professor > Division of Applied Acoustics > Chalmers University of Technology > 41296 Gothenburg > Sweden > +46 (0)31 772 2210 > http://www.ta.chalmers.se/people/jens-ahrens/ > > _______________________________________________ > Sursound mailing list > [email protected] > https://mail.music.vt.edu/mailman/listinfo/sursound - unsubscribe here, > edit account or options, view archives and so on. > _______________________________________________ > Sursound mailing list > [email protected] > https://mail.music.vt.edu/mailman/listinfo/sursound - unsubscribe here, > edit account or options, view archives and so on. > > _______________________________________________ > Sursound mailing list > [email protected] > https://mail.music.vt.edu/mailman/listinfo/sursound - unsubscribe here, edit > account or options, view archives and so on. _______________________________________________ Sursound mailing list [email protected] https://mail.music.vt.edu/mailman/listinfo/sursound - unsubscribe here, edit account or options, view archives and so on.
