Re: [music-dsp] FFT for realtime synthesis?
Am 25.10.2018 um 12:17 schrieb gm: (also I am doing the pitch shift the wrong way at the moment, first transpose in time domain, then FFT time stretch, cause that was easier to do for now but this shouldn't cause an audible problem here) Now I think that flaw is actually the way to go Instead of doing it the standard way, FFT time stretch & filtering -> time domain pitch shift where you need an uneven workload (not a fixed number of FFTs/Second) and additional latency to write the waveform before you can read and transpose it My proposal is Offlineprocess: FFT convert to spectrum with amplitude, phase and phase derivative -> create multisample (multispectra), one spectrogramme per half octave Realtimeprocess: select multispectrum -> iFFT timestretch and pitch shift in frequency domain (without moving content from bin to bin, hence the multispectrum for each 1/2 octave) this way you have an even workload (fixed number of FFTs/second), and latency is just the time you allow for the iFFT, can be as short as 1 sample 8-) Posting this here to prevent patents ;-) , but what do you think, do I make sense? ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
the same sample as before, rearranged and sequenced, transposed sound quality and latency aside, I think the idea has some potential https://soundcloud.com/traumlos_kalt/spectromat-test-4-01/s-7W2tR the second part is from Nancy Sinatras Summervine I am sorry it's all drenched in a resonant modualated delay effect, but I think you get the idea Am 25.10.2018 um 19:13 schrieb gm: here an example at 22050 hz sample rate, FFT size 1024, smoothing for the spectral envelope 10 bins, and simple phase realignment: when amplitude is greater than last frames amplitude phase is set to original phase, otherwise to the accumulated phase of the time stretch didn't expect this wo work but it seems to work It seems to sound better to me, but still not as good as required: https://soundcloud.com/traumlos_kalt/ft-resynth-test-3-phasealign-1-22k-01/s-KCHeV Am 25.10.2018 um 17:58 schrieb gm: One thing I noticed is that it seems to sound better at 22050 Hz sample rate so I assume 1024 FFT size is too small and you should use 2048. I dont know if that is because the DC band is too high or if the bins are too broadband with 1024, or both? I assume with this and some phase realingment and a better spectral envelope quality would be somewhat improved. Unfortunately Reaktor isn't the right tool at all to test these things, you have to hack around everything, just changing the FFT size will probably waste a whole day, so I probably won't investigate this further. ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
here an example at 22050 hz sample rate, FFT size 1024, smoothing for the spectral envelope 10 bins, and simple phase realignment: when amplitude is greater than last frames amplitude phase is set to original phase, otherwise to the accumulated phase of the time stretch didn't expect this wo work but it seems to work It seems to sound better to me, but still not as good as required: https://soundcloud.com/traumlos_kalt/ft-resynth-test-3-phasealign-1-22k-01/s-KCHeV Am 25.10.2018 um 17:58 schrieb gm: One thing I noticed is that it seems to sound better at 22050 Hz sample rate so I assume 1024 FFT size is too small and you should use 2048. I dont know if that is because the DC band is too high or if the bins are too broadband with 1024, or both? I assume with this and some phase realingment and a better spectral envelope quality would be somewhat improved. Unfortunately Reaktor isn't the right tool at all to test these things, you have to hack around everything, just changing the FFT size will probably waste a whole day, so I probably won't investigate this further. ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
One thing I noticed is that it seems to sound better at 22050 Hz sample rate so I assume 1024 FFT size is too small and you should use 2048. I dont know if that is because the DC band is too high or if the bins are too broadband with 1024, or both? I assume with this and some phase realingment and a better spectral envelope quality would be somewhat improved. Unfortunately Reaktor isn't the right tool at all to test these things, you have to hack around everything, just changing the FFT size will probably waste a whole day, so I probably won't investigate this further. Am 25.10.2018 um 12:17 schrieb gm: I made a quick test, original first, then resynthesized with time stretch and pitch shift and corrected formants: https://soundcloud.com/traumlos_kalt/ft-resynth-test-1-01/s-7GCLk https://soundcloud.com/traumlos_kalt/ft-resynth-test-2-01/s-2OJ2H sounds quite phasey and gurgely I am using 1024 FFT size and 5 bins moving average to extract a spectral envelope for formant preservation, which is probably not the best way to do this I assume you would need to realign phases at transients sound quality isn't what you would expect in 2018... (also I am doing the pitch shift the wrong way at the moment, first transpose in time domain, then FFT time stretch, cause that was easier to do for now but this shouldn't cause an audible problem here) about latency I dont now yet, I am using my FFT for NI Reaktor which has a latency of several times FFT size and is only good for proof of concept stuff ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
I made a quick test, original first, then resynthesized with time stretch and pitch shift and corrected formants: https://soundcloud.com/traumlos_kalt/ft-resynth-test-1-01/s-7GCLk https://soundcloud.com/traumlos_kalt/ft-resynth-test-2-01/s-2OJ2H sounds quite phasey and gurgely I am using 1024 FFT size and 5 bins moving average to extract a spectral envelope for formant preservation, which is probably not the best way to do this I assume you would need to realign phases at transients sound quality isn't what you would expect in 2018... (also I am doing the pitch shift the wrong way at the moment, first transpose in time domain, then FFT time stretch, cause that was easier to do for now but this shouldn't cause an audible problem here) about latency I dont now yet, I am using my FFT for NI Reaktor which has a latency of several times FFT size and is only good for proof of concept stuff ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
We did FFT noise synthesis for a participatory performance project "Open Band", you can read about it on this paper: https://qmro.qmul.ac.uk/xmlui/bitstream/handle/123456789/26169/13.pdf?sequence=1 the idea was to crate blocks of noise based on pre-determined frequencies to draw letters on the spectrum. I would be really interested also in develop some tool for FFT synthesis for web audio based on drawings also best, Ariane Em qua, 24 de out de 2018 às 12:22, Ethan Fenn escreveu: > I haven't thought through the details for any particular application, but > the chirp z-transform might be a useful trick to keep in mind for these > sorts of things. It lets you calculate an IFFT with an arbitrary spacing > between bins, or even an arbitrary fundamental in case you want to detune > the partials a bit. > > -Ethan > > > On Tue, Oct 23, 2018 at 9:37 PM, gm wrote: > >> >> >> Am 24.10.2018 um 02:48 schrieb gm: >> >>> two demo tracks https://soundcloud.com/transmortal/the-way-you-were-fake https://soundcloud.com/traumlos-kalt/the-way-we-were-iii they are mostly made from a snippet of Nancy Sinatras Fridays Child >>> >> I just realize in case s.o. is really interested, I have to be more >> precise: >> >> bass and synthstrings that come in later on the second track are ordinary >> synths >> >> the rest ist granular, samples snippets are from Fridays Child, Some >> Velvet Morning >> and Summer Vine by Nancy Sinatra, Robots by Balanscu Quartett and a synth >> sample >> >> I made so many demo tracks the past days, most of them were made with the >> Fridays Child sample >> which has the advantage of being old school hardcore stereo, so you get >> three >> different sources from the same time ... >> >> >> ___ >> dupswapdrop: music-dsp mailing list >> music-dsp@music.columbia.edu >> https://lists.columbia.edu/mailman/listinfo/music-dsp >> >> > ___ > dupswapdrop: music-dsp mailing list > music-dsp@music.columbia.edu > https://lists.columbia.edu/mailman/listinfo/music-dsp ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
I haven't thought through the details for any particular application, but the chirp z-transform might be a useful trick to keep in mind for these sorts of things. It lets you calculate an IFFT with an arbitrary spacing between bins, or even an arbitrary fundamental in case you want to detune the partials a bit. -Ethan On Tue, Oct 23, 2018 at 9:37 PM, gm wrote: > > > Am 24.10.2018 um 02:48 schrieb gm: > >> two demo tracks >>> https://soundcloud.com/transmortal/the-way-you-were-fake >>> https://soundcloud.com/traumlos-kalt/the-way-we-were-iii >>> >>> they are mostly made from a snippet of Nancy Sinatras Fridays Child >>> >> > I just realize in case s.o. is really interested, I have to be more > precise: > > bass and synthstrings that come in later on the second track are ordinary > synths > > the rest ist granular, samples snippets are from Fridays Child, Some > Velvet Morning > and Summer Vine by Nancy Sinatra, Robots by Balanscu Quartett and a synth > sample > > I made so many demo tracks the past days, most of them were made with the > Fridays Child sample > which has the advantage of being old school hardcore stereo, so you get > three > different sources from the same time ... > > > ___ > dupswapdrop: music-dsp mailing list > music-dsp@music.columbia.edu > https://lists.columbia.edu/mailman/listinfo/music-dsp > > ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
Am 24.10.2018 um 02:48 schrieb gm: two demo tracks https://soundcloud.com/transmortal/the-way-you-were-fake https://soundcloud.com/traumlos-kalt/the-way-we-were-iii they are mostly made from a snippet of Nancy Sinatras Fridays Child I just realize in case s.o. is really interested, I have to be more precise: bass and synthstrings that come in later on the second track are ordinary synths the rest ist granular, samples snippets are from Fridays Child, Some Velvet Morning and Summer Vine by Nancy Sinatra, Robots by Balanscu Quartett and a synth sample I made so many demo tracks the past days, most of them were made with the Fridays Child sample which has the advantage of being old school hardcore stereo, so you get three different sources from the same time ... ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
Am 24.10.2018 um 02:24 schrieb gm: Am 24.10.2018 um 00:46 schrieb robert bristow-johnson: > Does anybody know a real world product that uses FFT for sound synthesis? > Do you think its feasable and makes sense? so this first question is about synthesis, not modification for effects, right? and real-time, correct? so a MIDI Note-On is received and you want a note coming out as quickly as possible? yes exactly i don't know of a hardware product that does inverse FFT for synthesis. i do know of a couple of effects algorithms that go into products that use FFT. i think it's mostly about doing "fast convolution" for purposes of reverb. what are you intending to synthesize? notes? or something more wild than that? just curious. basically a sample "mangler", you load an arbitray sample, a loop of music for instance, and play back parts of it in real time, time streched, pitch shifted, with formants corrected or altered, backwards, forwards I dont need polyphonic playback, though that would be nice for some things right now I do this with a granular "cloud", that is many overlapping grains, which can play polyphonic or rather paraphonic, which means that the grains play pack at different pitches simultanously, depending on the chords you play but they all play back from the same sample position and have the same kind of "treatment" like envelope or filtering. I thought you could maybe do this and some other stuff in the spectral domain the idea is to change snippets / loops of existing music into new music, this idea is not new, two demo tracks https://soundcloud.com/transmortal/the-way-you-were-fake https://soundcloud.com/traumlos-kalt/the-way-we-were-iii they are mostly made from a snippet of Nancy Sinatras Fridays Child ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
Am 24.10.2018 um 02:12 schrieb gm: Am 24.10.2018 um 00:38 schrieb David Olofson: Simple demo song + some comments here: https://soundcloud.com/david-olofson/eelsynth-ifft-flutesong sounds quite nice actually ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
The Kurzweil K150 is the first product I can think of that did it. To create custom sounds for it required the use of software that modeled the sound using partial amplitudes over time. It's a very powerful technique for synthesising certain types of sound, such as a piano, where frequencies of partials do not change. The more rapidly partial frequencies change, the more complicated it becomes to model the resulting spectrum. It's fun stuff. On Tue, Oct 23, 2018 at 2:51 PM gm wrote: > Does anybody know a real world product that uses FFT for sound synthesis? > Do you think its feasable and makes sense? > > Totally unrelated to the recent discussion here I consider replacing > (WS)OLA > granular "clouds" with a spectral synthesis and was wondering if I > should use FFT for that. > > I want to keep all the musical artefacts of the granular approach when > desired > and I was thinking that you can get the "grain cloud" sound when you add > noise to the phases/frequencies > for instance and do similar things. > > > An advantage of using FFT instead of sinusoids would be that you dont > have to worry > about partial trajectories, residual noise components and that sort of > thing. > > Whether or not it would use much less CPU I am not sure, depends on how > much overlap > of frames you have. > > Disadvantages I see is latency, even more so if you want an even workload, > and that the implementation is somewhat fuzzy/messy when you do a > timestretch followed by resampling. > > Another disadvantage would be that you cant have immediate parameter > changes > since everything is frame based, and even though some granularity is > fine for me > the granularity of FFT would be fixed to the overlap/frame size, which > is another disadvantage. > > Another disadvantage I see is the temporal blur you get when you modify > the sound. > > Any thoughs on this? Experiences? > > > > ___ > dupswapdrop: music-dsp mailing list > music-dsp@music.columbia.edu > https://lists.columbia.edu/mailman/listinfo/music-dsp > > ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
Original Message Subject: [music-dsp] FFT for realtime synthesis? From: "gm" Date: Tue, October 23, 2018 5:51 pm To: music-dsp@music.columbia.edu -- > Does anybody know a real world product that uses FFT for sound synthesis? > Do you think its feasable and makes sense? � so this first question is about synthesis, not modification for effects, right?� and real-time, correct?� so a MIDI Note-On is received and you want a note coming out as quickly as possible? i don't know of a hardware product that does inverse FFT for synthesis.� i do know of a couple of effects algorithms that go into products that use FFT.� i think it's mostly about doing "fast convolution" for purposes of reverb. what are you intending to synthesize?� notes?� or something more wild than that?� just curious. � robert -- r b-j� � � � � � � � � � � � �r...@audioimagination.com "Imagination is more important than knowledge." � � � � ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
On Wed, Oct 24, 2018 at 12:14 AM gm wrote: > Am 23.10.2018 um 23:51 schrieb gm: [...] > I think I should add that I want to use it on polyphonic material or any > source material > so sinu oscillators are probably not the way to go cuase you would need > too many of them Sine oscillators can be implemented in various very efficient ways, and more importantly here; you only need to generate one "sample" per FFT window, so you can actually have insane numbers of them. And, if you implement more complex waveforms, and maybe various "transforms" in the render-to-FFT stage, there are probably many more cycles to save. -- //David Olofson - Consultant, Developer, Artist, Open Source Advocate .--- Games, examples, libraries, scripting, sound, music, graphics ---. | http://consulting.olofson.net http://olofsonarcade.com | '-' ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
On Tue, Oct 23, 2018 at 11:51 PM gm wrote: > > Does anybody know a real world product that uses FFT for sound synthesis? I believe AIR Loom does, and I suspect some wavetable synths use FFT as well; Waves Codex and DS Audio Thorn comes to mind. (Mentions of "spectral synthesis," and there's also something about the sound character of these, especially above 10kHz...) > Do you think its feasable and makes sense? Yes, and... I'm not sure. The method is essentially a highly optimized implementation of additive synthesis - but with that comes the same problem that always plagued additive synthesis: It's not very user friendly in its basic form. You need to construct more effective and/or comprehensible synthesis algorithms on top of it. Wavetable synthesis + various filters would seem like one sensible option. Either way, you can do a whole lot of interesting stuff with relative ease, without having to worry about aliasing distortion, undesired transients and whatnot, but there are some drawbacks as well, some of which you've already mentioned. One issue that I'm hearing in most implementations of FFT based synthesis, effects and the like, is the nature of the FFT; the frequency bins. Unless you're doing very trivial processing, avoid steep filters and the like, or keep your content nicely aligned to the FFT bins, the bins will become clearly audible in the form of metallic resonances, "wobbly" filter sweeps and whatnot. Many things are easy and intuitive to do in the frequency domain, but unfortunately, the FFT can't be viewed as a proper additive synthesis oscillator bank, except in trivial cases. [...] > Whether or not it would use much less CPU I am not sure, depends on how > much overlap of frames you have. It can be extremely efficient while still producing very high quality sound, especially if you don't need very fast transient response. I'd say performance is THE reason to use this approach, rather than just doing brute force additive synthesis. [...] > Another disadvantage would be that you cant have immediate parameter changes > since everything is frame based, and even though some granularity is > fine for me > the granularity of FFT would be fixed to the overlap/frame size, which > is another disadvantage. Actually, you can, and you can even render gradients through the FFT windows, but of course, that complicates "rendering" of the FFT bins a whole lot, compared to implementing straight sine oscillators. (Which is not entirely straightforward either, as soon as your frequencies land in between FFT bins.) [...] > Any thoughs on this? Experiences? I played around a bit with it a few years ago, and hacked a quick prototype in my own odd scripting language, EEL. (Old rubbish that I've been meaning to update or port to something any century now...) https://github.com/olofson/eelsynth Simple demo song + some comments here: https://soundcloud.com/david-olofson/eelsynth-ifft-flutesong -- //David Olofson - Consultant, Developer, Artist, Open Source Advocate .--- Games, examples, libraries, scripting, sound, music, graphics ---. | http://consulting.olofson.net http://olofsonarcade.com | '-' ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
Re: [music-dsp] FFT for realtime synthesis?
Am 23.10.2018 um 23:51 schrieb gm: An advantage of using FFT instead of sinusoids would be that you dont have to worry about partial trajectories, residual noise components and that sort of thing. I think I should add that I want to use it on polyphonic material or any source material so sinu oscillators are probably not the way to go cuase you would need too many of them ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
[music-dsp] FFT for realtime synthesis?
Does anybody know a real world product that uses FFT for sound synthesis? Do you think its feasable and makes sense? Totally unrelated to the recent discussion here I consider replacing (WS)OLA granular "clouds" with a spectral synthesis and was wondering if I should use FFT for that. I want to keep all the musical artefacts of the granular approach when desired and I was thinking that you can get the "grain cloud" sound when you add noise to the phases/frequencies for instance and do similar things. An advantage of using FFT instead of sinusoids would be that you dont have to worry about partial trajectories, residual noise components and that sort of thing. Whether or not it would use much less CPU I am not sure, depends on how much overlap of frames you have. Disadvantages I see is latency, even more so if you want an even workload, and that the implementation is somewhat fuzzy/messy when you do a timestretch followed by resampling. Another disadvantage would be that you cant have immediate parameter changes since everything is frame based, and even though some granularity is fine for me the granularity of FFT would be fixed to the overlap/frame size, which is another disadvantage. Another disadvantage I see is the temporal blur you get when you modify the sound. Any thoughs on this? Experiences? ___ dupswapdrop: music-dsp mailing list music-dsp@music.columbia.edu https://lists.columbia.edu/mailman/listinfo/music-dsp
