Re: Rational resample before FFT, and FFT rate
Okey-doke! Good luck! Gary On 5/3/24 10:54, John Ackermann N8UR wrote: (Off list) Hi Gary -- I didn't want you to think I've given up on this. I've decided that it might be simpler just to use Python and am working through the really excellent PySDR tutorials to get started building the program up a step at a time. Those appear to use numpy for all the processing. I don't know if I'll need to get into scipy for some of the fancier stuff. Thanks for your help! John On 4/30/24 21:40, Gary Schafer wrote: "I do understand the relationships between sample rate, FFT depth, time resolution, and frequency resolution. I used 512 samples/512 bins/1 second as a simple test case for the problem that I'm not getting one data point per frame, regardless of the frame rate." Sorry about my misunderstanding. I recreated a portion of your flowgraph just to see what it would do. I left the 16 kHz sample rate but with a 2^15 time record size. Once I ran the flowgraph, it was 17.5 seconds before the Number Sink updated, and 34 seconds before the spectral display (vector sink) updated. However, after that time, each update roughly twice per second. Is that different from what you're seeing? Gary On 4/30/24 20:17, John Ackermann N8UR wrote: On 4/30/24 18:35, Gary Schafer wrote: "I need data points at convenient intervals for time series plotting, e.g., 512 samples/second going into a 512 bin FFT to provide one maximum amplitude value per second." Let me answer that one at the same time as "Ultimately I want to plot both the amplitude and any frequency change to sub-Hz resolution." Okay, you've hit a contradiction. You can NOT get "sub-Hz" resolution using a sample rate, in Hz, that equals your time record size, in samples. Hi Gary -- sorry if my message was confusing. I do understand the relationships between sample rate, FFT depth, time resolution, and frequency resolution. I used 512 samples/512 bins/1 second as a simple test case for the problem that I'm not getting one data point per frame, regardless of the frame rate. Given how noisy this data is, I may well end up with something like one frame per 10 seconds, but before I can play with that I need to understand what is wrong with my processing to get the amplitude of the maximum bin per frame. John
Re: Rational resample before FFT, and FFT rate
On 4/30/24 21:40, Gary Schafer wrote: Sorry about my misunderstanding. I recreated a portion of your flowgraph just to see what it would do. I left the 16 kHz sample rate but with a 2^15 time record size. Once I ran the flowgraph, it was 17.5 seconds before the Number Sink updated, and 34 seconds before the spectral display (vector sink) updated. However, after that time, each update roughly twice per second. Is that different from what you're seeing? I disabled the rational resample so I'm using 500 samples/sec and 500 bins rather than 512/512, and that has broken the vector sink,. It's getting late here so let me experiment more tomorrow and get back with some results. Thanks! John
Re: Rational resample before FFT, and FFT rate
"I do understand the relationships between sample rate, FFT depth, time resolution, and frequency resolution. I used 512 samples/512 bins/1 second as a simple test case for the problem that I'm not getting one data point per frame, regardless of the frame rate." Sorry about my misunderstanding. I recreated a portion of your flowgraph just to see what it would do. I left the 16 kHz sample rate but with a 2^15 time record size. Once I ran the flowgraph, it was 17.5 seconds before the Number Sink updated, and 34 seconds before the spectral display (vector sink) updated. However, after that time, each update roughly twice per second. Is that different from what you're seeing? Gary On 4/30/24 20:17, John Ackermann N8UR wrote: On 4/30/24 18:35, Gary Schafer wrote: "I need data points at convenient intervals for time series plotting, e.g., 512 samples/second going into a 512 bin FFT to provide one maximum amplitude value per second." Let me answer that one at the same time as "Ultimately I want to plot both the amplitude and any frequency change to sub-Hz resolution." Okay, you've hit a contradiction. You can NOT get "sub-Hz" resolution using a sample rate, in Hz, that equals your time record size, in samples. Hi Gary -- sorry if my message was confusing. I do understand the relationships between sample rate, FFT depth, time resolution, and frequency resolution. I used 512 samples/512 bins/1 second as a simple test case for the problem that I'm not getting one data point per frame, regardless of the frame rate. Given how noisy this data is, I may well end up with something like one frame per 10 seconds, but before I can play with that I need to understand what is wrong with my processing to get the amplitude of the maximum bin per frame. John ackermanTest.pdf Description: Adobe PDF document
Re: Rational resample before FFT, and FFT rate
On 4/30/24 18:35, Gary Schafer wrote: "I need data points at convenient intervals for time series plotting, e.g., 512 samples/second going into a 512 bin FFT to provide one maximum amplitude value per second." Let me answer that one at the same time as "Ultimately I want to plot both the amplitude and any frequency change to sub-Hz resolution." Okay, you've hit a contradiction. You can NOT get "sub-Hz" resolution using a sample rate, in Hz, that equals your time record size, in samples. Hi Gary -- sorry if my message was confusing. I do understand the relationships between sample rate, FFT depth, time resolution, and frequency resolution. I used 512 samples/512 bins/1 second as a simple test case for the problem that I'm not getting one data point per frame, regardless of the frame rate. Given how noisy this data is, I may well end up with something like one frame per 10 seconds, but before I can play with that I need to understand what is wrong with my processing to get the amplitude of the maximum bin per frame. John
Re: Rational resample before FFT, and FFT rate
Let me answer some of your questions in reverse order: "My main question is why the displayed maximum bin value is updating much more slowly than once per second when FFT size = sample rate." That could be due to an issue in Gnu Radio. I've noticed that if the bin width (sample rate / N) is close to or less than 1 Hz, my flowgraphs will either crash or just stop running (freeze up). Perhaps that's an issue with this. I'd love to hear someone more knowledgeable than I on Gnu Radio chime in. Otherwise, you may have to resort to Python (or something similar) programming. "I need data points at convenient intervals for time series plotting, e.g., 512 samples/second going into a 512 bin FFT to provide one maximum amplitude value per second." Let me answer that one at the same time as "Ultimately I want to plot both the amplitude and any frequency change to sub-Hz resolution." Okay, you've hit a contradiction. You can NOT get "sub-Hz" resolution using a sample rate, in Hz, that equals your time record size, in samples. The most common measure of resolution for FFTs is the equivalent noise bandwidth (ENBW), and RBW = fs*B/N, where fs = sample rate, B = NENBW (normalized equivalent noise bandwidth) of the window used, and N = number of samples in the time record fed into the FFT. You can achieve sub-Hz *binwidth* by zero-padding the data, but that just really smooths out the spectral data. It doesn't really give you "sub-Hz resolution". Thinking it through, the ONLY way I can see you getting your sub-Hz resolution AND maintain a data point collection of 1 sample / second is to use overlapping FFTs. For example, see https://karc.ca/sites/default/files/EUCARA-2018%20-%20EUCARA2018_Dwingeloo_goes_SDR.pdf. Pe1nut (aka Paul Boven, the author of that presentation) is truly gifted in using Gnu Radio to do some great things in radio astronomy. That would require you to create multiple flows for your samples that allow for the data to overlap between FFTs. With four overlapping FFTs, for example, and with a window type that has a relatively small NENBW (Hamming, for example), you could get a resolution of roughly 0.33 Hz. Make sense? Gary On 4/29/24 22:18, John Ackermann N8UR wrote: Hi Gary -- Thanks for getting back to me. Maybe I can explain better what I'm doing. The data is 16 ksps complex IQ centered at 50.080 MHz. The target signal is an essentially unmodulated CW carrier about 427 below that. I have about 8 hours of recorded data to analyze. Ultimately I want to plot both the amplitude and any frequency change to sub-Hz resolution. To get sufficient resolution I decimate down to a bandwidth of only a few hundred Hz. That would put the target signal outside the passband, so the frequency translation first shifts it to the center. I need data points at convenient intervals for time series plotting, e.g., 512 samples/second going into a 512 bin FFT to provide one maximum amplitude value per second. I resampled so I could get that one-frame-per-second pacing with a power-of-two FFT size. If that's not a real concern, I can happily remove the resampling and work with the natural sample rate, e.g., 500 bin FFT fed by 500 samples/second. My main question is why the displayed maximum bin value is updating much more slowly than once per second when FFT size = sample rate. It's more like one per five seconds. In particular, am I doing something wrong with the stream and vector conversions around the FFT and MAX blocks? Thanks, John On 4/29/24 19:06, Gary Schafer wrote: It sounds to me as if you're trying to move the signal to the center of a FFT bin so that you don't have to deal with scalloping loss. Is that correct? If so, I strongly recommend that you skip all of this resampling and just use a "flattop" window in the FFT. That will allow you to measure the maximum amplitude with very high accuracy (roughly within 0.01 dB) regardless of where the signal resides within the spectrum. If that's not satisfactory, then I recommend that you skip the "having a sample rate that is a power-of-2 Hz" and just move the signal to the center of a frequency bin with whatever sample rate you have. The bin frequencies are simply (sample rate)/N, where N is the number of samples in the FFT. So with a sample rate of 16 kHz and a 512 pt FFT, you'd select a frequency that was an integer value of 16000/512 = 31.25 Hz. So, if you want to move it to, say, the 10th bin, move it to 31.25*10 = 312.5 Hz. Done. Good luck! Gary I am reading Digital RF data at 16k samples/second, and my goal is to get the power of the maximum frequency once per second. I start by resampling to a power-of-two rate, then translating to move the desired frequency to the center with further decimation, then doing an FFT, converting to log power and finally extracting the power of the loudest bin. The decimation and FFT size are calculated to yield one FFT
Re: Rational resample before FFT, and FFT rate
Hi Gary -- Thanks for getting back to me. Maybe I can explain better what I'm doing. The data is 16 ksps complex IQ centered at 50.080 MHz. The target signal is an essentially unmodulated CW carrier about 427 below that. I have about 8 hours of recorded data to analyze. Ultimately I want to plot both the amplitude and any frequency change to sub-Hz resolution. To get sufficient resolution I decimate down to a bandwidth of only a few hundred Hz. That would put the target signal outside the passband, so the frequency translation first shifts it to the center. I need data points at convenient intervals for time series plotting, e.g., 512 samples/second going into a 512 bin FFT to provide one maximum amplitude value per second. I resampled so I could get that one-frame-per-second pacing with a power-of-two FFT size. If that's not a real concern, I can happily remove the resampling and work with the natural sample rate, e.g., 500 bin FFT fed by 500 samples/second. My main question is why the displayed maximum bin value is updating much more slowly than once per second when FFT size = sample rate. It's more like one per five seconds. In particular, am I doing something wrong with the stream and vector conversions around the FFT and MAX blocks? Thanks, John On 4/29/24 19:06, Gary Schafer wrote: It sounds to me as if you're trying to move the signal to the center of a FFT bin so that you don't have to deal with scalloping loss. Is that correct? If so, I strongly recommend that you skip all of this resampling and just use a "flattop" window in the FFT. That will allow you to measure the maximum amplitude with very high accuracy (roughly within 0.01 dB) regardless of where the signal resides within the spectrum. If that's not satisfactory, then I recommend that you skip the "having a sample rate that is a power-of-2 Hz" and just move the signal to the center of a frequency bin with whatever sample rate you have. The bin frequencies are simply (sample rate)/N, where N is the number of samples in the FFT. So with a sample rate of 16 kHz and a 512 pt FFT, you'd select a frequency that was an integer value of 16000/512 = 31.25 Hz. So, if you want to move it to, say, the 10th bin, move it to 31.25*10 = 312.5 Hz. Done. Good luck! Gary I am reading Digital RF data at 16k samples/second, and my goal is to get the power of the maximum frequency once per second. I start by resampling to a power-of-two rate, then translating to move the desired frequency to the center with further decimation, then doing an FFT, converting to log power and finally extracting the power of the loudest bin. The decimation and FFT size are calculated to yield one FFT per second. I'm not sure if I'm doing this correctly and I've attached the flowgraph. (a) Should the FFT use only a power-of-two bin size? I am resampling to go from 16k to 8192 samples/second so that I ultimately decimate to 512 samples/second rather than 500 samples/second. Is that the right thing to do? Should I resample up to 16384 samples/second rather than down to 8192? Or should I just use a 500 bin FFT? (b) Given that the final sample rate and FFT depth are equal (512), I expected to get one vector per second, and thus one maximum value per second. Instead, I see a much slower update rate, about once per five seconds. I added a QT vector sink and while that takes a long time to get started, once going it does update about once per second. Am I doing something wrong that the maximum value doesn't update with each new vector? Thanks, John
Re: Rational resample before FFT, and FFT rate
It sounds to me as if you're trying to move the signal to the center of a FFT bin so that you don't have to deal with scalloping loss. Is that correct? If so, I strongly recommend that you skip all of this resampling and just use a "flattop" window in the FFT. That will allow you to measure the maximum amplitude with very high accuracy (roughly within 0.01 dB) regardless of where the signal resides within the spectrum. If that's not satisfactory, then I recommend that you skip the "having a sample rate that is a power-of-2 Hz" and just move the signal to the center of a frequency bin with whatever sample rate you have. The bin frequencies are simply (sample rate)/N, where N is the number of samples in the FFT. So with a sample rate of 16 kHz and a 512 pt FFT, you'd select a frequency that was an integer value of 16000/512 = 31.25 Hz. So, if you want to move it to, say, the 10th bin, move it to 31.25*10 = 312.5 Hz. Done. Good luck! Gary I am reading Digital RF data at 16k samples/second, and my goal is to get the power of the maximum frequency once per second. I start by resampling to a power-of-two rate, then translating to move the desired frequency to the center with further decimation, then doing an FFT, converting to log power and finally extracting the power of the loudest bin. The decimation and FFT size are calculated to yield one FFT per second. I'm not sure if I'm doing this correctly and I've attached the flowgraph. (a) Should the FFT use only a power-of-two bin size? I am resampling to go from 16k to 8192 samples/second so that I ultimately decimate to 512 samples/second rather than 500 samples/second. Is that the right thing to do? Should I resample up to 16384 samples/second rather than down to 8192? Or should I just use a 500 bin FFT? (b) Given that the final sample rate and FFT depth are equal (512), I expected to get one vector per second, and thus one maximum value per second. Instead, I see a much slower update rate, about once per five seconds. I added a QT vector sink and while that takes a long time to get started, once going it does update about once per second. Am I doing something wrong that the maximum value doesn't update with each new vector? Thanks, John
