Re: Questions On GNU Radio FFT Data logging

[Johannes Demel]

Hi Zen Chen,

I add the mailing list to the discussion again. Please reply to the 
mailing list for all mailing list discussions.


This is an example for a UDP client/server
https://pythontic.com/modules/socket/udp-client-server-example

Your visualization app is probably a server waiting for your GNU Radio 
flowgraph (the client) to send data.


Cheers
Johannes

On 13.12.21 03:38, Zen Chen wrote:

Hi Demel,

I sort of get what you meant . How do I read the data from UDP sink ? 
That is one of my questions.


Regards,
Chen Chong Zhi

On Fri, 10 Dec 2021 at 19:49, Johannes Demel > wrote:


Hi Chong Zhi,

I assume you want to observe the FM band ~90MHz to ~100MHz is that
correct?

If you want to listen to audio, have a look at gr-rds.
https://github.com/bastibl/gr-rds 
It helps quite a bit more to understand FM.

Since this discussion seems to have started in the GR Matrix chat, I
infer you actually want to re-implement the Qt GUI Frequency Sink block
outside of GNU Radio.
Further, this needs to happen online.

As Martin mentioned, at your current sample rate a CSV is a bad choice.
If CSV is a hard requirement, sent the resulting CSV to the person
requesting it and ask them to open it. I just hope they understand the
issue after their spreadsheet visualization tool crashed/is
unresponsive
or anything else.

The file format is explained in the GR wiki, as Marcus pointed out.

https://wiki.gnuradio.org/index.php/File_Sink#What_is_the_file_format_of_a_file_sink.3F_How_can_I_read_files_produced_by_a_file_sink.3F



Let's assume you use a binary file and want to open in in Python, in
your case this would be
```
import numpy as np
samples = np.fromfile('your_file.cdat', dtype=np.complex64)
```
Please mind that the File extension (.cdat) is an arbitrary choice. In
your flowgraph, the filename is just "Yes933FFT".

The most common tool for visualization in Python is matplotlib.
However,
this is more directed at drawing a static graph. It might be a good
idea
to look into e.g. pyqtgraph, if you want to continuously update your
graph. There are probably hundreds of libraries that will cater your
needs depending on what you want to do. e.g. there are libraries that
help you create an interactive website, if you want to do that.

The Qt GUI Frequency Sink block does quite a few more things. The
processing chain looks a bit like this:
1. FFT
2. compute magnitude squared
3. moving average
4. compute dB values, i.e. 10*log10(...)
5. Discard most samples and only draw a new line at the "update rate".

You might be able to swap step 4 and 5.
It might be interesting for you to perform some, most, or all of these
steps in your GNU Radio flowgraph. This might lower your sample rate.

Besides, if you want to visualize your data "online", a file is a bad
choice for data exchange. I would expect that you end up in
"concurrency
hell".
GNU Radio offers quite a few useful options here. e.g. UDP or TCP
sinks.
Or there are ZeroMQ sinks. With Python you should be able to receive
data from these sinks with just a few lines of code.
For example, you might use a UDP sink in your flowgraph and send all
data to a UDP port on your machine. Then, your visualization tool
listens on that UDP port, receives your data and does what it is
supposed to do.

Cheers
Johannes



On 10.12.21 09:59, Zen Chen wrote:
 > Yes933_10_12_20212nd.csv
 >

>
 > HI all,My name is Zen Chen , a GNU radio Novice and I tried to
create an
 > account on the GNU Radio .org website to post my questions on the
 > mailing list however I could not . I am using GNU radio and Hack
RF 1 to
 > design a power spectrum analyser and I am using the attached
flowgraph
 > to and python script to give me the attached CSV file however , the
 > results (FFT connect to file sink) is to large to be contained in a
 > single excel file . Is there a problem in my GNU Radio Flowgraph?
 >
 > Regards,
 > Chong Zhi

Re: Questions On GNU Radio FFT Data logging

[Johannes Demel]

Hi Chong Zhi,

I assume you want to observe the FM band ~90MHz to ~100MHz is that correct?

If you want to listen to audio, have a look at gr-rds.
https://github.com/bastibl/gr-rds
It helps quite a bit more to understand FM.

Since this discussion seems to have started in the GR Matrix chat, I 
infer you actually want to re-implement the Qt GUI Frequency Sink block 
outside of GNU Radio.

Further, this needs to happen online.

As Martin mentioned, at your current sample rate a CSV is a bad choice.
If CSV is a hard requirement, sent the resulting CSV to the person 
requesting it and ask them to open it. I just hope they understand the 
issue after their spreadsheet visualization tool crashed/is unresponsive 
or anything else.


The file format is explained in the GR wiki, as Marcus pointed out.
https://wiki.gnuradio.org/index.php/File_Sink#What_is_the_file_format_of_a_file_sink.3F_How_can_I_read_files_produced_by_a_file_sink.3F

Let's assume you use a binary file and want to open in in Python, in 
your case this would be

```
import numpy as np
samples = np.fromfile('your_file.cdat', dtype=np.complex64)
```
Please mind that the File extension (.cdat) is an arbitrary choice. In 
your flowgraph, the filename is just "Yes933FFT".


The most common tool for visualization in Python is matplotlib. However, 
this is more directed at drawing a static graph. It might be a good idea 
to look into e.g. pyqtgraph, if you want to continuously update your 
graph. There are probably hundreds of libraries that will cater your 
needs depending on what you want to do. e.g. there are libraries that 
help you create an interactive website, if you want to do that.


The Qt GUI Frequency Sink block does quite a few more things. The 
processing chain looks a bit like this:

1. FFT
2. compute magnitude squared
3. moving average
4. compute dB values, i.e. 10*log10(...)
5. Discard most samples and only draw a new line at the "update rate".

You might be able to swap step 4 and 5.
It might be interesting for you to perform some, most, or all of these 
steps in your GNU Radio flowgraph. This might lower your sample rate.


Besides, if you want to visualize your data "online", a file is a bad 
choice for data exchange. I would expect that you end up in "concurrency 
hell".
GNU Radio offers quite a few useful options here. e.g. UDP or TCP sinks. 
Or there are ZeroMQ sinks. With Python you should be able to receive 
data from these sinks with just a few lines of code.
For example, you might use a UDP sink in your flowgraph and send all 
data to a UDP port on your machine. Then, your visualization tool 
listens on that UDP port, receives your data and does what it is 
supposed to do.


Cheers
Johannes



On 10.12.21 09:59, Zen Chen wrote:
Yes933_10_12_20212nd.csv 

HI all,My name is Zen Chen , a GNU radio Novice and I tried to create an 
account on the GNU Radio .org website to post my questions on the 
mailing list however I could not . I am using GNU radio and Hack RF 1 to 
design a power spectrum analyser and I am using the attached flowgraph 
to and python script to give me the attached CSV file however , the 
results (FFT connect to file sink) is to large to be contained in a 
single excel file . Is there a problem in my GNU Radio Flowgraph?


Regards,
Chong Zhi

Re: Questions On GNU Radio FFT Data logging

[Martin Luelf]

Dear Zen Chen,

to subscribe to the mailing list you need to follow this link: 
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

I think the account on the GNURadio website is for editing the Wiki etc.

You are sampling your signal at 10 MSamples per second. Since you use a 
complex signal (blue input to the file sink) that means each sample is 
two 32 bit floating point numbers). Thus your generated binary file has 
a size of 80 MByte per second of recording.


If you convert that into a CSV file the default formater for 
numpy.savetxt() is .18e, meaning that each real and imaginary part will 
be written as 24 characters (one place before the decimal, the decimal 
point, the 18 places after the decimal and 4 chars for the exponential). 
These need to be delimited by a comma and finally a new line character. 
That makes 50 characters per measurement. For a regular ASCII file each 
char is 1 Byte so you are writing 50 Byte/sample * 10 MSample/s = 500 
MByte per second of recording.


That is a lot of data. For a 5 second recording for example you are 
asking Excel to import a 5GB file (!). I don't know Excel too much but 
that sounds like it might be to much. Possible remedies are: try to 
reduce the sampling rate (which will reduce the monitored bandwidth) or 
try to record very short bursts, or (my personal opinion) try to analyze 
your signal in a program that is more suitable to signal analysis than 
Excel. You most definitely want something that can work directly with 
the binary files from GNURadio's file sink rather than having to use a 
CSV file. GNURadio itself, python/numpy, Matlab/octave, C++ are all good 
choices for the analysis (and certainly many more).


Yours
Martin



On 10.12.21 09:59, Zen Chen wrote:
Yes933_10_12_20212nd.csv 

HI all,My name is Zen Chen , a GNU radio Novice and I tried to create an 
account on the GNU Radio .org website to post my questions on the 
mailing list however I could not . I am using GNU radio and Hack RF 1 to 
design a power spectrum analyser and I am using the attached flowgraph 
to and python script to give me the attached CSV file however , the 
results (FFT connect to file sink) is to large to be contained in a 
single excel file . Is there a problem in my GNU Radio Flowgraph?


Regards,
Chong Zhi

Questions On GNU Radio FFT Data logging

[Zen Chen]

 Yes933_10_12_20212nd.csv

HI all, My name is Zen Chen , a GNU radio Novice and I tried to create an
account on the GNU Radio .org website to post my questions on the mailing
list however I could not . I am using GNU radio and Hack RF 1 to design a
power spectrum analyser and I am using the attached flowgraph to and python
script to give me the attached CSV file however , the results (FFT connect
to file sink) is to large to be contained in a single excel file . Is there
a problem in my GNU Radio Flowgraph?

Regards,
Chong Zhi
import numpy
import time 


 f = numpy.fromfile(open(r"C:\Users\zench\Documents\Yes933FFT"),dtype=numpy.complex64)

  numpy.savetxt(r"C:\Users\zench\Documents\Yes933_10_12_2021.csv", f, delimiter=",")


Yes_933FMBackup3_29 Nov.pdf
Description: Adobe PDF document