Re: [Discuss-gnuradio] state of offline signal analysis tools in 2014?
Thanks for the recommendations. I should clarify that I am a software engineer, not a signals engineer, and my recurring need to visualize time-series data is often satisfied without having to invoke DSP. An example of the sort of thing I frequently want to do is to interactively explore data collected from multiple wireless sensors (e.g., six months of temperature and humidity data collected at 1-minute intervals). Technically signals, but simple shifts, scales, and basic windowed statistics are more useful than FFTs and complex filters. Sometimes I do need DSP techniques to extract the data from third-party wireless transmissions, hence my current dabbling with GNU Radio, but it's the data not the extraction process that's the primary focus. For example, the reason I'm using GNU Radio is my need to demodulate packets from a WS-2080 Weather Station and some other 433 MHz OOK sensors. Yes, I've tried rtl433 and rtlsdr-433m-sensor; the signal I'm capturing is too noisy or I don't have frequency/bandwidth/filter settings right. The specific use case that introduced my question about analysis tools is my desire to interactively identify regions of interest from wideband captures and then re-play that data repeatedly through various processing chains, tweaking parameters until I get something that reliably produces the underlying data. @mossman: The GSoC project is very close to what I'd want for DSP-oriented analysis. If it existed it'd probably handle the motivating example above. It's too domain-specific for generalized time-series visualization, though, so I'd still have an unmet need. @marcus.mueller: Thanks for the details. My experience is that a metered stream/dataflow-oriented architecture is simply unsuited to the sort of offline analysis I'm trying to do. The ability to jump forwards and backwards in time is crucial, as is the ability to decouple the signal rate from the data processing rate. Example: Qt Time Raster schedules its updates based on wall clock not sample time, so speeding up or slowing down the rate of data through the system changes the displayed images, and running unthrottled drops all the information. @dan.cajacob: pandas reminds me a lot of R. The intro video showed it'd be good for CLI-based data manipulation, but my initial need is to explore data graphically. @mdammer: kst-plot's web site shows some promising graphs, and it built cleanly (though it doesn't obey CMAKE_INSTALL_PREFIX), but I've been unable to locate examples and documentation that would allow me to evaluate its capabilities quickly. @madengr: Lack of source code for baudline and its focus on DSP knocks it out of contention for my general needs. On 07/16/2014 09:52 AM, Peter A. Bigot wrote: The sort of capabilities I'm looking for include: Read time-series data from files of different formats (some too large to fit in physical memory). Display the data, optionally applying linear transformations. Interactively pan and zoom. Jump forwards and backwards among time-registered events. Enable/disable/time-shift data overlays. Export selected data to new files. Calculate and display statistics and other non-linear transformations of selected data. A rough course towards the tool I imagine would be: * Collect/develop a suite of Qt/C++ widgets for graphical data display and manipulation that are not sensitive to the processing rate, don't have a unidirectional concept of time, can be accessed from Python, and can be combined to build something with the capabilities listed above. * Use a command-line interface like pandas/R to display original files, extract regions of interest, apply transformations, and repeat until satori. * Use GNU Radio Companion to glue components together to form domain-specific analysis applications. That's a lot of yak shaving just to get a reliable OOK packet extractor, so it probably won't happen. Thanks again for the suggestions. Peter ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] state of offline signal analysis tools in 2014?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Peter, You might have a look at Python with Numpy/Scipy/matplotlib. Processed data according to your needs may be observed interactively with matplotlib and numpy/scipy do a good job at offline analysis. You'd have to process your data again and again anyway whenever you want to check something else. happy hacking Johannes On 17.07.2014 11:52, Peter A. Bigot wrote: Thanks for the recommendations. I should clarify that I am a software engineer, not a signals engineer, and my recurring need to visualize time-series data is often satisfied without having to invoke DSP. An example of the sort of thing I frequently want to do is to interactively explore data collected from multiple wireless sensors (e.g., six months of temperature and humidity data collected at 1-minute intervals). Technically signals, but simple shifts, scales, and basic windowed statistics are more useful than FFTs and complex filters. Sometimes I do need DSP techniques to extract the data from third-party wireless transmissions, hence my current dabbling with GNU Radio, but it's the data not the extraction process that's the primary focus. For example, the reason I'm using GNU Radio is my need to demodulate packets from a WS-2080 Weather Station and some other 433 MHz OOK sensors. Yes, I've tried rtl433 and rtlsdr-433m-sensor; the signal I'm capturing is too noisy or I don't have frequency/bandwidth/filter settings right. The specific use case that introduced my question about analysis tools is my desire to interactively identify regions of interest from wideband captures and then re-play that data repeatedly through various processing chains, tweaking parameters until I get something that reliably produces the underlying data. @mossman: The GSoC project is very close to what I'd want for DSP-oriented analysis. If it existed it'd probably handle the motivating example above. It's too domain-specific for generalized time-series visualization, though, so I'd still have an unmet need. @marcus.mueller: Thanks for the details. My experience is that a metered stream/dataflow-oriented architecture is simply unsuited to the sort of offline analysis I'm trying to do. The ability to jump forwards and backwards in time is crucial, as is the ability to decouple the signal rate from the data processing rate. Example: Qt Time Raster schedules its updates based on wall clock not sample time, so speeding up or slowing down the rate of data through the system changes the displayed images, and running unthrottled drops all the information. @dan.cajacob: pandas reminds me a lot of R. The intro video showed it'd be good for CLI-based data manipulation, but my initial need is to explore data graphically. @mdammer: kst-plot's web site shows some promising graphs, and it built cleanly (though it doesn't obey CMAKE_INSTALL_PREFIX), but I've been unable to locate examples and documentation that would allow me to evaluate its capabilities quickly. @madengr: Lack of source code for baudline and its focus on DSP knocks it out of contention for my general needs. On 07/16/2014 09:52 AM, Peter A. Bigot wrote: The sort of capabilities I'm looking for include: Read time-series data from files of different formats (some too large to fit in physical memory). Display the data, optionally applying linear transformations. Interactively pan and zoom. Jump forwards and backwards among time-registered events. Enable/disable/time-shift data overlays. Export selected data to new files. Calculate and display statistics and other non-linear transformations of selected data. A rough course towards the tool I imagine would be: * Collect/develop a suite of Qt/C++ widgets for graphical data display and manipulation that are not sensitive to the processing rate, don't have a unidirectional concept of time, can be accessed from Python, and can be combined to build something with the capabilities listed above. * Use a command-line interface like pandas/R to display original files, extract regions of interest, apply transformations, and repeat until satori. * Use GNU Radio Companion to glue components together to form domain-specific analysis applications. That's a lot of yak shaving just to get a reliable OOK packet extractor, so it probably won't happen. Thanks again for the suggestions. Peter ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJTx6dJAAoJEO7fmkDsqywMQSgQAIEM6ujO7XUGXlKSiH0EFBAt 7zUBbERcjIdNCv4/15kuH/sjNTfk9tUhu5aIK8M3FyRKyC3kIBkDLJBiZhMdxQ14 sHldxKpt2kkKW6wYXLHfAV2qqauWM1lTfB9VYwjbtj8Wej55OwA6JUR6jtHS8F3O yZ+0obljFsChLSV8Y2DEHl66Of9SnyUciKIhBCt6MF4v48WeVwbkSeo04Ff31abE
Re: [Discuss-gnuradio] state of offline signal analysis tools in 2014?
On 7/16/14 10:52 AM, Peter A. Bigot wrote: GNU Radio is a great tool for applications and dynamic experimentation, but it doesn't have a lot of support for static/offline analysis of time-series data. I.e. I've captured some signal data and I want to explore its properties interactively so I can figure out what I want to do with it in GNU Radio. The sort of capabilities I'm looking for include: Read time-series data from files of different formats (some too large to fit in physical memory). Display the data, optionally applying linear transformations. Interactively pan and zoom. Jump forwards and backwards among time-registered events. Enable/disable/time-shift data overlays. Export selected data to new files. Calculate and display statistics and other non-linear transformations of selected data. If you have access to a Mac, DataGraph has much of this. We use it a lot to interactively manipulate large quantities of logged data. http://www.visualdatatools.com/DataGraph/ @(^.^)@ Ed ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] state of offline signal analysis tools in 2014?
On Wed, Jul 16, 2014 at 09:52:43AM -0500, Peter A. Bigot wrote: Is any such framework available now or in development? If not, is anybody interested in joining me offline to discuss the requirements and design for such a thing? Something like this very nearly happened as a GSoC project this year. See: http://gnuradio.org/redmine/projects/gnuradio/wiki/GSoC#Advanced-Signal-Visualization-and-Analysis Jared Boone has some preliminary work in Python, mostly on packet detection and demodulation: https://github.com/jboone/tpms Mike ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] state of offline signal analysis tools in 2014?
Hi Peter,
GNU Radio is based very much on the idea of a data stream, so it might
not actually be the tool of choice for static analysis.
However, there is quite a lot which can be done with on-board tools, so
let me comment in-text.
On 16.07.2014 16:52, Peter A. Bigot wrote:
GNU Radio is a great tool for applications and dynamic
experimentation, but it doesn't have a lot of support for
static/offline analysis of time-series data. I.e. I've captured some
signal data and I want to explore its properties interactively so I
can figure out what I want to do with it in GNU Radio.
The sort of capabilities I'm looking for include:
Read time-series data from files of different formats (some too large
to fit in physical memory).
So far, only raw samples in machine float format are supported, and the
GNU Radio-specific metadata/samples interleaved format along with the
Wav audio file format. As far as I know, all of the sources/sinks for
these file formats don't need to store data in RAM but read/write it
sequentially.
If you feel like there is something obviously missing in this list, you
could just use the awesome powers of python and/or C++ to read your
favourite file format, write a database adapter or a twitter source; the
reason we don't have things like native CSV or HDF support is that I
guess noone cared to implement a source for these formats, because they
don't lend themselves to streaming very well, and not because it's hard.
Anyway, there's a series of small tools for sample files, called
gr_{plot,plot_fft,spectrogram}_type, that at least allow you to
visualize recorded data easily, included in GNU Radio.
Display the data, optionally applying linear transformations.
Well the problem here is that our visual sinks usually want to
periodically update the display, and that GNU Radio flow graphs usually
terminate when sources have finished producing items (e.g. when the
source file has been read completely). Many of these issues can be
worked around be setting your file source to repeat and pausing the
graphical sink when you see something interesting, after throttling your
item flow enough to make the signal observable by the naked eye.
The linear transformation thing is something you'd have to implement in
a DSPish manner, and most probably can be done.
Interactively pan and zoom.
Most of the graphical sinks can do that
Jump forwards and backwards among time-registered events.
Nope, I'm afraid that won't work with the stream-oriented architecture
of GNU Radio.
Enable/disable/time-shift data overlays.
Again, if you feed a graphical sink with a signal and a time-shifted
version, you get a DSPified method of doing your visualization
Export selected data to new files.
not really available (yet?).
Calculate and display statistics and other non-linear transformations
of selected data.
Depends. Again, if you can translate your statistics to a signal
processing algorithm, then it's almost certainly already been done or is
easy to do.
Ideally I'd like an open-source analysis framework that I can extend
in Python or C++; something like the Midas DSP tool family.
Not aware of these, sorry, and google turned up some defense program
along with large audio mixers. Do you have a URL to refer to?
I'm aware of some Qt widgets like QtCustomPlot, and generic frameworks
like matplotlib and octave, but not of any ready-to-use applications
or frameworks that already provide the basic functionality described
above.
I think you should take a look at things like R, GNUplot etc.
Anyway, this is a very interesting topic, and I would really enjoy
hearing from cool software that does what you describes in a manner that
could e.g. be explained to EE first-semester students or so.
The keywords I've tossed at Google haven't produced any obvious
solutions, and discussions I find in the archives here are a couple
years old and seem to summarize as use maplotlib/octave.
I'm afraid my 2014 reply will disappoint you a little... it's if you
know what characteristics you're looking for, go for a few lines of
python; if you don't know, go for python and some additional lines.
Actually, I've grown so used to numpy/scipy/matplotlib/pyqtgraph that I
wouldn't trade it for Matlab (I have access to that and rarely use it),
especially because python is something I would consider a real language
whereas the matlab syntax and the matlab interpreter performance...
well, matlab has fantastic documentation.
Is any such framework available now or in development? If not, is
anybody interested in joining me offline to discuss the requirements
and design for such a thing?
Count me in, as this is relevant to my work.
Greetings,
Marcus
Thanks.
Peter
___
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
___
Discuss-gnuradio mailing list
Re: [Discuss-gnuradio] state of offline signal analysis tools in 2014?
I've never used it for RF work, but pandas is a very powerful framework for
working with timeseries and multi-dimensional data.
Very Respectfully,
Dan CaJacob
On Wed, Jul 16, 2014 at 11:20 AM, Marcus Müller marcus.muel...@ettus.com
wrote:
Hi Peter,
GNU Radio is based very much on the idea of a data stream, so it might
not actually be the tool of choice for static analysis.
However, there is quite a lot which can be done with on-board tools, so
let me comment in-text.
On 16.07.2014 16:52, Peter A. Bigot wrote:
GNU Radio is a great tool for applications and dynamic
experimentation, but it doesn't have a lot of support for
static/offline analysis of time-series data. I.e. I've captured some
signal data and I want to explore its properties interactively so I
can figure out what I want to do with it in GNU Radio.
The sort of capabilities I'm looking for include:
Read time-series data from files of different formats (some too large
to fit in physical memory).
So far, only raw samples in machine float format are supported, and the
GNU Radio-specific metadata/samples interleaved format along with the
Wav audio file format. As far as I know, all of the sources/sinks for
these file formats don't need to store data in RAM but read/write it
sequentially.
If you feel like there is something obviously missing in this list, you
could just use the awesome powers of python and/or C++ to read your
favourite file format, write a database adapter or a twitter source; the
reason we don't have things like native CSV or HDF support is that I
guess noone cared to implement a source for these formats, because they
don't lend themselves to streaming very well, and not because it's hard.
Anyway, there's a series of small tools for sample files, called
gr_{plot,plot_fft,spectrogram}_type, that at least allow you to
visualize recorded data easily, included in GNU Radio.
Display the data, optionally applying linear transformations.
Well the problem here is that our visual sinks usually want to
periodically update the display, and that GNU Radio flow graphs usually
terminate when sources have finished producing items (e.g. when the
source file has been read completely). Many of these issues can be
worked around be setting your file source to repeat and pausing the
graphical sink when you see something interesting, after throttling your
item flow enough to make the signal observable by the naked eye.
The linear transformation thing is something you'd have to implement in
a DSPish manner, and most probably can be done.
Interactively pan and zoom.
Most of the graphical sinks can do that
Jump forwards and backwards among time-registered events.
Nope, I'm afraid that won't work with the stream-oriented architecture
of GNU Radio.
Enable/disable/time-shift data overlays.
Again, if you feed a graphical sink with a signal and a time-shifted
version, you get a DSPified method of doing your visualization
Export selected data to new files.
not really available (yet?).
Calculate and display statistics and other non-linear transformations
of selected data.
Depends. Again, if you can translate your statistics to a signal
processing algorithm, then it's almost certainly already been done or is
easy to do.
Ideally I'd like an open-source analysis framework that I can extend
in Python or C++; something like the Midas DSP tool family.
Not aware of these, sorry, and google turned up some defense program
along with large audio mixers. Do you have a URL to refer to?
I'm aware of some Qt widgets like QtCustomPlot, and generic frameworks
like matplotlib and octave, but not of any ready-to-use applications
or frameworks that already provide the basic functionality described
above.
I think you should take a look at things like R, GNUplot etc.
Anyway, this is a very interesting topic, and I would really enjoy
hearing from cool software that does what you describes in a manner that
could e.g. be explained to EE first-semester students or so.
The keywords I've tossed at Google haven't produced any obvious
solutions, and discussions I find in the archives here are a couple
years old and seem to summarize as use maplotlib/octave.
I'm afraid my 2014 reply will disappoint you a little... it's if you
know what characteristics you're looking for, go for a few lines of
python; if you don't know, go for python and some additional lines.
Actually, I've grown so used to numpy/scipy/matplotlib/pyqtgraph that I
wouldn't trade it for Matlab (I have access to that and rarely use it),
especially because python is something I would consider a real language
whereas the matlab syntax and the matlab interpreter performance...
well, matlab has fantastic documentation.
Is any such framework available now or in development? If not, is
anybody interested in joining me offline to discuss the requirements
and design for such a thing?
Count me in, as this is
Re: [Discuss-gnuradio] state of offline signal analysis tools in 2014?
Two suggestions from my side: If you want to use Python, you can use the Spyder IDE (https://code.google.com/p/spyderlib/). Spyder is mainly designed for scientific programming. It even has built-in plotting capabilities. Another package I have used for that purpose is kst-plot. It is fast and has a lot of features: http://kst-plot.kde.org The good thing is that KST unlike many other KDE applications does not rely on a full KDE installation, but uses just QT libraries. There is a PPA for Ubuntu users: https://launchpad.net/~kst-plot/+archive/ubuntu/ppa Mark On 16/07/14 15:52, Peter A. Bigot wrote: GNU Radio is a great tool for applications and dynamic experimentation, but it doesn't have a lot of support for static/offline analysis of time-series data. I.e. I've captured some signal data and I want to explore its properties interactively so I can figure out what I want to do with it in GNU Radio. The sort of capabilities I'm looking for include: Read time-series data from files of different formats (some too large to fit in physical memory). Display the data, optionally applying linear transformations. Interactively pan and zoom. Jump forwards and backwards among time-registered events. Enable/disable/time-shift data overlays. Export selected data to new files. Calculate and display statistics and other non-linear transformations of selected data. Ideally I'd like an open-source analysis framework that I can extend in Python or C++; something like the Midas DSP tool family. I'm aware of some Qt widgets like QtCustomPlot, and generic frameworks like matplotlib and octave, but not of any ready-to-use applications or frameworks that already provide the basic functionality described above. The keywords I've tossed at Google haven't produced any obvious solutions, and discussions I find in the archives here are a couple years old and seem to summarize as use maplotlib/octave. Is any such framework available now or in development? If not, is anybody interested in joining me offline to discuss the requirements and design for such a thing? Thanks. Peter ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Re: [Discuss-gnuradio] state of offline signal analysis tools in 2014?
Peter, Have you tried Baudline? It's not open source, but it is free, and will do allot of what you want. Looks like there will be a new version soon with 64-bit support and I/Q demod. It's also quite easy to pipe GR file sink to it via a FIFO (gr-baz has a sink for just that), but it will also do off-line analysis also. http://www.baudline.com/ Thanks, Lou KD4HSO Peter A. Bigot wrote The sort of capabilities I'm looking for include: Read time-series data from files of different formats (some too large to fit in physical memory). Display the data, optionally applying linear transformations. Interactively pan and zoom. Jump forwards and backwards among time-registered events. Enable/disable/time-shift data overlays. Export selected data to new files. Calculate and display statistics and other non-linear transformations of selected data. -- View this message in context: http://gnuradio.4.n7.nabble.com/state-of-offline-signal-analysis-tools-in-2014-tp49376p49385.html Sent from the GnuRadio mailing list archive at Nabble.com. ___ Discuss-gnuradio mailing list Discuss-gnuradio@gnu.org https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
