On May 4, 2007, at 10:13 PM, Nathan Buchanan wrote:


I have a potential situation where I will have a lot of sensor data coming in very often. (every second or so) The sensor data is from physics type measurements, and will normally follow a slowly changing pattern with sinusoidal disturbances. The overall shape of the data is more important than catching high frequency disturbances.

I had the idea of taking the Fourier transform of the waveform and storing the waveform internally that way to reduce storage requirements. When I get some free cycles, I may try doing this. I would want it to be done in the database in such a way that the user can still join to a table using this internal storage scheme.

Why am I mailing this list? I'd like to ask if anyone has heard of someone doing anything like this. I did a small search of the lists and internet but didn't come up with anything. I just don't want to re-invent the wheel.

It's an interesting idea in theory (sounds like a simple low-pass filter would do most of what you want) but it doesn't sound like that great an idea in practice.

Sampling at 1Hz isn't going to generate very much data at all, unless you're capturing ridiculous amounts of data at each sample point (I have boxes averaging hundreds of pieces of data a second totaling 10s of k, along with a bunch of index updates without breaking much of an I/O sweat).

Unless you have some very tight storage constraints (embedded controllers, flash storage, ...) you're not going to see a problem with storing the actual data, and you're likely to regret both discarding the original data and expending much effort developing software low-pass pre-processing for the input signal.

(Processing it for analysis is a whole other thing, but even that will be much easier to do if you can apply variants on your algorithm to the untouched original data).


---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

Reply via email to