I sent in a patch here on the mailing list a few days ago on the
Spectrometer plugin. Between then and now I found another bug in the
old spectrometer code, so I fixed it up with this patch.
Screenshot of Spectrometer Plugin:
http://img295.imageshack.us/my.php?image=spectrogram3uj2.png
For those unfamiliar with the Spectrometer, a spectrometer is one way
to visualize sound. It is a 3D plot, the vertical axis represents
frequency, the horizontal axis represents time. The color changes
represent louder/softer. In the image above, you can see 11 piano
notes in the 500Hz to 1000Hz range (~A to high A) and then the
beginning of the vocal section. For those who have played the video
game "Rock Band", in Rock Band can be thought of as a simple
Spectrometer.
Here are a list of changes I made compared to the latest subversion
version (revision 1060):
1. The "STEP" constant was added to control the zoom of the
spectrometer. By default, the spectrometer in this patch is zoomed in
10x closer on the X Axis than previously. To help curb the
inefficiency of this zoom level, a buffer was added to the
SpectrogramFFT class.
2. The "color" value of the spectrometer is now based on a HSV color
model. White and Red are the strongest values, while black is the
weakest value. In between, the colors change according to Hue,
Saturation, and Value. Ultimately, the color scheme is more logical
and makes the spectrogram look significantly cleaner and more usable
compared to the old spectrogram. Further, this color value is based on
decibels as opposed to being proportional to the sound. This allows
the user to see a greater range of loudness in the sound. (This change
required the most code, with 4 new functions (to_hue, to_brightness,
to_saturation, and to_rgb)
3. The frequency markers on the left hand side were recalculated and
are accurate now. Before, the markers were buggy and the frequencies
did not actually match up with them. I tested this with a 375 Hz sine
wave and a 750 Hz sine wave using the synthesizer plugin. Both the
375Hz and 750Hz wave match up with their respective labels.
Additionally, more markers were added to the left (number of labels
increased from 6 to 20).
4. The SpectrogramFFT class was redone almost completely to take
advantage of the "fftw" library. As noted above, buffering was added
between read_samples for efficiency. SpectrogramFFT no longer inherits
from CrossfadeFFT, and adhears closer to the RAII paradigm now. (I
didn't bother to make it "clean" by removing copy constructors and
assignment operators... but none of the other classes in the codebase
seem to follow RAII anyway)
5. Off by one error with data was fixed in various places (like the
data buffer). A real to complex FFT returns an array of size (window/2
+1), so where appropriate I changed buffers from HALF_WINDOW or
WINDOW_SIZE/2 into WINDOW_SIZE/2 + 1.
6. After the forward FFT, the data is now normalized by dividing
1/sqrt(N) on each element.
7. The rendering of the data is more accurate / less buggy. Further
when a pixel corresponds to a point between array elements, the pixel
is now a weighted average of the two elements, offering a smoother
picture. (IE: with a 4096 window size and a sample rate of 48000, the
first 3 frequencies correspond to 11.7Hz, 23.4Hz, and 35.1Hz. If a
pixel corresponded to say, 30Hz, then it would be a weighted average
between the 2nd and 3rd elements)
I did the svn diff command inside of the trunk/plugin/spectrogram
directory. Questions / comments on the patch are welcome.
--Percival Ross Tiglao
Index: spectrogram.C
===================================================================
--- spectrogram.C (revision 1060)
+++ spectrogram.C (working copy)
@@ -10,15 +10,27 @@
#include <string.h>
+#include <cmath>
REGISTER_PLUGIN(Spectrogram)
+// 4096 to 16384 are decent values
+// Smaller values are more accurate with time, while larger values
+// offer more accuracy in frequency. This should be easily set by the user
+// eventually... Powers of Two are supposedly most efficient?
#define WINDOW_SIZE 4096
-#define HALF_WINDOW 2048
+#define HALF_WINDOW (WINDOW_SIZE/2)
+// Step can be seen as how zoomed in the spectrometer is. The higher the
+// step size, the more zoomed in on the x-axis the GUI will be.
+#define STEP 10
+// 1/4 Meg rendering buffer seems enough, ~3 second buffer @ 44.1k
+#define BUFFER (1 << 18)
+
+
SpectrogramConfig::SpectrogramConfig()
{
level = 0.0;
@@ -31,7 +43,7 @@
SpectrogramLevel::SpectrogramLevel(Spectrogram *plugin, int x, int y)
- : BC_FPot(x, y, plugin->config.level, INFINITYGAIN, 0)
+ : BC_FPot(x, y, plugin->config.level, -40, 10)
{
this->plugin = plugin;
}
@@ -75,7 +87,7 @@
void SpectrogramWindow::create_objects()
{
int x = 60, y = 10;
- int divisions = 5;
+ int divisions = 20;
char string[BCTEXTLEN];
add_subwindow(canvas = new BC_SubWindow(x,
@@ -85,11 +97,21 @@
BLACK));
x = 10;
- for(int i = 0; i <= divisions; i++)
- {
- y = (int)((float)(canvas->get_h() - 10) / divisions * i) + 10;
- sprintf(string, "%d",
- Freq::tofreq((int)((float)TOTALFREQS / divisions * (divisions - i))));
+ //A real FFT can only go up to the niquist frequency, higher
+ //values are redundant or don't make sense.
+ //Go up with a constant multiplier b/c frequency is logrithmic based
+ //The returned values after a FFT are 0 to .5 of the sampling rate.
+ //.5 of the sampling rate is commonly refered as niquist frequency.
+ // To sync up with the spectrogram, we have a starting point equal
+ // to the frequency associated with the first value in FFT output,
+ // which is the value of "starting" here;
+ int niquist = plugin->get_project_samplerate()/2;
+ double starting = (double)niquist / (WINDOW_SIZE/2);
+ double multiplier = pow(niquist/starting, 1./divisions);
+ int j=0;
+ for(double i=niquist; j <= divisions; i/=multiplier, j++){
+ y = (int)((float)(canvas->get_h()) / divisions * j)+10;
+ sprintf(string, "%d", (int)(i+.5));
add_subwindow(new BC_Title(x, y, string));
}
@@ -130,40 +152,59 @@
-SpectrogramFFT::SpectrogramFFT(Spectrogram *plugin)
- : CrossfadeFFT()
+/** See http://www.fftw.org/fftw3_doc/One_002dDimensional-DFTs-of-Real-Data.html#One_002dDimensional-DFTs-of-Real-Data for details*/
+SpectrogramFFT::SpectrogramFFT(Spectrogram *plugin, int win): window(win)
{
this->plugin = plugin;
+ in = (double*) fftw_malloc(sizeof(double) * window);
+ out = (fftw_complex*) fftw_malloc(sizeof(fftw_complex) * (window/2 + 1));
+ plan_forward = fftw_plan_dft_r2c_1d(window, in, out, FFTW_ESTIMATE);
+ plan_backward = fftw_plan_dft_r2c_1d(window, in, out, FFTW_ESTIMATE);
+ buffer = new double[BUFFER];
}
SpectrogramFFT::~SpectrogramFFT()
{
+ fftw_destroy_plan(plan_forward);
+ fftw_destroy_plan(plan_backward);
+ fftw_free(in);
+ fftw_free(out);
+ delete [] buffer;
}
+void SpectrogramFFT::process_buffer(int output_sample, long size,
+ double* output_ptr, int direction){
+ this->read_samples(output_sample, window, in);
+ fftw_execute(plan_forward);
-int SpectrogramFFT::signal_process()
-{
double level = DB::fromdb(plugin->config.level);
- for(int i = 0; i < HALF_WINDOW; i++)
- {
- plugin->data[i] += level *
- sqrt(freq_real[i] * freq_real[i] +
- freq_imag[i] * freq_imag[i]);
+ for(int i=0; i<(window/2 + 1); i++){
+ output_ptr[i] = sqrt(
+ (out[i][0] * out[i][0] + out[i][1]*out[i][1])
+ / window );
+ //printf("%.3f\n", output_ptr[i]);
}
-
- plugin->total_windows++;
- return 0;
}
int SpectrogramFFT::read_samples(int64_t output_sample,
int samples,
double *buffer)
{
- return plugin->read_samples(buffer,
- 0,
- plugin->get_samplerate(),
- output_sample,
- samples);
+ //if the data is in the buffer...
+ if(current <= output_sample
+ && output_sample+samples < current + BUFFER){
+
+ //simple memcpy
+ int64_t diff = output_sample - current;
+ memcpy(buffer, this->buffer + diff, samples * sizeof(double));
+ return 1;
+ } else { //read the data into the buffer...
+ current = output_sample;
+ plugin->read_samples(this->buffer, 0, plugin->get_samplerate(),
+ output_sample, BUFFER);
+ memcpy(buffer, this->buffer, samples * sizeof(double));
+ return 1;
+ }
}
@@ -210,23 +251,23 @@
load_configuration();
if(!fft)
{
- fft = new SpectrogramFFT(this);
- fft->initialize(WINDOW_SIZE);
+ fft = new SpectrogramFFT(this, WINDOW_SIZE);
}
if(!data)
{
- data = new float[HALF_WINDOW];
+ data = new double[HALF_WINDOW+1];
}
- bzero(data, sizeof(float) * HALF_WINDOW);
- total_windows = 0;
- fft->process_buffer(start_position,
- size,
- buffer,
- get_direction());
- for(int i = 0; i < HALF_WINDOW; i++)
- data[i] /= total_windows;
- send_render_gui(data, HALF_WINDOW);
+ read_samples(buffer, 0, sample_rate, start_position, size);
+ int inc = (size/STEP);
+ for(int i=0; i<size; i+= inc){
+ total_windows = 0;
+ fft->process_buffer(start_position+i,
+ size,
+ data,
+ get_direction());
+ send_render_gui(data, HALF_WINDOW);
+ }
return 0;
}
@@ -250,37 +291,130 @@
}
}
+//Scale brightness to db, -5 db is highest, -80 db is lowest
+//Returns a value from [0 to 1]
+double Spectrogram::to_brightness(double data){
+ double base = DB::todb(data); // base is now the db
+ base += 80; //move the zero point up, min == scalefactor
+ base += config.level; // Add a bit of configuration
+ base /= 75 ; //scale
+ base = base>1.0? 1.0 : base;
+ base = base<0.0? 0.0 : base;
+ return base;
+// int toReturn = (int)(base); //cap
+// toReturn = toReturn > 0xff ? 0xff : toReturn;
+// toReturn = toReturn < 0x00 ? 0x00 : toReturn;
+// //printf("%.3f %.3f\n", data, base);
+// return toReturn;
+}
+
+//Scale hue, [0 to 1]
+//From 15db highest, -50db lowest
+double Spectrogram::to_hue(double data){
+ double base = DB::todb(data);
+ base += 50;
+ base += config.level;
+ base /= 65;
+ base = base>1.0? 1.0 : base;
+ base = base<0.0? 0.0 : base;
+ return base;
+}
+
+//Scale Saturation to [0 to 1]
+//From 5db highest, -40db lowest
+double Spectrogram::to_saturation(double data){
+ double base = DB::todb(data);
+ base += 40;
+ base += config.level;
+ base /= 45;
+ base = base>1.0? 1.0 : base;
+ base = base<0.0? 0.0 : base;
+ return base;
+}
+
+//From hue, saturation, and value
+//Assume all are between 0 and 1
+//Returns int of form 0xRRGGBB...
+int64_t Spectrogram::to_rgb(double h, double s, double v){
+ double r, g, b;
+ r = g = b = v;
+ if (s !=0){
+ double var_h = h*6;
+ int var_i = (int)var_h;
+ double var1 = v * (1-s);
+ double var2 = v * (1-s * (var_h-var_i));
+ double var3 = v * (1-s * (1 - (var_h-var_i)));
+ switch(var_i){
+ case 0:
+ r = v;
+ g = var3;
+ b = var1;
+ break;
+ case 1:
+ r = var2;
+ g = v;
+ b = var1;
+ break;
+ case 2:
+ r = var1;
+ g = v;
+ b = var3;
+ break;
+ case 3:
+ r = var1;
+ g = var2;
+ b = v;
+ break;
+ case 4:
+ r = var3;
+ g = var1;
+ b = v;
+ break;
+ case 5:
+ r = v;
+ g = var1;
+ b = var2;
+ break;
+ }
+ }
+ int64_t color = 0;
+ int R = (int) (r*255);
+ int G = (int) (g*255);
+ int B = (int) (b*255);
+ color = ((0xff&R) << 16) | ((0xff&G)<<8) | (0xff&B);
+ return color;
+}
+
void Spectrogram::render_gui(void *data, int size)
{
if(thread)
{
thread->window->lock_window("Spectrogram::render_gui");
- float *frame = (float*)data;
- int niquist = get_project_samplerate();
+ double *frame = (double*)data;
+ int niquist = get_project_samplerate()/2;
BC_SubWindow *canvas = thread->window->canvas;
int h = canvas->get_h();
- int input1 = HALF_WINDOW - 1;
+ double input1 = WINDOW_SIZE/2+1;
double *temp = new double[h];
// Scale frame to canvas height
- for(int i = 0; i < h; i++)
+ double count=WINDOW_SIZE/2 + 1; //frequency counter
+ for(int i = 0; i < h; i++, count=pow(WINDOW_SIZE/2+1, (double)(h-i)/h))
{
- int input2 = (int)((h - 1 - i) * TOTALFREQS / h);
- input2 = Freq::tofreq(input2) *
- HALF_WINDOW /
- niquist;
- input2 = MIN(HALF_WINDOW - 1, input2);
+ double input2 = count+.5;
double sum = 0;
- if(input1 > input2)
+ if(static_cast<int>(input1 - input2)>0)
{
- for(int j = input1 - 1; j >= input2; j--)
+ for(int j =(int)(input1 - 1); j >= (int)(input2); j--)
sum += frame[j];
- sum /= input1 - input2;
+ sum /= static_cast<int>(input1 - input2);
}
else
{
- sum = frame[input2];
+ double weight = count - ((int)count);
+ sum = frame[(int)(count)]*(1-weight) +
+ frame[(int)(count+1)]*(weight);
}
temp[i] = sum;
@@ -295,14 +429,15 @@
canvas->get_w() - 1,
canvas->get_h());
int x = canvas->get_w() - 1;
- double scale = (double)0xffffff;
for(int i = 0; i < h; i++)
{
int64_t color;
- color = (int)(scale * temp[i]);
- if(color < 0) color = 0;
- if(color > 0xffffff) color = 0xffffff;
+ double bright = to_brightness(temp[i]);
+ double sat = to_saturation(temp[i]);
+ double hue = to_hue(temp[i]);
+ color = to_rgb(hue, sat, bright);
+
canvas->set_color(color);
canvas->draw_pixel(x, i);
}
Index: spectrogram.h
===================================================================
--- spectrogram.h (revision 1060)
+++ spectrogram.h (working copy)
@@ -14,6 +14,7 @@
#include "vframe.inc"
+#include <fftw3.h>
class Spectrogram;
@@ -51,18 +52,28 @@
-class SpectrogramFFT : public CrossfadeFFT
+class SpectrogramFFT
{
public:
- SpectrogramFFT(Spectrogram *plugin);
+ SpectrogramFFT(Spectrogram *plugin, int window);
~SpectrogramFFT();
-
- int signal_process();
- int read_samples(int64_t output_sample,
- int samples,
- double *buffer);
+ void process_buffer(int output_sample, long size, double* output_ptr,
+ int direction);
+
Spectrogram *plugin;
+
+private:
+ int read_samples(int64_t output_sample, int samples, double *buffer);
+//Thread unsafety of fftw ??
+// Mutex* lock;
+ fftw_plan plan_forward;
+ fftw_plan plan_backward;
+ double *in;
+ fftw_complex *out;
+ int window;
+ double *buffer;
+ int64_t current; //current buffer location
};
@@ -107,8 +118,13 @@
SpectrogramConfig config;
SpectrogramThread *thread;
SpectrogramFFT *fft;
- float *data;
+ double *data;
int total_windows;
+private:
+ int64_t to_rgb(double h, double sl, double l);
+ double to_brightness(double);
+ double to_hue(double);
+ double to_saturation(double);
};
