Dear all
I have been using fastbit since the initial release in the field of
network monitoring. While I'm impressed by fastbit performance for
counting records that match some criteria, when I actually request to
read the matching records the performance is not great.
For instance I can count matching records in a matter of msec whereas
retrieving the actual data takes 30 sec (when using 10 Million
records) or more (data was already indexed) using ibis or similar
tool. I was wondering if I make some mistakes when building the
fastbit archives. For this reason I have built a simple program
(enclosed below) that I have used to create dummy data value to query.
Changing the input parameters, I see that the indexing speed changes
significantly, but the query speed is still the same.
Question: is the obtained performance what you also expect, or did I
make some mistakes while building the fastbit archives?
Thanks in advance, Luca
----
#include <capi.h>
#include <ctype.h>
#include <string.h>
#include <stdlib.h>
/* ****************************************************** */
void timeval_diff(struct timeval *begin, struct timeval *end, struct
timeval *result) {
if(end->tv_sec >= begin->tv_sec) {
result->tv_sec = end->tv_sec-begin->tv_sec;
if((end->tv_usec - begin->tv_usec) < 0) {
result->tv_usec = 1000000 + end->tv_usec - begin->tv_usec;
if(result->tv_usec > 1000000) begin->tv_usec = 1000000;
result->tv_sec--;
} else
result->tv_usec = end->tv_usec-begin->tv_usec;
} else
result->tv_sec = 0, result->tv_usec = 0;
}
void append(char *dir, int num, int total) {
int *a_vals, *b_vals, i;
struct timeval begin, end, diff;
u_int32_t v, tot;
a_vals = (int*)malloc(sizeof(int)*num);
b_vals = (int*)malloc(sizeof(int)*num);
if(a_vals && b_vals) {
for (i = 0; i < num; i++)
a_vals[i] = i, b_vals[i] = i;
gettimeofday (& begin, NULL);
for(i=0; i<total; i++) {
fastbit_add_values("a", "int", a_vals, num, 0);
fastbit_add_values("b", "int", b_vals, num, 0);
fastbit_flush_buffer(dir);
}
gettimeofday (& end, NULL);
timeval_diff (& begin, & end, & diff);
v = diff.tv_sec*1000 + diff.tv_usec/1000;
tot = i*num;
printf("written %d records on disk [%.2f sec][%.2f insert/sec]\n",
tot, (float)v/1000, (float)tot*1000/v);
}
}
int main(int argc, char *argv[]) {
if(argc != 3) {
printf("fastbit_test <records x shot> <num shots>\n");
return(0);
}
append("fb", atoi(argv[1]), atoi(argv[2]));
return(0);
}
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
