1. is there XDR support for win32?
yes. you just need to make sure that you install the SunRPC package on cygwin. i have the actually source code for sunrpc as well which could be easily included in the source.
2. is it easy to change the format of the databases and maintain compatibility?
yes. i recommend that we use a discriminant union format something like the following...
enum RRDB_formats {
RRDB_NON_PORTABLE = 0,
RRDB_VERSION_1,
RRDB_VERSION_2
};
struct RRDB_2 {
int foo;
int bar;
};
struct RRDB_1 {
int foo;
};
union RRDB_body switch (RRDB_formats format) {
case RRDB_VERSION_2:
RRDB_2 db2;
case RRDB_VERSION_1:
RRDB_1 db1;
case RRDB_NON_PORTABLE:
void;
};
struct RRDB {
string signature<4>;
RRDB_body bdy;
};
the struct RRDB is the general definition of a database. it states that
all databases start with a signature that is at most 4 bytes in length
(and should be set to "RRD"). immediately after the signature is the
body of the database which starts with a enum (int) stating the format
of the data to come.
each database format change will require that an "upgrade/convert" function is written which takes an older version RRDB and converts it to a newer version. here are the steps...
a. decode the XDR data into a struct RRDB. b. check that the signature is correct !strcmp(rrdb.signature,"RRD")c. if rrdb.bdy.format == RRDB_VERSION_2 do nothing as we're at the latest version d. if rrdb.dby.format != RRDB_VERSION_2, call a function to update the structure of the struct RRDB. e. when encoding the XDR data next time it will be saved in the newer format (since the XDR library will process the disciminant which will be set to RRDB_VERSION_2 and then walk the data at rrdb.bdy.db2.
3. will the new XDR version be able to read the old rrd files?yes... we just keep the code that you already have for reading non-portable databases. this code will be called when rrdb.data.format == RRDB_NON_PORTABLE to decoding the non-portable database into a RRDB format. when this RRDB is later serializing/encoded it will be encoded in the new XDR format.
4. i will be checking in the mmap support tonight after some further testing.
you'll be able to drop mmap support in the future as it won't be necessary when using this XDR approach. please don't take what follows as a negative critique of rrdtool even though it might sound like it. i love rrdtool that is why i'm willing to invest time into making it even better.
the reason ganglia users have had to use ram-backed filesystems to store rrds is that rrdtool is very disk io intensive. after reading the rrd_update code i understand why now. rrd_update locks the file.. reads the file header, runs ftell to find where the header ends, and then proceeds to fseek/ftell/read/write within the file to search for where to update data and then fseeks back to the header and write new header info.
all those file seeks/writes/reads are very expensive.the XDR approach will not require people to use mmap() or ram-backed filesystems as a workaround. instead of fseek/ftell around a file, the new rrd_update would simply update the struct RRDB (in-memory) and then encode the update struct to disk. here are the steps for the new rrd_update..
a. decode the XDR data from a file/socket/memory/wherever into a struct RRDB.
b. walk the struct RRDB in memory (_not_ the file) and update the value arrays.
c. encode the updated RRDB structure to file/socket/memory/wherever using XDR
this three step approach will provide HUGE performance improvements over the current code and give developers the power to decide exactly when RRDB structures are synced/encoded to files/pipes/sockets/whatever.
i'll try to get an XDR RRDB backend written soon that will (i hope) put my code where my mouth is.
-matt
--
PGP fingerprint 'A7C2 3C2F 8445 AD3C 135E F40B 242A 5984 ACBC 91D3'
They that can give up essential liberty to obtain a little
temporary safety deserve neither liberty nor safety.
--Benjamin Franklin, Historical Review of Pennsylvania, 1759
signature.asc
Description: OpenPGP digital signature
