Hey,
One of our charges is to do auditing and accounting with our file
systems (we use the simplifying assumption that the users are non-
malicious).
Auditing can be done by going through the namenode logs and utilizing
the UGI information to track opens/reads/writes back to the users.
Accounting can be done by adding up the byte counts from the datanode
traces (or via the lovely metrics interfaces). However, joining them
together appears to be impossible! The namenode audits record
originating IP and UGI; the datanode audits contain the originating IP
and DFSClient ID. With 8 clients (and possibly 8 users) opening
multiple files all from the same IP, it becomes a mess to untangle.
For example, in other filesystems, we've been able to construct a
database with one row representing a file access from open-to-close.
We record the username, amount of time the file was open, number of
bytes read, the remote IP, and the server which served the file
(previous filesystem saved an entire file on server, not blocks).
Already, that model quickly is problematic as several servers take
part in serving the file to the client. The depressing, horrible file
access pattern (Worse than random! To read a 1MB record entirely with
a read-buffer size of 10MB, you can possibly read up to 2GB) of some
jobs means that recording each read is not practical.
I'd like to record audit records and transfer accounting (at some
level) into the DB. Does anyone have any experience in doing this?
It seems that, if I can add the DFSClient ID into the namenode logs, I
can record:
1) Each open (but miss the corresponding close) of a file at the
namenode, along with the UGI, timestamp, IP
2) Each read/write on a datanode records the datanode, remote IP,
DFSClient, bytes written/read, (but I miss the overall transaction
time! Possibly could be logged). Don't record the block ID, as I
can't map block ID -> file name in a cheap/easy manner (I'd have to
either do this synchronously, causing a massive performance hit -- or
do this asynchronously, and trip up over any files which were deleted
after they were read).
This would allow me to see who is accessing what files, and how much
that client is reading - but not necessarily which files they read
from, if the same client ID is used for multiple files. This also
will allow me to trace reads back to specific users (so I can tell who
has the worst access patterns and beat them).
So, my questions are:
a) Is anyone doing anything remotely similar which I can reuse?
b) Is there some hole in my logic which would render the approach
useless?
c) Is my approach reasonable? I.e., should I really be looking at
inserting hooks into the DFSClient, as that's the only thing which can
tell me information like "when did the client close the file?"?
Advise is welcome.
Brian
