The notion of a client/task ID, independent of IP or username seems
useful for log analysis. DFS's client ID is probably currently your
best bet, but we might improve its implementation, and make the notion
more generic.
It is currently implemented as:
String taskId = conf.get("mapred.task.id");
if (taskId != null) {
this.clientName = "DFSClient_" + taskId;
} else {
this.clientName = "DFSClient_" + r.nextInt();
}
This hardwires a mapred dependency, which is fragile, and it's fairly
useless outside of mapreduce, degenerating to a random number.
Rather we should probably have a configuration property that's
explicitly used to indicate the user-level task, that's different than
the username, IP, etc. For MapReduce jobs this could default to the
job's ID, but applications might override it. So perhaps we could add
static methods FileSystem.{get,set}TaskId(Configuration), then change
the logging code to use this?
What do others think?
Doug
Brian Bockelman wrote:
Hey,
One of our charges is to do auditing and accounting with our file
systems (we use the simplifying assumption that the users are
non-malicious).
Auditing can be done by going through the namenode logs and utilizing
the UGI information to track opens/reads/writes back to the users.
Accounting can be done by adding up the byte counts from the datanode
traces (or via the lovely metrics interfaces). However, joining them
together appears to be impossible! The namenode audits record
originating IP and UGI; the datanode audits contain the originating IP
and DFSClient ID. With 8 clients (and possibly 8 users) opening
multiple files all from the same IP, it becomes a mess to untangle.
For example, in other filesystems, we've been able to construct a
database with one row representing a file access from open-to-close. We
record the username, amount of time the file was open, number of bytes
read, the remote IP, and the server which served the file (previous
filesystem saved an entire file on server, not blocks). Already, that
model quickly is problematic as several servers take part in serving the
file to the client. The depressing, horrible file access pattern (Worse
than random! To read a 1MB record entirely with a read-buffer size of
10MB, you can possibly read up to 2GB) of some jobs means that recording
each read is not practical.
I'd like to record audit records and transfer accounting (at some level)
into the DB. Does anyone have any experience in doing this? It seems
that, if I can add the DFSClient ID into the namenode logs, I can record:
1) Each open (but miss the corresponding close) of a file at the
namenode, along with the UGI, timestamp, IP
2) Each read/write on a datanode records the datanode, remote IP,
DFSClient, bytes written/read, (but I miss the overall transaction
time! Possibly could be logged). Don't record the block ID, as I can't
map block ID -> file name in a cheap/easy manner (I'd have to either do
this synchronously, causing a massive performance hit -- or do this
asynchronously, and trip up over any files which were deleted after they
were read).
This would allow me to see who is accessing what files, and how much
that client is reading - but not necessarily which files they read from,
if the same client ID is used for multiple files. This also will allow
me to trace reads back to specific users (so I can tell who has the
worst access patterns and beat them).
So, my questions are:
a) Is anyone doing anything remotely similar which I can reuse?
b) Is there some hole in my logic which would render the approach useless?
c) Is my approach reasonable? I.e., should I really be looking at
inserting hooks into the DFSClient, as that's the only thing which can
tell me information like "when did the client close the file?"?
Advise is welcome.
Brian
