I am getting worried on the huge number of bytes written to local fs. I have a 2 machine cluster, one has 100% io util, one has 10-20% io util during map phase, the input data is replicated on both machines (replication = 2). So I suspect the extra 80-90% io on the first machine is caused by read/write to local fs.
Which machine and which directory does this "local fs" refer to? So that I can check myself if it is the culprit. Thanks. Haijun -----Original Message----- From: Haijun Cao [mailto:[EMAIL PROTECTED] Sent: Wednesday, June 04, 2008 10:44 PM To: [EMAIL PROTECTED] Subject: local bytes read/written Hi, I just started using pig, it is really fun to write pig query. I noticed in the map reduce job page, it reports bytes read/written from/to local file system, and the number is 2x, 3x of the bytes read/write to hadoop. Just want to understand the internal working of pig a little bit better, what operations read/write to local fs? For what purpose? Is it to the local fs of the data nodes? which directory? Thanks Haijun
