Andrzej Bialecki wrote:
Shawn Gervais wrote:
Greetings list,
This is my DFS report:
Total raw bytes: 709344133120 (660.62 Gb)
Used raw bytes: 302794461922 (281.99 Gb)
% used: 42.68%
Total effective bytes: 11826067632 (11.01 Gb)
Effective replication multiplier: 25.6039853097637
These numbers seem to me to be completely insane -- a 25 times
replication of blocks. I have my replication factor set to 3.
"Used raw bytes" goes up when I run jobs, and if I delete files those
jobs produce within DFS (e.g. a segment for a failed fetch), it
doesn't appear that hadoop immediately reclaims the space used by the
deleted files' blocks.
Am I right? Is this a bug?
What does 'hadoop fsck /' say?
Status: HEALTHY
Total size: 5265553979 B
Total blocks: 1952 (avg. block size 2697517 B)
Total dirs: 333
Total files: 1868
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Target replication factor: 3
Real replication factor: 3.0
For reference, the head of the "dfs -report" output taken at the same
time -- my dfs has changed since the above mail.
Total raw bytes: 709344133120 (660.62 Gb)
Used raw bytes: 68715927635 (63.99 Gb)
% used: 9.68%
Total effective bytes: 5265553979 (4.90 Gb)
Effective replication multiplier: 13.050085120967667
I am still confused by the wildly different replication factors seen here.
It occurs to me that Hadoop might be counting as "used raw bytes" other
files that are not in the DFS filestore, but are on the same partition.
For example, my Nutch installation and some other files are on the same
partition as the DFS filestore.
Regards,
-Shawn
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
