Hi Matthew, Are you certain that your configuration options have been copied throughout every node on the cluster? Can you test the mount point on each cluster and make sure all nodes are seeing the same data? Also check in /tmp. if for some reason the nodes are pulling default config options, sometimes data gets put there.
Travis Hegner http://www.travishegner.com/ -----Original Message----- From: Matthew Dunn [mailto:[email protected]] Sent: Monday, July 26, 2010 4:03 PM To: [email protected] Subject: Hadoop on PVFS2 Hello, I'm doing research comparing HDFS and PVFS2. Therefore, I need to get hadoop jobs to work off PVFS2. Currently, I have PVFS mounted to look like a local filesystem on all the computers in my cluster (7 total, 1 master, 6 slaves). So, in core-site.xml, fs.default.name is set to "file:///home/matt/mnt/pvfs2" (where pvfs2 is mounted), and hadoop.tmp.dir is set to a local folder. In mapred-site.xml, mapred.local.dir is in a local folder, and mapred.temp.dir and mapred.system.dir are in the shared pvfs2 directory. With this configuration, when I start up mapreduce daemons with bin/start-mapred.sh, the jobtracker and tasktrackers are starting properly, and I can run a wordcount. However, most other jobs, including the benchmarks I want to use, let's use mapredtest for example, do not work. For some reason, temporary files that should be saved by the jobtracker in the shared directory are being saved in the local hadoop folder, which is odd since I don't even have anything pointing there in the configuration files. The job fails because the tasktrackers can't find those files, since they should be shared (or at least that's what it seems is happening). Can anyone help me out? -- View this message in context: http://old.nabble.com/Hadoop-on-PVFS2-tp29270163p29270163.html Sent from the HBase User mailing list archive at Nabble.com. The information contained in this communication is confidential and is intended only for the use of the named recipient. Unauthorized use, disclosure, or copying is strictly prohibited and may be unlawful. If you have received this communication in error, you should know that you are bound to confidentiality, and should please immediately notify the sender or our IT Department at 866.459.4599.
