HI Alex HDFS-7651 is raised for same..Please have a look at once..
Thanks & Regards Brahma Reddy Battula _______________________________________ From: Alexander Striffeler [a.striffe...@students.unibe.ch] Sent: Thursday, July 23, 2015 1:00 PM To: user@hadoop.apache.org Subject: Re: NNBench on external HDFS Hi Chris, Wow, thanks a lot for your swift and extensive response! I'll try your suggestion with the local copy and in a second step I'll open a jira request... Have a good day, --Alex On 22.07.2015 20:14, Chris Nauroth wrote: > Hi Alexander, > > Your NNBench usage looks basically correct, but NNBench is not a standard > Hadoop tool. It does not implement the org.apache.hadoop.util.Tool > interface, it does not execute through org.apache.hadoop.util.ToolRunner, > and therefore it does not support the command line arguments that a lot of > other Hadoop tools like the FsShell support. Specifically, it does not > support passing -D arguments to override fs.defaultFS or any other > configuration properties. > > An alternative way to handle this would be to get a local copy of the > configuration directory from the remote cluster that you want to test. I > expect those configuration files would have fs.defaultFS set to the URL of > that remote cluster in core-site.xml. Before launching NNBench, run > "export HADOOP_CONF_DIR=<path to local copy of configuration files>". > After exporting that environment variable, you can run "hadoop classpath" > to print the classpath that will be used by all "hadoop" commands and > confirm that the correct configuration directory for the target cluster is > on the classpath. Then, you can run NNBench again, but drop the -D > argument, since it's going to get ignored anyway. > > I don't see any reason why NNBench shouldn't implement the standard Tool > interface and thus support the command line arguments that you were > expecting. If you'd like to request that as an enhancement, please go > ahead and file an HDFS jira to request it. Feel free to post a patch too > if you're inclined. Otherwise, someone else in the community can pick it > up. > > I hope this helps. > > --Chris Nauroth > > > > > On 7/22/15, 12:41 AM, "Alexander Striffeler" > <a.striffe...@students.unibe.ch> wrote: > >> Hi all >> >> I'm pretty new to the Hadoop environment and I'm about performing some >> micro benchmarks. In particular, I'm struggling with executing NNBench >> against an external File System: >> >> hadoop jar >> /usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t >> ests.jar >> nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation >> create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100 >> -baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s` >> >> yields in >> java.lang.IllegalArgumentException: Wrong FS: >> hdfs://<external.file.system>/user/username/nnbench-hostname/data, >> expected: hdfs://<native fs> >> >> If I neglect the ext FS prefix in the baseDir, NNBench simply ignores >> the -D option and writes the files to the native DFS. Does anyone have >> an idea how to solve this and nnbench an external DFS? >> >> Thanks a lot, any hints are very appreciated! >> Regards, >> Alex
