Are there any tips to reduce the latency in general for hdfs?
I noticed that when trying to copy a 30gb file it takes a "while". I know
its subjective but would like to know if anyone has any tricks to reduce
latency.
On Sat, Jan 29, 2011 at 12:58 PM, Nathan Rutman <nrut...@gmail.com> wrote:
>
> On Jan 28, 2011, at 9:53 PM, Harsh J wrote:
>
> > Moving discussion to hdfs-user mailing list: hdfs-u...@hadoop.apache.com
> >
> > On Sat, Jan 29, 2011 at 5:06 AM, Pei HE <pei...@gmail.com> wrote:
> >> Hi all,
> >> I want to know the detailed performance of Hadoop.
> >>
> >> I am writing a client code to test the latency, transfer speed of
> >> HDFS, and the time of initiating a JVM for a task.
> >>
> >> Does anyone know that which classes can be called without running a
> >> whole map/reduce job to find out the performance of Hadoop?
>
> You can get the performance of HDFS without Hadoop by using it directly:
>
> ######## Creating 1GB file ###########
>
> #ftg-007:/usr/local/hadoop # dd if=/dev/zero of=/hdfs/demofile bs=1M
> count=1000
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes (1.0 GB) copied, 8.78527 s, 119 MB/s
>
> ######### Copy from local to HDFS ########
> #ftg-007:/usr/local/hadoop # time ./bin/hadoop dfs -copyFromLocal
> /hdfs/demofile /
> real 0m51.911s
> user 0m9.945s
> sys 0m2.128s
>
> ######### Copy to Local from HDFS ########
> #ftg-007:/usr/local/hadoop # time ./bin/hadoop dfs -copyToLocal /demofile
> /hdfs/test/
> real 0m19.068s
> user 0m8.205s
> sys 0m4.948s
>
> >>
> >> I am trying to create a DFSClient in the following way.
> >> Configuration conf = new Configuration();
> >> DFSClient client = new DFSClient(conf);
> >> client.exists("hdfs://localhost:9900/home/");
> >> But, a RemoteException was thrown.
> >
> > Would help to know what the cause for the throw of a RemoteException was.
> >
> > --
> > Harsh J
> > www.harshj.com
>
>
--
--- Get your facts first, then you can distort them as you please.--
