> > A set of reasonable performance tests and results would be very helpful > in helping people decide whether to go with streaming or not. Hopefully > we can get some numbers from this thread and publish them? Anyone else > compared streaming with native java? >
I think this is a great idea. I think it'd also be nice to have a small cookbook-ish thing showing equivalent programs in each style. Perhaps a simple streaming example with python or something and it's equivalent using Java. I'd have written this already but I don't know java. On Mon, Mar 31, 2008 at 3:42 PM, Colin Freas <[EMAIL PROTECTED]> wrote: > Really? > > I would expect the opposite: for compressed files to process slower. > > You're saying that is not the case, and that compressed files actually > increase the speed of jobs? > > -Colin > > On Mon, Mar 31, 2008 at 4:51 PM, Andreas Kostyrka <[EMAIL PROTECTED]> > wrote: > > > Well, on our EC2/HDFS-on-S3 cluster I've noticed that it helps to > > provide the input files gzipped. Not great difference (e.g. 50% slower > > when not gzipped, plus it took more than twice as long to upload the > > data to HDFS-on-S3 in the first place), but still probably relevant. > > > > Andreas > > > > Am Montag, den 31.03.2008, 13:30 -0700 schrieb lin: > > > I'm running custom map programs written in C++. What the programs do > is > > very > > > simple. For example, in program 2, for each input line ID node1 > > node2 > > > ... nodeN > > > the program outputs > > > node1 ID > > > node2 ID > > > ... > > > nodeN ID > > > > > > Each node has 4GB to 8GB of memory. The java memory setting is > -Xmx300m. > > > > > > I agree that it depends on the scripts. I tried replicating the > > computation > > > for each input line by 10 times and saw significantly better speedup. > > But it > > > is still pretty bad that Hadoop streaming has such big overhead for > > simple > > > programs. > > > > > > I also tried writing program 1 with Hadoop Java API. I got almost > 1000% > > > speed up on the cluster. > > > > > > Lin > > > > > > On Mon, Mar 31, 2008 at 1:10 PM, Theodore Van Rooy < > [EMAIL PROTECTED]> > > > wrote: > > > > > > > are you running a custom map script or a standard linux command like > > WC? > > > > If > > > > custom, what does your script do? > > > > > > > > How much ram do you have? what are you Java memory settings? > > > > > > > > I used the following setup > > > > > > > > 2 dual core, 16 G ram, 1000MB Java heap size on an empty box with a > 4 > > task > > > > max. > > > > > > > > I got the following results > > > > > > > > WC 30-40% speedup > > > > Sort 40% speedup > > > > Grep 5X slowdown (turns out this was due to what you described > > above... > > > > Grep > > > > is just very highly optimized for command line) > > > > Custom perl script which is essentially a For loop which matches > each > > row > > > > of > > > > a dataset to a set of 100 categories) 60% speedup. > > > > > > > > So I do think that it depends on your script... and some other > > settings of > > > > yours. > > > > > > > > Theo > > > > > > > > On Mon, Mar 31, 2008 at 2:00 PM, lin <[EMAIL PROTECTED]> wrote: > > > > > > > > > Hi, > > > > > > > > > > I am looking into using Hadoop streaming to parallelize some > simple > > > > > programs. So far the performance has been pretty disappointing. > > > > > > > > > > The cluster contains 5 nodes. Each node has two CPU cores. The > task > > > > > capacity > > > > > of each node is 2. The Hadoop version is 0.15. > > > > > > > > > > Program 1 runs for 3.5 minutes on the Hadoop cluster and 2 minutes > > in > > > > > standalone (on a single CPU core). Program runs for 5 minutes on > the > > > > > Hadoop > > > > > cluster and 4.5 minutes in standalone. Both programs run as > map-only > > > > jobs. > > > > > > > > > > I understand that there is some overhead in starting up tasks, > > reading > > > > to > > > > > and writing from the distributed file system. But they do not seem > > to > > > > > explain all the overhead. Most map tasks are data-local. I > modified > > > > > program > > > > > 1 to output nothing and saw the same magnitude of overhead. > > > > > > > > > > The output of top shows that the majority of the CPU time is > > consumed by > > > > > Hadoop java processes (e.g. > > org.apache.hadoop.mapred.TaskTracker$Child). > > > > > So > > > > > I added a profile option (-agentlib:hprof=cpu=samples) to > > > > > mapred.child.java.opts. > > > > > > > > > > The profile results show that most of CPU time is spent in the > > following > > > > > methods > > > > > > > > > > rank self accum count trace method > > > > > > > > > > 1 23.76% 23.76% 1246 300472 > > > > java.lang.UNIXProcess.waitForProcessExit > > > > > > > > > > 2 23.74% 47.50% 1245 300474 java.io.FileInputStream.readBytes > > > > > > > > > > 3 23.67% 71.17% 1241 300479 java.io.FileInputStream.readBytes > > > > > > > > > > 4 16.15% 87.32% 847 300478 > java.io.FileOutputStream.writeBytes > > > > > > > > > > And their stack traces show that these methods are for interacting > > with > > > > > the > > > > > map program. > > > > > > > > > > > > > > > TRACE 300472: > > > > > > > > > > > > > > > java.lang.UNIXProcess.waitForProcessExit( > > UNIXProcess.java:Unknownline) > > > > > > > > > > java.lang.UNIXProcess.access$900(UNIXProcess.java:20) > > > > > > > > > > java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132) > > > > > > > > > > TRACE 300474: > > > > > > > > > > java.io.FileInputStream.readBytes( > > FileInputStream.java:Unknown > > > > > line) > > > > > > > > > > java.io.FileInputStream.read(FileInputStream.java:199) > > > > > > > > > > java.io.BufferedInputStream.read1(BufferedInputStream.java > > :256) > > > > > > > > > > java.io.BufferedInputStream.read(BufferedInputStream.java > > :317) > > > > > > > > > > java.io.BufferedInputStream.fill(BufferedInputStream.java > > :218) > > > > > > > > > > java.io.BufferedInputStream.read(BufferedInputStream.java > > :237) > > > > > > > > > > java.io.FilterInputStream.read(FilterInputStream.java:66) > > > > > > > > > > org.apache.hadoop.mapred.LineRecordReader.readLine( > > > > > LineRecordReader.java:136) > > > > > > > > > > org.apache.hadoop.streaming.UTF8ByteArrayUtils.readLine( > > > > > UTF8ByteArrayUtils.java:157) > > > > > > > > > > org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run( > > > > > PipeMapRed.java:348) > > > > > > > > > > TRACE 300479: > > > > > > > > > > java.io.FileInputStream.readBytes( > > FileInputStream.java:Unknown > > > > > line) > > > > > > > > > > java.io.FileInputStream.read(FileInputStream.java:199) > > > > > > > > > > java.io.BufferedInputStream.fill(BufferedInputStream.java > > :218) > > > > > > > > > > java.io.BufferedInputStream.read(BufferedInputStream.java > > :237) > > > > > > > > > > java.io.FilterInputStream.read(FilterInputStream.java:66) > > > > > > > > > > org.apache.hadoop.mapred.LineRecordReader.readLine( > > > > > LineRecordReader.java:136) > > > > > > > > > > org.apache.hadoop.streaming.UTF8ByteArrayUtils.readLine( > > > > > UTF8ByteArrayUtils.java:157) > > > > > > > > > > org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run( > > > > > PipeMapRed.java:399) > > > > > > > > > > TRACE 300478: > > > > > > > > > > > > > > > java.io.FileOutputStream.writeBytes( > > FileOutputStream.java:Unknownline) > > > > > > > > > > java.io.FileOutputStream.write(FileOutputStream.java:260) > > > > > > > > > > java.io.BufferedOutputStream.flushBuffer( > > > > BufferedOutputStream.java > > > > > :65) > > > > > > > > > > java.io.BufferedOutputStream.flush( > BufferedOutputStream.java > > :123) > > > > > > > > > > java.io.BufferedOutputStream.flush( > BufferedOutputStream.java > > :124) > > > > > > > > > > java.io.DataOutputStream.flush(DataOutputStream.java:106) > > > > > > > > > > org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java > > :96) > > > > > > > > > > org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > > > > > > > > > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:192) > > > > > org.apache.hadoop.mapred.TaskTracker$Child.main( > > TaskTracker.java > > > > > :1760) > > > > > > > > > > > > > > > I don't understand why Hadoop streaming needs so much CPU time to > > read > > > > > from > > > > > and write to the map program. Note it takes 23.67% time to read > from > > the > > > > > standard error of the map program while the program does not > output > > any > > > > > error at all! > > > > > > > > > > Does anyone know any way to get rid of this seemingly unnecessary > > > > overhead > > > > > in Hadoop streaming? > > > > > > > > > > Thanks, > > > > > > > > > > Lin > > > > > > > > > > > > > > > > > > > > > -- > > > > Theodore Van Rooy > > > > http://greentheo.scroggles.com > > > > > > > -- Travis Brady www.mochiads.com
