+100000 :) On Dec 15, 2007 2:25 PM, stack (JIRA) <[EMAIL PROTECTED]> wrote:
> > [ > https://issues.apache.org/jira/browse/HADOOP-2434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552145] > > stack commented on HADOOP-2434: > ------------------------------- > > +1 > > This patch works for me. > > > MapFile.get on HDFS in TRUNK is WAY!!! slower than 0.15.x > > --------------------------------------------------------- > > > > Key: HADOOP-2434 > > URL: https://issues.apache.org/jira/browse/HADOOP-2434 > > Project: Hadoop > > Issue Type: Bug > > Components: io > > Reporter: stack > > Attachments: TestMapFile.java, TestMapFile.java > > > > > > Stall happens down in SequenceFile in the first call to getPos inside > readRecordLength. I tried the johano patch from HADOOP-2172 that restores > the positional cache but that didn't seem to be the issue here. > > Here is data to support my assertion. > > I wrote a little program to make a MapFile of 1M records. I then did 1M > random reads from same file. Below are timings from a 0.15.0 and TRUNK as > of this afternoon run. > > 0.15.x branch: > > {code} > > [EMAIL PROTECTED] branch-0.15]$ ./bin/hadoop > org.apache.hadoop.io.TestMapFile > > .07/12/15 01:29:02 WARN util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > > 07/12/15 01:29:02 INFO io.TestMapFile: Writing 1000000 rows to > testMapFileRandomAccess > > 07/12/15 01:32:04 INFO io.TestMapFile: Writing 1000000 records took > 182009ms > > 07/12/15 01:32:04 INFO io.TestMapFile: Reading 1000000 random rows > > 07/12/15 01:48:02 INFO io.TestMapFile: Reading 1000000 random records > took 958243ms > > Time: 1,140.652 > > OK (1 test) > > {code} > > For the below test using TRUNK r604352, I amended the test so it output > a log message every 100k reads: > > {code} > > [EMAIL PROTECTED] hadoop-trunk]$ ./bin/hadoop > org.apache.hadoop.io.TestMapFile > > .07/12/15 01:56:34 WARN util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > > 07/12/15 01:56:34 INFO io.TestMapFile: Writing 1000000 rows to > testMapFileRandomAccess > > 07/12/15 01:59:38 INFO io.TestMapFile: Writing 1000000 records took > 183986ms > > 07/12/15 01:59:38 INFO io.TestMapFile: Reading 1000000 random rows > > ......... > > {code} > > After 20 minutes it still hadn't printed out the 'read 100k messages' (I > had to leave -- will fill in final figures later) > > -- > This message is automatically generated by JIRA. > - > You can reply to this email to add a comment to the issue online. > >
