Thanks shashwat .. After increasing the Size of disk MR works fine . There
are 6 mappers are running out of 6.first 5 map is talking 1L excuting in 20
min ... last 6th mapper is talking 5L record and taking long time to
excecute . which talks nearly 1.5 hours .. . Can any one give me an idea to
fix this .
Thanks and Regards,
S SYED ABDUL KATHER
On Wed, Jul 18, 2012 at 12:21 AM, shashwat shriparv <
[email protected]> wrote:
> Hi Syed,
>
> Do give try to this solution :
>
>
> http://agiletesting.blogspot.in/2011/11/troubleshooting-memory-allocation.html
>
> Regards
>
> ∞
> Shashwat Shriparv
>
>
>
> On Wed, Jul 18, 2012 at 12:18 AM, shashwat shriparv <
> [email protected]> wrote:
>
> > Hi Syed,
> >
> > The problem is with the disk space. as map-reduce keeps the intermediate
> > result on the local disk, just check if you have enough disk space. and
> > also make sure that you have cleared the tmp directory and its writable.
> > Just provide more space and try else try with small number of users and
> > check if its working
> >
> > Regards
> >
> > ∞
> > Shashwat Shriparv
> >
> >
> >
> >
> > On Tue, Jul 17, 2012 at 11:50 AM, syed kather <[email protected]>
> wrote:
> >
> >> Team ,
> >> i had wrote a mapreduce program . scenario of my program is to emit
> >> <userid,seqid> .
> >>
> >> Total no user : 825
> >> Total no seqid:6583100
> >>
> >> No of map which the program will emit is : 825 * 6583100
> >>
> >> I have Hbase table called ObjectSequence : which consist of
> >> 6583100(rows)
> >>
> >> i had use TableMapper and TableReducer for my map reduce program
> >>
> >>
> >> Problem definition :
> >>
> >> Processor : i7
> >> Replication Factor : 1
> >> Live Datanodes : 3
> >>
> >> Node Last
> >> Contact Admin State Configured
> >> Capacity (GB) Used
> >> (GB) Non DFS
> >> Used (GB) Remaining
> >> (GB) Used
> >> (%) Used
> >> (%) Remaining
> >> (%) Blocks chethan 1In Service 28.590.625.172.82 2.11
> >>
> >> 9.8773 shashwat<
> >>
> http://shashwat:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
> >> 2In Service28.980.87 22.016.13
> >>
> >> 21.0469 syed<
> >> http://syed:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
> >> 0In Service28.984.29 18.376.3214.8
> >>
> >> 21.82129
> >> When i run balancer in hadoop i had seen Blocks are not equally
> >> distributed
> >> . Can i know what may be the reason for this ..
> >>
> >>
> >> Kind% CompleteNum TasksPendingRunningComplete KilledFailed/Killed
> >> Task Attempts<
> >> http://chethan:50030/jobfailures.jsp?jobid=job_201207121836_0007>
> >> map<
> >>
> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=map&pagenum=1
> >> >
> >> 85.71%
> >>
> >> 701<
> >>
> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=map&pagenum=1&state=running
> >> >
> >> 6<
> >>
> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=map&pagenum=1&state=completed
> >> >
> >> 0 3<
> >>
> http://chethan:50030/jobfailures.jsp?jobid=job_201207121836_0007&kind=map&cause=failed
> >> >/
> >> 1<
> >>
> http://chethan:50030/jobfailures.jsp?jobid=job_201207121836_0007&kind=map&cause=killed
> >> >
> >> reduce<
> >>
> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=reduce&pagenum=1
> >> >
> >> 28.57%
> >>
> >> 101<
> >>
> http://chethan:50030/jobtasks.jsp?jobid=job_201207121836_0007&type=reduce&pagenum=1&state=running
> >> >
> >> 000 / 0
> >> i had seen only Number Task is allocated is 8 . Is there any possibility
> >> to
> >> increase the Map Number of Task
> >>
> >> Completed TasksTaskCompleteStatusStart TimeFinish TimeErrorsCounters
> >> task_201207121836_0007_m_000001<
> >>
> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000001
> >> >
> >> 100.00%
> >> UserID: 777 SEQID:415794
> >> 12-Jul-2012 21:35:48
> >> 12-Jul-2012 21:36:12 (24sec)
> >>
> >> 16<
> >>
> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000001>
> >> task_201207121836_0007_m_000002<
> >>
> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000002
> >> >
> >> 100.00%
> >> UserID: 777 SEQID:422256
> >> 12-Jul-2012 21:35:50
> >> 12-Jul-2012 21:36:47 (57sec)
> >>
> >> 16<
> >>
> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000002>
> >> task_201207121836_0007_m_000003<
> >>
> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000003
> >> >
> >> 100.00%
> >> UserID: 777 SEQID:563544
> >> 12-Jul-2012 21:35:50
> >> 12-Jul-2012 22:00:08 (24mins, 17sec)
> >>
> >> 16<
> >>
> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000003>
> >> task_201207121836_0007_m_000004<
> >>
> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000004
> >> >
> >> 100.00%
> >> UserID: 777 SEQID:592918
> >> 12-Jul-2012 21:35:50
> >> 12-Jul-2012 21:42:09 (6mins, 18sec)
> >>
> >> 16<
> >>
> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000004>
> >> task_201207121836_0007_m_000005<
> >>
> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000005
> >> >
> >> 100.00%
> >> UserID: 777 SEQID:618121
> >> 12-Jul-2012 21:35:50
> >> 12-Jul-2012 21:44:34 (8mins, 43sec)
> >>
> >> 16<
> >>
> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000005>
> >> task_201207121836_0007_m_000006<
> >>
> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000006
> >> >
> >> 100.00%
> >> UserID: 777 SEQID:685810
> >> 12-Jul-2012 21:36:12
> >> 12-Jul-2012 21:44:18 (8mins, 6sec)
> >>
> >> 16<
> >>
> http://chethan:50030/taskstats.jsp?tipid=task_201207121836_0007_m_000006>
> >> why for last Map task is talking nearly 2 hours .please give me some
> >> suggestion how to do an optimization
> >>
> >> TaskCompleteStatusStart Time Finish TimeErrorsCounters
> >> task_201207121836_0007_m_000000<
> >>
> http://chethan:50030/taskdetails.jsp?tipid=task_201207121836_0007_m_000000
> >> >
> >> 0.00%
> >> UserID: 482 SEQID:99596
> >> 12-Jul-2012 21:35:48
> >>
> >> java.io.IOException: Spill failed
> >> at
> >>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1029)
> >> at
> >>
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
> >> at
> >>
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> >> at
> >>
> org.pointcross.SearchPermission.MapReduce.NewObjectMapper.map(NewObjectMapper.java:205)
> >> at
> >>
> org.pointcross.SearchPermission.MapReduce.NewObjectMapper.map(NewObjectMapper.java:1)
> >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> >> at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> >> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:416)
> >> at
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> >> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException:
> >> Could not find any valid local directory for output/spill712.out
> >> at
> >>
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:381)
> >> at
> >>
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)
> >> at
> >>
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:127)
> >> at
> >>
> org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:121)
> >> at
> >>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1392)
> >> at
> >>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:853)
> >> at
> >>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1344)
> >>
> >> java.lang.RuntimeException: Error while running command to get file
> >> permissions : java.io.IOException: Cannot run program "/bin/ls":
> >> java.io.IOException: error=12, Cannot allocate memory
> >> at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
> >> at org.apache.hadoop.util.Shell.runCommand(Shell.java:200)
> >> at org.apache.hadoop.util.Shell.run(Shell.java:182)
> >> at
> >>
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
> >> at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
> >> at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
> >> at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:703)
> >> at
> >>
> org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:443)
> >> at
> >>
> org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getOwner(RawLocalFileSystem.java:426)
> >> at
> >> org.apache.hadoop.mapred.TaskLog.obtainLogDirOwner(TaskLog.java:251)
> >> at
> >>
> org.apache.hadoop.mapred.TaskLogsTruncater.truncateLogs(TaskLogsTruncater.java:124)
> >> at org.apache.hadoop.mapred.Child$4.run(Child.java:260)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:416)
> >> at
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> >> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
> >> allocate memory
> >> at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
> >> at java.lang.ProcessImpl.start(ProcessImpl.java:81)
> >> at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
> >> ... 15 more
> >>
> >> at
> >>
> org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:468)
> >> at
> >>
> org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getOwner(RawLocalFileSystem.java:426)
> >> at
> >> org.apache.hadoop.mapred.TaskLog.obtainLogDirOwner(TaskLog.java:251)
> >> at
> >>
> org.apache.hadoop.mapred.TaskLogsTruncater.truncateLogs(TaskLogsTruncater.java:124)
> >> at org.apache.hadoop.mapred.Child$4.run(Child.java:260)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:416)
> >> at
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> >> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >>
> >> java.io.IOException: Spill failed
> >> at
> >>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1029)
> >> at
> >>
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
> >> at
> >>
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> >> at
> >>
> org.pointcross.SearchPermission.MapReduce.NewObjectMapper.map(NewObjectMapper.java:205)
> >> at
> >>
> org.pointcross.SearchPermission.MapReduce.NewObjectMapper.map(NewObjectMapper.java:1)
> >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> >> at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> >> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:416)
> >> at
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> >> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException:
> >> Could not find any valid local directory for output/spill934.out
> >> at
> >>
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:381)
> >> at
> >>
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)
> >> at
> >>
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:127)
> >> at
> >>
> org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:121)
> >> at
> >>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1392)
> >> at
> >>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:853)
> >> at
> >>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1344)
> >>
> >>
> >>
> >> I had seen this error for last task what may be the reason for this
> error
> >> .
> >>
> >> NOTE: When i run import hbase table it takes 10 min .
> >>
> >>
> >> Team please give suggestion what to be done to solve these issue .
> >>
> >>
> >> Thanks and Regards,
> >> S SYED ABDUL KATHER
> >>
> >
> >
> >
> > --
> >
> >
> > ∞
> > Shashwat Shriparv
> >
> >
> >
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
