> > You would need to click on the map link on jobdetails.jsp and each task > > will say something like this: > > > > 11337 pages, 3748 errors, 1.7 pages/s, 329 kb/s, > > Okay. Now I see it. > > There are 97 map tasks. > > 96 map tasks state: > 0 threads, 0 pages, 0 errors, 0.0 pages/s, 0 kb/s, > All the above 96 map tasks state the same thing. > > 1 map task states: > 0 threads, 2888 pages, 54 errors, 3.3 pages/s, 1681 kb/s, > > There is definitely a problem here. > > How do I spread out the map tasks to all available slots? > > I found the errors by clicking on the task. > Then, I clicked on the task logs all link. > > It appeared to be running fine, until near the > end, I am seeing error messages like this: > > ----- > -activeThreads=16, spinWaiting=6, fetchQueues.totalSize=800 > fetch of http://www.example.com/public/page.asp/1618613 failed with: > java.lang.OutOfMemoryError: Java heap space > -activeThreads=16, spinWaiting=9, fetchQueues.totalSize=800 > -activeThreads=16, spinWaiting=0, fetchQueues.totalSize=800 > fetch of http://www.example.com/public/page.asp/118288 failed with: > java.lang.OutOfMemoryError: Java heap space > fetch of http://www.example.com/public/page.asp/74343 failed with: > java.lang.OutOfMemoryError: Java heap space > -activeThreads=16, spinWaiting=2, fetchQueues.totalSize=800 > fetch of http://www.example.com/public/page.asp/971779 failed with: > java.lang.OutOfMemoryError: Java heap space > -activeThreads=16, spinWaiting=2, fetchQueues.totalSize=800 > fetch of http://www.example.com/public/page.asp/1585170 failed with: > java.lang.OutOfMemoryError: Java heap space > -activeThreads=16, spinWaiting=1, fetchQueues.totalSize=800 > fetch of http://www.example.com/public/page.asp/82747 failed with: > java.lang.OutOfMemoryError: Java heap space > fetch of http://www.example.com/public/page.asp/687356 failed with: > java.lang.OutOfMemoryError: Java heap space > -activeThreads=16, spinWaiting=0, fetchQueues.totalSize=798 > fetch of http://www.example.com/public/page.asp/1425403 failed with: > java.lang.OutOfMemoryError: Java heap space > fetch of http://www.example.com/public/page.asp/1461659 failed with: > java.lang.OutOfMemoryError: Java heap space > -activeThreads=16, spinWaiting=3, fetchQueues.totalSize=800 > fetch of http://www.example.com/public/page.asp/1484234 failed with: > java.lang.OutOfMemoryError: Java heap space > fetch of http://www.example.com/public/page.asp/1096086 failed with: > java.lang.OutOfMemoryError: Java heap space > -activeThreads=16, spinWaiting=5, fetchQueues.totalSize=800 > fetch of http://www.example.com/public/page.asp/12074 failed with: > java.lang.OutOfMemoryError: Java heap space > fetch of http://www.example.com/public/page.asp/565789 failed with: > java.lang.OutOfMemoryError: Java heap space > fetch of http://www.example.com/public/page.asp/542157 failed with: > java.lang.OutOfMemoryError: Java heap space > fetch of http://www.example.com/public/page.asp/522730 failed with: > java.lang.OutOfMemoryError: Java heap space > fetch of http://www.example.com/public/page.asp/1617438 failed with: > java.lang.OutOfMemoryError: Java heap space > ----- > > How do I resolve this error? > The master has 4gb, this particular slave has 1.5gb. > Another slave has 1gb, another slave has 4gb. > Do I just need to add physical memory? > Or, is this something else in the configuration? > > Is this error the cause of only doing 3% of the 100k > urls I requested to be done? > > Or, is it a problem with the other 96 map tasks not doing > anything? > > Thanks again for all of your help. > > JohnM
Does anyone have any thoughts on how I can begin addressing the issues I am experiencing above? Thanks in advance for any pointers anyone can provide. JohnM -- john mendenhall [EMAIL PROTECTED] surf utopia internet services
