[
https://issues.apache.org/jira/browse/HADOOP-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12664066#action_12664066
]
Xing Shi commented on HADOOP-3961:
----------------------------------
double blowupOnThisTask = ts.getOutputSize() /
((double) tip.getMapInputSize() + 1);
for example the tip.getMapInputSize() =0, and the ts.getOutputSize() =15000(I
don't know why it is 15000)
I use streaming -mapper "cat" -reducer "cat".
here is the log:
2009-01-15 12:21:02,182 INFO mapred.ResourceEstimator
(ResourceEstimator.java:updateWithCompletedTask(78)) - measured blowup on tas
k_200901141908_0059_m_002291 was 15000/1 = 15000.0
2009-01-15 12:21:02,182 INFO mapred.ResourceEstimator
(ResourceEstimator.java:updateWithCompletedTask(90)) - new estimate is blowup
= 23.271476813887574
2009-01-15 12:21:02,183 INFO mapred.JobInProgress
(JobInProgress.java:findNewMapTask(1746)) - Choosing a non-local task
task_200901
141908_0059_m_002324
2009-01-15 12:21:02,183 INFO mapred.JobTracker
(JobTracker.java:createTaskEntry(1408)) - Adding task
'attempt_200901141908_0059_m_0
02324_0' to tip task_200901141908_0059_m_002324, for tracker
'tracker_xxxx:localhost.localdomain/127.0.0.1:
50961'
2009-01-15 12:21:02,183 INFO mapred.JobInProgress
(JobInProgress.java:addRunningTaskToTIP(1292)) - Choosing rack-local task
task_20
0901141908_0059_m_002324
2009-01-15 12:21:02,282 INFO mapred.JobInProgress
(JobInProgress.java:completedTask(1910)) - Task
'attempt_200901141908_0059_m_0022
92_0' has completed task_200901141908_0059_m_002292 successfully.
2009-01-15 12:21:02,282 INFO mapred.ResourceEstimator
(ResourceEstimator.java:updateWithCompletedTask(78)) - measured blowup on tas
k_200901141908_0059_m_002292 was 15000/1 = 15000.0
so the blowup is very large.
and then the new estimate newly computed also large, so:
2009-01-15 12:21:20,944 INFO mapred.ResourceEstimator
(ResourceEstimator.java:updateWithCompletedTask(90)) - new estimate is blowup =
819.2585444888223
2009-01-15 12:21:20,945 WARN mapred.JobInProgress
(JobInProgress.java:findNewMapTask(1652)) - No room for map task. Node
/default-rack/xxxx has 132707577856 bytes free; but we expect map to take
345805904529
> resource estimation works badly in some cases
> ---------------------------------------------
>
> Key: HADOOP-3961
> URL: https://issues.apache.org/jira/browse/HADOOP-3961
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: resourceEstFix.patch
>
>
> The disk space estimation introduced in HADOOP-657 performs badly under some
> circumstances. In particular, if maps have very small output, the initial
> estimate of output size equals input size throws everything off.
> Fix is to use a smaller initial estimate, and to update it appropriately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
