[jira] [Created] (MAPREDUCE-5660) Log info about possible thrashing (when using memory-based scheduling in Capacity Scheduler) is not printed

2013-12-01 Thread Adam Kawa (JIRA) 
Adam Kawa created MAPREDUCE-5660:


 Summary: Log info about possible thrashing (when using 
memory-based scheduling in Capacity Scheduler) is not printed
 Key: MAPREDUCE-5660
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5660
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: capacity-sched, mrv1, tasktracker
Affects Versions: 1.2.1
Reporter: Adam Kawa
Priority: Trivial


There is a tiny, but confusing when troubleshooting, bug in TaskTracker code:
{code}
if (totalMemoryAllottedForTasks  totalPhysicalMemoryOnTT) {
  LOG.info(totalMemoryAllottedForTasks  totalPhysicalMemoryOnTT.
  +  Thrashing might happen.);
} else if (totalMemoryAllottedForTasks  totalVirtualMemoryOnTT) {
  LOG.info(totalMemoryAllottedForTasks  totalVirtualMemoryOnTT.
  +  Thrashing might happen.);
}
{code}
totalMemoryAllottedForTasks is calculated in megabytes, while 
totalPhysicalMemoryOnTT (and totalVirtualMemoryOnTT) is calculated in bytes. 
totalMemoryAllottedForTasks should be converted to bytes for a correct 
comparison.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (MAPREDUCE-5652) ShuffleHandler should handle NM restarts

2013-12-01 Thread Alejandro Abdelnur (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13836088#comment-13836088
 ] 

Alejandro Abdelnur commented on MAPREDUCE-5652:
---

BTW, the {{ShuffleHandler}} is not aware of the cleanup. The clean up is done 
in the {{ResourceLocalizationService.java}} {{serviceInit()}} method.

 ShuffleHandler should handle NM restarts
 

 Key: MAPREDUCE-5652
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5652
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Karthik Kambatla
  Labels: shuffle

 ShuffleHandler should work across NM restarts and not require re-running 
 map-tasks. On NM restart, the map outputs are cleaned up requiring 
 re-execution of map tasks and should be avoided.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (MAPREDUCE-5660) Log info about possible thrashing (when using memory-based scheduling in Capacity Scheduler) is not printed

2013-12-01 Thread Adam Kawa (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Kawa updated MAPREDUCE-5660:
-

Attachment: MAPREDUCE-5660.1.patch.txt

 Log info about possible thrashing (when using memory-based scheduling in 
 Capacity Scheduler) is not printed
 ---

 Key: MAPREDUCE-5660
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5660
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: capacity-sched, mrv1, tasktracker
Affects Versions: 1.2.1
Reporter: Adam Kawa
Priority: Trivial
 Attachments: MAPREDUCE-5660.1.patch.txt


 There is a tiny, but confusing when troubleshooting, bug in TaskTracker code:
 {code}
 if (totalMemoryAllottedForTasks  totalPhysicalMemoryOnTT) {
   LOG.info(totalMemoryAllottedForTasks  totalPhysicalMemoryOnTT.
   +  Thrashing might happen.);
 } else if (totalMemoryAllottedForTasks  totalVirtualMemoryOnTT) {
   LOG.info(totalMemoryAllottedForTasks  totalVirtualMemoryOnTT.
   +  Thrashing might happen.);
 }
 {code}
 totalMemoryAllottedForTasks is calculated in megabytes, while 
 totalPhysicalMemoryOnTT (and totalVirtualMemoryOnTT) is calculated in bytes. 
 totalMemoryAllottedForTasks should be converted to bytes for a correct 
 comparison.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (MAPREDUCE-5660) Log info about possible thrashing (when using memory-based scheduling in Capacity Scheduler) is not printed

2013-12-01 Thread Adam Kawa (JIRA) 

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Kawa updated MAPREDUCE-5660:
-

Status: Patch Available  (was: Open)

 Log info about possible thrashing (when using memory-based scheduling in 
 Capacity Scheduler) is not printed
 ---

 Key: MAPREDUCE-5660
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5660
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: capacity-sched, mrv1, tasktracker
Affects Versions: 1.2.1
Reporter: Adam Kawa
Priority: Trivial
 Attachments: MAPREDUCE-5660.1.patch.txt


 There is a tiny, but confusing when troubleshooting, bug in TaskTracker code:
 {code}
 if (totalMemoryAllottedForTasks  totalPhysicalMemoryOnTT) {
   LOG.info(totalMemoryAllottedForTasks  totalPhysicalMemoryOnTT.
   +  Thrashing might happen.);
 } else if (totalMemoryAllottedForTasks  totalVirtualMemoryOnTT) {
   LOG.info(totalMemoryAllottedForTasks  totalVirtualMemoryOnTT.
   +  Thrashing might happen.);
 }
 {code}
 totalMemoryAllottedForTasks is calculated in megabytes, while 
 totalPhysicalMemoryOnTT (and totalVirtualMemoryOnTT) is calculated in bytes. 
 totalMemoryAllottedForTasks should be converted to bytes for a correct 
 comparison.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (MAPREDUCE-5660) Log info about possible thrashing (when using memory-based scheduling in Capacity Scheduler) is not printed

2013-12-01 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13836136#comment-13836136
 ] 

Hadoop QA commented on MAPREDUCE-5660:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12616489/MAPREDUCE-5660.1.patch.txt
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4234//console

This message is automatically generated.

 Log info about possible thrashing (when using memory-based scheduling in 
 Capacity Scheduler) is not printed
 ---

 Key: MAPREDUCE-5660
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5660
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: capacity-sched, mrv1, tasktracker
Affects Versions: 1.2.1
Reporter: Adam Kawa
Priority: Trivial
 Attachments: MAPREDUCE-5660.1.patch.txt


 There is a tiny, but confusing when troubleshooting, bug in TaskTracker code:
 {code}
 if (totalMemoryAllottedForTasks  totalPhysicalMemoryOnTT) {
   LOG.info(totalMemoryAllottedForTasks  totalPhysicalMemoryOnTT.
   +  Thrashing might happen.);
 } else if (totalMemoryAllottedForTasks  totalVirtualMemoryOnTT) {
   LOG.info(totalMemoryAllottedForTasks  totalVirtualMemoryOnTT.
   +  Thrashing might happen.);
 }
 {code}
 totalMemoryAllottedForTasks is calculated in megabytes, while 
 totalPhysicalMemoryOnTT (and totalVirtualMemoryOnTT) is calculated in bytes. 
 totalMemoryAllottedForTasks should be converted to bytes for a correct 
 comparison.



--
This message was sent by Atlassian JIRA
(v6.1#6144)