date:20130205


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Gorshkov updated MAPREDUCE-4905:


Attachment: MAPREDUCE-4905-trunk-a.patch

 test org.apache.hadoop.mapred.pipes
 ---

 Key: MAPREDUCE-4905
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4905
 Project: Hadoop Map/Reduce
  Issue Type: Test
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
Reporter: Aleksey Gorshkov
 Fix For: 3.0.0, 2.0.3-alpha, 0.23.6

 Attachments: MAPREDUCE-4905-trunk-a.patch, MAPREDUCE-4905-trunk.patch


 tests for  org.apache.hadoop.mapred.pipes
 patch MAPREDUCE-4905-trunk.patch for trunk, branch-2, branch-0.23

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4905) test org.apache.hadoop.mapred.pipes


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571391#comment-13571391
 ] 

Aleksey Gorshkov commented on MAPREDUCE-4905:
-

Ok. I've fixed those comments

 test org.apache.hadoop.mapred.pipes
 ---

 Key: MAPREDUCE-4905
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4905
 Project: Hadoop Map/Reduce
  Issue Type: Test
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
Reporter: Aleksey Gorshkov
 Fix For: 3.0.0, 2.0.3-alpha, 0.23.6

 Attachments: MAPREDUCE-4905-trunk-a.patch, MAPREDUCE-4905-trunk.patch


 tests for  org.apache.hadoop.mapred.pipes
 patch MAPREDUCE-4905-trunk.patch for trunk, branch-2, branch-0.23

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4905) test org.apache.hadoop.mapred.pipes

2013-02-05 Thread Robert Joseph Evans (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571438#comment-13571438
 ] 

Robert Joseph Evans commented on MAPREDUCE-4905:


Thanks for fixing all of my comments.  I do have one more final comment, and 
this is minor.  Inside PipeApplicationStub.java the code is sending a progress.

{code}
// progress
WritableUtils.writeVInt(dataOut, 53);
dataOut.writeFloat(5.5f);
{code}

Progress itself is intended to be between 0.0 and 1.0.  5.5, which equates to 
550% done, seems like a bit much. It would be nice to set it to 0.5 or 
something else more appropriate.

 test org.apache.hadoop.mapred.pipes
 ---

 Key: MAPREDUCE-4905
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4905
 Project: Hadoop Map/Reduce
  Issue Type: Test
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
Reporter: Aleksey Gorshkov
 Fix For: 3.0.0, 2.0.3-alpha, 0.23.6

 Attachments: MAPREDUCE-4905-trunk-a.patch, MAPREDUCE-4905-trunk.patch


 tests for  org.apache.hadoop.mapred.pipes
 patch MAPREDUCE-4905-trunk.patch for trunk, branch-2, branch-0.23

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events

2013-02-05 Thread Jason Lowe (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Lowe updated MAPREDUCE-4822:
--

Status: Open (was: Patch Available)

Hi Chu, thanks for picking this up.

The original intent of the JIRA was to fix more than just the one example
occurrence that Bobby pointed out. Looking through the source, I can see more
occurrences of unnecessary conversions in MapAttemptFinishedEvent and
TaskFinishedEvent.

There's a number of other event classes, like TaskStartedEvent, that are
converting back and forth between strings and enums since they store their
internal state in a single {{datum}} Avro record. We can refactor these to
build the datum on-demand like TaskAttemptFinishedEvent does, but that's
probably best left to another JIRA. We can keep this one short and simple.

Unnecessary conversions in History Events
-

Key: MAPREDUCE-4822
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
Labels: patch
Fix For: 0.24.0

Attachments: MAPREDUCE-4822.patch

There are a number of conversions in the Job History Event classes that are
totally unnecessary. It appears that they were originally used to convert
from the internal avro format, but now many of them do not pull the values
from the avro they store them internally.
For example:
{code:title=TaskAttemptFinishedEvent.java}
/** Get the task type */
public TaskType getTaskType() {
return TaskType.valueOf(taskType.toString());
}
{code}
The code currently is taking an enum, converting it to a string and then
asking the same enum to convert it back to an enum. If java work properly
this should be a noop and a reference to the original taskType should be
returned.
There are several places that a string is having toString called on it, and
since strings are immutable it returns a reference to itself.
The various ids are not immutable and probably should not be changed at this
point.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4905) test org.apache.hadoop.mapred.pipes


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571441#comment-13571441
 ] 

Hadoop QA commented on MAPREDUCE-4905:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12568010/MAPREDUCE-4905-trunk-a.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 6 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3303//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3303//console

This message is automatically generated.

 test org.apache.hadoop.mapred.pipes
 ---

 Key: MAPREDUCE-4905
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4905
 Project: Hadoop Map/Reduce
  Issue Type: Test
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
Reporter: Aleksey Gorshkov
 Fix For: 3.0.0, 2.0.3-alpha, 0.23.6

 Attachments: MAPREDUCE-4905-trunk-a.patch, MAPREDUCE-4905-trunk.patch


 tests for  org.apache.hadoop.mapred.pipes
 patch MAPREDUCE-4905-trunk.patch for trunk, branch-2, branch-0.23

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4967) TestJvmReuse fails on assertion

2013-02-05 Thread Chris Nauroth (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571513#comment-13571513
 ] 

Chris Nauroth commented on MAPREDUCE-4967:
--

+1

I applied the patch, ran {{TestJvmReuse}}, and confirmed that the test gets 
ignored.

I'm wondering if the class should be deleted at this point instead of leaving 
behind a test that's always ignored, or if we need to file a separate follow-up 
jira to figure out why it fails.  The only related jira I could find is 
MAPREDUCE-3480.  Does anyone else out there know additional history on this?


 TestJvmReuse fails on assertion
 ---

 Key: MAPREDUCE-4967
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4967
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker, test
Affects Versions: 1.1.2
Reporter: Chris Nauroth
Assignee: Karthik Kambatla
 Attachments: mr-4967.patch


 {{TestJvmReuse}} on branch-1 consistently fails on an assertion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service

[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571522#comment-13571522
]

Alejandro Abdelnur commented on MAPREDUCE-4049:
---

+1 from my side, regarding passing the ReduceTask, IMO, as discussed before,
these are unstable APIs enabling functionality in an experimental way. Creating
the right abstraction/API will require significant changes and we are not
attempting to do that right now. Because of this, I think is OK. Arun, what is
your take?

Last, while we all 3 agree to commit this to branch-1. Before we do so, I want
to have the discussion in the mapred-dev@ so everybody is on the same page and
to avoid issues now or in the future.

plugin for generic shuffle service
--

Key: MAPREDUCE-4049
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: performance, task, tasktracker
Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
Reporter: Avner BenHanoch
Assignee: Avner BenHanoch
Labels: merge, plugin, rdma, shuffle
Fix For: 2.0.3-alpha

Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf,
MAPREDUCE-4049--branch-1.patch, MAPREDUCE-4049--branch-1.patch,
MAPREDUCE-4049--branch-1.patch, mapreduce-4049.patch

Support generic shuffle service as set of two plugins: ShuffleProvider
ShuffleConsumer.
This will satisfy the following needs:
# Better shuffle and merge performance. For example: we are working on
shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE,
or Infiniband) instead of using the current HTTP shuffle. Based on the fast
RDMA shuffle, the plugin can also utilize a suitable merge approach during
the intermediate merges. Hence, getting much better performance.
# Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden
dependency of NodeManager with a specific version of mapreduce shuffle
(currently targeted to 0.24.0).
References:
# Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu
from Auburn University with others,
[http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
# I am attaching 2 documents with suggested Top Level Design for both plugins
(currently, based on 1.0 branch)
# I am providing link for downloading UDA - Mellanox's open source plugin
that implements generic shuffle service using RDMA and levitated merge.
Note: At this phase, the code is in C++ through JNI and you should consider
it as beta only. Still, it can serve anyone that wants to implement or
contribute to levitated merge. (Please be advised that levitated merge is
mostly suit in very fast networks) -
[http://www.mellanox.com/content/pages.php?pg=products_dynproduct_family=144menu_section=69]

[jira] [Commented] (MAPREDUCE-2264) Job status exceeds 100% in some cases

2013-02-05 Thread Arun C Murthy (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571538#comment-13571538
 ] 

Arun C Murthy commented on MAPREDUCE-2264:
--

If you can attach an addendum patch asap I'd be thrilled... thanks.

 Job status exceeds 100% in some cases 
 --

 Key: MAPREDUCE-2264
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2264
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.20.2, 0.20.205.0
Reporter: Adam Kramer
Assignee: Devaraj K
  Labels: critical-0.22.0
 Fix For: 1.2.0, 2.0.3-alpha

 Attachments: MAPREDUCE-2264-0.20.205-1.patch, 
 MAPREDUCE-2264-0.20.205.patch, MAPREDUCE-2264-0.20.3.patch, 
 MAPREDUCE-2264-branch-1-1.patch, MAPREDUCE-2264-branch-1-2.patch, 
 MAPREDUCE-2264-branch-1.patch, MAPREDUCE-2264-trunk-1.patch, 
 MAPREDUCE-2264-trunk-1.patch, MAPREDUCE-2264-trunk-2.patch, 
 MAPREDUCE-2264-trunk-3.patch, MAPREDUCE-2264-trunk-4.patch, 
 MAPREDUCE-2264-trunk-5.patch, MAPREDUCE-2264-trunk-5.patch, 
 MAPREDUCE-2264-trunk.patch, more than 100%.bmp


 I'm looking now at my jobtracker's list of running reduce tasks. One of them 
 is 120.05% complete, the other is 107.28% complete.
 I understand that these numbers are estimates, but there is no case in which 
 an estimate of 100% for a non-complete task is better than an estimate of 
 99.99%, nor is there any case in which an estimate greater than 100% is valid.
 I suggest that whatever logic is computing these set 99.99% as a hard maximum.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4977) Documentation for pluggable shuffle and pluggable sort

Alejandro Abdelnur created MAPREDUCE-4977:
-

 Summary: Documentation for pluggable shuffle and pluggable sort
 Key: MAPREDUCE-4977
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4977
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.0.3-alpha
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 2.0.3-alpha
 Attachments: MAPREDUCE-4877.patch

Add documentation with basic information on pluggable shuffle and sort.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4977) Documentation for pluggable shuffle and pluggable sort


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated MAPREDUCE-4977:
--

Attachment: MAPREDUCE-4877.patch

modifying documentation only, no code changes.

run site and verified correct generation.

 Documentation for pluggable shuffle and pluggable sort
 --

 Key: MAPREDUCE-4977
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4977
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.0.3-alpha
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-4877.patch


 Add documentation with basic information on pluggable shuffle and sort.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4977) Documentation for pluggable shuffle and pluggable sort


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated MAPREDUCE-4977:
--

Status: Patch Available  (was: Open)

 Documentation for pluggable shuffle and pluggable sort
 --

 Key: MAPREDUCE-4977
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4977
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.0.3-alpha
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-4877.patch


 Add documentation with basic information on pluggable shuffle and sort.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service

[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alejandro Abdelnur updated MAPREDUCE-4049:
--

Release Note: Allow ReduceTask loading a third party plugin for shuffle
(and merge) instead of the default shuffle. (was: Allow ReduceTask loading a
third party plugin for shuffle (and merge) instead of the default shuffle. A
corresponding ShuffleProvider is anyhow allowed to run in the NM as an
AuxiliaryService.
Use new config option: mapreduce.job.reduce.shuffle.consumer.plugin.class -
Name of the class whose instance will be used to send shuffle requests by
reducetasks of this job. The class must be an instance of
org.apache.hadoop.mapred.ShuffleConsumerPlugin.
)

Shortening release note to one liner. Detail documentation being introduced
with MAPREDUCE-4977.

plugin for generic shuffle service
--

Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf,
MAPREDUCE-4049--branch-1.patch, MAPREDUCE-4049--branch-1.patch,
MAPREDUCE-4049--branch-1.patch, mapreduce-4049.patch

[jira] [Commented] (MAPREDUCE-4807) Allow MapOutputBuffer to be pluggable


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571703#comment-13571703
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4807:
---

Asokan, would you please review the documentation, MAPREDUCE-4977, for 
correctness? THX

 Allow MapOutputBuffer to be pluggable
 -

 Key: MAPREDUCE-4807
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4807
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: 2.0.2-alpha
Reporter: Arun C Murthy
Assignee: Mariappan Asokan
 Fix For: 2.0.3-alpha

 Attachments: COMBO-mapreduce-4809-4807.patch, 
 COMBO-mapreduce-4809-4807.patch, COMBO-mapreduce-4809-4807.patch, 
 mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch, 
 mapreduce-4807.patch, mapreduce-4807.patch, mapreduce-4807.patch


 Allow MapOutputBuffer to be pluggable

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571705#comment-13571705
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4049:
---

Avner, would you please review the documentation, MAPREDUCE-4977, for 
correctness? THX

 plugin for generic shuffle service
 --

 Key: MAPREDUCE-4049
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: performance, task, tasktracker
Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
Reporter: Avner BenHanoch
Assignee: Avner BenHanoch
  Labels: merge, plugin, rdma, shuffle
 Fix For: 2.0.3-alpha

 Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, 
 MAPREDUCE-4049--branch-1.patch, MAPREDUCE-4049--branch-1.patch, 
 MAPREDUCE-4049--branch-1.patch, mapreduce-4049.patch


 Support generic shuffle service as set of two plugins: ShuffleProvider  
 ShuffleConsumer.
 This will satisfy the following needs:
 # Better shuffle and merge performance. For example: we are working on 
 shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, 
 or Infiniband) instead of using the current HTTP shuffle. Based on the fast 
 RDMA shuffle, the plugin can also utilize a suitable merge approach during 
 the intermediate merges. Hence, getting much better performance.
 # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden 
 dependency of NodeManager with a specific version of mapreduce shuffle 
 (currently targeted to 0.24.0).
 References:
 # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu 
 from Auburn University with others, 
 [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
 # I am attaching 2 documents with suggested Top Level Design for both plugins 
 (currently, based on 1.0 branch)
 # I am providing link for downloading UDA - Mellanox's open source plugin 
 that implements generic shuffle service using RDMA and levitated merge.  
 Note: At this phase, the code is in C++ through JNI and you should consider 
 it as beta only.  Still, it can serve anyone that wants to implement or 
 contribute to levitated merge. (Please be advised that levitated merge is 
 mostly suit in very fast networks) - 
 [http://www.mellanox.com/content/pages.php?pg=products_dynproduct_family=144menu_section=69]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4977) Documentation for pluggable shuffle and pluggable sort


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571712#comment-13571712
 ] 

Hadoop QA commented on MAPREDUCE-4977:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12568079/MAPREDUCE-4877.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation patch that doesn't require tests.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3304//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3304//console

This message is automatically generated.

 Documentation for pluggable shuffle and pluggable sort
 --

 Key: MAPREDUCE-4977
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4977
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.0.3-alpha
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-4877.patch


 Add documentation with basic information on pluggable shuffle and sort.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4970) Child tasks (try to) create security audit log files


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sandy Ryza updated MAPREDUCE-4970:
--

Attachment: MAPREDUCE-4970.patch

 Child tasks (try to) create security audit log files
 

 Key: MAPREDUCE-4970
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4970.patch


 After HADOOP-8552, MR child tasks will attempt to create security audit log 
 files with their user names.  On an insecure cluster, this has no effect, but 
 on a secure cluster, log4j will try to create log files for tasks with names 
 like SecurityAuth-joeuser.log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4970) Child tasks (try to) create security audit log files


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sandy Ryza updated MAPREDUCE-4970:
--

Status: Patch Available  (was: Open)

 Child tasks (try to) create security audit log files
 

 Key: MAPREDUCE-4970
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4970.patch


 After HADOOP-8552, MR child tasks will attempt to create security audit log 
 files with their user names.  On an insecure cluster, this has no effect, but 
 on a secure cluster, log4j will try to create log files for tasks with names 
 like SecurityAuth-joeuser.log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4970) Child tasks (try to) create security audit log files


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571797#comment-13571797
 ] 

Sandy Ryza commented on MAPREDUCE-4970:
---

This is difficult to write a test for, but I verified it on a 
pseudo-distributed cluster.

 Child tasks (try to) create security audit log files
 

 Key: MAPREDUCE-4970
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4970.patch


 After HADOOP-8552, MR child tasks will attempt to create security audit log 
 files with their user names.  On an insecure cluster, this has no effect, but 
 on a secure cluster, log4j will try to create log files for tasks with names 
 like SecurityAuth-joeuser.log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4970) Child tasks (try to) create security audit log files


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571804#comment-13571804
 ] 

Hadoop QA commented on MAPREDUCE-4970:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12568102/MAPREDUCE-4970.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3305//console

This message is automatically generated.

 Child tasks (try to) create security audit log files
 

 Key: MAPREDUCE-4970
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4970.patch


 After HADOOP-8552, MR child tasks will attempt to create security audit log 
 files with their user names.  On an insecure cluster, this has no effect, but 
 on a secure cluster, log4j will try to create log files for tasks with names 
 like SecurityAuth-joeuser.log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4643) Make job-history cleanup-period configurable


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sandy Ryza updated MAPREDUCE-4643:
--

Attachment: MAPREDUCE-4643-branch-1-3.patch

 Make job-history cleanup-period configurable
 

 Key: MAPREDUCE-4643
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 1.0.3
Reporter: Karthik Kambatla
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4643-branch1-2.patch, 
 MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch1.patch, 
 MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch


 Job history cleanup should be made configurable. Currently, it is set to 1 
 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it 
 to 20 minutes, but it should be configurable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chu Tong updated MAPREDUCE-4822:


Attachment: (was: MAPREDUCE-4822.patch)

 Unnecessary conversions in History Events
 -

 Key: MAPREDUCE-4822
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
  Labels: patch
 Fix For: 0.24.0


 There are a number of conversions in the Job History Event classes that are 
 totally unnecessary.  It appears that they were originally used to convert 
 from the internal avro format, but now many of them do not pull the values 
 from the avro they store them internally.
 For example:
 {code:title=TaskAttemptFinishedEvent.java}
   /** Get the task type */
   public TaskType getTaskType() {
 return TaskType.valueOf(taskType.toString());
   }
 {code}
 The code currently is taking an enum, converting it to a string and then 
 asking the same enum to convert it back to an enum.  If java work properly 
 this should be a noop and a reference to the original taskType should be 
 returned.
 There are several places that a string is having toString called on it, and 
 since strings are immutable it returns a reference to itself.
 The various ids are not immutable and probably should not be changed at this 
 point.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571820#comment-13571820
 ] 

Sandy Ryza commented on MAPREDUCE-4643:
---

uploaded refresh patch that includes test

 Make job-history cleanup-period configurable
 

 Key: MAPREDUCE-4643
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 1.0.3
Reporter: Karthik Kambatla
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4643-branch1-2.patch, 
 MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch1.patch, 
 MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch


 Job history cleanup should be made configurable. Currently, it is set to 1 
 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it 
 to 20 minutes, but it should be configurable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chu Tong updated MAPREDUCE-4822:


Attachment: MAPREDUCE-4822.patch

Changed all affected files under jobhistory directory

 Unnecessary conversions in History Events
 -

 Key: MAPREDUCE-4822
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
  Labels: patch
 Fix For: 0.24.0

 Attachments: MAPREDUCE-4822.patch


 There are a number of conversions in the Job History Event classes that are 
 totally unnecessary.  It appears that they were originally used to convert 
 from the internal avro format, but now many of them do not pull the values 
 from the avro they store them internally.
 For example:
 {code:title=TaskAttemptFinishedEvent.java}
   /** Get the task type */
   public TaskType getTaskType() {
 return TaskType.valueOf(taskType.toString());
   }
 {code}
 The code currently is taking an enum, converting it to a string and then 
 asking the same enum to convert it back to an enum.  If java work properly 
 this should be a noop and a reference to the original taskType should be 
 returned.
 There are several places that a string is having toString called on it, and 
 since strings are immutable it returns a reference to itself.
 The various ids are not immutable and probably should not be changed at this 
 point.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4822) Unnecessary conversions in History Events


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chu Tong updated MAPREDUCE-4822:


Status: Patch Available  (was: Open)

 Unnecessary conversions in History Events
 -

 Key: MAPREDUCE-4822
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
  Labels: patch
 Fix For: 0.24.0

 Attachments: MAPREDUCE-4822.patch


 There are a number of conversions in the Job History Event classes that are 
 totally unnecessary.  It appears that they were originally used to convert 
 from the internal avro format, but now many of them do not pull the values 
 from the avro they store them internally.
 For example:
 {code:title=TaskAttemptFinishedEvent.java}
   /** Get the task type */
   public TaskType getTaskType() {
 return TaskType.valueOf(taskType.toString());
   }
 {code}
 The code currently is taking an enum, converting it to a string and then 
 asking the same enum to convert it back to an enum.  If java work properly 
 this should be a noop and a reference to the original taskType should be 
 returned.
 There are several places that a string is having toString called on it, and 
 since strings are immutable it returns a reference to itself.
 The various ids are not immutable and probably should not be changed at this 
 point.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571828#comment-13571828
 ] 

Hadoop QA commented on MAPREDUCE-4643:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12568109/MAPREDUCE-4643-branch-1-3.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3307//console

This message is automatically generated.

 Make job-history cleanup-period configurable
 

 Key: MAPREDUCE-4643
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 1.0.3
Reporter: Karthik Kambatla
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4643-branch1-2.patch, 
 MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch1.patch, 
 MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch


 Job history cleanup should be made configurable. Currently, it is set to 1 
 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it 
 to 20 minutes, but it should be configurable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable

2013-02-05 Thread Karthik Kambatla (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571835#comment-13571835
 ] 

Karthik Kambatla commented on MAPREDUCE-4643:
-

+1. Thanks Sandy - the latest patch looks good to me.

 Make job-history cleanup-period configurable
 

 Key: MAPREDUCE-4643
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 1.0.3
Reporter: Karthik Kambatla
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4643-branch1-2.patch, 
 MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch1.patch, 
 MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch


 Job history cleanup should be made configurable. Currently, it is set to 1 
 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it 
 to 20 minutes, but it should be configurable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571841#comment-13571841
 ] 

Hadoop QA commented on MAPREDUCE-4822:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12568110/MAPREDUCE-4822.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3306//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3306//console

This message is automatically generated.

 Unnecessary conversions in History Events
 -

 Key: MAPREDUCE-4822
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
  Labels: patch
 Fix For: 0.24.0

 Attachments: MAPREDUCE-4822.patch


 There are a number of conversions in the Job History Event classes that are 
 totally unnecessary.  It appears that they were originally used to convert 
 from the internal avro format, but now many of them do not pull the values 
 from the avro they store them internally.
 For example:
 {code:title=TaskAttemptFinishedEvent.java}
   /** Get the task type */
   public TaskType getTaskType() {
 return TaskType.valueOf(taskType.toString());
   }
 {code}
 The code currently is taking an enum, converting it to a string and then 
 asking the same enum to convert it back to an enum.  If java work properly 
 this should be a noop and a reference to the original taskType should be 
 returned.
 There are several places that a string is having toString called on it, and 
 since strings are immutable it returns a reference to itself.
 The various ids are not immutable and probably should not be changed at this 
 point.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period configurable


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571899#comment-13571899
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4643:
---

patch looks good, 2 things though:

* mapred-default.xml should have the default values as values (instead of empty)
* The TestJobHistory test takes 333secs, this seems a bit too much.

 Make job-history cleanup-period configurable
 

 Key: MAPREDUCE-4643
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 1.0.3
Reporter: Karthik Kambatla
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4643-branch1-2.patch, 
 MAPREDUCE-4643-branch-1-3.patch, MAPREDUCE-4643-branch1.patch, 
 MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch


 Job history cleanup should be made configurable. Currently, it is set to 1 
 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it 
 to 20 minutes, but it should be configurable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4822) Unnecessary conversions in History Events


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571912#comment-13571912
 ] 

Chu Tong commented on MAPREDUCE-4822:
-

No testcase is included as the change is trivial.


 Unnecessary conversions in History Events
 -

 Key: MAPREDUCE-4822
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4822
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 0.23.4
Reporter: Robert Joseph Evans
Assignee: Chu Tong
Priority: Trivial
  Labels: patch
 Fix For: 0.24.0

 Attachments: MAPREDUCE-4822.patch


 There are a number of conversions in the Job History Event classes that are 
 totally unnecessary.  It appears that they were originally used to convert 
 from the internal avro format, but now many of them do not pull the values 
 from the avro they store them internally.
 For example:
 {code:title=TaskAttemptFinishedEvent.java}
   /** Get the task type */
   public TaskType getTaskType() {
 return TaskType.valueOf(taskType.toString());
   }
 {code}
 The code currently is taking an enum, converting it to a string and then 
 asking the same enum to convert it back to an enum.  If java work properly 
 this should be a noop and a reference to the original taskType should be 
 returned.
 There are several places that a string is having toString called on it, and 
 since strings are immutable it returns a reference to itself.
 The various ids are not immutable and probably should not be changed at this 
 point.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4970) Child tasks (try to) create security audit log files


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571919#comment-13571919
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4970:
---

Looks good, we need to update the docs mentioning this file and what is used 
for, in xdocs cluster_setup.xml.

 Child tasks (try to) create security audit log files
 

 Key: MAPREDUCE-4970
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4970.patch


 After HADOOP-8552, MR child tasks will attempt to create security audit log 
 files with their user names.  On an insecure cluster, this has no effect, but 
 on a secure cluster, log4j will try to create log files for tasks with names 
 like SecurityAuth-joeuser.log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-4955) NM container diagnostics for excess resource usage can be lost if task fails while being killed

2013-02-05 Thread Karthik Kambatla (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla reassigned MAPREDUCE-4955:
---

Assignee: Karthik Kambatla

 NM container diagnostics for excess resource usage can be lost if task fails 
 while being killed 
 

 Key: MAPREDUCE-4955
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4955
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am
Affects Versions: 2.0.3-alpha, 0.23.5
Reporter: Jason Lowe
Assignee: Karthik Kambatla

 When a nodemanager kills a container for being over resource budgets, it 
 provides a diagnostics message for the container status explaining why it was 
 killed.  However this message can be lost if the task fails during the 
 shutdown from the SIGTERM (e.g.: lost DFS leases because filesystem closed) 
 and notifies the AM via the task umbilical *before* the AM receives the NM's 
 container status message via the RM heartbeat.
 In that case the task attempt fails with the task's failure diagnostic, and 
 the user is left wondering exactly why the task failed because the NM's 
 diagnostics arrive too late, are not written to the history file, and are 
 lost.  If the AM receives the container status via the RM heartbeat before 
 the task fails during shutdown then the diagnostics are written properly to 
 the history file, and the user can see why the task failed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-2264) Job status exceeds 100% in some cases


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sandy Ryza updated MAPREDUCE-2264:
--

Attachment: MAPREDUCE-2264-trunk-addendum.patch

 Job status exceeds 100% in some cases 
 --

 Key: MAPREDUCE-2264
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2264
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.20.2, 0.20.205.0
Reporter: Adam Kramer
Assignee: Devaraj K
  Labels: critical-0.22.0
 Fix For: 1.2.0, 2.0.3-alpha

 Attachments: MAPREDUCE-2264-0.20.205-1.patch, 
 MAPREDUCE-2264-0.20.205.patch, MAPREDUCE-2264-0.20.3.patch, 
 MAPREDUCE-2264-branch-1-1.patch, MAPREDUCE-2264-branch-1-2.patch, 
 MAPREDUCE-2264-branch-1.patch, MAPREDUCE-2264-trunk-1.patch, 
 MAPREDUCE-2264-trunk-1.patch, MAPREDUCE-2264-trunk-2.patch, 
 MAPREDUCE-2264-trunk-3.patch, MAPREDUCE-2264-trunk-4.patch, 
 MAPREDUCE-2264-trunk-5.patch, MAPREDUCE-2264-trunk-5.patch, 
 MAPREDUCE-2264-trunk-addendum.patch, MAPREDUCE-2264-trunk.patch, more than 
 100%.bmp


 I'm looking now at my jobtracker's list of running reduce tasks. One of them 
 is 120.05% complete, the other is 107.28% complete.
 I understand that these numbers are estimates, but there is no case in which 
 an estimate of 100% for a non-complete task is better than an estimate of 
 99.99%, nor is there any case in which an estimate greater than 100% is valid.
 I suggest that whatever logic is computing these set 99.99% as a hard maximum.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2264) Job status exceeds 100% in some cases


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572030#comment-13572030
 ] 

Sandy Ryza commented on MAPREDUCE-2264:
---

Attached addendum patch

 Job status exceeds 100% in some cases 
 --

 Key: MAPREDUCE-2264
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2264
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.20.2, 0.20.205.0
Reporter: Adam Kramer
Assignee: Devaraj K
  Labels: critical-0.22.0
 Fix For: 1.2.0, 2.0.3-alpha

 Attachments: MAPREDUCE-2264-0.20.205-1.patch, 
 MAPREDUCE-2264-0.20.205.patch, MAPREDUCE-2264-0.20.3.patch, 
 MAPREDUCE-2264-branch-1-1.patch, MAPREDUCE-2264-branch-1-2.patch, 
 MAPREDUCE-2264-branch-1.patch, MAPREDUCE-2264-trunk-1.patch, 
 MAPREDUCE-2264-trunk-1.patch, MAPREDUCE-2264-trunk-2.patch, 
 MAPREDUCE-2264-trunk-3.patch, MAPREDUCE-2264-trunk-4.patch, 
 MAPREDUCE-2264-trunk-5.patch, MAPREDUCE-2264-trunk-5.patch, 
 MAPREDUCE-2264-trunk-addendum.patch, MAPREDUCE-2264-trunk.patch, more than 
 100%.bmp


 I'm looking now at my jobtracker's list of running reduce tasks. One of them 
 is 120.05% complete, the other is 107.28% complete.
 I understand that these numbers are estimates, but there is no case in which 
 an estimate of 100% for a non-complete task is better than an estimate of 
 99.99%, nor is there any case in which an estimate greater than 100% is valid.
 I suggest that whatever logic is computing these set 99.99% as a hard maximum.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4967) TestJvmReuse fails on assertion


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572032#comment-13572032
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4967:
---

+1. Lets open another JIRA to evaluate fix/removal of this test.

 TestJvmReuse fails on assertion
 ---

 Key: MAPREDUCE-4967
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4967
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker, test
Affects Versions: 1.1.2
Reporter: Chris Nauroth
Assignee: Karthik Kambatla
 Attachments: mr-4967.patch


 {{TestJvmReuse}} on branch-1 consistently fails on an assertion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4970) Child tasks (try to) create security audit log files


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sandy Ryza updated MAPREDUCE-4970:
--

Attachment: MAPREDUCE-4970-1.patch
MAPREDUCE-4970.patch

 Child tasks (try to) create security audit log files
 

 Key: MAPREDUCE-4970
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4970-1.patch, MAPREDUCE-4970.patch, 
 MAPREDUCE-4970.patch


 After HADOOP-8552, MR child tasks will attempt to create security audit log 
 files with their user names.  On an insecure cluster, this has no effect, but 
 on a secure cluster, log4j will try to create log files for tasks with names 
 like SecurityAuth-joeuser.log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4970) Child tasks (try to) create security audit log files


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572038#comment-13572038
 ] 

Sandy Ryza commented on MAPREDUCE-4970:
---

Latest patch includes update to cluster_setup.xml

 Child tasks (try to) create security audit log files
 

 Key: MAPREDUCE-4970
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4970-1.patch, MAPREDUCE-4970.patch, 
 MAPREDUCE-4970.patch


 After HADOOP-8552, MR child tasks will attempt to create security audit log 
 files with their user names.  On an insecure cluster, this has no effect, but 
 on a secure cluster, log4j will try to create log files for tasks with names 
 like SecurityAuth-joeuser.log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4967) TestJvmReuse fails on assertion


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated MAPREDUCE-4967:
--

   Resolution: Fixed
Fix Version/s: 1.2.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks Karthik. Committed to branch-1.

 TestJvmReuse fails on assertion
 ---

 Key: MAPREDUCE-4967
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4967
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker, test
Affects Versions: 1.1.2
Reporter: Chris Nauroth
Assignee: Karthik Kambatla
 Fix For: 1.2.0

 Attachments: mr-4967.patch


 {{TestJvmReuse}} on branch-1 consistently fails on an assertion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4970) Child tasks (try to) create security audit log files


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572045#comment-13572045
 ] 

Hadoop QA commented on MAPREDUCE-4970:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12568151/MAPREDUCE-4970-1.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3308//console

This message is automatically generated.

 Child tasks (try to) create security audit log files
 

 Key: MAPREDUCE-4970
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4970-1.patch, MAPREDUCE-4970.patch, 
 MAPREDUCE-4970.patch


 After HADOOP-8552, MR child tasks will attempt to create security audit log 
 files with their user names.  On an insecure cluster, this has no effect, but 
 on a secure cluster, log4j will try to create log files for tasks with names 
 like SecurityAuth-joeuser.log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4434) Backport MR-2779 (JobSplitWriter.java can't handle large job.split file) to branch-1


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572047#comment-13572047
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4434:
---

+1

 Backport MR-2779 (JobSplitWriter.java can't handle large job.split file) to 
 branch-1
 

 Key: MAPREDUCE-4434
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4434
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv1
Affects Versions: 1.1.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4434.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4434) Backport MR-2779 (JobSplitWriter.java can't handle large job.split file) to branch-1


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated MAPREDUCE-4434:
--

   Resolution: Fixed
Fix Version/s: 1.2.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks Karthik. Committed to branch-1.

 Backport MR-2779 (JobSplitWriter.java can't handle large job.split file) to 
 branch-1
 

 Key: MAPREDUCE-4434
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4434
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv1
Affects Versions: 1.1.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Fix For: 1.2.0

 Attachments: mr-4434.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4970) Child tasks (try to) create security audit log files


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572058#comment-13572058
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4970:
---

+1

 Child tasks (try to) create security audit log files
 

 Key: MAPREDUCE-4970
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4970-1.patch, MAPREDUCE-4970.patch, 
 MAPREDUCE-4970.patch


 After HADOOP-8552, MR child tasks will attempt to create security audit log 
 files with their user names.  On an insecure cluster, this has no effect, but 
 on a secure cluster, log4j will try to create log files for tasks with names 
 like SecurityAuth-joeuser.log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4970) Child tasks (try to) create security audit log files


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated MAPREDUCE-4970:
--

   Resolution: Fixed
Fix Version/s: 1.2.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks Sandy. Committed to trunk.

 Child tasks (try to) create security audit log files
 

 Key: MAPREDUCE-4970
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 1.2.0

 Attachments: MAPREDUCE-4970-1.patch, MAPREDUCE-4970.patch, 
 MAPREDUCE-4970.patch


 After HADOOP-8552, MR child tasks will attempt to create security audit log 
 files with their user names.  On an insecure cluster, this has no effect, but 
 on a secure cluster, log4j will try to create log files for tasks with names 
 like SecurityAuth-joeuser.log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (MAPREDUCE-4970) Child tasks (try to) create security audit log files


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572062#comment-13572062
 ] 

Alejandro Abdelnur edited comment on MAPREDUCE-4970 at 2/6/13 1:49 AM:
---

Thanks Sandy. Committed to branch-1.

  was (Author: tucu00):
Thanks Sandy. Committed to trunk.
  
 Child tasks (try to) create security audit log files
 

 Key: MAPREDUCE-4970
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4970
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 1.2.0

 Attachments: MAPREDUCE-4970-1.patch, MAPREDUCE-4970.patch, 
 MAPREDUCE-4970.patch


 After HADOOP-8552, MR child tasks will attempt to create security audit log 
 files with their user names.  On an insecure cluster, this has no effect, but 
 on a secure cluster, log4j will try to create log files for tasks with names 
 like SecurityAuth-joeuser.log.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4143) ApplicationMaster retry times should be set by Client

2013-02-05 Thread Hitesh Shah (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572070#comment-13572070
 ] 

Hitesh Shah commented on MAPREDUCE-4143:


Seems like a reasonable feature to have with a slight caveat that the retry 
limit should be bounded by the limit configured on the RM. A client should not 
be able to set retry limit to 1000 for example.

 ApplicationMaster retry times should be set by Client
 -

 Key: MAPREDUCE-4143
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4143
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: resourcemanager
Affects Versions: 0.23.1
 Environment: suse
Reporter: xieguiming
Priority: Minor

 We should support that different client or user have different 
 ApplicationMaster retry times. It also say that 
 yarn.resourcemanager.am.max-retries should be set by client. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4977) Documentation for pluggable shuffle and pluggable sort

2013-02-05 Thread Mariappan Asokan (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572090#comment-13572090
 ] 

Mariappan Asokan commented on MAPREDUCE-4977:
-

Hi Alejandro,
  Can we change the sentence ... replacing the sort logic with a custom 
algorithms such as Hash-Join or Limit-N query to ... replacing the sort logic 
with custom algorithms that enable Hash aggregation and Limit-N query ?

Thanks.

-- Asokan

PS. The patch file should be MAPREDUCE-4977.patch not MAPREDUCE-4877.patch.  
Sorry for nitpicking:)

 Documentation for pluggable shuffle and pluggable sort
 --

 Key: MAPREDUCE-4977
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4977
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.0.3-alpha
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-4877.patch


 Add documentation with basic information on pluggable shuffle and sort.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4977) Documentation for pluggable shuffle and pluggable sort


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572094#comment-13572094
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4977:
---

thx Asokan, will do. 

 Documentation for pluggable shuffle and pluggable sort
 --

 Key: MAPREDUCE-4977
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4977
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.0.3-alpha
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-4877.patch


 Add documentation with basic information on pluggable shuffle and sort.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4978) Add a updateJobWithSplit() method for new-api job

Liyin Liang created MAPREDUCE-4978:
--

 Summary: Add a updateJobWithSplit() method for new-api job
 Key: MAPREDUCE-4978
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4978
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Liyin Liang
Assignee: Liyin Liang
 Fix For: 1.2.0


HADOOP-1230 adds a method updateJobWithSplit(), which only works for old-api 
job. It's better to add another method for new-api job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4978) Add a updateJobWithSplit() method for new-api job


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liyin Liang updated MAPREDUCE-4978:
---

Fix Version/s: (was: 1.2.0)

 Add a updateJobWithSplit() method for new-api job
 -

 Key: MAPREDUCE-4978
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4978
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Liyin Liang
Assignee: Liyin Liang

 HADOOP-1230 adds a method updateJobWithSplit(), which only works for old-api 
 job. It's better to add another method for new-api job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4978) Add a updateJobWithSplit() method for new-api job


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liyin Liang updated MAPREDUCE-4978:
---

Attachment: 4978-1.diff

 Add a updateJobWithSplit() method for new-api job
 -

 Key: MAPREDUCE-4978
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4978
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 1.1.1
Reporter: Liyin Liang
Assignee: Liyin Liang
 Attachments: 4978-1.diff


 HADOOP-1230 adds a method updateJobWithSplit(), which only works for old-api 
 job. It's better to add another method for new-api job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4978) Add a updateJobWithSplit() method for new-api job


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liyin Liang updated MAPREDUCE-4978:
---

Affects Version/s: 1.1.1

 Add a updateJobWithSplit() method for new-api job
 -

 Key: MAPREDUCE-4978
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4978
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 1.1.1
Reporter: Liyin Liang
Assignee: Liyin Liang
 Attachments: 4978-1.diff


 HADOOP-1230 adds a method updateJobWithSplit(), which only works for old-api 
 job. It's better to add another method for new-api job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4978) Add a updateJobWithSplit() method for new-api job


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572109#comment-13572109
 ] 

Liyin Liang commented on MAPREDUCE-4978:


The attached patch add a new method updateJobWithSplit() only for new-api job. 
This patch also fixed MAPREDUCE-1743.

 Add a updateJobWithSplit() method for new-api job
 -

 Key: MAPREDUCE-4978
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4978
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 1.1.1
Reporter: Liyin Liang
Assignee: Liyin Liang
 Attachments: 4978-1.diff


 HADOOP-1230 adds a method updateJobWithSplit(), which only works for old-api 
 job. It's better to add another method for new-api job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4979) TestJvmReuse must be fixed or removed from branch-1

2013-02-05 Thread Chris Nauroth (JIRA)

Chris Nauroth created MAPREDUCE-4979:


 Summary: TestJvmReuse must be fixed or removed from branch-1
 Key: MAPREDUCE-4979
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4979
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Affects Versions: 1.2.0
Reporter: Chris Nauroth


{{TestJvmReuse}} on branch-1 is currently ignored.  If the test is not marked 
ignored, then it fails.  It's unclear if the test failure indicates that we 
need to fix a bug or if the test is irrelevant and needs to be deleted.  This 
jira tracks either fixing the test or removing it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4967) TestJvmReuse fails on assertion

2013-02-05 Thread Chris Nauroth (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572135#comment-13572135
 ] 

Chris Nauroth commented on MAPREDUCE-4967:
--

I submitted MAPREDUCE-4979 for follow-up.

 TestJvmReuse fails on assertion
 ---

 Key: MAPREDUCE-4967
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4967
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker, test
Affects Versions: 1.1.2
Reporter: Chris Nauroth
Assignee: Karthik Kambatla
 Fix For: 1.2.0

 Attachments: mr-4967.patch


 {{TestJvmReuse}} on branch-1 consistently fails on an assertion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-02-05 Thread Tsuyoshi OZAWA (JIRA)

Tsuyoshi OZAWA created MAPREDUCE-4980:
-

 Summary: Parallel test execution of hadoop-mapreduce-client-core
 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA


The maven surefire plugin supports parallel testing feature. By using it, the 
tests can be run more faster.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core

2013-02-05 Thread Tsuyoshi OZAWA (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi OZAWA updated MAPREDUCE-4980:
--

Attachment: MAPREDUCE-4980.1.patch

Attached the patch.

 Parallel test execution of hadoop-mapreduce-client-core
 ---

 Key: MAPREDUCE-4980
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
 Attachments: MAPREDUCE-4980.1.patch


 The maven surefire plugin supports parallel testing feature. By using it, the 
 tests can be run more faster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4905) test org.apache.hadoop.mapred.pipes


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Gorshkov updated MAPREDUCE-4905:


Attachment: MAPREDUCE-4905-trunk-b.patch

 test org.apache.hadoop.mapred.pipes
 ---

 Key: MAPREDUCE-4905
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4905
 Project: Hadoop Map/Reduce
  Issue Type: Test
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
Reporter: Aleksey Gorshkov
 Fix For: 3.0.0, 2.0.3-alpha, 0.23.6

 Attachments: MAPREDUCE-4905-trunk-a.patch, 
 MAPREDUCE-4905-trunk-b.patch, MAPREDUCE-4905-trunk.patch


 tests for  org.apache.hadoop.mapred.pipes
 patch MAPREDUCE-4905-trunk.patch for trunk, branch-2, branch-0.23

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4905) test org.apache.hadoop.mapred.pipes