date:20121207


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526506#comment-13526506
 ] 

Arun C Murthy commented on MAPREDUCE-4808:
--

On second thoughts, a simpler solution: why don't we use a simplified version 
of Merger as the interface?

This way Syncsort can just implement that, why bother with trying to deal with 
memory reservation etc.?

 Allow reduce-side merge to be pluggable
 ---

 Key: MAPREDUCE-4808
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4808
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: 2.0.2-alpha
Reporter: Arun C Murthy
Assignee: Mariappan Asokan
 Fix For: 2.0.3-alpha

 Attachments: COMBO-mapreduce-4809-4812-4808.patch, 
 mapreduce-4808.patch


 Allow reduce-side merge to be pluggable for MAPREDUCE-2454

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service

[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526516#comment-13526516
]

Arun C Murthy commented on MAPREDUCE-4049:
--

[~tucu00] I'm confused.

MAPREDUCE-2454 was a *huge* patch by a different contributor which got broken
up to aid through reviews.

MAPREDUCE-4049 is, effectively, a trivial patch from another contributor after
he (Avner) has very patiently taken in all feedback. We should be thankful.

I don't see why we need to block Avner's work on MAPREDUCE-2454. Furthermore,
Avner has made it crystal clear that he has issues working with people working
on MAPREDUCE-2454 (see http://s.apache.org/MRT, http://s.apache.org/6bh,
http://s.apache.org/fR4). Why coerce him?

Do you have any technical reason to revert the commit? Else, we can close this
discussion.

plugin for generic shuffle service
--

Key: MAPREDUCE-4049
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: performance, task, tasktracker
Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
Reporter: Avner BenHanoch
Assignee: Avner BenHanoch
Labels: merge, plugin, rdma, shuffle
Fix For: 3.0.0

Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch

Support generic shuffle service as set of two plugins: ShuffleProvider
ShuffleConsumer.
This will satisfy the following needs:
# Better shuffle and merge performance. For example: we are working on
shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE,
or Infiniband) instead of using the current HTTP shuffle. Based on the fast
RDMA shuffle, the plugin can also utilize a suitable merge approach during
the intermediate merges. Hence, getting much better performance.
# Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden
dependency of NodeManager with a specific version of mapreduce shuffle
(currently targeted to 0.24.0).
References:
# Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu
from Auburn University with others,
[http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
# I am attaching 2 documents with suggested Top Level Design for both plugins
(currently, based on 1.0 branch)
# I am providing link for downloading UDA - Mellanox's open source plugin
that implements generic shuffle service using RDMA and levitated merge.
Note: At this phase, the code is in C++ through JNI and you should consider
it as beta only. Still, it can serve anyone that wants to implement or
contribute to levitated merge. (Please be advised that levitated merge is
mostly suit in very fast networks) -
[http://www.mellanox.com/content/pages.php?pg=products_dynproduct_family=144menu_section=69]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service

[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526565#comment-13526565
]

Alejandro Abdelnur commented on MAPREDUCE-4049:
---

Arun, focusing on the technical side of your comments. My reasons to revert the
patch from trunk are:

All these components are highly interrelated as you know.

During the review of MAPREDUCE-4049 we found inconsistencies in the naming and
we aligned them with the other sub-tasks. We may need to do some more of that.
This was your motivation to create MAPREDUCE-2454 branch after a similar
comment I've made in MAPREDUCE-4809.

You want to have gridmix runs in a reasonable size cluster to ensure there are
not performance degradation due to the subtasks of MAPREDUCE-2454. I don' t see
why MAPREDUCE-4049 should be excluded from those tests. Personally I think this
is not needed for any of the patches as a change from 'new' to
'ReflectionUtils.newInstance()' outside of the processing loop cannot affect
things, but you strongly asked me for this over the phone.

Thus, I think your 'requirements' for the other tasks to MAPREDUCE-2454 do also
apply to MAPREDUCE-4049 and until they are satisfied, MAPREDUCE-2454 is not
ready for going to trunk.

Said this, again, please revert. I'm confident we can do a last push and get
the branch MAPREDUCE-2454 merge into trunk at fast pace.

plugin for generic shuffle service
--

Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service

[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526579#comment-13526579
]

Arun C Murthy commented on MAPREDUCE-4049:
--

I'm getting tired of the lawyering.

MAPREDUCE-2454 is a couple of orders of magnitude larger than MAPREDUCE-4049.

Anyway to stop wasting my time arguing, I just started a 300-node gridmix run
with MAPREDUCE-4049, I'll report back by eod.

plugin for generic shuffle service
--

Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch

[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun C Murthy updated MAPREDUCE-4049:
-

Issue Type: Improvement  (was: Sub-task)
Parent: (was: MAPREDUCE-2454)

 plugin for generic shuffle service
 --

 Key: MAPREDUCE-4049
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: performance, task, tasktracker
Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
Reporter: Avner BenHanoch
Assignee: Avner BenHanoch
  Labels: merge, plugin, rdma, shuffle
 Fix For: 3.0.0

 Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, 
 mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch, 
 mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch


 Support generic shuffle service as set of two plugins: ShuffleProvider  
 ShuffleConsumer.
 This will satisfy the following needs:
 # Better shuffle and merge performance. For example: we are working on 
 shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, 
 or Infiniband) instead of using the current HTTP shuffle. Based on the fast 
 RDMA shuffle, the plugin can also utilize a suitable merge approach during 
 the intermediate merges. Hence, getting much better performance.
 # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden 
 dependency of NodeManager with a specific version of mapreduce shuffle 
 (currently targeted to 0.24.0).
 References:
 # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu 
 from Auburn University with others, 
 [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
 # I am attaching 2 documents with suggested Top Level Design for both plugins 
 (currently, based on 1.0 branch)
 # I am providing link for downloading UDA - Mellanox's open source plugin 
 that implements generic shuffle service using RDMA and levitated merge.  
 Note: At this phase, the code is in C++ through JNI and you should consider 
 it as beta only.  Still, it can serve anyone that wants to implement or 
 contribute to levitated merge. (Please be advised that levitated merge is 
 mostly suit in very fast networks) - 
 [http://www.mellanox.com/content/pages.php?pg=products_dynproduct_family=144menu_section=69]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service

[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526598#comment-13526598
]

Alejandro Abdelnur commented on MAPREDUCE-4049:
---

Sounds good on the 300 node gridmix test, mind running a full MR-2454 patch if
Asokan posts a updated patch soon?

Still this is not addressing:

bq. During the review of MAPREDUCE-4049 we found inconsistencies in the naming
and we aligned them with the other sub-tasks. We may need to do some more of
that. This was your motivation to create MAPREDUCE-2454 branch after a similar
comment I've made in MAPREDUCE-4809.

And this was significant enough for you to create MR-2454 branch.

So again, revert MAPREDUCE-4049 from trunk until we iron out the whole branch.

PS: the common ratio for order of magnitude is 10. I think you are a bit off
with your comment. Lets keep things objective.

plugin for generic shuffle service
--

Key: MAPREDUCE-4049
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: performance, task, tasktracker
Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
Reporter: Avner BenHanoch
Assignee: Avner BenHanoch
Labels: merge, plugin, rdma, shuffle
Fix For: 3.0.0

Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch

[jira] [Created] (MAPREDUCE-4859) TestRecoveryManager fails on branch-1

Arun C Murthy created MAPREDUCE-4859:


 Summary: TestRecoveryManager fails on branch-1
 Key: MAPREDUCE-4859
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4859
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Arun C Murthy
Assignee: Arun C Murthy
 Fix For: 1.1.2


Looks like the tests are extremely flaky and just hang.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4858) TestWebUIAuthorization fails on branch-1


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526648#comment-13526648
 ] 

Arun C Murthy commented on MAPREDUCE-4858:
--

FYI - I had to add a minor addendum to fix the merge since HADOOP-8988 isn't in 
branch-1.1. My bad.


{noformat}
Index: src/test/org/apache/hadoop/mapred/TestWebUIAuthorization.java
===
--- src/test/org/apache/hadoop/mapred/TestWebUIAuthorization.java   
(revision 1418203)
+++ src/test/org/apache/hadoop/mapred/TestWebUIAuthorization.java   
(working copy)
@@ -782,9 +782,6 @@
 true);
 props.setProperty(PolicyProvider.POLICY_PROVIDER_CONFIG, 
 HadoopPolicyProvider.class.getName());
-props.setProperty(
-
CommonConfigurationKeys.HADOOP_SECURITY_INSTRUMENTATION_REQUIRES_ADMIN, 
-true);
 props.setProperty(JobConf.MR_ADMINS, mrAdminUser +   + mrAdminGroup);
 
 startCluster(true, props);
{noformat}

 TestWebUIAuthorization fails on branch-1
 

 Key: MAPREDUCE-4858
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4858
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Arun C Murthy
Assignee: Arun C Murthy
 Fix For: 1.1.2

 Attachments: MAPREDUCE-4858.patch, MAPREDUCE-4858.patch


 TestWebUIAuthorization fails on branch-1

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service

[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526653#comment-13526653
]

Alejandro Abdelnur commented on MAPREDUCE-4049:
---

Arun, until we address this, are you reverting the patch? Or I'll have to do it?

plugin for generic shuffle service
--

Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service

[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526677#comment-13526677
]

Arun C Murthy commented on MAPREDUCE-4049:
--

Alejandro - you committed the patch. I'm confused. We can change them via
MAPREDUCE-2454 if you are pedantic, or file a follow on issue.

I've already started the runs, I'm not going to be able to get cluster time to
re-run them.

plugin for generic shuffle service
--

Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service

[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526684#comment-13526684
]

Arun C Murthy commented on MAPREDUCE-4049:
--

I'm done discussing. Can we please move on? File a follow on issue for naming
nits.

plugin for generic shuffle service
--

Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch,
mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch

[jira] [Created] (MAPREDUCE-4860) Inconsistent synchronization in mapreduce.security.token.DelegationTokenRenewal

Karthik Kambatla created MAPREDUCE-4860:
---

 Summary: Inconsistent synchronization in 
mapreduce.security.token.DelegationTokenRenewal
 Key: MAPREDUCE-4860
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4860
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: security
Affects Versions: 2.0.2-alpha, 1.1.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla


mapreduce.security.token.DelegationTokenRenewal synchronizes on 
removeDelegationToken, but fails to synchronize on addToken, and renewing 
tokens in run().



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4860) Inconsistent synchronization in mapreduce.security.token.DelegationTokenRenewal


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526697#comment-13526697
 ] 

Karthik Kambatla commented on MAPREDUCE-4860:
-

In fact, I wonder why we can't use common.DelegationTokenRenewer for token 
renewal. That avoids duplicating code too.

 Inconsistent synchronization in 
 mapreduce.security.token.DelegationTokenRenewal
 ---

 Key: MAPREDUCE-4860
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4860
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: security
Affects Versions: 1.1.1, 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla

 mapreduce.security.token.DelegationTokenRenewal synchronizes on 
 removeDelegationToken, but fails to synchronize on addToken, and renewing 
 tokens in run().

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4860) Inconsistent synchronization in mapreduce.security.token.DelegationTokenRenewal


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4860:


Description: 
mapreduce.security.token.DelegationTokenRenewal synchronizes on 
removeDelegationToken, but fails to synchronize on addToken, and renewing 
tokens in run().

This inconsistency is exposed by frequent failures of 
TestDelegationTokenRenewal:
{noformat}
Error Message

renew wasn't called as many times as expected expected:4 but was:5
Stacktrace

junit.framework.AssertionFailedError: renew wasn't called as many times as 
expected expected:4 but was:5
at 
org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewal(TestDelegationTokenRenewal.java:317)
at 
org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewalAfterClose(TestDelegationTokenRenewal.java:338)

{noformat}


  was:
mapreduce.security.token.DelegationTokenRenewal synchronizes on 
removeDelegationToken, but fails to synchronize on addToken, and renewing 
tokens in run().




 Inconsistent synchronization in 
 mapreduce.security.token.DelegationTokenRenewal
 ---

 Key: MAPREDUCE-4860
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4860
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: security
Affects Versions: 1.1.1, 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla

 mapreduce.security.token.DelegationTokenRenewal synchronizes on 
 removeDelegationToken, but fails to synchronize on addToken, and renewing 
 tokens in run().
 This inconsistency is exposed by frequent failures of 
 TestDelegationTokenRenewal:
 {noformat}
 Error Message
 renew wasn't called as many times as expected expected:4 but was:5
 Stacktrace
 junit.framework.AssertionFailedError: renew wasn't called as many times as 
 expected expected:4 but was:5
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewal(TestDelegationTokenRenewal.java:317)
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewalAfterClose(TestDelegationTokenRenewal.java:338)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4812) Create reduce input merger plugin in ReduceTask.java and pass it to Shuffle


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526707#comment-13526707
 ] 

Arun C Murthy commented on MAPREDUCE-4812:
--

Alejandro - since you +1ed this jira, can you pls explain the rationale for the 
apis I've asked in MAPREDUCE-4808?

Maybe you and Asokan had an offline conversation?

 Create reduce input merger plugin in ReduceTask.java and pass it to Shuffle
 ---

 Key: MAPREDUCE-4812
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4812
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: 2.0.2-alpha
Reporter: Mariappan Asokan
Assignee: Mariappan Asokan
 Fix For: 2.0.3-alpha

 Attachments: COMBO-mapreduce-4809-4812.patch, 
 COMBO-mapreduce-4809-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch, 
 mapreduce-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch


 This is part of MAPREDUCE-2454.  This further breaks down MAPREDUCE-4808

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4859) TestRecoveryManager fails on branch-1


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526721#comment-13526721
 ] 

Arun C Murthy commented on MAPREDUCE-4859:
--

Sigh, I give up.

TestRecoveryManager is hopeless. Mainly in the sense that it uses the 
confounded UtilsForTests which are broken.

testJobTrackerRestartsWithMissingJobFile  testJobTrackerRestartWithBadJobs 
*hang* on both Linux and MacOSX.
testJobResubmission works on MacOSX and hangs on Linux similar to the other two.

I managed to track and fix one bug in testJobTrackerInfoCreation.

I'll ignore them for 1.1.2 (sad to have a stable release with unit-test 
failures due to flaky test code) so we can revisit them.

 TestRecoveryManager fails on branch-1
 -

 Key: MAPREDUCE-4859
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4859
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Arun C Murthy
Assignee: Arun C Murthy
 Fix For: 1.1.2

 Attachments: MAPREDUCE-4859.patch


 Looks like the tests are extremely flaky and just hang.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4859) TestRecoveryManager fails on branch-1


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun C Murthy updated MAPREDUCE-4859:
-

Attachment: MAPREDUCE-4859.patch

Fixed testJobTrackerInfoCreation, ignored others after converting them to 
junit4.

 TestRecoveryManager fails on branch-1
 -

 Key: MAPREDUCE-4859
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4859
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Arun C Murthy
Assignee: Arun C Murthy
 Fix For: 1.1.2

 Attachments: MAPREDUCE-4859.patch


 Looks like the tests are extremely flaky and just hang.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4859) TestRecoveryManager fails on branch-1

2012-12-07 Thread Matt Foley (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526725#comment-13526725
 ] 

Matt Foley commented on MAPREDUCE-4859:
---

+1.  Please commit to branch-1 and branch-1.1.  Thanks, Arun!

 TestRecoveryManager fails on branch-1
 -

 Key: MAPREDUCE-4859
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4859
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Arun C Murthy
Assignee: Arun C Murthy
 Fix For: 1.1.2

 Attachments: MAPREDUCE-4859.patch


 Looks like the tests are extremely flaky and just hang.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-4859) TestRecoveryManager fails on branch-1


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun C Murthy resolved MAPREDUCE-4859.
--

Resolution: Fixed

I just committed this. Thanks for the quick check Matt!

 TestRecoveryManager fails on branch-1
 -

 Key: MAPREDUCE-4859
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4859
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.1.1
Reporter: Arun C Murthy
Assignee: Arun C Murthy
 Fix For: 1.1.2

 Attachments: MAPREDUCE-4859.patch


 Looks like the tests are extremely flaky and just hang.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526773#comment-13526773
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4049:
---

My +1 was for the work in the branch.

My rationale it only echoing yours:

https://issues.apache.org/jira/browse/MAPREDUCE-4809?focusedCommentId=13501245page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13501245

 Arun C Murthy added a comment - 20/Nov/12 07:50
 I've also created a MR-2454 branch in svn, let's commit to that branch first.
 This way we can change our mind before we do the final merge if necessary.

MAPREDUCE-4812 will require some changes in MAPREDUCE-4049

So, revert from trunk, thx.

 plugin for generic shuffle service
 --

 Key: MAPREDUCE-4049
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: performance, task, tasktracker
Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
Reporter: Avner BenHanoch
Assignee: Avner BenHanoch
  Labels: merge, plugin, rdma, shuffle
 Fix For: 3.0.0

 Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, 
 mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch, 
 mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch


 Support generic shuffle service as set of two plugins: ShuffleProvider  
 ShuffleConsumer.
 This will satisfy the following needs:
 # Better shuffle and merge performance. For example: we are working on 
 shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, 
 or Infiniband) instead of using the current HTTP shuffle. Based on the fast 
 RDMA shuffle, the plugin can also utilize a suitable merge approach during 
 the intermediate merges. Hence, getting much better performance.
 # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden 
 dependency of NodeManager with a specific version of mapreduce shuffle 
 (currently targeted to 0.24.0).
 References:
 # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu 
 from Auburn University with others, 
 [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
 # I am attaching 2 documents with suggested Top Level Design for both plugins 
 (currently, based on 1.0 branch)
 # I am providing link for downloading UDA - Mellanox's open source plugin 
 that implements generic shuffle service using RDMA and levitated merge.  
 Note: At this phase, the code is in C++ through JNI and you should consider 
 it as beta only.  Still, it can serve anyone that wants to implement or 
 contribute to levitated merge. (Please be advised that levitated merge is 
 mostly suit in very fast networks) - 
 [http://www.mellanox.com/content/pages.php?pg=products_dynproduct_family=144menu_section=69]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-12-07 Thread Kihwal Lee (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526775#comment-13526775
]

Kihwal Lee commented on MAPREDUCE-1700:
---

{quote}
bq. Tom, one thing I've forgot to mention in my previous comment, we should see
how to enable the classloader on the client side as well as it may be required
(to use different JARs) for the submission code.

I think this is a slightly different problem, since users generally have more
control over the JVM they submit from than the JVM the task runs in. So, yes,
another JIRA would be appropriate.
{quote}

I think AM also runs user code, if a custom output format is defined.

User supplied dependencies may conflict with MapReduce system JARs
--

Key: MAPREDUCE-1700
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Reporter: Tom White
Assignee: Tom White
Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch,
MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch,
MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch

If user code has a dependency on a version of a JAR that is different to the
one that happens to be used by Hadoop, then it may not work correctly. This
happened with user code using a different version of Avro, as reported
[here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
The problem is analogous to the one that application servers have with WAR
loading. Using a specialized classloader in the Child JVM is probably the way
to solve this.

[jira] [Commented] (MAPREDUCE-4812) Create reduce input merger plugin in ReduceTask.java and pass it to Shuffle


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526783#comment-13526783
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4812:
---


MAPREDUCE-4812 (this JIRA) makes the MergerManager is made pluggable, nothing 
else.

In the current code, in the local case where Shuffle is not involved, the 
MergerManager is not being used; instead the Merger class is used directly.

With MAPREDUCE-4808, the MergerManager is augmented to also handle the local 
case by adding a new method and moving the Merger.merge() invocation to it.

MAPREDUCE-4808 introduces the merge pluggability for the local case when 
shuffle is not in the picture.

My understanding is that Asokan split the pluggability of MergeManager from 
augmenting its  functionality to handle the local case to keep the changes 
focused.

I'm OK with folding both in a single JIRA if you think it makes more sense..



 Create reduce input merger plugin in ReduceTask.java and pass it to Shuffle
 ---

 Key: MAPREDUCE-4812
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4812
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: 2.0.2-alpha
Reporter: Mariappan Asokan
Assignee: Mariappan Asokan
 Fix For: 2.0.3-alpha

 Attachments: COMBO-mapreduce-4809-4812.patch, 
 COMBO-mapreduce-4809-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch, 
 mapreduce-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch


 This is part of MAPREDUCE-2454.  This further breaks down MAPREDUCE-4808

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4795) TestDelegationTokenRenewal should not use static variables in Renewer


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4795:


Resolution: Not A Problem
Status: Resolved  (was: Patch Available)

Spoke to Sandy offline about this: the static variables in renewer don't seem 
to be causing the test failures we noticed.

 TestDelegationTokenRenewal should not use static variables in Renewer
 -

 Key: MAPREDUCE-4795
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4795
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 1.0.4
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-4795-branch-1.patch


 TestDelegationTokenRenewal uses static variables to access what's going on 
 inside its Renewer class, making it so problems can occur if the tests are 
 run in parallel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4808) Allow reduce-side merge to be pluggable


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526795#comment-13526795
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4808:
---

Arun, what do you exactly mean?

Both Merger and MergeManager are classes. 

The reducer side of logic in the Merger class is used only in 2 places, within 
the MergeManager (used by the Shuffle) and in the ReduceTask (for the local 
case).

This patch is moving  the later use into the MergeManager as well. By doing 
this all reduce merge logic, for the local and the distributed case, is 
encapsulated in the MergerManager.

Then, when an alternate implementation is provided, it can handle both cases, 
local and distributed case.

The resulting interface being introduced as ReduceInputMerger is quite simple:

{code}
  public void init(ContextK, V reduceMergerContext);
  public void waitForResource() throws InterruptedException;
  public MapOutputK, V reserve(TaskAttemptID mapId, long requestedSize,
 int fetcher) throws IOException;
  public RawKeyValueIterator close() throws Throwable;

  // To merge files created for a local job.
  public RawKeyValueIterator mergeLocalFiles(Path localFiles[])
throws IOException;
{code}

I think this is much simpler than trying modify things in the Merger, given 
that the merger is not directly used by the Shuffle, but through the 
MergeManager.


 Allow reduce-side merge to be pluggable
 ---

 Key: MAPREDUCE-4808
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4808
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: 2.0.2-alpha
Reporter: Arun C Murthy
Assignee: Mariappan Asokan
 Fix For: 2.0.3-alpha

 Attachments: COMBO-mapreduce-4809-4812-4808.patch, 
 mapreduce-4808.patch


 Allow reduce-side merge to be pluggable for MAPREDUCE-2454

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4812) Create reduce input merger plugin in ReduceTask.java and pass it to Shuffle

2012-12-07 Thread Mariappan Asokan (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526802#comment-13526802
 ] 

Mariappan Asokan commented on MAPREDUCE-4812:
-

Hi Arun,
  Sorry I did not get back sooner.  The intention of {{ReduceInputMerger}} 
interface is to have a pluggable {{MergeManager}} implementation.  For a 
non-local job, {{Shuffle}} and {{MergeManager}} interact and synchronize with 
each other using the three methods {{waitForInMemoryMerge(),}} {{reserve(),}} 
and {{close()}}.  So in order to use the {{Shuffle}} these methods are captured 
in {{ReduceInputMerger}} interface.  I renamed {{waitForInMemoryMerge()}} to a 
generic name {{waitForResource()}} since the plugin implementation may not have 
the concept of in-memory merge.
Since the return value from {{reserve()}} is {{MapOutput}}, I did some 
refactoring of {{MapOutput}} so that plugin can return its own implementation 
of it.  I kept the refactoring done on {{MapOutput}} in MAPREDUCE-4808.  With 
just MAPREDUCE-4812, an external plugin is not possible, but it has the core 
part of the concepts so that it is easy to review just {{ReduceInputMerger}} 
design.  Similarly, for a local job the input is coming from local files.  I 
enhanced {{ReduceInputMerger}} with one more method for this.  It is also kept 
in MAPREDUCE-4808.

Hope I explained well.  Please let me know if you have any more questions.

Thanks.

-- Asokan


 Create reduce input merger plugin in ReduceTask.java and pass it to Shuffle
 ---

 Key: MAPREDUCE-4812
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4812
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Affects Versions: 2.0.2-alpha
Reporter: Mariappan Asokan
Assignee: Mariappan Asokan
 Fix For: 2.0.3-alpha

 Attachments: COMBO-mapreduce-4809-4812.patch, 
 COMBO-mapreduce-4809-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch, 
 mapreduce-4812.patch, mapreduce-4812.patch, mapreduce-4812.patch


 This is part of MAPREDUCE-2454.  This further breaks down MAPREDUCE-4808

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4860) Inconsistent synchronization in mapreduce.security.token.DelegationTokenRenewal


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4860:


Affects Version/s: (was: 2.0.2-alpha)

 Inconsistent synchronization in 
 mapreduce.security.token.DelegationTokenRenewal
 ---

 Key: MAPREDUCE-4860
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4860
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: security
Affects Versions: 1.1.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla

 mapreduce.security.token.DelegationTokenRenewal synchronizes on 
 removeDelegationToken, but fails to synchronize on addToken, and renewing 
 tokens in run().
 This inconsistency is exposed by frequent failures of 
 TestDelegationTokenRenewal:
 {noformat}
 Error Message
 renew wasn't called as many times as expected expected:4 but was:5
 Stacktrace
 junit.framework.AssertionFailedError: renew wasn't called as many times as 
 expected expected:4 but was:5
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewal(TestDelegationTokenRenewal.java:317)
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewalAfterClose(TestDelegationTokenRenewal.java:338)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4861) Cleanup: mapreduce.security.token.DelegationTokenRenewal is not used in branch-2. Can remove it.

Karthik Kambatla created MAPREDUCE-4861:
---

 Summary: Cleanup: mapreduce.security.token.DelegationTokenRenewal 
is not used in branch-2. Can remove it.
 Key: MAPREDUCE-4861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla


mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in 
branch-2 at all. grep on trunk yields no results, not even ReflectionUtils 
related suff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4861) Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4861:


Summary: Cleanup: Remove unused 
mapreduce.security.token.DelegationTokenRenewal  (was: Cleanup: 
mapreduce.security.token.DelegationTokenRenewal is not used in branch-2. Can 
remove it.)

 Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
 --

 Key: MAPREDUCE-4861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4861.patch


 mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in 
 branch-2 at all. grep on trunk yields no results, not even ReflectionUtils 
 related suff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4861) Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4861:


Status: Patch Available  (was: Open)

 Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
 --

 Key: MAPREDUCE-4861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4861.patch


 mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in 
 branch-2 at all. grep on trunk yields no results, not even ReflectionUtils 
 related suff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4861) Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4861:


Attachment: mr-4861.patch

 Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
 --

 Key: MAPREDUCE-4861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4861.patch


 mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in 
 branch-2 at all. grep on trunk yields no results, not even ReflectionUtils 
 related suff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4861) Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal

2012-12-07 Thread Sandy Ryza (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526818#comment-13526818
 ] 

Sandy Ryza commented on MAPREDUCE-4861:
---

+1

 Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
 --

 Key: MAPREDUCE-4861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4861.patch


 mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in 
 branch-2 at all. grep on trunk yields no results, not even ReflectionUtils 
 related suff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-461) Enable ServicePlugins for the JobTracker


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun C Murthy updated MAPREDUCE-461:


Fix Version/s: 1.2.0

 Enable ServicePlugins for the JobTracker
 

 Key: MAPREDUCE-461
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-461
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Fredrik Hedberg
Assignee: Fredrik Hedberg
Priority: Minor
 Fix For: 0.23.0, 1.2.0

 Attachments: MAPREDUCE-461.branch-1.patch, MAPREDUCE-461.patch, 
 sp-jt-1.diff


 Allow ServicePlugins (see HADOOP-5257) for the JobTracker.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4861) Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal

2012-12-07 Thread Daryn Sharp (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526841#comment-13526841
 ] 

Daryn Sharp commented on MAPREDUCE-4861:


Presumably the code was copied elsewhere?  If so, were the tests also copied?

 Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
 --

 Key: MAPREDUCE-4861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4861.patch


 mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in 
 branch-2 at all. grep on trunk yields no results, not even ReflectionUtils 
 related suff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4861) Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526856#comment-13526856
 ] 

Hadoop QA commented on MAPREDUCE-4861:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12559961/mr-4861.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3108//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3108//console

This message is automatically generated.

 Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
 --

 Key: MAPREDUCE-4861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4861.patch


 mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in 
 branch-2 at all. grep on trunk yields no results, not even ReflectionUtils 
 related suff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4860) Inconsistent synchronization in mapreduce.security.token.DelegationTokenRenewal


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4860:


Attachment: mr-4860.patch

Taking a closer look, the problem seems to be with the use of TimerTask/Timer.

If a task is already scheduled, calling task.cancel() from outside the task 
doesn't stop the scheduled task, but prevents the task from getting scheduled 
another time. However, the task can cancel itself.

The patch:
# adds a check within the task if it has been stopped from outside
# synchronizes changes to delegationTokens 

 Inconsistent synchronization in 
 mapreduce.security.token.DelegationTokenRenewal
 ---

 Key: MAPREDUCE-4860
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4860
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: security
Affects Versions: 1.1.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4860.patch


 mapreduce.security.token.DelegationTokenRenewal synchronizes on 
 removeDelegationToken, but fails to synchronize on addToken, and renewing 
 tokens in run().
 This inconsistency is exposed by frequent failures of 
 TestDelegationTokenRenewal:
 {noformat}
 Error Message
 renew wasn't called as many times as expected expected:4 but was:5
 Stacktrace
 junit.framework.AssertionFailedError: renew wasn't called as many times as 
 expected expected:4 but was:5
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewal(TestDelegationTokenRenewal.java:317)
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewalAfterClose(TestDelegationTokenRenewal.java:338)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4861) Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526868#comment-13526868
 ] 

Karthik Kambatla commented on MAPREDUCE-4861:
-

Daryn, the code has been copied to 
yarn.server.rm.security.DelegationTokenRenewer and the corresponding tests have 
been copied to rm.security.TestDelegationTokenRenewer (except one test - 
testing renewal after close()). Do you suggest I add the missing test before 
deleting these?

The copied code suffers from issues similar to MAPREDUCE-4860. However, I still 
think o.a.h.fs.DelegationTokenRenewer can be re-used here instead. Comments?

 Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
 --

 Key: MAPREDUCE-4861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4861.patch


 mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in 
 branch-2 at all. grep on trunk yields no results, not even ReflectionUtils 
 related suff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4861) Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4861:


Attachment: mr-4861.patch

Updated the patch to copy testDTRenewalAfterClose to TestDelegationTokenRenewer

 Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
 --

 Key: MAPREDUCE-4861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4861.patch, mr-4861.patch


 mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in 
 branch-2 at all. grep on trunk yields no results, not even ReflectionUtils 
 related suff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4860) Inconsistent synchronization in mapreduce.security.token.DelegationTokenRenewal


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4860:


Status: Patch Available  (was: Open)

 Inconsistent synchronization in 
 mapreduce.security.token.DelegationTokenRenewal
 ---

 Key: MAPREDUCE-4860
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4860
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: security
Affects Versions: 1.1.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4860.patch


 mapreduce.security.token.DelegationTokenRenewal synchronizes on 
 removeDelegationToken, but fails to synchronize on addToken, and renewing 
 tokens in run().
 This inconsistency is exposed by frequent failures of 
 TestDelegationTokenRenewal:
 {noformat}
 Error Message
 renew wasn't called as many times as expected expected:4 but was:5
 Stacktrace
 junit.framework.AssertionFailedError: renew wasn't called as many times as 
 expected expected:4 but was:5
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewal(TestDelegationTokenRenewal.java:317)
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewalAfterClose(TestDelegationTokenRenewal.java:338)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4860) Inconsistent synchronization in mapreduce.security.token.DelegationTokenRenewal


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526886#comment-13526886
 ] 

Hadoop QA commented on MAPREDUCE-4860:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12559974/mr-4860.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3110//console

This message is automatically generated.

 Inconsistent synchronization in 
 mapreduce.security.token.DelegationTokenRenewal
 ---

 Key: MAPREDUCE-4860
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4860
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: security
Affects Versions: 1.1.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4860.patch


 mapreduce.security.token.DelegationTokenRenewal synchronizes on 
 removeDelegationToken, but fails to synchronize on addToken, and renewing 
 tokens in run().
 This inconsistency is exposed by frequent failures of 
 TestDelegationTokenRenewal:
 {noformat}
 Error Message
 renew wasn't called as many times as expected expected:4 but was:5
 Stacktrace
 junit.framework.AssertionFailedError: renew wasn't called as many times as 
 expected expected:4 but was:5
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewal(TestDelegationTokenRenewal.java:317)
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewalAfterClose(TestDelegationTokenRenewal.java:338)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4862) Inconsistent synchronization in yarn.server.rm.security.DelegationTokenRenewal

Karthik Kambatla created MAPREDUCE-4862:
---

 Summary: Inconsistent synchronization in 
yarn.server.rm.security.DelegationTokenRenewal
 Key: MAPREDUCE-4862
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4862
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: security
Affects Versions: 1.1.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla


mapreduce.security.token.DelegationTokenRenewal synchronizes on 
removeDelegationToken, but fails to synchronize on addToken, and renewing 
tokens in run().

This inconsistency is exposed by frequent failures of 
TestDelegationTokenRenewal:
{noformat}
Error Message

renew wasn't called as many times as expected expected:4 but was:5
Stacktrace

junit.framework.AssertionFailedError: renew wasn't called as many times as 
expected expected:4 but was:5
at 
org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewal(TestDelegationTokenRenewal.java:317)
at 
org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewalAfterClose(TestDelegationTokenRenewal.java:338)

{noformat}


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4862) Inconsistent synchronization in yarn.server.rm.security.DelegationTokenRenewal

[
https://issues.apache.org/jira/browse/MAPREDUCE-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Karthik Kambatla updated MAPREDUCE-4862:

Description:
yarn.s.rm.security.DelegationTokenRenewer uses TimerTask/Timer. When such a
timer task is canceled, already scheduled tasks run to completion. The task
should check for such cancellation before running. Also, delegationTokens needs
to be synchronized on all accesses.

was:
mapreduce.security.token.DelegationTokenRenewal synchronizes on
removeDelegationToken, but fails to synchronize on addToken, and renewing
tokens in run().

This inconsistency is exposed by frequent failures of
TestDelegationTokenRenewal:
{noformat}
Error Message

renew wasn't called as many times as expected expected:4 but was:5
Stacktrace

junit.framework.AssertionFailedError: renew wasn't called as many times as
expected expected:4 but was:5
at
org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewal(TestDelegationTokenRenewal.java:317)
at
org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewalAfterClose(TestDelegationTokenRenewal.java:338)

{noformat}

Inconsistent synchronization in yarn.server.rm.security.DelegationTokenRenewal
--

Key: MAPREDUCE-4862
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4862
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: security
Affects Versions: 1.1.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla

yarn.s.rm.security.DelegationTokenRenewer uses TimerTask/Timer. When such a
timer task is canceled, already scheduled tasks run to completion. The task
should check for such cancellation before running. Also, delegationTokens
needs to be synchronized on all accesses.

[jira] [Updated] (MAPREDUCE-4860) Inconsistent synchronization in mapreduce.security.token.DelegationTokenRenewal


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4860:


Attachment: mr-4860.patch

 Inconsistent synchronization in 
 mapreduce.security.token.DelegationTokenRenewal
 ---

 Key: MAPREDUCE-4860
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4860
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: security
Affects Versions: 1.1.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4860.patch, mr-4860.patch


 mapreduce.security.token.DelegationTokenRenewal synchronizes on 
 removeDelegationToken, but fails to synchronize on addToken, and renewing 
 tokens in run().
 This inconsistency is exposed by frequent failures of 
 TestDelegationTokenRenewal:
 {noformat}
 Error Message
 renew wasn't called as many times as expected expected:4 but was:5
 Stacktrace
 junit.framework.AssertionFailedError: renew wasn't called as many times as 
 expected expected:4 but was:5
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewal(TestDelegationTokenRenewal.java:317)
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewalAfterClose(TestDelegationTokenRenewal.java:338)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4861) Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526923#comment-13526923
 ] 

Hadoop QA commented on MAPREDUCE-4861:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12559978/mr-4861.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

  
org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3109//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3109//console

This message is automatically generated.

 Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
 --

 Key: MAPREDUCE-4861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4861.patch, mr-4861.patch


 mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in 
 branch-2 at all. grep on trunk yields no results, not even ReflectionUtils 
 related suff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4860) Inconsistent synchronization in mapreduce.security.token.DelegationTokenRenewal


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526924#comment-13526924
 ] 

Hadoop QA commented on MAPREDUCE-4860:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12559989/mr-4860.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3111//console

This message is automatically generated.

 Inconsistent synchronization in 
 mapreduce.security.token.DelegationTokenRenewal
 ---

 Key: MAPREDUCE-4860
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4860
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: security
Affects Versions: 1.1.1
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4860.patch, mr-4860.patch


 mapreduce.security.token.DelegationTokenRenewal synchronizes on 
 removeDelegationToken, but fails to synchronize on addToken, and renewing 
 tokens in run().
 This inconsistency is exposed by frequent failures of 
 TestDelegationTokenRenewal:
 {noformat}
 Error Message
 renew wasn't called as many times as expected expected:4 but was:5
 Stacktrace
 junit.framework.AssertionFailedError: renew wasn't called as many times as 
 expected expected:4 but was:5
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewal(TestDelegationTokenRenewal.java:317)
   at 
 org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewalAfterClose(TestDelegationTokenRenewal.java:338)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4861) Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526945#comment-13526945
 ] 

Karthik Kambatla commented on MAPREDUCE-4861:
-

Looks like I ran the wrong TestDelegationTokenRenewer locally before uploading. 
Taking a closer look, I don't think the new test applies to YARN as the Renewer 
can't be stopped and restarted. Shall upload the first patch again.

 Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
 --

 Key: MAPREDUCE-4861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4861.patch, mr-4861.patch


 mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in 
 branch-2 at all. grep on trunk yields no results, not even ReflectionUtils 
 related suff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4861) Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4861:


Attachment: mr-4861.patch

 Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
 --

 Key: MAPREDUCE-4861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4861.patch, mr-4861.patch, mr-4861.patch


 mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in 
 branch-2 at all. grep on trunk yields no results, not even ReflectionUtils 
 related suff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated MAPREDUCE-4049:
--

Issue Type: Sub-task  (was: Improvement)
Parent: MAPREDUCE-2454

 plugin for generic shuffle service
 --

 Key: MAPREDUCE-4049
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: performance, task, tasktracker
Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
Reporter: Avner BenHanoch
Assignee: Avner BenHanoch
  Labels: merge, plugin, rdma, shuffle
 Fix For: 3.0.0

 Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, 
 mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch, 
 mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch


 Support generic shuffle service as set of two plugins: ShuffleProvider  
 ShuffleConsumer.
 This will satisfy the following needs:
 # Better shuffle and merge performance. For example: we are working on 
 shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, 
 or Infiniband) instead of using the current HTTP shuffle. Based on the fast 
 RDMA shuffle, the plugin can also utilize a suitable merge approach during 
 the intermediate merges. Hence, getting much better performance.
 # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden 
 dependency of NodeManager with a specific version of mapreduce shuffle 
 (currently targeted to 0.24.0).
 References:
 # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu 
 from Auburn University with others, 
 [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
 # I am attaching 2 documents with suggested Top Level Design for both plugins 
 (currently, based on 1.0 branch)
 # I am providing link for downloading UDA - Mellanox's open source plugin 
 that implements generic shuffle service using RDMA and levitated merge.  
 Note: At this phase, the code is in C++ through JNI and you should consider 
 it as beta only.  Still, it can serve anyone that wants to implement or 
 contribute to levitated merge. (Please be advised that levitated merge is 
 mostly suit in very fast networks) - 
 [http://www.mellanox.com/content/pages.php?pg=products_dynproduct_family=144menu_section=69]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526953#comment-13526953
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4049:
---

Arun, 

I'd appreciate not to remove again this JIRA from being a subtask of 
MAPREDUCE-2454. 

You are leaving with no choice but to *-1* this patch to go in trunk until the 
work in the branch is completed.

My rationale for this -1 it is yours:

https://issues.apache.org/jira/browse/MAPREDUCE-4809?focusedCommentId=13501245page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13501245

 Arun C Murthy added a comment - 20/Nov/12 07:50
 I've also created a MR-2454 branch in svn, let's commit to that branch first.
 This way we can change our mind before we do the final merge if necessary.

In addition, as we may need to make further changes to MAPREDUCE-4049, I'd like 
to see the whole scope of changes before merging back to trunk.



 plugin for generic shuffle service
 --

 Key: MAPREDUCE-4049
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: performance, task, tasktracker
Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
Reporter: Avner BenHanoch
Assignee: Avner BenHanoch
  Labels: merge, plugin, rdma, shuffle
 Fix For: 3.0.0

 Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, 
 mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch, 
 mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch


 Support generic shuffle service as set of two plugins: ShuffleProvider  
 ShuffleConsumer.
 This will satisfy the following needs:
 # Better shuffle and merge performance. For example: we are working on 
 shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, 
 or Infiniband) instead of using the current HTTP shuffle. Based on the fast 
 RDMA shuffle, the plugin can also utilize a suitable merge approach during 
 the intermediate merges. Hence, getting much better performance.
 # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden 
 dependency of NodeManager with a specific version of mapreduce shuffle 
 (currently targeted to 0.24.0).
 References:
 # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu 
 from Auburn University with others, 
 [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
 # I am attaching 2 documents with suggested Top Level Design for both plugins 
 (currently, based on 1.0 branch)
 # I am providing link for downloading UDA - Mellanox's open source plugin 
 that implements generic shuffle service using RDMA and levitated merge.  
 Note: At this phase, the code is in C++ through JNI and you should consider 
 it as beta only.  Still, it can serve anyone that wants to implement or 
 contribute to levitated merge. (Please be advised that levitated merge is 
 mostly suit in very fast networks) - 
 [http://www.mellanox.com/content/pages.php?pg=products_dynproduct_family=144menu_section=69]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (MAPREDUCE-4049) plugin for generic shuffle service


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur reopened MAPREDUCE-4049:
---


 plugin for generic shuffle service
 --

 Key: MAPREDUCE-4049
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: performance, task, tasktracker
Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
Reporter: Avner BenHanoch
Assignee: Avner BenHanoch
  Labels: merge, plugin, rdma, shuffle
 Fix For: 3.0.0

 Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, 
 mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch, 
 mapreduce-4049.patch, mapreduce-4049.patch, mapreduce-4049.patch


 Support generic shuffle service as set of two plugins: ShuffleProvider  
 ShuffleConsumer.
 This will satisfy the following needs:
 # Better shuffle and merge performance. For example: we are working on 
 shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, 
 or Infiniband) instead of using the current HTTP shuffle. Based on the fast 
 RDMA shuffle, the plugin can also utilize a suitable merge approach during 
 the intermediate merges. Hence, getting much better performance.
 # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden 
 dependency of NodeManager with a specific version of mapreduce shuffle 
 (currently targeted to 0.24.0).
 References:
 # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu 
 from Auburn University with others, 
 [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
 # I am attaching 2 documents with suggested Top Level Design for both plugins 
 (currently, based on 1.0 branch)
 # I am providing link for downloading UDA - Mellanox's open source plugin 
 that implements generic shuffle service using RDMA and levitated merge.  
 Note: At this phase, the code is in C++ through JNI and you should consider 
 it as beta only.  Still, it can serve anyone that wants to implement or 
 contribute to levitated merge. (Please be advised that levitated merge is 
 mostly suit in very fast networks) - 
 [http://www.mellanox.com/content/pages.php?pg=products_dynproduct_family=144menu_section=69]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4861) Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526978#comment-13526978
 ] 

Hadoop QA commented on MAPREDUCE-4861:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12559994/mr-4861.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3112//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3112//console

This message is automatically generated.

 Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
 --

 Key: MAPREDUCE-4861
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: mr-4861.patch, mr-4861.patch, mr-4861.patch


 mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in 
 branch-2 at all. grep on trunk yields no results, not even ReflectionUtils 
 related suff.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4594) Add init/shutdown methods to mapreduce Partitioner

2012-12-07 Thread Radim Kolar (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radim Kolar updated MAPREDUCE-4594:
---

Attachment: partitioner2.txt

now with unit tests

 Add init/shutdown methods to mapreduce Partitioner
 --

 Key: MAPREDUCE-4594
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4594
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: client
Affects Versions: trunk
Reporter: Radim Kolar
 Attachments: partitioner1.txt, partitioner2.txt, partitioner2.txt


 The Partitioner supports only the Configurable API, which can be used for 
 basic init in setConf(). Problem is that there is no shutdown function.
 I propose to use standard setup() cleanup() functions like in mapper / 
 reducer.
 Use case is that I need to start and stop spring context and datagrid client.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4594) Add init/shutdown methods to mapreduce Partitioner


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526992#comment-13526992
 ] 

Hadoop QA commented on MAPREDUCE-4594:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12560007/partitioner2.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3113//console

This message is automatically generated.

 Add init/shutdown methods to mapreduce Partitioner
 --

 Key: MAPREDUCE-4594
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4594
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: client
Affects Versions: trunk
Reporter: Radim Kolar
 Attachments: partitioner1.txt, partitioner2.txt, partitioner2.txt


 The Partitioner supports only the Configurable API, which can be used for 
 basic init in setConf(). Problem is that there is no shutdown function.
 I propose to use standard setup() cleanup() functions like in mapper / 
 reducer.
 Use case is that I need to start and stop spring context and datagrid client.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4594) Add init/shutdown methods to mapreduce Partitioner

2012-12-07 Thread Radim Kolar (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13527010#comment-13527010
 ] 

Radim Kolar commented on MAPREDUCE-4594:


hmm, it fails to compile with javac. In eclipse it works. Its problem with 
generic.

i can not cast class Partitioner.class to Class? extends Partitioner?, ? 
because of type erasure.

 Add init/shutdown methods to mapreduce Partitioner
 --

 Key: MAPREDUCE-4594
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4594
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: client
Affects Versions: trunk
Reporter: Radim Kolar
 Attachments: partitioner1.txt, partitioner2.txt, partitioner2.txt


 The Partitioner supports only the Configurable API, which can be used for 
 basic init in setConf(). Problem is that there is no shutdown function.
 I propose to use standard setup() cleanup() functions like in mapper / 
 reducer.
 Use case is that I need to start and stop spring context and datagrid client.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4594) Add init/shutdown methods to mapreduce Partitioner

2012-12-07 Thread Radim Kolar (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Radim Kolar updated MAPREDUCE-4594:
---

Attachment: partitioner3.txt

 Add init/shutdown methods to mapreduce Partitioner
 --

 Key: MAPREDUCE-4594
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4594
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: client
Affects Versions: trunk
Reporter: Radim Kolar
 Attachments: partitioner1.txt, partitioner2.txt, partitioner2.txt, 
 partitioner3.txt


 The Partitioner supports only the Configurable API, which can be used for 
 basic init in setConf(). Problem is that there is no shutdown function.
 I propose to use standard setup() cleanup() functions like in mapper / 
 reducer.
 Use case is that I need to start and stop spring context and datagrid client.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4594) Add init/shutdown methods to mapreduce Partitioner