subject:"\[jira\] \[Commented\] \(MAPREDUCE\-1700\) User supplied dependencies may conflict with MapReduce system JARs"

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2015-05-31 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566844#comment-14566844
 ] 

Guozhang Wang commented on MAPREDUCE-1700:
--

[~sjlee0] I have sent my questions to the users mailing list with a more 
detailed description. I would appreciate it if you can help resolving my 
confusions.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2015-05-27 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14561473#comment-14561473
 ] 

Guozhang Wang commented on MAPREDUCE-1700:
--

Hello,

We are seeing some similar issues in Samza usage and trying to borrow some 
ideas from Hadoop. I read through this ticket and YARN-286 but am still trying 
to figure out how it resolves the (e.g. Avro) conflict? For us we have a single 
thread that is similar to JobImpl executing user code like map/reduce as well 
as some system logic such as ser-de. Let's say we have a Avro 1.4 jar in 
CLASSPATH and another Avro 1.7 jar in APP_CLASSPATH. The task's 
ApplicationClassLoader would then first load Avro 1.7 from CLASSPATH, but then 
when it needs Avro 1.4 for ser-de it will fail because it has already loaded 
Avro 1.7 which is backward incompatible. Am I missing anything here?

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2015-05-27 Thread Sangjin Lee (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14561603#comment-14561603
 ] 

Sangjin Lee commented on MAPREDUCE-1700:


The idea of the application classloader is that there are two classloaders (the 
application classloader and the JVM's classloader) which can keep its copies of 
the same class. It is possible for the application classloader to load and use 
avro 1.7 (from the app classpath) and for the JVM classloader to load and use 
avro 1.4 (from the main classpath). It is essentially achieved by reversing the 
standard parent-first loading approach.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2015-05-27 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14561768#comment-14561768
 ] 

Guozhang Wang commented on MAPREDUCE-1700:
--

Thanks [~sjlee0] for the quick replies. I think my main confusion is coming 
from setContextClassLoader() for the thread.

More specifically let's say the thread does both some framework logic and 
some user logic like:

{code}
message = system.getMessage().deserialize();  // this is system code
result = process(message); // this is user code that could introduce other 
dependencies
system.send(result.serialize); // this is system code again
{code}

then setting this thread's context class loader to the application class load 
seems not be sufficient to me since the thread would also need to execute some 
framework logic that would require a different class loader?

Sorry about the consecutive questions and if people feel strong about it, I can 
move to the mailing list. Just feel it is more efficient asking the questions 
here with patches at hand.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2015-05-27 Thread Sangjin Lee (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562103#comment-14562103
]

Sangjin Lee commented on MAPREDUCE-1700:

I would also advise moving this discussion to the mailing group. That way
you'll get a bigger audience. Also, this is a resolved JIRA, so I don't think
this is the best place to conduct the discussion.

The thread context classloader is a poor abstraction that's fraught with
issues. There is no well-defined contract on how to set and use it. The current
hadoop code does pretty much what you're describing above. However, it's
difficult to pinpoint always when the user code runs and when the system code
runs, so it's bound to be imprecise at some point.

Normally the TCCL is not the main way to load classes. So it would be a problem
only if some code explicitly invokes Thread.getContextClassLoader() to retrieve
the TCCL and uses it to load classes. So in practice, it's uncommon we see
issues due to this although it's possible in principle.

User supplied dependencies may conflict with MapReduce system JARs
--

Key: MAPREDUCE-1700
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Reporter: Tom White
Assignee: Tom White
Fix For: 2.0.3-alpha

Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch,
MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch,
MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch,
MAPREDUCE-1700.patch, MAPREDUCE-1700.patch

If user code has a dependency on a version of a JAR that is different to the
one that happens to be used by Hadoop, then it may not work correctly. This
happened with user code using a different version of Avro, as reported
[here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
The problem is analogous to the one that application servers have with WAR
loading. Using a specialized classloader in the Child JVM is probably the way
to solve this.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2015-05-27 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14562034#comment-14562034
 ] 

Guozhang Wang commented on MAPREDUCE-1700:
--

More specifically, I feel for it to really work, one has to do sth. like:

{code}
Thread thread = Thread.currentThread();
ClassLoader old = thread.getContextClassLoader();
thread.setContextClassLoader(newClassLoader);

try {
// user code like process(), map(), reduce(), etc..
} finally {
thread.setContextClassLoader(old);
}
{code}

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2013-10-08 Thread James Xu (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788955#comment-13788955
 ] 

James Xu commented on MAPREDUCE-1700:
-

Hi Tom, can you elaborate more on why 
org.apache.commons.logging.,org.apache.log4j. should be blacklisted? we are 
trying to do the similar class loader thing for another software, so want to 
learn some experience here :)

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2013-10-08 Thread Tom White (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13789117#comment-13789117
 ] 

Tom White commented on MAPREDUCE-1700:
--

[~xumingming], the set of packages to blacklist came from the Jetty 
(http://docs.codehaus.org/display/JETTY/Classloading), which is why Commons 
Logging and Log4j were included. Excluding logging classes prevents inadvertant 
double initialization of the logging system - once when the task JVM starts and 
again when the user code is loaded. Note that you can change the system default 
by setting mapreduce.job.classloader.system.classes.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2013-04-09 Thread Sangjin Lee (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627020#comment-13627020
 ] 

Sangjin Lee commented on MAPREDUCE-1700:


This is an awesome change. I've always wondered why this support wasn't there. 
Thanks!

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2013-01-30 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13566423#comment-13566423
 ] 

Hudson commented on MAPREDUCE-1700:
---

Integrated in Hadoop-Hdfs-0.23-Build #510 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/510/])
merge -r 1430928:1430929 Merging MAPREDUCE-1700 to branch-0.23 (Revision 
1440100)

 Result = SUCCESS
kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1440100
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml


 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2013-01-29 Thread Kihwal Lee (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565758#comment-13565758
 ] 

Kihwal Lee commented on MAPREDUCE-1700:
---

Merged to branch-0.23.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2013-01-14 Thread Scott Carey (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553420#comment-13553420
 ] 

Scott Carey commented on MAPREDUCE-1700:


Awesome!

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2013-01-10 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549519#comment-13549519
 ] 

Hudson commented on MAPREDUCE-1700:
---

Integrated in Hadoop-Yarn-trunk #92 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/92/])
MAPREDUCE-1700. User supplied dependencies may conflict with MapReduce 
system JARs. (Revision 1430929)

 Result = SUCCESS
tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430929
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml


 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2013-01-10 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549601#comment-13549601
 ] 

Hudson commented on MAPREDUCE-1700:
---

Integrated in Hadoop-Mapreduce-trunk #1309 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1309/])
MAPREDUCE-1700. User supplied dependencies may conflict with MapReduce 
system JARs. (Revision 1430929)

 Result = FAILURE
tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430929
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml


 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2013-01-10 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549614#comment-13549614
 ] 

Hudson commented on MAPREDUCE-1700:
---

Integrated in Hadoop-Hdfs-trunk #1281 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1281/])
MAPREDUCE-1700. User supplied dependencies may conflict with MapReduce 
system JARs. (Revision 1430929)

 Result = FAILURE
tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430929
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml


 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2013-01-09 Thread Kihwal Lee (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13547831#comment-13547831
 ] 

Kihwal Lee commented on MAPREDUCE-1700:
---

+1 The patch looks good. I hope people try this with many different use cases.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2013-01-09 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548644#comment-13548644
 ] 

Hudson commented on MAPREDUCE-1700:
---

Integrated in Hadoop-trunk-Commit #3203 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/3203/])
MAPREDUCE-1700. User supplied dependencies may conflict with MapReduce 
system JARs. (Revision 1430929)

 Result = SUCCESS
tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430929
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml


 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.3-alpha

 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2013-01-04 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13543836#comment-13543836
 ] 

Hadoop QA commented on MAPREDUCE-1700:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12563283/MAPREDUCE-1700.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3194//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3194//console

This message is automatically generated.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-12-21 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538117#comment-13538117
 ] 

Hadoop QA commented on MAPREDUCE-1700:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12562082/MAPREDUCE-1700.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3157//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3157//console

This message is automatically generated.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-12-07 Thread Kihwal Lee (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13526775#comment-13526775
]

Kihwal Lee commented on MAPREDUCE-1700:
---

{quote}
bq. Tom, one thing I've forgot to mention in my previous comment, we should see
how to enable the classloader on the client side as well as it may be required
(to use different JARs) for the submission code.

I think this is a slightly different problem, since users generally have more
control over the JVM they submit from than the JVM the task runs in. So, yes,
another JIRA would be appropriate.
{quote}

I think AM also runs user code, if a custom output format is defined.

User supplied dependencies may conflict with MapReduce system JARs
--

Key: MAPREDUCE-1700
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Reporter: Tom White
Assignee: Tom White
Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch,
MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch,
MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-12-04 Thread Kihwal Lee (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509888#comment-13509888
 ] 

Kihwal Lee commented on MAPREDUCE-1700:
---

Now that we have a much better way of dealing with dependency conflicts, what 
will be the fate of mapreduce.job.user.classpath.first feature? Is there any 
use case where this feature works but the CCL approach don't or somehow is 
preferred over CCL for some reason? If none, shall we deprecate it?

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-11-21 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501874#comment-13501874
 ] 

Hadoop QA commented on MAPREDUCE-1700:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12554482/MAPREDUCE-1700.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3046//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3046//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3046//console

This message is automatically generated.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-11-21 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501913#comment-13501913
 ] 

Steve Loughran commented on MAPREDUCE-1700:
---

# {{Task}} should get the string {{APP_CLASSPATH}} from the 
{{ApplicationConstants}}.
# the test dir logic won't work on windows if {{test.build.data}} isn't set 
:{{System.getProperty(test.build.data, /tmp)}} -that default should be 
replaced with {{System.getProperty(java.io.tmpdir')}}
# in {{ApplicationClassLoader.loadClass()}} it looks to me like it is possible 
to have the situation {{c==null}}  {{ex==null}} at the {{if (c==null} throw 
ex;}} clause -if {{parent.loadClass() = null}}. Some check for a null {{ex}} 
value and setting to (something?) would avoid this.
# the tests should look for resource loading too, just to be thorough.

Other than that, with my finite classloader knowledge -looks good. Someone who 
understands OSGi should do  quick review too.


 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-11-21 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501911#comment-13501911
 ] 

Hadoop QA commented on MAPREDUCE-1700:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12554490/MAPREDUCE-1700.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3047//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3047//console

This message is automatically generated.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-11-21 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13502092#comment-13502092
 ] 

Hadoop QA commented on MAPREDUCE-1700:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12554512/MAPREDUCE-1700.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3049//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3049//console

This message is automatically generated.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-10-08 Thread Alejandro Abdelnur (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13471808#comment-13471808
 ] 

Alejandro Abdelnur commented on MAPREDUCE-1700:
---

Tom, one thing I've forgot to mention in my previous comment, we should see how 
to enable the classloader on the client side as well as it may be required (to 
use different JARs) for the submission code. May be as another JIRA. 

Also, don't recall now if it is there or not, we may want o have a job config 
property to disable it in case some app runs into funny issues with it.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-10-04 Thread Alejandro Abdelnur (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469203#comment-13469203
 ] 

Alejandro Abdelnur commented on MAPREDUCE-1700:
---

Nice.

What I would add is the capability of blacklisting packages. This is, if a 
package is blacklisted and a class under that package hierarchy is found in the 
job JARs, the job should fail. This is something avail in webapp classloaders 
to avoid webapps for bundling things like servlet/jsp JARs that would break 
things. In our case we would blacklist common/hdfs/yarn/mapred packages and 
log4j (the factory is a singleton and if present in the job JARs will trash the 
log configuration of hadoop). I could see other JARs fitting this blacklist, 
thus I'd suggest that we have a config property with the list of blacklisted 
packages.

This is isolating MR jobs from Hadoop JARs. I think we should do the same at 
YARN level to isolate YARN JARs from AM JARs. Because of this, the 
JobClassLoader should be in common and probably have a different name, like 
IsolationClassLoader. Also it should receive, in the constructor, the blacklist.


 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, 
 MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-09-06 Thread Luke Lu (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449917#comment-13449917
 ] 

Luke Lu commented on MAPREDUCE-1700:


I agree that the new (much) lighter weight approach is worth exploring. Thanks 
Tom! 

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-09-06 Thread Steve Loughran (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450030#comment-13450030
 ] 

Steve Loughran commented on MAPREDUCE-1700:
---

no, we don't want to go anywhere near servlet classloaders, because you end up 
in WAR EAR and app server trees. The app server takes priority, except in the 
special case of JBoss in the past, which shared classes across webapps

https://community.jboss.org/wiki/JBossClassLoaderHistory
http://docs.jboss.org/jbossweb/2.1.x/class-loader-howto.html

people will hit walls when they try to do things like upgrade the XML parser or 
try and add a new URL handler. 

I'll look at the patch, but classloaders are a mine of grief. That's the 
strength of OSGi: the grief is standardised an someone else has done the grief 
mining already


 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700.patch, 
 MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-09-06 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450074#comment-13450074
]

Luke Lu commented on MAPREDUCE-1700:

There is no need to be scared of classloaders, especially for the simple load
only and then exit scenarios that we're talking about. Most of the class
loader issues stem from long running containers that need to dynamically
load/unload classes. OSGi is an overkill for MR tasks. To be clear, I'm not
anti-OSGi, which I think is perfectly fine for managing server-side plugins.

User supplied dependencies may conflict with MapReduce system JARs
--

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-09-05 Thread Tom White (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448595#comment-13448595
 ] 

Tom White commented on MAPREDUCE-1700:
--

Scott makes a good case for why some kind of classloader isolation is needed.

The patch is still a work in progress, but the idea is that the OSGi support is 
optional - so if you use a regular (non-OSGi) job JAR then it works like it 
does today, while if your job JAR is an OSGi bundle (basically a JAR with extra 
headers in the manifest, and possibly some embedded dependencies) then it is 
loaded in an OSGi container in the task JVM. This allows folks who want to use 
OGSi to do so while not impacting others. (Hopefully this answers Steve's 
question.)

From the point of view of this JIRA, OSGi is simply a means to ensure 
classloader isolation. That means that if Jigsaw became a reality, then we 
could use that instead or as well. OSGi has many other features, but they are 
not used for this change. (Note that there are other ongoing efforts to make 
Hadoop more OSGi-friendly, covered in HADOOP-7977, and while some might be 
helpful for this JIRA (such as HADOOP-6484), none is required.)

Also, in the future OSGi containers could improve container reuse by providing 
better isolation between jobs, since bundles can be unloaded, although I 
haven't spent any time looking at how that would work in the context of MR.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-09-05 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448882#comment-13448882
]

Luke Lu commented on MAPREDUCE-1700:

bq. The conflict might require a user jar that is not compatible with one
needed by the framework, either order breaks something

You can always change the client framework and make it work with user code, per
job, with class path ordering. There is currently always a way in both Hadoop 1
and 2 to submit a job with arbitrary dependencies, even though it might not be
pretty (may require change to client framework).

bq. The user might override a system jar and alter functionality in a way that
breaks the framework, or subverts security.

The client framework code can always be changed per job to accommodate new
dependencies. MR security is done at protocol level, i.e. no amount class path
ordering can subvert security.

I agree with Arun that this is a nice to have feature to improve usability.
Advanced users can already achieve whatever that can be achieved (including
running an OSGi container) per job.

User supplied dependencies may conflict with MapReduce system JARs
--

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-09-05 Thread Scott Carey (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449335#comment-13449335
 ] 

Scott Carey commented on MAPREDUCE-1700:


{quote}
You can always change the client framework and make it work with user code, per 
job, with class path ordering. There is currently always a way in both Hadoop 1 
and 2 to submit a job with arbitrary dependencies, even though it might not be 
pretty (may require change to client framework).{quote}

Without a user doing classloader gymnasitics and fancy packaging themselves, 
there is not always a way.  A user cannot simply package a jar up and ask 
hadoop to execute it and expose to the user's execution environment only the 
public Hadoop API.


 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-09-05 Thread Luke Lu (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449430#comment-13449430
]

Luke Lu commented on MAPREDUCE-1700:

bq. Without a user doing classloader gymnasitics and fancy packaging
themselves, there is not always a way.

That's an interesting way to say that except for some ways that would always
work, there is not always a way. Using the standard task API to bootstrap an
OSGi container is reasonably straight forward :)

bq. A user cannot simply package a jar up and ask hadoop to execute it and
expose to the user's execution environment only the public Hadoop API.

I do agree that there is a usability issue for certain (and arguably less
common) use cases, where a user wants to use dependencies that conflict with
client framework. However the proposed OSGi approach makes the usability worse
for common cases: You'll always need OSGi bundles, which is a form of fancy
packaging, to run your jobs.

A more reasonable (and less heavy) solution would not require users to make any
change (including adding metadata to their jars) to their existing code.

User supplied dependencies may conflict with MapReduce system JARs
--

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-09-04 Thread Arun C Murthy (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447911#comment-13447911
]

Arun C Murthy commented on MAPREDUCE-1700:
--

Tom, I don't understand specific advantages of OSGI or Felix, so please pardon
some of my questions.

However, with MR being an application in YARN (see MAPREDUCE-4421) we can just
add user jars in front of the classpath for the tasks (we already allow it).
This isn't the same Map/Reduce child inherits the TT classpath problem in MR1
(actually even in MR1 you can put child jars ahead in the classpath for a long
while now). Given this, do we need to bring in OSGI or Felix, what do else do
they provide? Thanks.

User supplied dependencies may conflict with MapReduce system JARs
--

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-09-04 Thread Steve Loughran (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447996#comment-13447996
]

Steve Loughran commented on MAPREDUCE-1700:
---

Arun,

I see where Tom is coming from. Irrespective of how the Hadoop services are
deployed, you need to be able to do things like submit jobs from OSGi
containers (e.g Spring others) which is what this patch appears to offer. And
if Oracle finally commit to OSGi now that Java 8 is being redefined, it'd be
good from all clients.

I would like to see a way to support this which doesn't put an OSGi JAR on the
classpath of everything.

Tom -is there a way to abstract away OSGi support so that it's optional, even
if its a subclass of JobSubmitter? An
{{org.apache.hadoop.mapreduce.osgi.OSGiJobSubmitter}} could override some new
specific protect methods to enable this.

User supplied dependencies may conflict with MapReduce system JARs
--

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-09-04 Thread Scott Carey (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448178#comment-13448178
 ] 

Scott Carey commented on MAPREDUCE-1700:


Putting user jars before/after the application dependencies doesn't actually 
solve the problem.  

* The conflict might require a user jar that is not compatible with one needed 
by the framework, either order breaks something
* The user might override a system jar and alter functionality in a way that 
breaks the framework, or subverts security.

Both the host container and the user code need to be able to be certain of what 
code they are executing without stepping on each other's toes.  This is _not 
possible_ with one classpath.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-1700.patch, MAPREDUCE-1700.patch


 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2012-09-04 Thread Scott Carey (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448205#comment-13448205
]

Scott Carey commented on MAPREDUCE-1700:

If we are lucky, projecct jigsaw will be pulled back into Java 8. According
to: http://mreinhold.org/blog/late-for-the-train-qa it has not yet been decided.

If it is brought back in, then perhaps we can wait until Java has a module
system 1 to 1.5 years from now. If not, I do not think Hadoop can wait until
Java 9, sometime 2015 to 2016 ish.

User supplied dependencies may conflict with MapReduce system JARs
--

[jira] Commented: (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2010-11-04 Thread Henning Blohm (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12928208#action_12928208
 ] 

Henning Blohm commented on MAPREDUCE-1700:
--

The patch in MAPREDUCE-1938 does unfortunately not solve the issue when the job 
implementation uses custom class loaders to load dependency classes. The 
proposed patch only addresses the issue when no custom class loaders are in the 
picture.

As a first step, it would really help (us anyway), if jobs would not be started 
with stuff on the classpath that is not at all required for job execution per 
se (e.g. jetty libs, the eclipse java compiler, jasper...).

Secondly, hadoop could actually start with only the hadoop api types on the 
classpath plus a small launcher that would load hadoops implementation in an 
isolated child loader, so that implementation dependencies do not leak through 
to the job's implementation. I am not sure if the hadoop implementation is 
ready for implementation/api separation via class loaders though. 

I patched hadoop 0.20.2 to exclude all libs in lib/server from the jobs 
classpath and I move all non-job related jars into that server folder in my 
hadoop installation. That helped somewhat. 

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White

 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2010-08-12 Thread David Rosenstrauch (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12897933#action_12897933
 ] 

David Rosenstrauch commented on MAPREDUCE-1700:
---

I also just ran into this issue.  (Again, due to using a recent release of avro 
+ jackson.)

Is there any workaround for this?  (Short of having to go into every node on 
the cluster and removing the jackson jar from the hadoop installation?

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White

 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs

2010-08-12 Thread Arun C Murthy (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898029#action_12898029
 ] 

Arun C Murthy commented on MAPREDUCE-1700:
--

We have a patch for yahoo-hadoop-0.20 in MAPREDUCE-1938 to help solve this.

 User supplied dependencies may conflict with MapReduce system JARs
 --

 Key: MAPREDUCE-1700
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Reporter: Tom White

 If user code has a dependency on a version of a JAR that is different to the 
 one that happens to be used by Hadoop, then it may not work correctly. This 
 happened with user code using a different version of Avro, as reported 
 [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
 The problem is analogous to the one that application servers have with WAR 
 loading. Using a specialized classloader in the Child JVM is probably the way 
 to solve this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

41 matches

Mail list logo