[jira] [Commented] (YARN-2092) Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 2.5.0-SNAPSHOT

2015-04-27 Thread Hitesh Shah (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14515828#comment-14515828
 ] 

Hitesh Shah commented on YARN-2092:
---

Closing this out as it is no longer an issue for tez. 

 Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 
 2.5.0-SNAPSHOT
 

 Key: YARN-2092
 URL: https://issues.apache.org/jira/browse/YARN-2092
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah

 Came across this when trying to integrate with the timeline server. Using a 
 1.8.8 dependency of jackson works fine against 2.4.0 but fails against 
 2.5.0-SNAPSHOT which needs 1.9.13. This is in the scenario where the user 
 jars are first in the classpath.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-2092) Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 2.5.0-SNAPSHOT

2014-05-28 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010906#comment-14010906
 ] 

Steve Loughran commented on YARN-2092:
--

# code using Apache Curator was one -in YARN and on the client if you picked up 
the locally installed hadoop CP. And we couldn't upload a newer version of 
Jackson, because again, what's on the CP is what you get. 
# If, client-side you abandon that CP and use/redist your entire hadoop binary 
set then you can reduce the risk here, at the expense of taking away from ops 
any control of the versions of things you run on the cluster, but then you now 
have to deal with older files in the cluster.
# or you ignore yarn.lib.classpath entirely, *somehow* work out the values of 
yarn-site.xml c, and re-upload every single hadoop-*.jar and its chosen 
binaries into every single container. having a YARN artifact repo will reduce 
the cost of that, but add a new one: bug fixes in hadoop will only propagate 
when the apps are rebuilt.
# ..if you look at the HADOOP-9991 issue you can see links to some places where 
the outdated JARs in Hadoop cause problems for other ASF projects.
# Tez appears to have broken because it was explicity putting the 1.8.x JARs on 
its list of binaries to upload. It only worked because it was using exactly the 
same version. 
# if you adopt a policy of change no dependencies that break apps that upload 
duplicate JARs to the CP -then this goes beyond Jackson, it says hadoop cannot 
update any of its dependencies. That would go for 2.x and no doubt even if we 
did update things for 3.x, then we'll still get you broke my code that 
uploaded jackson 
1.8 issues.
# ...and we haven't gone near Guava yet, which is frozen because it really is 
so brittle, but that means we can't pick up the guava 16.x-only fixes needed to 
work with the latest JVMs. 

If you do want to revoke jackson, I'm not going to veto it -but it goes beyond 
YARN, and we may as well revert every single HADOOP-9991-related upgrade. 

 Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 
 2.5.0-SNAPSHOT
 

 Key: YARN-2092
 URL: https://issues.apache.org/jira/browse/YARN-2092
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah

 Came across this when trying to integrate with the timeline server. Using a 
 1.8.8 dependency of jackson works fine against 2.4.0 but fails against 
 2.5.0-SNAPSHOT which needs 1.9.13. This is in the scenario where the user 
 jars are first in the classpath.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2092) Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 2.5.0-SNAPSHOT

2014-05-23 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007269#comment-14007269
 ] 

Steve Loughran commented on YARN-2092:
--

I think this should be a wontfix, as the underlying problem was trying to get 
an older version of Jackson onto the classpath. Admittedly, this probably 
worked on 2.2-2.4, but that is because the code was pushing up the same version 
of jackson that was there. If Tez wasn't trying to push up any jackson JARs, 
but instead take what was there, it would work (which is essentially what has 
been done).

Unless/Until we can isolate YARN apps from the classpath other than 
org.apache.hadoop.*, this problem will arise. Which implies we need OSGI 
support in YARN

 Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 
 2.5.0-SNAPSHOT
 

 Key: YARN-2092
 URL: https://issues.apache.org/jira/browse/YARN-2092
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah

 Came across this when trying to integrate with the timeline server. Using a 
 1.8.8 dependency of jackson works fine against 2.4.0 but fails against 
 2.5.0-SNAPSHOT which needs 1.9.13. This is in the scenario where the user 
 jars are first in the classpath.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2092) Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 2.5.0-SNAPSHOT

2014-05-23 Thread Sangjin Lee (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007281#comment-14007281
 ] 

Sangjin Lee commented on YARN-2092:
---

+1 on exploring the OSGi option.

Until/unless it happens, one option might be to consider an expanded use for 
the mapreduce.job.classloader config? Currently it only works within a MR app 
(AM and tasks). However, one could argue that the functionality it provides is 
a generic one in that it creates a somewhat isolated class space. Perhaps we 
could rename the job classloader to something like an app classloader, and make 
it available for any place wherever user code needs to run in isolation.

 Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 
 2.5.0-SNAPSHOT
 

 Key: YARN-2092
 URL: https://issues.apache.org/jira/browse/YARN-2092
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah

 Came across this when trying to integrate with the timeline server. Using a 
 1.8.8 dependency of jackson works fine against 2.4.0 but fails against 
 2.5.0-SNAPSHOT which needs 1.9.13. This is in the scenario where the user 
 jars are first in the classpath.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2092) Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 2.5.0-SNAPSHOT

2014-05-23 Thread Sangjin Lee (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007284#comment-14007284
 ] 

Sangjin Lee commented on YARN-2092:
---

... user code or any code that needs to run in isolation.

 Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 
 2.5.0-SNAPSHOT
 

 Key: YARN-2092
 URL: https://issues.apache.org/jira/browse/YARN-2092
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah

 Came across this when trying to integrate with the timeline server. Using a 
 1.8.8 dependency of jackson works fine against 2.4.0 but fails against 
 2.5.0-SNAPSHOT which needs 1.9.13. This is in the scenario where the user 
 jars are first in the classpath.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2092) Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 2.5.0-SNAPSHOT

2014-05-23 Thread Hitesh Shah (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007291#comment-14007291
 ] 

Hitesh Shah commented on YARN-2092:
---

Isolating apps is just one aspect. The bigger issue is the provisioning of 
thinner client-api jars so that the hadoop internals and their dependencies do 
not need be pulled into the classpath of an app.


 Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 
 2.5.0-SNAPSHOT
 

 Key: YARN-2092
 URL: https://issues.apache.org/jira/browse/YARN-2092
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah

 Came across this when trying to integrate with the timeline server. Using a 
 1.8.8 dependency of jackson works fine against 2.4.0 but fails against 
 2.5.0-SNAPSHOT which needs 1.9.13. This is in the scenario where the user 
 jars are first in the classpath.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2092) Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 2.5.0-SNAPSHOT

2014-05-22 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005718#comment-14005718
 ] 

Steve Loughran commented on YARN-2092:
--

This seems like from the HADOOP-10104 patch.  Which went in because the 2.2+ 
version of jackson was so out of date is was breaking other things.

I'm not sure its so much incompatible as that TEZ is trying to push in its own 
version of jackon, which is then leading to classpath mixing problems. Even if 
you try to push in one set of the JARs ahead of the other, things are going to 
break. I know, I've tried.

jackson 1.x should be compatible at run time with code build for previous 
versions. If there's a link problem there then it's something we can take up 
with the Jackson team. 



 Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 
 2.5.0-SNAPSHOT
 

 Key: YARN-2092
 URL: https://issues.apache.org/jira/browse/YARN-2092
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah

 Came across this when trying to integrate with the timeline server. Using a 
 1.8.8 dependency of jackson works fine against 2.4.0 but fails against 
 2.5.0-SNAPSHOT which needs 1.9.13. This is in the scenario where the user 
 jars are first in the classpath.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2092) Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 2.5.0-SNAPSHOT

2014-05-22 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005804#comment-14005804
 ] 

Steve Loughran commented on YARN-2092:
--

I should add that the underlying issue is that the AM gets then entire CP from 
the {{yarn.lib.classpath}}. That's mandatory to pick up a version of the hadoop 
binaries (and -site.xml files) compatible with the rest of the cluster. But it 
brings in all the other dependencies which hadoop itself relies on. As hadoop 
evolves, this problem will only continue.

The only viable long-term solution is to somehow support OSGi-launched AMs, so 
the AM only gets the org.apache.hadoop classes from the hadoop JARs, and has to 
explicitly add everything itself. See HADOOP-7977 for this -maybe it's 
something we could target for hadoop 3.0 driven by the needs of AMs

 Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 
 2.5.0-SNAPSHOT
 

 Key: YARN-2092
 URL: https://issues.apache.org/jira/browse/YARN-2092
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah

 Came across this when trying to integrate with the timeline server. Using a 
 1.8.8 dependency of jackson works fine against 2.4.0 but fails against 
 2.5.0-SNAPSHOT which needs 1.9.13. This is in the scenario where the user 
 jars are first in the classpath.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (YARN-2092) Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 2.5.0-SNAPSHOT

2014-05-21 Thread Zhijie Shen (JIRA) 

[ 
https://issues.apache.org/jira/browse/YARN-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005335#comment-14005335
 ] 

Zhijie Shen commented on YARN-2092:
---

{code}
2014-05-19 20:09:07,933 FATAL [HistoryEventHandlingThread] 
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread 
Thread[HistoryEventHandlingThread,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError: 
org.codehaus.jackson.map.ObjectMapper.setSerializationInclusion(Lorg/codehaus/jackson/map/annotate/JsonSerialize$Inclusion;)Lorg/codehaus/jackson/map/ObjectMapper;
at 
org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider.locateMapper(YarnJacksonJaxbJsonProvider.java:54)
at 
org.codehaus.jackson.jaxrs.JacksonJsonProvider.writeTo(JacksonJsonProvider.java:488)
{code}

JacksonJsonProvider is in jackson-jaxrs while ObjectMapper is in 
jackson-mapper-asl. If I understand correctly, it looks like the two libs' 
versions don't match.

Hadoop uses the following 4 jackson libs.
{code}
  dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-mapper-asl/artifactId
version1.9.13/version
  /dependency
  dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-core-asl/artifactId
version1.9.13/version
  /dependency
  dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-jaxrs/artifactId
version1.9.13/version
  /dependency
  dependency
groupIdorg.codehaus.jackson/groupId
artifactIdjackson-xc/artifactId
version1.9.13/version
  /dependency
{code}

Given Tez includes all these 4 jars of 1.8.8 in its classpath, either putting 
it before or after Hadoop classpath, there shouldn't be the mismatch. On the 
other side, if Tez includes part of these 4 jars and puts the libs before 
hadoop libs, this problem will occur. Say:

{code}
cp=...:jackson-jaxrs-1.8.8.jar:jackson-xc-1.8.3.jar:jackson-jaxrs-1.9.13.jar:jackson-xc-1.9.13.jar:jackson-mapper-asl-1.9.13.jar:jackson-xc-1.9.13.jar:...
{code}

 Incompatible org.codehaus.jackson* dependencies when moving from 2.4.0 to 
 2.5.0-SNAPSHOT
 

 Key: YARN-2092
 URL: https://issues.apache.org/jira/browse/YARN-2092
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Hitesh Shah

 Came across this when trying to integrate with the timeline server. Using a 
 1.8.8 dependency of jackson works fine against 2.4.0 but fails against 
 2.5.0-SNAPSHOT which needs 1.9.13. This is in the scenario where the user 
 jars are first in the classpath.  



--
This message was sent by Atlassian JIRA
(v6.2#6252)