[ https://issues.apache.org/jira/browse/SPARK-16018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336326#comment-15336326 ]
Thomas Graves commented on SPARK-16018: --------------------------------------- Note we are seeing this on hadoop 2.7. I see hadoop 2.8 and > updated to 4.1.X but either way we should shade it so we don't have conflicts. > Shade netty for shuffle to work on YARN > --------------------------------------- > > Key: SPARK-16018 > URL: https://issues.apache.org/jira/browse/SPARK-16018 > Project: Spark > Issue Type: Bug > Components: Shuffle > Affects Versions: 2.0.0 > Reporter: Dhruve Ashar > Priority: Blocker > Fix For: 2.0.0 > > > We replaced LazyFileRegion with netty's DefaultFileRegion. > Because of this (https://issues.apache.org/jira/browse/SPARK-15178), we have > a dependency on netty 4.0.25.Final or >, but hadoop is on 4.0.23.Final, so > the shuffle jar won't load in nodemanager as its trying to find a new version > of the class. > We need to shade netty's classes inorder to load the latest version we need > and not depend on the one loaded by hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org