Dylan1312 edited a comment on issue #8339: Don't inadvertently exclude hadoop-client-core dependency URL: https://github.com/apache/incubator-druid/pull/8339#issuecomment-524797775 Hey @clintropolis, no problem at all and thanks for looking! Happy to bump the milestone on this since we haven't heard of anyone else that's affected. We're using a Cloudera distro of 2.6. I can get a full example of an indexing spec but I think the most pertinent thing that's potentially different is setting mapreduce.job.classloader = true (to avoid a jackson dependency conflict). Guessing that without classloader isolation the missing classes would be present in the hadoop install itself. Example below showing the difference between a build of this branch and our latest release, note hadoop-mapreduce-client-core-2.8.3.jar isn't present within druid-hdfs-storage in the latter. ``` ➜ ls apache-druid-0.16.0-incubating-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-mapreduce* apache-druid-0.16.0-incubating-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-mapreduce-client-app-2.8.3.jar apache-druid-0.16.0-incubating-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-mapreduce-client-common-2.8.3.jar apache-druid-0.16.0-incubating-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-mapreduce-client-core-2.8.3.jar apache-druid-0.16.0-incubating-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-mapreduce-client-jobclient-2.8.3.jar apache-druid-0.16.0-incubating-SNAPSHOT/extensions/druid-hdfs-storage/hadoop-mapreduce-client-shuffle-2.8.3.jar ➜ ls ~/Downloads/apache-druid-0.15.1-incubating/extensions/druid-hdfs-storage/hadoop-mapreduce* /Users/dylan/Downloads/apache-druid-0.15.1-incubating/extensions/druid-hdfs-storage/hadoop-mapreduce-client-app-2.8.3.jar /Users/dylan/Downloads/apache-druid-0.15.1-incubating/extensions/druid-hdfs-storage/hadoop-mapreduce-client-common-2.8.3.jar /Users/dylan/Downloads/apache-druid-0.15.1-incubating/extensions/druid-hdfs-storage/hadoop-mapreduce-client-jobclient-2.8.3.jar /Users/dylan/Downloads/apache-druid-0.15.1-incubating/extensions/druid-hdfs-storage/hadoop-mapreduce-client-shuffle-2.8.3.jar ``` That JAR is present in the hadoop-dependencies folder but as far as I can tell we don't include that folder in the classloader that's set up for the hadoop container.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
