Hi Matt, we don’t have any class path separation in Sqoop at the moment. Hence pretty much “any” version of the library will be loaded depending on which version will appear in the class path first. That is obviously very error prone as the class path can have different order on every cluster. This is very common problem in Hadoop space as Hadoop is not isolating running map reduce jobs from it’s own class path, so solving the problem on Sqoop side might not help unless Hadoop solves the problem first.
Luckily there is a workaround - you can “shade” your version of the library to ensure that it will be always used. I’ve seen several project using maven-shade-plugin to do exactly that. The downside is that your binary will get considerably bigger (as shaded libraries will get copied to your own jar). Jarcec On Sep 12, 2014, at 2:24 AM, Matt Ingenthron <[email protected]> wrote: > Hi all, > > Back before sqoop was contributed to ASF, we’d authored a sqoop connector to > work with Couchbase. Recently, when updating this, I found that we’ve moved > on to Apache httpcore 4.3 while some distros of Hadoop are shipping httpcore > 4.2. So, our choices to update are to either backport, fork the dependency > that uses httpcore 4.3 (since it’s not really needed here) to remove that > functionality or embed the dependency, unless… > > The classloader when running a sqoop job shouldn’t be picking up httpcore > from the common distro? > > Right now, when a job runs it’s clearly picking up the older version. > > What is the intended behavior here? Single classloader for hadoop and when > the sqoop M-R job runs it merely appends the libs the sqoop job requested? > Or something fancier? > > Thanks in advance, > > Matt > > -- > Matt Ingenthron > Couchbase, Inc. >
