Anyone? or is this question nonsensical... and I am doing something fundamentally wrong?
On Mon, Mar 16, 2015 at 5:33 PM, Jacob Abraham <abe.jac...@gmail.com> wrote: > Hi Folks, > > I have a situation where I am getting a version conflict between java > libraries that is used by my application and ones used by spark. > > Following are the details - > > I use spark provided by Cloudera running on the CDH5.3.2 cluster (Spark > 1.2.0-cdh5.3.2). The library that is causing the conflict is commons-net. > > In our spark application we use commons-net with version 3.3. > > However I found out that spark uses commons-net version 2.2. > > Hence when we try to submit our application using spark-submit, I end up > getting, a NoSuchMethodError() > > > Error starting receiver 5 - > > > java.lang.NoSuchMethodError: > org.apache.commons.net.ftp.FTPClient.setAutodetectUTF8(Z)V > > at ZipStream.onStart(ZipStream.java:55) > at > org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:121) > at > org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:106) > at > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:277) > at > org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:269) > > ..... > > > > > > > Now, if I change the commons-net version to 2.2, the job runs fine (expect > for the fact that some of the features we use from the commons-net 3.3 are > not there). > > > > How does one resolve such an issue where sparks uses one set of libraries > and our user application requires the same set of libraries, but just a > different version of it (In my case commons-net 2.2 vs 3.3). > > > I see that there is a setting that I can supply - > "spark.files.userClassPathFirst", but the documentation says that it is > experimental and for us this did not work at all. > > > Thanks in advance. > > > Regards, > > -Jacob > > > > >
