Just found this issue and wanted to link it here, in case somebody finds this thread later:
https://spark-project.atlassian.net/browse/SPARK-939 On Thursday, March 20, 2014 at 11:14 AM, Matei Zaharia wrote: > Hi Jaka, > > I’d recommend rebuilding Spark with a new version of the HTTPClient > dependency. In the future we want to add a “put the user’s classpath first” > option to let users overwrite dependencies. > > Matei > > On Mar 20, 2014, at 8:42 AM, Jaka Jančar <j...@kubje.org > (mailto:j...@kubje.org)> wrote: > > Could the executor use isolated classloaders, in order not to pullute the > > environment with it's own stuff? > > > > > > > > > > > > On Wednesday, March 19, 2014 at 8:30 AM, Jaka Jančar wrote: > > > > > Hi, > > > > > > I'm getting the following error: > > > > > > java.lang.NoSuchMethodError: > > > org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V > > > at > > > org.apache.http.impl.conn.PoolingClientConnectionManager.createConnectionOperator(PoolingClientConnectionManager.java:140) > > > at > > > org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:114) > > > at > > > org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:99) > > > at > > > org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:85) > > > at > > > org.apache.http.impl.conn.PoolingClientConnectionManager.<init>(PoolingClientConnectionManager.java:93) > > > at > > > com.amazonaws.http.ConnectionManagerFactory.createPoolingClientConnManager(ConnectionManagerFactory.java:26) > > > at > > > com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:96) > > > at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:155) > > > at > > > com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:119) > > > at > > > com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:103) > > > at > > > com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:334) > > > at > > > com.celtra.analyzer.TrackingLogRDD.createClient(TrackingLogRDD.scala:131) > > > at com.celtra.analyzer.TrackingLogRDD.compute(TrackingLogRDD.scala:117) > > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237) > > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:226) > > > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:107) > > > at org.apache.spark.scheduler.Task.run(Task.scala:53) > > > at > > > org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:215) > > > at > > > org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:50) > > > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:182) > > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > at java.lang.Thread.run(Thread.java:744) > > > > > > My app uses the AWS SDK, which requires > > > org.apache.httpcomponents:httpclient:4.2. > > > > > > I believe the error is caused by the fact that an older version of the > > > package is already present on the classpath: > > > > > > Spark -> Akka -> sjson -> org.apache.httpcomponents:httpclient:4.1 > > > Spark -> jets3t -> commons-httpclient:commons-httpclient:3.1 > > > > > > > > > What are my options if I need to use a newer version of the library in my > > > app? > > > > > > Thanks, > > > Jaka > > > > > > > > > > > > > > > > > >
